AI - Natural Language Processing - Lexical Processing

What is NLP ?

In previous article I was talking about Training Computer Systems or Machines like how we are training kids. The computer system or the Machine should perform the task based on the interaction with the human being and the instructions provided. These interaction can be in the form of inputs which are not precoded and can be given in the form of Text, Speech, Action, Gesture, sound etc., These are part of NLP or the Natural Language Processing and so Machines can not only receive processing instructions in the form of Machine Language but also in the form of Natural Language. Text for of Natural Language contains Words, Grammar, Sentences and also context that are powerful enough to change the meaning and the processing of same can be taken care by Lexical Processing.

What is Lexical Processing ?

Words are the building blocks of comprehension and properties the word represent provides primary data. The retrieval of these data is what Lexical Processing does. In simpler terms Lexical Process - Word Knowledge. The knowledge or process can be further classified into further 3 forms : Meaning of words(lexical semantics), Sound of each words(Speech lexicon), written form (Orthographic lexicon). An NLP engineer aims to make the computer system or the machine to retrieve the data from given comprehension through Lexical Processing and perform text analytic .

Basic Lexical Processing

Lexical processing is based on

Ø Tokenisation
Text into smaller terms. This will help in extracting features from the text targeting text analytic.

Ø Stop Words removal
Stop Words are the ones which are less important in given text. ‘The’,‘is’,’a’,’an’
are few examples of stop words.

Ø Stemming
A rule based technique that cut suffix of a word to get it’s root form called ‘stem’. For example the word “Running” can be reduced to it’s stem form “Run”

Ø Lemmatization
Which is an intelligent technique. It takes the input word do recursive search in the dictionary to identify the base word which is called as Lemma.

After basic lexical processing you will get a list of words called as Bag-of-Words Representation. Frequency of terms in the comprehension weights are assigned and text analytic is performed.

Advanced Lexical Processing

Advanced Lexical Processing deals with difference in spelling , colloquial form of words, terms comprise of more than one words etc., by applying Phonetic Hashing,SoundEx, Edit distance, Pointwise Mutual Information etc.

Each of the above can be discussed in detail in future article.

Lexical Processing - Applications

Lexical Processing is the building block of NLP and is the pre processing tool for text analytic, Machine Learning and AI. Spell corrector, Automatic spell check application are built applying Lexical Processing.

Lexical Processing - Tools available
NLTK The natural Language Tool Kit is utilized widely in Python.
spaCy is a library of Python built on very recent research.
CoreNLP From Stanford supports languages like Arabic, Chinese, German, Spanish, French other than English.
Flair Build by Zalando Research. Has One Model, Many languages concept. Python based.

More tools are available and they include many other NLP functionalities. We will be discussing about them in forthcoming articles.

Note :
Planning for a series of Articles on AI and this is Article no.2
Article no.1 : https://databaseandaiml.blogspot.com/2019/05/artificial-intelligence-like-toddler.html

Comments

  1. Marta HugoOctober 6, 2021 at 6:26 PM

    AI & ML in Dubai
    https://www.nsreem.com/ourservices/ai-ml/
    Artificial intelligence is very widespread today. In at least certainly considered one among its various forms has had an impact on all major industries in the world today, NSREEM is #1 AI & ML Service Provider in Dubai
    1633569987566-9

    ReplyDelete

Post a Comment

Popular posts from this blog

Recover Dropped table - Available Solutions - Oracle Database

Recent discussion was how to recover a dropped table and various factors were touched upon and I thought of sharing my insights which may be useful for someone who is heading a largepool of DBA and Developers. For anyone who ’s into the IT industry and supporting production databases for more than 10 year would have came across various complex, challenging scenarios in BAU support or during database migration as access or performance issues. Table drop that too from production database is not something that would happen everyday and administrators with due diligence are managing databases. But we can’t write-off saying that the scenario never happens. When being reported about a table drop few hours from production database; as an accountable person you may need to update stakeholders who would be directly affected by the incident. Rather than reporting this as an incident, the recommendation is to suggest optimal solutions handy. Scenario : Table dropped from the datab...
Read more

Machine Learning - A High level overview

Image
Machine Learning is automated learning through algorithms by machines or by computer systems. It'd data analytics technique and is a subset of Artificial Intelligence. It is a methodology that makes computer system or machine to learn from experience like how human beings do. In CRISP - DM the Cross-Industry Standard Process framework for Data Mining the fourth stage is the Modeling stage. In the Modeling stage Machine Learning Models are prepared.   And where the Machine Learning models are being used? Machine learning is applied where there is need for computer system to learn and to interpret without precoded programs. Machine Learning Models are used in Predictive analytics. In day today life we are asking Apple ’s Siri or Google Assistant in the mobile phone “Get me the route to Central Library” and we have the route map displayed on the screen from the current location. The mobile phone has recognized the voice and it responded. The process of recognizing vo...
Read more