Natural Language Processing with NLTK

1. Introduction

Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. NLTK (Natural Language Toolkit) is one of the most widely-used libraries for NLP in Python. It provides easy-to-use interfaces to over 50 corpora and lexical resources, along with a suite of text processing libraries.

2. Installation

To start using NLTK, you need to install it via pip. Run the following command in your terminal:

pip install nltk

3. Tokenization

Tokenization is the process of breaking a text into individual words, phrases, symbols, or other meaningful elements (tokens). This is often the first step in NLP.

Here's how to perform tokenization using NLTK:

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize

# Sample text
text = "Natural Language Processing with NLTK is fun. Let's learn tokenization!"

# Sentence Tokenization
sentences = sent_tokenize(text)
print(sentences)

# Word Tokenization
words = word_tokenize(text)
print(words)

4. Stemming and Lemmatization

Stemming reduces words to their root form, while lemmatization reduces words to their base or dictionary form. Both are essential for normalization in NLP.

from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer

# Initialize stemmer and lemmatizer
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

word = "running"
print("Stem:", stemmer.stem(word))
print("Lemma:", lemmatizer.lemmatize(word, pos='v'))

5. Part of Speech Tagging

Part of Speech (POS) tagging assigns parts of speech to each word (noun, verb, adjective, etc.). NLTK provides tools to perform POS tagging easily.

nltk.download('averaged_perceptron_tagger')

sentence = "NLTK is a great library for NLP."
tokens = word_tokenize(sentence)
pos_tags = nltk.pos_tag(tokens)
print(pos_tags)

6. FAQ