Text Preprocessing

Text preprocessing is a crucial step in Natural Language Processing (NLP) that involves transforming raw text into a format suitable for analysis. This process helps improve the performance of machine learning models by cleaning and structuring the data.

• Tokenization: Splitting text into words or phrases.
• Stop Words Removal: Eliminating common words that add little meaning.
• Stemming: Reducing words to their base or root form.
• Lemmatization: Similar to stemming but considers the context and converts words to their meaningful base form.
• Normalization: Standardizing text, including lowercasing and removing punctuation.

3.1. Tokenization

import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize

text = "Hello, world! Welcome to Text Preprocessing."
tokens = word_tokenize(text)
print(tokens)  # Output: ['Hello', ',', 'world', '!', 'Welcome', 'to', 'Text', 'Preprocessing', '.']

3.2. Stop Words Removal

from nltk.corpus import stopwords

nltk.download('stopwords')
stop_words = set(stopwords.words('english'))

filtered_tokens = [word for word in tokens if word.lower() not in stop_words]
print(filtered_tokens)  # Output: ['Hello', 'world', 'Welcome', 'Text', 'Preprocessing', '.']

3.3. Stemming

from nltk.stem import PorterStemmer

ps = PorterStemmer()
stemmed_tokens = [ps.stem(word) for word in filtered_tokens]
print(stemmed_tokens)  # Output: ['Hello', 'world', 'welcom', 'text', 'preprocess', '.']

3.4. Lemmatization

from nltk.stem import WordNetLemmatizer

nltk.download('wordnet')
lemmatizer = WordNetLemmatizer()
lemmatized_tokens = [lemmatizer.lemmatize(word) for word in filtered_tokens]
print(lemmatized_tokens)  # Output: ['Hello', 'world', 'Welcome', 'Text', 'Preprocessing', '.']

3.5. Normalization

normalized_text = ' '.join(stemmed_tokens).lower()
print(normalized_text)  # Output: 'hello world welcom text preprocess .' 

• Always analyze the text data to understand its structure before preprocessing.
• Use lemmatization over stemming when accuracy is crucial.
• Consider language-specific stop words for better results.
• Maintain a balance between data cleaning and preserving the original meaning.