Word Embeddings

In our previous notebook, we built a solid sentiment classifier using logistic regression and simple count vectorizer. While that model performed well by focusing on word counts and phrases, it treated each word largely as an independent entity without understanding its underlying meaning or relationship to other words.

This time, we’ll delve into word embeddings – a technique that represents words as dense vectors, capturing their semantic relationships and context within a large body of text. Our today’s goal is to integrate them into our logistic regression pipeline and see how they will affect our sentiment analysis score.

Data Preparation¶

We are going to use the same dataset as before.

This time, we will do some data preprocessing. For each text sample, we are going to apply a technique called semantic vectorization. Its core idea is to vectorize separate words into a thing called word embedding instead of a simple index.

What is a word embedding? Essentially, it is a mathematical representation of a word (or phrase) as a vector (a numerical array) in a high-dimensional space. These vectors capture the semantic meaning of the word by representing its relationships to other words in a corpus of text. That means that semantically similar words would be close in this high-dimensional space.

This approach requires a separate vectorization model - we could start with a pre-trained model called GoogleNews300. It contains 300-dimensional vectors for 3 million words and phrases.

Let’s see it in action by loading it and comparing the similarities of different words.

That makes some sense, right? Communism has poor relations with cars.
But what do those vectors look like?

Now we need to build a vectorization routine. For each sequence we will perform a simple tokenization, extract embeddings, and then squash them into a single averaged vector.

But why?

The reason is simple - most traditional classifiers (like logistic regression) are fundamentally designed to work with fixed-size, flat feature vectors. They don’t have an inherent mechanism to understand or process sequences of varying lengths or the temporal relationships within those sequences.

Building and Training the Model¶

Our final pipeline will remain almost the same - vectorizer, followed by a logistic regression classifier. Defining a simple cross-validation grid is a nice idea as well - but it will contain only one parameter now (regularisation strength).

The model is ready to be trained. This time, we may use grid search - our parameters matrix is so tiny that we could afford a full hyperplanar parameter search instead of a randomized one.

Result¶

Conclusion¶

The model achieved a final accuracy of 86%. While this demonstrates the basic application of word embeddings, it significantly underperforms compared to the previous approach using a simple count vectorizer with n-grams (which reached 90%).

This suggests that for this specific dataset and task, the simple averaging of word embeddings, which loses word order and contextual nuances, is less effective than a feature representation that explicitly captures local phrases (like n-grams).

This highlights a limitation of simple averaging - it discards crucial word order and local contextual information, which n-grams successfully captured. To overcome this and fully leverage the semantic power of word embeddings without losing sequential context, we need models that can learn to understand the relationships between words in a sentence.