Text Representation: From Vectors to Context
Represent text as numbers, from one-hot vectors, bags of words and TF-IDF to word2vec, fastText and the contextual embeddings of transformers, and learn to visualise and question what they encode.
Intermediate · Machine Learning · about 17 hours · 5 modules · certificate on completion
Taught by Amiraà Hassàn, Arohà Ngàta, AI Learning Companions.
About this course
Every NLP system has to turn text into numbers, and the choice of representation decides much of what it can learn. This course follows that choice from the simplest vectors to the contextual ones inside today's language models. You'll start with one-hot vectors and see why they can't express similarity, build bags of words and n-gram features, store them sparsely and reproduce scikit-learn's TF-IDF by hand. Then you'll learn dense embeddings by prediction: you'll write skip-gram with negative sampling in PyTorch, train word2vec with gensim and compare it with pre-trained GloVe vectors.
Next you'll handle the words a model has never seen with fastText's character n-grams, and move from words to documents with averaged vectors and Doc2Vec. You'll open up a transformer encoder, learning the idea of self-attention and why positions must be added, and use DistilBERT's contextual vectors to tell the river bank from the high-street bank. Finally you'll map embedding spaces with PCA and t-SNE, test analogies and see where they mislead, and probe the social biases embeddings absorb. Every module has a lab with exact expected output, and the capstone trains word2vec and fastText on financial news and explores what they learned. It builds on NLP Foundations with Python and Corpora and prepares you for Transformers and Large Language Models.
What you'll learn
- Compare one-hot, bag-of-words, TF-IDF and hashed representations, and reproduce scikit-learn's TF-IDF by hand
- Implement skip-gram with negative sampling, and train and evaluate word2vec and fastText embeddings with gensim
- Handle out-of-vocabulary words with character n-gram embeddings, and build document-level representations
- Explain how static and contextual embeddings differ, and use a transformer's contextual vectors to separate word senses
- Visualise embedding spaces with PCA and t-SNE, and interpret similarities, analogies and bias with care
Skills you'll gain
Text vectorisation, TF-IDF, Sparse matrices, Word embeddings, word2vec, fastText, Contextual embeddings, Dimensionality reduction, Embedding evaluation, Bias in embeddings
Tools
Python 3, scikit-learn, PyTorch, gensim, Hugging Face Transformers, Matplotlib, NLTK
Before you start
NLP Foundations with Python and Corpora, or the same skills: Python, NLTK corpora, tokenisation and subword ideas; Statistical NLP: Models, Parsing and Information Retrieval helps but isn't required. You need Python 3.10 or later with scikit-learn, PyTorch (CPU is fine), gensim, transformers 4.46 or later and Matplotlib. The labs download the Brown corpus, GloVe vectors (66 MB) and DistilBERT (about 270 MB), with the commands given.
Modules
Vector space and basic vectorisation
Turn text into vectors a model can use: one-hot word vectors and why they fail, bags of words and n-grams, sparse storage, TF-IDF exactly as scikit-learn computes it, and feature hashing.
- One-hot vectors and their limits (10 min)
- Bags of words and n-gram features (12 min)
- Sparsity and sparse matrices (12 min)
- TF-IDF as a document representation (12 min)
- Hands-on lab: Vectorise a corpus four ways
Word embeddings
Learn dense word vectors by prediction: word2vec's skip-gram and CBOW models, the negative sampling that makes them fast, and GloVe's global co-occurrence objective, trained from scratch and with gensim.
- From counts to learned embeddings (12 min)
- word2vec: skip-gram and CBOW (12 min)
- Negative sampling (14 min)
- GloVe: global co-occurrence statistics (12 min)
- Hands-on lab: Train word2vec from scratch and with gensim
Beyond words and universal representations
Handle words a model has never seen with subword embeddings, the fastText approach, then move from words to whole documents with averaged vectors, Doc2Vec and pre-trained sentence encoders.
- The out-of-vocabulary problem (10 min)
- Character n-grams and fastText (14 min)
- Document vectors: averages, SIF and Doc2Vec (12 min)
- Universal sentence representations (10 min)
- Hands-on lab: fastText, unseen words and document vectors
Transformer anatomy for contextual embeddings
See how transformers give every token a vector that depends on its sentence: the idea of self-attention, why positions must be added, how layers refine meaning, and how contextual vectors tell the senses of a word apart.
- Static versus contextual embeddings (12 min)
- Self-attention, intuitively (12 min)
- Positional encodings (10 min)
- Resolving polysemy with contextual vectors (12 min)
- Hands-on lab: Contextual vectors for an ambiguous word
Visualising and interpreting embeddings
Look inside embedding spaces with care: project them to two dimensions with PCA and t-SNE, measure similarity, test analogies and their pitfalls, and probe embeddings for social bias.
- Principal component analysis (12 min)
- t-SNE and the limits of maps (12 min)
- Similarity, analogies and their pitfalls (14 min)
- Bias in embeddings and responsible use (12 min)
- Hands-on lab: Map and probe an embedding space
Project: Embedding space explorer
Explore how well static embeddings capture a specialised domain: financial news. Train a word2vec model and a fastText model, both skip-gram with negative sampling, on NLTK's Reuters corpus (nltk.download('reuters'), about 1.3 million words), with the same settings so the comparison is fair. Evaluate both on the starter file reuters_analogies.txt, 184 questions in four sections (country and currency, plurals, past tenses and comparatives), and add a section of at least 20 questions of your own from the domain. Show how fastText handles words it never saw, such as the misspellings 'governmnet', 'tradeing' and 'comapny', and where its character n-grams mislead. Then visualise the space with t-SNE and PCA maps of at least five domain clusters, such as currencies, commodities, countries, months and verbs of rising and falling, with notes on what they show and hide. In a reference run with 100 dimensions, a window of 5 and 5 epochs, fastText answered about 37% of the starter questions and word2vec about 27%, with the biggest gap on comparatives; explain the differences you find.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.