Statistical NLP: Models, Parsing and Information Retrieval

Classify text with Naive Bayes and logistic regression, build word vectors from counts, tag with hidden Markov models, parse with probabilistic grammars, and rank documents with TF-IDF and BM25.

Intermediate · Machine Learning · about 22 hours · 5 modules · certificate on completion

Taught by Ravi Chandrasekharàn, Yildizè Demirèl, AI Learning Companions.

About this course

Before neural networks, the field learned to process language with probability and counting, and those methods still power search engines, baselines and many production systems. This course teaches them by building them. You'll write a Naive Bayes classifier from scratch and train logistic regression by gradient descent on cross-entropy, then represent words by the contexts they occur in, weighting co-occurrence counts with PPMI and compressing them with SVD into dense vectors.

Next come sequences and structure: smoothing n-gram language models properly, hidden Markov models for part-of-speech tagging with Viterbi decoding, and probabilistic context-free grammars parsed with the CKY algorithm and the inside algorithm. The last module builds the heart of a search engine: an inverted index, Boolean queries, ranking with TF-IDF and BM25, and the precision, recall and mean average precision that tell you whether it works. Every module has a lab on real data, such as the Penn Treebank sample, the Brown corpus and movie-review snippets, with exact expected output, and the capstone combines your tagger and a classifier to test, honestly, whether part-of-speech tags help tell opinion from fact. It builds on NLP Foundations with Python and Corpora and prepares you for Transformers and Large Language Models.

What you'll learn

  • Train and evaluate Naive Bayes and logistic regression text classifiers, implementing both from scratch
  • Build count-based word vectors with PPMI and SVD, and compare words with cosine similarity
  • Smooth n-gram language models and implement Viterbi decoding for hidden Markov model POS tagging
  • Parse sentences with probabilistic context-free grammars and the CKY algorithm
  • Build an inverted index and rank documents with TF-IDF and BM25
  • Evaluate statistical NLP models rigorously with precision, recall, F1, MAP and error analysis

Skills you'll gain

Text classification, Naive Bayes, Logistic regression, Vector semantics, N-gram language models, Hidden Markov models, Viterbi algorithm, Probabilistic parsing, Information retrieval, Evaluation and error analysis

Tools

Python 3, NLTK, NumPy, SciPy, scikit-learn

Before you start

NLP Foundations with Python and Corpora, or the same skills: Python, NLTK corpora, tokenisation and a bigram language model. Conditional probability, Bayes' rule and logarithms help (Statistics for Data Science and AI covers them), as does basic NumPy. You need Python 3.10 or later with NLTK, NumPy, SciPy and scikit-learn; the labs use small NLTK datasets, about 6 MB in all, and give the download commands.

Modules

  1. Text classification

    Classify documents with two classic models: Naive Bayes, which models how each class generates words, and logistic regression, which learns feature weights directly by minimising cross-entropy, and evaluate both properly.

    • Generative and discriminative classifiers (12 min)
    • Multinomial Naive Bayes for text (14 min)
    • Logistic regression: features, weights and the sigmoid (12 min)
    • Training with cross-entropy, and evaluating classifiers (14 min)
    • Hands-on lab: Naive Bayes from scratch, then logistic regression

  2. Vector semantics and embeddings

    Represent words by the company they keep: build term-document and word-context matrices, weight them with PPMI, compare words with cosine similarity, and compress the counts into dense vectors with SVD, the bridge to learned embeddings.

    • The distributional hypothesis and co-occurrence matrices (12 min)
    • Weighting with PMI and PPMI (12 min)
    • Cosine similarity and nearest neighbours (12 min)
    • Dense vectors from counts: SVD and latent semantic analysis (12 min)
    • Hands-on lab: Word vectors from co-occurrence counts

  3. Markov models and POS tagging

    Model sequences with the Markov assumption: smooth n-gram language models properly, then build hidden Markov models for part-of-speech tagging, decode them with the Viterbi algorithm and handle words never seen in training.

    • N-gram language models and smoothing (14 min)
    • Hidden Markov models: transitions and emissions (12 min)
    • Viterbi decoding (14 min)
    • Unknown words and evaluating taggers (12 min)
    • Hands-on lab: An HMM part-of-speech tagger with Viterbi

  4. PCFGs and probabilistic parsing

    Describe sentence structure with context-free grammars, parse efficiently with the CKY algorithm, choose between ambiguous parses with probabilistic grammars, and see how the inside-outside algorithm learns rule probabilities.

    • Context-free grammars and constituency (12 min)
    • Chomsky normal form and the CKY algorithm (14 min)
    • Probabilistic CFGs: choosing the best parse (12 min)
    • The inside-outside algorithm, in outline (12 min)
    • Hands-on lab: Probabilistic CKY parsing

  5. Information retrieval foundations

    Build the core of a search engine: an inverted index for Boolean queries, ranked retrieval with TF-IDF in the vector space model and with BM25, and the measures that tell you whether the results are any good.

    • Boolean retrieval and the inverted index (12 min)
    • Ranked retrieval: TF-IDF and the vector space model (12 min)
    • BM25: a better term weighting (12 min)
    • Evaluating search: precision, recall, F1 and ranked metrics (14 min)
    • Hands-on lab: Build and evaluate a small search engine

Project: Integrated POS tagger and text classifier

Build a two-stage NLP system and find out honestly whether the first stage helps the second. Train your hidden Markov model tagger, with smoothing, unknown-word classes and Viterbi decoding, on the Penn Treebank sample with universal tags, and report its accuracy on held-out sentences. Then tag every sentence of NLTK's subjectivity dataset (5,000 subjective review snippets and 5,000 objective plot sentences; nltk.download('subjectivity')) and train logistic regression classifiers to tell the two apart with three feature sets: words only, part-of-speech unigrams and bigrams only, and both. Compare them on one stratified test split with accuracy, precision, recall, F1 and confusion matrices, then analyse the errors: which sentences does each model get wrong, and do the tags capture anything the words don't, such as adjectives, first- and second-person pronouns or past-tense narration? In a reference run with an 80/20 split, words alone reached about 91% accuracy, tags alone about 73% and the combination no better than words, so don't assume the tags will help: a clear answer backed by evidence is what earns the marks.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free