📘 Lecture 7 — Parsing Documents
📖 Overview: This lecture covers the fundamental techniques for building and using inverted indexes in information retrieval systems. It explains how to efficiently store and compute with sparse document vectors, create inverted indexes, compute cosine similarity, and understand the time complexity of indexing and retrieval operations.
🗂️ Topics Covered
The lecture covers the basic indexing pipeline, sparse vector representations (lists, trees, hashtables), inverted index creation and implementation, computing IDF and document vector lengths, time complexity analysis of indexing, and retrieval with inverted indexes including query efficiency.
📝 Lecture Summary
Basic indexing pipeline
The basic indexing pipeline involves processing documents to create a searchable index. Documents are first parsed to extract tokens, then a sparse vector representation is created where most entries are zero since documents typically contain only a small subset of the total vocabulary. The vocabulary and dimensionality of vectors can be very large (up to 10⁴), so efficient storage methods are needed.
🔑 Definition — Sparse Vector: A vector where most entries are zero, representing document-term weights only for terms that actually appear in the document.
Sparse Vectors as Lists
Linked lists can store vectors as pairs of non-zero-weight tokens paired with their weights. Space is proportional to the number of unique tokens (n) in the document. However, this requires linear search O(n) to find or change the weight of a specific token, and worst-case quadratic time O(n²) to construct the vector.
📐 Formula: O(n²) = n(n+1)/2 ≈ Σᵢ₌₁ⁿ i → The time required grows quadratically with the number of unique tokens.
Sparse Vectors as Trees
Balanced binary trees or tries index tokens with weights stored at the leaves. Space overhead is ~2n nodes (contains data and pointers). This provides O(log n) time to find or update token weights and O(n log n) time to construct the vector, requiring a software package to support such structures.
Sparse Vectors as HashTables
Hash tables store tokens with token string as key and weight as value. Storage overhead is ~1.5n, and the table must fit in main memory. This provides constant time O(1) to find or update token weights (ignoring collisions) and O(n) time to construct the vector.
💡 Why this matters: Hash tables offer the best performance trade-off for in-memory document indexing, but the table must fit entirely in RAM.
Implementation Based on Inverted Files
In practice, document vectors are not stored directly; an inverted organization provides much better efficiency. The keyword-to-document index can be implemented as a hash table, sorted array, or tree-based data structure (trie, B-tree). The critical issue is logarithmic or constant-time access to token information.
Inverted Index
An inverted index maps each token to a list of documents containing that token. Creating one involves:
- Create empty HashMap H
- For each document D: create HashMapVector V, then for each token T in V: if T not in H, create TokenInfo for T and insert it; create TokenOccurrence for T in D and add it to T's occurrence list
- Compute IDF for all tokens
- Compute vector lengths for all documents
🔑 Definition — TokenOccurrence: A record indicating a token's occurrence in a specific document, containing the term frequency and document reference.
Computing IDF
Let N be the total number of documents. For each token T: determine M (number of documents where T occurs, i.e., length of T's occurrence list). The IDF for T is log(N/M). This requires a second pass through all tokens after all documents have been indexed.
📐 Formula: IDF(T) = log(N/M) → Inverse Document Frequency measures how rare or common a term is across the entire document collection.
Cosine Similarity Measure
The document vector length is the square root of the sum of squares of token weights. The weight of a token is TF × IDF. Document lengths cannot be determined until IDFs are known (after all documents are indexed).
📐 Formula: Document length = √(Σ (I × C)²) where I = IDF of token T and C = count of T in document D.
Computing Document Lengths
- Initialize all document vector lengths to 0.0
- For each token T: let I be its IDF; for each TokenOccurrence of T in document D: let C be the count of T in D; increment D's length by (I × C)²
- After processing all tokens, set each document's length to the square root of its accumulated value
Time Complexity of Indexing
- Creating a vector and indexing a document with n tokens: O(n)
- Indexing m documents: O(m × n)
- Computing token IDFs for vocabulary V: O(|V|)
- Computing vector lengths: O(m × n)
- Since |V| ≤ m × n, complete process is O(m × n) — same complexity as reading in the corpus
Retrieval with an Inverted Index
Tokens not present in both query and document do not affect cosine similarity because their weight product is zero and doesn't contribute to the dot product. Since queries are typically short with extremely sparse vectors, the inverted index is used to find only the limited set of documents containing at least one query word.
💡 Why this matters: The inverted index dramatically reduces computation by only considering documents that contain query terms, avoiding the need to scan all documents.
Inverted Query Retrieval Efficiency
The retrieval process:
- Prompt user for query Q
- Compute ranked array of retrievals R for Q
- Print top N documents in R
- User can:
- Show next N retrievals
- Show the Mth retrieved document
- Continue until user enters empty query
⭐ Key Takeaways
The most critical concepts from this lecture are the three sparse vector implementations (lists O(n²) construction, trees O(n log n), and hashtables O(n) with O(1) lookup), the two-pass process for creating an inverted index where IDF computation requires a second pass after all documents are indexed, and the cosine similarity formula where document vector length = √(Σ(TF × IDF)²). The time complexity of full indexing is O(m × n), and retrieval efficiency comes from using the inverted index to only examine documents containing query terms. Understanding the trade-offs between list, tree, and hashtable implementations is essential for choosing the right data structure.
🧠 Quick Revision Questions
- What are the three implementations for sparse vectors and their time complexities for construction?
- Why must IDF computation be done as a second pass after all documents have been indexed?
- What is the formula for computing document vector length in cosine similarity?
- How does an inverted index improve retrieval efficiency compared to scanning all documents?
- What is the overall time complexity of indexing m documents, each with n tokens?
📘 Lecture 8 — Information Retrieval Techniques
📖 Overview: Lecture 8 delves into the foundational steps of the indexing pipeline, focusing on how raw documents are parsed and tokenized into searchable units. It explains the critical handling of numbers, the rationale behind stop words, and the complexities introduced by different document formats, languages, and character sets. Understanding these processes is essential for building any effective information retrieval system.
🗂️ Topics Covered
The lecture begins by explaining the basic indexing pipeline and the process of parsing a document to determine its format, language, and character set. It then discusses complications arising from multi-language documents and the challenge of defining what constitutes a single "document" for indexing. The core topics of tokenization are introduced, including issues with punctuation like apostrophes and hyphens, followed by specific considerations for indexing numbers. The lecture concludes by examining language-specific tokenization challenges (French, German, Chinese, Japanese, Arabic) and the function and modern usage of stop words.
📝 Lecture Summary
Parsing a Document
The first step in the indexing pipeline is to parse the document. This involves identifying its format (e.g., PDF, Word, Excel, HTML), its language (e.g., English, French), and the character set in use (e.g., CP1252, UTF-8). This initial identification is necessary for the correct extraction of text.
💡 Why this matters: A system cannot index a PDF the same way as a Word document, and it cannot process UTF-8 text as if it were ASCII without causing errors.
Complications: Format/Language
Documents being indexed can include content from many different languages, and a single index may contain terms from many languages. A document or its components can contain multiple languages or formats, such as a French email with a German PDF attachment or a French email quoting clauses from an English-language contract. There are commercial and open-source libraries that can handle much of this complexity.
Complications: What is a Document?
When a query is issued returning "documents," there are often interesting questions of grain size: What is a unit document? Is it a single file? An email (perhaps one of many in a single mbox file)? An email with five attachments? A group of files (e.g., a PPT presentation or LaTeX document split over HTML pages)? These tasks are often done heuristically.
Precision and Recall
Precision and recall are two key measures for evaluating the effectiveness of an information retrieval system. Precision measures the proportion of retrieved documents that are relevant (i.e., how many results are correct). Recall measures the proportion of relevant documents that are retrieved (i.e., how many of the correct results you've found).
Tokenization
The input text (e.g., “Friends, Romans and Countrymen”) is processed to output tokens (e.g., Friends, Romans, Countrymen). A token is an instance of a sequence of characters. Each such token is now a candidate for an index entry, after further processing.
🔑 Definition — Token: An instance of a sequence of characters in a particular document that is grouped together as a useful semantic unit for processing.
Issues in tokenization include handling:
- Apostrophes:
Finland's capitalcould becomeFinlandANDs,Finlands, orFinland's. - Hyphens:
Hewlett-Packardcould be one token or two (HewlettandPackard).state-of-the-artcould be broken up.co-educationvslowercase,lower-case,lower case. - Multi-word terms:
San Franciscois one token or two? How is it decided?
Numbers
Numbers present their own tokenization challenges, e.g., 3/20/91, Mar. 12, 1991, 55 B.C., B-52, (800) 234-2333. They often have embedded spaces. Older IR systems may not index numbers, but they are often very useful, for example, looking up error codes or stacktraces on the web. One solution is using n-grams. Metadata (creation date, format, etc.) is often indexed separately.
Tokenization: Language Issues
Different languages present unique tokenization issues:
- French:
L'ensemble— should it be one token or two (Landensemble)? The goal is forl'ensembleto matchun ensemble. - German: Noun compounds are not segmented, e.g.,
Lebensversicherungsgesellschaftsangestellter('life insurance company employee'). German retrieval systems benefit greatly from a compound splitter module, which can give a 15% performance boost. - Chinese and Japanese: These languages have no spaces between words, so tokenization is not always guaranteed and unique. Japanese has the further complication of multiple alphabets intermingled.
- Arabic (or Hebrew): These languages are written right to left, but with certain items like numbers written left to right. Words are separated, but letter forms within a word form complex ligatures.
Stop Words
With a stop list, you add the commonest words (e.g., the, a, and, to, be) to a list and exclude them from the dictionary entirely. The intuition is that they have little semantic content and represent a large volume of postings (~30% of postings for the top 30 words).
🔑 Definition — Stop List: A list of common words that are excluded from the index entirely because they are believed to carry little semantic content.
However, the trend is away from doing this because good compression techniques mean the space for including stop words is very small, and good query optimization techniques mean the cost at query time is low. Stop words are needed for phrase queries (“King of Denmark”), various song titles (“Let it be”, “To be or not to be”), and “relational” queries (“flights to London”).
⭐ Key Takeaways
The indexing pipeline begins with parsing a document to determine its format, language, and character set, which introduces complications when dealing with multi-language content. Tokenization is the process of breaking text into individual tokens, but it is fraught with issues related to punctuation (apostrophes, hyphens), multi-word names, and the specific rules of different human languages (e.g., German compounding, Chinese segmentation). While numbers were often ignored in older systems, they are now considered highly valuable for queries like error code lookups. Finally, the modern approach to stop words is to include them rather than exclude them, as they are essential for phrase and relational queries, and efficient compression and optimization techniques make their inclusion inexpensive.
🧠 Quick Revision Questions
- What are the three primary characteristics of a document that must be identified during the parsing stage before indexing can begin?
- Give two examples of complexities that arise when tokenizing text that contains apostrophes or hyphens.
- Explain the main challenge in tokenizing Chinese text that does not exist in English text.
- What is the classic argument against using a stop list for words like "the" and "a"?
- Why are numbers like
3/20/91or(800) 234-2333considered challenging to tokenize?
📘 Lecture 9 — Terms Normalization
📖 Overview: This lecture focuses on the crucial step of normalizing tokens into standardized terms for effective information retrieval. It explores various normalization techniques, including case folding, handling diacritics, and using thesauri and soundex, to ensure that queries match relevant documents despite variations in word forms, capitalization, and spelling.
🗂️ Topics Covered
The lecture covers normalization of terms for indexing and querying, including case folding to handle capitalization variations, normalization for different languages (accents, umlauts, date forms, character sets), asymmetric expansion as an alternative to equivalence classes, and the use of thesauri for synonyms and soundex for phonetic spelling corrections.
📝 Lecture Summary
Normalization
We may need to “normalize” words in indexed text as well as query words into the same form. We want to match U.S.A. and USA. Tokens are transformed to terms which are then entered into the index. A term is a (normalized) word type, which is an entry in our IR system dictionary. We most commonly implicitly define equivalence classes of terms by deleting periods to form a term (e.g., U.S.A., USA → USA) or deleting hyphens to form a term (e.g., anti-discriminatory, antidiscriminatory → antidiscriminatory).
1. Normalization: other languages
For accents, such as French résumé vs. resume, a simple remedy is to remove the accent, but this is not good in cases like Resume (with and without accent having different meanings) vs. Cliché and Cliche (same meaning). An important consideration is: Are the users going to use accents while writing queries? For umlauts, e.g., German: Tuebingen vs. Tübingen, these should be equivalent. Often, it is best to normalize to a de-accented term (e.g., Tuebingen, Tübingen, Tubingen → Tubingen).
🔑 Definition — Normalization: The process of transforming tokens into a standard form (terms) so that different surface forms of the same word can be matched.
💡 Why this matters: Even in languages that standardly have accents, users often may not type them, so normalization improves recall.
Normalization also applies to things like date forms: e.g., 7月30日 vs. 7/30 (date or mathematical expression). This leads to diversification where 7/30 = 7/30, July 30, 7-30. In Japanese, there are several different character sets, and normalization needs to take care of this fact, resolving queries entered using any char-set. In German, MIT is a word, so the system must differentiate if it refers to the university or the word "with". Tokenization and normalization may depend on the language and is intertwined with language detection. The same method of normalization should be used while indexing as well as while query processing.
2. Case folding
Case folding means reducing all letters to lower case. An exception is upper case in mid-sentence: e.g., General Motors, Fed vs. fed, SAIL vs. sail. It is often best to lower case everything, since users will use lowercase regardless of ‘correct’ capitalization. A word starting with a capital letter in the middle of a sentence is for nouns, so case folding may be given importance in this case. However, if users are not going to use capital letters, then there is no point in improving the index. A longstanding Google example [fixed in 2011] was the query "C.A.T." which returned "#1 result is for 'cats' not Caterpillar Inc."
🔑 Definition — Case folding: The process of converting all letters in a token to a single case (usually lowercase) to create equivalence classes for indexing and querying.
3. Normalization to terms
An alternative to equivalence classing is to do asymmetric expansion. An example: Enter: window → Search: window, windows; Enter: windows → Search: Windows, windows, window; Enter: Windows → Search: Windows. This is potentially more powerful but less efficient. It increases the size of the postings list but gives more control in query processing.
🔑 Definition — Asymmetric expansion: A normalization technique where a query term is expanded to include related forms (e.g., plurals, capitalizations) in an asymmetric way, rather than merging them all into a single equivalence class.
4. Thesauri and soundex
We need to handle synonyms and homonyms. Synonyms are different words with same meanings (e.g., Automobile / Car). Homonyms are same words with different meanings (e.g., Jaguar, Blackberry). For homonyms, postings for all variants are stored against the same index word (term). For synonyms, we can use hand-constructed equivalence classes (e.g., car = automobile, color = colour). We can rewrite to form equivalence-class terms: when a document contains automobile, index it under car-automobile (and vice-versa). Or we can expand a query: when the query contains automobile, look under car as well.
For spelling mistakes (e.g., Chebichev), one approach is Soundex, which forms equivalence classes of words based on phonetic heuristics. It groups words that sound similar into the same equivalence class.
🔑 Definition — Soundex: A phonetic algorithm that indexes words by their pronunciation, forming equivalence classes of words that sound similar, typically used to handle spelling mistakes and variations.
⭐ Key Takeaways
Normalization is critical for matching different surface forms of words, such as "U.S.A." and "USA." Case folding to lowercase is generally recommended, except in special cases like proper nouns. For multilingual contexts, normalization must handle accents, umlauts, date forms, and different character sets. Asymmetric expansion offers more control than simple equivalence classes but is less efficient. Thesauri handle synonyms by creating equivalence classes, while Soundex handles spelling errors by grouping phonetically similar words.
🧠 Quick Revision Questions
- What is the difference between a token and a term in the context of information retrieval normalization?
- Give an example of why simple accent removal might be problematic (i.e., two words with and without accent having different meanings).
- What is the general recommendation for case folding in IR systems, and what is a potential downside?
- Explain how asymmetric expansion differs from creating equivalence classes for normalization.
- What problem does the Soundex algorithm solve, and how does it achieve this?
📘 Lecture 10 — Lemmatization
📖 Overview: This lecture covers two core text normalization techniques in information retrieval: lemmatization and stemming. It explains how these methods reduce words to their base or root forms to improve retrieval, and critically evaluates their effectiveness across different languages and applications.
🗂️ Topics Covered
The lecture introduces Lemmatization as a dictionary-based NLP tool that returns the grammatical base form of a word, and Stemming as a crude affix-chopping process that reduces terms to their roots before indexing. It then details Porter’s algorithm, the most common English stemmer, including its conventions, phases, and typical rules. The lecture also discusses other stemmers like Lovins and Paice/Husk, provides a stemming example, addresses language-specificity of these transformations, and concludes with an analysis of whether stemming/lemmatization actually helps retrieval performance.
📝 Lecture Summary
Lemmatization
Lemmatization is an NLP tool that uses dictionaries and morphological analysis of words in order to return the base or dictionary form of a word. It reduces inflectional or variant forms to their base form. For example, “am, are, is” are all reduced to “be”, while “car, cars, car’s, cars’” all become “car”. The sentence “the boy’s cars are different colors” is lemmatized to “the boy car be different color”. Proper nouns like “Pakistan” remain unchanged. Lemmatization implies doing a “proper” reduction to the dictionary headword form. A key example is the lemmatization of “saw”: it attempts to return either “see” (if the token is a verb) or “saw” (if the token is a noun) depending on the part of speech.
🔑 Definition — Lemmatization: A dictionary-based NLP process that returns the base or dictionary form of a word by considering its morphological analysis and part of speech.
Stemming
Stemming involves reducing terms to their “roots” before indexing. It suggests a crude affix-chopping process and is language-dependent. For example, “automate(s)”, “automatic”, and “automation” are all reduced to “automat”. Similarly, “computation”, “computing”, and “computer” all reduce to “comput”. Unlike lemmatization, stemming does not use a dictionary and may not produce a real word.
🔑 Definition — Stemming: The process of reducing inflected or derived words to their stem, base, or root form, typically by removing affixes.
Porter’s algorithm
Porter’s algorithm is the commonest algorithm for stemming English, and results suggest it is at least as good as other stemming options. It follows conventions and has 5 phases of reductions applied sequentially. Each phase consists of a set of commands. A sample convention is: “Of the rules in a compound command, select the one that applies to the longest suffix.”
Typical rules in Porter include:
- sses → ss: “Processes” becomes “Process”
- ies → i: “Skies” becomes “Ski”; “ponies” becomes “poni”
- ational → ate: “Rotational” becomes “Rotate”
- tional → tion: “national” becomes “nation”
- S → “”: “cats” becomes “cat”
- There are also weight of word sensitive rules, such as: (m>1) EMENT → null (if whatever comes before “ement” has length greater than 1, replace “ement” with null). For example, “replacement” becomes “replac”, but “cement” remains “cement”.
📌 Example: Using Porter’s rule sses → ss, the word “processes” is stemmed to “process”. The rule ies → i stems “ponies” to “poni”. The weight-sensitive rule (m>1) EMENT → null stems “replacement” to “replac” because the stem before “ement” has more than one syllable, but leaves “cement” unchanged.
Other stemmers
Other stemmers that exist include the Lovins stemmer, which is a single-pass, longest suffix removal system with about 250 rules (available at http://www.comp.lancs.ac.uk/computing/research/stemming/general/lovins.htm). There is also the Paice/Husk stemmer and Snowball. Full morphological analysis (lemmatization) is also an option, but it offers at most modest benefits for retrieval.
Language-specificity
The above methods (lemmatization and stemming) embody transformations that are language-specific and often application-specific. These are “plug-in” addenda to the indexing process. Both open source and commercial plug-ins are available for handling these language-specific transformations.
Does stemming/lemmatization help?
For English, the results are very mixed. Stemming helps recall for some queries but harms precision on others. For example, stemming “operative” (dentistry) to “oper”, “operational” (research) to “oper”, and “operating” (systems) to “oper” increases recall but reduces precision. Such normalization is not very useful in English language. However, stemming is definitely useful for Spanish, German, Finnish, and other morphologically rich languages. There are 30% performance gains for Finnish. The reason is that these languages have very clear morphological rules for forming words. Domain-specific normalization may also be helpful, such as normalizing words with respect to their usage in a particular domain.
💡 Why this matters: Understanding when stemming and lemmatization are beneficial versus harmful is critical for designing effective information retrieval systems. The decision to use these techniques depends heavily on the language of the documents and the specific retrieval goals (recall vs. precision).
⭐ Key Takeaways
A student must remember that lemmatization is a dictionary-based NLP process that returns the grammatical base form of a word, requiring part-of-speech knowledge, while stemming is a crude, rule-based affix removal that does not rely on a dictionary. Porter’s algorithm is the most common English stemmer and uses 5 sequential phases of reduction commands with weight-sensitive rules. The effectiveness of these techniques is highly language-specific: they provide limited or mixed benefits for English but significant gains (up to 30%) for morphologically rich languages like Finnish. Finally, these transformations are plug-in components to the indexing process and must be chosen based on the language and application domain.
🧠 Quick Revision Questions
- What is the key difference between lemmatization and stemming in terms of dictionary usage and part-of-speech awareness?
- Give two examples of typical rules in Porter’s algorithm and show how they transform specific words.
- Why does stemming/lemmatization provide a 30% performance gain for Finnish but mixed results for English?
- In the context of Porter’s weight-sensitive rule (m>1) EMENT → null, why does “replacement” become “replac” but “cement” remains “cement”?
- What are the three specific other stemmers mentioned in the lecture besides Porter’s algorithm?
📘 Lecture 11 — Compression
📖 Overview: This lecture addresses the critical need for compression in information retrieval systems, specifically focusing on compressing inverted indexes. It covers techniques for efficient dictionary storage, including dictionary-as-a-string and blocking methods, to reduce memory and disk space usage, thereby improving system performance.
🗂️ Topics Covered
The lecture begins by explaining why compression is essential for inverted indexes, focusing on both the dictionary and postings files. It then explores the drawbacks of a fixed-width array for dictionary storage. The primary techniques covered are dictionary-as-a-string storage and blocking, which are methods to compress the term list and pointers, reducing overall memory footprint. The lecture concludes with references for further study.
📝 Lecture Summary
Why compression for inverted indexes?
Compression is crucial for both the dictionary and the postings file to improve efficiency. For the dictionary, compression makes it small enough to fit in main memory and can even free up space to keep some postings lists in main memory as well. For the postings file(s), compression reduces disk space needed and decreases the time required to read postings lists from disk. Large search engines keep a significant part of the postings in memory, and compression allows them to keep more data in memory, which is faster to access.
Dictionary storage - first cut
A naive approach to dictionary storage is using an array of fixed-width entries. For example, storing 500,000 terms with 28 bytes per term would require 14 MB of space. The lecture notes that fixed-width terms are wasteful because term lengths vary.
🔑 Definition — Fixed-width entries: A storage method where each term entry in the dictionary occupies the same, predetermined amount of memory, regardless of the actual length of the term.
Dictionary-as-a-String
To save space, the dictionary is stored as a (long) string of characters. A pointer to the next word shows the end of the current word. This technique can save up to 60% of dictionary space. The total space for a compressed list is calculated as:
- 4 bytes per term for Freq (frequency)
- 4 bytes per term for pointer to Postings
- 3 bytes per term pointer (to the term string)
- Average 8 bytes per term in term string
📐 Formula: Total Space = 500K terms × (4 bytes + 4 bytes + 3 bytes + 8 bytes) = 500K × 19 bytes = 9.5 MB. The average storage is now 11 bytes per term, compared to 20 bytes without compression.
💡 Why this matters: This technique demonstrates a significant reduction from 20 bytes per term to 11 bytes per term, saving nearly half the dictionary space by eliminating fixed-width waste.
📌 Example: Storing 500,000 terms. Without compression, each term used 28 bytes = 14 MB. With dictionary-as-a-string, it uses 19 bytes per term = 9.5 MB, saving 4.5 MB.
Blocking
Blocking is a technique where pointers are stored to every k-th term in the term string, rather than to every term. The example in the lecture uses k=4. This requires storing term lengths (one extra byte per term). The net effect is that where 3 bytes per pointer were used without blocking, for k=4 pointers, the storage changes from 3 × 4 = 12 bytes to 3 + 4 = 7 bytes for 4 pointers, saving ~0.5 MB.
📐 Formula: Storage for k pointers: Without blocking = k × (bytes per pointer). With blocking = (bytes for one pointer) + k (for term lengths).
📌 Example: For k=4, without blocking: 4 pointers × 3 bytes/pointer = 12 bytes. With blocking: 1 pointer (3 bytes) + 4 term lengths (4 × 1 byte = 4 bytes) = 7 bytes total.
💡 Why this matters: Blocking saves space by reducing the number of pointers needed. Larger k values can save more, but too large a k value is not recommended because it makes accessing terms slower.
Why not go with larger k?
While larger k saves more space, it makes searching for a term slower. With a larger block size, you need to scan more terms within a block to find the correct one, increasing the time for dictionary lookup.
⭐ Key Takeaways
Compression is vital for making inverted indexes efficient by reducing memory and disk usage. The dictionary-as-a-string method is a primary technique that saves significant space by storing terms contiguously with pointers instead of in fixed-width arrays. Blocking further compresses the dictionary by storing pointers only to every k-th term and using term lengths for navigation, though too large a block size can reduce search speed. The primary goal is to fit the dictionary and as many postings lists as possible into main memory to accelerate query processing. Students should understand the trade-off between compression ratio (space savings) and access speed (time).
🧠 Quick Revision Questions
- What are the two main components of an inverted index that benefit from compression, and what are the primary benefits for each?
- In the dictionary-as-a-string approach, what data is stored for each term, and how does this reduce space compared to a fixed-width array?
- How does the blocking technique reduce storage, and what is the role of the "k" parameter?
- What is the drawback of using a very large value for 'k' in blocking?
- Calculate the total dictionary space using dictionary-as-a-string for 250,000 terms, assuming the same per-term byte costs (4 bytes Freq, 4 bytes Postings pointer, 3 bytes term pointer, 8 bytes in term string). Show your calculation.
📘 Lecture 12 — Compression
📖 Overview: This lecture explores compression techniques for information retrieval systems, focusing on reducing storage requirements for dictionaries and postings lists. It covers blocking, front coding, postings compression, and variable byte codes—essential methods for efficient index storage and retrieval.
🗂️ Topics Covered
The lecture covers four main compression techniques: blocking, which stores pointers to every kth term string; front coding, which exploits common prefixes in sorted words; postings compression, which reduces the size of document ID lists by storing gaps; and variable byte (VB) codes, which encode gap values using a continuation bit mechanism.
📝 Lecture Summary
Blocking
Blocking groups term strings into blocks and stores pointers to every kth term string only. For example, with k=4, pointers are stored for every 4th term. This technique requires storing term lengths (1 extra byte) to allow traversal within blocks.
🔑 Definition — Blocking: A compression technique where pointers are stored to every kth term string, reducing dictionary storage by grouping terms into blocks.
📐 Formula: k = block size → pointer stored for every kth term
📌 Example: With k=4 in a sorted dictionary, pointers are stored for terms at positions 1, 5, 9, etc. Each term's length requires 1 extra byte for navigation.
Front Coding
Front coding exploits common prefixes in sorted words by storing only the differences from the previous term. For the last k-1 terms in a block of k, only the suffix after the common prefix is stored.
🔑 Definition — Front coding: A compression technique that stores only the differences between consecutive sorted words, leveraging their long common prefixes.
📐 Formula: Store [length of common prefix][suffix length][suffix characters]
📌 Example: For the sequence "8automata8automate9automatic10automation":
- "automata" is stored fully (length 8)
- "automate" stores only "e" (8 characters common, length difference stored)
- "automatic" stores only "ic" (9 characters common)
- "automation" stores only "ion" (10 characters common)
Postings Compression
The postings file is much larger than the dictionary (factor of at least 10). The goal is to store each posting (docID) compactly. For Reuters (800,000 documents), 32 bits per docID are used with 4-byte integers, or about 20 bits with log₂ 800,000. The aim is to use much less than 20 bits per docID.
🔑 Definition — Postings compression: Techniques to reduce storage requirements for document ID lists by encoding gaps between consecutive docIDs rather than absolute values.
💡 Why this matters: Different terms have vastly different frequencies—rare terms like "arachnocentric" occur in ~1 in a million docs (want ~20 bits), while common terms like "the" occur in virtually every doc (20 bits/posting is too expensive, prefer bitmap).
Postings: Two Conflicting Forces
- Rare terms (e.g., "arachnocentric"): occur in ~1 in a million docs → want log₂ 1M ≈ 20 bits per posting
- Common terms (e.g., "the"): occur in virtually every doc → 20 bits/posting is too expensive → prefer 0/1 bitmap vector
Postings File Entry
Postings are stored in increasing order of docID (e.g., computer: 33, 47, 154, 159, 202...). This ordering allows storing gaps instead of absolute values: 33, 14, 107, 5, 43... Most gaps can be encoded with far fewer than 20 bits.
🔑 Definition — Gap encoding: Storing the differences between consecutive docIDs rather than absolute values, exploiting the fact that most gaps are small.
📌 Example: For docIDs [33, 47, 154, 159, 202], store gaps [33, 14, 107, 5, 43] instead of absolute values.
Variable Byte (VB) Codes
Variable Byte (VB) codes encode gap values G using close to the fewest bytes needed to hold log₂ G bits. The method uses one byte with a continuation bit (c). If G ≤ 127, binary-encode it in 7 bits with c=1. Otherwise, encode lower-order 7 bits first, then additional bytes for higher-order bits, setting c=1 on the last byte and c=0 on others.
🔑 Definition — Variable Byte (VB) codes: A compression scheme that encodes gap values using a variable number of bytes, with 7 bits per byte for data and 1 continuation bit indicating whether more bytes follow.
📐 Encoding Algorithm:
- Start with G (gap value)
- If G ≤ 127: store in 7 bits, set continuation bit to 1
- Else: store lower 7 bits with continuation bit=0, then continue with remaining bits
- Final byte always has continuation bit=1
📌 Example: For gap value G=5:
- 5 ≤ 127, so encode as binary 0000101 in 7 bits
- Set continuation bit to 1
- Result: 1 byte = [1][0000101] = 10000101 (binary)
For a larger value (e.g., G=131):
- First byte: lower 7 bits of 131 (131 mod 128 = 3 → 0000011) with c=0 → 00000011
- Second byte: higher bits (131 ÷ 128 = 1 → 0000001) with c=1 → 10000001
- Result: 2 bytes [00000011, 10000001]
⭐ Key Takeaways
The lecture emphasizes that compression is critical for efficient information retrieval, with postings files being at least 10 times larger than dictionaries. Blocking reduces dictionary storage by grouping terms and storing pointers to every kth term. Front coding exploits common prefixes in sorted words to store only differences. The key insight for postings compression is storing gaps between consecutive docIDs instead of absolute values, as most gaps are small. Variable byte codes provide a practical variable-length encoding using a continuation bit, where values ≤127 fit in one byte and larger values spread across multiple bytes with the last byte marked by continuation bit=1.
🧠 Quick Revision Questions
- How does blocking reduce dictionary storage, and what extra information must be stored for navigation within a block?
- In front coding, how is the term "automatic" stored if the previous term was "automata" (length 8)?
- Why is storing postings as gaps more efficient than storing absolute docIDs, and what are the "two conflicting forces" in postings compression?
- How does variable byte encoding handle a gap value of 200? Show the continuation bits and the resulting byte sequence.
- For a term occurring in 500,000 out of 1,000,000 documents, explain whether gap encoding or bitmap representation would be more efficient and why.
📘 Lecture 13 — Compression
📖 Overview: This lecture explores techniques for compressing postings lists in information retrieval systems. It covers variable byte codes, unary codes, and gamma codes, explaining how these methods reduce storage space for document gap values while maintaining efficient decoding.
🗂️ Topics Covered
The lecture examines variable byte (VB) codes as a byte-aligned compression method using continuation bits, then introduces unary code as a simple representation where numbers are encoded as ones followed by a zero. Gamma codes are presented as bit-level compression using length-offset pairs, with detailed properties and examples. The lecture concludes with compression statistics from the Reuters RCV1 collection.
📝 Lecture Summary
Variable Byte (VB) codes
For a gap value G, variable byte encoding aims to use close to the fewest bytes needed to hold (\log_2 G) bits. The method begins with one byte to store G and dedicates 1 bit as a continuation bit c. If G ≤ 127, it binary-encodes the value in 7 available bits and sets c = 1. If G > 127, the algorithm encodes G’s lower-order 7 bits and uses additional bytes for higher-order bits, repeating the process. At the end, the continuation bit of the last byte is set to 1 (c = 1), while preceding bytes have c = 0.
🔑 Definition — Continuation bit: A dedicated bit in each byte indicating whether more bytes follow (c = 0) or the byte is the last in the sequence (c = 1).
📐 Formula: G encoded in VB → Use 7 bits per byte for data, 1 bit per byte for continuation flag.
📌 Example: For G = 130 (binary 10000010), the lower-order 7 bits are 0000010, and the higher-order bits are 1. First byte: higher-order 1 with continuation bit 0 → 10000001 (c=0). Second byte: lower-order 0000010 with continuation bit 1 → 10000010. Final VB code: 10000001 10000010.
Other variable unit codes use different alignment units: 32 bits (words), 16 bits, or 4 bits (nibbles). Variable byte alignment wastes space with many small gaps; nibbles perform better in such cases. VB codes are used by many commercial/research systems as a good blend of variable-length coding and sensitivity to computer memory alignment. Recent work also exists on word-aligned codes that pack a variable number of gaps into one word.
Unary code
Unary code represents a number n as n 1s followed by a final 0. This method is extremely simple but produces very long codes for large numbers.
🔑 Definition — Unary code: A representation where the integer n is encoded as n consecutive 1 bits terminated by a single 0 bit.
📐 Formula: unary(n) = (n) ones followed by one zero → total of (n+1) bits.
📌 Example: Unary code for 3 is 1110. Unary code for 40 is 40 ones followed by a zero: 11111111111111111111111111111111111111110. Unary code for 80 requires 81 bits. This approach appears impractical for large gaps but forms the foundation for gamma codes.
Gamma codes
Gamma codes provide better compression using bit-level encoding. The code represents a gap G as a pair: length and offset. The offset is G in binary with the leading bit removed. For example, 13 → binary 1101 → offset 101. The length is the number of bits in the offset. For 13 (offset 101), length = 3. The length is encoded using unary code: 1110. The gamma code of 13 is the concatenation of length and offset: 1110101.
🔑 Definition — Gamma code: A prefix-free bit-level encoding where a gap G is represented as the unary-coded length of its offset (binary representation minus leading 1) concatenated with the offset itself.
📐 Formula: gamma(G) = unary(len(offset)) + offset, where offset = binary(G) with leading 1 removed.
📌 Example: For G = 5, binary is 101, offset is 01 (length 2). Unary(2) = 110. Gamma code = 11001. For G = 40, binary is 101000, offset is 01000 (length 5). Unary(5) = 111110. Gamma code = 11111001000.
Gamma code properties
G is encoded using (2 \lfloor \log_2 G \rfloor + 1) bits. The length of offset is (\lfloor \log_2 G \rfloor) bits. The length of length is (\lfloor \log_2 G \rfloor + 1) bits. All gamma codes have an odd number of bits. This encoding is almost within a factor of 2 of the best possible (\log_2 G) bits. The gamma code is uniquely prefix-decodable, like VB, meaning no code is a prefix of another. It can be used for any distribution and is parameter-free.
🔑 Definition — Prefix-decodable: A code where no valid codeword is a prefix of any other valid codeword, enabling unambiguous decoding without separators.
📐 Formula: Bits used for G = (2 \lfloor \log_2 G \rfloor + 1) bits.
📌 Example: For G = 6 (binary 110), offset = 10 (length 2), unary(2) = 110, gamma = 11010 (5 bits). Calculation: (2 \lfloor \log_2 6 \rfloor + 1 = 2 \times 2 + 1 = 5) bits. This matches.
Reuters RCV1
Despite theoretical efficiency, gamma codes are seldom used in practice because machines have word boundaries (8, 16, 32, 64 bits). Operations that cross word boundaries are slower. Compressing and manipulating at the granularity of bits can be slow. Variable byte encoding is aligned and thus potentially more efficient. Regardless of efficiency, variable byte is conceptually simpler at little additional space cost.
RCV1 compression statistics show that index compression techniques allow creation of an index for highly efficient Boolean retrieval that is very space efficient — only 4% of the total size of the collection, or 10-15% of the total size of the text in the collection. However, positional information has been ignored; space savings are less for indexes used in practice, though techniques remain substantially the same.
💡 Why this matters: Understanding the tradeoff between compression ratio and decoding speed (bit-level vs. byte/word-aligned) is crucial for designing practical search engines that balance storage efficiency with query response time.
⭐ Key Takeaways
Variable byte codes use continuation bits to achieve byte-aligned compression of gap values, making them practical for modern systems despite space inefficiency with small gaps. Gamma codes achieve better compression ratios using bit-level encoding with unary length and trimmed binary offset, but suffer from slower decoding due to cross-word-boundary operations. Gamma codes are parameter-free, prefix-decodable, and work for any distribution, yet are seldom used in practice because byte-aligned codes offer better performance. Compression of postings lists can reduce index size to only 4% of the collection size for Boolean retrieval, though this figure increases when including positional information. The choice between compression methods involves a fundamental tradeoff between space efficiency and decoding speed.
🧠 Quick Revision Questions
- How does the continuation bit work in Variable Byte encoding, and what values indicate whether more bytes follow?
- What is the unary code representation for the number 5, and why is this method impractical for large gap values?
- For a gap value G = 25, show step-by-step how to construct its gamma code, including the offset, length, and final concatenated bits.
- According to the gamma code property formula, how many bits are required to encode the gap value G = 100?
- Why are gamma codes seldom used in practice despite their theoretical compression efficiency, and what alternative does the lecture recommend?