π Lecture 14 β Index Constructions
π Overview: This lecture addresses the critical challenge of scaling index construction for very large document collections. It explores hardware constraints, memory hierarchy, and disk access mechanics, then introduces distributed computing approaches like MapReduce to overcome the limitations of in-memory indexing.
ποΈ Topics Covered
The lecture covers scaling index construction for large collections, the memory hierarchy and Mooreβs Law, hard disk tracks, sectors, and blocks, disk access time components, hardware basics including memory vs. disk access speeds and block-based I/O, inverted index structure, and hardware assumptions for distributed computing systems.
π Lecture Summary
Scaling index construction
In-memory index construction does not scale for very large collections because you cannot fit the entire collection into memory, sort it, and then write it back. To construct an index for very large collections, we must take into account hardware constraints such as memory, disk, and speed.
π Definition β Scaling index construction: The process of building an inverted index for collections too large to fit entirely in main memory, requiring disk-based and distributed methods.
Memory Hierarchy
The memory hierarchy organizes storage by speed and capacity, from fastest/smallest (registers, cache) to slowest/largest (hard disk, tape). Access times vary dramatically: registers (~1 ns), cache (~2 ns), main memory (~10 ns), and hard disk (~10 ms). This hierarchy dictates that algorithms must minimize slow disk accesses.
π Definition β Memory hierarchy: A structured ordering of storage levels by speed, cost, and capacity, where each level acts as a cache for the next slower level.
Mooreβs Law
Mooreβs Law observes that the number of transistors on a microchip doubles approximately every two years, leading to exponential growth in processing power and memory capacity. However, disk seek times have not improved at the same rate, creating a growing performance gap between computation and data access.
π Definition β Mooreβs Law: The empirical observation that transistor density on integrated circuits doubles roughly every two years, driving exponential growth in computing capability.
Hard Disk Tracks and Sectors
Hard disks store data on platters divided into concentric tracks (rings) and sectors (arcs within tracks). The disk head must move to the correct track (seek) and wait for the correct sector to rotate under it (rotational delay) before data can be read or written.
π Definition β Track: A concentric circular path on a hard disk platter where data is stored. π Definition β Sector: A subdivision of a track, typically 512 bytes, representing the smallest addressable unit of data on a disk.
Hard Disk Blocks
Operating systems group sectors into larger blocks (typically 8KB to 256KB) for efficient I/O. Reading and writing occur in whole blocks, meaning even if you need only a few bytes, the entire block is transferred. Block-based I/O is fundamental to disk performance optimization.
π Definition β Block: The minimum unit of data transferred between disk and memory during a single I/O operation, consisting of multiple sectors.
Disk Access Time
The total time to read data from disk is: Access time = Seek time + Rotational delay + Transfer time. Seek time is the time to move the disk head to the correct track. Rotational delay is the wait for the right sector to spin under the head. Transfer time is the time to actually read and transfer the data.
π Formula: Disk Access Time = Seek Time + Rotational Delay + Transfer Time β Total time from issuing a read request to having data available in memory. π‘ Why this matters: Seek time (typically 5-10 ms) dominates access time, so minimizing seeks is critical for performance.
Hardware basics
Access to data in memory is much faster than access to data on disk. During disk seeks, no data is transferred while the head is being positioned. Therefore, transferring one large chunk of data is faster than many small chunks. Disk I/O is block-based, reading/writing entire blocks (8KB to 256 KB) rather than individual bytes.
π Definition β Block-based I/O: The property that disk controllers read and write data in fixed-size blocks, not arbitrary byte ranges.
Inverted Index
The inverted index is the core data structure for information retrieval, mapping each term to a list of documents (postings list) containing that term. For large collections, the index itself can be many gigabytes, requiring disk-based construction and storage.
π Definition β Inverted index: A data structure associating each term in a collection with the list of documents where that term appears, enabling fast full-text search.
Hardware basics (Continued)
Servers used in IR systems typically have several GB of main memory, sometimes tens of GB. Available disk space is several orders of magnitude larger (2β3 orders). Fault tolerance is very expensive, so it is much cheaper to use many regular machines rather than one fault-tolerant machine.
π Definition β Fault tolerance: The ability of a system to continue operating properly in the event of a failure of some of its components.
Distributed computing
Distributed computing is a field of computer science that studies distributed systems, where components located on networked computers communicate and coordinate their actions by passing messages. For large-scale indexing, distributed systems allow splitting the work across many machines.
π Definition β Distributed computing: A model in which components of a software system run on multiple networked computers, coordinating through message passing to achieve a common goal.
Hardware assumptions
For efficient indexing, reasonable hardware assumptions include: main memory of several GB, disk space of several TB, multiple CPU cores, and network connectivity for distributed processing. Key papers on the topic include the original MapReduce publication (Dean and Ghemawat, 2004) and SPIMI (Heinz and Zobel, 2003).
π Definition β MapReduce: A programming model for processing large datasets in parallel across distributed clusters, using map and reduce operations.
β Key Takeaways
The most critical concepts from this lecture are that in-memory index construction fails for large collections due to memory limitations, and understanding the memory hierarchy is essential since disk access is millions of times slower than memory. Disk access time is dominated by seeks, making block-based I/O and large sequential transfers critical for performance. For very large collections, distributed computing with approaches like MapReduce provides scalability, and it is more cost-effective to use many regular machines than one fault-tolerant supercomputer.
π§ Quick Revision Questions
- Why does in-memory index construction fail for very large document collections?
- What are the three components of disk access time, and which one typically dominates?
- Why is transferring one large chunk of data from disk faster than many small chunks?
- What is the advantage of using many regular machines over one fault-tolerant machine in distributed indexing?
- What does the acronym SPIMI stand for, and what problem does it solve?
π Lecture 15 β Merge Sort
π Overview: This lecture covers the fundamental sorting algorithm Merge Sort and its application in information retrieval for inverted index construction. It introduces two key indexing strategiesβSingle-pass in-memory indexing (SPIMI) and its block-based variantβthat enable efficient processing of large document collections on a single machine.
ποΈ Topics Covered
The lecture explains the Two-Way Merge Sort algorithm with an example, introduces Single-pass in-memory indexing (SPIMI) as an alternative to traditional sorting-based methods, details the SPIMI-Invert process for generating per-block inverted indexes, and discusses merging these blocks into a single global index. It concludes by comparing SPIMI with Block Sort-Based Indexing (BSBI) and outlining future scalability challenges.
π Lecture Summary
Two-Way Merge Sort
A sorting algorithm that repeatedly divides a list into two halves, recursively sorts each half, then merges the two sorted halves into one sorted list. In information retrieval, this is used to sort postings during index construction.
π Definition β Merge Sort: A divide-and-conquer algorithm that splits a list, sorts each half recursively, and merges the sorted halves. π Formula: split list β sort left half β sort right half β merge sorted halves β produces sorted list. π Example: Given list [3, 1, 4, 2], split into [3, 1] and [4, 2]; sort each to [1, 3] and [2, 4]; merge to [1, 2, 3, 4].
Single-pass in-memory indexing (SPIMI)
A memory-efficient indexing technique that processes document blocks one at a time without requiring a global term-termID mapping.
π Definition β SPIMI (Single-Pass In-Memory Indexing): An indexing method that builds separate inverted indexes for each block of documents, using per-block dictionaries and no sorting of postings. π Key Idea 1: Generate separate dictionaries for each block β no need to maintain term-termID mapping across blocks. π Key Idea 2: Donβt sort. Accumulate postings in postings lists as they occur. π‘ Why this matters: These two ideas allow generating a complete inverted index for each block with minimal memory overhead.
SPIMI-Invert
The process of creating per-block inverted indexes using SPIMIβs dictionary-per-block approach, then merging these indexes into one global inverted index.
π Definition β SPIMI-Invert: The algorithm that builds an inverted index for each document block in a single pass, then merges all block indexes into a single index. π Example: Process block 1 β generate index A; process block 2 β generate index B; merge A and B into global index. π Note: Merging of blocks is analogous to BSBI (Block Sort-Based Indexing).
β Key Takeaways
Students must remember that SPIMI eliminates the need for global term-termID mapping by creating per-block dictionaries, and it avoids sorting by accumulating postings as they occur, making it more memory-efficient than traditional methods. The key difference from BSBI is that SPIMI builds indexes without sorting within blocks, while BSBI sorts postings per block. Both approaches produce separate block indexes that are later merged into one global inverted index. Understanding these tradeoffs is critical for designing scalable indexing solutions. The lecture concludes that these methods work when data fits on a single machine; larger collections require distributed approaches like MapReduce.
π§ Quick Revision Questions
- What are the two key ideas that make Single-pass in-memory indexing (SPIMI) efficient?
- How does SPIMI-Invert differ from Block Sort-Based Indexing (BSBI)?
- Why does SPIMI not need to maintain a global term-termID mapping across blocks?
- What is the analogy between merging blocks in SPIMI and the merging step in BSBI?
- When does SPIMI become insufficient, and what alternative approaches are needed?
π Lecture 16 β Phrase queries
π Overview: This lecture covers techniques for handling phrase queries in information retrieval, which require exact matches of word sequences. It introduces biword indexes, extended biwords, and positional indexes as solutions, while discussing their trade-offs in precision and index size. Understanding these approaches is critical for building search engines that can handle nuanced user queries.
ποΈ Topics Covered
The lecture begins by categorizing types of queries (phrase, proximity, wild card), then focuses on phrase queries and the limitations of simple term-document indexes. It presents the biword index approach for two-word phrases and its extension to longer phrases, followed by extended biwords using part-of-speech tagging to reduce false positives. Finally, it introduces positional indexes as a more robust solution, storing term positions within documents for precise phrase matching.
π Lecture Summary
Types of Queries
Three main query types are discussed:
- Phrase Queries: Require exact word order, e.g.,
"The crops in pakistan"must appear as a contiguous sequence. A sentence like βI went to university at Stanfordβ would not match the phrase query"Stanford university". - Proximity Queries: Specify distance constraints between terms, e.g.,
LIMIT! /3 STATUTE /3 FEDERAL /2 TORTβ meaning βLIMITβ within 3 words of βSTATUTEβ, etc. - Wild Card Queries: Use asterisks for pattern matching, e.g.,
Results*matches βResultsβ, βResults-basedβ, etc.
π‘ Why this matters: Simple term-document indexes only tell you if a word appears in a document, not where. Phrase queries require positional information to distinguish βI went to university at Stanfordβ from βStanford Universityβ.
Phrase queries
The fundamental problem: a standard inverted index storing <term : docs> pairs cannot verify exact phrase matches. For a query like "Stanford university", we need to ensure the two terms appear adjacent and in that order within the document. Storing only document IDs is insufficient.
A first attempt: Biword indexes
To handle two-word phrases, the lecture introduces biword indexes, which index every consecutive pair of terms in the text as a single phrase unit.
- Example: For the text βFriends, Romans, Countrymenβ, the biwords generated are:
friends romansromans countrymen
- Each biword becomes a dictionary term, so two-word phrase queries can be answered immediately by looking up the biword.
π Definition β Biword Index: An index where dictionary entries are consecutive word pairs (bigrams) from the text, allowing direct lookup of two-word phrases.
Longer phrase queries
For longer phrases like "stanford university palo alto", the query is processed by breaking it into a Boolean conjunction of overlapping biwords:
"stanford university" AND "university palo" AND "palo alto"- Important caveat: Without checking the original document positions, we cannot guarantee that a document matching this Boolean query actually contains the full phrase. False positives occur if the biwords appear far apart in different parts of the text.
π Example: A document containing βstanford university has a campus in palo altoβ would match all three biwords but not the exact phrase "stanford university palo alto" β itβs a false positive.
Extended biwords
To reduce false positives, extended biwords use part-of-speech tagging (POST) to preprocess text:
- Parse the indexed text and perform part-of-speech tagging.
- Bucket terms into (say) Nouns (N) and articles/prepositions (X).
- Deem any string of terms of the form NX*N (Noun, zero or more Xβs, Noun) to be an extended biword.
- Each extended biword becomes a dictionary term.
- Examples:
catcher in the rye(N X X N β extended biword);Capital of Pakistan(N X N β extended biword)
Query processing:
- Given a query, parse it into Nβs and Xβs
- Segment query into enhanced biwords
- Look up index
- Issues:
- Parsing longer queries into conjunctions, e.g.,
tangerine trees and marmalade skiesis parsed intotangerine trees AND trees and marmalade AND marmalade skies - False positives still occur (as with biwords)
- Index blowup due to larger dictionary (more terms)
- Parsing longer queries into conjunctions, e.g.,
Positional indexes
A more robust solution: positional indexes store term positions within documents. For each term, the index records:
<number of docs containing term; doc1: position1, position2 ... ; doc2: position1, position2 ... ; etc.>
- This allows precise verification of phrase order and adjacency by comparing position lists.
- Example: For term βStanfordβ in doc1 at positions [5, 10] and term βuniversityβ in doc1 at positions [6, 11], a query for
"Stanford university"matches because positions 5 and 6 are consecutive.
π Definition β Positional Index: An inverted index where for each term, we store a list of document IDs and, for each document, a list of word positions where the term occurs.
Resources: The lecture references MG 3.6, 4.3; MIR 7.2; and a paper by Williams, Zobel, and Bahle on fast phrase querying.
β Key Takeaways
- Simple term-document indexes cannot handle phrase queries because they lack word position information; storing only document IDs leads to false positives for multi-word phrases.
- Biword indexes provide a direct but limited solution for two-word phrases, but for longer phrases require Boolean conjunctions that still produce false positives.
- Extended biwords use part-of-speech tagging to create meaningful multi-word units (NX*N pattern), reducing but not eliminating false positives, at the cost of a larger dictionary.
- Positional indexes are the most accurate method for phrase queries, storing exact word positions per document, enabling precise verification of adjacency and order.
- The trade-off between index size and query precision is central: biword indexes are compact but less accurate, while positional indexes are larger but more powerful for proximity and phrase queries.
π§ Quick Revision Questions
- Why canβt a simple
<term : list of docs>inverted index correctly answer the query"catcher in the rye"? - How does a biword index represent the phrase βFriends, Romans, Countrymenβ?
- What is the main limitation of using biword indexes for a query like
"stanford university palo alto"? - In extended biwords, what pattern of part-of-speech tags defines a valid extended biword? Provide an example.
- What additional information does a positional index store compared to a standard inverted index, and how does this solve the phrase query problem?
π Lecture 17 β Processing a Phrase Query
π Overview: This lecture covers the processing of phrase queries and proximity queries using positional indexes. It explains how to merge inverted index entries to find exact phrase matches and adapts this method for proximity searches where terms must appear within a specified word distance.
ποΈ Topics Covered
The lecture covers processing a phrase query by extracting and merging inverted index entries, proximity queries using /k notation, positional index size considerations, combination schemes that blend biword and positional indexes, and a brief introduction to wildcard queries.
π Lecture Summary
Processing a phrase query
To process a phrase query like "to be or not to be", extract the inverted index entries for each distinct term: to, be, or, not. Then merge the doc:position lists to enumerate all positions where the phrase appears consecutively. For example, the entry for "to" might include: 2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191; ... and for "be": 1:17,19; 4:17,191,291,430,434; 5:14,19,101; ... The same general method is used for proximity searches.
π‘ Why this matters: This technique allows retrieval systems to find exact phrase matches, which is essential for search queries that require specific word order.
Proximity queries
A proximity query uses the /k notation, meaning "within k words of." For example, LIMIT! /3 STATUTE /3 FEDERAL /2 TORT requires the words to appear within the specified distances. Positional indexes can handle such queries naturally, whereas biword indexes cannot. The exercise is to adapt the linear merge of postings to handle proximity queries for any value of k.
π‘ Why this matters: Proximity queries enable flexible matching where words must be near each other but not necessarily adjacent, which is useful for legal or medical document searches.
Positional index size
Positional indexes can compress position values/offsets similarly to how documents were compressed in previous lectures. However, this expands postings storage substantially because an entry is needed for each occurrence, not just once per document. Index size depends on average document size: the average web page has less than 1000 terms, while SEC filings, books, and epic poems can easily reach 100,000 terms. For a term with frequency 0.1%, the positional index grows proportionally.
Rules of thumb
As a rule of thumb, the positional index size factor is 2-4 times larger than a non-positional index. It typically accounts for 35-50% of the volume of the original text. These figures hold for "English-like" languages.
Combination schemes
A positional index expands postings storage because it stores every occurrence. However, biword indexes and positional indexes can be profitably combined. For particular phrases like βMichael Jacksonβ or βBritney Spears,β it is inefficient to keep merging positional postings lists. This inefficiency is even more pronounced for phrases like βThe Who,β where common words appear frequently.
π‘ Why this matters: Combining both index types allows the system to handle common phrases efficiently while still supporting general proximity queries.
Wild Card Queries
Wildcard queries allow pattern matching using wildcards. Examples include: Stan* matches Standard, Stanford; ST matches Start; ion matches Option; Paan matches Pakistan; Pat*an matches Pakistan, etc.
π‘ Why this matters: Wildcard queries enable users to search for terms when they are unsure of the exact spelling or want to find all variations.
β Key Takeaways
You must remember the three key methods for handling phrase and proximity queries: positional indexes store every word occurrence and allow phrase and proximity matching; biword indexes are efficient for common phrases but cannot handle general proximity; and combination schemes merge both approaches for optimal performance. Positional indexes are 2-4 times larger than non-positional ones and occupy 35-50% of the original text volume. Wildcard queries expand search flexibility by allowing pattern matching on unknown or variable terms.
π§ Quick Revision Questions
- How do you process a phrase query like "to be or not to be" using inverted index entries?
- What does the /k notation mean in a proximity query, and how does it differ from exact phrase matching?
- Why are positional indexes significantly larger than non-positional indexes?
- What is the advantage of combining biword indexes with positional indexes?
- How do wildcard queries like "Stan*" or "Patan" work in terms of pattern matching?
π Lecture 18 β Information Retrieval Techniques (Wild Card Queries & B-Tree)
π Overview: This lecture focuses on techniques for handling wild-card queries in information retrieval systems, covering how to search for word prefixes, suffixes, and complex patterns. It also introduces the B-Tree and B+ Tree data structures that support efficient lexicon lookup.
ποΈ Topics Covered
The lecture covers wild-card query handling using B-Trees, Permuterm Index, K-Grams, and Soundex Algorithms. It explains how to handle prefix queries like "mon*" and suffix queries like "*mon", and describes the B-Tree and B+ Tree structures used for efficient term retrieval.
π Lecture Summary
How to Handle Wild-Card Queries
Wild-card queries allow users to search for terms when they only know part of the word. The lecture outlines several approaches: B-Trees, Permuterm Index, K-Grams, and Soundex Algorithms. These techniques enable the system to find all matching terms in the lexicon efficiently.
π Definition β B-Tree: A self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time.
Wild-card queries: *
Wild-card queries use the asterisk () as a placeholder for any sequence of characters. The query mon finds all documents containing any word beginning with "mon". This is easy to handle with a sorted lexicon: retrieve all words in the range mon β€ w < moo.
However, the query *mon β finding words ending in "mon" β is harder. The solution is to maintain an additional B-tree for terms stored backwards. This allows retrieval of all words in the range nom β€ w < non.
π Formula: For prefix query mon* β range search: mon β€ w < moo
For suffix query *mon β reverse term to nom β range search: nom β€ w < non
π Example: To enumerate all terms meeting the wild-card query pro*cent:
- Split the query into two parts:
pro*and*cent - Use the forward B-tree to find all terms starting with
pro(range:pro β€ w < prp) - Use the backward B-tree to find all terms ending with
cent(reverse totnec, range:tnec β€ w < tned) - Intersect the two result sets to find terms satisfying both conditions
π‘ Why this matters: This approach allows efficient retrieval even for complex wild-card patterns, which is essential for search engines and database systems.
B-Tree
The B-Tree is a fundamental data structure for lexicon storage. It maintains terms in sorted order and supports efficient range queries. The slides show a B-Tree structure where each node contains multiple keys and pointers, allowing fast insertion, deletion, and search operations.
B+ Tree
The B+ Tree is a variant of the B-Tree where all data is stored in the leaves, and internal nodes only contain keys for routing. This structure is particularly efficient for range queries and sequential access, as all leaf nodes are linked together.
Wild-card queries example
The lecture provides an example of the B-Tree structure for handling wild-card queries. It also references resources for further reading:
- IIR 3, MG 4.2 for information retrieval concepts
- K. Kukich (1992) on automatic text correction
- J. Zobel and P. Dart (1995) on approximate matching in large lexicons
- Peter Norvig's guide on how to write a spelling corrector
β Key Takeaways
For wild-card queries, prefix searches (mon) are straightforward using range queries on a forward B-tree, while suffix searches (mon) require a separate backward B-tree. Complex queries like procent* require intersecting results from both forward and backward trees. The B-Tree and B+ Tree are the primary data structures for efficient lexicon storage and retrieval, supporting logarithmic-time searches and range queries. Understanding these techniques is critical for implementing search engines and database systems that handle partial-match queries.
π§ Quick Revision Questions
- How would you handle a wild-card query like "comp*" using a B-Tree?
- What is the key difference between handling prefix queries and suffix queries?
- How do you enumerate all terms matching the wild-card query "pro*cent"?
- What is the advantage of storing terms in a backward B-tree for suffix queries?
- How does a B+ Tree differ from a standard B-Tree in terms of data storage?
π Lecture 19 β Permuterm Index, k-gram, Soundex
π Overview: This lecture covers techniques for handling wildcard queries in information retrieval systems. It introduces the permuterm index and k-gram indexes as methods for efficient wildcard query processing, and concludes with Soundex, a phonetic algorithm for matching names. Understanding these techniques is essential for building robust search systems that handle imprecise queries.
ποΈ Topics Covered
The lecture covers three main topics: the Permuterm index, which uses a specialized dictionary structure for wildcard queries; k-gram (specifically bigram) indexes, which offer a more space-efficient alternative; and the Soundex algorithm, which matches terms based on their phonetic similarity rather than exact spelling.
π Lecture Summary
Permuterm index
The Permuterm index is a technique for handling wildcard queries in information retrieval. It works by creating a special index of all rotations (permutations) of each term in the dictionary, where a special symbol $ marks the end of a word. For example, the word hello generates rotations: hello$, ello$h, llo$he, lo$hel, o$hell, and $hello. To process a wildcard query like hel*o, you rotate the query to o$hel* and look up o$hel* in the permuterm index to find matches like hello$. This technique can handle both single and multiple wildcard characters, but it is space-intensive because the permuterm index can be roughly four times larger than the original dictionary.
π Definition β Permuterm index: A wildcard query processing technique that indexes all rotations of dictionary terms, using a special end-of-word marker, to allow wildcard queries to be executed as exact-match lookups.
π Example: For the query mon*, the system rotates the query to mon* and looks up all entries starting with mon in the permuterm index. It would find terms like moon and money.
Bigram (k-gram) indexes
A k-gram index is an alternative to the permuterm index for processing wildcard queries. It indexes all contiguous sequences of ( k ) characters (substrings) from each dictionary term. For example, with k=2 (bigrams), the term moon generates bigrams: $m, mo, oo, on, n$. To process a query like mon*, the system converts the query to bigram constraints: $m AND mo AND on, then finds dictionary terms that contain all three bigrams. This is more space-efficient than the permuterm index but requires a post-filtering step to remove false positives (e.g., moon would be found but must be checked against the original query). Typically, bi-grams (k=2) or tri-grams (k=3) are used; larger k-grams reduce flexibility in query processing.
π Definition β k-gram index: A wildcard query processing technique that indexes all substrings of length ( k ) from dictionary terms to find candidate terms that satisfy bigram constraints.
π Formula: For a query mon*, the bigram query becomes: $m AND mo AND on β these are all bigrams from the query, with $ as start-of-word marker.
π Example: Query mon* is processed by:
- Break the query into bigrams:
$m,mo,on - Look up the posting list for each bigram
- Find terms that contain all three bigrams: e.g.,
moon(contains$m,mo,oo,on,n$) - Post-filter: check if the candidate term
moonmatchesmon*(yes, it matchesmonthbut notmoon) - Surviving terms are looked up in the term-document inverted index.
π‘ Why this matters: The bigram index is fast and space-efficient compared to the permuterm index, making it a practical choice for many search systems.
Soundex
Soundex is a phonetic algorithm that maps words to a code based on their pronunciation, designed to match names that sound alike but are spelled differently. It was invented for the U.S. census in 1918. For example, chebyshev maps to tchebycheff. The algorithm is language-specific, mainly for names, and is used by databases (Oracle, Microsoft, etc.) to provide phonetic matching. It increases recall (finds more relevant results) but lowers precision (includes some irrelevant results). For instance, a Soundex query for SMITH might return Lindsay and William. It is used by Interpol for name matching but is typically tailored to European names.
π Definition β Soundex: A phonetic algorithm that converts a word into a four-character code (<uppercase letter><digit><digit><digit>) to match words that sound alike.
π Algorithm (typical):
- Retain the first letter of the word as the first character of the code.
- Replace remaining letters with digits (1-6) based on phonetic groups (e.g., b,f,p,v β 1; c,g,j,k,q,s,x,z β 2; d,t β 3; l β 4; m,n β 5; r β 6). Vowels, h, w, and y are ignored (coded as 0).
- Remove all pairs of consecutive identical digits.
- Remove all zeros from the resulting string.
- Pad the resulting string with trailing zeros and return the first four positions.
π Example: Herman becomes H655:
- H is retained β
H e(ignored) β0,rβ6,mβ5,a(ignored) β0,nβ5- String:
H06505 - Remove consecutive duplicates:
H06505 - Remove zeros:
H655 - Pad with trailing zeros (already 4 characters):
H655
β Key Takeaways
Students must remember the differences between permuterm index and k-gram indexes for handling wildcard queries: permuterm is more flexible but space-intensive, while k-gram (especially bigram) is more space-efficient but requires post-filtering. The Soundex algorithm is a four-character phonetic code designed for name matching in databases, increasing recall at the cost of precision. For the exam, be able to process a wildcard query using both permuterm and bigram methods, and compute a Soundex code for a given name. The key applications are wildcard query processing in search engines and phonetic name matching in database systems.
π§ Quick Revision Questions
- How does the permuterm index process a wildcard query like
hel*o? (Include the query rotation step.) - For the term
moon, list all its bigrams (k=2) with start and end markers. - What are the steps to compute a Soundex code for the name
Johnson? - Which is more space-efficient: the permuterm index or the bigram index? Why?
- Why does Soundex increase recall but lower precision when used in name searches?
π Lecture 20 β Spelling Correction
π Overview: This lecture covers techniques for correcting spelling errors in both documents and user queries within information retrieval systems. It explains how spell correction improves search accuracy by handling misspellings through isolated word and context-sensitive methods, focusing on edit distance and n-gram overlap as core approaches.
ποΈ Topics Covered
This lecture covers the two principal uses of spell correction in information retrieval: correcting documents during indexing (especially for OCR) and correcting user queries to retrieve the right results. It distinguishes between isolated word correction and context-sensitive correction, explains edit distance (Levenshtein distance) and weighted edit distance, introduces n-gram overlap, and includes a discussion on dynamic programming with a Fibonacci series example as a problem-solving technique.
π Lecture Summary
Spell correction
Spell correction has two principal uses in information retrieval: correcting documents being indexed and correcting user queries to retrieve the βrightβ answers. There are two main flavors: Isolated word correction checks each word on its own for misspelling but will not catch typos resulting in correctly spelled words (e.g., βfromβ β βformβ). Context-sensitive correction looks at surrounding words to catch such errors (e.g., βI flew form Lahore to Dubaiβ).
π‘ Why this matters: Context-sensitive correction is critical for understanding user intent when a typo produces a valid word.
Document correction
Document correction is especially needed for OCRβed documents. Correction algorithms are tuned for specific confusions, like rn/m (e.g., βmodernβ may be considered as βmodemβ). Algorithms can use domain-specific knowledge β for example, OCR can confuse O and D more often than O and I (adjacent on the QWERTY keyboard, so more likely interchanged in typing). However, web pages and even printed material have typos. The goal is that the dictionary contains fewer misspellings, but often we donβt change the documents and instead fix the query-document mapping.
π Definition β Document correction: The process of fixing spelling errors in the document corpus, especially needed for OCR output, to improve indexing accuracy.
Query mis-spellings
Query misspellings are the principal focus here. For example, the query βFasbinderβ vs. βFassbinderβ (correct German surname). We can either retrieve documents indexed by the correct spelling, OR return several suggested alternative queries with the correct spelling β as seen in Googleβs βDid you mean...?β feature.
π Definition β Query misspelling: An incorrectly spelled word in a userβs search query that must be corrected to retrieve relevant documents.
Isolated word correction
The fundamental premise is that there is a lexicon from which the correct spellings come. Two basic choices for this: a standard lexicon (e.g., Websterβs English Dictionary), or an βindustry-specificβ lexicon β hand-maintained. Additionally, the lexicon of the indexed corpus can be used (e.g., all words on the web, including misspellings). Given a lexicon and a character sequence Q, we must return the words in the lexicon closest to Q. βClosestβ is determined using several alternatives: Edit distance (Levenshtein distance), Weighted edit distance, and n-gram overlap.
π Definition β Isolated word correction: A spell correction method that checks each word on its own for misspelling against a lexicon without considering context. π Definition β Lexicon: A dictionary of correct words used as a reference for spell correction.
Edit distance
Edit distance (also known as Levenshtein distance) is the minimum number of operations to convert one string Sβ to another Sβ. Operations are typically character-level: Insert, Delete, Replace, and sometimes Transposition. For example, the edit distance from βdofβ to βdogβ is 1 (replace f with g). From βcatβ to βactβ is 2 (delete c, insert a before t) β but just 1 with transpose (swap c and a). From βcatβ to βdogβ is 3. Generally, edit distance is found by dynamic programming.
π Formula: Edit distance = minimum number of insert, delete, replace (and optionally transpose) operations to transform one string into another. π Example: From βcatβ to βactβ:
- Without transpose: delete βcβ (1), insert βaβ at start (1) = 2 operations
- With transpose: swap βcβ and βaβ = 1 operation
Problem Solving Techniques
Dynamic programming solves sub-problems bottom up. The problem cannot be solved until we find all solutions of sub-problems. The solution comes up when the whole problem appears. In contrast, Divide and conquer works by dividing the problem into sub-problems, conquering each sub-problem recursively, and combining these solutions.
π Definition β Dynamic programming: A problem-solving technique that solves sub-problems bottom-up, building solutions incrementally until the full problem is solved.
Fibonacci series
Definition: The first two numbers in the Fibonacci sequence are 1 and 1, or 0 and 1, depending on the chosen starting point, and each subsequent number is the sum of the previous two (Wikipedia). The algorithm is:
Fib(n) {
if (n == 1) return 0;
if (n == 2) return 1;
else return Fib(n-1) + Fib(n-2);
}
The Fibonacci Tree shows the recursive calls for computing Fibonacci numbers, demonstrating the overlapping sub-problems that make dynamic programming an efficient approach.
π Definition β Fibonacci series: A sequence where each number is the sum of the two preceding ones, typically starting with 0 and 1.
β Key Takeaways
Spell correction in information retrieval has two main uses: correcting documents (especially OCR output) and correcting user queries. The two main approaches are isolated word correction (checking each word independently against a lexicon) and context-sensitive correction (considering surrounding words). Isolated word correction relies on edit distance (Levenshtein distance) to find the closest matching word in the lexicon, using operations like insert, delete, and replace. Edit distance is calculated using dynamic programming, which solves sub-problems bottom-up. Understanding these concepts is essential for implementing spell checkers in search engines and document processing systems.
π§ Quick Revision Questions
- What are the two principal uses of spell correction in information retrieval?
- What is the difference between isolated word correction and context-sensitive correction?
- What is edit distance and what operations does it use?
- Why is dynamic programming used to calculate edit distance?
- In the context of document correction, what does the confusion "rn/m" mean and why does it occur?
π Lecture 21 β Spelling Correction
π Overview: This lecture covers techniques for spelling correction in information retrieval systems, focusing on how to handle user query errors. It introduces edit distance and n-gram overlap methods to find and rank potential corrections, which is essential for improving search accuracy and user experience.
ποΈ Topics Covered
The lecture covers edit distance as a metric for measuring string similarity, methods for using edit distances to generate spelling suggestions, weighted edit distance for capturing common errors like OCR or keyboard mistakes, n-gram overlap as an alternative approach, and the Jaccard coefficient as a normalized measure of similarity.
π Lecture Summary
Edit distance
Edit distance is a fundamental concept for measuring similarity between two strings. It is defined as the minimum number of operations required to convert one string into another. The standard operations are character-level: Insert, Delete, Replace, and optionally Transposition (swapping two adjacent characters). This distance is typically computed using dynamic programming.
π Definition β Edit distance: The minimum number of character-level operations (Insert, Delete, Replace, Transposition) needed to transform one string into another.
π Formula: No single formula, but computed via dynamic programming table. For strings S1 and S2, the cost at position (i,j) is: min(cost of insertion, cost of deletion, cost of substitution)
π Example: The edit distance from dof to dog is 1 (replace 'f' with 'g'). From cat to act is 2 (transposition, or 2 operations without transpose). From cat to dog is 3 (replace each character).
π‘ Why this matters: Edit distance provides a principled way to quantify how "far apart" two words are, which is crucial for suggesting corrections when a user misspells a query term.
Using edit distances
This section discusses practical strategies for applying edit distance in a search system. The primary approach is to enumerate all character sequences within a preset (or weighted) edit distance (commonly 2) from the query. This set is then intersected with a list of "correct" words (the lexicon). The matching terms are shown to the user as suggestions. Alternatively, the system can look up all possible corrections in the inverted index and return documents, though this is slow. Another approach is to run with a single most likely correction, which disempowers the user but saves a round of interaction.
Weighted edit distance
Weighted edit distance is an extension of basic edit distance where the cost of each operation depends on the specific characters involved. This is designed to capture realistic error patterns such as OCR (Optical Character Recognition) errors or keyboard typing errors. For example, the letter 'm' is more likely to be mis-typed as 'n' (adjacent on keyboard) than as 'q' (distant), so replacing 'm' with 'n' should have a smaller cost than replacing 'm' with 'q'. This approach can be formulated as a probability model and requires a weight matrix as input. The dynamic programming algorithm is modified to incorporate these weights.
π Definition β Weighted edit distance: A variant of edit distance where the cost of each operation (insert, delete, replace) depends on the specific characters involved, rather than being uniform.
n-gram overlap
n-gram overlap provides an alternative method for finding spelling corrections. The process involves enumerating all the n-grams (contiguous subsequences of n characters) in the query string and in the lexicon. The system uses an n-gram index (similar to that used for wild-card searches) to retrieve all lexicon terms that match any of the query's n-grams. Results are then thresholded by the number of matching n-grams. Variants can weight matches by keyboard layout or other factors.
π Example with trigrams: Suppose the text is november. Its trigrams are: nov, ove, vem, emb, mbe, ber. The query is december. Its trigrams are: dec, ece, cem, emb, mbe, ber. The overlapping trigrams are emb, mbe, ber β a total of 3 out of 6 in each term.
One option β Jaccard coefficient
To turn raw n-gram overlap counts into a normalized measure of similarity, the Jaccard coefficient is used. It calculates the ratio of the size of the intersection of two sets to the size of their union. For the trigram example: Jaccard coefficient = (size of intersection) / (size of union) = 3 / (6 + 6 - 3) = 3/9 = 1/3.
π Definition β Jaccard coefficient: A normalized measure of similarity between two sets, calculated as the size of their intersection divided by the size of their union.
π Formula: Jaccard(A, B) = |A β© B| / |A βͺ B| β The fraction of total unique elements that are shared between two sets.
π Example: For trigram sets of "november" and "december": intersection size = 3 (emb, mbe, ber), union size = 9 (nov, ove, vem, emb, mbe, ber, dec, ece, cem). Jaccard = 3/9 = 0.333.
β Key Takeaways
The most critical concepts from this lecture are understanding edit distance as the minimum number of operations to convert one string to another, and knowing how to apply it for spelling correction by enumerating sequences within a threshold. Weighted edit distance is crucial for modeling realistic errors like OCR mistakes, where operation costs depend on character pairs. N-gram overlap provides an efficient alternative, especially when combined with an n-gram index, and the Jaccard coefficient normalizes this overlap into a proper similarity metric. The trade-off between returning multiple suggestions vs. a single correction is important for system design.
π§ Quick Revision Questions
- What are the four standard operations used in edit distance, and which one is optional?
- How does weighted edit distance differ from standard edit distance, and what type of errors does it model?
- What is the edit distance from "cat" to "act" when transposition is allowed versus when it is not?
- How many trigrams does the word "november" have, and what are they?
- How is the Jaccard coefficient calculated for n-gram overlap, and what does a value of 0.333 indicate?