π Lecture 22 β Spelling Correction
π Overview: This lecture focuses on techniques for correcting spelling errors in information retrieval systems, particularly for query terms. It covers matching trigrams, the use of the Jaccard coefficient to identify candidate terms, context-sensitive spell correction, and general issues in spell correction.
ποΈ Topics Covered
The lecture covers matching trigrams using Jaccard coefficient to identify candidate terms, computing the Jaccard coefficient with n-grams, joining n-grams with edit distance for efficient correction, context-sensitive spell correction using surrounding context and query logs, and general issues in spell correction with relevant resources.
π Lecture Summary
Matching trigrams
The process of spelling correction often begins with matching trigrams (3-letter n-grams) to find candidate terms from a dictionary. A heuristic approach is used to improve efficiency when computing the Jaccard coefficient (J.C.).
π Definition β Heuristic for Jaccard Coefficient: While computing the Jaccard coefficient, we may disregard the repeating n-grams in the query term as well as in the current dictionary term. The reason is that we are computing a candidate term in any case, which we shall process later using edit distance.
Computing Jaccard coefficient
The Jaccard coefficient is a measure used to compare the similarity and diversity of sample sets. In spell correction, it helps identify candidate terms by comparing the n-gram sets of the query term and dictionary terms.
π Formula: Jaccard coefficient = (|A β© B|) / (|A βͺ B|) β The number of n-grams common to both sets divided by the total number of unique n-grams in the union of both sets.
π Example: For query term "form" and dictionary term "from":
- Trigrams of "form": {"for", "orm"}
- Trigrams of "from": {"fro", "rom"}
- Common trigrams: none
- Jaccard coefficient = 0/4 = 0
Joining N-grams with Edit Distance
The n-gram approach will give an Answer list which contains candidate terms. These terms can then be checked for Edit Distance. This process helps avoid checking the edit distance for all dictionary terms and each term, but restricts it to only candidate terms in the answer list.
π‘ Why this matters: This two-step approach significantly reduces computational overhead by filtering the dictionary using n-gram matching before applying the more expensive edit distance calculation.
Context-sensitive spell correction
Context-sensitive correction requires the surrounding context to catch errors that are context-dependent, such as in the phrase "flew form heathrow" (where "form" should be "from"). Natural Language Processing (NLP) is considered too heavyweight for this task in information retrieval.
The first idea is to retrieve dictionary terms close (in weighted edit distance) to each query term. Then, try all possible resulting phrases with one word "fixed" at a time. For example:
- flew from heathrow
- fled form heathrow
- flea form heathrow
- etc.
Suggest the alternative that has lots of hits. It is more appropriate to look for hits in Query Logs rather than in the corpus.
π Definition β Context-sensitive correction: Suppose that for "flew form Heathrow", we have 7 alternatives for "flew", 19 for "form", and 3 for "heathrow". Instead of checking 7193 = 399 combinations, we look for the most frequent (in corpus/in query log) replacement of the first word, combine it with the second to formulate a bigram, then choose the most frequent and then combine it with the third one. This reduces it to much less than 399.
π Alternative approaches: Correct each word separately, which will result in 7+19+3 = 29 operations. Another alternative is to only correct the misspelled words.
General issues in spell correction
General issues in spell correction include the efficiency of retrieval, the accuracy of suggestions, and the choice of resources (corpus vs. query logs) for frequency-based decisions.
Resources for further reading include:
- IIR 3, MG 4.2 (textbooks)
- K. Kukich. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4), Dec 1992.
- J. Zobel and P. Dart. Finding approximate matches in large lexicons. Software - practice and experience 25(3), March 1995.
- Mikael Tillenius: Efficient Generation and Ranking of Spelling Error Corrections. Master's thesis at Sweden's Royal Institute of Technology.
- Peter Norvig: How to write a spelling corrector (http://norvig.com/spell-correct.html) β a nice, easy reading on spell correction.
β Key Takeaways
The most critical concepts for the exam are: (1) The Jaccard coefficient is used to measure n-gram similarity between query terms and dictionary terms to identify candidate corrections, with a heuristic to disregard repeating n-grams. (2) N-gram matching filters the dictionary before applying edit distance, significantly improving efficiency. (3) Context-sensitive correction requires surrounding context and can be computationally expensive; using query logs is more appropriate than corpus hits. (4) The phrase approach can be optimized by checking alternatives incrementally (bigram by bigram) rather than evaluating all possible combinations. (5) Spell correction resources include key research papers by Kukich, Zobel and Dart, and a practical introduction by Peter Norvig.
π§ Quick Revision Questions
- What is the heuristic used when computing the Jaccard coefficient for n-gram matching, and why is it applied?
- How does joining n-grams with edit distance improve the efficiency of spell correction?
- In context-sensitive correction, why is it more appropriate to look for hits in query logs rather than in a corpus?
- For the phrase "flew form heathrow" with 7 alternatives for "flew", 19 for "form", and 3 for "heathrow", describe two different approaches to reduce the search space from 399 combinations.
- Which researcher's work on "Techniques for automatically correcting words in text" is referenced as a resource for efficient spell retrieval?