π Lecture 1 β Information Retrieval Techniques
π Overview: This lecture introduces the foundational concepts of Information Retrieval (IR), covering the transition from structured to unstructured data storage, the digitization of libraries, and the core models used in modern search engines. It lays the groundwork for understanding how systems like Google find and rank relevant documents from vast datasets.
ποΈ Topics Covered
The lecture covers the limitations of traditional library and database search, contrasting them with modern IR systems. It introduces the shift from structured data in tables to semi-structured and unstructured text, the digitization of libraries, and the concept of IR models. Key topics include the use of indexes and inverted indexes in search engines like Google/Bing/Yahoo, the importance of test corpora for evaluation, and an introduction to the probabilistic model for relevance.
π Lecture Summary
Introduction
The lecture begins by contextualizing Information Retrieval (IR) techniques within the evolution of information storage. It contrasts traditional Library Management Systems that rely on structured data (like tables) with the need to search and manage vast amounts of semi-structured and unstructured data that does not fit neatly into database rows and columns.
Structured Data Storage / Tables
Traditional systems, like an Employee database, store information in structured tables with rows (records) and columns (fields). For example, a table might have columns for Employee, Department, and Salary. Searching requires queries on these specific fields, which is effective for well-defined data but fails for free-form text or document content.
Semi-Structured and Unstructured
Modern IR deals heavily with semi-structured data (e.g., web pages with HTML tags) and unstructured data (e.g., plain text documents, emails, books). This type of data has no pre-defined schema, making it difficult to search with simple database queries. IR techniques are specifically designed to handle this complexity.
Library Digitization
The digital transformation of physical libraries into digital repositories is a core challenge. This involves converting physical books and documents into searchable digital formats. The goal is to provide access to the content, not just the metadata (title, author). This shift is what makes techniques like full-text search and relevance ranking essential.
Information Retrieval Models
To find relevant documents from a large collection, IR systems use mathematical models. The lecture explicitly introduces:
- Indexes / Inverted Indexes: A fundamental data structure. An inverted index maps words back to the documents they appear in, along with their position. This is how search engines can quickly find all documents containing a specific word without scanning every file.
- Test Corpus: A standard collection of documents (e.g., a specific book or set of books) used to evaluate and compare the effectiveness of different IR models.
- The Probabilistic Model: This model computes a similarity coefficient between a query and each document. It estimates the probability that a document will be relevant to a given query. It is a core statistical approach for ranking results.
π Definition β Probabilistic Model: A model that computes the similarity coefficient between queries and documents as the probability that a document will be relevant to a query.
Search Computing: Books
The lecture uses the example of searching for computing books to illustrate how these models are applied. The process involves taking a user's query, using an inverted index to locate candidate documents (books), and then applying an IR model (like the probabilistic model) to rank these books by their estimated relevance to the user's search terms.
β Key Takeaways
- Information Retrieval is fundamentally about searching semi-structured and unstructured data, which is a different problem from searching structured databases.
- The inverted index is the core data structure enabling fast search in large document collections, acting as a lookup table from words to documents.
- The probabilistic model is a classic approach to ranking, which estimates the likelihood of a document being relevant to a user's query.
- Academic research in IR relies on test corpora to objectively evaluate and compare the performance of different search algorithms.
- The lecture contrasts the limitations of traditional library catalogs (metadata search) with the power of modern search engines (full-text content search).
π§ Quick Revision Questions
- What is the fundamental difference between searching a structured database (like a library management system) and searching unstructured text?
- Explain the purpose of an inverted index in Information Retrieval.
- How does the probabilistic model determine which documents are more relevant to a query?
- What is a "test corpus" and why is it important in the context of evaluating IR models?
- Give an example of "semi-structured data" vs. "unstructured data" as mentioned in the lecture context.
π Lecture 2 β Information Retrieval Techniques
π Overview: This lecture introduces the fundamental concepts of Information Retrieval (IR) , exploring its definitions, core models, and the foundational Boolean Retrieval Model. It explains how IR systems represent, store, and organize unstructured information to satisfy user information needs, focusing on the Boolean model's mechanics, query processing, and its pros and cons.
ποΈ Topics Covered
The lecture begins by defining Information Retrieval from multiple perspectives and discussing the general concept of IR models, including their components and ranking functions. It then differentiates between Ad Hoc retrieval and Filtering tasks. The core of the lecture is dedicated to the Boolean Retrieval Model, explaining its binary weighting scheme, query evaluation (including posting list intersection and query optimization), and concludes with a discussion of its advantages and significant drawbacks, such as its lack of document ranking.
π Lecture Summary
What is Information Retrieval?
Information Retrieval (IR) is the field concerned with the representation, storage, organization of, and access to information items. It aims to find relevant material, typically unstructured text documents, that satisfy an information need from within large collections stored on computers. IR is about finding relevant documents, not just simple pattern matches.
IR Models
Modeling in IR is a complex process aimed at producing a ranking function, which assigns scores to documents with regard to a given query. An IR model is formally defined as a quadruple [D, Q, F, R(qi, dj)]:
- D: A set of logical views for the documents.
- Q: A set of logical views for the user queries.
- F: A framework for modeling documents and queries.
- R(qi, dj): A ranking function.
There are two main retrieval tasks: Ad Hoc Retrieval, where a user submits a new query to a relatively static collection; and Filtering, where a user has a long-term information need and is notified of new documents matching a profile.
The Boolean Model
The Boolean retrieval model is a simple model based on set theory and Boolean algebra. In this model, the significance of an index term for a document is represented by a binary weight (Wkj β {0,1}), indicating the presence (1) or absence (0) of a term in a document. Queries are expressed as Boolean expressions using the operators AND, OR, and NOT (e.g., "Brutus AND Caesar but NOT Calpurnia").
π Definition β Boolean Model: A retrieval model where documents are represented as binary vectors of index terms, and queries are Boolean expressions. A document is considered relevant if its set of terms satisfies the query's logical conditions.
π Formula: The ranking function R(qi, dj) returns a binary similarity score of 0 or 1. The query q = tβ β§ t_b retrieves the intersection of the posting lists for tβ and t_b: R(q, dβ±Ό) = 1 if the document vector satisfies the Boolean expression, else 0.
π Example: Given three documents dβ = [1,1,1]α΅, dβ = [1,0,0]α΅, dβ = [0,1,0]α΅ for terms (tβ, tβ, tβ), the postings are R_tβ = {dβ, dβ}, R_tβ = {dβ, dβ}, R_tβ = {dβ}. For the query q = tβ β§ tβ, we intersect R_tβ and R_tβ: {dβ, dβ} β© {dβ, dβ} = {dβ}. Only document dβ is considered relevant. The merge operation on sorted posting lists takes O(x+y) operations, where x and y are the list lengths.
Query Optimization
Query optimization is the process of selecting how to organize the work of answering a query with the goal of minimizing the total amount of work. This often involves evaluating the query in a different order to reduce the size of intermediate results.
π Example: Given tβ = {dβ, dβ, dβ , dβ}, tβ = {dβ, dβ, dβ, dβ , dβ}, tβ = {dβ, dβ, dβ}, and query q = tβ AND tβ OR tβ:
- Left-first evaluation: (tβ AND tβ) = {dβ, dβ } β {dβ, dβ } OR tβ = {dβ, dβ , dβ, dβ, dβ} (5 documents)
- Right-first evaluation: (tβ OR tβ) = {dβ, dβ, dβ, dβ , dβ, dβ} β {dβ, dβ, dβ, dβ , dβ, dβ} AND tβ = {dβ, dβ } (2 documents) The standard evaluation priority is: AND, NOT, then OR.
Considerations on the Boolean Model
The Boolean Model is a data retrieval model rather than an information retrieval model because it uses a binary decision criterion (relevant or not) with no notion of a grading scale or partial matching.
Pros:
- Boolean expressions have precise semantics.
- Allows for structured queries.
- Intuitive for expert users.
- Simple and neat formalism.
Drawbacks:
- Retrieval is based on binary decision criteria with no notion of partial matching.
- No ranking of documents is provided (absence of a grading scale).
- Users find it awkward to translate an information need into a Boolean expression.
- User queries are often too simplistic.
- The model frequently returns either too few or too many documents.
β Key Takeaways
This lecture establishes that Information Retrieval is about finding relevant documents from large, unstructured collections. The Boolean Retrieval model is the simplest IR model, using binary term weights and Boolean algebra; this means a document is either retrieved or not with no ranking. While it offers precise semantics, its major failing is the lack of partial matching and ranking, often producing results that are "too few or too many". You must understand how to evaluate Boolean queries using logical operators and intersection of sorted posting lists, and the principle of query optimization to minimize computational work. The Boolean model is more of a data retrieval than an information retrieval model due to its all-or-nothing approach.
π§ Quick Revision Questions
- What is the fundamental difference between the goal of Information Retrieval and simple pattern matching?
- What are the four components of an IR model (the quadruple)?
- In the Boolean Model, what does a binary weight of '0' and '1' for an index term represent?
- A search system counts all documents in an inverted list for "cat" and all documents in the list for "dog", then adds the two sets together. Is this most likely the AND or OR operation?
- Why is the Boolean Model considered more of a "data" retrieval than an "information" retrieval model?
π Lecture 3 β Boolean Retrieval Model & Rank Retrieval Model
π Overview: This lecture introduces two fundamental information retrieval models: the Boolean Retrieval Model, which uses exact set-based matching with AND/OR/NOT operators, and the Ranked Retrieval Model, which scores and orders documents by relevance. Understanding both models is critical for comparing how professional search systems (like Westlaw) and modern web search engines work.
ποΈ Topics Covered
The lecture covers the Boolean Retrieval Model including its definition, set-based document representation, Boolean queries with AND/OR/NOT operators, and its commercial application in Westlaw. It then examines the problems with Boolean search (feast or famine) and transitions to Ranked Retrieval Models, which use free text queries and scoring to produce ordered result sets without overwhelming users.
π Lecture Summary
Boolean Retrieval Model
The Boolean Retrieval Model views each document as a set of terms (unique words), not a bag (which allows duplicates). For example, documents D1 = {This is a pen} and D2 = {It is a pen} produce the set {This, It, is, a, pen}, where order and duplicates are irrelevant. A set is like {a, b, c} = {b, a, c}, while a bag is like {a, a, b, c}.
π Definition β Boolean queries: Queries that use AND, OR, and NOT to join query terms. The model is precise: each document either matches the condition or not.
π Model: Set membership β Document either contains the term(s) or does not.
π Example: Query "pen AND (this OR it)" would return both D1 and D2 because each contains "pen" and at least one of "this" or "it".
The Boolean model was the primary commercial retrieval tool for 3 decades. Many professional searchers (e.g., lawyers) still prefer Boolean queries because you know exactly what you are getting. Many search systems you use are also Boolean: spotlight, email, intranet, etc.
Information Retrieval Ingredients
The three key ingredients for any information retrieval system are:
- Documents representation β how documents are modeled (e.g., as sets of terms)
- Query formulation β how users express their information need
- Query processing β how the system matches queries to documents
Westlaw
Westlaw is the largest commercial legal search service in terms of the number of paying subscribers, with over half a million subscribers performing millions of searches a day over tens of terabytes of text data. The service started in 1975. In 2005, Boolean search (called "Terms and Connectors" by Westlaw) was still the default, used by a large percentage of users, although ranked retrieval has been available since 1992.
π Example queries:
- Information need: Legal theories on preventing disclosure of trade secrets by former employees of a competing company β Query:
"trade secret" /s disclos! /s prevent /s employe! - Information need: Requirements for disabled people to access a workplace β Query:
disab! /p access! /s work-site work-place (employment /3 place) - Information need: Cases about a host's responsibility for drunk guests β Query:
host! /p (responsib! liab!) /p (intoxicat! drunk!) /p guest
Problem with Boolean search: feast or famine
Boolean queries have a significant feast or famine problem. They require query writing skills, and often result in either too few (=0) or too many (1000s) results. It takes a lot of skill to come up with a query that produces a manageable number of hits. AND gives too few; OR gives too many.
Ranked retrieval models
In ranked retrieval, rather than a set of documents satisfying a query expression, the system returns an ordering over the (top) documents in the collection for a query. Free text queries use one or more words in a human language, rather than a query language of operators and expressions.
π‘ Why this matters: Ranked retrieval solves the feast or famine problem because large result sets are not an issue β we just show the top k (β10) results and don't overwhelm the user, assuming the ranking algorithm works.
Scoring as the basis of ranked retrieval
We wish to return in order the documents most likely to be useful to the searcher. To rank-order documents, we assign a score β say in [0, 1] β to each document. This score measures how well the document and query "match".
β Key Takeaways
The Boolean Retrieval Model treats documents as term sets and uses exact Boolean logic (AND/OR/NOT), which is precise but suffers from feast or famine β either too few or too many results. Westlaw demonstrates successful commercial Boolean search in legal applications, but it requires significant query writing skill. The Ranked Retrieval Model overcomes this by assigning a relevance score to each document and returning a ranked list of top results, typically with free text queries. This scoring-based approach is the foundation of modern web search engines, eliminating the need for users to construct complex Boolean expressions.
π§ Quick Revision Questions
- What is the fundamental difference between a "set" and a "bag" in document representation? Give an example.
- What does the Boolean query
"trade secret" /s disclos! /s prevent /s employe!mean in Westlaw, and what information need does it address? - Describe the "feast or famine" problem in Boolean retrieval and explain why ranked retrieval avoids it.
- How does a ranked retrieval model determine which documents to show to the user? What role does "scoring" play?
- What were the three key ingredients of an information retrieval system mentioned in the lecture?
π Lecture 4 β Vector Space Retrieval Model
π Overview: This lecture introduces the Vector Space Retrieval Model, a fundamental approach in information retrieval that represents documents and queries as vectors in a term space. It explains how term weighting schemes, particularly tf-idf, are used to compute document-query similarity scores, enabling ranked retrieval that is more precise than Boolean models.
ποΈ Topics Covered
The lecture covers the Vector Model's core concepts, including binary term-document incidence matrices and term-document count matrices, the bag of words model, term frequency (tf) with log-frequency weighting and normalization techniques, document frequency and its relationship to term informativeness, idf (inverse document frequency), and finally the tf-idf weighting scheme.
π Lecture Summary
The Vector Model (Salton, 1968)
The Vector Space Model represents documents and queries as vectors in the term space. Term weights are used to compute the degree of similarity (or score) between each document stored in the system and the user query. By sorting in decreasing order of this degree of similarity, the vector model takes into consideration documents which match the query terms only partially. This ranked document answer set is a lot more precise (in the sense that it better matches the user information need) than the document answer set retrieved by the Boolean model. A measure of the similarity between the two vectors is then computed, and documents whose content (terms in the document) correspond most closely to the content of the query are judged to be the most relevant.
Binary term-document incidence matrix
In this representation, each document is represented by a binary vector β {0,1}^|V|, where 1 indicates the presence of a term and 0 indicates its absence. The lecture presents a table showing binary incidence for terms (Antony, Brutus, Caesar, Calpurnia, Cleopatra, mercy, worser) across documents (Antony and Cleopatra, Julius Caesar, The Tempest, Hamlet, Othello, Macbeth).
Term-document count matrices
Instead of just presence/absence, the count matrix considers the number of occurrences of a term in a document. Each document is a count vector in β^V (a column below). The lecture provides an example table showing actual counts, e.g., "Antony" appears 157 times in "Antony and Cleopatra", 73 times in "Julius Caesar", etc.
Bag of words model
The vector representation doesn't consider the ordering of words in a document. For example, "John is quicker than Mary" and "Mary is quicker than John" have the same vectors. This is called the bag of words model. In a sense, this is a step back: The positional index is required to distinguish these two documents. The course will look at "recovering" positional information later.
Term frequency tf
The term frequency tf_t,d of term t in document d is defined as the number of times that t occurs in d. Raw term frequency is not what we want: a document with 10 occurrences of the term is more relevant than a document with 1 occurrence, but not 10 times more relevant. Relevance does not increase proportionally with term frequency.
π Definition β Term Frequency (tf): The number of times a term occurs in a document.
π Formula (Log-frequency weighting): The log frequency weight of term t in d is:
w_t,d = { 1 + log10(tf_t,d), if tf_t,d > 0; 0, otherwise }
β Plain-English meaning: 0 β 0, 1 β 1, 2 β 1.3, 10 β 2, 1000 β 4, etc. This reduces the impact of very high term frequencies.
π Example: If a term appears 10 times in a document, its log-frequency weight would be 1 + log10(10) = 1 + 1 = 2. If it appears 1000 times, the weight would be 1 + log10(1000) = 1 + 3 = 4.
Term Frequency Normalization
We can use another normalization method for tf: Find the term frequencies of all terms in a document, find out the maximum tf_max from all those terms, then the tf of term 'i' referred to as tf_i is obtained by dividing the tf_i by tf_max: tf_i = tf_i / tf_max. This brings the term frequencies of each term in a document between 1 and 0, irrespective of the size of a document.
Document frequency
Rare terms are more informative than frequent terms. Consider a term in the query that is rare in the collection (e.g., "arachnocentric") β a document containing this term is very likely to be relevant to the query. We want a high weight for rare terms like "arachnocentric". Conversely, frequent terms are less informative than rare terms β consider a query term that is frequent in the collection (e.g., "high", "increase", "line") β while a document containing such a term is more likely to be relevant than one that doesn't, it's not a sure indicator of relevance. For frequent terms, we want high positive weights but lower than for rare terms.
π‘ Why this matters: Document frequency (df) captures how common or rare a term is across the entire collection, helping distinguish informative terms from common ones.
idf example, suppose N = 1 million
The inverse document frequency (idf) formula is:
idf_t = log10(N / df_t)
where N is the total number of documents and df_t is the document frequency of term t.
π Formula: idf_t = log10(N / df_t) β Plain-English meaning: Terms that appear in fewer documents get higher idf values.
π Example (with N = 1 million):
- "calpurnia": df = 1, idf = 6
- "animal": df = 100, idf = 4
- "sunday": df = 1,000, idf = 3
- "fly": df = 10,000, idf = 2
- "under": df = 100,000, idf = 1
- "the": df = 1,000,000, idf = 0
There is one idf value for each term t in a collection.
tf-idf weighting
The tf-idf weight of a term is the product of its tf weight and its idf weight:
w_t,d = (1 + log10(tf_t,d)) Γ log10(N / df_t)
π Definition β tf-idf: A term weighting scheme that is the product of term frequency and inverse document frequency.
π Formula: w_t,d = (1 + log10(tf_t,d)) Γ log10(N / df_t) β Plain-English meaning: The weight of a term in a document increases with the number of occurrences within that document and increases with the rarity of the term across the collection.
This is the best known weighting scheme in information retrieval. Note: the "-" in tf-idf is a hyphen, not a minus sign. Alternative names include tf.idf, tf x idf.
β Key Takeaways
The Vector Space Model represents documents and queries as vectors in term space, using the bag of words approach that ignores word order. Term frequency (tf) measures how often a term appears in a document, but raw counts are log-transformed because relevance doesn't increase linearly with frequency. Document frequency (df) captures how common a term is across the collection, with rare terms being more informative (higher idf values). The tf-idf weighting scheme combines both factors β it gives high weight to terms that appear frequently in a document but rarely in the collection. Students must remember the log-frequency weighting formula, the idf calculation, and the final tf-idf product formula, as these are fundamental to ranked retrieval.
π§ Quick Revision Questions
- What is the bag of words model, and what information does it deliberately ignore?
- Why is raw term frequency not a good measure of relevance, and how does log-frequency weighting address this?
- How is term frequency normalization performed using tf_max?
- Calculate the idf value for a term with df = 250 in a collection of N = 50,000 documents.
- Write the complete tf-idf formula and explain what each component represents in plain English.
π Lecture 5 β Information Retrieval Techniques
π Overview: This lecture dives into the core of modern information retrieval: how to transform text documents and user queries into weighted numerical vectors and compute their similarity. It focuses on the TF-IDF weighting scheme, vector space representation, and key similarity measures like the Jaccard coefficient and inner product, which are fundamental for ranking search results.
ποΈ Topics Covered
The lecture covers computing TF-IDF with a worked example, mapping documents and queries to vectors in a high-dimensional space, understanding term weight coefficients, and comparing two similarity measures: the Jaccard coefficient and the inner product (dot product), including its normalized form.
π Lecture Summary
Computing TF-IDF -- An Example
The lecture provides a step-by-step example of calculating TF-IDF weights for terms in a document. Given a document with terms A, B, C and their frequencies, and their document frequencies across a collection of 10,000 documents, we calculate each component. The term frequency (tf) is normalized by the maximum frequency in the document, and the inverse document frequency (idf) is the log of the ratio of total documents to the number of documents containing the term. The final tf-idf weight is the product of these two values, giving high weight to terms that are frequent in a specific document but rare across the collection.
π Definition β Term Frequency (tf): A measure of how frequently a term occurs in a document. It is often normalized (e.g., by dividing by the maximum frequency of any term in the document) to prevent bias towards longer documents.
π Definition β Inverse Document Frequency (idf): A measure of how much information a term provides, i.e., whether it is common or rare across all documents. It is calculated as log(N/df), where N is the total number of documents and df is the number of documents containing the term.
π Formula: tf-idf = tf * idf where tf = term frequency in document / max term frequency in document and idf = logβ(N / document frequency of term). β This weights terms by how descriptive they are of a specific document.
π Example: For a document with terms A(3), B(2), C(1) in a collection of 10,000 documents where A appears in 50, B in 1300, and C in 250:
- A: tf = 3/3 = 1; idf = logβ(10000/50) β 7.6; tf-idf = 7.6
- B: tf = 2/3 β 0.67; idf = logβ(10000/1300) β 2.9; tf-idf β 2.0
- C: tf = 1/3 β 0.33; idf = logβ(10000/250) β 5.3; tf-idf β 1.8 π‘ Why this matters: This example shows how a rare term (A, appearing in only 50 documents) gets a much higher tf-idf weight than a common term (B), making it more influential in identifying the document's topic.
Binary β count β weight matrix
The lecture illustrates the transformation of a document-term matrix from a binary representation (1 if a term is present, 0 if not), to a count matrix (raw term frequencies), and finally to a weight matrix (with tf-idf values). For example, in the binary matrix, 'Antony' has a 1 in 'Antony and Cleopatra' and 'Julius Caesar', but 0 in others. In the count matrix, its frequency is 157 in one and 73 in another. In the final weighted matrix, the tf-idf weight for 'Antony' in 'Antony and Cleopatra' is 5.25, and in 'Julius Caesar' is 3.18. This progression shows how raw data is refined into meaningful importance scores.
π Definition β Document-Term Matrix: A matrix where rows represent terms, columns represent documents, and each cell contains a weight (binary, count, or tf-idf) indicating the term's importance in that document.
Mapping to vectors
The concept of mapping documents and queries to a vector space is introduced. Each unique term in the collection defines an axis in this high-dimensional space, making the vectors for the terms themselves the canonical basis vectors. A document is represented as a vector, which is the sum of the vectors (weighted by their tf-idf scores) of all the terms it contains. A query is treated in the exact same way: it is mapped to a vector in the same space using the same weighting scheme. This allows the system to mathematically compare the query with every document.
π Definition β Vector Space Model: An algebraic model for representing text documents (and queries) as vectors of identifiers, such as index terms. Similarity between documents and a query is assessed by the proximity of their vectors. π‘ Why this matters: This mapping is the core transformation that enables mathematical calculation of similarity, moving from raw text to a numerical representation.
Coefficients
The lecture discusses how the coefficients (the weights or values along each dimension of the vector) represent a term's presence, importance, or "aboutness". The model itself gives no guidance on how to set these weights, so several common choices exist:
- Binary: A simple 1 if the term is present in the document, 0 if not.
- tf (Term Frequency): The raw frequency of the term in the document.
- tfΒ·idf: The product of term frequency and inverse document frequency, which is the most common and effective choice as it accounts for both local and global importance.
Query Vector
The query vector is treated the same as a document vector and is typically tf-idf weighted. An alternative approach is for the user to manually supply weights for the query terms, but the standard approach is to compute them automatically.
Similarity Measure
A similarity measure is a function that computes the degree of similarity between two vectors (e.g., query vector and document vector). It is crucial for ranking: the system can rank retrieved documents in order of presumed relevance based on their similarity score and can also enforce a threshold to control the size of the retrieved set.
π Definition β Similarity Measure: A function that computes a score representing the degree of similarity between two vectors.
Jaccard coefficient
The Jaccard coefficient is a commonly used measure for the overlap of two sets A and B.
π Formula: Jaccard(A, B) = |A β© B| / |A βͺ B|
This score always falls between 0 and 1.
Jaccard(A, A) = 1(a set is perfectly similar to itself).Jaccard(A, B) = 0if the sets have no intersection (A β© B = 0).- The sets don't have to be the same size.
Issues with Jaccard for scoring
The lecture points out critical issues with using the Jaccard coefficient for document scoring:
- It does not consider term frequency (tf) β it only checks if a term is present or absent.
- It does not account for the informativeness of rare terms (idf). Jaccard treats all terms equally.
- It lacks a sophisticated method for length normalization. Instead of the Jaccard formula, a more advanced normalization technique is needed, leading to the inner product measure.
Similarity Measure - Inner Product
The inner product (or dot product) is a more sophisticated similarity measure for vectors.
π Formula: sim(dβ±Ό, q) = dβ±Ό β’ q = β wα΅’β±Ό * wα΅’q (where wα΅’β±Ό is the weight of term i in document j, and wα΅’q is the weight of term i in the query).
- For binary vectors, the inner product is simply the number of query terms that match in the document (the size of the intersection).
- For weighted term vectors (like tf-idf), it is the sum of the products of the weights of the matched terms.
- The lecture also mentions "Similarity, Normalized", implying the use of the cosine similarity (the normalized version of the inner product) to account for document length, though the formula is not fully detailed in this text.
β Key Takeaways
The most critical concepts from this lecture are the TF-IDF weighting scheme, which balances term frequency with inverse document frequency to give high weights to distinctive terms. You must understand how to map documents and queries into a vector space, where each unique term is a dimension. The inner product (dot product) is the preferred similarity measure over the simpler Jaccard coefficient because it incorporates non-binary weights (like tf-idf), unlike Jaccard which only handles set membership. Finally, remember the progression from binary to count to weight matrices as a fundamental data preparation step for using these mathematical models.
π§ Quick Revision Questions
- In the TF-IDF example, why did term A (with 3 occurrences) receive a higher tf-idf weight (7.6) than term B (with 2 occurrences and a weight of 2.0)?
- What are the three stages of a document-term matrix transformation discussed in the lecture, and what does each stage represent about a term in a document?
- Explain the fundamental difference between the Jaccard coefficient and the Inner Product when used as a similarity measure for weighted document vectors.
- Why is the "Jaccard coefficient" considered insufficient for scoring documents in a modern IR system, specifically regarding term frequency and term informativeness?
- How is a query represented in the "Vector Space Model," and is its weighting scheme typically the same as or different from a document's?
π Lecture 6 β Similarity Measures and Cosine Similarity Measure
π Overview: This lecture explores how documents and queries are compared in vector space retrieval, focusing on the cosine similarity measure as the primary similarity metric. It also covers the basic indexing pipeline, the challenge of sparse vectors, and how inverted indexes efficiently store and retrieve document-term information in information retrieval systems.
ποΈ Topics Covered
The lecture covers cosine similarity as the most common similarity measure for comparing vectors in vector space retrieval, along with its advantages (partial matching, ranked results) and disadvantages (term independence assumption, lack of model justification). It then discusses the basic indexing pipeline, the nature of sparse vectors with very large vocabularies, and three methods for storing sparse vectors (as lists, trees, or hash tables). Finally, it introduces the inverted index as the practical, efficient implementation for document-term storage.
π Lecture Summary
Cosine Similarity Measure
The cosine similarity measure is the most frequently used similarity metric in the standard vector space retrieval model. In this model, each dimension corresponds to a term in the vocabulary, and vector elements are real-valued numbers reflecting term importance. Any vector β whether representing a document, query, or other object β can be compared to any other vector using cosine correlation. This approach provides partial matching (documents need not contain all query terms) and ranked results (documents are ordered by similarity score).
π Definition β Cosine Similarity: A metric that measures the cosine of the angle between two vectors, indicating how similar they are. It equals the dot product of the vectors divided by the product of their magnitudes.
π Formula: cos(ΞΈ) = (AΒ·B) / (||A|| Γ ||B||) β This computes the cosine of the angle between vectors A and B, where AΒ·B is the dot product and ||A|| and ||B|| are the Euclidean lengths (magnitudes) of the vectors.
π Example: If document vector A = [1, 0, 2] and query vector B = [0, 1, 1], then AΒ·B = (1Γ0) + (0Γ1) + (2Γ1) = 2, ||A|| = β(1Β² + 0Β² + 2Β²) = β5 β 2.236, ||B|| = β(0Β² + 1Β² + 1Β²) = β2 β 1.414, so cos(ΞΈ) = 2 / (2.236 Γ 1.414) = 2 / 3.162 β 0.632.
Vector Space Model: Disadvantages
Despite its widespread use, the vector space model has several significant disadvantages. It assumes independence among terms, meaning it treats each term as unrelated to others β though this is a common assumption in retrieval models. There is a lack of justification for some vector operations, such as the choice of similarity function (e.g., why cosine and not another metric) and the choice of term weights (e.g., why certain weighting schemes are used). The model is barely a retrieval model because it does not explicitly model relevance, a person's information need, or language models. Additionally, it assumes a symmetric treatment of queries and documents, treating them as interchangeable, which may not reflect real retrieval scenarios.
π‘ Why this matters: Understanding these limitations helps you recognize when vector space retrieval might fail β for example, when term relationships are important (e.g., synonyms like "car" and "automobile" are treated as independent) or when relevance is not simply about term overlap.
Basic Indexing Pipeline
The basic indexing pipeline is the process by which documents are converted from raw text into a structured, searchable index. This pipeline typically involves steps such as tokenization (breaking text into words), normalization (e.g., lowercasing), and building term-document mappings. The pipeline prepares documents for vector space representation and subsequent similarity calculations.
Sparse Vectors
Sparse vectors arise because the vocabulary (and therefore the dimensionality of vectors) in information retrieval can be very large β approximately 10β΄ or more terms. However, most documents and queries do not contain most words, so vectors are sparse, meaning most entries are 0. Efficient methods are needed for storing and computing with sparse vectors to avoid wasting memory and computation time.
π Definition β Sparse Vector: A vector in which most elements are zero, common in IR because documents typically contain only a small subset of the total vocabulary.
Sparse Vectors as Lists
One method for storing sparse vectors is to use linked lists of non-zero-weight tokens paired with their weights. The space required is proportional to the number of unique tokens (n) in the document. However, this method requires linear search of the list to find or change the weight of a specific token, and in the worst case, it requires quadratic time β O(nΒ²) β to compute a vector for a document, because the sum of 1 to n equals n(n+1)/2, which is O(nΒ²).
π Definition β O(nΒ²) Quadratic Time: The time grows quadratically with the number of tokens; for n=1000 tokens, about 500,000 operations may be needed.
π Formula: β(i=1 to n) i = n(n+1)/2 = O(nΒ²) β This shows that constructing a vector using a list requires summing from 1 to n, resulting in quadratic time complexity.
Sparse Vectors as Trees
An alternative is to index tokens in a document using a balanced binary tree or trie, with weights stored at the leaves. The space overhead for the tree structure is approximately 2n nodes. This approach provides O(log n) time to find or update the weight of a specific token, and O(n log n) time to construct the entire vector. However, it requires a software package to support such data structures, which may not be readily available.
Sparse Vectors as Hash Tables
A more efficient approach is to store tokens in a hash table, using the token string as the key and the weight as the value. The storage overhead for a hash table is approximately 1.5n. The table must fit in main memory for optimal performance. This method provides constant time (O(1)) to find or update the weight of a specific token (ignoring collisions), and O(n) time to construct the vector (again ignoring collisions).
Implementation Based on Inverted Files
In practice, document vectors are not stored directly; instead, an inverted organization provides much better efficiency. The keyword-to-document index (inverted index) can be implemented as a hash table, a sorted array, or a tree-based data structure (trie, B-tree). The critical issue is achieving logarithmic or constant-time access to token information, which these structures provide.
π Definition β Inverted Index: A data structure that maps each term (keyword) to a list of documents containing that term, along with positions or weights, enabling fast retrieval of documents matching query terms.
Inverted Index
The inverted index is the standard implementation for storing document-term information in information retrieval systems. Instead of storing vectors per document (forward index), the inverted index stores for each term a list of documents in which that term appears. This structure supports efficient query processing by quickly identifying which documents contain query terms, rather than scanning all document vectors.
β Key Takeaways
The most critical concept from this lecture is that cosine similarity is the primary metric for comparing documents and queries in vector space retrieval, providing ranked, partial matching results despite its limitations like term independence and lack of relevance modeling. Sparse vectors are the norm in IR due to large vocabularies, and efficient storage is crucial β hash tables offer O(1) access and O(n) construction, making them faster than lists (O(nΒ²)) or trees (O(n log n)). The inverted index is the practical, real-world implementation that reverses the document-term mapping, allowing logarithmic or constant-time access to term information rather than storing vectors per document. For exams, remember that cosine similarity measures the angle between vectors (not distance), that sparse vectors necessitate efficient data structures, and that the inverted index is the industry standard for large-scale retrieval systems.
π§ Quick Revision Questions
- What is the formula for cosine similarity, and what do each of its components represent?
- List three disadvantages of the vector space model and explain why each is a problem.
- Compare the time complexities for constructing a vector using lists, trees, and hash tables β which is fastest and why?
- What is an inverted index, and how does it differ from storing document vectors directly?
- Why are vectors in information retrieval described as "sparse"? Give an approximate value for vocabulary size.