π Lecture 23 β Performance Evaluation of Information Retrieval Systems
π Overview: This lecture explores how we evaluate the effectiveness and performance of Information Retrieval (IR) systems. It identifies the challenges in measuring system quality, defines various performance metrics, and explains how to assess user happiness with search engines across different contexts like web, eCommerce, and enterprise settings.
ποΈ Topics Covered
The lecture begins by asking why we need to evaluate IR systems and highlights the difficulties involved, such as the subjective and dynamic nature of relevancy. It then details the specific measures for a search engine, including indexing speed, search latency, and query expressiveness. The core of the lecture focuses on the concept of measuring user happiness, examining how this varies depending on the setting (web, eCommerce, enterprise). Finally, it discusses practical ways to tell if users are happy, such as analyzing click-through rates, purchase behavior, and user return rates.
π Lecture Summary
Why System Evaluation?
There are many different retrieval models, algorithms, and systems, so we need a method to determine which one is best. Evaluation helps us choose the best component for tasks like selecting a ranking function (e.g., dot-product, cosine), choosing term selection methods (e.g., stopword removal, stemming), or deciding on term weighting strategies (e.g., TF, TF-IDF). A key question evaluation answers is how far down a ranked list a user must look to find enough relevant documents.
π‘ Why this matters: Without systematic evaluation, we cannot objectively improve search engines or compare different research approaches.
Difficulties in Evaluating IR Systems
Effectiveness is directly related to the relevancy of retrieved items. However, measuring relevancy is inherently difficult. Relevancy is usually not a simple binary classification (relevant/not relevant) but a continuous scale. Even when treated as binary, it can be a difficult judgment for a human to make. From a human perspective, relevancy has four key characteristics:
- Subjective: It depends on a specific user's own judgment.
- Situational: It relates to the user's current needs and context.
- Cognitive: It depends on human perception and behavior.
- Dynamic: It can change over time as the user's knowledge or situation evolves.
Measures for a search engine
Performance can be measured by several factors:
- Indexing Speed: How fast does the system index documents? Measured by the number of documents indexed per hour (considering average document size).
- Search Speed: How fast does it search? Measured by latency as a function of index size.
- Expressiveness of Query Language: The system's ability to allow users to express complex information needs and its speed when processing such complex queries.
- User Interface (UI): An uncluttered and intuitive UI is a crucial performance factor.
- Cost: Is it free or what is the pricing model?
Measuring user happiness
A central issue is who is the user we are trying to make happy, as this depends on the setting.
- Web Engine: The user finds what they want and returns to the engine. Happiness can be measured by the rate of return users. The user's goal is to complete a task, with search being a means, not an end.
- eCommerce Site: The user finds what they want and buys it. The question is whether we measure happiness for the end-user or the eCommerce site. Metrics include time to purchase or the fraction of searchers who become buyers.
- Enterprise (company/govt/academic): The focus is on "user productivity". The key question is how much time users save when looking for information. Other criteria involve breadth of access and secure access.
How do you tell if users are happy?
Several observable behaviors can indicate user happiness at scale:
- Relevance: The search returns products relevant to users.
- Click-through Rate: Search results get clicked a lot. However, be cautious as misleading titles or summaries can cause users to click on irrelevant results.
- Conversion: Users buy after using the search engine, or more generally, users spend a lot of money after using the search engine.
- Repeat Behavior: The presence of repeat visitors or repeat buyers is a strong indicator of happiness.
- Bounce Rate & Retention: Observing if users leave soon after searching, or if they return within a week, month, or year.
β Key Takeaways
Evaluating IR systems is a complex but essential task. The primary challenge is that relevancy is subjective, situational, cognitive, and dynamic, making it difficult to quantify. A system's performance is measured by more than just speed, including indexing rate, search latency, and query expressiveness. The definition of "user happiness" is highly context-dependent, varying between web search, eCommerce, and enterprise settings. Ultimately, user happiness is inferred from behavior like click-throughs, purchases, and repeat visits, but these signals must be interpreted carefully to avoid being misled by, for example, clickbait titles.
π§ Quick Revision Questions
- What are the four human-centric characteristics that make relevancy difficult to measure?
- Name three types of performance measures for a search engine, other than user happiness.
- How does the definition of "user happiness" differ between a web search engine and an eCommerce site?
- Explain why a high click-through rate is not always a reliable indicator of user happiness.
- In an enterprise setting, what is the primary measure of user happiness?
π Lecture 24 β Information retrieval techniques
π Overview: This lecture focuses on how to evaluate Information Retrieval (IR) systems, focusing on benchmarks, relevance judgments, and the challenges of measuring search quality. Understanding these methods is essential for building and testing effective search algorithms.
ποΈ Topics Covered
The lecture covers the elusive nature of measuring happiness (relevance), the concept of a Gold Standard (human-labeled corpora), how to measure the quality of a new search algorithm, issues with relevance judgments, crowd-sourcing for relevance, standard relevance benchmarks (like TREC), and finally how to evaluate an IR system.
π Lecture Summary
Happiness: elusive to measure
The most common proxy for "happiness" in an IR system is the relevance of search results. Measuring relevance requires three elements: (1) a benchmark document collection, (2) a benchmark suite of queries, and (3) a binary assessment (Relevant or Nonrelevant) for each query-document pair. While some work exists on more-than-binary scales, binary assessment is the standard.
Human Labeled Corpora (Gold Standard)
A Gold Standard is created by starting with a corpus of documents, collecting a set of queries for that corpus, and having one or more human experts exhaustively label the relevant documents for each query. This typically assumes binary relevance judgments. This process requires considerable human effort for large document/query corpora.
Relevance judgments
Relevance judgments are usually binary (relevant vs. non-relevant), but can be more nuanced (0, 1, 2, 3...) in other cases. The main issue is the scale: 5 million documents times 50K queries leads to a quarter trillion judgments. Even at 2.5 seconds per judgment, this would take 10ΒΉΒΉ seconds and cost nearly $300 million at $10 per hour. Additionally, 10,000 new products (documents) are added every day.
π Definition β Gold Standard: A human-labeled corpus where experts have assessed the relevance of documents to specific queries, used as a ground truth for evaluating IR systems.
Crowd source relevance judgments?
One approach to reduce costs is to use crowd-sourcing platforms (like Amazon Mechanical Turk). This involves presenting query-document pairs to low-cost labor online, hoping it's cheaper than hiring qualified assessors. The main takeaway is that while you get some signal, the variance in the resulting judgments is very high.
What else?
Still, you need test queries that are germane to the available documents and representative of actual user needs. Random query terms from the documents are generally not a good idea. Instead, sample from query logs if available. Classically (non-Web), with low query rates and not enough query logs, experts hand-craft "user needs."
Standard relevance benchmarks
The TREC (Text REtrieval Conference) by the National Institute of Standards and Technology (NIST) has run a large IR test bed for many years. It uses Reuters and other benchmark doc collections. "Retrieval tasks" are specified, sometimes as queries. Human experts mark, for each query and for each doc, Relevant or Nonrelevant (or at least for the subset of docs that some system returned for that query).
π Example: Public test collections include TREC, Reuters, and others collected by NIST.
Evaluating an IR system
Key insight: The user's information need is translated into a query. Yet, relevance is assessed relative to the user need, not the query itself. For example, the information need might be "My swimming pool bottom is becoming black and needs to be cleaned," while the query is "pool cleaner." The assessor must determine if the document addresses the underlying need, not just whether it contains the words "pool" and "cleaner."
β Key Takeaways
The key point is that evaluating an IR system requires a Gold Standard of human-labeled relevance judgments, but this is extremely costly at scale. Crowd-sourcing offers a cheaper alternative but introduces high variance. The most critical concept is that relevance must be judged against the user's true information need, not merely the query terms they typed, and benchmarks like TREC provide standardized resources for this purpose. Understanding these evaluation methods is fundamental to developing and validating effective search algorithms.
π§ Quick Revision Questions
- What are the three essential elements required for measuring relevance in an IR evaluation?
- What is a "Gold Standard" in the context of IR evaluation, and what is its main limitation?
- Why is crowd-sourcing for relevance judgments considered problematic, despite being cheaper?
- What is the key difference between assessing relevance to a "query" versus assessing relevance to a "user need"?
- Name one major standard relevance benchmark mentioned in the lecture and describe its role.
π Lecture 25 β Benchmarks for the Evaluation of IR Systems
π Overview: This lecture introduces key evaluation measures for information retrieval (IR) systems, focusing on how to assess system performance. It explains why accuracy is not suitable for IR and details the critical concepts of precision and recall, including their trade-off and calculation methods.
ποΈ Topics Covered
The lecture covers evaluation measures such as precision, recall, accuracy, mean average precision, F-measure/E-measure, non-binary relevance, discounted cumulative gain, and normalized discounted cumulative gain. It then focuses on unranked retrieval evaluation, the trade-off between recall and precision, and computing recall/precision points with an example.
π Lecture Summary
Evaluation Measures
This section lists the key evaluation measures used in IR systems. These include Precision, Recall, Accuracy, Mean Average Precision, F-Measure/E-Measure, Non-Binary Relevance, Discounted Cumulative Gain, and Normalized Discounted Cumulative Gain. Each measure provides a different perspective on how well an IR system retrieves relevant documents.
Precision and Recall
Precision is defined as the ability to retrieve top-ranked documents that are mostly relevant. Recall is defined as the ability of the search to find all of the relevant items in the corpus. Determining Recall is difficult because the total number of relevant items is sometimes not available. One method to address this is to sample across the database and perform relevance judgment on these items. Another method is to apply different retrieval algorithms to the same database for the same query, taking the aggregate of relevant items as the total relevant set.
Unranked retrieval evaluation: Precision and Recall
The lecture questions whether accuracy should be used for evaluation. Given a query, an engine classifies each document as "Relevant" or "Nonrelevant." The accuracy of an engine is the fraction of these classifications that are correct.
π Definition β Accuracy: (tp + tn) / (tp + fp + fn + tn), where tp = true positives, tn = true negatives, fp = false positives, fn = false negatives.
Accuracy is a commonly used evaluation measure in machine learning classification work. However, it is not a very useful evaluation measure in IR because it treats all documents equally, and in IR, the focus is on retrieving the most relevant documents from a large corpus where most documents are non-relevant. A system that returns no documents could achieve high accuracy if the query has few relevant documents.
Trade-off between Recall and Precision
There is a fundamental trade-off between recall and precision. Typically, when you try to increase recall (by retrieving more documents), precision tends to decrease. Conversely, when you aim for high precision (by being very selective), recall often suffers because you might miss some relevant documents.
π‘ Why this matters: Understanding this trade-off is essential for tuning an IR system to meet specific user needs, such as high precision for a focused search or high recall for a comprehensive search.
Computing Recall/Precision Points
For a given query, you produce a ranked list of retrievals. Adjusting a threshold on this ranked list produces different sets of retrieved documents, and therefore different recall/precision measures. To compute recall/precision points, you mark each document in the ranked list that is relevant according to the gold standard. Then, compute a recall/precision pair for each position in the ranked list that contains a relevant document.
π Formula: Recall = (Number of relevant items retrieved) / (Total number of relevant items in the corpus) π Formula: Precision = (Number of relevant items retrieved) / (Total number of items retrieved)
Computing Recall/Precision Points: Example 1
The lecture provides a visual example (not fully detailed in text but described conceptually) where a ranked list of documents is assessed against a gold standard. For each relevant document found at a given rank, recall (cumulative relevant found / total relevant) and precision (cumulative relevant found / rank) are calculated. This creates a series of points that can be plotted to show how precision changes as recall increases.
β Key Takeaways
The most critical concepts from this lecture are the definitions of precision and recall and their trade-off. Accuracy is rejected for IR because it is misleading in large, imbalanced corpora. Computing recall/precision points involves evaluating a ranked list against a gold standard, marking relevant documents, and calculating pairs at each relevant rank. Students must understand that recall is difficult to measure due to unknown total relevant items. The listed evaluation measures, including F-measure and NDCG, are essential for comprehensive IR system benchmarking.
π§ Quick Revision Questions
- What are the two main evaluation measures for unranked retrieval discussed in the lecture?
- Why is accuracy not a useful evaluation measure for Information Retrieval?
- What is the fundamental trade-off between recall and precision?
- How do you compute a recall/precision pair for a position in a ranked list?
- What are two methods suggested for determining the total number of relevant items when it is not available?
π Lecture 26 β Precision and Recall
π Overview: This lecture explores the fundamental evaluation metrics used in information retrieval systems: precision and recall. It covers how to compute these metrics, interpolate recall/precision curves, and interpret system performance through various derived measures like the F-measure and E-measure, which are essential for comparing and optimizing search engines.
ποΈ Topics Covered
The lecture covers computing recall/precision points with examples, interpolating recall/precision curves, average recall/precision curves for system evaluation across multiple queries, R-Precision, Precision@K, the F-Measure (harmonic mean of recall and precision), and the E-Measure (a parameterized variant of F-Measure that allows weighting emphasis between precision and recall).
π Lecture Summary
Computing Recall/Precision Points: Example 1 & 2
To evaluate an IR system, we compute recall and precision at various points in the ranked list of retrieved documents. Recall measures the proportion of relevant documents retrieved, while precision measures the proportion of retrieved documents that are relevant. These points are calculated by examining the ranked output and noting the recall and precision values at each position where a relevant document is found.
π Definition β Recall: The fraction of relevant documents that have been retrieved out of all relevant documents in the collection. π Definition β Precision: The fraction of retrieved documents that are relevant out of all documents retrieved.
π Formula:
- Recall = (Number of relevant documents retrieved) / (Total number of relevant documents in collection)
- Precision = (Number of relevant documents retrieved) / (Total number of documents retrieved)
π Example: For a query with 5 relevant documents (R=5), if the system returns 10 documents and 4 of them are relevant, then Precision = 4/10 = 0.4 and Recall = 4/5 = 0.8.
Interpolating a Recall/Precision Curve
Interpolation is used to create a smooth recall/precision curve by estimating precision values at standard recall levels (typically 0.0, 0.1, 0.2, ..., 1.0). This allows comparison across different queries or systems. The standard interpolation method assigns the maximum precision observed at or beyond a given recall level.
π Definition β Interpolation: A technique to estimate precision values at standard recall points by taking the maximum precision achieved at any recall level greater than or equal to the target recall level.
π Example 1: If at recall levels 0.2 and 0.4 we have precisions 0.8 and 0.6 respectively, the interpolated precision at recall 0.1 would be 0.8 (maximum precision at or above 0.1).
π Example 2: For a system with computed points: (0.0, 1.0), (0.2, 0.75), (0.4, 0.5), (1.0, 0.25), the interpolated precision at recall 0.3 becomes max(0.5, 0.25) = 0.5.
Average Recall/Precision Curve
To evaluate overall system performance across a large set of queries, we compute the average precision at each standard recall level. This creates an average recall/precision curve that represents typical system behavior, smoothing out variations from individual queries.
π Definition β Average Precision: The mean of precision values obtained at each standard recall level across all queries in the test set.
π Why this matters: The curve closest to the upper right-hand corner of the graph indicates the best performance, as it shows high precision across all recall levels.
π Sample RP Curve for CF Corpus: The curve typically shows high precision at low recall levels, decreasing as recall increases. A system with a curve that remains high across recall levels is superior.
R-Precision
R-Precision is precision measured at the R-th position in the ranked results, where R is the total number of relevant documents for the query. This metric provides a single-point evaluation that balances recall and precision considerations.
π Definition β R-Precision: The precision at rank position R, where R is the total number of relevant documents for the query.
π Formula: R-Precision = (Number of relevant documents in top R results) / R
π Example: If a query has 10 relevant documents (R=10), and among the top 10 results, 7 are relevant, then R-Precision = 7/10 = 0.7.
Precision@K
Precision@K (P@K) measures precision at a fixed rank position K, regardless of how many relevant documents exist. This is particularly useful in web search where users typically only look at the top few results (e.g., top 10 or top 20).
π Definition β Precision@K: The proportion of relevant documents among the top K retrieved documents.
π Formula: Precision@K = (Number of relevant documents in top K) / K
π Example: If a web search returns 20 results and 12 are relevant in the top 20, then Precision@20 = 12/20 = 0.6.
F-Measure
The F-Measure (also called F1-score) combines both recall and precision into a single metric using the harmonic mean. Unlike the arithmetic mean, both recall and precision need to be high for the harmonic mean to be high, making it a balanced evaluation.
π Definition β F-Measure: A single performance metric that combines recall and precision using their harmonic mean.
π Formula: F = 2 Γ (Precision Γ Recall) / (Precision + Recall)
π Why this matters: The harmonic mean penalizes extreme values more than the arithmetic mean. For example, if Precision=1.0 and Recall=0.0, the arithmetic mean is 0.5 but the harmonic mean is 0.0, correctly indicating poor performance.
E Measure (Parameterized F Measure)
The E Measure is a variant of the F-Measure that allows weighting emphasis on either precision or recall through a parameter Ξ² (beta). This provides flexibility for applications where one metric is more important than the other.
π Definition β E Measure: A parameterized version of the F-Measure that allows controlling the trade-off between precision and recall.
π Formula: E = (1 + Ξ²Β²) Γ (Precision Γ Recall) / (Ξ²Β² Γ Precision + Recall)
π Parameter Interpretation:
- Ξ² = 1: Equal weight to precision and recall (E = F)
- Ξ² > 1: Weight recall more heavily (emphasizes finding all relevant documents)
- Ξ² < 1: Weight precision more heavily (emphasizes finding only relevant documents)
π Example: For a medical diagnosis system with Ξ² = 0.5 (emphasizing precision to avoid false positives), if Precision=0.8 and Recall=0.6, then E = (1+0.25) Γ (0.48) / (0.25Γ0.8 + 0.6) = 1.25 Γ 0.48 / 0.8 = 0.75.
β Key Takeaways
The most critical concepts from this lecture are: (1) Precision and recall form the foundation of IR evaluationβprecision measures accuracy of results while recall measures completeness. (2) Interpolation creates standardized recall/precision curves by taking maximum precision at or beyond each recall level, enabling fair system comparison. (3) Average recall/precision curves across multiple queries provide a robust evaluation of overall system performance, with the best system having a curve closest to the upper-right corner. (4) Specialized metrics like R-Precision, Precision@K, F-Measure, and E-Measure offer different perspectives for evaluation depending on the application needs. (5) The E-Measure with parameter Ξ² allows flexible weighting between precision and recall, making it adaptable to different search scenarios.
π§ Quick Revision Questions
-
How do you compute precision and recall at a given point in a ranked retrieval list, and what do these metrics represent differently?
-
What is the standard interpolation method for recall/precision curves, and why must you use the maximum precision at or beyond a given recall level?
-
How is the average recall/precision curve constructed across multiple queries, and what does a curve in the upper-right corner indicate about system performance?
-
What is the difference between R-Precision and Precision@K, and when would you choose one over the other?
-
How does the E-Measure differ from the F-Measure, and what happens to the emphasis when Ξ² is set to 0.5 versus 2.0?
π Lecture 27 β Mean Average Precision, Non Binary Relevance, DCG, NDCG
π Overview: This lecture covers advanced evaluation metrics for information retrieval systems that go beyond simple precision and recall. It introduces the Mean Average Precision (MAP) for evaluating ranked retrieval results, Mean Reciprocal Rank (MRR) for systems with a single relevant item, and most importantly, graded relevance metrics like Cumulative Gain (CG), Discounted Cumulative Gain (DCG), and Normalized Discounted Cumulative Gain (NDCG) that handle non-binary relevance judgments. These metrics are essential for evaluating modern search engines where relevance is not simply binary.
ποΈ Topics Covered
The lecture covers Mean Average Precision and how it computes average precision across multiple recall points; Mean Reciprocal Rank which measures the rank position of the first relevant document; the concept of Non-Binary Relevance where documents have varying degrees of relevance; Cumulative Gain as a simple sum of relevance scores; Discounted Cumulative Gain which reduces the contribution of documents found at lower ranks; and Normalized Discounted Cumulative Gain which standardizes DCG scores against an ideal ranking for cross-query comparison.
π Lecture Summary
Mean Average Precision (MAP)
Average Precision is the average of the precision values calculated at each point where a relevant document is retrieved. The computation considers only the ranks at which relevant documents appear. For example, if in a search ranking, relevant documents appear at positions 1, 2, 4, 6, 11, and 26, the precision at each of these points is calculated (1.0 at rank 1, 1.0 at rank 2, 0.75 at rank 4, 0.667 at rank 6, 0.38 at rank 11, and 0 at rank 26 because the last document is not retrieved), and these are averaged: (1 + 1 + 0.75 + 0.667 + 0.38 + 0)/6 = 0.633.
In another example with relevant documents at positions 1, 3, 5, 8, 9, and 18, the precisions are 1.0, 0.667, 0.6, 0.5, 0.556, and 0.429, giving an average of (1 + 0.667 + 0.6 + 0.5 + 0.556 + 0.429)/6 = 0.625. Mean Average Precision (MAP) is then the average of these average precision values across a set of queries. If a relevant document never gets retrieved, its corresponding precision is assumed to be zero.
π‘ Why this matters: MAP uses macro-averaging, meaning each query counts equally regardless of how many relevant documents it has. MAP is perhaps the most commonly used evaluation measure in research papers. However, it assumes the user is interested in finding many relevant documents for each query, which may not always be true for web search where users often want just one good result. MAP also requires many relevance judgments in the text collection, making it expensive to compute.
π Definition β Average Precision: The average of precision values computed at each point where a relevant document is retrieved in a ranked list. π Formula: AP = (sum of precision at each relevant rank) / (total number of relevant documents) π Example: For a query with 6 relevant documents retrieved at ranks 1,2,4,6,11,26: AP = (1.0 + 1.0 + 0.75 + 0.667 + 0.38 + 0)/6 = 0.633
Mean Reciprocal Rank
Mean Reciprocal Rank (MRR) focuses on the rank position K of the first relevant document retrieved. This is particularly useful when users are typically looking for a single relevant result, such as in question answering or navigational queries where the user wants the specific website. The Reciprocal Rank for a single query is calculated as 1/K. The Mean Reciprocal Rank (MRR) is then the average of these reciprocal rank scores across multiple queries.
π Definition β Mean Reciprocal Rank (MRR): The average of the reciprocal ranks of the first relevant document across multiple queries. π Formula: MRR = (1/N) * sum(1 / rank_of_first_relevant_doc) for N queries π Example: If for query 1 the first relevant doc is at rank 2 (RR=1/2=0.5), for query 2 at rank 1 (RR=1/1=1.0), and for query 3 at rank 5 (RR=1/5=0.2), then MRR = (0.5 + 1.0 + 0.2)/3 = 0.567
Non-Binary Relevance
Documents are rarely entirely relevant or non-relevant to a query, which is why Non-Binary Relevance assigns graded relevance judgments rather than simple binary (relevant/not relevant) categories. There are many sources of graded relevance judgments, including using a 5-point scale for relevance, having multiple judges evaluate the same document, and analyzing click distribution and deviation from expected click levels. However, the lecture notes caution that click-through data is not equivalent to true relevance judgments.
π Definition β Non-Binary Relevance: A graded or multi-level relevance judgment system where documents can have varying degrees of relevance to a query, rather than being classified as simply relevant or non-relevant.
Cumulative Gain
Cumulative Gain (CG) is the simplest measure for handling graded relevance. It is calculated by summing the relevance scores of documents at each rank position. The gain is accumulated starting at the top of the ranking and moving downward. For example, if documents at the top 10 positions have relevance scores of 3, 2, 3, 0, 1, 2, 3, 2, 1, 0 on a relevance scale, then the cumulative gain at position 10 would be the sum of all these scores.
π Definition β Cumulative Gain (CG): The sum of the graded relevance values of all documents in a ranked list up to a given rank position. π Formula: CG = sum of relevance scores up to rank k π Example: For relevance scores [3, 2, 3, 0, 1] at ranks 1-5, CG at rank 5 = 3 + 2 + 3 + 0 + 1 = 9
Discounted Cumulative Gain
Discounted Cumulative Gain (DCG) improves upon Cumulative Gain by recognizing that users are less likely to examine documents at lower ranks. DCG uses graded relevance as a measure of usefulness, or gain, from examining a document. The gain is accumulated starting at the top of the ranking but is reduced, or discounted, at lower ranks. The typical discount factor is 1/log(rank) using base 2. With base 2, the discount at rank 4 is 1/2 (because logβ(4) = 2), and at rank 8 it is 1/3 (because logβ(8) = 3). This means a relevant document found at position 4 contributes only half as much to the total score as if it were found at position 1.
π‘ Why this matters: Discounting Based on Position reflects real user behavior where users scan results from top to bottom, finding documents lower down less useful even if they are highly relevant. DCG penalizes relevant documents that appear at lower ranks.
π Definition β Discounted Cumulative Gain (DCG): A measure of ranking quality that accumulates gain (relevance) from documents while applying a discount factor that decreases with rank position. π Formula: DCG = sum of (relevance of document at rank i) / logβ(i+1) for all documents up to rank position π Example: For relevance scores [3, 2, 3, 0, 1] at ranks 1-5: DCG = 3/logβ(2) + 2/logβ(3) + 3/logβ(4) + 0/logβ(5) + 1/logβ(6) = 3/1 + 2/1.585 + 3/2 + 0/2.322 + 1/2.585 = 3 + 1.262 + 1.5 + 0 + 0.387 = 6.149
Normalized Discounted Cumulative Gain (NDCG)
Normalized Discounted Cumulative Gain (NDCG) addresses the challenge of comparing DCG values across different queries, which may have different numbers of relevant documents and different relevance distributions. To make comparisons meaningful, DCG values are normalized so that an ideal ranking would have an NDCG of 1.0. The ideal ranking is created by sorting all documents in the collection by their relevance scores in descending order, so the most relevant documents appear first. The NDCG is then calculated as the ratio of the actual DCG to the ideal DCG for the same query.
π‘ Why this matters: NDCG allows fair comparison of search system performance across queries with different characteristics. A perfect ranking gets a score of 1.0, and real rankings get scores between 0 and 1.
π Definition β Normalized Discounted Cumulative Gain (NDCG): DCG normalized by the ideal DCG (DCG of a perfectly ranked list) to produce a score between 0 and 1. π Formula: NDCG = DCG / Ideal DCG π Example: If a query has actual DCG of 6.149 and the ideal DCG (when documents are sorted by descending relevance) is 8.912, then NDCG = 6.149/8.912 = 0.690, meaning the ranking achieves 69% of the ideal performance.
β Key Takeaways
The most critical concept for exams is understanding that MAP handles binary relevance and averages precision across all relevant documents, while MRR only considers the first relevant document's rank. For graded relevance, CG simply adds up relevance scores, DCG applies a logarithmic discount based on rank position to reflect diminishing user attention at lower ranks, and NDCG normalizes DCG against an ideal ranking to enable cross-query comparison. The discount factor for DCG is typically 1/logβ(rank), with base 2. Students must memorize that NDCG always falls between 0 and 1, with 1.0 representing a perfect ranking. The key distinction is that MAP is for binary relevance and macro-averaging across queries, while DCG/NDCG handle graded relevance with position-based discounting.
π§ Quick Revision Questions
- How does the Mean Average Precision (MAP) handle relevant documents that are never retrieved by the search system?
- What is the discount factor used in Discounted Cumulative Gain (DCG) and why does it use a logarithmic function?
- Why is Normalized Discounted Cumulative Gain (NDCG) preferred over raw DCG when comparing search engine performance across different queries?
- What is the fundamental difference between Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) in terms of what each metric measures?
- Why does the lecture caution against using click-through data as a direct substitute for relevance judgments in non-binary relevance evaluations?
π Lecture 28 β Using User Clicks
π Overview: This lecture explores how user click behavior can be used to evaluate and improve Information Retrieval systems. It covers methods for extracting pairwise relevance judgments from click data, metrics for comparing rankings, and practical techniques like interleaved experiments and A/B testing that large search engines use to measure user satisfaction.
ποΈ Topics Covered
The lecture covers what clicks tell us about relevance, relative vs absolute ratings, pairwise relative ratings derived from user clicks, interleaved documents as a method for obtaining pairwise orderings, the Kendall tau distance for comparing rankings, critique of additive relevance in favor of marginal relevance, the Kappa measure for inter-judge agreement, and A/B testing for evaluating search engine innovations.
π Lecture Summary
What do clicks tell us?
User clicks on search results provide implicit feedback about relevance. However, clicks are not absolute relevance judgments; they are relative. For a given query, a user clicking on Result A but not Result B suggests A is preferred, but we cannot conclude that the unclicked result is irrelevant.
π‘ Why this matters: Clicks are abundant and free, unlike manual relevance judgments, but they need careful interpretation.
Relative vs absolute ratings
Absolute ratings (e.g., "Result 1 is relevant") are hard to derive from clicks with certainty. For example, it is difficult to conclude Result1 > Result3 from click data alone. However, relative judgments are more reliable: if a user clicks Result3 but not Result2, we can probably conclude Result3 > Result2.
π Definition β Relative rating: A judgment that one document is better than another for a given query, without requiring an absolute relevance label.
Pairwise relative ratings
These are pairs of the form: DocA is better than DocB for a query. Importantly, this does not mean DocA is relevant to the query β only that it is preferred over DocB. Pairwise preferences shift evaluation from per-doc relevance assessments to conformance with historical pairwise preferences recorded from user clicks.
A critical caution: Do not learn and test on the same ranking algorithm. If you learn historical clicks from one search engine (e.g., "nozama") and then compare another engine (e.g., "Sergey") against that history, the evaluation is biased.
Interleaved docs (Joachims 2002)
One approach to obtain pairwise orderings is to interleave results from two ranking engines A and B. Users are shown a combined list where results from both engines are mixed. Their clicks then implicitly indicate which engine's results are preferred, generating pairwise comparisons.
Kendall tau distance
This metric measures the agreement between two rankings. Let ( X ) be the number of agreements (pairs in the same order) between a ranking A and a reference ranking P, and ( Y ) be the number of disagreements. The Kendall tau distance is:
π Formula: Kendall tau = (X - Y) / (X + Y)
This value ranges from -1 to +1. A value of +1 means perfect agreement, 0 means random agreement, and -1 means complete disagreement.
π Example: Say P = {(1,2), (1,3), (1,4), (2,3), (2,4), (3,4)} and A = (1,3,2,4). P lists all 6 possible pairs in the correct order. For A: pairs (1,2), (1,3), (1,4), (3,2), (3,4), (2,4). Compared to P, we find X=5 agreements and Y=1 disagreement (the pair (3,2) is reversed). Then Kendall tau = (5-1)/(5+1) = 4/6 = 0.667.
Critique of additive relevance
Traditional evaluation assumes additive relevance: the total relevance of a result list is the sum of individual document relevance scores. This is flawed because:
- Relevance vs Marginal Relevance: A document can be redundant even if it is highly relevant (e.g., duplicates, same information from different sources).
- Marginal relevance β the new information a document adds beyond what the user has already seen β is a better measure of utility.
- However, evaluating marginal relevance is harder because it requires considering the entire set of documents shown together.
- This pushes us to assess a slate of results (the whole list) rather than summing individually assessed relevance scores.
- One method: raters are shown two result lists and asked to pick the better one β reminiscent of the interleaved doc idea.
Kappa measure for inter-judge (dis)agreement
The Kappa measure quantifies agreement among judges for categorical judgments, correcting for chance agreement.
π Formula: Kappa = [P(A) β P(E)] / [1 β P(E)]
- P(A) = proportion of time judges agree
- P(E) = what agreement would be by chance
Interpretation: Kappa = 0 for chance agreement, 1 for total agreement. Kappa > 0.8 is considered good agreement; 0.67 < Kappa < 0.8 allows "tentative conclusions" (Carletta '96). For more than 2 judges, average pairwise kappas are used.
π Example: From the contingency table in the lecture, with 400 total judgments:
- P(A) = 370/400 = 0.925
- P(nonrelevant) = (10+20+70+70)/800 = 0.2125
- P(relevant) = (10+20+300+300)/800 = 0.7878
- P(E) = (0.2125)^2 + (0.7878)^2 = 0.665
- Kappa = (0.925 β 0.665) / (1 β 0.665) = 0.260 / 0.335 = 0.776
This value (0.776) falls in the "tentative conclusions" range.
A/B testing
A/B testing is a practical evaluation method for testing a single innovation in a large search engine.
- Prerequisite: You have a large search engine up and running.
- Have most users use the old system.
- Divert a small proportion of traffic (e.g., 1%) to the new system that includes the innovation.
- Evaluate with an automatic measure like clickthrough on the first result (percentage of users that click on the top result or any result on the first page).
- This directly shows if the innovation improves user happiness.
- Probably the evaluation methodology large search engines trust most.
- In principle less powerful than multivariate regression analysis, but easier to understand and implement.
β Key Takeaways
User clicks provide abundant but noisy relative relevance judgments β we can conclude Document A is preferred to B, but not that A is relevant. To avoid bias, pairwise preferences should be learned from one ranking algorithm and tested on a different one, often using interleaved experiments. The Kendall tau distance quantifies agreement between rankings, while additive relevance is critiqued because it ignores redundancy; marginal relevance and slate-based evaluation are superior but harder. The Kappa measure corrects for chance in inter-judge agreement, and A/B testing with clickthrough metrics is the gold standard for evaluating innovations in large-scale search engines.
π§ Quick Revision Questions
- Why is it harder to conclude "Result1 > Result3" than "Result3 > Result2" from click data?
- What is the key caution when using historical pairwise preferences from clicks to compare two ranking algorithms?
- In the Kendall tau example, why was the pair (3,2) considered a disagreement with the reference P?
- Why is marginal relevance considered a better measure of utility than additive relevance, and what makes it harder to evaluate?
- In the Kappa example, what does a Kappa value of 0.776 tell us about the agreement between the two judges?
π Lecture 29 β Cosine Ranking
π Overview: This lecture focuses on efficient computation of cosine-based ranking for information retrieval. It covers strategies to speed up the ranking process by reducing the number of cosine computations and dimensions, while maintaining high-quality results for top-K document retrieval.
ποΈ Topics Covered
The lecture covers computing cosine-based ranking with methods to speed up the process, efficient cosine ranking as a K-nearest neighbor problem, computing the K largest cosines using selection vs. sorting, using a heap for selecting top K documents, understanding cosine similarity as a proxy for user satisfaction, generic approach of pruning non-contenders, index elimination techniques, considering high-idf query terms only, and considering documents containing many query terms.
π Lecture Summary
Computing cosine-based ranking
Speeding up cosine ranking involves reducing the number of cosine computations. This can be achieved through methods like union of term-wise candidates, sampling and pre-grouping, and reducing the number of dimensions using techniques like random projection and latent semantic indexing. The goal is to efficiently find the most relevant documents without computing all possible cosine similarities.
Efficient cosine ranking
This approach effectively solves the K-nearest neighbor problem for a query vector. In general, high-dimensional spaces make this problem computationally difficult. However, for short queries, it becomes solvable because standard inverted indexes can efficiently identify candidate documents containing query terms.
Computing the K largest cosines: selection vs. sorting
Typically, we want to retrieve the top K docs in the cosine ranking, not to order all documents. This raises the question: can we pick off the K docs with the highest cosines without full sorting? Let J represent the number of documents with nonzero cosines. We seek the K best of these J documents. π Formula: J = number of docs with nonzero cosines β K = number of top docs we want from J.
Measuring the distance between 2 vectors
Distance between two vectors in V-dimensional space is based on the angle between them, assuming each vector has unit length. In 2D, this is visualized as angles on a circle, where the cosine of the angle ΞΈ directly measures similarity. π Definition β Cosine similarity: a measure of similarity between two vectors calculated as the cosine of the angle between them, ranging from 0 to 1 for positive vectors, where 1 indicates identical direction.
Use heap for selecting top K
A heap is a binary tree where each node's value is greater than the values of its children (max-heap). Building a heap takes 2J operations, and each of K winners is read off in 2log J steps. For example, with J=1,000,000 and K=100, this is about 10% of the cost of sorting. The heap building process takes O(n) time. Deleting the max element takes constant time, but fixing the rest of the heap takes log(n) time. With n replaced by J, building takes O(J), each deletion takes log(J) time, and for K operations we need K log(J) operations.
Cosine similarity is only a proxy
The user has a task and a query formulation. Cosine similarity matches documents to the query, but it is only a proxy for user happiness. If we retrieve a list of K documents "close" to the top K by cosine measure, that should be acceptable. π‘ Why this matters: The goal is not perfect ranking but good enough retrieval that satisfies the user's information need.
Generic approach
Find a set A of contenders, where K < |A| << N (N is total documents). Set A does not necessarily contain the top K, but has many documents from among the top K. Then return the top K documents in A. This approach prunes non-contenders and is also used for other (non-cosine) scoring functions. Several schemes follow this approach.
Index elimination
The basic cosine computation algorithm only considers documents containing at least one query term. Index elimination takes this further by: only considering high-idf query terms, and only considering documents containing many query terms. π Example: For query "catcher in the rye", only accumulate scores from "catcher" and "rye", ignoring "in" and "the".
High-idf query terms only
For a query such as catcher in the rye, only accumulate scores from catcher and rye. The intuition is that in and the contribute little to the scores and don't alter rank-ordering much. The benefit is that postings of low-idf terms have many documents, so these (many) documents get eliminated from the set A of contenders.
Docs containing many query terms
Any document with at least one query term is a candidate for the top K output list. For multi-term queries, only compute scores for documents containing several of the query terms β for example, at least 3 out of 4. This imposes a "soft conjunction" on queries, as seen on web search engines (early Google). It is easy to implement in postings traversal.
β Key Takeaways
The most critical concept is that computing cosine ranking efficiently requires strategic pruning rather than exhaustive calculation. The heap data structure provides an efficient way to select the top K documents without sorting all candidates, using O(J) to build and O(K log J) to extract winners. Cosine similarity is recognized as a proxy for user satisfaction, not an exact measure of relevance. Index elimination techniques like considering only high-idf terms or requiring documents to contain multiple query terms significantly reduce the candidate set. The generic approach of creating a contender set A with |A| much smaller than N while still capturing most top K documents is the fundamental strategy behind all these methods.
π§ Quick Revision Questions
- Why is cosine similarity described as "only a proxy" for user happiness?
- How does using a heap for selecting top K documents compare in complexity to full sorting?
- What is the benefit of considering only high-idf query terms in cosine ranking?
- What does the "soft conjunction" mean in the context of considering docs containing many query terms?
- What are the two main strategies mentioned for reducing the number of dimensions in cosine ranking?