📘 Lecture 37 — Markov chains
📖 Overview: This lecture introduces Markov chains as a foundational concept for understanding PageRank and web information retrieval. It explains how random surfer models can be used to compute long-term visit rates for web pages, and how these concepts extend to query processing and personalized search results.
🗂️ Topics Covered
The lecture covers Markov chains and their transition probability matrices, ergodic Markov chains and steady-state probability distributions, Markov chains with teleporting using the random surfer model, query processing by integrating PageRank with relevance, personalized PageRank through biased random jumps, and Google PageRank-biased spidering for focused crawling.
📝 Lecture Summary
Markov chains
A Markov chain consists of n states, plus an n × n transition probability matrix P. At each step, we are in one of the states. For 1 ≤ i,j ≤ n, the matrix entry P_ij tells us the probability of j being the next state, given we are currently in state i.
🔑 Definition — Markov chain: A stochastic model consisting of n states and an n × n transition probability matrix P, where each entry P_ij represents the probability of moving from state i to state j in one step.
📐 Formula: P = [P_ij] for 1 ≤ i,j ≤ n → The matrix entry at row i, column j gives the probability of transitioning to state j from state i.
📌 Example: If there are 3 web pages (states A, B, C), and from page A there is a 0.5 probability of going to B and 0.5 to C, then P_AB = 0.5 and P_AC = 0.5 in the transition matrix.
Ergodic Markov chains
For any ergodic Markov chain, there is a unique long-term visit rate for each state. This is called the steady-state probability distribution. Over a long time-period, we visit each state in proportion to this rate. It does not matter where we start.
🔑 Definition — Ergodic Markov chain: A Markov chain that is irreducible (every state can be reached from every other state) and aperiodic (no cycles that force regular patterns), guaranteeing a unique steady-state distribution.
📐 Formula: Steady-state distribution π = πP, where π is a probability vector and P is the transition matrix → The stationary distribution remains unchanged after one transition step.
💡 Why this matters: Ergodic chains ensure that PageRank calculations converge to a unique ranking regardless of the starting point, making web ranking stable and predictable.
Markov Chain with Teleporting: Random Surfer Model
The random surfer model adds a teleporting mechanism to overcome the problem of dead ends (pages with no outgoing links) and spider traps (pages that trap the surfer). At each step, with probability (1 - β), the surfer "teleports" to a random page uniformly, and with probability β, follows an outgoing link. This creates an ergodic Markov chain.
🔑 Definition — Teleporting: A mechanism in the random surfer model where the surfer, with probability (1 - β), jumps to a random page instead of following a link, ensuring all pages are reachable.
📐 Formula: Modified transition matrix: P' = βP + (1 - β)E, where E is a matrix of uniform teleport probabilities (1/n for all entries) → Combines link-following with random jumps to ensure ergodicity.
📌 Example: If β = 0.85 and there are 10 pages, with 85% probability the surfer follows links, and with 15% probability they teleport to any of the 10 pages equally (each with 0.15/10 = 0.015 probability).
Query Processing
The query processing approach involves: first, compute PageRank for all pages in the index; second, run the query against the index to find relevant pages; third, sort the obtained pages based on their PageRank scores; and fourth, integrate PageRank and relevance to produce the final ranking of results.
🔑 Definition — Query Processing: The multi-step process of ranking search results by combining traditional relevance scoring with PageRank importance scores.
📌 Example: For a query "machine learning," the system first retrieves all pages containing these terms, then sorts them by PageRank, and finally blends relevance scores (e.g., TF-IDF) with PageRank using a weighted formula like 0.6 × relevance + 0.4 × PageRank to produce the final result list.
Personalized PageRank
Personalized PageRank can be achieved by changing the teleport distribution E to a non-uniform distribution. This restricts "random jumps" to a set of specified relevant pages. For example, let E(p) = 0 except for one's own home page, for which E(p) = α. This results in a bias towards pages that are closer in the web graph to your own homepage. Another example is using 0.6 × sports + 0.4 × news as the teleport distribution.
🔑 Definition — Personalized PageRank: A modified PageRank algorithm where the teleport probability vector E is customized to reflect personal preferences, biasing the ranking toward pages related to specific topics or user interests.
📐 Formula: Personalized PageRank: R' = βPR' + (1 - β)E', where E' is a non-uniform personalization vector → The teleport distribution is tailored to user-specific interests.
📌 Example: For a sports fan with E(sports_page) = 0.6 and E(news_page) = 0.4 (and 0 for all others), the random surfer teleports to sports pages 60% of the time and news pages 40% of the time, causing sports-related pages to receive higher PageRank scores.
Google PageRank-Biased Spidering
PageRank-biased spidering uses PageRank to direct a spider on "important" pages. The system computes PageRank using the current set of crawled pages. It then orders the spider's search queue based on current estimated PageRank. This approach includes Topic Directed PageRank, where crawling is focused on pages relevant to specific topics.
🔑 Definition — PageRank-Biased Spidering: A web crawling strategy that prioritizes pages with higher estimated PageRank scores, ensuring that important pages are discovered and indexed earlier.
📌 Example: If a spider has two uncrawled URLs—one with estimated PageRank 0.05 and another with 0.001—it will crawl the first page first because of its higher importance score, leading to more efficient resource utilization.
⭐ Key Takeaways
Markov chains provide the mathematical foundation for PageRank, with transition probability matrices modeling the random surfer's behavior across web pages. Ergodic Markov chains guarantee a unique steady-state probability distribution, which is essential for stable PageRank calculations that converge regardless of starting point. The random surfer model with teleporting solves the problems of dead ends and spider traps by allowing random jumps, ensuring the Markov chain is ergodic. Query processing combines PageRank scores with relevance scores to produce final search rankings, balancing importance with topical relevance. Personalized PageRank customizes the teleport distribution to user preferences, biasing results toward specific topics or personal home pages.
🧠 Quick Revision Questions
- What are the two key conditions for a Markov chain to be considered ergodic?
- How does the teleporting mechanism in the random surfer model solve the problem of dead ends and spider traps?
- What is the formula for the modified transition matrix in a Markov chain with teleporting?
- How does personalized PageRank differ from standard PageRank in terms of the teleport distribution?
- What is the purpose of PageRank-biased spidering in web crawling, and how does it prioritize pages?
📘 Lecture 38 — HITS
📖 Overview: This lecture introduces Hyperlink-Induced Topic Search (HITS), an algorithm that models web pages as hubs and authorities based on link analysis. It explains how to compute these scores iteratively, addresses practical limitations, and demonstrates applications like similar-page finding and clustering ambiguous queries.
🗂️ Topics Covered
The lecture covers the HITS algorithm including hubs and authorities definitions and their circular relationship, building a base set from a root set, iterative computation of hub and authority scores with normalization, practical limitations and scaling, proof of convergence using matrix notation, finding similar pages via link structure, using HITS for clustering ambiguous queries, and issues like topic drift and mutually reinforcing affiliates.
📝 Lecture Summary
Hyperlink-Induced Topic Search (HITS)
In response to a query, instead of an ordered list of pages each meeting the query, find two sets of inter-related pages: Hub pages are good lists of links on a subject (e.g., “Bob’s list of cancer-related links”). Authority pages occur recurrently on good hubs for the subject. HITS is best suited for “broad topic” queries (informational queries) rather than page-finding queries, and gets at a broader slice of common opinion.
Hubs and Authorities
A good hub page for a topic points to many authoritative pages for that topic. A good authority page for a topic is pointed to by many good hubs for that topic. This is a circular definition that will be turned into an iterative computation.
Authorities are pages recognized as providing significant, trustworthy, and useful information on a topic. In-degree (number of pointers to a page) is one simple measure of authority, but it treats all links as equal. Should links from pages that are themselves authoritative count more?
Hubs are index pages that provide lots of useful links to relevant content pages (topic authorities). For example, hub pages for information retrieval are included in course home pages.
The hope
High-level scheme: extract from the web a base set of pages that could be good hubs or authorities, then from these identify a small set of top hub and authority pages using an iterative algorithm.
Base set: Given a text query (e.g., “browser”), use a text index to get all pages containing the query term — call this the root set of pages. Add any page that either points to a page in the root set, or is pointed to by a page in the root set. Call this the base set.
Distilling hubs and authorities
Compute, for each page x in the base set, a hub score h(x) and an authority score a(x). Initialize: for all x, h(x)←1; a(x)←1. Iteratively update all h(x), a(x). After iterations, output pages with highest h() scores as top hubs and highest a() scores as top authorities.
Base limitations — to limit computational expense: limit number of root pages to the top 200 pages retrieved for the query; limit number of “back-pointer” pages to a random set of at most 50 pages. To eliminate purely navigational links: eliminate links between two pages on the same host. To eliminate “non-authority-conveying” links: allow only m (m≃4–8) pages from a given host as pointers to any individual page.
Iterative update: Repeat the following updates for all x:
- a(x) ← sum of h(y) over all pages y that link to x
- h(x) ← sum of a(z) over all pages z that x links to
Scaling: To prevent the h() and a() values from getting too big, scale down after each iteration. The scaling factor doesn’t matter because we only care about the relative values of the scores.
How many iterations? Claim: relative values of scores will converge after a few iterations. Suitably scaled, h() and a() scores settle into a steady state. In practice, ~5 iterations get you close to stability.
Results: For query “Java” — authorities include java.sun.com, comp.lang.java FAQ. For query “search engine” — authorities include Yahoo.com, Excite.com, Lycos.com, Altavista.com. For query “Gates” — authorities include Microsoft.com, roadahead.com.
Proof of convergence
n × n adjacency matrix A: each of the n pages in the base set has a row and column in the matrix. Entry Aij = 1 if page i links to page j, else = 0.
Hub/authority vectors: View the hub scores h() and authority scores a() as vectors with n components. Recall the iterative updates: a(x) ← sum of h(y) for y linking to x, and h(x) ← sum of a(z) for z that x links to.
Rewrite in matrix form: Let a = A^T · h and h = A · a, where A is the adjacency matrix. Then a = A^T · A · a and h = A · A^T · h, showing that a is the principal eigenvector of A^T A and h is the principal eigenvector of A A^T.
Finding Similar Pages Using Link Structure
Given a page P, let R (the root set) be t (e.g., 200) pages that point to P. Grow a base set S from R. Run HITS on S. Return the best authorities in S as the best similar-pages for P. Finds authorities in the “link neighborhood” of P.
Similar Page Results: Given “honda.com” — similar pages include toyota.com, ford.com, bmwusa.com, saturncars.com, nissanmotors.com, audi.com, volvocars.com.
HITS for Clustering
An ambiguous query can result in the principal eigenvector only covering one of the possible meanings. Non-principal eigenvectors may contain hubs and authorities for other meanings. Example: query “jaguar” — Atari video game (principal eigenvector), NFL Football team (2nd non-principal eigenvector), Automobile (3rd non-principal eigenvector).
Issues
Topic Drift: Off-topic pages can cause off-topic “authorities” to be returned (e.g., the neighborhood graph can be about a “super topic”).
Mutually Reinforcing Affiliates: Affiliated pages/sites can boost each others’ scores. Linkage between affiliated pages is not a useful signal.
💡 Why this matters: Understanding HITS reveals how early search engines used link structure beyond simple in-degree, and how eigenvector analysis can reveal multiple themes in ambiguous queries through non-principal components.
⭐ Key Takeaways
The HITS algorithm models web pages as hubs (pages pointing to many good sources) and authorities (pages pointed to by many good hubs), computed iteratively with mutual reinforcement. Starting from a query’s root set, a base set is built by including linked and linking pages, then hub and authority scores converge after about 5 iterations via eigenvector computation. Practical limitations require capping root set size (200 pages) and back-links (50 pages), removing same-host navigational links, and limiting cross-host pointers. HITS can find similar pages by analyzing link neighborhoods, and non-principal eigenvectors can disambiguate multiple meanings of ambiguous queries. However, topic drift and mutually reinforcing affiliates can degrade results.
🧠 Quick Revision Questions
- What is the circular definition that connects hubs and authorities in HITS?
- How is the base set constructed starting from a root set?
- Write the mathematical expression for the iterative update of authority scores.
- How many iterations are typically sufficient for HITS to reach stability?
- What issue arises when using HITS for clustering an ambiguous query like “jaguar”?
📘 Lecture 39 — Search Computing
📖 Overview: This lecture introduces Search Computing, a paradigm for answering multi-domain queries that span multiple, distinct data sources (e.g., finding a theatre, a recent thriller movie, and a good steak house nearby). It explains why traditional search engines fail at such tasks, presents the observed trends enabling this approach, and details the architecture and manifesto of Search Computing.
🗂️ Topics Covered
The lecture begins by motivating multi-domain queries with ranking, explains why search engines cannot handle them, and reviews observed trends like Web APIs and linked data. It then defines Search Computing as service composition on demand, presents the Search Computing “Manifesto,” and concludes with the overall Search Computing architecture.
📝 Lecture Summary
Motivation: multi-domain queries with ranking
The lecture opens with a scenario: a user wants to watch a recent thriller at a close theatre and then eat at a good steak house. A single keyword search fails because the required information (movie listings, theatre locations, restaurant reviews) is spread across different web pages. Users must manually split the task, search for theatres, inspect details, find a thriller is missing, then independently search for a steak house, and finally integrate and rank results manually in their brain. This process is inefficient and highlights the need for automated multi-domain search.
Why Search Engines can’t do it?
Traditional search engines fail for two key reasons:
- The query is about distinct domains that should be linked (e.g., movies, locations, restaurants).
- The query deals with multiple rankings that are hard to compute together (e.g., "close" theatre, "recent" thriller, "good" steak house). Although enough data exists on the Web, it is not on a single web page, making integration and ranking impossible for standard search engines.
🔑 Definition — Multi-domain query: A query that spans multiple, conceptually different domains (e.g., entertainment, dining, location).
Observed trends
Several trends enable the Search Computing approach:
- More data sources become accessible through Web APIs (as services), including both surface and deep Web content.
- Data sources are often coupled with search APIs, making them programmatically queryable.
- Publishing of structured and interconnected data is becoming popular (e.g., Linked Open Data). This creates an opportunity to build focused search systems that compose results from several data sources. Such systems are easy-to-build, easy-to-query, easy-to-maintain, and easy-to-scale, covering the functionality of vertical search systems (e.g., Expedia, Amazon) on more focused application domains like localized real estate, leisure planning, or biomedical research.
🔑 Definition — Vertical search systems: Specialized search engines focused on a specific domain (e.g., travel, e-commerce).
Search Computing = service composition “on demand”
Search Computing is defined as the composition of search services on demand. Key aspects of composition abstractions include:
- Service invocations: calling individual data sources.
- Fundamental operations: parallel invocations, joins, pipelining.
- Global constraints on execution: ensuring overall coherence. Data composition should be search-driven, aimed at producing few top results very fast. For example, a query like "a house in a walkable area, close to public transportation, and located in a pleasant neighborhood" requires composing results from real estate, transit, and neighborhood rating services.
🔑 Definition — Search Computing: A paradigm for building global answers by combining data from multiple search services, ranking them according to a global ranking criteria.
The Search Computing “Manifesto”
The manifesto outlines the core goals:
- Build theories, methods, and tools to support search-oriented multi-domain queries.
- Given a multi-domain query over a set of search services, build global answers by combining data from each service.
- Rank global answers according to a global ranking and output results in ranking order.
- Support user-friendly query formulation and browsing of results.
- Include new domains while the search process proceeds.
- Possibly change the relative weight of each partial ranking. This is described as "searching via interactive/dynamic mashups of ranked data sources."
Search Computing architecture: overall view
The architecture is presented in a diagram (not fully described in text). It suggests an overall system where user queries are processed by combining multiple ranked data sources, with components for service invocation, data composition, and global ranking. The architecture supports the dynamic inclusion of new domains and adjustment of ranking weights.
💡 Why this matters: Search Computing directly addresses the limitation of current search engines for complex, real-world queries that require integrating and ranking information from multiple, independent sources.
⭐ Key Takeaways
Search Computing solves the problem of multi-domain queries that traditional search engines cannot handle because the required data is spread across multiple pages and domains. It leverages trends like Web APIs, deep Web data, and linked open data to compose results on demand. The Search Computing Manifesto emphasizes building global ranked answers from multiple services, supporting dynamic domain inclusion, and adjusting ranking weights. The architecture supports service invocations, joins, pipelining, and global constraints to produce top results fast. The ultimate goal is user-friendly query formulation and browsing for complex, multi-faceted searches like leisure planning or real estate.
🧠 Quick Revision Questions
- What are the two main reasons traditional search engines cannot handle multi-domain queries like "find a recent thriller at a close theatre and a good steak house nearby"?
- List three observed trends that enable the Search Computing approach.
- What does "Search Computing = service composition on demand" mean in practice?
- According to the Search Computing Manifesto, what are the six key methods or tools that should be built?
- What is the primary goal of the Search Computing architecture?
📘 Lecture 40 — Top-k Query Processing
📖 Overview: This lecture introduces Top-k Query Processing, a set of techniques for efficiently retrieving the highest-ranked objects from multiple sorted lists. It covers two main algorithms — Fagin’s Algorithm (FA) and the Threshold Algorithm (TA) — and compares their performance in terms of objects seen, random accesses, and buffer requirements. The lecture also explores extensions of TA for scenarios with restricted or impossible random access.
🗂️ Topics Covered
The lecture begins with an introduction to Top-k Query Processing and a Simple Database model. It then presents Fagin’s Algorithm, followed by the Threshold Algorithm as a new idea. A detailed Comparison of Fagin’s and Threshold Algorithm is provided, including discussions on the best algorithm based on instance optimality and aggregation function characteristics. The lecture concludes with extensions of TA for various real-world constraints.
📝 Lecture Summary
Top-k Query Processing
Top-k Query Processing is a technique for finding the top k objects (e.g., documents, items) that have the highest combined score across multiple sorted lists. The goal is to minimize the number of accesses to the database, especially when random access is expensive.
🔑 Definition — Top-k Query: A query that retrieves the k objects with the highest overall score, where the score is computed from multiple attributes via an aggregation function.
Simple Database model
The database model consists of m sorted lists, each containing object IDs and their scores for a specific attribute. Objects have a grade (score) in each list. The overall grade for an object is computed by a monotone aggregation function t that combines the individual grades.
Fagin’s Algorithm
Fagin’s Algorithm (FA) works by doing sorted access in parallel to all m lists until at least k objects have been seen in all lists (i.e., k objects are "common" across all lists). At that point, random access is performed to retrieve the missing grades for those common objects, and the top k are determined.
📌 Example: Suppose we have 2 lists of movies (A and B) sorted by rating. We want top 3 movies. FA reads movies from both lists until 3 movies appear in both lists. Then, it randomly accesses the missing ratings for those 3 movies and computes their total score to select the top 3.
Threshold Algorithm (TA)
The Threshold Algorithm (TA) improves on FA by maintaining a threshold value T, which is the maximum possible grade any unseen object can achieve. TA does sorted access and for each newly seen object, performs random access to get its missing grades and computes its overall grade. It keeps a buffer of the top k objects seen so far. The algorithm stops when the scores of the top k objects are all greater than or equal to the current threshold T.
🔑 Definition — Threshold (T) in TA: The sum of the last scores seen in each sorted list during sorted access. It represents the maximum possible score of any object not yet fully seen.
💡 Why this matters: TA stops earlier than FA because it uses the threshold to know when it is safe to stop, without needing to wait for k common objects.
Comparison of Fagin’s and Threshold Algorithm
The lecture compares FA and TA across several dimensions:
- Objects seen: TA sees fewer objects than FA.
- Stopping point: TA stops at least as early as FA.
- Common objects grade: When k objects are in common in FA, their grades are higher than or equal to the threshold in TA.
- Random accesses: TA may perform more random accesses (m-1 per object) than FA, which does all random accesses only at the end.
- Buffer space: TA requires only bounded buffer space of size k, while FA needs unbounded buffers (to store all seen objects).
- Cost trade-off: TA uses fewer random accesses overall but more random seeks, while FA uses more buffer space.
The best algorithm
The "best" algorithm depends on the middleware cost model and the concept of instance optimality. Instance optimality means that the algorithm performs within a constant factor of the optimal algorithm for any data instance.
Key factors in choosing the best algorithm:
- Wild guesses: Random accesses without prior sorted access.
- Aggregation function characteristics:
- Monotone: t(x1,...,xm) ≤ t(x'1,...,x'm) if xi ≤ x'i for every i
- Strictly monotone: t(x1,...,xm) < t(x'1,...,x'm) if xi < x'i for every i
- Strict: t(x1,...,xm) = 1 precisely when xi = 1 for every i
- Database restrictions: e.g., the distinctness property of objects across lists.
🔑 Definition — Instance optimality: An algorithm A is instance optimal if for every instance I, the cost of A on I is within a constant factor of the optimal algorithm's cost on I.
Extending TA
The lecture presents several extensions of TA for real-world constraints:
- TAz: For restricted sorted access (e.g., distance databases).
- No Random Access Algorithm (NRA): For cases where random access is impossible (e.g., web search engines).
- TAθ: For approximate top-k results (only approximate answers needed).
- Combined Algorithm: A hybrid between TA and NRA that considers the relative costs of random and sorted access.
A Taxonomy of Top-k Joins is also mentioned, which categorizes these algorithms based on access patterns and constraints.
⭐ Key Takeaways
The most critical concepts from this lecture are: Top-k query processing efficiently finds the highest-ranked objects from multiple sorted lists. Fagin’s Algorithm waits until k objects appear in all lists, while the Threshold Algorithm uses a threshold to stop earlier, at the cost of more random accesses. TA requires only bounded buffer space (k), whereas FA needs unbounded buffers. The choice between TA and FA depends on the middleware cost, aggregation function properties (monotone, strictly monotone, strict), and database restrictions like the distinctness property. Extensions like NRA and TAθ adapt the core TA idea for scenarios with restricted random access or approximate results.
🧠 Quick Revision Questions
- What is the main difference between Fagin’s Algorithm (FA) and the Threshold Algorithm (TA) in terms of when they stop?
- In the Threshold Algorithm, what does the “threshold” T represent, and how is it calculated?
- Why might TA perform more random accesses than FA, even though it sees fewer objects?
- What is instance optimality, and why is it important when comparing algorithms like FA and TA?
- Name two extensions of TA and describe the real-world constraints they address.
📘 Lecture 41 — Clustering
📖 Overview: This lecture introduces clustering, an unsupervised learning technique that groups similar documents together. It covers the fundamental concepts, applications in Information Retrieval, issues to consider, similarity/distance measures, and two major clustering approaches: flat (K-means) and hierarchical algorithms.
🗂️ Topics Covered
This lecture covers the definition and purpose of clustering, its applications in IR such as improving search recall, navigation, and visualization. It discusses key issues like representation, similarity, and cluster count, explains hard vs. soft clustering, and provides an in-depth look at the K-means clustering algorithm, its convergence, and seed choice, followed by an introduction to hierarchical clustering.
📝 Lecture Summary
What is clustering?
Clustering is the process of grouping a set of objects into classes of similar objects. Documents within a cluster should be similar, while documents from different clusters should be dissimilar. It is the most common form of unsupervised learning, meaning it learns from raw data without pre-labeled examples, as opposed to supervised learning. Clustering is a common and important task in Information Retrieval and other fields.
🔑 Definition — Clustering: The process of grouping a set of objects into classes of similar objects.
📌 Example: A data set with clear cluster structure shows natural groupings of points that a clustering algorithm would identify.
Applications of clustering in IR
Clustering has several applications in IR:
- Whole corpus analysis/navigation: Providing a better user interface for browsing without typing.
- For improving recall in search applications: Getting better search results, similar to pseudo-relevance feedback.
- For better navigation of search results: Effective “user recall” will be higher.
- For speeding up vector space retrieval: Cluster-based retrieval gives faster search.
For improving search recall
The cluster hypothesis states that documents in the same cluster behave similarly with respect to relevance to information needs. To improve search recall, documents in the corpus are clustered a priori. When a query matches document D, other documents in the cluster containing D are also returned. The hope is that a query for “car” will also return documents containing “automobile” because clustering grouped those documents together.
🔑 Definition — Cluster hypothesis: Documents in the same cluster behave similarly with respect to relevance to information needs.
📌 Example: A search for “car” returns not only documents with the word “car” but also documents containing “automobile” that were grouped into the same cluster.
💡 Why this matters: This technique effectively increases recall by leveraging the semantic relationships discovered by clustering.
Yippy.com is mentioned as an example of a search engine that groups search results into clusters.
Visualization is another application, helping users understand the structure of a document collection.
Issues for clustering
Key issues to consider for clustering include:
- Representation for clustering: How to represent documents (e.g., vector space model, normalization).
- Need a notion of similarity/distance: A method to measure how close or far apart documents are.
- How many clusters?: This number can be fixed a priori or determined completely from the data.
- Avoid “trivial” clusters: Clusters that are too large or too small are undesirable. If a cluster is too large, it wastes a user’s click for navigation without significantly narrowing down the document set.
Notion of similarity/distance
The ideal measure is semantic similarity, but the practical approach uses term-statistical similarity, where documents are treated as vectors. The most common measure is cosine similarity. For many algorithms, it is easier to think in terms of a distance (the opposite of similarity), such as Euclidean distance. However, real implementations often use cosine similarity.
🔑 Definition — Cosine similarity: A measure of similarity between two non-zero vectors that measures the cosine of the angle between them.
Hard vs. soft clustering
- Hard clustering: Each document belongs to exactly one cluster. This is more common and easier to do.
- Soft clustering: A document can belong to more than one cluster. This is more appropriate for applications like creating browsable hierarchies.
📌 Example of Soft Clustering: A pair of sneakers could be put in two clusters: (i) sports apparel and (ii) shoes. This is only possible with a soft clustering approach.
Clustering Algorithms
Two main types of clustering algorithms:
- Flat algorithms: Usually start with a random (partial) partitioning and refine it iteratively. The key example is K-means clustering.
- Hierarchical algorithms: Build a tree of clusters (bottom-up, agglomerative, or top-down, divisive).
Partitioning Algorithms
A partitioning method constructs a partition of n documents into a set of K clusters. Given a set of documents and the number K, the goal is to find a partition of K clusters that optimizes a chosen partitioning criterion. Finding the globally optimal solution is intractable for many objective functions, so effective heuristic methods like K-means and K-medoids algorithms are used.
K-Means
K-means assumes documents are real-valued vectors. Clusters are based on centroids (the center of gravity or mean) of points in a cluster. The reassignment of instances to clusters is based on distance to the current cluster centroids.
🔑 Definition — Centroid: The center of gravity or mean of all points in a cluster, represented by c.
📐 Formula: Centroid c = (1/|S|) * Σ(d in S) d → The centroid is the average position of all documents in the cluster.
Termination conditions for K-means include:
- A fixed number of iterations.
- Document partition unchanged.
- Centroid positions don’t change.
Convergence of K-means is guaranteed because it is a special case of the Expectation Maximization (EM) algorithm. While the number of iterations could be large, in practice it usually isn’t.
Convergence of K-Means
The Residual Sum of Squares (RSS) is a goodness measure of a cluster, defined as the sum of squared distances from the cluster centroid.
- For a cluster j: RSS_j = Σ_i |d_i – c_j|² (sum over all d_i in cluster j)
- For all clusters: RSS = Σ_j RSS_j
The reassignment step monotonically decreases RSS because each vector is assigned to the closest centroid. The recomputation step also monotonically decreases each RSS_j.
🔑 Definition — Residual Sum of Squares (RSS): A measure of cluster goodness, representing the sum of squared distances of all points in a cluster from its centroid.
📐 Formula: RSS = Σ_j Σ_i |d_i – c_j|² → The total RSS is the sum of all cluster RSS values.
Seed Choice
The choice of initial centroids (seeds) can affect the final clustering result.
Hierarchical Clustering
Hierarchical clustering builds a tree-based hierarchical taxonomy, known as a dendrogram, from a set of documents. One approach is the recursive application of a partitional clustering algorithm.
🔑 Definition — Hierarchical clustering: An approach that builds a tree-based hierarchy (dendrogram) of document clusters.
Final word and resources
In clustering, clusters are inferred from the data without human input (unsupervised learning). However, in practice, many choices influence the outcome, including number of clusters, similarity measure, and representation of documents.
- Resources: IIR 16 (except 16.5) and IIR 17.1–17.3.
⭐ Key Takeaways
Clustering is a powerful unsupervised learning technique for grouping similar documents, based on the cluster hypothesis that improves search recall. The lecture distinguishes between hard and soft clustering and covers key similarity measures like cosine and Euclidean distance. The K-means algorithm is the primary flat clustering method, iteratively assigning points to the nearest centroid and updating centroids, with convergence guaranteed because RSS monotonically decreases. Hierarchical clustering offers an alternative approach by building a tree structure (dendrogram) of documents.
🧠 Quick Revision Questions
- What is the difference between supervised and unsupervised learning in the context of clustering?
- Explain the cluster hypothesis and how it can be used to improve search recall.
- What is the difference between hard clustering and soft clustering, and why would you use one over the other?
- Describe the steps of the K-means algorithm and state two termination conditions.
- What is the Residual Sum of Squares (RSS) and why does its monotonic decrease guarantee convergence of K-means?
📘 Lecture 42 — Classification
📖 Overview: This lecture introduces classification techniques in Information Retrieval, focusing on the probabilistic foundation for document ranking and categorization. It explains why probabilities are essential for handling uncertainty in IR, and presents key classification methods including the Naïve Bayes classifier, Bayes’ Rule, Bernoulli Random Variables, and smoothing functions.
🗂️ Topics Covered
The lecture covers the role of probabilities in information retrieval, the document ranking problem, three main classification methods (manual, rule-based, and supervised learning), document classification using Bayes’ Rule, Bernoulli Random Variables for modeling term presence/absence, smoothing functions to handle zero probabilities, and an alternative version of Naïve Bayes using term frequency.
📝 Lecture Summary
Why probabilities in IR?
In traditional IR systems, matching between each document and query is attempted in a semantically imprecise space of index terms. Probabilities provide a principled foundation for uncertain reasoning. We can use probabilities to quantify our uncertainties about document relevance.
💡 Why this matters: Probabilistic models allow IR systems to rank documents by their estimated relevance, moving beyond simple keyword matching.
The document ranking problem
- We have a collection of documents and a user issues a query
- A list of documents needs to be returned
- Ranking method is the core of an IR system: in what order do we present documents to the user?
- We want the “best” document to be first, second best second, etc.
- Idea: Rank by probability of relevance of the document w.r.t. information need
- P(R=1|documentᵢ, query)
Classification Methods (1): Manual classification
- Used by the original Yahoo! Directory, Looksmart, about.com, ODP, PubMed
- Accurate when job is done by experts
- Consistent when the problem size and team is small
- Difficult and expensive to scale
- Means we need automatic classification methods for big problems
Classification Methods (2): Hand-coded rule-based classifiers
- Rules are defined by subject experts
- Commercial systems have complex query languages
- Accuracy can be high if a rule has been carefully refined over time by a subject expert
- Building and maintaining these rules is expensive
Classification Methods (3): Supervised learning
Given:
- A document d
- A fixed set of classes: C = {c₁, c₂, ..., cⱼ}
- A training set D of documents each with a label in C
Determine:
- A learning method or algorithm which will enable us to learn a classifier γ
- For a test document d, we assign it the class γ(d) ∈ C
Bayes’ Rule For Text Classification
- There are “V” terms in the vocabulary. Vocabulary = |V|
- The Document “D” contains “k” words: D = <w₁,w₂,...,wk>
- K << |V|
🔑 Definition — Bayes’ Rule: P(C|D) = P(D|C) × P(C) / P(D)
Where:
- P(C|D) = probability that document D belongs to class C (posterior)
- P(D|C) = probability of observing document D given class C (likelihood)
- P(C) = prior probability of class C
- P(D) = evidence or marginal probability of document D
Bernoulli Random Variables
- There are a₁, a₂, a₃, ..., a_L “L” words which are not present in the document “D” from the vocabulary |V|
- <w₁,w₂,...,wk> = {P(ā₁|C) P(ā₂|C) P(ā₃|C) ... P(ā_L|C)}
- <w₁,w₂,...,wk> = [1 – {P(a₁|C) P(a₂|C) P(a₃|C) ... P(a_L|C)}]
Training Set of Documents
| Doc ID | Cheap | Buy | Banking | Dinner | The | Class |
|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 0 | 1 | Not spam |
| 2 | 1 | 0 | 1 | 0 | 1 | Spam |
| 3 | 0 | 0 | 0 | 0 | 1 | Not spam |
| 4 | 1 | 0 | 1 | 0 | 1 | Spam |
| 5 | 1 | 1 | 0 | 0 | 1 | spam |
| 6 | 0 | 0 | 1 | 0 | 1 | Not spam |
| 7 | 0 | 1 | 1 | 0 | 1 | Not spam |
| 8 | 0 | 0 | 0 | 0 | 1 | Not spam |
| 9 | 0 | 0 | 0 | 0 | 1 | Not spam |
| 10 | 1 | 1 | 0 | 1 | 1 | Not spam |
The numbers (0 or 1) indicate absence or presence of each term in the document.
Smoothing Function
Smoothing is used to handle the problem of zero probabilities when a term does not appear in any training document of a particular class. Without smoothing, a missing term would make the entire probability zero.
📐 Formula: Add-one smoothing (Laplace smoothing): P(t|c) = (count(t,c) + 1) / (∑count(t',c) + |V|) → Plain English: Add 1 to every term count to avoid zero probabilities
Another version of Naïve Bayes
- In the previous version we consider the document frequency of a term (i.e., in how many documents is it present)
- In this newer version we consider term frequency in the document and in the corpus
Naïve Bayes Characteristics
- Underflow error may occur in both cases when we implement it on a machine
- We take log of both sides which converts multiplication into addition and avoids underflow
- Feature Selection: cut down the size of vocabulary
- Consider the subset of terms, i.e., drop rare terms and stop words
⭐ Key Takeaways
Probabilistic classification is essential for modern IR because it provides a principled way to handle uncertainty in document-query matching. The Naïve Bayes classifier uses Bayes’ Rule with a strong independence assumption to compute the probability of a document belonging to a class. Three classification paradigms exist: manual (expert-based, expensive to scale), rule-based (accurate but expensive to maintain), and supervised learning (automatic, scalable with training data). Bernoulli models treat terms as binary (present/absent), while the alternative version considers term frequency. Smoothing (like add-one smoothing) is critical to prevent zero probabilities from making entire calculations invalid, and log-transformation prevents underflow errors in computation.
🧠 Quick Revision Questions
- What is the core idea behind ranking documents using probabilities?
- List the three main classification methods discussed in this lecture and one advantage of each.
- Write Bayes’ Rule for text classification and explain each term.
- Why is smoothing necessary in Naïve Bayes classification?
- What is the difference between the Bernoulli version of Naïve Bayes and the version that uses term frequency?
📘 Lecture 43 — Classification
📖 Overview: This lecture covers classification techniques in information retrieval, focusing on vector space methods. It explains how to classify documents based on their similarity to prototype vectors or nearest neighbors, and discusses the trade-offs between different algorithms, including Rocchio classification and k-Nearest Neighbors.
🗂️ Topics Covered
The lecture covers Rocchio classification using centroids and prototype vectors, the k-Nearest Neighbor (kNN) algorithm including its lazy learning paradigm, decision boundaries for kNN, and the important concept of bias vs. variance in model selection. It also includes a discussion on evaluating categorization performance.
📝 Lecture Summary
Classification Using Vector Spaces
In vector space classification, the training set is represented as a labeled set of points (vectors) in a multidimensional space. The core premise is that documents in the same class form a contiguous region of space, while documents from different classes don’t overlap much. Learning a classifier involves building surfaces to delineate classes in this space.
💡 Why this matters: This geometric view allows us to use distance and similarity measures to automatically determine which class a new document belongs to.
Rocchio Algorithm
The Rocchio algorithm adapts relevance feedback methods for text categorization. It uses standard tf-idf weighted vectors to represent documents. For each category, a prototype vector (centroid) is computed by summing the vectors of all training documents in that category. A test document is assigned to the category with the closest prototype vector based on cosine similarity.
The algorithm works by finding the centroid of members of each class. However, one issue is that if there are many more vectors along one axis (e.g., y-axis) in a class, they will drift the centroid toward that axis, potentially misclassifying an "awkward" red document that lies near the blue centroid.
🔑 Definition — Rocchio classification: A classification method that computes a prototype vector (centroid) for each class and assigns test documents to the nearest centroid using cosine similarity.
📐 Formula: μ(c) = (1/|D_c|) * Σ( d in D_c ) v(d) → The centroid μ of class c is the average of all document vectors in that class.
📌 Example: If class "Government" has two documents with vectors [1,0] and [0.5,0.5], the centroid would be the average: [0.75, 0.25]. A test document with vector [0.8, 0.2] would be classified as "Government" because it is closest to this centroid by cosine similarity.
k Nearest Neighbor Classification
Nearest-Neighbor Learning is also called case-based learning or memory-based learning because it simply stores all labeled training examples. For a test instance x under 1NN, it computes similarity between x and all examples, then assigns x the category of the most similar example.
k Nearest Neighbor (kNN) improves robustness by finding the k closest examples and returning the majority category of these k neighbors. k is typically odd (3 and 5 are most common) to avoid ties. The rationale is the contiguity hypothesis: documents near a given input document will decide its class.
🔑 Definition — k Nearest Neighbor (kNN): A lazy learning algorithm that classifies a new document based on the majority class of its k most similar training examples.
📌 Example: With k=3, if a test document's three nearest neighbors are from classes "Sports", "Sports", and "Politics", the test document is classified as "Sports" (majority vote).
kNN decision boundaries
kNN decision boundaries are non-linear and adapt to the local structure of data. Unlike Rocchio, kNN tends to handle polymorphic categories better because it doesn't assume a single prototype can represent an entire class.
kNN Discussion:
- Advantages: No feature selection necessary, no training required, scales well with large number of classes, and is more accurate than NB or Rocchio in most cases.
- Disadvantages: May be expensive at test time (must compare to all training examples).
Evaluating Categorization
Evaluation must be done on test data that are independent of the training data. Sometimes cross-validation is used (averaging results over multiple training and test splits). It is easy to get good performance on a test set that was available to the learner during training (e.g., just memorize the test set).
Bias vs. variance: Choosing the correct model capacity
This concept addresses the trade-off between bias (error due to overly simplistic model assumptions) and variance (error due to model sensitivity to small fluctuations in training data). A model with high bias may underfit, while one with high variance may overfit. The goal is to find the correct model capacity that minimizes total error.
🔑 Definition — Bias vs. Variance: Bias measures how far a model's predictions are from correct values on average, while variance measures how much predictions vary for different training sets. The optimal model balances these two sources of error.
⭐ Key Takeaways
The most critical concepts from this lecture are understanding Rocchio classification using centroids versus kNN using multiple neighbors. Rocchio is cheap to train but may fail with polymorphic categories, while kNN is more accurate but expensive at test time. The contiguity hypothesis underpins kNN, and k=3 or 5 with majority voting handles noise better than 1NN. Finally, evaluation must use independent test data, and the bias-variance trade-off is essential for choosing model complexity.
🧠 Quick Revision Questions
- What is a prototype vector in Rocchio classification, and how is it computed?
- Why does kNN use k=3 or 5 instead of just 1 nearest neighbor?
- According to the contiguity hypothesis, what determines the class of a test document?
- Why is kNN considered a "lazy learning" algorithm?
- What is the trade-off between bias and variance in model selection?