๐ Lecture 30 โ Sampling and pre-grouping
๐ Overview: This lecture covers techniques for efficient information retrieval by pre-computing term-wise candidate lists and using sampling-based pre-grouping methods. These approaches reduce query processing time by narrowing the search space to a small set of promising documents rather than scanning the entire collection.
๐๏ธ Topics Covered
The lecture examines term-wise candidate preprocessing where each termโs nearest documents are pre-computed and stored as preferred lists. It then discusses storage strategies for these preferred lists and how to handle cases where preferred lists fail to produce enough results. The main focus is on sampling and pre-grouping using leaders and followers, along with general variants that allow multiple leader attachments and recursive construction.
๐ Lecture Summary
Term-wise candidates
Preprocess: Pre-compute, for each term, its k nearest docs.
- Treat each term as a 1-term query.
- Requires lots of preprocessing.
- Result: "preferred list" for each term.
Search:
- For a t-term query, take the union of their t preferred lists โ call this set S.
- Compute cosines from the query to only the docs in S, and choose top k.
Preferred List storage
- Store postings in descending order of similarity to the term.
- This would disturb the postings merging, as it wonโt be stored in a sorted order of DocID.
- Store separate postings list, which is the preferred list.
- Store the postings in the ascending order of DocID.
- This would keep the merging of posting lists easy.
- Requires extra storage.
Number of Required Results:
- In case the preferred lists are not able to produce required number of top results, go to the normal postings lists.
- Generate estimated number of total results against a query by using total number of postings against the terms.
- Produce top results by using preferred lists.
Sampling and pre-grouping
First run a pre-processing phase:
- Pick โN docs at random: call these leaders.
- For each other doc, pre-compute nearest leader.
- Docs attached to a leader: its followers.
- Likely: each leader has ~ โN followers.
Process a query as follows:
- Given query Q, find its nearest leader L.
- Seek k nearest docs from among L's followers.
๐ก Why this matters: This technique dramatically reduces the number of documents that must be scored during query processing, from the entire collection (N documents) to only about โN followers of a single leader.
Why use random sampling:
- Fast
- Leaders reflect data distribution
- Random sampling is computationally efficient and statistically representative.
Visualization: The lecture includes a diagram showing leaders (represented as larger icons) with their followers (smaller icons grouped around each leader), and a query Q with arrows pointing to its nearest leader L.
General variants
- Have each follower attached to a=3 (say) nearest leaders.
- From query, find b=4 (say) nearest leaders and their followers.
- Can recur on leader/follower construction.
๐ก Why this matters: Multiple attachments improve recall by ensuring a queryโs nearest documents arenโt missed if theyโre followers of a leader that isnโt the queryโs absolute nearest. Recursive construction allows hierarchical partitioning for even faster search.
โญ Key Takeaways
The lecture presents two main strategies for efficient approximate nearest neighbor search in information retrieval. Term-wise candidates pre-compute preferred lists for each term, requiring union operations at query time and extra storage. Sampling and pre-grouping randomly selects โN documents as leaders, assigns each remaining document to its nearest leader as a follower, and at query time only searches followers of the queryโs nearest leader. General variants improve this by attaching each follower to multiple nearest leaders and finding multiple nearest leaders at query time, while recursion enables hierarchical clustering. Both methods dramatically reduce the number of documents that need cosine computations, trading some precision for significant speed gains.
๐ง Quick Revision Questions
- How many leaders are selected in the sampling and pre-grouping approach, and roughly how many followers does each leader typically have?
- What is the difference between storing a preferred list in descending order of similarity versus ascending order of DocID?
- When would a retrieval system need to fall back from using preferred lists to using normal posting lists?
- In the general variant with multiple attachments, what do the parameters a=3 and b=4 represent?
- Why is random sampling preferred over other methods for selecting leaders in the pre-grouping approach?
๐ Lecture 31 โ Dimensionality reduction
๐ Overview: This lecture explores techniques for reducing the number of dimensions in vector space representations of documents, which speeds up cosine similarity computations and improves retrieval efficiency. It covers two main approachesโrandom projection and Latent Semantic Indexing (LSI)โand explains the underlying linear algebra concepts, including Singular-Value Decomposition (SVD) and eigenvalue analysis.
๐๏ธ Topics Covered
The lecture covers two dimensionality reduction methods: random projection onto fewer axes and its computation, and Latent Semantic Indexing (LSI) which is data-dependent. It then reviews notions from linear algebra including matrix rank, eigenvectors, eigenvalues, and Singular-Value Decomposition (SVD). Finally, it discusses dimension reduction by zeroing out all but the largest eigenvalues, document-document similarities, and query processing.
๐ Lecture Summary
Dimensionality reduction
What if we could take our vectors and โpackโ them into fewer dimensions (say 10000โ100) while preserving distances? (Well, almost.) This speeds up cosine computations. Two methods: Random projection and โLatent semantic indexingโ.
Random projection onto k<<m axes.
- Choose a random direction xโ in the vector space.
- For i = 2 to k, choose a random direction xแตข that is orthogonal to xโ, xโ, ... xแตขโโ.
- Project each doc vector into the subspace xโ, xโ, ... xโ.
Guarantee: With high probability, relative distances are (approximately) preserved by projection. Pointer to precise theorem in Resources.
Computing the random projection
Projecting n vectors from m dimensions down to k dimensions:
- Start with m ร n matrix of terms ร docs, A.
- Find random k ร m orthogonal projection matrix R.
- Compute matrix product W = R ร A.
- jth column of W is the vector corresponding to doc j, but now in k << m dimensions.
Cost of computation: This takes a total of kmn multiplications. Expensive - see Resources for ways to do essentially the same thing, quicker.
- Exercise: by projecting from 10000 dimensions down to 100, are we really going to make each cosine computation faster? Size of the vectors would decrease. Will result into smaller postings.
Latent semantic indexing (LSI)
Another technique for dimension reduction. Random projection was data-independent. LSI on the other hand is data-dependent. It eliminates redundant axes and pulls together โrelatedโ axesโfor example, car and automobile.
Notions from linear algebra: Matrix, vector, matrix transpose and product, rank, eigenvalues and eigenvectors.
The matrix
The matrix
has rank 2: the first two rows are linearly independent, so the rank is at least 2, but all three rows are linearly dependent (the first is equal to the sum of the second and third) so the rank must be less than 3.
- The matrix
has rank 1: there are nonzero columns. So the rank is positive, but any pair of columns is linearly dependent.
Similarly, the transpose of A has rank 1. Indeed, since the column vectors of A are the row vectors of the transpose of A, the statement that the column rank of a matrix equals its row rank is equivalent to the statement that the rank of a matrix is equal to the rank of its transpose, i.e., rk(A) = rk(Aแต).
Singular-Value Decomposition: Recall m ร n matrix of terms ร docs, A. A has rank r โค m,n.
- Define term-term correlation matrix T = AAแต
- Aแต denotes the matrix transpose of A.
- T is a square, symmetric m ร m matrix.
Eigenvectors: Denote by P the m ร r matrix of eigenvectors of T. Denote by R the n ร r matrix of eigenvectors of D. It turns out A can be expressed (decomposed) as A = PQRแต. Q is a diagonal matrix with the eigenvalues of AAแต in sorted order.
๐ Definition โ Eigenvectors and Eigenvalues: For a square matrix, eigenvectors are non-zero vectors that, when multiplied by the matrix, result in a scalar multiple of themselves. The scalar is called the eigenvalue.
๐ก Why this matters: This decomposition allows us to identify which dimensions capture the most variance in the document-term relationships, enabling effective compression.
Dimension reduction
For some s << r, zero out all but the s biggest eigenvalues in Q. Denote by Qโ this new version of Q. Typically s in the hundreds while r could be in the (tens of) thousands. Let Aโ = P Qโ Rแต. Turns out Aโ is a pretty good approximation to A.
Guarantee: Relative distances are (approximately) preserved by projection. Of all m ร n rank s matrices, Aโ is the best approximation to A. Pointer to precise theorem in Resources.
Doc-doc similarities: Aโ Aโแต is a matrix of doc-doc similarities: the (j,k) entry is a measure of the similarity of doc j to doc k.
Semi-precise intuition: We accomplish more than dimension reduction here:
- Docs with lots of overlapping terms stay together
- Terms from these docs also get pulled together.
- Thus car and automobile get pulled together because both co-occur in docs with tires, radiator, cylinder, etc.
Query processing
View a query as a (short) doc: call it row 0 of Aโ. Now the entries in row 0 of Aโ Aโแต give the similarities of the query with each doc. Entry (0,j) is the score of doc j on the query. Exercise: fill in the details of scoring/ranking.
- LSI is expensive in terms of computation...
- Randomly choosing a subset of documents for dimensional reduction can give a significant boost in performance.
โญ Key Takeaways
The two primary methods for dimensionality reduction in information retrieval are random projection (data-independent, fast, preserves distances with high probability) and Latent Semantic Indexing (data-dependent, uses SVD to capture term co-occurrence patterns). Random projection reduces dimensions by projecting vectors onto a random k-dimensional subspace, requiring kmn multiplications. LSI reduces dimensions by zeroing out all but the s largest eigenvalues in the singular-value decomposition of the term-document matrix, with s typically in the hundreds. LSI not only reduces dimensions but also semantically pulls together related terms like car and automobile, improving query-document matching. Both methods preserve relative distances approximately, but LSI requires expensive SVD computation, though performance can be boosted by using a random subset of documents.
๐ง Quick Revision Questions
- What are the two main methods for dimensionality reduction covered in this lecture, and how do they differ in their data dependence?
- In random projection, what is the cost of projecting n vectors from m dimensions down to k dimensions, and what is the formula for the matrix product?
- In LSI, what is the role of eigenvalues and the diagonal matrix Q in the Singular-Value Decomposition (SVD) of the term-document matrix A?
- How does the dimension reduction step work in LSI: what does Aโ = P Qโ Rแต represent, and what is the guarantee about Aโ?
- How does LSI pull together related terms like "car" and "automobile," and how is a query processed using the resulting doc-doc similarity matrix?
๐ Lecture 32 โ Web Search
๐ Overview: This lecture explores the evolution and challenges of web search, from the early days of the World Wide Web to modern search engines. It covers the historical development of browsers and search engines, the unique challenges the web poses for information retrieval, the graph structure of the web, and the business models that power web search.
๐๏ธ Topics Covered
The lecture covers the World Wide Web, its pre-history, and the history of web browsers and search engines. It then addresses the specific challenges the web presents for information retrieval, including distributed, volatile, and large-volume data. The graph structure of the web is analyzed using the Bowtie model, and Zipf's law is applied to web page characteristics. The lecture concludes with a discussion of manual hierarchical web taxonomies and various business models for web search.
๐ Lecture Summary
The World Wide Web
The World Wide Web was developed by Tim Berners-Lee in 1990 at CERN to organize research documents available on the Internet. It combined the idea of documents available by FTP with the concept of hypertext to link documents. Berners-Lee developed the initial HTTP network protocol, URLs, HTML, and the first โweb server.โ
Web Pre-History
The concept of hypertext was first developed by Ted Nelson in 1965. Doug Engelbart invented the mouse and built the first implementation of hypertext in the late 1960โs at SRI. The ARPANET was developed in the early 1970โs. While the basic technology was in place in the 1970โs, it took the PC revolution and widespread networking to inspire the web and make it practical.
Web Browser History
Early browsers like Erwise and ViolaWWW were developed in 1992. In 1993, Marc Andreessen and Eric Bina at UIUC NCSA developed the Mosaic browser and distributed it widely. Andreessen joined with James Clark to form Mosaic Communications Inc. in 1994, which became Netscape. Microsoft later licensed the original Mosaic from UIUC to build Internet Explorer in 1995.
Search Engine Early History
By the late 1980โs, many files were available by anonymous FTP. In 1990, Alan Emtage of McGill Univ. developed Archie (short for โarchivesโ), which assembled lists of files available on many FTP servers and allowed regex search of these file names. In 1993, Veronica and Jughead were developed to search names of text files available through Gopher servers.
Web Search History
In 1993, early web robots (spiders) were built to collect URLโs, including Wanderer, ALIWEB, and the WWW Worm. In 1994, Stanford grad students David Filo and Jerry Yang started manually collecting popular web sites into a topical hierarchy called Yahoo. Later that year, Brian Pinkerton developed WebCrawler as a class project, and Fuzzy Maudlin developed Lycos, which was the first to use a standard IR system and index a large set of pages. In late 1995, DEC developed Altavista, which supported boolean operators, phrases, and "reverse pointer" queries.
Web Search Recent History
In 1998, Larry Page and Sergey Brin, Ph.D. students at Stanford, started Google. Its main advance was the use of link analysis to rank results based partially on authority.
Web Challenges for IR
The web presents several challenges for information retrieval:
- Distributed Data: Documents are spread over millions of different web servers.
- Volatile Data: Many documents change or disappear rapidly (e.g., dead links).
- Large Volume: There are billions of separate documents.
- Unstructured and Redundant Data: No uniform structure exists, with HTML errors and up to 30% (near) duplicate documents.
- Quality of Data: There is no editorial control, leading to false information, poor quality writing, and typos.
- Heterogeneous Data: Multiple media types (images, video, VRML), languages, and character sets are present.
Zipfโs Law on the Web
Zipf's Law applies to the web in several ways:
- Number of in-links/out-links to/from a page has a Zipfian distribution.
- Length of web pages has a Zipfian distribution.
- Number of hits to a web page has a Zipfian distribution.
Manual Hierarchical Web Taxonomies
The Yahoo approach used human editors to assemble a large hierarchically structured directory of web pages (http://www.yahoo.com/). The Open Directory Project (dmoz.org) is a similar approach based on the distributed labor of volunteer editors, where โnet-citizens provide the collective brain.โ It was started by Netscape and is used by most other search engines.
Business Models for Web Search
Several business models are used for web search:
- Advertisers pay for banner ads on the site that do not depend on a userโs query.
- CPM (Cost Per Mille): Pay for each ad display (thousand impressions).
- CPC (Cost Per Click): Pay only when a user clicks on an ad.
- CTR (Click Through Rate): Fraction of ad impressions that result in clicks throughs. CPC = CPM / (CTR * 1000)
- CPA (Cost Per Action/Acquisition): Pay only when a user actually makes a purchase on the target site.
- Advertisers bid for keywords. Ads for the highest bidders are displayed when a user query contains a purchased keyword.
- PPC (Pay Per Click): CPC for bid word ads (e.g., Google AdWords).
โญ Key Takeaways
The web evolved from Tim Berners-Lee's initial system through key milestones like the Mosaic browser and the early search engines such as Archie, Yahoo, and Lycos. Google's major innovation was its use of link analysis (PageRank) to rank results by authority, moving beyond simple text matching. The web presents unique challenges for IR, including distributed, volatile, massive, unstructured, low-quality, and heterogeneous data. The graph structure of the web is not arbitrary; it follows the Bowtie model and exhibits Zipfian distributions for links, page length, and traffic. Finally, web search is funded by advertising business models like CPM, CPC, and CPA, with keyword bidding being a central mechanism.
๐ง Quick Revision Questions
- Who developed the World Wide Web, and what core technologies did they create?
- What was the main innovation that Google introduced for web search ranking?
- List four specific challenges that the web presents for information retrieval.
- According to the lecture, what characteristics of the web follow a Zipfian distribution?
- Explain the difference between CPM, CPC, and CPA in the context of web search advertising.
๐ Lecture 33 โ Spidering
๐ Overview: This lecture focuses on the fundamental techniques for web crawling, also known as spidering, which is the process by which search engines automatically discover and download web pages. It covers the essential protocols and strategies that crawlers must follow to be polite, robust, and efficient, including search strategies for traversing the web graph, mechanisms for avoiding duplicate pages, and methods for respecting website owner restrictions via the Robots Exclusion Protocol and META tags.
๐๏ธ Topics Covered
This lecture covers web search using IR, defines spiders (robots/bots/crawlers) and their basic operations, outlines what any crawler must do (polite and robust) and what any crawler should do (distributed, scalable, high-quality). It explains search strategies comparing breadth-first and depth-first search, presents the basic crawl architecture including the spidering algorithm and queueing strategy, covers avoiding page duplication, and details methods for restricting spidering through Robot Exclusion Protocol and Robots META Tags.
๐ Lecture Summary
Spiders (Robots/Bots/Crawlers)
Spiders are automated programs that systematically browse the World Wide Web to discover and download web pages. They start with a comprehensive set of root URLs from which to begin the search. The spider then follows all links on these pages recursively to find additional pages. As novel pages are encountered, they are indexed, typically in an inverted index. Spiders may also allow users to directly submit pages to be indexed, which then become part of the crawl starting set.
What any crawler must do
A crawler must be Polite by respecting both implicit and explicit politeness considerations. This includes only crawling allowed pages and respecting the robots.txt file. A crawler must also be Robust, meaning it must be immune to spider traps and other malicious behavior from web servers that could cause it to get stuck or waste resources.
What any crawler should do
An ideal crawler should be capable of distributed operation, designed to run on multiple distributed machines. It should be scalable, designed to increase the crawl rate simply by adding more machines. It must have high performance/efficiency to permit full use of available processing and network resources. Additionally, it should fetch pages of higher quality first, support continuous operation by fetching fresh copies of previously fetched pages, and be extensible to adapt to new data formats and protocols.
Search Strategy Trade-Offโs
There is a fundamental trade-off between breadth-first and depth-first search strategies. Breadth-first explores uniformly outward from the root page but requires memory of all nodes on the previous level, which is exponential in depth, making it the standard spidering method. Depth-first requires memory of only depth times branching-factor (linear in depth) but can get "lost" pursuing a single thread. Both strategies are implementable using a queue of links (URLs).
๐ Definition โ Breadth-first search: Explores all nodes at the present depth level before moving on to nodes at the next depth level. ๐ Definition โ Depth-first search: Explores as far as possible along each branch before backtracking.
Avoiding Page Duplication
Since the web is a graph, not a tree, a crawler must detect when it is revisiting a page that has already been spidered. It must efficiently index visited pages to allow rapid recognition. One method is to index pages using the URL as a key, using data structures like a tree index (e.g., trie) or a hashtable. URLs must be canonicalized (e.g., deleting ending "/") to treat equivalent URLs as the same page. However, this method does not detect duplicated or mirrored pages with different URLs. An alternative is to index pages using textual content as a key, but this requires first downloading the page before determining if it is a duplicate.
Spidering Algorithm
The spidering algorithm is a step-by-step process for downloading and indexing web pages:
- Initialize a queue (Q) with an initial set of known URLs.
- Until the queue is empty or a page/time limit is exhausted:
- Pop a URL, L, from the front of Q.
- If L is not to an HTML page (e.g., .gif, .jpeg, .ps, .pdf, .ppt...), continue the loop.
- If L has already been visited, continue the loop.
- Download page, P, for L.
- If P cannot be downloaded (e.g., 404 error, robot excluded), continue the loop.
- Index P (e.g., add to inverted index or store cached copy).
- Parse P to obtain a list of new links N.
- Append N to the end of Q.
Queueing Strategy
How new links are added to the queue determines the search strategy. Using FIFO (First-In, First-Out), appending to the end of the queue, gives breadth-first search. Using LIFO (Last-In, First-Out), adding to the front of the queue, gives depth-first search. Heuristically ordering the queue gives a "focused crawler" that directs its search towards "interesting" pages based on specific criteria.
๐ Definition โ Focused crawler: A crawler that uses heuristics to prioritize the fetching of pages that are likely to be of higher quality or relevance to a specific topic.
Restricting Spidering
Spidering can be restricted in several ways: restricting the spider to a particular site by removing links to other sites from the queue; restricting the spider to a particular directory by removing links not in the specified directory; and obeying page-owner restrictions through robot exclusion.
Link Extraction
To follow links, a crawler must find all links in a page and extract their URLs. This includes <a href="URL"> tags and <frame src="URL"> tags. The crawler must also complete relative URLs using the current page's URL. For example, <a href="proj3"> relative to http://www.cs.utexas.edu/users/mooney/ir-course/ becomes http://www.cs.utexas.edu/users/mooney/ir-course/proj3. Another example: <a href="../cs343/syllabus.html"> becomes http://www.cs.utexas.edu/users/mooney/cs343/syllabus.html.
URL Syntax
A URL has the following syntax: <scheme>://<authority><path>?<query>#<fragment>. An authority has the syntax <host>:<port-number>. A query passes variable values from an HTML form and has the syntax <variable>=<value>&<variable>=<value>.... A fragment, also called a reference or ref, is a pointer within the document to a point specified by an anchor tag of the form <A NAME="<fragment>">.
Robot Exclusion
Web sites and pages can specify that robots should not crawl or index certain areas. This involves two components: the Robots Exclusion Protocol, which is a site-wide specification of excluded directories, and the Robots META Tag, which is an individual document tag to exclude indexing or following links.
๐ Definition โ Robots.txt: A file placed at the root of a host's web directory (e.g., http://www.ebay.com/robots.txt) that lists excluded directories for a given robot (user-agent). ๐ Example: To exclude all robots from the entire site, the robots.txt file contains:
User-agent: *
Disallow: /
๐ Example: To exclude specific directories from all robots:
User-agent: *
Disallow: /tmp/
Disallow: /cgi-bin/
Disallow: /users/paranoid/
๐ Example: To exclude a specific robot (e.g., GoogleBot):
User-agent: GoogleBot
Disallow: /
๐ Example: To allow a specific robot while excluding all others:
User-agent: GoogleBot
Disallow:
User-agent: *
Disallow: /
Robots META Tag
This META tag is included in the HEAD section of a specific HTML document. The format is <meta name="robots" content="value">. The content value is a pair of values for two aspects: index | noindex (allow/disallow indexing of this page) and follow | nofollow (allow/disallow following links on this page).
- Special values: all = index,follow; none = noindex,nofollow
๐ Example:
<meta name="robots" content="noindex,follow">โ allows following links but does not index the page. ๐ Example:<meta name="robots" content="index,nofollow">โ indexes the page but does not follow its links. ๐ Example:<meta name="robots" content="none">โ neither indexes the page nor follows its links.
โญ Key Takeaways
The most critical concepts from this lecture for the exam are the fundamental differences between breadth-first and depth-first search strategies for crawling, including their memory requirements and the risk of getting "lost". You must understand the complete spidering algorithm, the role of the queue (FIFO for breadth-first, LIFO for depth-first), and how to canonicalize URLs and avoid page duplication. The two components of the Robot Exclusion protocolโthe site-wide robots.txt file and the per-page robots META tagโwith their specific syntax and values (User-agent, Disallow, index/noindex, follow/nofollow) are essential. Finally, recognizing spider traps, understanding URL syntax (scheme, authority, path, query, fragment), and being able to complete relative URLs are all examinable skills.
๐ง Quick Revision Questions
- What are the two mandatory behaviors every crawler must exhibit, and what does each entail?
- Compare breadth-first and depth-first search strategies in terms of memory requirements and the risk of getting "lost".
- Explain the difference between the Robots Exclusion Protocol and the Robots META Tag, and provide an example of a robots.txt file that blocks all robots from the entire site.
- What is the purpose of URL canonicalization, and what is one common example of this process?
- Describe the spidering algorithm step-by-step, including what happens when a non-HTML file or a page that cannot be downloaded is encountered.
๐ Lecture 34 โ Web Crawler
๐ Overview: This lecture focuses on the architecture and operation of web crawlers, essential tools for information retrieval that systematically browse the web to collect pages. It covers crawling politeness, freshness, the Mercator URL frontier scheme, and techniques for detecting duplicate and near-duplicate documents.
๐๏ธ Topics Covered
The lecture begins with basic crawl architecture and communication between nodes. It then discusses the URL frontier with a focus on politeness and freshness, followed by a detailed explanation of the Mercator scheme including front queues, back queues, and back queue heap. Finally, it covers duplicate document detection, specifically near-duplicate detection using shingles and set intersection.
๐ Lecture Summary
Basic crawl architecture
The lecture introduces the basic architecture of a web crawler. The output of the URL filter at each node is sent to the Dup URL Eliminator of the appropriate node. This ensures that duplicate URLs are removed before crawling.
Communication between nodes
The communication flow involves passing filtered URLs from one node to another for duplicate elimination. This coordination is crucial for efficient distributed crawling.
URL frontier: two main considerations
The URL frontier manages the queue of URLs to be crawled. Two main considerations guide its design:
- Politeness: Do not hit a web server too frequently to avoid overloading it.
- Freshness: Crawl some pages more often than others, e.g., pages (such as News sites) whose content changes often.
๐ก Why this matters: These goals may conflict each other. For example, a simple priority queue fails because many links out of a page go to its own site, creating a burst of accesses to that site.
Politeness โ challenges
Even if we restrict only one thread to fetch from a host, it can still hit the host repeatedly. A common heuristic is to insert a time gap between successive requests to a host that is much larger than the time for the most recent fetch from that host.
URL frontier: Mercator scheme
The Mercator URL frontier is a sophisticated scheme to handle both politeness and freshness. It uses two sets of queues:
- Front queues: Manage prioritization
- Back queues: Enforce politeness
- Each queue is First-In-First-Out (FIFO)
URLs flow in from the top into the frontier.
Front queues
The Prioritizer assigns to each URL an integer priority between 1 and K and appends the URL to the corresponding queue. Heuristics for assigning priority include:
- Refresh rate sampled from previous crawls
- Application-specific rules (e.g., โcrawl news sites more oftenโ)
Back queue invariants
Back queues maintain three invariants:
- Each back queue is kept non-empty while the crawl is in progress
- Each back queue only contains URLs from a single host
- A table is maintained from hosts to back queues
Biased front queue selector
When a back queue requests a URL, the biased front queue selector picks a front queue from which to pull a URL. This choice can be:
- Round robin biased to queues of higher priority
- Some more sophisticated variant
- Randomized
Back queue heap
The back queue heap contains one entry for each back queue. The entry is the earliest time tโ at which the host corresponding to the back queue can be hit again. This earliest time is determined from:
- Last access to that host
- Any time buffer heuristic we choose
Back queue processing
When a crawler thread seeks a URL to crawl:
- Extracts the root of the heap
- Fetches URL at head of corresponding back queue q (look up from table)
- Checks if queue q is now empty โ if so, pulls a URL v from front queues
- If thereโs already a back queue for vโs host, append v to q and pull another URL from front queues, repeat
- Else add v to q
- When q is non-empty, create heap entry for it
Number of back queues B
To keep all threads busy while respecting politeness, the Mercator recommendation is three times as many back queues as crawler threads.
Duplicate documents
The web is full of duplicated content.
- Strict duplicate detection = exact match โ not as common
- But many, many cases of near duplicates, e.g., Last modified date the only difference between two copies of a page
๐ Definition โ Duplicate Detection: Identifying two or more documents that are identical or nearly identical in content.
Duplicate/Near-Duplicate Detection
- Duplication: Exact match can be detected with fingerprints
- Near-Duplication: Approximate match
- Overview: Compute syntactic similarity with an edit-distance measure. Use a similarity threshold to detect near-duplicates.
- E.g., Similarity > 80% => Documents are โnear duplicatesโ
- Not transitive though sometimes used transitively
Computing Similarity
Features used for similarity computation:
- Segments of a document (natural or artificial breakpoints)
- Shingles (Word N-Grams)
- Example: "a rose is a rose is a rose" โ 4-grams are:
- a_rose_is_a
- rose_is_a_rose
- is_a_rose_is
- a_rose_is_a
- Example: "a rose is a rose is a rose" โ 4-grams are:
๐ Formula: Similarity Measure between two docs (= sets of shingles) โ Set Intersection. Specifically: Size_of_Intersection / Size_of_Union
Shingles + Set Intersection
Computing exact set intersection of shingles between all pairs of documents is expensive and intractable. Instead, approximate using a cleverly chosen subset of shingles from each document (a sketch). Estimate (size_of_intersection / size_of_union) based on a short sketch.
Sketch of a document
Create a โsketch vectorโ (of size ~200) for each document. Documents that share โฅ t (say 80%) corresponding vector elements are deemed near duplicates.
For document D, sketch_D[i] is computed as follows:
- Let f map all shingles in the universe to 0..2^m (e.g., f = fingerprinting)
- Let ฯ_i be a random permutation on 0..2^m
- Pick MIN {ฯ_i(f(s))} over all shingles s in D
Final notes
- Shingling is a randomized algorithm
- The analysis did not presume any probability model on the inputs
- It will give us the right (wrong) answer with some probability on any input
- Weโve described how to detect near duplication in a pair of documents
- In โreal lifeโ weโll have to concurrently look at many pairs โ Use Locality Sensitive Hashing for this
โญ Key Takeaways
The Mercator URL frontier is essential for balancing politeness (not overwhelming servers) and freshness (crawling frequently updated pages). It uses two tiers of queues: front queues for priority and back queues for host-specific politeness, managed by a heap of earliest access times. For duplicate detection, exact matches are identified with fingerprints, while near-duplicates use shingle-based similarity (word n-grams) computed via set intersection. Since exact set intersection is expensive, a sketch vector of about 200 elements is used, comparing the minimum of randomly permuted shingle fingerprints. Locality Sensitive Hashing is the practical method for scaling near-duplicate detection across many document pairs concurrently.
๐ง Quick Revision Questions
- What are the two main considerations (goals) of a URL frontier, and why might they conflict?
- In the Mercator scheme, what is the role of front queues versus back queues?
- How does the back queue heap determine when a host can be accessed again?
- What is a shingle in the context of near-duplicate detection, and how is similarity measured between two documents using shingles?
- How is a sketch vector for a document created, and what threshold is typically used to deem documents as near-duplicates?
๐ Lecture 35 โ Distributed Index
๐ Overview: This lecture explores how large-scale information retrieval systems handle indexing across multiple machines, including distributed indexing using MapReduce, dynamic indexing strategies for updating documents, and the challenges of maintaining collection statistics in dynamic environments. It provides essential techniques for building scalable search engines.
๐๏ธ Topics Covered
The lecture covers distributed indexing concepts, the MapReduce framework for parallel index construction, mapping IR tasks onto Hadoop MapReduce, dynamic indexing approaches including simple and logarithmic merge strategies, and key issues such as maintaining collection statistics and handling deletions with invalidation bit-vectors.
๐ Lecture Summary
Distributed Index
In large-scale IR systems, the inverted index is not stored on a single machine due to storage and performance limitations. Instead, it is distributed across multiple machines in cluster computing or grid computing environments. Large-scale IR systems typically operate data centers containing thousands of machines.
Map Reduce
MapReduce is a programming model that utilizes several machines to construct indexes in parallel. It consists of two phases:
- Map Phase: Takes documents and parses them. Documents can be split into shards based on criteria such as alphabetical ranges (e.g., a-h, i-p, q-z).
- Reduce Phase: Performs inversion โ building the inverted index from the mapped data.
Apache Hadoop is an open-source framework for implementing MapReduce. Each phase of an IR system can be mapped to the MapReduce paradigm.
๐ Definition โ MapReduce: A programming model for processing large datasets in parallel across distributed clusters, consisting of Map (parsing/filtering) and Reduce (aggregation/inversion) phases.
Mapping onto Hadoop Map Reduce
The Map phase takes documents and produces (TermId, DocId) pairs. The Reducer receives these pairs and, based on a defined mapping criterion (e.g., DocId ranges or alphabetical groups like a-g), assigns pairs to different reducers. The reducers then sort the information by DocId and generate postings lists for different terms.
๐ Formula: Map โ (TermId, DocId) pairs โ Reducer โ Sorted postings lists
Dynamic Indexing
Documents in a dynamic corpus are frequently updated or deleted. A simple approach involves:
- Maintaining a big main index on disk.
- Storing new documents in an auxiliary small index built in RAM.
- Answering queries by searching both indexes and merging results.
- For deletions: Use invalidation bit-vectors for deleted documents. The size of the bit-vector equals the number of documents in the corpus. Search results are filtered using this bit-vector.
- Periodically merge the main and auxiliary indexes.
๐ก Why this matters: Without dynamic indexing, every document update or deletion would require rebuilding the entire index, which is computationally infeasible for large collections.
Dynamic Indexing (Logarithmic Merge)
The logarithmic merge strategy improves merge efficiency. Indexes are sized as powers of two:
- Z0: index in memory
- I1, I2, I3...: indexes of increasing sizes (2ยน, 2ยฒ, 2ยณ...)
When Z0 exceeds a threshold 'n' (number of tokens), it merges with I1. If I1 already exists, the merged result becomes I1, and Z0 resets. This continues, with Z0 growing and merging into progressively larger indexes.
Searching involves querying all existing indexes. If there are T terms in the index, there will be at most log(T) indexes. Query processing requires O(log T) merges.
๐ Formula: Number of indexes โค logโ(T) where T = total terms ๐ Example: If there are 1,000,000 terms, there will be at most logโ(1,000,000) โ 20 indexes.
Issues in Dynamic Indexing
Key challenges include:
- Collection Statistics: IDF (Inverse Document Frequency) is harder to maintain with multiple indexes because term frequency across the entire corpus is distributed.
- Spelling Correction: Traditional spelling correction assumes a single unified index.
- Invalidation bit-vectors must be incorporated when computing word statistics.
Solutions include:
- A simple heuristic: use the LARGEST index for statistics, though it reflects inaccurate information.
- Large search engines build a new index on separate machines and switch to it once complete.
- Google Dance: The historical practice of Google periodically updating its entire index.
๐ก Why this matters: Accurate collection statistics are critical for ranking relevance; maintaining them in dynamic environments is a complex engineering challenge.
โญ Key Takeaways
Students must understand that distributed indexing using MapReduce enables parallel construction of inverted indexes across clusters. Dynamic indexing requires strategies like maintaining an auxiliary index in RAM with periodic merges to handle document updates and deletions. The logarithmic merge approach keeps merge costs at O(log T) by maintaining indexes of exponentially increasing sizes. Key challenges include maintaining accurate collection statistics like IDF and handling deletions via invalidation bit-vectors. Large search engines solve these issues by building complete new indexes on separate machines before switching over.
๐ง Quick Revision Questions
- What are the two phases of MapReduce, and what does each phase do in the context of building an inverted index?
- How does the logarithmic merge strategy keep the number of indexes at most log(T)?
- What is an invalidation bit-vector, and how is its size determined?
- Why is IDF harder to maintain in a dynamic indexing environment with multiple indexes?
- What is the "Google Dance" and what problem does it solve?
๐ Lecture 36 โ Link Analysis
๐ Overview: This lecture shifts focus from content-based retrieval to link analysis, exploring how hyperlinks between web pages serve as powerful sources of authority and ranking. It introduces foundational concepts like the web as a directed graph, anchor text indexing, and PageRank scoring, which are critical for modern search engines and information retrieval.
๐๏ธ Topics Covered
The lecture covers hypertext and links and their role in authority, the web as a directed graph with adjacency lists and gap encodings, anchor text indexing for improved search accuracy, citation analysis concepts, and PageRank scoring with teleportation to handle dead-ends, along with resource references.
๐ Lecture Summary
Hypertext and Links
This section explains that we move beyond document content to examine hyperlinks between documents. Links address questions about authority (conferral of authority to pages), relevance (e.g., a page linked from CERN likely about high energy physics), and are applicable to the web, email, and social networks. Links are powerful sources of authenticity and authority used for spam detection, host quality assessment, and analysis of phone call logs.
The lecture introduces the Good, Bad, Unknown model: Good nodes wonโt point to Bad nodes. All other combinations (GoodโGood, GoodโUnknown, BadโBad, BadโGood, BadโUnknown, Unknownโany) are plausible. Simple iterative logic is applied: if you point to a Bad node, youโre Bad; if a Good node points to you, youโre Good. Sometimes probabilistic analogs are needed (e.g., for mail spam). Primary interests are link analysis for IRโscoring, ranking, clustering (topical structure from links), classification (linked documents likely on same subject), and crawling (where to go next based on links).
The Web as a Directed Graph
The web is represented as a directed graph where web pages are nodes and hyperlinks are directed edges. This graph structure enables analysis of connectivity, authority flow, and ranking.
Anchor Text
Anchor text is the visible, clickable text in a hyperlink. The WWW Worm (McBryan, 1994) used anchor text to distinguish between pages about a topic like "IBM": IBMโs home page (mostly graphical), IBMโs copyright page (high term frequency for "ibm"), and a rivalโs spam page (arbitrarily high term frequency). This shows how anchor text provides better context than raw page content.
๐ Definition โ Anchor Text: The clickable text in a hyperlink that describes the target page, used as a feature for indexing and ranking.
๐ Example: For "ibm," anchor text like "IBM Home Page" helps identify the real IBM site even if the target page is mostly graphical with low text content, while a spam page with artificially high "ibm" frequency can be identified as less relevant.
Indexing anchor text
When indexing a document D, include (with some weight) anchor text from links pointing to D. This improves search accuracy because anchor text often describes the target page more reliably than its own content.
Adjacency lists
Adjacency lists represent the set of neighbors of a node. Assume each URL is represented by an integer (e.g., 32 bits for 4 billion pages). Naively, each hyperlink needs 64 bits, but Boldi/Vigna achieved ~3 bits/link on average, with further work reducing to 2 bits/link.
Main ideas of Boldi/Vigna: Consider a lexicographically ordered list of all URLs (e.g., www.stanford.edu/alchemy, www.stanford.edu/biology, etc.). This ordering enables efficient compression.
Gap encodings: Given a sorted list of integers x, y, z..., represent by x, y-x, z-y... Compress each integer using a code:
- ฮณ code: Number of bits = 1 + 2โlg xโ
- ฮด code: Number of bits = ...
- Information theoretic bound: 1 + โlg xโ bits
- ฮถ code: Works well for integers from a power law (Boldi Vigna DCC 2004)
Citation Analysis
Citation analysis includes:
- Citation frequency: How often a document is cited
- Bibliographic coupling frequency: Articles that co-cite the same articles are related
- Citation indexing: Who is this author cited by? (Garfield 1972)
- Pagerank preview: Pinsker and Narin (1960s) asked which journals are authoritative
The web differs from scholarly citation: millions of self-interested participants, widespread spamming, and link farms (groups of websites that heavily link to one another) complicate analysis.
PageRank scoring
PageRank scoring imagines a browser doing a random walk on web pages:
- Start at a random page
- At each step, go out of the current page along one of the links on that page, equiprobably
- โIn the long runโ each page has a long-term visit rateโuse this as the pageโs score
Not quite enough: The web is full of dead-ends where the random walk can get stuck, making long-term visit rates meaningless.
Teleporting solves this:
- At a dead end, jump to a random web page
- At any non-dead end, with probability 10%, jump to a random web page
- With remaining probability (90%), go out on a random link
- 10% is a parameter
Result of teleporting: Cannot get stuck locally; there is a long-term rate at which any page is visited. This visit rate is computed to derive the PageRank score.
๐ก Why this matters: PageRank was the foundational algorithm that made Googleโs search results significantly better than competitors by leveraging link structure rather than just text content.
โญ Key Takeaways
A student must remember that link analysis uses hyperlinks as sources of authority, with the "Good, Bad, Unknown" model and iterative logic for classifying nodes. The web as a directed graph enables PageRank scoring via random walk with teleportation (10% jump probability) to handle dead-ends, producing long-term visit rates as page scores. Anchor text indexing improves search by including link descriptions when indexing target pages. Adjacency lists with gap encodings (ฮณ, ฮด, ฮถ codes) efficiently compress link data. Citation analysis concepts (bibliographic coupling, citation indexing) preceded web link analysis but the web differs due to spam and self-interested participants.
๐ง Quick Revision Questions
- What is the "Good, Bad, Unknown" model, and what rule governs Good nodes regarding Bad nodes?
- How does teleporting solve the dead-end problem in PageRank scoring, and what is the typical teleportation probability used?
- Explain how anchor text indexing works and why it is more reliable than raw page content for some queries.
- What are gap encodings, and which codes (ฮณ, ฮด, ฮถ) are mentioned for compressing adjacency lists?
- How does citation analysis differ from web link analysis regarding participant behavior and spam?