π Lecture 36 β Running Time Analysis, Union by Size, Union by Height, Sprucing up Find, Timing with Optimization
π Overview: This lecture focuses on optimizing the disjoint set (union-find) data structure. It introduces union by size and union by height to keep trees shallow, and path compression in
findto flatten the tree for faster future lookups. The goal is to achieve nearly linear time performance for a sequence of union and find operations.
ποΈ Topics Covered
The lecture covers the running time analysis of union and find operations on up-trees. It then introduces union by size (weight) to maintain shallow trees, analyzes its depth guarantee (O(log n)), and presents union by height as an alternative. The key optimization of path compression in the find operation is explained in detail with figures, and the overall timing with all optimizations is stated via a theorem involving the inverse Ackermann function.
π Lecture Summary
Running Time Analysis
In the previous lecture, disjoint sets and up-trees implemented via arrays were discussed. While union is a constant-time operation, the running time of find(i) is proportional to the height of the tree containing node i. In the worst case, this height can be n (e.g., performing unions on a sorted list: union(1,2), union(2,3), etc., creates a tree of height n). The goal is to modify union to keep heights small, ensuring find operations are fast. Balancing techniques from binary search trees are not suitable here due to array implementation complexity, so a simpler method called Union by Size is used.
Union by Size
This method maintains the size (number of nodes) of every tree. During a union, the smaller tree is made a subtree of the larger one. In the implementation, for each root node i, instead of setting parent[i] to -1, we set it to -k if the tree rooted at i has k nodes. This allows the find loop's terminating condition (checking for a negative value) to still work correctly.
π Definition β Union by Size (or Union by Weight): A strategy for merging two disjoint sets where the root of the tree with fewer nodes is made a child of the root of the tree with more nodes.
π‘ Why this matters: This ensures the tree's height does not grow as quickly, directly improving the performance of find.
π Algorithm for union(i, j) based on size:
root1 = find(i);
root2 = find(j);
if (root1 != root2)
if (parent[root1] <= parent[root2]) // root1 has more or equal nodes (more negative)
parent[root1] += parent[root2]; // update size
parent[root2] = root1; // attach smaller tree
else // root2 has more nodes
parent[root2] += parent[root1]; // update size
parent[root1] = root2; // attach smaller tree
π Example: Consider an array of 8 elements, all initially -1 (singleton trees). union(4,6): Both trees have size 1. The condition parent[4] <= parent[6] is true (both are -1). The tree with root 4 absorbs node 6. parent[4] becomes -2 (size 2), and parent[6] becomes 4. union(1,4): The tree with root 1 has size 1 (-1), and tree with root 4 has size 2 (-2). Since root4's tree is larger (-2 <= -1), node 1 becomes a child of node 4. The new tree at root 4 has size 3 (-3). This process maintains a tree of height 2.
Analysis of Union by Size
If unions are done by weight (size), the depth of any element is never greater than logβ n.
- Intuitive Proof: An elementβs depth only increases when it is in the smaller tree during a union. In such a union, the resulting tree is at least twice as large as the element's previous tree. Since the treeβs size doubles each time the elementβs depth increases, the depth can increase at most logβ n times before the tree contains all n elements. This is a significant improvement from a worst-case depth of n.
Union by Height
This is an alternative to union-by-size. The strategy maintains the height of each tree. During a union, the tree with a smaller height is made a subtree of the one with a larger height. The implementation details are left as an exercise, but the concept is very similar to union-by-size and is essentially equivalent in terms of performance.
Sprucing up Find
This optimization focuses on the find operation using path compression. During find(i), as we traverse the path from node i to the root, we update the parent entry of every node on that path to directly point to the root.
π Definition β Path Compression: A technique used in the find operation that flattens the tree by making every node on the path from a node to the root become a direct child of the root.
π Algorithm for find(i) with path compression (recursive):
find(i) {
if (parent[i] < 0) // i is the root
return i;
else
return parent[i] = find(parent[i]); // compress path
}
π Example: In a deep tree, calling find(1) traverses nodes 1 -> 2 -> 9 -> 4 -> 13 -> 7 (root). The recursive call reaches the root (7) and then returns. As the recursion unwinds, the call find(13) sets parent[13] = 7, find(4) sets parent[4] = 7, find(9) sets parent[9] = 7, and so on. After this single find, nodes 1, 2, 4, 9, and 13 all have their parent set directly to the root (7). This drastically reduces the height of the tree for future find operations on any of these nodes.
π‘ Why this matters: This is a "pay now, reap the benefits later" optimization. A single find operation may do more work, but subsequent find operations on compressed paths become nearly constant time.
Timing with Optimization
Theorem: A sequence of m union and find operations, n of which are find operations, can be performed on a disjoint-set forest with union by rank (weight or height) and path compression in worst-case time proportional to (m * Ξ±(n)), where Ξ±(n) is the inverse Ackermann function. For all practical purposes, Ξ±(n) β€ 4, which means the average time per operation is essentially constant. The union-find structure is therefore extremely efficient for a sequence of m operations, achieving nearly linear time in m.
β Key Takeaways
- To improve
findperformance in up-trees, we must keep tree heights small by optimizing theunionoperation. - Union by Size/Weight and Union by Height are simple strategies that guarantee the depth of any node is at most O(log n), a huge improvement from the potential O(n) worst case.
- Path compression in the
findoperation further flattens the tree, making subsequent finds significantly faster by directly connecting nodes on the traversal path to the root. - By combining union by rank (size or height) with path compression, a sequence of
munion and find operations runs in essentially O(m) time (linear), making it one of the most efficient data structures for maintaining disjoint sets. - The inverse Ackermann function confirms the incredible efficiency of the union-find algorithm, which is practical for real-world applications requiring many set operations.
π§ Quick Revision Questions
- What is the goal of "union by size/weight" and how does it achieve this?
- In union by size, what value is stored in the parent array of a root node, and why is it negative?
- What is the maximum possible depth of any node after performing unions strictly by size, and why?
- Explain how the path compression technique in the
findoperation changes the structure of a tree. - What is the approximate worst-case running time (in terms of m and n) for a sequence of m union and find operations using union by rank and path compression, and what function makes it so efficient?
π Lecture 37 β Image Segmentation and Maze Example
π Overview: This lecture explores practical applications of disjoint sets and the union/find algorithm. It demonstrates how these data structures can efficiently segment digital images based on pixel intensity thresholds and generate random mazes by systematically removing walls between cells until a path exists from entrance to exit.
ποΈ Topics Covered
The lecture begins with a review of union/find optimization techniques, including union by size and path compression. It then introduces image segmentation as a key application, explaining how pixel intensities are thresholded to create binary images and how union/find groups connected regions. The maze generation problem is presented next, where cells are initially isolated by walls and randomly merged using union operations until the entrance and exit cells belong to the same set. A pseudo code algorithm for maze generation is provided, followed by step-by-step pictorial illustrations of the process.
π Lecture Summary
Review
The lecture reviews union and find methods with special reference to optimization. Union by size or union by weight reduces tree size, while path optimization in the find method further reduces tree traversal. The time required by the find/union algorithm is proportional to m+n when there are m unions and n finds. Union is a constant time operation that links two trees, whereas find involves tree traversal. Disjoint sets increase as the forest decreases.
Image Segmentation
Image segmentation divides an image into different parts based on pixel intensity. An image is a collection of pixels, each with a value representing intensity (e.g., 0 for black, 255 for white). Pixels can be grouped using a threshold valueβfor example, replacing all pixel values β₯ 4 with 1 and values < 4 with 0 to create a binary image. The lecture demonstrates this with a 5Γ5 grid of pixel values (0, 2, 4) representing different gray levels.
π Definition β Threshold: A cutoff value used to classify pixels into two groups (e.g., above or below the threshold). π Example: Given a 5Γ5 matrix with values 0, 2, and 4, applying threshold = 4 replaces all 4s with 1 and all other values with 0, producing a binary matrix. Using the union/find algorithm, we group adjacent 1s into disjoint sets to identify connected regions in the image.
The union/find algorithm processes each row of the matrix. Initially, there are 25 individual sets (one per pixel). When two adjacent pixels both have value 1, a union operation combines their sets. This process continues across all rows, eventually forming disjoint sets of connected 1s. For example, in the first row, two 1s at columns 3 and 4 are unioned; in the second row, 1s at columns 2 and 3 are unioned with the set above, merging into a set of four 1s. Continuing this process yields two disjoint sets of 1sβone large set on the right side and a smaller set on the left.
When the threshold is changed to 2, more pixels become 1s, and the union operation merges them into a single large blob of connected 1s.
π‘ Why this matters: The union/find algorithm segments images quickly without requiring extra memoryβall processing occurs in a single array representing the up-tree.
Maze Example
A maze is a puzzle where a user enters from one side and finds a path to the exit, with many paths leading to dead ends. Maze generation can be done using the disjoint sets algorithm.
π Definition β Equivalence relation: Two cells are equivalent if they can be reached from each other (walls removed so there is a path from one to the other).
Consider a 5Γ5 grid with 25 cells numbered 0 to 24. Each cell is isolated by walls from others. The entrance is cell 0 and the exit is cell 24. The algorithm randomly removes walls until the entrance and exit cells are in the same set. Removal of a wall is equivalent to a union operation. A randomly chosen wall is not removed if the cells it separates are already in the same set.
π Example: Initially, there are 25 sets, each containing one cell. Randomly choose cell 11 and its right wall, merging cell 11 and cell 12 into one set {11,12}. Then choose cell 6 and its bottom wall, merging cell 6 with the set containing 11 and 12 to form {6,11,12}. Continue this processβwhen cell 8 and its top wall (cell 3) are chosen and are in different sets, they are merged into set {3,8}. This continues until cells 0 and 24 eventually belong to the same set, indicating a path exists from entrance to exit.
Pseudo Code of the Maze Generation
The pseudo code for the MakeMaze function takes a size parameter and generates a maze of that size.
MakeMaze(int size) {
entrance = 0;
exit = size-1;
while (find(entrance) != find(exit)) {
cell1 = randomly chosen cell
cell2 = randomly chosen adjacent cell
if (find(cell1) != find(cell2)) {
knock down wall between cells
union(cell1, cell2)
}
}
}
The while loop continues until the find operations on entrance and exit return the same set. Inside the loop, two adjacent cells are randomly chosen. If they are in different sets, the wall between them is removed and union is applied to combine them into one set.
π Example: In the 5Γ5 grid, the loop might first choose cell 11 and cell 12 (adjacent horizontally). Since find(11) and find(12) return different sets initially, the wall is removed and union is performed. Later, cell 6 and cell 11 might be chosen; if they are in different sets, the wall is removed and they merge. The loop stops when cell 0 and cell 24 are in the same set.
β Key Takeaways
The union/find algorithm efficiently segments images by grouping adjacent pixels with similar intensity values using thresholding, requiring no extra memory beyond the up-tree array. Maze generation becomes straightforward using the same algorithm: randomly remove walls between cells, applying union operations until the entrance and exit cells belong to the same set. The algorithm is simpleβa while loop checks if entrance and exit are in the same set, and if not, randomly selects adjacent cells to union if they are in different sets. This demonstrates the power of disjoint sets for solving real-world problems involving connectivity and equivalence relations.
π§ Quick Revision Questions
- How does the union/find algorithm segment a binary image into connected regions?
- In maze generation, what condition causes the while loop to terminate?
- Why should a wall not be removed if the two cells it separates are already in the same set?
- What is the role of the threshold value in image segmentation, and how does changing it affect the result?
- How many sets initially exist in a 5Γ5 maze generation problem, and what happens to this number as walls are removed?
π Lecture 38 β Tables and Dictionaries
π Overview: This lecture introduces the Table Abstract Data Type (ADT) and Dictionaries, fundamental data structures for organizing information in rows and columns. It covers the core operations on tables and their implementations using unsorted and sorted sequential arrays, culminating in the binary search algorithm for efficient data retrieval. Understanding these concepts is crucial for database systems, compiler design, and any application requiring organized data storage and retrieval.
ποΈ Topics Covered
The lecture begins by defining Tables and Dictionaries as abstract data types consisting of rows (records) and columns (fields), with a key field uniquely identifying each record. It then details the three primary operations on the Table ADT: insert, find, and remove. Two implementations are discussed: the unsorted sequential array, where insertion is fast but searching is slow, and the sorted sequential array, where searching becomes fast using binary search but insertion requires data shifting. The lecture concludes with a detailed explanation of the binary search algorithm and its efficiency in O(log n) time.
π Lecture Summary
Tables and Dictionaries
A table, as an abstract data type, is a collection of rows and columns of information. Unlike a simple two-dimensional array, a table can have columns of different data types (e.g., integer, string). Each column is known as a field, and each row is called a record or tuple. For example, a telephone directory has three fields: name, address, and phone number. Another example is a bank account table with fields like account number, account title, account type, and balance. Tables are widely used in databases for applications like payroll software, and in compilers where symbol tables store information about program variables, including their name, type, and scope.
The key is a special field used to uniquely identify an entry in a table. In databases, this is known as the primary key. Only the key value is needed to find, insert, or delete a specific record. For example, in a telephone directory, the name field serves as the key, and searching for "Imran Ahmad" returns the complete record containing the address and phone number.
π Definition β Key: A field in a table that uniquely identifies a record (entry), ensuring that no two records have the same key value.
π‘ Why this matters: The concept of a key is foundational for efficient data retrieval in databases and data structures, as it allows operations to target specific records without scanning the entire table.
Operations on Table ADT
insert This method is used to add a new record to the table. It requires both the key and the entry (the complete record data). The insert operation places the key and its associated fields into the table.
find This method searches for an entry in the table given a key value. It locates the complete record that matches the provided key. For example, in an employees table where employee id is the key, providing id 15466 returns the entire employee record.
remove This method is given a key value to find and delete the associated entry from the table.
Implementation of Table
The choice of implementation for the Table ADT depends on several factors:
- How often entries are inserted, found, and removed
- How many key values are likely to be used
- The likely pattern of searching for keys
- Whether the table fits in memory
- How long the table will exist
In a table, it is best practice to store the key and the entry (complete record) separately, even though the key's value may be part of the entry. A TableNode represents a single row and contains both the key and entry components.
π Definition β TableNode: The structural element of a table that holds both the key and the entry (complete record) for a single row.
Unsorted Sequential Array
In this implementation, TableNodes are stored consecutively in an array in any order. Each element of the array contains both a key and an entry. For practical implementation, a class (e.g., Employee or PhoneDirectoryEntry) can be created to represent each record type, and an array of objects is used to store them.
- insert: Fast β data is added at the back of the array, requiring O(1) time.
- find: Slow β must search through keys one at a time, potentially checking all n entries. Time is proportional to n.
- remove: Slow β must first perform a find (O(n)), then remove the entry. Time is proportional to n.
π‘ Why this matters: The unsorted sequential array provides fast insertion but slow searching and deletion, making it suitable for applications where data is frequently added but rarely searched or removed.
Sorted Sequential Array
This implementation keeps data in the array in sorted order based on the key field. For example, in a telephone directory, names are stored alphabetically. When a new entry is inserted, the array must be re-sorted or elements shifted to maintain the sorted order.
- insert: Slow β requires shifting existing entries to find the correct sorted position. Time is proportional to n.
- find: Fast β uses binary search, requiring O(log n) time.
- remove: Slow β first finds the entry in O(log n) time, then shifts remaining elements to maintain sorted order. Time is proportional to n.
Binary Search
Binary search is a searching algorithm that works on sorted data, finding an element in O(log n) time. If there are 100,000 elements, binary search requires at most 20 steps (since logβ(100,000) β 17).
The algorithm works like looking up a word in a dictionary:
- Start from the middle of the sorted array.
- If the target word comes before the middle word, search in the first half.
- If the target word comes after, search in the second half.
- Repeat the process on the remaining half until the target is found or the search space is empty.
With each step, the search space is halved, making the algorithm extremely efficient compared to linear search.
π Definition β Binary search: A search algorithm that repeatedly divides a sorted array in half to find a target value, achieving O(log n) time complexity.
π Formula: Time complexity of binary search β O(logβ n), where n is the number of elements in the sorted array.
π Example: Searching for "Salman Akhtar" in a sorted telephone directory with 100,000 entries. Start at entry 50,000; if "Salman Akhtar" alphabetically comes before, search the first half (entries 1-50,000). Continue halving until the target is found. Maximum steps required = 17 (since 2ΒΉβ· = 131,072 > 100,000).
β Key Takeaways
The Table ADT is a fundamental data structure consisting of records (rows) and fields (columns), where a unique key field is used for all operations including insert, find, and remove. Two primary implementations exist: unsorted sequential arrays (fast insertion, slow search in O(n)) and sorted sequential arrays (slow insertion due to shifting, but fast search using binary search in O(log n)). Binary search is a critical algorithm that repeatedly halves the search space, making it exponentially faster than linear search for large datasets. The choice between implementations depends on the expected frequency of operationsβwhether insertion or searching is more common in the specific application.
π§ Quick Revision Questions
- What is the difference between a two-dimensional array and a table (Table ADT)?
- What is a key in a table, and why must it be unique?
- How does the insertion time complexity differ between unsorted and sorted sequential array implementations of a table?
- Explain the binary search algorithm and why its time complexity is O(log n).
- In what real-world applications would you prefer an unsorted sequential array over a sorted sequential array for table implementation?
π Lecture 39 β Searching an Array: Binary Search and Skip Lists
π Overview: This lecture covers binary search algorithm for sorted arrays, its implementation in C++, and its efficiency analysis. It then introduces alternative implementations of the Table ADT using linked lists and the novel skip list data structure that overcomes limitations of previous approaches through hierarchical chains.
ποΈ Topics Covered
The lecture explores binary search algorithm with three detailed examples demonstrating different search scenarios, followed by its C++ implementation and efficiency analysis showing O(logβN) performance. It then examines linked list implementation of Table ADT and introduces the skip list data structure as a more efficient alternative, including its representation, higher-level chains, and formal definition.
π Lecture Summary
Searching an Array: Binary Search
Binary search is an efficient algorithm for finding an item in a sorted array by repeatedly dividing the search interval in half. The algorithm compares the target value to the middle element of the array; if they are equal, the search is complete. If the target value is less than the middle element, the search continues in the left half; otherwise, it proceeds in the right half. This process is independent of data type and works for both numeric and string data.
The algorithm in pseudocode:
if (value == middle element)
value is found
else if (value < middle element)
search left half of list with the same method
else
search right half of list with the same method
Binary Search β Example 1
Case 1: val == a[mid] β The target value equals the middle element.
Given array a with indices 0 to 8 containing sorted values: 1, 5, 7, 9, 10, 13, 17, 19, 27.
π Value (val): The item being searched for in the array.
π Formula: mid = (low + high) / 2
π Example: Searching for val = 10
- low = 0, high = 8
- mid = (0 + 8) / 2 = 4
- a[mid] = 10, which equals val = 10
- Result: Found immediately at position 4
Binary Search β Example 2
Case 2: val > a[mid] β The target value is greater than the middle element.
π Example: Searching for val = 19
- low = 0, high = 8, mid = (0 + 8) / 2 = 4
- a[mid] = 10, val (19) > a[mid] (10)
- Since array is sorted, the left half cannot contain 19
- New low = mid + 1 = 5, high remains 8
- Search continues in the right half (positions 5 to 8)
Binary Search β Example 3
Case 3: val < a[mid] β The target value is less than the middle element.
π Example: Searching for val = 7
- low = 0, high = 8, mid = (0 + 8) / 2 = 4
- a[mid] = 10, val (7) < a[mid] (10)
- New high = mid - 1 = 3, low remains 0
- Now searching in left half (positions 0 to 3)
Step 2 in left half:
- low = 0, high = 3, mid = (0 + 3) / 2 = 1
- a[mid] = 5, val (7) > a[mid] (5)
- New low = mid + 1 = 2, high remains 3
Step 3:
- low = 2, high = 3, mid = (2 + 3) / 2 = 2
- a[mid] = 7, val (7) = a[mid] (7)
- Found at position 2 after three comparisons
π‘ Why this matters: Without binary search, finding value 27 would require 9 comparisons sequentially, but with binary search it requires only 3 comparisons. The efficiency gain increases dramatically with larger arrays.
Binary Search β C++ Code
The function isPresent implements binary search for an integer array.
π isPresent function: A C++ function that performs binary search on a sorted array and returns 1 if the value is found, 0 otherwise.
int isPresent(int *arr, int val, int N)
{
int low = 0;
int high = N - 1;
int mid;
while (low <= high)
{
mid = (low + high) / 2;
if (arr[mid] == val)
return 1; // found!
else if (arr[mid] < val)
low = mid + 1;
else
high = mid - 1;
}
return 0; // not found
}
The function receives an array pointer arr, the value to search val, and array size N. It initializes low=0 and high=N-1. The while loop continues as long as low <= high. Inside the loop, mid is calculated, and comparisons determine whether to search the left or right half. Returns 1 if found, 0 if not found after loop termination.
Important: This function requires the data to be sorted to work properly; otherwise it will fail.
Binary Search β Binary Tree
Binary search can be visualized as a binary tree structure. The search process divides a list into two smaller sub-lists until a sub-list is no longer divisible. This corresponds to a fully balanced binary tree where each node represents a decision point, and left/right subtrees represent the two halves.
Binary Search - Efficiency
The efficiency of binary search is analyzed by counting the number of bisections needed.
| Number of Bisections | Remaining Items |
|---|---|
| After 1 bisection | N/2 items |
| After 2 bisections | N/4 = N/2Β² items |
| After i bisections | N/2β± = 1 item |
π Formula: i = logβN
This shows that after a maximum of logβN bisections, either the item is found or determined to be absent.
Comparison of Array Implementation:
- Insert: Must find sorted position and shift elements β O(N) worst case
- Remove: Proportional to N β O(N)
- Search: Maximum logβN comparisons β O(logβN)
π‘ Why this matters: Keeping data sorted for binary search pays off during search operations, making search extremely efficient at the cost of slightly more expensive insertions and deletions.
Implementation 3 (of Table ADT): Linked List
Linked list implementation of the Table ADT stores nodes that may be scattered in memory (not contiguous like arrays).
- TableNodes are stored consecutively (unsorted or sorted)
- insert: Add to front requires O(1) for unsorted; O(n) for sorted list to find correct position
- find: Search through potentially all keys one at a time β O(n) for both unsorted and sorted
- remove: Find the element, then remove using pointer alterations β O(n)
The fixed size of arrays becomes a constraint, while linked lists have no such limitation. However, the find operation using linked lists becomes slower because binary search works only for arrays (since linked list nodes are not contiguous in memory).
Implementation 4 (of Table ADT): Skip List
Skip list is a data structure introduced by Professor Bill Pugh in 1990 to overcome basic limitations of previous lists.
Key characteristics:
- Overcomes limitations where search and update require linear time
- Provides fast searching of sorted chain
- Offers alternative to BST and related tree structures (where balancing can be expensive)
- Relatively recent data structure
Skip List - Representation
A skip list consists of head and tail special nodes at the start and end of the list respectively. The key innovation is adding additional pointers to speed up search.
Using a pointer to the middle element, we can perform binary search-like operations. For example, to find 60 in a sorted skip list, we first compare with the middle element (40), determine that 60 is greater, then search the right half β similar to binary search in arrays.
Skip List - Higher Level Chains
- Level 0 chain includes all elements (the original linked list)
- Level 1 chain includes every other element
- Level 2 chain includes every fourth element
- Level i chain includes every 2β±-th element
The skip list becomes a hierarchy of chains where level i contains a subset of elements in level i-1. Using this structure, elements can be found in O(logβn) time.
However, the frequency of pointers becomes so high compared to data size that management becomes difficult. Insert and remove operations become complex because single insertion or removal requires adjusting many pointers.
Professor Pugh suggested that instead of leveling in powers of 2, it should be done randomly, which makes the skip list easier to manage.
Skip List - Formally
A skip list for a set S of distinct (key, element) items is a series of lists Sβ, Sβ, ..., Sβ such that:
- Each list Sα΅’ contains the special keys +β and -β
- List Sβ contains the keys of S in non-decreasing order
- Each list is a subsequence of the previous one: Sβ β Sβ β ... β Sβ
- List Sβ contains only the two special keys
π‘ Why this matters: The idea of randomness in skip lists is new and will be explored in the next lecture to understand how it makes the skip list data structure easy and useful.
β Key Takeaways
Binary search is a powerful O(logβN) algorithm that works only on sorted arrays by repeatedly dividing the search space in half. The C++ implementation uses three variables (low, high, mid) and a while loop to efficiently locate elements. While linked lists offer flexibility without size constraints, they cannot support binary search because their nodes are not contiguous in memory. The skip list data structure overcomes this limitation by adding hierarchical chains of pointers, achieving O(logβn) search time similar to binary search trees. Randomness in skip lists makes insertions and deletions practical by eliminating the complexity of maintaining exact power-of-two leveling.
π§ Quick Revision Questions
- What is the formula to calculate the middle position in binary search, and why must the array be sorted for this algorithm to work?
- In binary search, after how many bisections will you be left with exactly one item from an array of N elements?
- Why can't binary search be applied to linked lists, and what is the time complexity of search in a linked list implementation of Table ADT?
- What are the four components of the formal definition of a skip list, and what special keys does each list contain?
- What was Professor Bill Pugh's key insight about level distribution in skip lists, and how does it address the complexity of insert and remove operations?
π Lecture 40 β Skip List
π Overview: This lecture introduces the skip list data structure, a probabilistic alternative to balanced trees that enables efficient search (O(log n)) on linked lists. By using multiple levels of additional pointers, skip lists provide binary-search-like performance without the complexity of rotations or array-based limitations. The lecture covers the structure, search algorithm, randomized insertion, and deletion from skip lists.
ποΈ Topics Covered
The lecture covers the definition and structure of skip lists as a series of sorted linked lists with special keys, the search algorithm that moves across and down through levels, randomized insertion using coin tossing to determine node height, and deletion by finding and removing nodes from all levels they occupy. The key insight is that skip lists achieve logarithmic performance through probabilistic balancing without requiring rotations.
π Lecture Summary
Skip List
A skip list for a set S of distinct (key, element) items is a series of lists Sβ, Sβ, ..., Sβ such that:
- Each list Sα΅’ contains the special keys +β and -β
- List Sβ contains the keys of S in non-decreasing order
- Each list is a subsequence of the previous one, i.e., Sβ β Sβ β ... β Sβ
- List Sβ contains only the two special keys
In a skip list, nodes exist only once but have multiple forward pointers to higher levels. For example, node 23 may have one pointer to node 26 in Sβ and another pointer to node 31 in Sβ. The levels are created randomly, not by taking every 2nd or 4th node like in a perfect skip list.
π Definition β Skip List: A linked list with multiple levels of additional pointers where each level is a subsequence of the level below, allowing binary-search-like traversal.
π Example: In a skip list with nodes 12, 23, 26, 31, 34, 56, 64, 78 in Sβ, only nodes 23, 31, 34, 64 appear in Sβ, only node 31 appears in Sβ, and Sβ contains only -β and +β.
π‘ Why this matters: Unlike arrays, skip lists don't require contiguous memory or shifting elements during insertion/deletion, while still providing O(log n) search time.
Skip List Search
The search algorithm for a key x follows these rules: β’ Start at the first position of the top list β’ At current position p, compare x with y β key(after(p)) β’ x = y: return element(after(p)) β’ x > y: "scan forward" (move to next node in same list) β’ x < y: "drop down" (move to the level below at the same position) β’ If we try to drop down past the bottom list, return NO_SUCH_KEY
π Example: Searching for 78 in the skip list:
- Start at Sβ's first node (-β), next is +β > 78 β drop down to Sβ
- In Sβ, next is 31 < 78 β scan forward to +β > 78 β drop down to Sβ
- In Sβ, next is 34 < 78 β scan forward to 64 < 78 β scan forward to +β > 78 β drop down to Sβ
- In Sβ, next after 64 is 78 β found!
This requires far fewer steps than traversing Sβ linearly through 12, 23, 26, 31, 34, 56, 64 to reach 78. The search time is O(log n), similar to binary search trees.
Insertion in Skip List
Insertion uses a randomized algorithm with coin tossing:
Step 1: Repeatedly toss a coin until we get tails, and denote with i the number of times the coin came up heads.
Step 2: If i > h (current height), add new lists Sβββ, ..., Sα΅’ββ, each containing only the two special keys.
Step 3: Search for x and find positions pβ, pβ, ..., pα΅’ of items with largest key less than x in each list Sβ, Sβ, ..., Sα΅’.
Step 4: For j β 0, ..., i, insert item (x, o) into list Sβ±Ό after position pβ±Ό.
π Definition β Randomized Algorithm: An algorithm that performs coin tosses (uses random bits) to control its execution. Its running time depends on the outcome of the coin tosses.
π Formula: Coin toss logic:
b β random()
if b <= 0.5 // head: increment i, continue tossing
else // tail: stop tossing
π Example: Inserting 15 into a skip list containing 10, 23, 36:
- Coin toss results: i = 2 (two heads before tail)
- Current h = 2, so no new lists needed
- Search: start from Sβ top, drop to Sβ (next is 23 > 15), drop to Sβ
- In Sβ: scan past 10 (10 < 15), stop before 23 (23 > 15)
- Insert 15 after 10 in Sβ, Sβ, and Sβ (positions pβ, pβ, pβ)
The new node appears in Sβ always, plus in as many higher levels as the coin toss determined (i levels).
π‘ Why this matters: The random height ensures the skip list remains probabilistically balanced without needing complex rebalancing operations like AVL rotations.
Deletion from Skip List
To remove an item with key x from a skip list: β’ Search for x and find positions pβ, pβ, ..., pα΅’ of items with key x, where pβ±Ό is in list Sβ±Ό β’ Remove positions pβ, pβ, ..., pα΅’ from lists Sβ, Sβ, ..., Sα΅’ β’ Remove any empty lists except the one containing only the two special keys
π Example: Removing node 34 from skip list:
- Search from top list: drop from Sβ to Sβ where 34 is found (pβ)
- Continue to Sβ where 34 is found (pβ)
- Continue to Sβ where 34 is found (pβ)
- Remove 34 from Sβ, Sβ, and Sβ
- Since Sβ now contains only -β and +β (like Sβ), keep only one top list
Unlike insertion, deletion involves no randomness or coin tossing β only pointer manipulation.
β Key Takeaways
A skip list is a multi-level linked list structure that achieves O(log n) search time through probabilistic balancing, making it a practical alternative to balanced trees without requiring rotations or array-based storage. The search algorithm combines horizontal scanning with vertical dropping, moving from the topmost sparse list down to the densest bottom list until the target is found or determined absent. Insertion uses a randomized coin-tossing process to determine how many levels a new node should occupy, ensuring the data structure remains probabilistically balanced without deterministic rebalancing. Deletion is straightforward: find the node in all levels it occupies, remove it, and clean up any now-empty top levels. Unlike AVL trees, skip lists trade deterministic guarantees for simpler implementation while still providing expected logarithmic performance.
π§ Quick Revision Questions
- What are the four properties that define a skip list's structure, and what special keys appear in every list?
- In the skip list search algorithm, when do you "scan forward" versus "drop down," and what happens if you need to drop past the bottom list?
- How does the randomized insertion algorithm determine how many levels a new node should occupy, and what step involves coin tossing?
- In the insertion example with value 15, why was the node inserted into Sβ even though Sβ already had only -β and +β?
- How does deletion from a skip list differ from insertion in terms of randomness, and what cleanup step is needed after removing a node that was the only non-special node in a level?
π Lecture 41 β Data Structures Lecture No. 41
π Overview: This lecture reviews skip list implementation using TowerNode and QuadNode structures, then analyzes skip list performance. It introduces AVL trees as balanced binary search trees and transitions into hashing as a method to achieve constant-time operations for find, insert, and remove. The lecture concludes with examples of hash functions and their implementation.
ποΈ Topics Covered
The lecture covers review of skip list structure and implementation using TowerNode with dynamic arrays of next pointers, the QuadNode approach with four pointers and node copying, performance analysis of skip lists showing expected space proportional to n and search/insert/delete proportional to log n, introduction to AVL trees as balanced BSTs, the concept of hashing as a methodology for constant-time table operations, and examples of hash functions including ASCII summation and base-b conversion.
π Lecture Summary
Review
In the previous lecture, three methods of skip list were studied: insert, find, and remove. The skip list has nodes at 0th, 1st, and 2nd levels containing actual values like 12, 23, 34, and 45. Node 34 appears in three nodes. Implementation requires a structure with next pointers. For example, data values 20, 26, 30, 40, 50, 57, 60 are stored at different levels, with nodes like 26 and 57 having two next pointers while node 40 has three next pointers.
π Definition β TowerNode: A node in a skip list that contains an array of next pointers, with the actual number of pointers decided by a random procedure.
π Definition β MAXLEVEL: An upper limit on the number of levels (and thus next pointers) in a node, imposed to prevent memory allocation problems from excessive coin-flip heads.
π Example: When inserting data and heads appear six times, six next pointers are needed. The TowerNode factory allocates space for six next pointers dynamically. Each pointer points to nodes at their own level β for node 40, its 0-level pointer points to node 50, its 2nd pointer points to node 57, and its third pointer points to tail.
Quad Node
Another method for skip list implementation uses the QuadNode structure, which does not use an array of pointers but instead contains four next pointers.
π Definition β QuadNode: A node that stores item, link to node before, link to node after, link to node below, and link to node above.
π Example: In a QuadNode structure, values 23, 34, and 64 are copied two times, and value 31 is copied three times. The bottom layer has nil down pointers, the top layer has nil up pointers, and rightmost column has nil right pointers. This creates a doubly skip list with backward movement capability.
π‘ Why this matters: QuadNode avoids dynamic array allocation β every list node contains four fixed pointers, and the QuadNode factory returns a node with these four pointers that must be linked up, bottom, left, and right.
Performance of Skip Lists
The analysis is probability-based. In a skip list with n items, the expected space used is proportional to n. For n items, we need n memory locations for data plus approximately 2n memory locations for next pointers. The proportionality constant is around 15 to 20, but it cannot be nΒ² or nΒ³.
The expected search, insertion, and deletion time is proportional to log n. This resembles binary search tree performance. With 100,000 nodes, log n is about 20, so approximately 20 steps are needed for any operation.
π Definition β log n performance: For n items, the number of steps required for search, insert, or delete operations grows logarithmically with n.
π‘ Why this matters: Skip list implementation is simple (similar to linked lists) and provides balanced-tree performance without explicit balancing algorithms.
AVL Tree
Insertion, deletion, and searches are performed based on key values. Each node contains key and data together (like telephone directory with name as key and address/number as entry). The AVL tree ensures the tree remains balanced and does not become degenerated.
π Definition β AVL Tree: A balanced binary search tree where the find, insert, and remove operations are all proportional to log n.
π Example: In an AVL tree, search is on the key (e.g., person's name) while the entry contains address, telephone number, and remaining information. Being balanced, it maintains log n performance for all operations.
Hashing
Hashing is not a new data structure but an algorithmic procedure and methodology for using existing data structures. It aims to make find, insert, and remove operations constant time β a single step operation.
π Definition β Hash function: A mathematical function that takes a key (like a name or roll number) and returns an array index as an integer number.
The process: Key β Hash Function β Array Index. Data is stored in an array (static or dynamic), but not in consecutive locations. The storage place is calculated using the key and hash function.
π Example: To insert employee data with name as key, pass the name to hash function to get an integer, then use that integer as array index to insert the data. For find, pass key to hash function, obtain array index, and retrieve data from that position. If data is not present at that array position, it means data is not found. For remove, pass key to hash function, get array index, and set that position to null.
π‘ Why this matters: All three operations (insert, find, remove) become constant time operations, requiring only one step each, unlike tree or list structures requiring traversal.
Examples of Hashing
Using fruit names as keys, the hash function hashCode returns specific integer values:
hashCode("apple") = 5 hashCode("watermelon") = 3 hashCode("grapes") = 8 hashCode("cantaloupe") = 7 hashCode("kiwi") = 0 hashCode("strawberry") = 9 hashCode("mango") = 6 hashCode("banana") = 2
The resulting array (size 10): index 0 = kiwi, index 1 = empty, index 2 = banana, index 3 = watermelon, index 4 = empty, index 5 = apple, index 6 = mango, index 7 = cantaloupe, index 8 = grapes, index 9 = strawberry.
This creates an associative array where users can think of table["apple"], table["watermelon"], etc., while internally integer indices are used via hashCode.
π Definition β Associative array: An array where keys (like strings) are used as indices instead of numbers, implemented internally using hash functions.
For string keys, one hash function implementation adds ASCII values of characters:
h(str) = (sum of str[i] for i=0 to length-1) % TableSize
π Formula: h(str) = (β str[i]) % TableSize β Add all character ASCII values, then take modulo with table size.
π Example: h("ABC") = (65 + 66 + 67) % 55 = 198 % 55 = 33
C++ implementation of hashCode:
int hashCode(char* s) {
int i, sum = 0;
for(i = 0; i < strlen(s); i++)
sum = sum + s[i]; // ASCII value
return sum % TABLESIZE;
}
Another possibility converts string into some number in arbitrary base b (b might be a prime number):
h(str) = (β str[i] Γ b^i) % T
π Formula: h(str) = (β str[i] Γ b^i) % TableSize β Multiply each character's ASCII value by base raised to position power, sum, then take modulo.
π Example: h("ABC") with b=7 and T=55 = (65 Γ 7β° + 66 Γ 7ΒΉ + 67 Γ 7Β²) % 55 = (65 + 462 + 3283) % 55 = 3810 % 55 = 45
If keys are integers, key % T is generally a good hash function unless data has undesirable features.
π Example: If T=10 and all keys end in zero, then key%T = 0 for all keys. With employee IDs ending in zero, all would hash to index 0, making the hash function useless. Therefore, T should be a prime number to avoid such situations.
β Key Takeaways
Skip lists can be implemented using either TowerNode (with dynamic array of next pointers) or QuadNode (with four fixed pointers: up, down, left, right). Skip lists provide expected search, insert, and delete time proportional to log n with space proportional to n, making them efficient and simple to implement. Hashing is a methodology rather than a data structure, designed to achieve constant-time operations for find, insert, and remove by using a hash function to compute storage locations. The hash function must return an integer index from a key, and common implementations include summing ASCII values or using a base conversion formula. For integer keys, key % table size is effective when the table size is a prime number to avoid undesirable patterns.
π§ Quick Revision Questions
- What are the two different node structures discussed for implementing skip lists, and what are their key differences?
- What is the expected time complexity for search, insert, and remove operations in a skip list?
- How does hashing achieve constant-time operations for find, insert, and remove?
- What problem occurs if table size is 10 and all keys end in zero, and how should table size be chosen?
- Write the formula for converting a string to a hash value using base b, and compute h("ABC") with b=7 and T=55.