π Lecture 30 β Inserting into a Min-Heap, Deleting from a Min-Heap, Building a Heap
π Overview: This lecture covers fundamental heap operations in data structures. It explains how to insert elements into a min-heap while maintaining the heap order, how to delete the minimum element (deleteMin), and how to efficiently build a heap from an unordered array using the percolateDown method.
ποΈ Topics Covered
The lecture covers three main topics: inserting into a min-heap using the percolate-up technique where new elements are placed at the bottom and moved upward; deleting from a min-heap (deleteMin) where the root is removed and a hole percolates downward; and building a heap using the percolateDown method starting from position N/2 to achieve linear time complexity instead of NlogβN.
π Lecture Summary
Inserting into a Min-Heap
When inserting a new element into a min-heap implemented with an array, the element is placed at the last position of the array. Since this insertion may violate the heap order (where parent values must be less than children values), we start moving this element upward. Only one branch of the tree is affected due to this upward movement, making the process localized.
π Definition β Min-Heap: A complete binary tree where the minimum value is always at the root, and every parent node has a value less than or equal to its children.
π Formula: Parent-child relationship in array β parent at position i, left child at 2i, right child at 2i+1
π Example: Inserting element 15 into a min-heap:
- Initial array: [13, 14, 16, 24, 21, 19, 68, 65, 26, 32, 31] (positions 1-11)
- Insert 15 at position 12 (left child of node at position 6, which is 19)
- Step 1: Compare 15 with parent 19 β 15 < 19, so swap (15 goes to position 6, 19 goes to position 12)
- Step 2: Compare 15 with new parent 16 β 15 < 16, so swap (15 goes to position 3, 16 goes to position 6)
- Step 3: Compare 15 with parent 13 β 15 > 13, so stop
- Final position: 15 is at position 3, maintaining min-heap order
π‘ Why this matters: The maximum number of exchanges is logβN, making insertion efficient even for large trees.
Deleting from a Min-Heap (deleteMin)
The deleteMin operation finds and removes the minimum number from the tree. Finding the minimum is easy because it is at the top of the heap (the root). Deletion causes a hole which needs to be filled while maintaining the complete binary tree property.
π Definition β Percolate Down: The process where a hole moves downward in the tree, with smaller children moving upward to fill the hole, maintaining the min-heap order.
π Example: Deleting the minimum element from a min-heap:
- Initial tree: root = 13, children = 14 and 16
- Step 1: Remove root (13), creating a hole at position 1
- Step 2: Compare children 14 and 16 β 14 is smaller, so place 14 in the hole (position 1)
- Step 3: Hole now at previous position of 14. Compare its children 24 and 21 β 21 is smaller, so place 21 in the hole
- Step 4: Hole now at previous position of 21. Compare its children 32 and 31 β 31 is smaller, so place 31 in the hole
- Step 5: Hole has reached the bottom and can be deleted. Array size is reduced by 1
π‘ Why this matters: The hole follows the 2i or 2i+1 scheme, moving from top to bottom until it reaches a position where it can be safely removed.
Building a Heap (buildHeap)
When given N keys, instead of doing N successive inserts (which could take NlogβN time in worst case), we can build a heap in linear time using the percolateDown method.
π Definition β buildHeap: An algorithm that constructs a heap from an unordered array of N keys by applying percolateDown from position N/2 down to 1.
π Formula: Algorithm β for(i = N/2; i > 0; i--) percolateDown(i);
π Example: Building a min-heap from initial data of 15 elements:
- Initial array (positions 1-15): [65, 31, 32, 26, 21, 19, 68, 13, 24, 15, 14, 16, 5, 70, 12]
- Start with i = 15/2 = 7 (position 7 has value 68)
- Children of position 7: positions 14 (70) and 15 (12)
- Apply percolateDown(7): 12 is smaller than 68, so swap β position 7 becomes 12, position 15 becomes 68
- Next iteration: i = 6 (position 6 has value 19)
- Children of position 6: positions 12 (16) and 13 (5)
- Apply percolateDown(6): 5 is smaller than 19, so swap β position 6 becomes 5, position 13 becomes 19
- Continue until i = 1 is reached
π Definition β PercolateDown(p): A method that moves the key in node p downward to its correct position in the min-heap, with smaller values moving upward automatically.
π‘ Why this matters: This algorithm achieves linear time O(N) instead of O(NlogβN), making it significantly more efficient for large datasets.
β Key Takeaways
For the exam, you must understand that insertion in a min-heap involves placing the new element at the bottom and percolating it upward until the heap order is restored, affecting only one branch of the tree with maximum logβN exchanges. The deleteMin operation is straightforward β you remove the root (minimum element), create a hole, and percolate the hole downward by comparing children and moving the smaller one up, continuing until the hole reaches the bottom where it can be deleted. The critical insight for building a heap is that starting from position N/2 and applying percolateDown gives linear time complexity O(N) rather than NlogβN, with the loop moving through all non-leaf nodes from bottom to top. Remember that the parent-child relationship in the array uses the 2i and 2i+1 scheme for children, and i/2 for parent.
π§ Quick Revision Questions
- When inserting an element into a min-heap, at which position in the array is the new element initially placed?
- In the deleteMin operation, what fills the hole created at the root, and how does the hole move through the tree?
- Why does the buildHeap algorithm start the percolateDown loop from position N/2 instead of N?
- What is the worst-case time complexity of building a heap using N successive inserts, and what is the time complexity using the percolateDown method?
- In the percolateDown method, if a node has two children, which child is moved upward to fill the hole?
π Lecture 31 β Data Structures Lecture No. 31
π Overview: This lecture focuses on the BuildHeap algorithm, a more efficient method for constructing a heap from an existing dataset compared to repeated insert() calls. It also covers other essential heap operations like
decreaseKey,increaseKey, andremove, and presents the complete C++ implementation of the Heap class, including the constructor, insert, and deleteMin methods.
ποΈ Topics Covered
This lecture covers the rationale and step-by-step execution of the BuildHeap algorithm, demonstrating how it transforms an unordered binary tree into a min-heap using the percolateDown procedure. It then introduces other important heap methods, including decreaseKey, increaseKey, and remove, explaining their utility, especially in the context of priority queues. The lecture concludes with a detailed walkthrough of the C++ code for the Heap class, covering the constructor, insert, and deleteMin methods.
π Lecture Summary
BuildHeap
In the previous lecture, we discussed the BuildHeap method of the heap abstract data structure. In this handout, you are going to know why a programmer adopts this method. Suppose we have some data that can be numbers or characters or in some other form and want to build a min-heap or max-heap out of it. One way is to use the insert() method to build the heap by inserting elements one by one. In this method, the heap property will be maintained. However, the analysis shows that it is NlogN algorithm i.e. the time required for this will be proportional to NlogN. Secondly, if we have all the data ready, then it will be better to build a heap at once as it is a better option than NlogN.
In the delete procedure, when we delete the root, a new element takes the place of root. In case of min-heap, the minimum value will come up and the larger elements will move downwards. In this regard, percolate procedures may be of great help. The percolateDown procedure will move the smaller value up and bigger value down. This way, we will create the heap at once, without calling the insert() method internally. Letβs revisit it again:
- The general algorithm is to place the N keys in an array and consider it to be an unordered binary tree.
- The following algorithm will build a heap out of N keys.
for ( i = N/2; i > 0; i-- )
percolateDown(i);
Suppose, there are N data elements (also called as N Keys). We will put this data in an array and call it as a binary tree. As discussed earlier, a complete binary tree can be stored in an array. We have a tree in an array but it is not a heap yet. It is not necessary that the heap property is satisfied. In the next step, we apply the algorithm. Why did we start the i from N/2?
π Definition β BuildHeap: A method to construct a heap from an unordered array of N keys by applying percolateDown() on each internal node, starting from the last internal node (at index N/2) up to the root.
π Formula:
for ( i = N/2; i > 0; i-- )
percolateDown(i);
β This loop starts at the last non-leaf node and moves up to the root, ensuring each subtree becomes a valid heap.
π Example: Build a min-heap from the array: [0, 65, 31, 32, 26, 21, 19, 68, 13, 24, 15, 14, 16, 5, 70, 12] (index 0 is unused, N=15).
- Step 1 (i = N/2 = 7):
percolateDown(7).- Node at index 7 is 68. Its children are 70 (index 14) and 12 (index 15).
- 12 is the smallest. Swap 68 and 12.
- Subtree with root 12 now satisfies the min-heap property.
- Step 2 (i = 6):
percolateDown(6).- Node at index 6 is 19. Its children are 16 (index 12) and 5 (index 13).
- 5 is the smallest. Swap 19 and 5.
- Step 3 (i = 5):
percolateDown(5).- Node at index 5 is 21. It may be swapped with a smaller child.
- Continue for i = 4, 3, 2, 1.
- Final min-heap array: [0, 5, 13, 12, 24, 14, 16, 65, 26, 31, 15, 21, 32, 19, 70, 68]. The root is the minimum element (5).
Why start from N/2? All leaf nodes (indices > N/2) are already valid min-heaps by themselves (they have no children). Starting from N/2, the first level above the leaves, avoids unnecessary work on leaf nodes and makes the algorithm more efficient. The complexity of BuildHeap is O(N), which is better than the O(N log N) of N insert operations.
π‘ Why this matters: The BuildHeap algorithm demonstrates that if you have all your data upfront, you can construct a heap more efficiently than inserting elements one by one. This is a classic example of how a clever algorithm can improve time complexity.
Other Heap Methods
Letβs have a look on some more methods of heap and see the C++ codes.
decreaseKey(p, delta) This method lowers the value of the key at position βpβ by the amount βdeltaβ. Since this might violate the heap order, so it (the heap) must be reorganized with percolate up (in min-heap) or down (in max-heap).
This method takes a pointer to the node that may be the array position as we are implementing it as an array internally. The user wants to decrease the value of this node by delta. Suppose we have a node with value 17 and want to decrease it by 10. Its new value will be 10. By decreasing the value of a node, the heap order can be violated. If heap order is disturbed, then we will have to restore it. We may not need to build the whole tree. The node value may become smaller than that of the parents, so it is advisable to exchange these nodes. We use percolateUp and percolateDown methods to maintain the heap order. Here the question arises why we want to decrease the value of some node? The major use of heap is in priority queues. Priority queues are not FIFO or LIFO. The elements are taken out on some key value, also known as priority value. Suppose we have some value in the priority queue and want to decrease its priority. If we are using heap for the priority queue, the priority of the some elements can be decreased so that it could be taken out later and some other element that now has higher priority will be taken out first.
increaseKey(p, delta)
This method is the opposite of decreaseKey. It will increase the value of the element by delta. These methods are useful while implementing the priority queues using heap.
remove(p)
This method removes the node at position p from the heap. This is done first by decreaseKey(p, -β) and then performing deleteMin(). First of all, we will decrease the value of the node by -β and call the method deleteMin. The deleteMin method deletes the root. If we have a min-heap, the root node contains the smallest value of the tree. After deleting the node, we will use the percolateDown method to restore the order of the heap.
The user can delete any node from the tree. We can write a special procedure for this purpose. Here we will use the methods which are already available. At first, the decreaseKey method will be called and value of node decreased by -β. It will result in making the value of this node smallest of all the nodes. If the value is in integers, this node will have the smallest integer. Now this node has the minimum value, so it will become the root of the heap. Now we will call the deleteMin() and the root will be deleted which is the required node. The value in this node is not useful for us. The -β is a mathematical notation. It is not available in the C++. Actually we want to make the minimum possible value of this node supported by the computer.
C++ Code
Now we will look at the C++ code of the Heap class.
Heap.h file
/* The heap class. This is heap.h file */
template <class eType>
class Heap
{
public:
Heap( int capacity = 100 );
Void insert( const eType & x );
Void deleteMin( eType & minItem );
Const eType & getMin( );
Bool isEmpty( );
Bool isFull( );
int Heap<eType>::getSize( );
private:
int currentSize; // Number of elements in heap
eType* array; // The heap array
int capacity;
void percolateDown( int hole );
};
We may like to store different type of data in the heap like integers, strings, floating-point numbers or some other data type etc. For this purpose, template is used. With the help of template, we can store any type of object in the heap. Therefore first of all we have:
template <class eType>
Here eType will be used as a type parameter. We can use any meaningful name. Then we declare the Heap class. In the public part of the class, there is a constructor as given below.
Heap( int capacity = 100 );
We have a parameter capacity in the constructor. Its default value is 100. If we call it without providing the parameter, the capacity will be set to 100. If we call the constructor by providing it some value like 200, the capacity in the Heap object will be 200.
Next we have an insert method as:
void insert( const eType & x );
Here we have a reference element of eType which is of constant nature. In this method, we will have the reference of the element provided by the caller. The copy of element is not provided through this method. We will store this element in the Heap.
Similarly we have a delete method as:
void deleteMin( eType & minItem );
This method is used to delete the element from the heap.
If we want to know the minimum value of the heap, the getMin method can be useful. The signature is:
const eType & getMin( );
This method will return the reference of the minimum element in the Heap.
We have some other methods in the Heap class to check whether it is empty or full. Similarly, there is a method to check its size.
bool isEmpty( );
bool isFull( );
int Heap<eType>::getSize( );
When will Heap become full? As we are implementing the Heap with the help of an array, the fixed data type, so the array may get full at some time. This is the responsibility of the caller not to insert more elements when the heap is full.
In the private part, we have some data elements and methods. At first, we have currentSize element. It will have the number of elements in the heap. Then we have an array of eType. This array will be dynamically allocated and its size depends on the capacity of the Heap. We have one private method as percolateDown.
Heap.cpp file
/* heap.cpp file */
#include "Heap.h"
template <class eType>
Heap<eType>::Heap( int capacity )
{
array = new eType[capacity + 1];
currentSize=0;
}
/* Insert item x into the heap, maintaining heap order. Duplicates
are allowed. */
template <class eType>
bool Heap<eType>::insert( const eType & x )
{
if( isFull( ) ) {
cout << "insert - Heap is full." << endl;
return 0;
}
// Percolate up
int hole = ++currentSize;
for(; hole > 1 && x < array[hole/2 ]; hole /= 2)
array[ hole ] = array[ hole / 2 ];
array[hole] = x;
}
template <class eType>
void Heap<eType>::deleteMin( eType & minItem )
{
if( isEmpty( ) ) {
cout << "heap is empty." << endl;
return;
}
minItem = array[ 1 ];
array[ 1 ] = array[ currentSize-- ];
percolateDown( 1 );
}
We include the heap.h file before having the constructor of the heap. In the constructor, we are dynamically creating our array. We have added 1 to the capacity of the array as the first position of the array is not in use. We also initialize the currentSize to zero because initially Heap is empty.
Next we have the insert method. Inside the insert method, we will call the isFull() method. If the heap is full, we will display a message and return 0. If the heap is not full, we will insert the element in the heap and take a variable hole and assign it currentSize plus one value. Then we will have a βfor loopβ which will be executed as long as hole is greater than 1. This is due to the fact that at position one, we have root of the array. Secondly, the element which we want to insert in the array is smaller than the array[hole/2]. In the loop, we will assign the array[hole/2] to array[hole]. Then we divide the hole by 2 and again check the loop condition. After exiting from the loop, we assign x to the array[hole]. Instead of performing multiple swaps which is expensive, we find the final position of the new node and then insert it at that position. The hole moves upward, and the parent is moved down until the final position is achieved.
The next method is deleteMin. First of all, it calls the isEmpty() method. If heap is empty, it will display a message and return. If the heap is not empty, it will delete the node. As the minimum value lies at the first position of the array, we save this value in minItem variable and store the currentSize at the first position of the array. At the same time, we reduce the currentSize by one as one element is being deleted. Then we call the percolateDown method, providing it the new root node. This method will make the tree a min-heap again.
π Definition: template <class eType> : A C++ feature that allows the Heap class to work with any data type (e.g., integers, strings, objects) without rewriting the code for each type.
π Definition: hole : A variable used in the insert method that represents the position in the array where the new element might be placed. It starts at the next available position and "percolates up" as parent values are moved down.
π Example: Insert value 3 into a min-heap [0, 5, 13, 12, 24] (currentSize=4).
- Check if the heap is full. It is not.
hole = ++currentSize = 5.- Loop:
hole > 1 (5 > 1) && x (3) < array[5/2] (array[2] = 13)β true.array[5] = array[2](13).hole = 5/2 = 2.
- Loop:
hole > 1 (2 > 1) && x (3) < array[2/2] (array[1] = 5)β true.array[2] = array[1](5).hole = 2/2 = 1.
- Loop:
hole > 1is false. Exit loop. array[1] = 3.- The final array is [0, 3, 5, 12, 24, 13], which is a valid min-heap.
β Key Takeaways
The BuildHeap algorithm constructs a heap from an unordered array of N elements in O(N) time, which is significantly faster than inserting N elements one by one using O(N log N). The key to its efficiency is starting the percolateDown loop from the last internal node (N/2), because leaf nodes are already valid heaps. Other important heap operations, such as decreaseKey, increaseKey, and remove, are essential for implementing priority queues, where element priorities need dynamic adjustments. The C++ template implementation of the Heap class provides a reusable data structure, where insert uses "percolate up" to find the correct position for the new element and deleteMin removes the root and uses "percolate down" to restore the heap order. For the remove(p) method, a clever trick is to first decrease the node's value to negative infinity to make it the new root, then call deleteMin().
π§ Quick Revision Questions
- What is the time complexity of the BuildHeap algorithm compared to building a heap by calling insert() N times?
- Explain why the
forloop in the BuildHeap algorithm starts fromi = N/2instead ofi = N. - In a min-heap, describe the steps to implement the
decreaseKey(p, delta)operation, assuming the new value breaks the heap property. - How does the
remove(p)method usedecreaseKeyanddeleteMinto delete an arbitrary node from the heap? - In the C++
insertmethod, why is finding the final position of the new node using a "hole" more efficient than repeatedly swapping elements?
π Lecture 32 β Data Structures
π Overview: This lecture completes the implementation of the heap class by discussing the
percolateDown,getMin, andbuildHeapmethods. It then provides a mathematical and visual proof thatbuildHeapruns in linear time, demonstrating its superiority over N logβN algorithms.
ποΈ Topics Covered
This lecture covers the percolateDown method used in deleteMin and buildHeap operations, the getMin method for retrieving the minimum without deletion, the buildHeap method for constructing a heap from an unsorted array, and a detailed proofβboth mathematical and non-mathematicalβthat buildHeap runs in linear time (O(N)). Small utility methods isEmpty, isFull, and getSize are also presented.
π Lecture Summary
perculateDown Method
This method takes an array index (called hole) as an argument and restores the heap order starting from that index downward. It stores the value at hole in a temporary variable tmp, then uses a for loop that continues as long as hole * 2 <= currentSize (i.e., the node has a left child). Inside the loop, the smaller of the two children is identified; if the left child is larger than the right child, child is incremented to point to the right child. If the smaller child's value is less than tmp, the child's value moves up to hole, and hole moves down to child. Otherwise, the loop breaks. Finally, tmp is placed in its correct position (array[hole]).
This method uses a single assignment inside the loop instead of three swap statements, making it more efficient. The "hole" (empty position) percolates downward until the correct location for tmp is found.
π Definition β percolateDown: A method that restores the heap order by moving a value downward from a given index, comparing it with its children until the correct position is found.
π Formula: hole * 2 (left child), hole * 2 + 1 (right child) β used to find children of a node in an array-based heap.
π Example: In a min-heap, if the root (at index 1) is removed and the last element is moved to index 1, percolateDown(1) is called. If array[1] = 15, and its children are array[2] = 10 and array[3] = 12, the left child (10) is smaller. Since 10 < 15, the value 10 moves to index 1, and the hole moves to index 2. The process continues until 15 is placed correctly.
getMin Method
This method returns the minimum value in the heap (the root) without removing it, similar to the top method in a stack. It simply checks if the heap is not empty and returns array[1].
template <class eType>
const eType& Heap<eType>::getMin( )
{
if( !isEmpty( ) )
return array[ 1 ];
}
π‘ Why this matters: Unlike deleteMin, getMin allows you to inspect the smallest element without altering the heap structure, which is useful for priority queue peek operations.
buildHeap Method
This method takes an external array anArray and its size n, copies it into the internal heap array (shifting from 0-based to 1-based indexing), sets currentSize = n, and then calls percolateDown(i) for every node from index currentSize / 2 down to 1. This bottom-up approach ensures all subtrees become valid heaps.
template <class eType>
void Heap<eType>::buildHeap(eType* anArray, int n )
{
for(int i = 1; i <= n; i++)
array[i] = anArray[i-1];
currentSize = n;
for( int i = currentSize / 2; i > 0; i-- )
percolateDown( i );
}
The isEmpty, isFull, and getSize utility methods simply check or return currentSize and capacity values.
buildHeap in Linear Time
The buildHeap algorithm runs in linear time O(N), which is better than N logβN. This is proven by showing that the sum of heights of all nodes (S) is a linear function of N. For a perfect binary tree of height h with N = 2^(h+1) β 1 nodes, the sum S = N β h β 1. For large N, the log term becomes insignificant, making S β N. Since each percolateDown operation for a node at height h takes at most h steps, the total work is proportional to S, which is O(N).
Theorem
Statement: "For a perfect binary tree of height h containing 2^(h+1) β 1 nodes, the sum of the heights of nodes is 2^(h+1) β 1 β (h+1), or N β h β 1."
Mathematical Proof: In a perfect binary tree, there are 2β± nodes at height hβi. The sum S = Ξ£ 2β± (h β i) for i = 0 to hβ1. Multiplying by 2 and subtracting yields S = 2^(h+1) β 1 β (h+1) = N β (h+1). Since a complete tree has nodes between 2^h and 2^(h+1), S β N β logβ(N+1). For large N, logβ(N+1) is insignificant, so S β N.
Non-Mathematical (Visual) Proof: For each node of height h, darken h tree edges (first a left edge, then subsequent right edges). There are N β 1 total edges and h edges on the right path, so the number of darkened edges is N β 1 β h, which equals the sum of heights. In a 31-node perfect tree of height 4, there are 30 total edges, 4 unmarked (dotted) edges on the right path, and 26 marked edgesβmatching the sum of heights formula S = 31 β 4 β 1 = 26.
π Definition β Sum of Heights (S): The total number of edges traversed if every node descends to its deepest leaf; it represents the upper bound on the number of value movements in buildHeap.
β Key Takeaways
The percolateDown method is the core operation for maintaining heap order, used in both deleteMin and buildHeap. It avoids costly three-way swaps by using a single assignment per loop iteration. The buildHeap method constructs a heap from an unsorted array in linear time by calling percolateDown on all non-leaf nodes in reverse level order. The sum of heights of all nodes in a perfect binary tree equals N β h β 1, which is approximately N for large N, proving that buildHeap is O(N) rather than O(N log N). Utility methods like getMin, isEmpty, isFull, and getSize complete the heap class interface.
π§ Quick Revision Questions
- What is the termination condition of the
forloop inpercolateDown, and why does it usehole * 2 <= currentSize? - Explain why
percolateDownuses a single assignment inside the loop instead of three swap statements. - In
buildHeap, why does the loop start fromcurrentSize / 2and go down to 1? - Prove mathematically that
buildHeapruns in linear time using the sum of heights theorem. - For a perfect binary tree with 31 nodes and height 4, how many edges are marked according to the visual proof, and what does this number represent?
π Lecture 33 β Priority Queue Using Heap, The Selection Problem, Heap Sort, Disjoint Set ADT, Equivalence Relations
π Overview: This lecture explores the implementation of a priority queue using a heap data structure, demonstrating how heaps improve efficiency over array-based implementations. It also introduces the selection problem, heap sort, and the Disjoint Set ADT with its applications in computer vision and equivalence relations.
ποΈ Topics Covered
The lecture covers implementing a priority queue using heap, including the PriorityQueue class with heap-based operations. It then discusses the selection problem for finding the kth smallest element, introduces heap sort as a sorting algorithm, and concludes with the Disjoint Set ADT, blob coloring in computer vision, and equivalence relations in mathematics.
π Lecture Summary
Priority Queue Using Heap
The lecture begins by implementing a priority queue using a heap instead of an array. The buildHeap method is preferred over insert when all data is available, as it takes less time than NlogβN. The code shows a PriorityQueue class that internally uses a heap of Event objects with a maximum size of 30 (PQMAX). The constructor creates a new Heap object, and the destructor deletes it. The remove() method calls deleteMin() on the heap, which returns the minimum element. The insert() method calls the heap's insert() method, which performs percolate up and down operations to place the new element at its correct position.
π Definition β Heap: A complete binary tree where each node's key is either greater than or equal to (max-heap) or less than or equal to (min-heap) its children's keys.
The full() method calls isFull() on the heap, and length() calls getSize(). This implementation is more efficient than the array-based priority queue because the heap can readjust itself in logβN time, compared to sorting the array after each insertion.
The Selection Problem
The selection problem involves finding the kth smallest (or largest) element from a list of N elements that can be totally ordered. For example, given 1000 numbers, finding the 10th smallest number.
One approach is to sort the array, taking NlogβN time. A faster approach uses the heap:
- Put N elements into an array and apply the buildHeap algorithm
- Perform k deleteMin operations
- The last element extracted is the answer (the kth smallest)
π Definition β Median: The value where half the numbers are greater and half are smaller. Mathematically, k = βN/2β for the median.
π Formula: buildHeap time complexity = O(N) (linear time) π Example: To find the median of final marks (max 100) for N students, use buildHeap to construct a min-heap, then call deleteMin N/2 times. The N/2th marks extracted would be the median.
Heap Sort
If we call deleteMin N times (where k = N) and record the elements as they come off a min-heap, we get all elements sorted in ascending order. This is the basis of heapsort, a fast sorting algorithm that will be fine-tuned later in the course.
π Formula: Heap Sort time complexity = O(N logβN) π Example: For a min-heap of 100 elements, calling deleteMin 100 times and storing each removed element produces a sorted list in ascending order.
Disjoint Set ADT
The Disjoint Set ADT is used to handle relationships and answer queries like "Is Haaris related to Ahmad?" It works on the key property: If Haaris is related to Saad and Saad is related to Ahmad, then Haaris is related to Ahmad (transitivity).
π Definition β Disjoint Set ADT: A data structure that maintains a collection of disjoint (non-overlapping) sets, supporting operations to find which set an element belongs to and to union two sets together.
Blob Coloring is a computer vision application where we partition pixels into disjoint sets (one set per blob). For a black and white image with five non-overlapping black blobs, the ADT creates one set per blob. This is also used in image segmentation problems, MRI scans, and CT scans.
Equivalence Relations
A binary relation R over a set S is called an equivalence relation if it has three properties:
- Reflexivity: for all element x β S, x R x
- Symmetry: for all elements x and y, x R y if and only if y R x
- Transitivity: for all elements x, y and z, if x R y and y R z then x R z
π Definition β Equivalence Relation: A relation that is reflexive, symmetric, and transitive. The relation "is related to" over the set of people is an example of an equivalence relation.
π‘ Why this matters: Understanding equivalence relations is fundamental to the Disjoint Set ADT implementation, as the relation "is in the same set" forms an equivalence relation.
β Key Takeaways
The heap-based priority queue implementation is more efficient than the array-based version, with O(logβN) readjustment time compared to O(N) sorting. The selection problem can be solved using buildHeap (O(N)) followed by k deleteMin operations, which is faster than sorting for small k. Heap sort emerges naturally from heap operations by extracting all elements. The Disjoint Set ADT is crucial for relationship tracking, computer vision (blob coloring, image segmentation), and medical imaging. Equivalence relations form the mathematical foundation for understanding how disjoint sets partition elements.
π§ Quick Revision Questions
- What is the time complexity advantage of using buildHeap instead of N insert operations when constructing a heap from N elements?
- How would you find the 10th smallest element from 1000 numbers using a min-heap? What operations are required?
- What happens if you call deleteMin N times on a min-heap and record all removed elements?
- In the Disjoint Set ADT, what are the three properties of an equivalence relation, and how do they apply to the "is related to" relation?
- How does blob coloring in computer vision relate to the Disjoint Set ADT? Give an example.
π Lecture 34 β Equivalence Relations and Disjoint Sets
π Overview: This lecture introduces the mathematical concept of equivalence relations and their connection to disjoint sets and equivalence classes. It lays the foundation for the dynamic equivalence problem, explaining how we can efficiently determine whether elements belong to the same set using union and find operationsβa key data structure concept for solving real-world grouping problems.
ποΈ Topics Covered
The lecture covers equivalence relations with their three defining properties (reflexivity, symmetry, transitivity) and provides examples from family relationships and electrical circuits. It then discusses how equivalence relations partition sets into disjoint equivalence classes, introduces the dynamic equivalence problem, and presents the union/find operations as a solution. The lecture concludes by contrasting a naive matrix-based approach with a more efficient graph-based representation.
π Lecture Summary
Equivalence Relations
The lecture begins by defining equivalence relations mathematically. A binary relation R over a set S is called an equivalence relation if it satisfies three specific properties. The first property is reflexivity, meaning for all element x in S, x R x. The second is symmetry, meaning for all elements x and y, x R y if and only if y R x. The third is transitivity, meaning for all elements x, y and z, if x R y and y R z then x R z.
The lecture uses the example of family relationships to illustrate these properties. Haris, Saad, and Ahmed are used as examples: Haris and Saad are related as brothers, Saad and Ahmed are related as cousins. The relationship "is related to" satisfies reflexivity (everyone is related to themselves), symmetry (if Haris is Saad's brother, Saad is Haris's brother), and transitivity (if Haris is related to Saad and Saad to Ahmed, then Haris is related to Ahmed).
π Definition β Equivalence Relation: A binary relation R over a set S that satisfies reflexivity (x R x for all x in S), symmetry (x R y if and only if y R x), and transitivity (if x R y and y R z, then x R z).
π Example: The relationship "β€" is NOT an equivalence relation. While it satisfies reflexivity (x β€ x) and transitivity (if x β€ y and y β€ z, then x β€ z), it fails symmetry since x β€ y does not imply y β€ x.
π Example: Electrical connectivity is an equivalence relation. If component a is connected to b by metal wire, then: a is connected to itself (reflexivity), if a is connected to b then b is connected to a (symmetry), and if a is connected to b and b is connected to c, then a is connected to c (transitivity).
π‘ Why this matters: Understanding equivalence relations helps us mathematically model real-world grouping problems like family relationships, pixel connectivity in images, and circuit connectivity.
Disjoint Sets
An equivalence relation R over a set S can be viewed as a partitioning of S into disjoint sets. Each set of the partition is called an equivalence class of R, containing all elements that are related to each other.
π Definition β Equivalence Class: A subset of S containing all elements that are related to each other through the equivalence relation R. Every member of S appears in exactly one equivalence class.
The lecture illustrates this with the analogy of grouping related people at a gathering. Initially, people are in separate groups, but when a marriage occurs between two groups, they merge into a larger familyβmirroring the union operation on disjoint sets.
π Example: Given 1000 people, initially each person is in their own set. As family relationships are discovered, sets merge. To determine if person a is related to person b, we only need to check if they are in the same equivalence class.
Dynamic Equivalence Problem
The dynamic equivalence problem asks how to efficiently determine whether two elements belong to the same set given a set of binary relations. The lecture presents two approaches to solve this.
First approach: Store the relation as a two-dimensional boolean array. If we have n elements, we need an nΓn matrix. For 1000 people, this requires 1,000,000 entries. For 1,000,000 people, it requires 10ΒΉβ΄ entriesβprohibitively expensive in memory.
Second approach: Use a graph-based representation where:
- Each element is a node
- Each relation creates an edge between nodes
- Connected components represent equivalence classes
π Example: Given five elements {a1, a2, a3, a4, a5} with relations a1 R a2, a3 R a4, a5 R a1, and a4 R a2, we can determine that a3 R a5 even though this relation wasn't explicitly givenβthey are connected through the graph.
The lecture introduces two fundamental operations for this problem:
π Definition β Find: Returns the name of the set (equivalence class) that contains a given element. Formally, Si = find(a).
π Definition β Union: Merges two sets to create a new set. Formally, Sk = Si βͺ Sj.
Algorithm for adding a relation a R b:
- Perform
find(a)to get the set containing a - Perform
find(b)to get the set containing b - If
find(a) == find(b): a and b are already related (no action needed) - If
find(a) != find(b): performunion(a, b)to merge the two sets
The lecture notes that all elements can be numbered sequentially from 1 to n, with initially Si = {i} for i = 1 through n. The name of the set returned by find is arbitraryβwhat matters is that find(x) = find(y) if and only if x and y are in the same set.
π‘ Why this matters: For any sequence of at most m finds and up to n-1 unions, this algorithm will require time proportional to (m + n), making it highly efficient for large-scale problems.
β Key Takeaways
Equivalence relations must satisfy reflexivity, symmetry, and transitivityβif any property is missing, the relation is not an equivalence relation. An equivalence relation partitions a set into disjoint equivalence classes where every element appears in exactly one class. The dynamic equivalence problem can be solved using the union/find algorithm: find determines which set an element belongs to, and union merges two sets. The naive matrix approach for determining relations is space-prohibitive for large datasets (O(nΒ²) space), while the graph-based approach is more efficient. Elements can be numbered sequentially, and the find operation only needs to return consistent set identifiers to determine equivalence.
π§ Quick Revision Questions
- What are the three properties that a binary relation must satisfy to be an equivalence relation?
- Why is the "β€" relation NOT an equivalence relation?
- In the dynamic equivalence problem, what two operations are used to manage disjoint sets?
- If we have 1000 people and want to use a boolean matrix to store all relations, how many entries would the matrix have?
- When adding a new relation a R b, what must we check before performing a union operation?
π Lecture 35 β Dynamic Equivalence Problem
π Overview: This lecture explores the dynamic equivalence problem and its solution using disjoint set data structures. It explains how to efficiently implement the union and find operations using a parent array, with trees representing sets. Understanding these concepts is crucial for managing equivalence relations in various algorithms and data structures.
ποΈ Topics Covered
The lecture covers the dynamic equivalence problem, including the use of trees to represent sets and forests to represent collections of sets. It details the implementation of the parent array for storing set relationships, with examples of union and find operations. The discussion includes initialization of sets, performing unions to merge trees, executing finds to determine set membership, and analyzing the running time of these operations.
π Lecture Summary
Dynamic Equivalence Problem
The dynamic equivalence problem involves managing disjoint sets where elements are unique and belong to exactly one set. A tree is used to represent each set, with the root serving as the name of the set. The find operation returns the root (set name) for a given element, while the union operation merges two sets by making the root of one tree point to the root of another. A collection of trees is called a forest, and these trees are not necessarily binaryβnodes may have more than two children. Initially, N elements form N separate trees, each with one node.
π Definition β Forest: A collection of trees, used to represent multiple disjoint sets in this data structure.
Example 1
This example demonstrates union operations on eight elements (1 to 8). Initially, each element is in its own set. After union(5,6), set 6 becomes a child of set 5, making the new set named 5. Similarly, union(7,8) joins 8 as a child of 7. When union(5,7) is called, the set containing 7 (root 7) is merged into set 5, making 7 a child of 5. After union(3,4), set 4 becomes a child of 3. Finally, union(4,5) finds the root of 4 (which is 3) and merges set 5 into set 3, showing that union automatically finds the correct roots before merging.
π‘ Why this matters: In the union operation, the caller does not need to provide the rootsβthe function automatically finds the set roots of the given elements before merging them.
Parent Array
The parent array stores the tree structure using a one-dimensional array. For each node i, Parent[i] contains the index of its parent, or -1 if it is the root. This approach uses array indices as pointers instead of actual memory addresses.
π Definition β Parent Array: An array where each index represents an element, and the value at that index stores the parent of the element (or -1 for roots).
π Formula: Parent[i] = -1 if i is the root, otherwise Parent[i] = parent index.
Initialization
Initially, all elements are roots of their own sets. Using a for loop, each position in the parent array is set to -1.
π Formula: for (i = 0; i < n; i++) Parent[i] = -1 β This initializes all n elements as separate sets with no parents.
π Example: For 8 elements (1 to 8), the initialization results in: Parent[1]=-1, Parent[2]=-1, ..., Parent[8]=-1. All eight positions contain -1, representing eight separate trees with single nodes.
Find (i)
The find operation traverses from the given element i up to the root of its tree. It starts with j=i and follows parent pointers until reaching a node with parent value less than 0 (i.e., -1, the root).
π Formula: for(j=i; parent[j] >= 0; j=parent[j]) ; return j β This loop continues as long as parent[j] is not -1, moving j to its parent each iteration, eventually returning the root.
π Example: In the array after several unions, find(8) starts at position 8. Since Parent[8]=7, j becomes 7. Parent[7]=5, so j becomes 5. Parent[5]=3, so j becomes 3. Parent[3]=-1, so the loop ends, returning 3. Thus, 8 belongs to set 3.
Union (i, j)
The union operation merges two sets. It first finds the roots of both i and j using the find function. If the roots are different, it makes the root of j a child of the root of i by setting Parent[root_j] = root_i.
π Formula: root_i = find(i); root_j = find(j); if (root_i != root_j) parent[root_j] = root_i; β This merges the set containing j into the set containing i, with root_i becoming the new root.
π Example: After calling union(4,5), find(4) returns 3 (the root of 4's set), and find(5) returns 5 (itself a root). Since 3 β 5, Parent[5] is set to 3. Now the tree has root 3 with children 4 and 5, and 5 has children 6, 7, and 8 in its subtree.
Example 2
This example traces the same union operations using the parent array representation. After all unions, the array shows: Parent[1]=-1, Parent[2]=-1, Parent[3]=-1, Parent[4]=3, Parent[5]=3, Parent[6]=5, Parent[7]=5, Parent[8]=7. The number of -1 values (four: positions 1, 2, 3, and none at 4,5,6,7,8) indicates four trees in the forest.
π Example: To verify find(6) in this array: Start at 6, Parent[6]=5 β j=5. Parent[5]=3 β j=3. Parent[3]=-1, return 3. So 6 belongs to set 3.
Running Time Analysis
The parent array implementation is more space-efficient than a Boolean matrix. For N elements, a Boolean matrix requires NΒ² locations, while the parent array needs only N locations. The union operation is a constant time operation (O(1)) once roots are found. The running time of find(i) is proportional to the height of the tree containing node i, which can be O(N) in the worst case. The goal is to modify union to ensure that tree heights stay small.
π‘ Why this matters: The parent array approach uses significantly less space than a two-dimensional matrix while still efficiently handling equivalence relations. However, tree height can degrade performance, motivating improvements like union by size or rank.
β Key Takeaways
The dynamic equivalence problem is solved using trees to represent sets, with a forest representing all sets and the root serving as the set name. The parent array provides an efficient one-dimensional storage structure using array indices as pointers, with -1 indicating roots. Union is a constant-time operation once roots are found, while find's running time depends on tree height and can be O(N) in the worst case. The number of trees in the forest can be determined by counting the number of -1 values in the parent array. The key trade-off is between the simplicity of this array-based approach and the potential for performance degradation due to tree height, motivating future improvements.
π§ Quick Revision Questions
- In the parent array representation, what does the value -1 at position i indicate?
- After performing union(5,6) followed by union(5,7), what would be the parent of 7 in the array?
- What is the running time complexity of the find operation in the worst case, and what factor determines it?
- How many trees exist in a forest represented by a parent array with the following values: [-1, -1, -1, 3, 5, 5, 7]?
- In the union function implementation, why is it necessary to first call find on both arguments before merging the sets?