π Lecture 13 β Instruction Level Parallelism (Dynamic Scheduling - Scoreboard Approach)
π Overview: This lecture introduces dynamic scheduling as a hardware approach to exploit instruction-level parallelism by allowing out-of-order execution. It focuses on the Scoreboard technique from the CDC 6600, detailing how it resolves data hazards (RAW, WAR, WAW) through a centralized hardware structure that manages instruction issue, operand reading, execution, and result writing.
ποΈ Topics Covered
The lecture begins by recapping in-order execution limitations and the problems of out-of-order execution. It then defines dynamic scheduling versus static scheduling. The core of the lecture explains the Scoreboard technique, including its three main data structures (Instruction Status, Functional Unit Status, Register Result Status) and the four pipeline stages it controls (Issue, Read Operand, Execute, Write Result). A detailed, cycle-by-cycle example of scoreboard operation is provided to show how instructions are issued in-order but executed and completed out-of-order, with specific attention to structural, RAW, and WAR hazard handling.
π Lecture Summary
Recap: Lecture 12
The recap reviews that in the standard FP pipeline, only one instruction is issued per clock cycle. Integer ADD instructions have zero latency and pass through the pipeline normally. However, FP add, multiply, and divide instructions have longer latencies, causing them to loop in the EX-stage and increasing stalls. RAW and WAR hazards can occur because instructions of varying lengths may reach the Write Back (WB) stage out-of-order. The WAW hazard occurs when the j-th instruction writes before the i-th instruction, and the i-th instruction overwrites the result. WAW hazards are resolved by either delaying the issue of the j-th instruction until i-th writes, or by stalling the i-th instruction's write.
In-Order Execution
In a simple pipelined datapath, In-order execution is enforced: instructions are fetched, decoded, and issued strictly in program sequence. If an instruction stalls due to a structural hazard or data dependence, no later instruction can proceed. In the MIPS 5-stage pipeline, both structural and data hazards are checked during the Instruction Decode (ID) stage. The issue process at ID is separated into two parts: checking for structural hazards and waiting for the absence of data hazards.
For example, in the code:
DIV.D F0, F2, F4
ADD.D F10, F0, F8
SUB.D F12, F8, F14
ADD.D must wait for DIV.D because of the RAW hazard on F0.
Basic Problems of Out-of-order Execution
Out-of-order execution allows instructions to execute when their operands are available, not necessarily in program order. This introduces problems like WAR and WAW hazards and complicates precise exceptions. Consider this FP code:
DIV.D F0, F2, F4
ADD.D F6, F0, F8
SUB.D F8, F10, F14
MUL.D F6, F10, F8
Here, SUB.D writes F8 (a source for MUL.D), creating a possible WAR. MUL.D writes F6 (the same destination as ADD.D), creating a WAW.
Dynamic Scheduling
- Static Scheduling: The compiler rearranges instruction execution to avoid hazards.
- Dynamic Scheduling: The hardware rearranges instruction execution at runtime to avoid stalls. This allows the processor to issue instructions while previous ones are stalled, handling hazards that are not known at compile time.
π‘ Why this matters: Dynamic scheduling is critical for modern processors because it can deal with unpredictable runtime conditions (like cache misses) that static scheduling cannot.
Dynamic Scheduling: Scoreboarding Technique
The Scoreboard technique was introduced in the CDC 6600, which contained 4 FP units, 5 Memory Reference Units, and 7 integer operation units. In the MIPS processor, a Scoreboard is a centralized hardware structure that tracks all instructions and functional units.
Features of the Scoreboard: The Scoreboard maintains all necessary information to control and coordinate instruction execution.
Components of Scoreboard:
- Instruction Status: Tracks which of four stages each instruction is in.
- Functional Unit Status: Tracks the state of each functional unit (FU).
- Register Result Status: Tracks which functional unit will write to each register.
Instruction Status
Each instruction goes through four stages, controlled by the Scoreboard:
- Issue: If a functional unit for the instruction is free and no other active instruction has the same destination register, the Scoreboard issues the instruction to the FU and updates its internal data structure. This guarantees that WAW hazards cannot occur.
- Read Operand: The Scoreboard monitors the availability of source operands. It checks if no earlier issued active instruction is going to write to the source register. This resolves RAW hazards.
- Execute: The FU begins execution and notifies the Scoreboard when it has completed.
- Write Result: The Scoreboard checks for WAR hazards. If a stall is necessary, the result write is delayed. Otherwise, the result is written.
The Instruction Status data structure is a table with rows for each instruction and columns for each stage (Issue, Read, Execution complete, Write result). Checkmarks (β) indicate the stage is complete. For example:
| Instruction | Issue | Read | Execution | Write result |
|---|---|---|---|---|
| MUL.D | β | β | ||
| ADD.D | β | β | β | β |
Functional Unit Status
This table tracks the state of each FU. Its fields are:
- Busy: A single bit indicating if the FU is busy (Y or N).
- OP: Specifies the operation (ADD, SUB, etc.).
- Registers: Fi (Destination register), Fj (first source register), Fk (second source register).
- Qj and Qk: The name of the FU that will produce source registers Fj and Fk. A value of "--" means the register is already available.
- Rj, Rk: Flags indicating whether source registers Fj and Fk are ready and not yet read. Set to "No" when the operands are read.
Example of a functional unit status table:
| FU name | Busy | OP | Fi | Fj | Fk | Qj | Qk | Rj | Rk |
|---|---|---|---|---|---|---|---|---|---|
| MUL1 | Y | Mul | F0 | F2 | F4 | -- | -- | No | No |
| DIVIDE | Y | Div | F10 | F0 | F6 | MUL1 | -- | No | Yes |
In the DIVIDE row, Qj = MUL1 indicates that the first source register (F0) is waiting for the result from the MUL1 unit. Rj = No means the operand has not been read yet (or is waiting). Rk = Yes means the second source register (F6) is ready and has not been read.
π Definition β Qj/Qk: These fields hold the identifier of the functional unit that will write the result needed by the source registers (Fj, Fk). This is the key mechanism for resolving RAW hazards dynamically.
Register Result Status
This table has rows for each functional unit and columns for each register (F0 to F30). The entry indicates which functional unit (if any) will write its result to that register. For example:
| F0 | F2 | F4 | F6 | F8 | F10 | F12 | ... | F30 | |
|---|---|---|---|---|---|---|---|---|---|
| FU | Mul1 | Add | Divide |
This shows that MUL1 will write to F0, Add will write to F6, and Divide will write to F10.
Detailed Scoreboard Pipeline Control
The lecture presents a diagram showing how the Scoreboard sits between the instruction fetch/decode stage and the functional units. The Scoreboard reads the instruction from the ID stage, checks for hazards using its data structures, and either issues the instruction to a FU (if safe) or stalls it. The FUs then communicate back to the Scoreboard when execution is complete and when writing results.
Scoreboard Example
The lectures run a detailed example using a code sequence with FP Add latency = 2 clocks, Multiply = 10 clocks, and Divide = 40 clocks. The example traces the pipeline state cycle by cycle.
Scoreboard Example Code:
LD F6, 34(R2)
LD F2, 45(R3)
MULTD F0, F2, F4 ; Multiply
SUBD F8, F6, F2 ; Subtract
DIVD F10, F0, F6 ; Divide
ADDD F6, F8, F2 ; Add
The following is a summary of the cycle-by-cycle trace:
π Definition β Structural hazard: Occurs when a required functional unit is already busy.
- Cycle 1: LD F6 issues to the load unit.
- Cycle 2: LD F2 issues to the load unit.
- Cycle 3: The Scoreboard tries to issue MULTD. However, note that the code is LD, MULTD, SUBD... The issue attempt for MULTD might stall if the FU (MUL1) is busy? No, in this cycle, the issue for MULTD is checked. The example shows that in Cycle 3, the issue of MULTD is stalled on a structural hazard because the MUL unit is not yet available? Wait, the cycle-by-cycle trace in the lecture shows:
- Cycle 3: Issue MULT? No, stall on structural hazard. This is because the MULTD instruction requires a FU, but the scoreboard assumes only one MUL unit. Since MULTD needs the MUL unit, and it might be free, why the stall? The lecture text for Cycle 3 states: "Issue MULT? No, stall on structural hazard". Let's follow the exact sequence.
- Cycle 4: The MULTD instruction is now issued to the MUL1 unit. The Functional Unit Status and Register Result Status are updated to reflect that MUL1 is busy with MULTD, writing to F0.
- Cycle 5: SUBD issues to the Add unit. LD F2 writes its result (or passes to the next stage). The example shows the flow of instructions to their respective FUs.
- Cycle 6: DIVD issues to the Divide unit. The example shows how the scoreboard tracks the WAW hazard? No, there is no WAW here yet. The FUs are all busy.
- Cycle 7: The scoreboard checks if it can read multiply operands. Since F2 and F4 are ready (from the load), the operands for MULTD are read.
- Cycle 8a (first half): The scoreboard checks if the next instruction (SUBD) can read its operands. F6 is still being loaded? In this cycle, SUBD's source F6 might be waiting. The scoreboard also tries to issue ADDD? No, ADDD's destination is F6, which will be written by SUBD, so this creates a WAW hazard on F6.
- Cycle 8b (second half): The scoreboard decides to stall the issue of ADDD due to the WAW hazard.
- Cycle 9: The scoreboard checks if it can read operands for MULT and SUBD. It attempts to issue ADDD, but it is still stalled.
- Cycle 11: SUBD's operands (F6, F2) become available. The scoreboard reads operands for SUBD.
- Cycle 12: The scoreboard tries to read operands for DIVD (F0, F6). F0 is waiting on MULTD. DIVD's operand read is blocked (RAW hazard).
- Cycle 13: ADDD's issue is still stalled due to the WAW hazard.
- Cycle 14: The MULTD instruction completes execution (having taken its latency). The scoreboard is notified.
- Cycle 15: MULTD is ready to write its result. The scoreboard checks for WAR hazards. Is there any instruction that needs to read F0 (the destination)? Yes, DIVD needs F0. If MULTD writes F0 now, and DIVD has not yet read it, DIVD would get the old value of F0, causing a WAR hazard. Therefore, MULTD's write result is stalled.
- Cycle 16: The stall continues. The scoreboard waits for all instructions that need to read F0 to do so.
- Cycle 17: SUBD (which does not need F0) might be ready to write its result. The scoreboard again checks for WAR on F8 (SUBD's destination). If any instruction needs F8 (like ADDD), the write could be stalled. In this cycle, the lecture asks: "Write result of ADDD?" but ADDD is still stalled in issue. The example shows SUBD writing its result.
- Cycle 18: The WAR hazard on MULTD is resolved (DIVD has read its operands). MULTD now writes its result to F0. The Register Result Status for F0 is cleared.
- Cycle 19: Now that F0 is available, DIVD can read its operands (F0 is now ready). The scoreboard reads operands for DIVD.
- Cycle 20-21: The pipeline progresses. ADDD might finally be issued (since the WAW hazard with SUBD is resolved after SUBD writes).
- Cycle 22: ADDD is issued. Its operands are available.
- Cycle 61: DIVD completes execution (after 40 cycles). It notifies the scoreboard.
- Cycle 62: DIVD writes its result. The scoreboard checks for WAR hazards. Since all subsequent instructions have been issued, there is no WAR, and the write completes.
π Formula/Key Concept: Hazard Resolution Rules in Scoreboard
- WAW: Prevented by checking the destination register against the Register Result Status table during the Issue stage. If another active instruction will write to the same register, the issue is stalled.
- RAW: Handled by monitoring Qj/Qk fields in the Functional Unit Status. An instruction cannot read its operands (Read Operand stage) until the FU producing the operand (identified by Qj/Qk) has written the result (or the operand is ready).
- WAR: Handled during the Write Result stage. A result write is stalled if the destination register is needed as a source by any instruction that has not yet read its operands.
π Example (RAW hazard): DIVD F10, F0, F6. F0 is produced by MULTD. Qj for DIVD is set to MUL1. DIVD's Read Operand stage stalls until MUL1 writes F0. This happens at Cycle 18, allowing DIVD to read at Cycle 19.
π Example (WAW hazard): ADDD F6, F8, F2 wants to write to F6. However, SUBD F8, F6, F2 is already active and will also write to F8 (No, SUBD writes F8). Wait, the lecture example shows ADDD F6, F8, F2 and SUBD F8, F6, F2. The WAW is on F6? No, SUBD writes F8. The WAW in this code is: Before ADDD, there is no other instruction writing to F6? Actually, there is no WAW on F6 in this code from the load? Let's re-check. The code is:
- LD F6, 34(R2)
- LD F2, 45(R3)
- MULTD F0, F2, F4
- SUBD F8, F6, F2
- DIVD F10, F0, F6
- ADDD F6, F8, F2 Here, LD F6 writes to F6 (the first load). ADDD F6 also writes to F6. This is a WAW hazard between LD F6 and ADDD F6. The scoreboard stalls the issue of ADDD (instruction 6) until the previous write to F6 (from LD F6) is no longer active in the pipeline.
π Example (WAR hazard): MULTD F0, F2, F4 completes execution and is about to write F0. But DIVD F10, F0, F6 still needs to read F0 as a source. If MULTD writes F0 now, DIVD would read the old F0? No, it would read the new F0, but that is a RAW, not a WAR. Wait. The WAR in this code is actually on F6. SUBD F8, F6, F2 writes to F8. ADDD F6, F8, F2 wants to read F8. But DIVD F10, F0, F6 wants to read F6. The WAR occurs when an earlier instruction (DIVD) reads a register (F6) that a later instruction (ADDD) will write. So, when ADDD is ready to write F6, the scoreboard stalls that write until DIVD (which was issued earlier but still executing) reads its source F6. In the cycle-by-cycle, the WAR on MULTD's write of F0 is that DIVD needs F0. If MULTD writes F0 (Cycle 15-17), DIVD (which hasn't read F0 yet due to its own RAW) would get the new F0, but DIVD is an earlier instruction that should read the old F0. So yes, MULTD's write is stalled because DIVD (an earlier instruction) hasn't read its source F0 yet.
β Key Takeaways
The Scoreboard technique enables in-order issue but out-of-order execution and out-of-order completion. It uses three tables to track all instructions and functional units, resolving WAW hazards by stalling issue, RAW hazards by stalling the read of operands, and WAR hazards by stalling the write of results. The critical design point is that a result write is delayed until all earlier instructions that need to read that destination register have done so, preventing WAR. This lecture demonstrates through a cycle-by-cycle example that dynamic scheduling with a scoreboard can significantly reduce pipeline stalls compared to in-order execution, but it still incurs penalties from structural, RAW, and WAR hazards. The key trade-off is the increased hardware complexity and the need for sophisticated control logic.
π§ Quick Revision Questions
- What are the four stages that the Scoreboard controls for each instruction?
- How does the Scoreboard detect and prevent a WAW hazard?
- How does the Scoreboard handle a RAW hazard? Explain the role of the Qj/Qk fields.
- Describe exactly when a WAR hazard is detected and resolved in the Scoreboard pipeline.
- In the example, why was MULTD's write result stalled in Cycle 15, even though MULTD had finished execution at Cycle 14?
π Lecture 14 β Instruction Level Parallelism (Dynamic Scheduling β Tomasuloβs Approach)
π Overview: This lecture introduces Tomasulo's Algorithm, a hardware-based dynamic scheduling technique that overcomes limitations of the Scoreboard method. It explains how register renaming and reservation stations enable out-of-order execution while eliminating WAW and WAR hazards, and demonstrates the algorithm through detailed cycle-by-cycle examples and loop unrolling.
ποΈ Topics Covered
The lecture begins with a recap of the Scoreboard method and its limitations, then introduces Tomasulo's Algorithm as an alternative dynamic scheduling approach. It covers the organizational structure including reservation stations, load/store buffers, and the Common Data Bus. The three stages of Tomasulo's algorithm (Issue, Execution, Write Result) are explained through a detailed step-by-step example, followed by a comparison with Scoreboard execution. A second example demonstrates loop unrolling and register renaming in action.
π Lecture Summary
Recap - Lecture 13
The Scoreboard method divides the ID stage into two parts: Issue (in-order) and Read Operands (out-of-order). It has three parts: Instruction status, Functional unit status (with 9 fields including Busy, Op, Fi, Fj, Fk, Qj, Qk, Rj, Rk), and Register result status. The method enables out-of-order execution resulting in out-of-order completion.
However, Scoreboard has several limitations: no forwarding (first write register then read it), limited to instructions in basic block (small window), number of functional units (structural hazards), waiting for WAR hazards, and preventing WAW hazards. A comparison example showed Scoreboard completing at cycle 62 while Tomasulo completed earlier.
Dynamic Scheduling
Dynamic scheduling allows hardware to reorder instructions at runtime to exploit ILP when dependencies cannot be determined at compile time. This is accomplished by dividing instruction processing into stages that allow instructions behind stalls to proceed.
Tomasulo's Approach
Tomasulo's Algorithm was introduced for the IBM 360/91 about 3 years after CDC 6600 (1966), with the goal of high performance without special compilers. Key differences from Scoreboard include: IBM has only 2 register specifiers per instruction vs. 3 in CDC 6600; IBM has 4 FP registers vs. 8 in CDC 6600.
The algorithm uses distributed control and buffers with functional units instead of centralized control in Scoreboard. The FU buffers are called reservation stations which hold pending operands. Register renaming replaces registers in instructions with values or pointers to reservation stations, which avoids WAR and WAW hazards. More reservation stations than registers enable optimizations that compilers cannot achieve.
Results go from FU to reservation stations over the Common Data Bus (CDB) which broadcasts results to all FUs, unlike Scoreboard where results go through registers. The CDB differs from a normal bus: normal data bus has data + destination ("go to" bus), while CDB has data + source ("come from" bus). The CDB carries 64 bits of data + 4 bits of Functional Unit source address.
π Definition β Reservation Station: A buffer at each functional unit that holds the operation, actual operands, and information to resolve hazards. Components include Op, Busy, Vj/Vk (value of source operands), Qj/Qk (reservation stations producing source values), and Qi (for store buffers only). Note: No ready flags as in Scoreboard; Qj,Qk=0 means ready.
Load Buffers have three functions: hold components of effective address until computed, track outstanding loads waiting on memory, and hold the result of completed load waiting for CDB.
Store Buffers have three functions: hold components of effective address until computed, hold destination memory address of outstanding store instructions, and hold address and value of store until memory unit is available.
Three Stages of Tomasulo's Algorithm
- Issue β Get instruction from FP Op Queue. If reservation station is free (no structural hazard), control issues instruction and sends operands to the RS (renames registers).
- Execution β Operate on operands (EX). When both operands ready then execute; if not ready, watch Common Data Bus for result.
- Write Result β Finish execution (WB). Write on Common Data Bus to all awaiting units; mark reservation station available.
Tomasulo Example Cycles 0-16
The example sequence shows instructions: L.D F6, 34(R2); L.D F2, 45(R3); MULT.D F0, F2, F4; SUB.D F8, F6, F2; DIV.D F10, F0, F6; ADDD F6, F8, F2.
Key observations from the cycle-by-cycle execution:
- Cycle 1: L.D F6 issues and uses Load1
- Cycle 2: L.D F2 issues and uses Load2; unlike Scoreboard, can have multiple loads outstanding
- Cycle 3: MULT.D issued (vs. Scoreboard stalled); registers names removed ("renamed") in Reservation Stations
- Cycle 4-5: Loads completing; broadcasts on CDB to waiting units
- Cycle 6: ADDD issued (vs. Scoreboard where it was delayed)
- Cycle 7: Add1 completing; broadcasts result
- Cycle 10-11: Add2 completing; writes result (vs. Scoreboard where WAW hazard prevented writing)
- Cycle 14-16: MULT1 completing; broadcasts result faster than Scoreboard's cycle 62
The algorithm demonstrates in-order issue, out-of-order execution and completion.
Comparing to Scoreboard Cycle 62
Tomasulo completes earlier because it eliminates WAW and WAR hazards through register renaming, allows multiple loads outstanding, and uses CDB broadcast for forwarding. Scoreboard takes longer due to no forwarding, waiting for WAR hazards, and preventing WAW hazards.
Another Example: Loop Unrolling
This example multiplies an array (F0) by a scalar F2 using a loop. Tomasulo's algorithm eliminates WAW and WAR hazards through register renaming. The reservation stations allow multiple executions of the loop by dynamic unrolling when branches are predicted taken.
The loop shows two active iterations based on branch prediction taken. Key observations:
- Cycle 3: MULT1 has no register names in RS
- Cycle 6: F0 never sees Load1 result (register renaming prevents WAW)
- Cycle 7: MULT2 has no register names in RS
- Cycle 9-10: Load1 and Load2 completing; broadcasting results
- Cycle 14-15: MULT1 and MULT2 completing
- Cycle 16-21: Remaining instructions complete with CDB broadcasting results
π‘ Why this matters: Register renaming prevents the WAW hazard that would occur when multiple iterations try to write to F0, and prevents WAR hazards when instructions read registers that later iterations modify.
β Key Takeaways
Tomasulo's Algorithm is a dynamic scheduling technique that uses reservation stations for distributed control and register renaming to eliminate WAW and WAR hazards. The three-stage pipeline (Issue, Execution, Write Result) enables in-order issue with out-of-order execution and completion, where the Common Data Bus broadcasts results to all awaiting units. Unlike Scoreboard, Tomasulo supports multiple outstanding loads, forwarding through CDB, and more reservation stations than registers for compiler-like optimizations. The algorithm achieves faster execution than Scoreboard by eliminating structural hazards for independent operations and allowing parallel execution across multiple iterations through dynamic loop unrolling. Register renaming replaces architectural register names with pointers to reservation stations, effectively removing name dependencies and enabling greater ILP exploitation.
π§ Quick Revision Questions
- What are the three stages of Tomasulo's Algorithm and what happens in each?
- How does register renaming eliminate WAW and WAR hazards?
- What is the difference between the Common Data Bus and a normal data bus?
- Why can Tomasulo issue MULT.D in cycle 3 while Scoreboard had to stall?
- How does Tomasulo's algorithm achieve dynamic loop unrolling?
π Lecture 15 β Instruction Level Parallelism (Dynamic Branch Prediction)
π Overview: This lecture explores dynamic branch prediction techniques essential for high-performance pipelined processors. It introduces Branch History Tables (BHT) with 1-bit and 2-bit predictors, explains correlating branch predictors that exploit relationships between branches, and discusses accuracy improvements and implementation through Branch Target Buffers (BTB).
ποΈ Topics Covered
The lecture begins with a recap of Tomasulo's approach, then covers Dynamic Branch Prediction through Branch History Table concepts, 1-bit and 2-bit dynamic branch prediction with their accuracy limitations, Branch Prediction Buffer (BPB) implementation, correlating branches using (m,n) predictors, and concludes with Branch Target Buffer techniques and accuracy comparisons across different schemes.
π Lecture Summary
Recap: Lecture 14
Tomasulo's approach for IBM 360/91 achieves high performance without special compilers by distributing control and buffers with Function Units (FU). Registers in instructions are replaced by values or pointers to reservation stations (RS) , implementing register renaming. Unlike Scoreboard, Tomasulo can have multiple loads outstanding. These properties allow issuing instructions with name dependence (e.g., MULT issued with name dependence on register F2). Tomasulo eliminates WAR hazards (ADD.D writes result at Cycle 11 even though DIV.D starts execution at Cycle 16). Tomasulo issues instructions in-order but may execute them out-of-order. Integer instructions SUBI and BNEZ are executed out-of-order to evaluate conditions, and branch-taken prediction is implemented by repeating loop instructions.
Introduction to Dynamic Branch Prediction
In the previous lecture, a loop-based example demonstrated Tomasulo's approach for overcoming WAW and WAR hazards. Dynamically scheduled pipelines can yield high performance provided branches are predicted accurately.
π Definition β Branch History Table (BHT) : A small memory structure that stores prediction bits for recently executed branches. If the prediction is wrong, the prediction-bit is inverted.
1-bit Dynamic Branch Prediction
Problem: In a loop, a 1-bit BHT causes two mispredictions in a row. A 1-bit predictor mispredicts at twice the rate that the branch is not-taken.
π Example: Consider a loop branch (For i=1 to 10): the branch is taken 9 times and not-taken once. Performance = Ζ(accuracy, cost of mispredictions). The expected accuracy should match the taken-branch frequency (9 out of 10 = 90%). But the 1-bit prediction achieves only 8 out of 10 (80%).
π‘ Why this matters: The 1-bit predictor fails because after the loop exits (not-taken), on the next loop iteration it predicts not-taken again, causing two mispredictions per loop completion.
2-bit Dynamic Branch Prediction
Two bits encode four states in a saturating counter system:
- States 00 and 01: Predict Not-Taken
- States 10 and 11: Predict Taken
In a saturating counter implementation, the 2-bit counter saturates at 00 (Predict Taken) or 11 (Predict Not Taken). The counter is incremented when a branch is taken and decremented when it is not taken.
π Definition β Saturating Counter: A counter that stops incrementing or decrementing when it reaches its minimum (00) or maximum (11) value.
π Example: State transitions:
- 00 β 01 for Taken when predicted not taken
- 10 β 11 for Taken when predicted taken
When the counter is β₯ Β½ of its maximum value (β₯10, meaning states 01 and 11), the branch is predicted as taken. Otherwise (<10: states 10 and 00), the branch is predicted as not taken.
π Example: Loop For i=1 to 10 with 2-bit predictor:
| Iteration | Previous State | Branch | Next State | Prediction |
|---|---|---|---|---|
| 0 | -- | not Taken | 11 | Taken |
| 1 | 11 | Taken | 11 | Taken |
| 2 | 11 | Taken | 11 | Taken |
| ... | ... | ... | ... | ... |
| 9 | 11 | Taken | 11 | Taken |
| 10 | 11 | Not taken | 10 | Taken |
Prediction fails only once. This is a major improvement over the 1-bit predictor which failed twice.
Branch Prediction Buffer (BPB) or BHT Implementation: If prediction is wrong, prediction bits are changed. In case of predicted taken, state changes 11β10. In case of predicted not taken, state changes 00β01.
Branch History Table Accuracy
For SPEC89 benchmark, a branch prediction buffer with 4096 entries results in prediction accuracy ranging from 99% to 82%, or misprediction rates of 1% to 18%.
Impact of Size on Accuracy of BHT
As we try to exploit more ILP, the accuracy of the branch predictor becomes critical. Accuracy is shown by increasing buffer size:
- 4096 Entries with 2-bit BHT
- Unlimited Entries with 2-bit BHT
π‘ Why this matters: Simply increasing the number of bits per predictor without changing the predictor structure has little impact. Alternative methods are needed to increase prediction accuracy.
Correlating Branches
The 2-bit predictor scheme uses only the recent behavior of a single branch to predict future behavior. In practice, the behavior of other branches may also influence prediction accuracy.
π Example: SPEC92 benchmark worst case for 2-bit predictor:
Assume aa is assigned R1 and bb is assigned R2
IF (aa==2) DSUBUI R3, R1, #2
aa=0; BNEZ R3, L1 ; branch b1 (aa!=2)
DADD R1, R0, R0 ; aa=0 Not Branch
IF (bb==2) L1: DSUBUI R3, R2, #2
bb=0; BNEZ R3, L2 ; branch b2 (bb!=2)
DADD R2, R0, R0 ; bb=0 Not Branch
IF (aa!=bb) L2: DSUBU R3, R1, R2
{ BEQZ R3, L3 ; branch b3 (aa=bb)
Here, the behavior of b3 (L2) is correlated with the behavior of b1 and b2. If b1 and b2 are both not-taken (aa=0, bb=0), then b3 is taken. A predictor using only single branch behavior cannot capture this.
Correlating Branch Predictors
Hypothesis: Recent branches are correlated; behavior of recently executed branches affects prediction of current branch.
π Definition β (m,n) Predictor: A predictor that records the last m branches to select between 2^m history tables, each with n-bit counters. The old 2-bit BHT is a (0,2) predictor.
π Example: Illustrative code (d is assigned to R1):
IF (d==0) BNEZ R1, L1 ; branch b1 (d!=0)
d=1; DADDIU R1,R0,#1 ; branch not taken, d=1
IF (d==1) L1: DADDIU R3, R1, #-1
BNEZ R3, L2 ; branch b2 (d!=1)
Working of correlating predictor:
| Initial d | d==0? | b1 | d before b2 | d==1? | b2 |
|---|---|---|---|---|---|
| 0 | yes | NT | 1 | yes | NT |
| 1 | no | T | 1 | yes | NT |
| 2 | no | T | 2 | no | T |
If b1 is not taken, b2 will not be taken. Prediction bits are written as: Prediction if last branch is not-taken / Prediction if last branch is taken.
Four possible combinations:
| Prediction bits | New Prediction if last branch Not Taken | New Prediction if last branch Taken |
|---|---|---|
| NT/NT | NT | NT |
| NT/T | NT | T |
| T/NT | T | NT |
| T/T | T | T |
For the (1,1) predictor (1-bit predictor with 1-bit of correlation), the only misprediction is on the first iteration when d=2, as this is not correlated with the previous prediction.
π Example: A (2,2) branch prediction buffer uses 2-bit global history to choose from among 4 predictors for each branch address. Behavior of recent branches selects between four predictions of the next branch, updating just that prediction.
π‘ Why this matters: Correlating predictors significantly reduce misprediction rates by capturing patterns that single-branch predictors cannot detect.
Dynamic Branch Prediction Summary
- Branch History Table: Uses 2 bits for loop accuracy
- Correlation: Recently executed branches are correlated with the next branch
- Branch Target Buffer: Includes both branch address and prediction
- Predicated Execution: Can reduce the number of branches and number of mispredicted branches
β Key Takeaways
The 1-bit branch predictor causes two mispredictions per loop completion (one for exit, one for re-entry), achieving only 80% accuracy compared to the expected 90%. The 2-bit saturating counter predictor improves this to a single misprediction per loop using four states (00, 01, 10, 11) and saturating behavior. Correlating branch predictors, represented as (m,n) predictors, use the history of the last m branches to select between 2^m tables of n-bit counters, capturing relationships between branches that single-branch predictors cannot. The accuracy of predictors depends on both the size of the Branch History Table and the predictor structure, with simply increasing bits per predictor having little impact. Branch Target Buffers combine branch address storage with prediction information to further improve performance.
π§ Quick Revision Questions
- Why does a 1-bit dynamic branch predictor cause two mispredictions in a row for a loop branch?
- How many states does a 2-bit saturating counter predictor have, and what are the state transitions when a branch is taken vs. not taken?
- In the correlating branch example with branches b1, b2, and b3, why can't a single-branch predictor correctly predict b3?
- What does the notation (m,n) mean in the context of correlating branch predictors?
- What is the prediction accuracy range reported for a 4096-entry 2-bit BHT on SPEC89 benchmarks?
π Lecture 16 β Instruction Level Parallelism (Dynamic Branch Prediction .... Contβd)
π Overview: This lecture continues the exploration of dynamic branch prediction techniques in instruction-level parallelism. It covers correlating branch predictors, tournament predictors, and the Branch Target Buffer (BTB) for high-performance instruction delivery. These concepts are critical for reducing branch penalties in modern pipelined processors.
ποΈ Topics Covered
The lecture recaps dynamic scheduling and branch prediction, then examines correlating branch predictors with (m,n) notation and examples. It introduces multilevel tournament predictors with state transition diagrams, followed by a detailed discussion of the Branch Target Buffer (BTB) including its structure, operation flow, penalty calculations, and improvement techniques.
π Lecture Summary
Recap: Dynamic Scheduling and Branch Prediction
- Static scheduling relies on the software (compiler), while dynamic scheduling uses hardware-intensive approaches for branch prediction.
Important questions: Branch-Prediction Buffer
Q1: What is the impact of increasing the size of branch-prediction buffer on two branches in a program?
- A single predictor predicting a single branch is generally more accurate than the same predictor serving more than one instruction.
- It is less likely that two branches in a program share a single predictor.
- Therefore, increasing the size of predictor buffer does not have significant effect on two branches in a program.
Q2: How does sharing a predictor affect the misprediction rate?
Example: Sequence 1
- Here, branches B1 and B2 share a 1-bit predictor.
- B1 is always TAKEN, and B2 is always NOT-TAKEN.
Example: Sequence 2
- Another sequence of branches sharing the same predictor.
Example: Conclusion
- Why does sharing a predictor increase misprediction rate?
- If a predictor is shared by a set of branch instructions, then the members of the set of branch instructions may change over the course of execution of a long program.
- Hence, the branch action history changes and the predictor is likely to mispredict more often.
Correlating Branch Predictors Re-visited
- In the program segment:
IF (d==0) Branch b1 for d!=0 d=1;IF (d==1) Branch b2 for b!=1 d=2; - This problem may be resolved in a Correlating-Branch Predictor by recording the m most recently executed branches as taken or not taken (in 2^m branch-history tables for 1-, 2-, or n-bit predictors), and using the branch-pattern to select the proper branch history table for the current branch.
π Definition β (m, n) predictor: Records the last m branches to select between 2^m history tables, each with n-bit counters (2^m n-bit predictors).
- A 2-bit BHT is regarded as a (0,2) correlating predictor.
Example:
-
A 1-bit predictor with 1-bit correlation is written as (1,1) predictor.
-
Here, we have two (2^1) separate prediction bits (i.e., two 1-bit BHTs):
- One prediction bit is used if the last branch executed was not-taken.
- The other prediction bit is used if the last branch executed was taken.
-
Denoted as: (New prediction when last NT / New prediction when last T).
- E.g., T/NT stands for: New prediction is TAKEN if previous was NOT-TAKEN, and is NOT-TAKEN if previous was TAKEN.
-
In an (m,n) predictor, the global history of the most recent m branches is recorded in an m-bit shift register.
-
Each bit records whether the branch was taken or not taken.
-
The branch-prediction buffers are indexed using concatenation of low-order bits from branch-address with m-bit global history.
(2, 2) Correlating Branches Predictor:
- The buffer is drawn as a 2-dimensional object; each buffer is 2 bits wide. In reality, they are arranged linearly.
- A (2, 2) branch prediction buffer uses 2-bit global history to choose from among 4 predictors for each branch address of 4-bit (among the 16 entries in each of the 4 predictors).
- Behavior of recent branches selects between four predictions of the next branch, updating just that prediction.
- Indexing is done by concatenation of 4 lower-order address bits of the branch (word address) and 2 global bits to form a 6-bit address to select a 2-bit prediction from 64 entries in 4 buffers, each of 16 entries.
Comparison of (0,2) and (2,2) predictors β The (2,2) predictor shows higher prediction accuracy (e.g., 99.7% for eqntott benchmark vs. 96.5% for (0,2)), demonstrating that correlating predictors perform better than non-correlating ones.
Multilevel Branch Predictors (Tournament Predictors)
- Multilevel branch prediction (nested branches) involves information at local and global levels to predict correctly.
- It uses several levels of Branch-Prediction Tables and an algorithm to choose among different predictors.
π Definition β Tournament Predictor: A multilevel branch predictor that uses a selector (often a 2-bit saturating counter) to choose between two or more different predictors (e.g., a global predictor and a local predictor) based on which one has been more accurate recently.
State Transition Diagram of Tournament Predictor
- The transition for the predicted predictor is specified by:
- Correct = 1
- Incorrect = 0
- The state transition diagram shows that from the saturating state:
- The counter is incremented whenever the predicted predictor is correct and the other is incorrect (i.e., for 1/0).
- The counter is decremented in the reverse direction (i.e., for 0/1).
- The counter does not change for all other predictions for non-saturating present state.
- For the saturating state 00 (Use predictor 1):
- It increments to state 01 (use predictor 1) for 1/0.
- It decrements to state 11 (use predictor 2) for 0/1.
- For the saturating state 11 (Use predictor 2):
- It increments to state 00 (use predictor 1) for 1/0.
- It decrements to state 10 (use predictor 2) for 0/1.
π‘ Why this matters: Tournament predictors can achieve higher prediction accuracy by dynamically selecting the best predictor based on recent behavior, reducing misprediction penalties.
High Performance Instruction Delivery
- In the MIPS 5-stage pipeline, we need to know the address of the next instruction fetch at the end of the current IF cycle.
- For zero branch penalty, we need to know whether the as-yet undecoded instruction is a branch, and if yes, then what is the next-PC.
Branch Target Buffer
- This is accomplished by introducing a Cache that contains the address of the next instruction if a branch is taken, as well as not-taken.
- This cache is known as the Branch-Target Cache or Branch-Target Buffer (BTB).
Branch-Prediction Buffer vs. Branch-Target Buffer
- Recall that a branch-prediction buffer is accessed during the ID stage, after instruction decode.
- We know the branch-target address at the end of the ID stage to fetch the next predicted instruction.
Branch Target Buffer
- The BTB has three fields:
- Lookup: addresses of the known branch instructions (predicted as taken).
- Predicted PC: PC of the fetched instruction predicted taken-branch.
- Prediction State: Optional extra prediction state bits.
Complications:
- Complications arise in using a 2-bit predictor because it uses information for both the branches taken and not-taken.
- This complication is resolved in PowerPC processors by using both the Target-buffer and Prediction-buffer.
Steps involved in using Branch Target Buffer at IF, ID, and EXE pipeline stages:
- IF Stage
- ID Stage
- EXE Stage
Branch-Target Buffer β Flow Chart Explanation
IF Stage:
- The PC of an instruction is compared with the contents of the buffer.
- If it is found, then the instruction must be a branch instruction predicted taken.
- Else, it may be a branch predicted not-taken or a normal instruction.
ID Stage:
- Decode the instruction. If, in the IF Stage, an entry was found in the Target-buffer as predicted-branch, then begin fetching immediately from the predicted PC.
- Check the decoded instruction. If it is a Taken-branch...
EX Stage performs one of four possible functions:
i. Where in the IF stage, entry was not found in the target buffer, and in the ID stage:
- If it is found to be Taken-branch:
- (i-a) Enter branch-instruction address and next PC into branch-target buffer.
- Else:
- (i-b) Proceed as normal instruction execution.
ii. Where in the IF stage, the entry was found in the target-buffer, and in the ID stage:
-
If it is found to be Taken-branch:
- (ii-a) Correctly predicted, so execute normally without stall.
-
Else:
- (ii-b) It is mispredicted, so kill the fetched instruction, restart fetching at another address, and delete the entry from the target-buffer.
-
If the correctly predicted branch entry is found in the buffer, there will be no branch penalty.
-
Else, it suffers at least 2 clock cycle delay as a misprediction penalty:
- One clock delay for fetching the wrong instruction.
- One clock cycle to restart the fetch.
Branch-Target Buffer β Examples
| Inst. in Buffer | Prediction | Actual Branch | Penalty Cycles |
|---|---|---|---|
| Yes | Taken | Taken | 0 |
| Yes | Taken | Not-Taken | 2 |
| No | - | Taken | 2 |
| No | - | Not Taken | 0 |
Branch-Target Buffer β Solution We can compute the penalty by looking at the probability of two events:
- i) Branch predicted taken but ends up not taken: = % buffer hit rate Γ % incorrect prediction = 0.95 Γ 0.1 = 0.095
- ii) Branch is taken but is not found in the buffer: = % incorrect prediction = 0.1
The penalty in both cases is 2 cycles, therefore: Branch Penalty = (0.095 + 0.1) Γ 2 = 0.195 Γ 2 = 0.39
Example: Branch-Target Buffer Problem: Consider a branch-target buffer implemented for conditional branches only for a pipelined processor.
Assuming that:
- Misprediction penalty = 4 cycles
- Buffer miss-penalty = 3 cycles
- Hit rate and accuracy each = 90%
- Branch Frequency = 15%
Solution: The speedup with Branch Target Buffer versus no BTB is expressed as: Speedup = CPI_no_BTB / CPI_BTB = (CPI_base + Stalls_no_BTB) / (CPI_base + Stalls_BTB)
The stalls are determined as: Stalls = Ξ£ (Frequency_s Γ Penalty_s) for all s in stall cases.
The sum over all the stall cases as the product of frequency of the stall cases and the stall-penalty:
i. Stalls_no_BTB = 0.15 Γ 2 = 0.30
ii. To find Stalls_BTB, we consider each output from BTB. There exist three possibilities:
a) Branch misses the BTB:
- Frequency = 15% Γ 0.1 = 1.5% = 0.015
- Penalty = 3
- Stalls = 0.045
b) Branch can hit and correctly predicted:
- Frequency = 15% Γ 0.9 (hit) Γ 0.9 (prediction) = 12.1% = 0.121
- Penalty = 0
- Stalls = 0
c) Branch can hit but incorrectly predicted:
- Frequency = 15% Γ 0.9 (hit) Γ 0.1 (misprediction) = 1.3% = 0.013
- Penalty = 4
- Stalls = 0.052
Stalls_BTB = 0.045 + 0 + 0.052 = 0.097
Speedup = (CPI_base + Stalls_no_BTB) / (CPI_base + Stalls_BTB) = (1.0 + 0.3) / (1.0 + 0.097) = 1.3 / 1.097 = 1.2
Improvement in BTB
- To achieve more instruction delivery, one possible variation in the Branch Target Buffer is:
- To store one or more target instructions, instead of, or in addition to, the predicted Target Address.
Advantages:
- It possibly allows a larger BTB as it permits access to take longer than the time between successive instruction fetches.
- Buffering the actual Target-Instructions allows Branch Folding, i.e., ZERO cycle Unconditional Branching or sometimes ZERO cycle conditional branching.
β Key Takeaways
Correlating branch predictors use history of recent branches to improve accuracy, with (m,n) notation indicating how many past branches are recorded and how many bits per predictor. Tournament predictors dynamically select between multiple predictors using a saturating counter, achieving superior accuracy. The Branch Target Buffer (BTB) caches branch targets to enable zero-cycle branch prediction when hits occur, but mispredictions or misses incur penalties (typically 2-4 cycles). The performance benefit of BTB can be quantified through speedup calculations considering hit rates, accuracy, and branch frequencies, with branch folding offering potential for zero-cycle branching.
π§ Quick Revision Questions
- What does the notation (m, n) mean in correlating branch predictors, and how does a (2,2) predictor differ from a (0,2) predictor?
- How does a tournament predictor decide which predictor to use, and what is the role of the saturating counter in its state transition?
- What are the three fields in a Branch Target Buffer, and what are the penalty cycles for the four possible BTB scenarios?
- Calculate the branch penalty given a 90% hit rate, 90% accuracy, 3-cycle miss penalty, and 4-cycle misprediction penalty with 15% branch frequency.
- What is branch folding, and what advantage does storing target instructions instead of target addresses provide in BTB design?