π Lecture 23 β Instruction Level Parallelism (Hardware Support at Compile Time)
π Overview: This lecture explores hardware support techniques used at compile time to overcome the limitations of static scheduling in exploiting Instruction-Level Parallelism (ILP). It focuses on two key methods: conditional/predicated instructions to eliminate branches and convert control dependence into data dependence, and hardware-based compiler speculation to safely move instructions across branches while preserving program behavior.
ποΈ Topics Covered
This lecture begins with a recap of dynamic and static ILP exploitation techniques and their limitations due to control dependences and ambiguous memory references. It then introduces hardware support for VLIW processors, discussing Conditional/Predicated Instructions as an Instruction Set Extension, including Conditional Move, Conditional ADD, and Conditional Load with examples like absolute value and if-conversion. The lecture covers the advantages and limitations of predicated code, introduces Compiler Speculation, and details hardware support methods for speculative execution, including methods to preserve exceptions using poison bits and hardware buffering.
π Lecture Summary
Recap: H/W and S/W Exploitation
Both dynamic and static scheduling techniques aim to exploit ILP for single or multiple instruction issue per clock cycle. Dynamic approaches use hardware modifications resulting in superscalar and VLIW processors. Pipeline enhancements include Tomasuloβs pipeline to overcome structural and data hazards, and branch predictors to minimize control hazard stalls. Static scheduling approaches include Loop Unrolling, Software Pipelining, Trace Scheduling, and Superblock Scheduling. These techniques give better performance when branch behavior is correctly predictable at compile time. Otherwise, parallelism cannot be completely exposed due to two reasons: (1) control dependences limit the amount of parallelism that can be exploited, and (2) dependence between memory reference instructions could prevent code movement necessary to increase parallelism.
Hardware Support for VLIW
These limitations, particularly for VLIW processors, can be overcome by providing hardware support at the compile time. The most commonly used techniques are:
- Extension of the Instruction Set by including Conditional or Predicated Instructions
- Hardware speculation to enhance the compiler's ability to move code over branches while preserving exceptional behavior
- Allowing the compiler to reorder load/store instructions when no conflict is suspected but not certain
1: Instruction Set Extension
The extended instruction set including Conditional or Predicated Instructions allows the compiler to group instructions across branches, eliminate branches, and convert control dependence into data dependence. These approaches are equally useful for hardware-intensive (dynamic) and software-intensive (static) scheduling schemes. Predicate registers are included in the IA64 processor to implement predicated instructions.
π Definition β Predicate Register: A one-bit register that controls whether a predicated instruction executes normally (if the condition is true) or acts as a no-operation (if the condition is false).
Conditional Instructions
Conditional instructions have an extra operandβa one-bit predicate register. A condition is evaluated as part of instruction execution to set the value of the predicate register. In HPL-PD from HP Lab, the value of the predicate register is typically set by a Compare-to-predicate operation, e.g., p1 = CMPP <= r1, r2 (here predicate register p1 is set if r2 <= r1). If the condition is true (p1=1), the instruction is executed normally. If false (p1=0), the instruction execution continues as if it were a no-operation. Typical conditional instructions include:
- Conditional Move β
CMOVZ R1, R2, R3: moves the value from one register to another if the condition is true (the third operandβpredicate register R3βis Zero) - Conditional ADD β
(R8) ADD R1, R2, R3: assumes R1 = R2 + R3 occurs if the predicate register R8 is 1 - Conditional Load β
LWC R1, 0(R2), R3: assumes the load occurs unless the third operand R3 is Zero
Example 1: Conditional or Predicated Instructions
Consider the conditional statement: If (A==0) { S=T; }. Assuming registers R1, R2, R3 hold A, S, and T respectively:
Traditional code:
BNEZ R1, L ; No-op if A (R1)!= 0
ADDU R2, R3, R0 ; Else replace S (R2) by T (R3)L
Using Conditional Move: CMOVZ R2, R3, R1 (Move R3 to R2 if the third operand R1=0). Using CMOVZ converts the control dependence into a data dependence, resolved where the register-write occurs rather than near the front of the pipe (as with branches). This transformation is used in vector computers as if-conversion, which replaces conditional branches with predicated operations.
Example 2: Absolute Value Function
Conditional move implements the absolute value function: A = abs(B), implemented as:
if B<0 {A=-B;} else {A=B;}
This can be implemented as a pair of conditional moves:
CMOVZ R2, -R3, R1
CMOVZ R2, R3, R4
Or one unconditional move A=B and one conditional move A=-B.
π Formula: Conditional Move operation: CMOVZ dest, src, pred β destination register gets source value only if the predicate register is zero (condition true).
π Example: For A = abs(B): If R1 holds a comparison flag (B<0), CMOVZ R2, -R3, R1 moves -B to A if B<0; CMOVZ R2, R3, R4 moves B to A if Bβ₯0.
Conditional MOVE Instruction
Conditional moves eliminate branches and improve pipeline behavior but are useful only for short sequences. When predication eliminates branches guarding large blocks of code, many conditional moves are needed. To remedy this, some architectures support full predication, where execution of all instructions is controlled by a predicate. Full prediction allows conversion of large blocks of branch-dependent code. For example, an if-then-else statement within a loop is converted to predicated execution, where code in the if case executes only if the condition is true, and code in the else case executes only if the condition is false.
Predicated LOAD Instructions
Consider a two-issue superscalar that can issue a combination of one memory reference and one ALU operation or a branch every cycle. Original code sequence:
| First instruction slot | Second instruction slot |
|---|---|
| LW R1, 40(R2) | ADD R3,R4,R5 |
| ADD R6,R3,R7 | |
| BEQZ R10,L | |
| LW R8, 0(R10) | |
| LW R9, 0(R8) |
Here, the second LW (LW R9, 0(R8)) depends on the prior load (LW R8, 0(R10)), causing a data dependence stall if the branch is not taken. Using the predicated version LWC: LWC R8, 0(R10), R10 (load occurs unless R10 is 0). This instruction is moved up to the second issue slot:
| First instruction slot | Second instruction slot |
|---|---|
| LW R1, 40(R2) | ADD R3,R4,R5 |
| LWC R8,0(R10),R10 | ADD R6,R3,R7 |
| BEQZ R10,L | |
| LW R9, 0(R8) |
π‘ Why this matters: This improves execution time by eliminating one instruction issue slot and reducing the pipeline stall for the last instruction. However, if the compiler mispredicts the branch, the mispredicated instruction has no effect and does not improve running timeβmaking the transformation speculative.
Advantages of Predicated Code
Conditional or predicated instructions are extremely useful for: implementing short alternative control flows, eliminating some unpredictable branches, and reducing the overhead of global code scheduling.
Limitations on Conditional Instructions
Moving an instruction across a branch and making it conditional will slow the program whenever the moved instruction would not have been normally executed. Predicating a control dependent and eliminating a branch may slow down the processor if that code would not have been executed. Key limitations include:
- Predicated instructions are useful only when the predicate can be evaluated early
- Conditional instructions may cause a stall for data hazard if the condition evaluation and predicated instructions are not separated
- Conditional instructions may have speed penalty compared to unconditional instructions
- Use is limited when control flow involves more than a simple alternative sequenceβmoving an instruction across multiple branches requires making it conditional on both, requiring additional instructions to compute the controlling predicate
Architectures with Conditional Instructions
Due to these limitations, most architectures include only a few conditional instructions, mostly CMOV. The MIPS, Alpha, PowerPC, SPARC, and Intel (Pentium) all support Conditional Move. The IA-64 Micro Architecture supports full predication of all instructions.
Introduction to Compiler Speculation
Where programs have branches predictable at compile time, the compiler speculates to either improve scheduling and/or increase the issue rate. Predicated instructions may help speculate, but they are more useful when they can eliminate control dependence by if-conversion. However, in most cases, speculated instructions must be moved before the condition evaluationβsomething that cannot be done by predication alone. This motivates three capabilities for ambitious speculation:
- The ability to find instructions that can be speculatively moved without affecting program data flow
- The ability to ignore exceptions in speculated instructions until we know such exception would really occur
- The ability to speculatively interchange loads and stores, or stores and stores, which may have address conflicts
Note: The first is a compiler capability; the other two can be achieved by hardware support.
π Formula: Hardware speculation approach: Supports reordering loads and stores by checking for potential address conflicts at runtime, allowing the compiler to reorder loads and stores when it suspects they do not conflict.
Methods to Preserve Exceptions
Four hardware methods support more ambitious speculation without introducing erroneous exception behavior. The key is that results of a mispredicted speculated sequence will not be used in the final computation, so exceptions are preserved.
Method 1: The hardware and operating system cooperatively ignore exceptions for speculative instructions in the incorrect program path. Exception behavior for the correct program is preserved; for the incorrect one it is ignored. This is used as a "fast mode" under program control. Examples: Memory protection violation (indicates program error, normally causes termination) and Page fault (handles program error, normally resumed).
Method 2: Speculative instructions that never raise an exception are used, and checks are introduced to determine when an exception should occur.
Method 3: A set of bits called poison bits are attached to the result register. These bits are written by speculated instructions when the instruction causes exceptions. The poison bits cause a fault when a normal instruction attempts to use the register. This approach tracks exceptions as they occur but postpones any terminating exception until a value is actually used. The scheme adds a poison bit to every register and another bit to every instruction to indicate if it is speculative. The poison bit of the destination register is set whenever a speculative instruction results in a terminating exception. All other exceptions are handled immediately. If a speculative instruction uses a register with poison bit on, the destination register has its poison bit on. If a normal instruction attempts a register source with poison bit on, the instruction causes a fault.
π Definition β Poison Bits: Bits attached to result registers that are set when a speculative instruction causes an exception; they cause a fault only when a normal instruction attempts to use the register, postponing terminating exceptions until the value is actually used.
Method 4: A mechanism is provided to indicate that an instruction is speculative, and the hardware buffers the instruction result until it is certain that the instruction is no longer speculative.
Summary
Both hardware and software mechanisms provide approaches to exploit ILP. There are certain limitations on both mechanisms.
β Key Takeaways
The critical concepts from this lecture are that conditional/predicated instructions convert control dependence into data dependence by using an extra predicate register to determine execution, allowing branches to be eliminated and ILP to be increased at compile time. The Conditional Move (CMOV) is the most common implementation, but full predication (as in IA-64) is needed for large blocks of code. Hardware-based compiler speculation allows the compiler to move instructions across branches more ambitiously by providing runtime capabilities to check for address conflicts and preserve exception behavior. The key hardware mechanisms for preserving exceptions include ignoring exceptions for mispredicted paths, using speculative instructions that never raise exceptions, poison bits that track exceptions until a register is actually used, and hardware buffering of speculative results until they are confirmed.
π§ Quick Revision Questions
- How do conditional/predicated instructions convert control dependence into data dependence?
- What are the three typical conditional instructions given in the lecture, and what does each do?
- What are the four hardware methods to preserve exceptions in speculative execution?
- Explain how poison bits work and when they cause a fault.
- Why can't predication alone solve all speculative code movement problems, and what additional capabilities does hardware provide?
π Lecture 24 β Instruction Level Parallelism (Concluding Instruction Level Parallelism)
π Overview: This lecture concludes the discussion on Instruction Level Parallelism (ILP) by exploring compile-time hardware support methods to preserve exceptions during speculation and to handle memory reference speculation. It details four key methods for preserving exception behavior and compares hardware versus software-based speculation approaches, highlighting their respective advantages and limitations for modern computer architecture.
ποΈ Topics Covered
This lecture begins with a recap of previously discussed hardware support for exposing parallelism, including predicated instructions and methods for preserving exception behavior. It then provides an in-depth analysis of four methods to preserve exceptions: Fast Mode, Speculative Instructions, Poison-bit Register, and Hardware (Re-order) Buffering. The lecture proceeds to discuss hardware support for memory reference speculation, including memory disambiguation using speculative load and verify instructions. Finally, it concludes with a comprehensive summary comparing hardware versus software speculation approaches.
π Lecture Summary
Recap: Compile Time H/W Support
Last time, we discussed methods to provide hardware support for exposing more parallelism at compile time. This included extending the instruction set with conditional or predicated instructions to eliminate branches and convert control dependence into data dependence, improving processing performance. We also introduced hardware and software-based abilities required to move speculated instructions before branch condition evaluation while preserving exception behavior. We distinguished between exceptions that indicate program error and cause termination (e.g., memory protection violation) and those that can be handled and resumed normally (e.g., page fault in virtual memory). For terminating exceptions in speculated instructions, we cannot take the exception, while for resumable exceptions, they are preserved and processed when the instruction is no longer speculative.
Methods to preserve exception behavior
The four methods to preserve exceptions are: Fast Mode, Speculative-instructions method, Poison-bit Register method, and Hardware (Re-order) buffering.
Methods 1: Fast Mode
The Fast Mode approach is the simplest method where hardware works cooperatively with the operating system to handle presumable exceptions under program control. Here, hardware and software preserve exception behavior for the correct program but ignore it for incorrect programs.
π Definition β Correct Program: A program where the instruction generating a terminating exception is speculative, and the speculative result is simply unused and not harmful.
π Definition β Incorrect Program: A program that previously received a terminating exception and will get an incorrect result in the present instruction, meaning the exception behavior is mispredicted.
π Example: Consider the IF-THEN-ELSE code fragment: If (A==0) A = B; Else A = A + 4;
Assuming A is at 0(R3) and B is at 0(R2), and speculating the THEN clause is almost always executed:
LD R1,0(R3) ;load A
BNEZ R1,L1 ;test A
LD R1,0(R2) ;then clause (speculative)
J L2 ;skip else
L1: DADDUI R1,R1,#4 ;else clause
L2: SD R1,0(R3) ;store A
To preserve speculation, use a temporary register (R14) to avoid destroying R1 when B is loaded:
LD R1,0(R3) ;load A
LD R14,0(R2) ;speculative load B
BEQZ R1,L3 ;other branch of IF
DADDUI R14,R1,#4 ;else clause
L3: SD R14,0(R3) ;non-speculative store A
Method 2: Speculative Instructions
This approach introduces speculative versions of instructions, such as Speculative Load (sLD) and Speculative Check (SPECCK). These instructions don't generate terminating exceptions but rather check for such exceptions, preserving exception behavior exactly.
π Example: Reconsidering the IF-THEN-ELSE statement using speculative instructions:
LD R1,0(R3) ;load A
sLD R14,0(R2) ;speculative, no termination
BNEZ R1,L1 ;test A
SPECCK 0(R2) ;speculative check
J L2 ;skip else
L1: DADDUI R1,R1,#4 ;else clause
L2: SD R1,0(R3) ;store A
The load instruction speculates whether the branch will be taken or not-taken. The speculation check maintains a basic block for the THEN clause, preserving exception behavior. However, extra code is required to check for possible exceptions, resulting in overhead.
π‘ Why this matters: This method preserves exception behavior exactly rather than speculatively, making it more precise but introducing code overhead.
Method 3: Poison Bit
The Poison Bit approach attaches a set of bits to every instruction and result register to indicate whether the instruction is speculative. These bits track exceptions as they occur but postpone any terminating exception until a value is actually used. When a bit is set in the result register, it indicates an exceptional condition. When a non-speculative operation encounters a register with the speculative bit set, an exception is raised.
Sequence of steps:
- The poison bit of the destination register is set whenever a speculative instruction results in a terminating exception
- All other exceptions are handled immediately
- If a speculative instruction uses a register with poison-bit turned ON, the destination register simply has its poison-bit turned ON
- If a normal instruction attempts to use a register source with poison-bit ON, the instruction causes a fault
Resulting Behavior: Any program that would have generated an exception still generates one, but at the first instant where a result is used by a non-speculative instruction. STORES are never speculative as poison-bits exist on registers only.
π Example: Using SLD and register R14 with poison-bit for the IF-THEN-ELSE statement:
LD R1,0(R3) ;load A
sLD R14,0(R2) ;speculative load B
BEQZ R1,L3
DADDUI R14,R1,#4
L3: SD R14,0(R3) ;store A
If the speculative sLD generates a terminating exception, the poison bit of R14 is turned ON. When the non-speculative SW instruction occurs, it raises an exception if the poison bit for R14 is ON.
Method 4: Hardware Buffering
A hardware buffer, such as the reorder buffer, is provided. The compiler marks instructions as speculative and includes an indicator of how many branches the instruction was moved speculatively. This can be done using either a single bit for branch taken/not-taken path, or a sentinel (guard) marking the original location. Instructions marked as speculative are placed in the re-order buffer, which tracks when instructions are ready to commit and delays write-back. Speculative instructions are not allowed to commit until:
- Either the branches have been speculatively moved and are ready to commit
- Or the corresponding sentinel is reached
Memory Reference Speculation
Hardware support for memory disambiguation (Address Certainty) is needed when optimizing codes by moving LOADs across STOREs. If the latency of LOAD is high, it may be desirable to move LOAD before STORE. However, this optimization is not valid if LOAD and STORE reference the same location and the compiler is uncertain at compile time.
To resolve this, two special instructions are included in the ISA:
- LDS R1, 0(R2) β Load Speculatively: Initiates a load like a normal load instruction, but a log entry is made in a table to store the memory location
- LDV R2, 0(R2) β Load Verify: Checks if a store to the memory location has occurred since the LDS. If so, the new load is issued and the pipeline stalls; otherwise, it's a NO-OP
Process: When LDS is executed, the hardware saves the address in a log table. If a subsequent STORE changes the location before LDV, speculation has failed. If the STORE doesn't touch the location, speculation is successful.
Handling speculation failure can be done in two ways:
- If only the load was speculated: redo the load using LDV, which supplies the target register in addition to memory address
- If the additional instruction (LDV) is also speculated: re-execute all speculated instructions starting with the LOAD using a fix-up sequence, where LDV specifies the address of the fix-up code
Example: Conditional Moving up the Branch
Using conditional move instructions can eliminate branches and guard against memory access violations. Consider a 2-issue superscalar code where LW after Branch depends on prior load:
LW R1,40(R2) ADD R3,R4,R5
ADD R6,R3,R7
BEQZ R10,L
LW R8,0(R10)
LW R9,0(R8)
Revised code using conditional move: Assuming R29 contains a safe address, R30 saves original R8:
DADDI R29,R0,#1000 ;initialize R29 to safe address
LW R1,40(R2) ;first load
MOV R30,R8 ;save R8 in unused R30
CMOVNZ R29,R10,R10 ;R29=R10 if R10 is safe and non-zero
LW R8,0(R29) ;speculative load
CMOVZ R8,R30,R10 ;restore R8 if R10=0
BEQZ R10,L
LW R9,0(R8)
Branch-free code using two additional registers (R30, R31):
ADDI R29,R0,#1000
LW R1,40(R2)
MOV R30,R8
MOV R31,R9 ;save R9 in unused R31
CMOVNZ R29,R10,R10
LW R8,0(R29)
LW R9,0(R8) ;load speculated
CMOVZ R8,R30,R10
CMOVZ R9,R31,R10 ;restore R9 if needed
Summary: Hardware versus software speculation
The limitations of both approaches are summarized as follows:
-
Memory reference certainty: To speculate extensively, we must ascertain memory references, which is difficult at compile time. In hardware-based schemes, dynamic run-time certainty of memory address uses Tomasulo's pipelined structure to move loads past stores at run time.
-
Control flow unpredictability: Hardware-based speculation works better when control flow is unpredictable and when hardware-based branch prediction is superior to compile-time software-based branch prediction.
-
Precise exception model: Hardware-based speculation maintains a completely precise exception model for speculated instructions.
-
No compensation needed: Hardware-based speculation does not require compensation needed by ambitious software speculation mechanisms.
-
Compiler advantage: Compiler-based approaches have the ability to see further in the code sequence, potentially providing better code scheduling than purely hardware-driven approaches (e.g., conditional move instructions).
-
Implementation independence: Hardware-based speculation with dynamic scheduling does not require different code sequences for different architecture implementations. However, the major disadvantage is the extra hardware resources and complexity of the structure.
β Key Takeaways
A student must remember the four distinct methods for preserving exception behavior during speculation: Fast Mode (cooperative OS-HW ignoring incorrect program exceptions), Speculative Instructions (using sLD and SPECCK for exact preservation), Poison Bits (attaching bits to results to postpone exceptions until use), and Hardware Buffering (using reorder buffers with sentinels). The critical distinction between correct programs (where speculative result is unused) and incorrect programs (where exception is mispredicted) is essential. For memory reference speculation, the LDS/LDV instruction pair provides runtime memory disambiguation, allowing loads to be moved speculatively before stores with verification. Hardware speculation excels with unpredictable control flow and provides precise exception models but requires significant hardware complexity, while compiler speculation can see further ahead in code and uses techniques like conditional move instructions for branch elimination.
π§ Quick Revision Questions
- What are the four methods discussed for preserving exception behavior during speculation, and how does the Fast Mode approach differ from the others?
- Explain the role of poison bits in preserving exception behavior. When is an exception actually raised when using poison bits?
- How do the LDS and LDV instructions work together to support memory reference speculation, and what happens when speculation fails?
- Compare hardware-based speculation with compiler-based speculation regarding memory reference certainty, control flow handling, and hardware complexity.
- In the conditional moving example, why is an unused register (R29) initialized to a safe address, and what happens to the original register value (R8) during speculative execution?
π Lecture 25 β Memory Hierarchy Design (Storage Technologies Trends and Caching)
π Overview: This lecture explores what lies outside the processor, focusing on the memory hierarchy and storage technologies. It explains how the principle of locality allows systems to achieve fast access speeds at low cost by organizing memory into levels, from fast but expensive SRAM to slow but cheap disk storage, and details the design and operation of key memory types.
ποΈ Topics Covered
The lecture begins by recapping processor performance and defining the I/O system outside the processor. It introduces the memory hierarchy and the principle of locality. It then categorizes storage systems, with a deep dive into Random-Access Memory (RAM) types: Static RAM (SRAM) and Dynamic RAM (DRAM), including their cell structures and organization. Enhanced DRAM types are covered, followed by nonvolatile memories, disk geometry and capacity, disk access time with a concrete example, and logical disk blocks. It concludes with the CPU-memory gap and a summary.
π Lecture Summary
Recap: Processor Performance
The lecture recaps that processor performance involves the design of the data path and control, utilizing hardware-based and software-based techniques to expose Instruction-Level Parallelism (ILP).
What is outside processor?
The performance of a computer is heavily influenced by what is outside the processor, referred to as the I/O system. This system includes the memory system, buses, and controllers. The design goal of the memory system for high-performance computers is to present the user with as much memory as is available. Since fast memory is expensive and cheap memory is slow, a memory hierarchy is organized into several levels. This hierarchy aims to achieve a cost as low as the cheapest memory and a speed as fast as the fastest memory.
Principle of Memory hierarchy
The principle of memory hierarchy is based on "The principle of locality", which determines where each type of module is located in the memory system. The advantage of this principle is that it provides an average access speed very close to that of the fastest technology, at a cost very close to that of the cheapest technology.
Storage Type in Memory System
Semiconductor memories such as registers, Static RAM, and Dynamic RAM are fast in access speed but expensive. They are used in small sizes and are placed either inside or closest to the processor. Secondary storage devices, which offer huge storage at the lowest cost per bit, are placed farthest away from the processor.
Storage Systems
Memory systems can be classified based on different attributes:
- Material: Semiconductor, magnetic, optical
- Accessing: Random, Sequential, Hybrid
- Store/Retrieve: ROM (Read Only Memory), RMM (Read Mostly Memory), and RWM (Read Write Memory)
Random-Access Memory (RAM)
RAM is packaged as a chip, with the basic storage unit being a cell (one bit per cell). Multiple RAM chips form a memory. The two main types are Static RAM (SRAM) and Dynamic RAM (DRAM).
Static RAM: Basic Cell
SRAM uses a flip-flop circuit (typically 6 transistors) to store a bit. The state is stable as long as power is applied.
- Write: Drive the bit lines (bit=1, bit=0) and then select the row.
- Read: Pre-charge bit and bit to Vdd, then select the row. The cell pulls one line low, and a sense amplifier on the column detects the difference between bit and bit.
π Definition β SRAM (Static RAM): A type of semiconductor memory that uses a bistable latching circuit (flip-flop) to store each bit. It is faster and more expensive than DRAM but does not require refreshing.
Dynamic Random Access Memory (DRAM)
Each DRAM cell stores a bit using a capacitor and a transistor. The value must be refreshed every 10-100 ms. DRAM is sensitive to disturbances and is slower and cheaper than SRAM.
π Definition β DRAM (Dynamic RAM): A type of semiconductor memory that stores each bit as a charge on a tiny capacitor. It requires periodic refreshing to maintain the stored data, making it slower but denser and cheaper than SRAM.
Basic DRAM Cell
- Write: Drive the bit line and select the row by turning on the pass transistor.
- Read: Pre-charge the bit line to Vdd and select the row. The cell and bit line share charges, causing a very small voltage change on the bit line. A sensitive sense amplifier detects this change. The read operation is destructive, so an automatic write-back (restore) must be performed at the end of every read.
- Refresh is achieved by performing a dummy read to every cell.
π Formula: The operation of a DRAM cell is based on charge sharing between the cell's capacitor and the bit line capacitance.
Reading DRAM Supercell (2,1)
A DRAM chip is organized as a matrix of supercells.
- Step 1(a): The Row Access Strobe (RAS) selects row 2.
- Step 1(b): The entire row 2 is copied from the DRAM array to the row buffer.
- Step 2(a): The Column Access Strobe (CAS) selects column 1.
- Step 2(b): Supercell (2,1) is copied from the buffer to the data lines, and eventually back to the CPU.
Enhanced DRAMs
Several enhancements have been made to standard DRAM to improve performance:
- Fast Page Mode DRAM (FPM DRAM): In normal DRAM, each M-bit access requires a RAS/CAS cycle. In FPM DRAM, a row is saved into an N x M register. Subsequent accesses to the same row (page) can be performed by only providing a column address, which is faster.
- Extended Data Out DRAM (EDO DRAM): An enhanced FPM DRAM with more closely spaced CAS signals, allowing for faster burst transfers.
- Synchronous DRAM (SDRAM): Driven with the rising clock edge instead of asynchronous control signals, simplifying the interface and increasing speed.
- Double Data Rate synchronous DRAM (DDR SDRAM): An enhancement of SDRAM that uses both the rising and falling edges of the clock as control signals, effectively doubling the data transfer rate.
- Video RAM (VRAM): Like FPM DRAM, but output is produced by shifting the row buffer. It is dual-ported to allow concurrent reads and writes, useful for video graphics.
Nonvolatile Memories
DRAM and SRAM are volatile memories; they lose information when powered off. Nonvolatile memories retain their value even when powered off. The generic name for these is Read-Only Memory (ROM), which is misleading as some types can be read and modified.
- Types of ROMs: Programmable ROM (PROM), Erasable Programmable ROM (EPROM), Electrically Erasable PROM (EEPROM), and Flash memory.
- Firmware is a program stored in a ROM, such as boot-time code (BIOS) or code for graphics cards and disk controllers.
Disk Geometry
A disk consists of one or more platters, each with a magnetic surface. The surface is divided into concentric circles called tracks, and each track is further divided into sectors. A cylinder is the set of aligned tracks across multiple platters.
Disk Capacity
Capacity is the maximum number of bits that can be stored on a disk. It is determined by:
- Recording density (bits/in): number of bits in a 1-inch segment of a track.
- Track density (tracks/in): number of tracks in a 1-inch radial segment.
- Areal density (bits/inΒ²): the product of recording and track density.
Disk Access Time
The average time to access a target sector is approximated by: Taccess = Tavg seek + Tavg rotation + Tavg transfer
- Seek time (Tavg seek): Time to position the read/write head over the correct cylinder. Typical Tavg seek = 9 ms.
- Rotational latency (Tavg rotation): Time waiting for the first bit of the target sector to rotate under the read/write head. Tavg rotation = 1/2 Γ 1/RPMs Γ 60 sec/1 min.
- Transfer time (Tavg transfer): Time to read the bits in the target sector. Tavg transfer = 1/RPM Γ 1/(avg # sectors/track) Γ 60 secs/1 min.
π Formula: Taccess = Tavg seek + Tavg rotation + Tavg transfer Plain-English Meaning: The total time to read a sector is the sum of the time to move the head to the right track, the time to wait for the disk to spin the sector under the head, and the time to read the data.
π Example: Disk Access Time Calculation
- Given: Rotational rate = 7,200 RPM, Avg seek time = 9 ms, Avg # sectors/track = 400.
- Derived:
- Tavg rotation = 1/2 Γ (60 secs / 7200 RPM) Γ 1000 ms/sec = 4 ms.
- Tavg transfer = 60/7200 RPM Γ 1/400 secs/track Γ 1000 ms/sec = 0.02 ms.
- Taccess = 9 ms + 4 ms + 0.02 ms = 13.02 ms.
- Important Points: Access time is dominated by seek time and rotational latency. The first bit in a sector is the most expensive; the rest are free. SRAM access time is about 4 ns/doubleword, and DRAM about 60 ns. A disk is about 40,000 times slower than SRAM and 2,500 times slower than DRAM.
Logical Disk Blocks
The set of available sectors is modeled as a sequence of b-sized logical blocks. The mapping between logical blocks and actual (physical) sectors is maintained by the disk controller, a hardware/firmware device. It converts requests for logical blocks into (surface, track, sector) triples.
CPU-Memory Gap
The gap between the speed of the processor and the speed of storage devices (DRAM, SRAM, and Disk) is increasing with time, making memory hierarchy design a critical challenge for computer architects.
π‘ Why this matters: The CPU-memory gap is the fundamental reason why memory hierarchy and caching are necessary for modern computer performance. Without them, the processor would spend most of its time waiting for data.
β Key Takeaways
The memory hierarchy uses the principle of locality to create a system that appears as fast as the fastest level and as cheap as the cheapest level. SRAM is fast, expensive, and does not require refreshing, while DRAM is slower, cheaper, and must be refreshed, making it suitable for main memory. Disk storage is orders of magnitude slower than DRAM and SRAM, with access time dominated by mechanical seek time and rotational latency. Enhanced DRAMs like FPM, EDO, SDRAM, and DDR improve upon standard DRAM to reduce access latency. The growing CPU-memory gap is the primary motivation for sophisticated memory hierarchies and caching strategies.
π§ Quick Revision Questions
- What is the principle of locality and how does it benefit memory hierarchy design?
- Explain the difference in the read and write operations of an SRAM cell versus a DRAM cell.
- What are the components of disk access time, and which of them contribute the most to the total time?
- How does Fast Page Mode (FPM) DRAM improve upon the performance of regular DRAM?
- What is the "CPU-memory gap" and why is it a significant problem in computer architecture?
π Lecture 26 β Memory Hierarchy Design (Concept of Caching and Principle of Locality)
π Overview: This lecture introduces the foundational concepts of memory hierarchy design, focusing on caching mechanisms and the principle of locality. It explains how different memory technologies (SRAM, DRAM, disk) are organized hierarchically to bridge the growing speed gap between the CPU and main memory, and why caching is essential for modern computer performance.
ποΈ Topics Covered
The lecture begins with a recap of storage trends and device characteristics, then introduces the concept of cache memory as a staging area for frequently used data. It explains the principle of locality (temporal and spatial) with detailed examples, covers cache addressing techniques including direct mapping, and concludes with an analysis of cache transactions (hits, misses, and penalty calculations) along with placement and replacement policies.
π Lecture Summary
Recap: Storage Devices
The lecture reviews semiconductor memories including SRAM, DRAM, and magnetic disk storage. DRAM is slow but cheap relative to SRAM and serves as main memory for holding moderately large amounts of data and instructions. Disk storage is the slowest and cheapest, used for secondary storage to hold bulk data and instructions. The CPU-Memory Gap β the speed difference between DRAM/Disk and the processor compared to SRAM β is increasing very rapidly over time.
Memory Hierarchy Principles
The speed of DRAM and CPU complement each other. Memory is organized in a hierarchy based on two concepts: Concept of Caching and Principle of Locality.
1: Concept of Caching
A cache is a staging area or temporary place to store a frequently-used subset of data or instructions from the relatively cheaper, larger, and slower memory. This avoids having to go to main memory every time the information is needed.
Memory devices of different types are used for each level k. The faster, smaller device at level k serves as a cache for the larger, slower device at level k+1. Programs tend to access data at level k more often than at level k+1. Storage at level k+1 can be slower, but larger and cheaper per bit. The result is a large pool of memory that costs as much as the cheap storage at the highest level (near bottom in hierarchy) but serves data at the rate of fast storage at the lowest level (near top in hierarchy).
2: Principle of Locality
Programs access a relatively small portion of the address space at any instant of time. This principle has two types:
π Definition β Temporal locality: If an item is referenced, it will tend to be referenced again soon (locality in time).
π Definition β Spatial locality: If an item is referenced, items whose addresses are close by tend to be referenced soon (locality in space).
A well-written program tends to reuse data and instructions that are either near those used recently or that were recently referenced themselves.
Locality Example β Program:
sum = 0;
for (i = 0; i < n; i++)
sum += a[i];
return sum;
- Spatial Locality: All array elements
a[i](data) are referenced in succession at each loop iteration, so all array elements should be located at the same level. All instructions of the loop are referenced repeatedly in sequence, therefore should be located at the same level. - Temporal Locality: The variable
sumis referenced each iteration (recently referenced data is used again). The instructions of the loopsum += a[i]cycle through the loop repeatedly.
π‘ Why this matters: The memory hierarchy keeps more recently accessed data items closer to the processor because the processor will likely access them again soon. Not only do we move the item just accessed closer, but we also move adjacent data items.
Hierarchy List
- Register File β Level 0 β Datapath
- L1 β Level 1 β Cache on Chip
- L2 β Level 2 β External Cache
- Main memory β Level 3 β System Board DRAM
- Disk cache β Level 4 β Disk drive
- Disk β Level 5 β Magnetic disk
- Optical β Level 6 β CDs etc. β bulk storage
- Tape β Level 7 β Huge cheapest storage
Intel Processor Cache Evolution
- 80386 β No on-chip cache
- 80486 β 8K byte lines
- Pentium (all versions) β Two on-chip L1 caches for Data & Instructions
- Pentium 4 β L1 caches: Two 8K bytes; L2 cache: 256K feeding both L1 caches
Cache Devices
Cache device is a small SRAM made directly accessible to the processor. DRAM, accessible by the cache as well as by the user or programmer, is placed at the next higher level as Main Memory. Larger storage such as disk is placed away from the main memory.
Memory Hierarchy Terminology
π Definition β Hit: Data the processor wants to access appears in some block in the upper level.
- Hit Rate: The fraction of memory accesses that are found in the upper level (HIT).
- Hit Time: Time to access the upper level, consisting of: (i) RAM access time, (ii) Time to determine if this is a hit or miss.
π Definition β Miss: Data needed by the processor is not found in the upper level and must be retrieved from a block in the lower level.
- Miss Rate = 1 - (Hit Rate)
- Miss Penalty: The sum of time to replace a block in the upper level and to deliver the block to the processor.
π Formula: Miss Rate = 1 - Hit Rate
π Recommendation: Hit Time must be much much smaller than Miss Penalty, otherwise no need for memory hierarchy.
Cache Hit
When the CPU needs object d stored in block b (e.g., block 14) of the level k+1 memory, and the corresponding block (e.g., block 2) exists in the cache at level k, the program finds block b in the cache and object d is transferred directly to the CPU.
Cache Miss
When the program needs object A stored in block C (e.g., block 12) at level k+1, but block C is not at level k, a cache miss occurs. Level k cache must fetch it from level k+1 and then transfer object A to the CPU.
Placement and Replacement Policies
If level k cache is full, some current block must be replaced (evicted). The "victim" depends on:
- Cache design defining the relationship of cache addresses with higher level memory addresses
- Placement policy: Determines where the new block can go
- Replacement policy: Defines which block should be evicted
Types of Misses
π Definition β Cold (compulsory) miss: Occurs when the cache is empty, at the beginning of cache access.
π Definition β Capacity miss: Occurs when the set of active cache blocks (working set) is larger than the cache.
π Definition β Conflict miss: Occurs when the level k cache is large enough, but multiple data objects all map to the same level k block.
π Example: Conflict Miss
If the placement policy is based on direct addressing, then Block n at level k+1 must be placed in block (n mod 4) at level k. In this case, referencing blocks 0, 8, 0, 8, 0, 8... would miss every time because 8 mod 4 = 0, as both blocks 0 and 8 of level k+1 are placed at location 00 at level k.
Cache Design β Tags
More than one block from level k+1 memory (main memory, with N blocks) may be placed at the same location (given by N MOD M) in level-k memory (cache, with M blocks). Therefore, a tag must be associated with each block in the level-k (cache) memory to identify its position in the level k+1 memory (Main memory).
Direct Mapping Example
A 16 MB main memory has a 24-bit address bus. It is organized in 32-bit blocks. A 16K word (64 KB) cache requires a 16-bit address and 8-bit tag.
π Address Structure (24-bit):
- 2 bits: Word identifier (4-byte block)
- 22 bits: Block identifier for main memory
- 8 bits: Tag (= 22 - 14)
- 14 bits: Slot or line or index value for cache
π Key Rule: No two blocks in the same line have the same Tag field. Check contents of cache by finding line and checking Tag.
Cache Design β Another Example
Assume a 1 KB direct mapped cache with block size = 32 bytes. Each block associated with the cache tag will have 32 bytes.
Address Translation β Direct Mapped Cache
Assume level k+1 main memory of 4 GB, with Block Size = 32 bytes, and level k cache of 1 Kbyte.
π Address Breakdown:
- 5 least significant bits (bits 0-4): Used as byte select within the cache block (since 2β΅ = 32 bytes)
- 32 - 10 = 22 bits: The upper bits stored as cache tag (since 1 KB = 2ΒΉβ° bytes)
- Bits 5 through 9 (the middle bits): Used as Cache Index to select the proper cache block entry
β Key Takeaways
The memory hierarchy works because of the principle of locality: programs tend to reuse recently accessed data (temporal locality) and nearby data (spatial locality). Cache acts as a small, fast staging area between the processor and slower main memory, storing frequently used subsets to minimize access time. Understanding hit rate, miss rate, and miss penalty is critical because the penalty for a miss must be much larger than hit time for caching to be beneficial. Direct mapping uses a simple address structure where block identifier bits are split into tag and index, with the index determining the cache line and the tag verifying whether the correct block is present. The three types of misses (cold, capacity, and conflict) arise from different limitations in cache design and program behavior.
π§ Quick Revision Questions
- What are the two types of locality, and how do they affect memory hierarchy design?
- Why must Hit Time be much smaller than Miss Penalty for memory hierarchy to be effective?
- How does direct mapping determine which cache line a main memory block will occupy?
- What is the difference between a compulsory miss, a capacity miss, and a conflict miss?
- In the direct mapping example with a 1 KB cache and 32-byte blocks, which bits of the address are used for the byte offset, cache index, and tag?