π Lecture 27 β Memory Hierarchy Design (Cache Design Techniques)
π Overview: This lecture explores the critical aspects of cache memory design within the memory hierarchy. It covers cache performance metrics like miss rate and miss penalty, and delves into three fundamental cache organization techniques: direct mapped, fully associative, and set associative caches. Understanding these designs is crucial for optimizing data access speed in modern computer architecture.
ποΈ Topics Covered
The lecture begins with a recap of caching principles and locality, then introduces key performance metrics such as miss rate, miss penalty, and average access time. It presents a detailed performance example. The core of the lecture covers block size trade-offs and the three categories of cache organization based on block placement: direct mapped, fully associative, and set associative mapping, including their address structures, advantages, and disadvantages.
π Lecture Summary
Recap: Memory Hierarchy Principles
The memory hierarchy is designed to provide high-speed storage at the cheapest cost per byte. It organizes different memory modules based on the concept of caching and the principle of locality. Caching uses a small, fast, and expensive storage as a staging area for a frequently-used subset of data from larger, slower memory. The principle of locality states that a processor accesses a relatively small portion of the address space at any time, which includes temporal locality (recently accessed items will be accessed again) and spatial locality (items near accessed items will be accessed soon).
Recap: Working and Operation of Memory Hierarchy & Cache
The memory hierarchy moves recently accessed data items closer to the processor, along with their adjacent items. A cache device is a small SRAM that sits between the CPU and main memory. Data transfer between cache and CPU is word transfer, while between cache and main memory it is block transfer. The cache controller checks for data on a CPU request. If present, it's a HIT and data is delivered fast. If not present, it's a MISS, and the required block is read from main memory to cache, then delivered to the CPU.
Cache Memory Performance
Cache memory performance is a trade-off primarily measured by miss rate, miss penalty, and average access time. Miss Rate is the fraction of memory accesses not found in the cache, calculated as number of misses / total memory accesses. Since Hit Rate is the fraction found, Miss Rate = 1 β Hit Rate. Miss Penalty is the number of stall cycles incurred during a miss, including time to replace a block and deliver it to the processor. The formula for performance is:
- Average Access Time = Hit Time x (Hit Rate) + Miss Penalty x Miss Rate
- CPU Execution Time = (CPU Clock Cycles + Memory Stall Cycles) x clock cycle time
- Memory stall cycles = Number of Misses x Miss Penalty =
IC x (Misses / Instructions) x Miss Penalty=IC x (Memory Access per Instruction) x Miss Rate x Miss Penalty
π‘ Why this matters: These formulas are essential for quantifying the impact of cache misses on overall CPU performance, allowing architects to make informed design choices.
Cache Performance Example
- Assume: CPI=1.0 (all hits), 50% of instructions are load/store, miss rate = 2%, miss penalty = 25 clock cycles.
- Execution Time for all Hit:
IC x 1.0 x cycle time - Memory Stall Cycles:
IC x (1 + 0.5) x 0.02 x 25 = IC x 0.75 - CPU Execution Time (with cache):
(IC x 1.0 + IC x 0.75) x cycle time = 1.75 x IC x cycle time - Conclusion: The computer with no cache misses is 1.75 times faster.
Block Size Tradeoff: Miss Rate, Miss Penalty, and Average Access Time
As block size increases, the miss rate initially decreases due to spatial locality but will eventually increase, a phenomenon called the ping pong effect, where data bounces in and out of the cache. The miss penalty also increases with block size because more data must be transferred. Therefore, beyond a certain point, the average access time becomes a more comprehensive performance metric than either miss rate or miss penalty alone.
How do you Design a Cache?
The design process involves decoding the memory address from the CPU into a physical address. For a read, data is loaded from the memory location at that address. For a write, data is stored to that address. The core decision is how to place a block from main memory into the cache.
Categories of Cache Organization
There are three block placement policies for cache organization:
- Direct Mapped: Each block has only one place it can appear in the cache.
- Fully Associative: A block can be placed anywhere in the cache.
- Set Associative: A block can be placed in a restricted set of places (a group of blocks).
Direct Mapped Cache Organization
In a direct mapped cache, each main memory block maps to only one specific cache line. The mapping is determined by: (Block address) MOD (Number of blocks in the cache). For example, with 16 MB main memory and 1 MB cache organized in 4-byte blocks, main memory (1M blocks) is divided into 16 sections, each with 256K blocks. A block at line 0CE7 from any section will always map to line 0CE7 in the cache.
π Definition β Direct Mapping Address Structure: A 24-bit address is divided into a 2-bit word identifier (for 4-byte block), an 8-bit tag, and a 14-bit slot/line/index for the cache. The tag for blocks in the same cache line must be different.
π Formula: (Block address) MOD (Number of Blocks in the cache) β This determines the unique cache line where a block can be placed.
π Example: For a 1 KB direct mapped cache with a 32-byte block size, the address is split into a Byte Select field (5 bits for 32 bytes), a Cache Index field (5 bits for 32 sets, as 1 KB / 32 bytes = 32), and a Tag field (the remaining bits from a 32-bit address, e.g., 22 bits for a 4GB main memory).
Direct Mapping pros & cons
- Pros: Simple and inexpensive.
- Cons: If a program repeatedly accesses two blocks that map to the same line, cache misses are very high, a problem known as conflict misses. A valid bit is included to check if the cache contents are valid.
Associative Mapping (Fully Associative)
In associative mapping, a main memory block can load into any line of the cache. The memory address is interpreted as tag and word. The tag uniquely identifies the block of memory, and every line's tag is examined for a match. This eliminates the cache index.
Fully Associative Cache Organization
This design forgets about the cache index, allowing any block to be placed anywhere. It stores all upper bits of the address (except byte select) as the cache tag and uses one comparator for every cache entry. The address is sent to all entries in parallel for an associative lookup.
π Definition β Associative Mapping Address Structure: A 22-bit tag is stored with each 32-bit block of data. The tag field is compared with the tag entry in the cache to check for a hit. The least significant 2 bits identify the word within the block.
π Example: The address FFFFC has a tag 3FFFFC and data 24682468 stored in cache line 3FFF. There is no index; the tag must match any entry.
Characteristics of Fully Associative Cache
- This is the most complex end of the cache design spectrum (hardware intensive) and is limited to 64 or fewer entries.
- Conflict miss is zero, as any block can go anywhere.
- When more blocks are accessed than the cache can hold, a Capacity Miss occurs, requiring an existing block to be evicted.
Set Associative Mapping Summary
- Address length:
(s + w)bits; Number of addressable units:2^(s+w); Block size:2^wwords/bytes. - Number of blocks in main memory:
2^d; Number of lines in set:k; Number of sets:v = 2^d. - Number of lines in cache:
kv = k * 2^d; Size of tag:(s β d)bits.
Set Associative Mapping
The cache is divided into a number of sets, each containing a number of lines. A given block maps to any line within a specific set, chosen by: (block address) MOD (Number of sets in cache). If there are n blocks in a set, it's an n-way set associative cache. For example, a 2-way set associative cache has two cache entries for each cache index.
π Definition β Set Associative Mapping Address Structure: The address is divided into a Tag field, a Set field (used to determine the cache set), and a Word field (for the specific word within the block). The tags in the selected set are compared in parallel.
π Example (2-way): The address 1FF 7FFC has tag 1FF and data 12345678 in set number 1FFF. The address 001 7FFC has tag 001 and can be placed in the same set (1FFF) but in a different line within that set.
Working and Disadvantages of a 2-way Set Associative Cache
- How it works: The cache index selects a set. The two tags in that set are compared in parallel with the upper bits of the address. If neither matches, it's a miss. If one matches, the data from that side is selected.
- Disadvantages:
- Requires
Ncomparators instead of one. - Slower due to extra multiplexer delay.
- Data is available only after the hit/miss signal is valid. In a direct mapped cache, data is available before the hit/miss signal, allowing the processor to speculatively use the data (which is safe 90% of the time due to high hit rates). This speculation is not possible with a set-associative cache.
- Requires
β Key Takeaways
A student must understand that cache performance is governed by the trade-off between miss rate, miss penalty, and block size, which is encapsulated in the average access time formula. The three fundamental cache organizationsβdirect mapped, fully associative, and set associativeβoffer a spectrum of complexity and performance, balancing hardware cost against conflict misses. Direct mapped caches are simple and fast but prone to high conflict misses, while fully associative caches eliminate conflict misses but are hardware-intensive and slow. Set associative caches provide a practical compromise, with the number of ways being a key design parameter that affects both miss rates and access latency.
π§ Quick Revision Questions
- Define the three key performance metrics for a cache and write the formula for Average Access Time.
- How does the direct mapping function
(Block address) MOD (Number of blocks in the cache)determine where a main memory block is placed? - In a fully associative cache, what is the primary hardware requirement that makes it difficult to scale to large sizes?
- What is the key advantage of a direct mapped cache over a set-associative cache regarding data availability?
- A 2-way set associative cache has a miss rate of 1% and a miss penalty of 20 cycles. Calculate the Average Access Time if the hit time is 1 cycle.
π Lecture 28 β Memory Hierarchy Design (Cache Design and policies)
π Overview: This lecture explores memory hierarchy design, specifically focusing on cache memory design and policies. It covers block placement, identification, replacement, and write strategies, explaining how these policies impact cache performance and overall system efficiency in modern computer architectures.
ποΈ Topics Covered
Recap of cache addressing techniques and organizations (Direct Mapped, Fully Associative, Set Associative). Detailed coverage of memory hierarchy designer's concerns including block placement, identification, replacement policies, and write strategies. Discussion on write buffer management and write-miss policies with examples comparing no-write allocate versus write allocate.
π Lecture Summary
Recap: Block Size Trade off
The lecture reviews the impact of block size on cache performance, noting that larger block sizes reduce miss rate, but if block size is too big relative to cache size, miss rate increases. Miss penalty also increases with block size, and these parameters combine to affect Average Access Time.
Recap: Cache Organizations
Three cache organizations are reviewed based on block placement policy:
- Direct Mapped: each block has only one place it can appear in the cache, leading to Conflict Miss
- Fully Associative Mapped: any block of main memory can be placed anywhere in the cache
- Set Associative Mapped: allows placement of a block in a set of places in the cache
Memory Hierarchy Designerβs Concerns
Four key concerns for memory hierarchy designers:
- Block placement: Where can a block be placed in the upper level?
- Block identification: How is a block found if it is in the upper level?
- Block replacement: Which block should be replaced on a miss?
- Write strategy: What happens on a write?
Block Placement Policy
-
Fully Associative: Block can be placed anywhere in the upper level (Cache). Example: Block 12 from main memory can be placed at block 2, 6 or any of the 8 block locations in cache.
-
Set Associative: Block can be placed anywhere in a set in upper level (cache). The set number is given by: (Block No) MOD (number of sets). Example: an 8-block, 2-way set associative mapped cache has 4 sets [0-3] each of two blocks; block 12 or 16 of main memory can go anywhere in set #0 as (12 MOD 4 = 0) and (16 MOD 4 = 0). Block 14 can be placed at any of the 2 locations in set#2 (14 MOD 4 = 2).
-
Direct Mapped (1 way associative): Block can be placed at only one specific location in upper level (Cache). The location is given by: Block number MOD No. of cache blocks. Example: block 12 or block 20 can be placed at location 4 in cache having 8 blocks as (12 MOD 8 = 4).
Block Identification
A TAG is associated with each block frame giving the block address. All possible tags where a block may be placed are checked in parallel. A Valid bit is used to identify whether the block contains correct data. No need to check index or block offset.
π Definition β Direct Mapped Identification: Lower Level memory of 4GB uses a 32-bit address, with fields split between tag, index, and block offset for identification purposes.
Block Replacement Policy
On a cache miss, a new block needs to be brought in. If existing block locations are filled, an existing block must be replaced based on cache mapping and replacement policy.
Three main replacement policies:
-
Random: Replace any block. Simple and easiest to implement. The candidate for replacement is randomly selected. Some designers use pseudo random block numbers.
-
Least Recently Used (LRU): Replace the block either never used or used long ago. Reduces chances of throwing out information that may be needed soon. Access time and number of times a block is accessed is recorded. The block replaced is one that has not been used for the longest time. Example: if blocks are accessed in sequence 0,2,3,0,4,3,0,1,8,0, the victim to replace is block 2.
-
First-in, First-out (FIFO): The block first placed in the cache is thrown out first. Example: if blocks are accessed in sequence 2,3,4,5,3,4, then to bring in a new block, block 2 will be thrown out as it is the oldest accessed block. FIFO is used as approximation to LRU as LRU can be complicated to calculate.
π Conclusion - Miss Rate Comparison Table:
| Associativity | 2-way (LRU) | 2-way (Random) | 4-way (LRU) | 4-way (Random) | 8-way (LRU) | 8-way (Random) |
|---|---|---|---|---|---|---|
| 16 KB | 5.0% | 5.7% | 4.7% | 5.3% | 4.4% | 5.0% |
| 64 KB | 1.9% | 2.0% | 1.5% | 1.7% | 1.4% | 1.5% |
| 256 KB | 1.15% | 1.17% | 1.13% | 1.13% | 1.12% | 1.12% |
π‘ Why this matters: For larger caches (256 KB), LRU and Random perform similarly, but for smaller caches (16 KB), LRU provides better miss rates, making it the preferred choice for performance-critical systems.
Write Strategy
Memory hierarchy must not overwrite a cache block unless main memory is up to date. Multiple CPUs may have individual caches, and I/O may address main memory directly. Instruction cache accesses are read, writes are typically 10% of cache access. Data cache writes are 10%-20% of overall memory access.
-
Write back: Information is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced.
- Pros: No write to lower level for repeated writes to cache; a dirty bit indicates if cache block is modified (dirty) or not modified (clean)
- Cons: More complex replacement procedure; reduces memory-bandwidth and power requirements
-
Write through: Information is written to both the block in the cache and to the block in the lower-level memory.
- Pros: Simplifies replacement procedure; block is always clean; simplifies data-coherency
- Cons: Higher memory traffic; always combined with write buffers so CPU doesn't wait for lower level memory
Write Buffer for Write Through
A write buffer is a FIFO with typical number of entries: 4. Once data is written into the write buffer (assuming cache hit), CPU is done with the write. The memory controller moves the write buffer's contents to real memory behind the scene. DRAM cycle time sets the upper limit on write frequency.
π Definition β Write Buffer Saturation: Occurs when Store frequency approaches 1 over DRAM Write Cycle Time, i.e., CPU Cycle Time <= DRAM Write Cycle Time. In saturation, no matter how big the write buffer, it will still overflow because items are fed in faster than they can be emptied.
Two solutions to Write Buffer Saturation:
- Replace write through cache with write back cache
Write-Miss Policy
Two options for handling write-misses:
-
Write Allocate: A block is allocated on a write-miss, followed by the write hit action. The rest of the block is read from memory (Byte 2, 3, ... Byte 31) after writing the tag and data.
-
No-write Allocate: Write-misses do not affect the cache; the block is modified only in the lower level memory. The blocks stay out of the cache until the program tries to read them. This practice uses sub-blocking to tell the processor the rest of the block is no longer valid.
π Example: No write-allocate vs write allocate
Consider a fully associative write-back cache with empty entries and the following sequence:
- Write Mem [100]
- Write Mem [100]
- Read Mem [200]
- Write Mem [200]
- Write Mem [100]
For no-write allocate:
- First write [100]: MISS (tag not in cache)
- Second write [100]: MISS
- Read [200]: MISS
- Write [200]: HIT
- Write [100]: MISS
- Result: 4 MISSes and 1 HIT
For write-allocate:
- First access to [100]: MISS
- First access to [200]: MISS
- Rest are HITS (both [100] and [200] found in cache)
- Result: 2 MISSes and 3 HITs
Conclusion:
- Write-back caches normally use write-allocate, hoping subsequent writes to the block will be captured by the cache
- Write-through caches often use No Write Allocate, because even if subsequent writes occur, they must go to lower level memory
β Key Takeaways
This lecture establishes that cache design involves four critical decisions: block placement (determining where blocks go), block identification (using tags and valid bits), replacement policy (LRU generally outperforms Random for smaller caches but performs similarly for larger ones), and write strategy (write-back with dirty bits reduces memory traffic while write-through simplifies coherency). Write buffer management is crucial to prevent saturation, especially when CPU cycle time approaches DRAM write cycle time. For write-miss policies, write-back caches typically use write-allocate while write-through caches prefer no-write allocate, and these choices significantly impact hit/miss ratios as demonstrated by the example showing 4 misses vs 2 misses for the same access sequence.
π§ Quick Revision Questions
- What are the four main concerns of a memory hierarchy designer when designing cache?
- How does LRU replacement policy differ from FIFO, and why is LRU generally preferred for smaller caches?
- What is the key difference between write-back and write-through strategies, and what advantage does each provide?
- What is write buffer saturation, and what are the two solutions to address it?
- In the example comparing no-write allocate versus write allocate for a sequence of five memory operations, why does write-allocate result in fewer misses (2 vs 4)?
π Lecture 29 β Memory Hierarchy Design
Cache Performance Enhancement by: Reducing Cache Miss Penalty
π Overview: This lecture focuses on techniques to reduce the cache miss penalty, a critical factor in improving overall CPU performance. Starting with a recap of cache design fundamentals, it explores five key methods: multilevel caches, critical word first/early restart, priority to read misses, merging write buffers, and victim caches. These strategies aim to minimize the time penalty incurred when a cache miss occurs, directly impacting the average memory access time and CPU execution time.
ποΈ Topics Covered
Recap of cache design concepts including block placement, identification, replacement, and write strategies, along with write buffer and write miss policies. The lecture then derives CPU performance impact using miss rate and miss penalty, introduces the average memory access time formula, and details five specific techniques for reducing miss penalty: multilevel caches, critical word first and early restart, priority to read misses over write misses, merging write buffers, and victim caches.
π Lecture Summary
Recap: Memory Hierarchy Designerβs Concerns
The designer must consider four key concerns for the memory hierarchy. Block placement determines where a block can be placed in the upper level. Block identification determines how a block is found if it is in the upper level. Block replacement decides which block should be replaced on a miss. Write strategy governs what happens on a write.
Recap: Write Buffer for Write Through
Cache write strategies include write back and write through. A write-buffer is used in write through to hold data waiting to be written to memory, allowing the processor to continue. A level-2 cache can be introduced between the L1 cache and DRAM. Two write miss policies are Write Allocate, where a block is allocated in the cache on a write miss, and No-Write Allocate, where the block stays out of the cache until the program tries to read it.
π Definition β Write Allocate: A block is allocated in the cache on a write miss, i.e., the block to be written is available in the cache. π Definition β No-Write Allocate: The blocks stay out of the cache until the program tries to read the blocks; i.e., the block is modified only in the lower level memory.
Impact of Caches on CPU Performance
The CPU execution time is given by: π Formula: CPU Execution Time = (CPU Execution clock cycles + Memory Stall cycles) x Clock Cycle Time
Example:
- Assumptions: cache miss penalty = 100 clock cycles, all instructions take 1 clock cycle, average miss rate = 2%, average memory references per instruction = 1.5, average number of cache misses per 1000 instructions = 30.
- Find the impact of cache on CPU performance considering both misses per instruction and miss rate.
π Solution (misses per instruction): CPU Time = IC x (1.0 + (30/1000 x 100)) x clock cycle time = IC x 4.00 x clock cycle time
π Solution (miss rate): CPU Time = IC x (1.0 + (1.5 x 2% x 100)) x clock cycle time = IC x 4.00 x clock cycle time
Cache Performance (Review)
Memory stall clock cycles are the sum of:
- IC x Reads per instruction x Read miss rate x Read Miss Penalty
- IC x Writes per instruction x Write Miss Rate x Write Miss Penalty
By averaging read and write miss rates: π Memory stall clock cycles = Number of memory accesses x Miss rate x Miss penalty
π Average Memory Access Time = Hit Time + Miss rate x Miss penalty π‘ Why this matters: The average memory access time is an indirect measure of CPU performance and is not a substitute for execution time. However, this formula can decide about split caches (instruction and data) vs. unified caches.
Example: Split vs. Unified Caches
- Statement: Consider a 32KB unified cache with misses per 1000 instructions = 43.3, and 16KB instruction/data split caches with instruction misses per 1000 = 3.82 and data misses per 1000 = 40.9. Assume 36% of instructions are data transfer, 74% of memory references are instruction references, hit takes 1 clock cycle, miss penalty = 100 cycles, and a load/store takes one extra cycle on unified cache. Find the average memory access time for each case (write-through with write-buffer, ignoring write buffer stalls).
π Solution:
-
Miss Rate = (Misses/1000) / (Accesses/instruction)
- Miss RateββKB Inst = (3.82/1000) / 1.0 = 0.0038
- Miss RateββKB data = (40.9/1000) / 0.36 = 0.114
- Overall miss rate for split caches = (74% x 0.0038) + (26% x 0.114) = 0.0324
- Miss RateββKB unified = (43.3/1000) / (1+0.36) = 0.0318
-
Average Memory Access Time = %inst x (Hit time + Inst. Miss rate x miss penalty) + %data x (Hit time + data Miss rate x miss penalty)
- Average Memory Access Time_split = 74% x (1 + 0.0038 x 100) + 26% x (1 + 0.114 x 100) = 4.24
- Average Memory Access Time_unified = 74% x (1 + 0.0318 x 100) + 26% x (1+1+0.0318 x 100) = 4.44
- Result: Split caches have slightly better average access time (4.24 vs. 4.44) and avoid structural hazards, even though the unified cache has a slightly lower miss rate.
Improving Cache Performance
The average memory access time formula provides a framework to optimize cache performance: Average Memory Access Time = Hit Time + Miss Rate x Miss Penalty. Four general options exist: 1) Reduce the miss penalty, 2) Reduce the miss rate, 3) Reduce miss penalty or miss rate via parallelism, and 4) Reduce the time to hit in the cache.
Reducing Miss Penalty
Five techniques are covered: 1) Multilevel Caches, 2) Critical Word First and Early Restart, 3) Priority to Read Misses Over Write Misses, 4) Merging Write Buffers, and 5) Victim Caches.
1: Multilevel Caches
This technique ignores the CPU but concentrates on the interface between cache and main memory. Multiple levels of caches create a tradeoff between cache size (effectiveness) and cost (access time), with a small fastest memory used as level-1 cache.
Performance Analysis: π Average Access Time = Hit TimeL1 + Miss RateL1 x Miss PenaltyL1 Where, Miss PenaltyL1 = Hit TimeL2 + Miss RateL2 x Miss PenaltyL2
π Average memory access time = Hit TimeL1 + Miss RateL1 x (Hit TimeL2 + Miss RateL2 x Miss PenaltyL2)
π Stall per instruction_average = Misses per instructionL1 x Hit TimeL2 + Misses per instructionL2 x Miss PenaltyL2
π Definition β Local Miss Rate: Measure of misses in a cache divided by the total number of misses in this cache. π Definition β Global Miss Rate: Measure of the number of misses in the cache divided by the total number of memory accesses generated by the CPU. For L1: Miss RateL1. For L2: Miss RateL1 x Miss RateL2.
Example:
- Given: For 1000 references with 40 misses in L1 and 20 in L2. Miss penalty for L2 cache-memory = 100 clock cycles, hit time for L2 = 10 clock cycles, hit time for L1 = 1 clock cycle, memory references per instruction = 1.5.
π Solution:
- L1 cache miss rate = 4% (40/1000 x 100)
- Local Miss Rate for L2 = 50% (20/40)
- Global Miss Rate for L2 = 2% (20/1000 x 100)
- Average Memory Access Time = 1 + 4% x (10 + 50% x 100) = 1 + 0.04 x (10 + 50) = 1 + 2.4 = 3.4 cycles
- Misses per instruction for L1 = 40 x 1.5 = 60 per 1000 instructions
- Misses per instruction for L2 = 20 x 1.5 = 30 per 1000 instructions
- Average Memory Stalls per instruction = (60/1000) x 10 + (30/1000) x 100 = 0.6 + 3.0 = 3.6 clock cycles
- Result: The average miss penalty using multilevel caches reduces by a factor of 100 / 3.6 β 28 relative to the single level cache.
2: Critical Word First and Early Restart
Donβt wait for the full block to be loaded before restarting the CPU. The CPU normally needs one word of a block at a time.
π Definition β Early Restart: Request the words in a block in normal order. As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution. π Definition β Critical Word First: Request the missed word from memory first; the memory sends it to the CPU as soon as it arrives. The CPU continues filling the rest of the words in the block afterward.
Example:
- A computer uses 64-byte (8 word) cache blocks. An L2 cache takes 11 clock cycles to get the first 8-byte (critical word) and then 2 clock cycles per 8-byte word for the rest of the block (2 issues per cycle). Compare:
- With critical word first (assuming no other access to the rest of the block)
- Without critical word first (assuming instructions read data sequentially 8-byte words at a time from the rest of the block, requiring a block load)
π Solution:
- With Critical Word First: Average miss penalty = Miss Penalty of critical word + Miss penalty of remaining words = 11 x 1 + (8-1) x 2 = 11 + 14 = 25 clock cycles
- Without Critical Word First: Average miss penalty = [Miss Penalty of first word + miss penalty of remaining words] + clock cycles to issue the load = [11 x 1 + (8-1) x 2] + 8/4 = 25 + 4 = 29 clock cycles (8 issues / 2 issues per cycle = 4 cycles)
Merit: This technique doesnβt require extra hardware. Drawback: It is generally useful only in large blocks; programs exhibiting spatial locality may face a problem accessing data or instructions from memory if the next miss is to the remainder of the block.
3: Priority to Read Miss over the Write Misses
This technique reduces the average miss penalty by considering the overlap between the CPU and cache miss penalty. Write-buffers ensure that writes to memory do not stall the processor. However, a write-buffer may hold the updated value of a location needed on a read miss, and the processor is blocked until the read returns. Write buffers complicate memory access.
π Definition β Raw Data Hazard (RAW): A situation where a read after a write to the same location may get the old value if the write has not completed.
Example Program Segment:
SW R3, 512(R0) ; M[512] β R3 (Cache index 0)
LW R1, 1024(R0) ; R1 β M[1024] (Cache index 0)
LW R2, 512(R0) ; R2 β M[512] (Cache index 0)
Assume a direct-mapped, write-through cache that maps 512 and 1024 to the same block, and a 4-word write buffer. Find if the value in R2 will always equal the value in R3.
π Analysis: The data in R3 is placed in the write-buffer after the first instruction. The second load (to 1024) is a miss. The third load (to 512) also results in a miss. If the write buffer hasnβt completed writing to location 512 in memory, the read of location 512 will put the old value into the cache and then into R2. Thus, R3 would not equal R2.
Solution: Give priority to read misses. This can be done by either waiting for the write buffer to empty, or by checking write buffer contents on a read miss. If there are no conflicts and the memory system is available, let the memory access continue for the read. By giving priority to read misses, the cost (penalty) of writes can be reduced. In write-back caches, a better alternative to writing a dirty block to memory before a read is to copy the dirty block to a write-buffer, then do the read, then do the write. The CPU stalls less since it restarts as soon as the read is done.
4: Merging Write Buffer
Write-through caches rely on write-buffers as all stores must be sent to the lower level. Even write-back caches use a simple buffer when a block is replaced. A small write-buffer may end up stalling the processor if it fills up. This is resolved by merging cache-block entries in the write buffer. Multiword writes are faster than writes performed one at a time. If a write buffer already contains some words from a given data block, the current modified word can be merged with the block parts already in the buffer.
π Definition β Write Merge: If the buffer contains other modified blocks, the address is checked to see if the address of the new data matches the address of a valid write buffer entry. If so, the new data are combined with the existing entry.
π‘ Why this matters: This technique reduces the number of stalls due to the write-buffer being full, thereby reducing the miss penalty through improvement in the efficiency of the write-buffer.
5: Victim Caches
Another way to reduce the miss penalty is to remember what was discarded, as it may be needed again. The victim cache contains only discarded blocks because of some earlier miss. On another miss, the victim cache is checked to see if it has the desired data before going to the next lower-level memory. If the desired data is found, the victim block and cache block are swapped. This recycling requires a small fully associative cache between a cache and its refill path, called the victim cache.
β Key Takeaways
The first and most important technique to reduce miss penalty is the use of multilevel caches, which can dramatically reduce the effective miss penalty by a factor proportional to the product of local miss rates. The Critical Word First and Early Restart technique reduces the penalty by not waiting for the entire block to load before restarting the CPU. Giving priority to read misses over write misses avoids RAW hazards and reduces stalls in both write-through and write-back caches. Merging write buffers improves efficiency by combining multiple writes to the same block, reducing the number of stalls. Finally, victim caches provide a small, fast buffer for recently evicted blocks, reducing the penalty for blocks that are quickly re-referenced. All these methods aim to optimize the average memory access time by directly reducing the miss penalty component of the formula: Average Memory Access Time = Hit Time + Miss Rate x Miss Penalty.
π§ Quick Revision Questions
- What is the formula for CPU Execution Time, and how does the cache miss penalty affect it?
- Explain the difference between local and global miss rates in a multilevel cache hierarchy, and calculate them for a system with 50 L1 misses and 10 L2 misses out of 1000 memory accesses.
- How does the "Critical Word First" technique reduce the average miss penalty, and what is a potential drawback of this technique?
- Describe the RAW data hazard in a write-through cache with a write buffer. How does giving priority to read misses resolve this hazard?
- What is a victim cache, and how does it reduce the miss penalty compared to directly accessing the next level of memory?
π Lecture 30 β Memory Hierarchy Design Cache Performance Enhancement (Reducing Miss Rate)
π Overview: This lecture focuses on reducing cache miss rate as a key strategy for improving cache performance. It classifies the three types of cache misses and presents five hardware and software techniques to minimize them, including larger block sizes, larger caches, higher associativity, way prediction, and compiler optimizations.
ποΈ Topics Covered
The lecture begins with a recap of reducing miss penalty through multilevel caches, critical word first, priority to read misses, merging write buffers, and victim caches. It then introduces the three Cβs classification of cache misses: compulsory, capacity, and conflict misses. The main focus is on five techniques for reducing miss rate: larger block size, larger cache size, higher associativity, way prediction and pseudo-associativity, and compiler optimization including loop interchange and blocking. A final example compares average memory access time across different associativity levels.
π Lecture Summary
Recap: Improving Cache Performance
Cache performance can be improved by addressing miss penalty, miss rate, parallelism, or hit time. The miss penalty can be reduced through several techniques: Multilevel Caches (more cache levels reduce penalty), Critical Word First and Early Restart (request the needed word first), Priority to Read Misses Over Writes (read misses get higher priority), Merging Write Buffers (combining writes to same block), and Victim Caches (small fully-associative cache to hold evicted blocks).
Cache Misses
Cache misses are classified into three types:
- Compulsory Misses (cold start or first reference misses): occur when a block is first accessed and must be brought into the cache
- Capacity Misses: occur in a fully associative cache when the working set exceeds cache size
- Conflict Misses (collision or interference misses): occur when multiple blocks map to the same cache set or address
π‘ Why this matters: Understanding miss classification helps target the right technique for each type.
1: Larger Block Size
Larger blocks reduce miss rate by exploiting spatial locality β larger blocks bring more data or instructions per access. However, in small caches, larger blocks may increase miss rate due to fewer total blocks available.
π Definition β Miss Penalty: the time to fetch a block from lower memory levels π Formula: Average Memory Access Time = Hit time + Miss Rate Γ Miss Penalty π Example: With 80 clock cycles overhead, 16 bytes every 2 clock cycles delivery, hit time = 1 clock cycle
- For a 4KB cache, miss rate = 7.24%
- Miss penalty = 80 + 4 = 84 clocks (80 overhead + 4 for 16 bytes/2 cycles = 8 cycles, wait β correction: 80 + 4? Actually the text says: Miss penalty = 80 + 4 = 84 clocks)
- Average Memory Access Time = 1 + (7.25% Γ 84) = 7.082 Clock cycles
- Larger blocks exploit high latency and high bandwidth of lower level memory
2: Large Cache Size
Larger caches reduce capacity misses. By 2001, processors used 2nd-level and 3rd-level caches. Drawbacks include longer hit time and higher cost (access time increases with cache size).
3: Higher Associativity
Higher associativity reduces conflict misses but increases hit time. The cache cycle time (CCT) ratios relative to direct-mapped (1-way):
- CCT 2way = 1.36 Γ CCT 1way
- CCT 4way = 1.44 Γ CCT 1way
- CCT 8way = 1.56 Γ CCT 1way
4: Way Prediction and Pseudo-associativity
This technique combines the fast hit time of direct-mapped caches with the lower conflict misses of set-associative caches.
Way Prediction predicts which block in a set will be accessed. Steps:
- Extra bits β 2-way prediction or 4-way prediction
- Multiplexer β single tag check
Other blocks are checked for matches in subsequent clock cycles. The Alpha 21264 uses this with 1 clock cycle latency and 3 clock cycles for miss handling.
Pseudo-associative (column associative) caches β on a miss, the cache checks a "pseudo-set" to find the block. Performance involves a "slower hit" when the prediction is wrong.
5: Compiler Optimization
Compiler optimizations reduce both data and instruction cache misses.
Instruction cache optimization:
- Code reordering β determines conflicts between procedures
- Code-line alignment β decreases cache miss by ensuring entry points align with cache block boundaries
Data cache optimization:
- Improves spatial locality and temporal locality
- Array calculations β using loop interchange and blocking
π Example β Loop Interchange: The first version accesses data non-sequentially for j (0β100) and sequentially for i (0β5000):
/* First Version */
for (k = 0; k < 100; k = k+1)
for (j = 0; j < 100; j = j+1)
for (i = 0; i < 5000; i = i+1)
x[i][j] = 2 * x[i][j];
Reordered version swaps j and i loops to access memory sequentially:
/* Reordered version */
for (k = 0; k < 100; k = k+1)
for (i = 0; i < 5000; i = i+1)
for (j = 0; j < 100; j = j+1)
x[i][j] = 2 * x[i][j];
π Example β Blocking for Matrix Multiplication: Blocking improves temporal locality. The example uses row-major order and column-major order for matrix multiplication:
/* Initial version of matrix multiplication code */
for (i = 0; i < N; i = i+1)
for (j = 0; j < N; j = j+1)
{r = 0;
for (k = 0; k < N; k = k+1) {
r = r + y[i][k]*z[k][j]; };
x[i][j] = r;
};
β Key Takeaways
The most critical points for exam understanding: (1) Cache misses are classified into three types β compulsory, capacity, and conflict β each requiring different reduction strategies. (2) Larger block sizes reduce compulsory misses but can increase miss rate in small caches; larger caches reduce capacity misses but increase hit time. (3) Higher associativity reduces conflict misses but increases cache cycle time, with ratios of 1.36x for 2-way to 1.56x for 8-way relative to direct-mapped. (4) Way prediction techniques check a section of cache first (for fast hit) then fall back to checking the rest on a miss, balancing speed and miss rate. (5) Compiler optimizations like loop interchange and blocking are software approaches that improve spatial and temporal locality without hardware changes.
π§ Quick Revision Questions
- What are the three types of cache misses (the "Three C's") and what causes each?
- Calculate the average memory access time for a 4KB cache with 7.24% miss rate, 80 clock overhead, 16 bytes/2 cycles delivery, and 1 clock hit time.
- What is the cache cycle time (CCT) ratio for an 8-way set-associative cache compared to direct-mapped?
- How does way prediction differ from pseudo-associativity in handling cache accesses?
- Explain how loop interchange reduces cache misses in nested loops that access a 2D array non-sequentially.