π Lecture 31 β Memory Hierarchy Design
π Overview: This lecture explores advanced cache performance enhancement techniques focused on reducing miss penalty and miss rate through parallelism, as well as reducing hit time. Building on earlier concepts of the 3C model, it introduces non-blocking caches, hardware and software prefetching, and practical methods to minimize the time to access the cache, all crucial for modern high-performance processors.
ποΈ Topics Covered
The lecture begins with a recap of reducing miss rate using the 3C model (compulsory, capacity, conflict misses). It then introduces parallelism-based techniques: non-blocking caches (hit under miss/multiple misses), hardware prefetching with stream buffers (including a detailed UltraSPARC III example), and compiler-controlled prefetching (register and cache variants). Finally, it covers four techniques to reduce hit time: small and simple caches, avoiding address translation during indexing (virtual caches), pipelined cache access, and trace caches, concluding with a comprehensive summary table of cache optimization techniques.
π Lecture Summary
Recap: 3 C Model and Reducing Miss Rate
- Large block size reduces compulsory misses.
- Large cache size reduces capacity misses.
- Higher associativity reduces conflict misses.
- Way-prediction checks a section of cache for hit first; on miss, it checks the rest.
- Compiler-based techniques include loop interchange and blocking to optimize cache performance.
- Today's focus is on other enhancement methods: 1. Reducing Miss Penalty or Miss Rate via parallelism β overlapping execution of instructions with memory hierarchy activities. 2. Reducing the hit time.
Reducing Miss Penalty or Rate via Parallelism
- The basic idea is to reduce miss penalty or miss rate by performing multiple outstanding memory operations through overlapping memory activities and instruction execution activities in the processor.
- This can be accomplished using three techniques:
- Non-blocking Caches: reduces stall on misses.
- Hardware Prefetch: reduces number of misses.
- Software (compiler controlled) Prefetch: reduces number of misses.
Non-blocking Caches
- Memory hierarchy involves decoupled instruction and data caches in an out-of-order execution CPU.
- Non-blocking or lockup-free caches allow the processor to continue during a miss.
- Key concepts include "hit under miss", "hit under multiple miss", and "miss under miss".
- Complexity: For example, a miss to address 1000 and later a miss to address 1032; the complexity of the cache controller increases.
- The memory system can service multiple misses.
- Example processor: Pentium Pro.
Hardware Prefetch: Reduces Misses
- Prefetches 2 blocks: the requested block and the next sequential block.
- Uses a "stream buffer" : if the next requested block is already available in the stream buffer:
- Original cache request is cancelled.
- Block is read from the stream buffer.
- Next prefetch request is issued.
- Reduces "demand misses" .
- Jouppi in 1990 found: 15% - 25% of misses reduced with 1 block stream buffer; 4 blocks stream buffer achieved 43% for stream fetching; 50% at the same address; 16-block stream buffer achieved 72% miss reduction.
- Works efficiently for data caches.
- Hardware identifies the stream of accesses.
- Palacharla & Kessler in 1994: using 8 stream buffers achieved 50% to 70% miss reduction for 64KB, 4-way set associative caches.
π Example: For UltraSPARC III, 64 KB data cache and 256 KB cache have average misses per 1000 instructions of 36.9 and 326 respectively, assuming:
- prefetching hit rate equals 20%
- hit time is 1 clock cycle
- miss penalty is 15 cycles
- 1 extra clock cycle if data misses the cache but found in prefetch buffer
- data references equal 22%
Solution:
-
Miss rate_prefetch = [Misses/1000] / data references = [36.9 / 1000] / [22 / 100] = 36.9 / 220 = 16.7%
-
Average memory access time_prefetch = Hit Time + Miss rate Γ Prefetch hit time Γ 1 + Miss rate Γ (1 - prefetch hit time) Γ miss penalty Average memory access time_prefetch = 1 + (16.7% Γ 20% Γ 1) + (16.7% Γ (1-20%) Γ 15) = 3.046
a) Effective Miss rate_prefetched 64K = [Average memory access time β Hit time] / Miss Penalty = [3.064 β 1] / [15] = 2.064 / 15 = 13.6%
b) From the given data, 256KB data cache yields miss rate: Miss rate_256KB = 33.6 / (22% Γ 1000) = 14.8%
Software (Compiler Controlled) Prefetch:
-
Two variants of prefetch:
- Register Prefetch: Load data into register (HP RISC).
- Cache Prefetch: Load data only into cache, not the register (MIPS IV, PowerPC, SPARC v.9).
-
Issues and limitations of compiler prefetching:
- Faulting prefetch vs. Non-faulting prefetch.
- Compiler prefetching is semantically invisible; it cannot cause virtual memory faults.
- Caches do not stall.
- Overhead of issuing the prefetch instructions.
Summary: Reducing Cache Miss Penalty or Miss Rate via Parallelism
- The non-blocking caches.
- Bandwidth behind the cache and Instruction-level parallelism.
- Hardware and Software prefetching.
Final Component to Reduce Average Memory Access Time - Hit Time
- Five techniques to reduce the miss penalty.
- Five methods to reduce the miss rate.
- Three approaches to reduce the miss penalty and miss rate in parallel to reduce the average memory access time.
- Hit time is the final component to optimize.
Reducing Hit Time
- Clock rate of the processor.
- Cache Access time.
The four commonly used techniques:
- Small and Simple Caches
- Avoiding Address Translation during Indexing
- Pipelined Cache Access
- Trace Cache
Small and Simple Caches
- The most time-consuming operation in a cache hit is comparing the long address tag.
- This operation requires a bigger circuit which is always slower compared to smaller circuits.
- An obvious approach to keep hit time small is to keep the cache smaller and simpler.
- The cache should be kept small to fit on the same chip as the processor to avoid off-chip time penalty; and simpler, such as direct-mapped, where tag comparison can be overlapped with data transmission.
- The impact of cache size and complexity (associativity and number of read/write ports) is significant.
- Size of L1 caches: 16KB L1 cache on Pentium III, 8KB on Pentium IV.
Avoiding Address Translation during Indexing
- For a small and simple cache, Translation Look-aside Buffer (TLB) is used. The virtual address from the CPU goes through the virtual memory system.
- Two levels of address mapping: address from virtual memory to main memory, and main memory to cache.
- Using a virtual address for the cache allows directly mapped access.
- Virtual caches vs. Physical cache: the virtual address translation step is eliminated from cache hit time (Amdahl's rule).
- Limitation of virtually addressed caches: issues include protection, page address, physical address difference (a virtual address may refer to different physical addresses), and aliasing/synonyms (two different virtual addresses or two processes pointing to the same physical address).
- Hardware solution to synonyms: 2-way set associative cache.
- Software solution: share some address bits.
Pipelined Cache Access
- Cache hit time can be improved by pipelining the cache access.
- This increases the latency of the first level cache-hit but enables a faster cycle time which increases the bandwidth of instructions.
- Example: Pentium I takes 1 clock cycle to access the instruction cache; Pentium Pro through Pentium II takes 2 clock cycles; Pentium IV takes 4 clock cycles.
Trace Caches
- For multiple-issue processors, trace caches store the dynamic sequence of instructions.
- Instructions are loaded into the cache block based on branch prediction and instruction prefetching.
- Advantages: No wasted words and no conflicts; conflicts are avoided.
- Demerit: Complicates the address mapping because blocks are no longer aligned to power-of-2 multiples of words. Requires a more complex address mapping.
- Disadvantage: The same instruction may be stored multiple times.
Summary β Cache Optimization
- 5 methods to reduce the miss penalty.
- 7 ways to reduce 3Cs (compulsory, capacity, conflict misses).
- 3 methods for reducing miss rate and miss penalty via parallelism.
- 4 techniques to reduce hit time.
Table 1: Cache Optimization Techniques β Miss Penalty, Miss Rate, and Hit Time
| Technique | Miss Penalty | Miss Rate | Hit Time | Complexity | Comments |
|---|---|---|---|---|---|
| Miss Penalty | |||||
| Multilevel caches | + | 2 | Costly H/W | ||
| Early Restart & Critical Word 1st | + | 2 | Widely Used | ||
| Priority to Read Misses | + | 1 | Trivial for Uni-processor | ||
| Merging write buffer | + | 1 | used with w/t through | ||
| Victim Caches | + | + | 2 | ||
| Miss Rate | |||||
| Larger Block Size | - | + | 0 | Trivial | |
| Larger Cache Size | - | + | 1 | Widely Used | |
| Higher Associativity | + | - | 1 | Widely Used | |
| Pseudo-Associative | + | 2 | Used in L2 | ||
| Way Predicted | + | 2 | Used in I-Cache | ||
| Compiler Reduce Misses | + | 0 | S/W approach |
Table 2: Cache Optimization Techniques β Parallelism and Hit Time
| Technique | Miss Penalty | Miss Rate | Hit Time | Complexity | Comments |
|---|---|---|---|---|---|
| Parallelism | |||||
| Non-Blocking | + | 3 | out-of-order CPU | ||
| HW Prefetching | + | + | 2 inst of Instr/Data, 3 data | ||
| Compiler Controlled Prefetching | + | + | 3 | need non-blocking cache | |
| Hit Time | |||||
| Avoiding Address Translation | + | 2 | Widely Used | ||
| Trace Cache | + | 3 | Used in P |
β Key Takeaways
This lecture presents a comprehensive toolkit for improving cache performance. The primary strategies target three key metrics: miss penalty, miss rate, and hit time. To reduce miss penalty and miss rate through parallelism, the core techniques are non-blocking caches (allowing hits/misses under outstanding misses), hardware prefetching (using stream buffers to anticipate data needs and reduce demand misses), and compiler-controlled prefetching (inserting explicit prefetch instructions). To reduce hit time, the essential approaches are keeping caches small and simple (to minimize tag comparison time), using virtual addresses to avoid costly TLB translation during indexing, pipelining cache access (to support higher clock frequencies), and employing trace caches (to store dynamic instruction sequences and improve instruction fetch bandwidth in wide-issue processors). The final summary tables categorize all optimization techniques by their impact on miss penalty, miss rate, and hit time, along with their complexity, providing a clear decision framework for architects.
π§ Quick Revision Questions
- What is the fundamental difference between "hit under miss" and "hit under multiple miss" in a non-blocking cache?
- In hardware prefetching with a stream buffer, what three steps are taken if a requested block is found in the buffer?
- For the UltraSPARC III example, calculate the effective miss rate for a 64KB cache with prefetching, given the average memory access time is 3.046 cycles, hit time is 1 cycle, and miss penalty is 15 cycles.
- What are the two main issues that arise when using virtually addressed caches, and what is the hardware solution for one of them?
- How does pipelining cache access improve performance even though it increases the latency of the first-level cache hit?
π Lecture 32 β Memory Hierarchy Design (Main and Virtual Memories)
π Overview: This lecture completes the memory hierarchy discussion by examining main memory and virtual memory performance. It explains how DRAM organization impacts memory bandwidth, introduces techniques for improving main memory performance, and contrasts virtual memory with cache memory. Understanding these concepts is critical for designing systems that balance speed, cost, and capacity across the memory hierarchy.
ποΈ Topics Covered
The lecture recaps cache design and performance metrics, then dives into main memory organization and DRAM types (Fast Page Mode, SDRAM, DDR). It presents three techniques for improving main memory performance: wider main memory, interleaved memory, and independent memory banks, with detailed bandwidth calculations. The final sections introduce virtual memory concepts, protection, relocation, page faults, and compare virtual memory design issues with cache memory.
π Lecture Summary
Recap: Memory Hierarchy
The design goal of a memory system is to achieve the low cost of the cheapest memory combined with the fast speed of the fastest memory. The fastest, smallest, and most costly memories sit at the top, while the slowest, biggest, and cheapest memories sit at the bottom. Key parameters are average access speed, cost, and the cheapest technology available. Semiconductor memories include Static and Dynamic RAMs, which occupy the upper levels of the memory hierarchy.
π‘ Why this matters: This hierarchy concept explains why computers use multiple memory types rather than relying on a single technology.
Recap: Caches Design
Caches use Static Random Access Memory (SRAM) , while Main Memory is Dynamic Random Access Memory (DRAM) (with access time ~8 ms, occurring <5% of the time). The magnetic, optical, or other media form the lower levels (e.g., disk). Virtual memory extends this hierarchy further. Cache and main memory are organized in equal sized blocks. Word transfer is fast for data requested by the CPU, while block transfer moves larger chunks between levels.
Recap: Cache Performance
If misses occur, the miss penalty (time to fetch a block from lower level) becomes critical. Cache design and performance depend on techniques that optimize miss rate, miss penalty, and hit time.
Main Memory Organization
Main memory serves as the source for caches (providing data on cache misses) and the destination for virtual memory (holding pages from disk). The DRAM logical organization (4 M Bit) consists of a memory array arranged in rows and columns, controlled by RAS (Row Address Strobe) and CAS (Column Address Strobe).
Main Memory Performance
Performance of DRAM is characterized by three types:
- Fast page mode DRAM β Optimizes sequential access by keeping the row address constant while changing column addresses.
- Synchronous DRAM (SDRAM) β Avoids handshaking by using a clock to synchronize operations.
- Double Data Rate (DDR) DRAM β Transmits data on both rising and falling clock edges.
Key performance metrics are Latency (average memory access time) and Bandwidth (number of bytes read/write per unit time). These depend on Access Time and Cycle Time. Inputs/outputs and multiprocessors require higher bandwidth; low-latency memory helps single-thread performance. Multiprocessors demand higher bandwidth, often using 2nd level caches with larger block size to reduce the effective miss penalty.
Improving Main Memory Performance
The most commonly used techniques are: β Wider Main Memory β Simple Interleaved Memory β Independent Memory Banks
1: Wider Main Memory β Example
Consider a 4-word block (i.e., 32 bytes). Timing parameters:
- Time to send address = 4 clock cycles
- Time to send the data word = 4 clock cycles
- Access time per word = 56 clock cycles
Miss Penalty Formula: No. of words Γ [time to: send address + send data word + access word]
For 1-word organization: Miss Penalty = 4 Γ (4 + 4 + 56) = 4 Γ (64) = 256 Clock Cycles
Memory bandwidth = bytes/clock cycle = 32/256 = 1/8 byte/cycle (0.125)
For 4-word organization: Miss Penalty = 1 Γ (4 + 4 + 56) = 64 Clock Cycles
Memory bandwidth = 32/64 = 1/2 bytes/cycle (0.5)
π‘ Why this matters: Wider memory delivers 4Γ the bandwidth by transferring the entire block in a single access, dramatically reducing miss penalty.
2: Interleaved Memory
Memory is divided into banks based on address:
- bank 0 has all words whose Address MOD 4 = 0
- bank 1 has all words whose Address MOD 4 = 1
- bank 2 has all words whose Address MOD 4 = 2
- bank 3 has all words whose Address MOD 4 = 3
Bandwidth Calculation Example: Using the same timing model, the miss penalty for 4-word interleaved memory is: = time to send address + time to access + number of banks Γ time to send data = 4 + 56 + 4 Γ 4 = 76 clock cycles
Bandwidth = 32/76 = 0.4 byte per clock cycle
Compare to 1-word organization: Bandwidth = 32/256 = 1/8 = 0.125 byte per clock cycle
π‘ Why this matters: Interleaving achieves 3.2Γ bandwidth improvement over the 1-word organization by overlapping accesses across banks.
3: Independent Memory Banks
Memory banks offer independent accesses, allowing simultaneous operations in:
- Multiprocessors (each processor can access a different bank)
- I/O (input/output devices can use separate banks)
- CPU with Hit under n Misses (the CPU can continue processing hits while servicing a miss)
- Non-blocking Caches (cache continues to serve hits during a miss)
An input device may use one controller and one bank, the cache read may use another bank, and the cache write still another bank.
Summary: Main Memory Bandwidth
Three approaches can be combined:
- Using memory banks
- Making memory and its bus wider
- Doing both (wider memory with banks)
How many banks should there be? This decision is essential to ensure that if memory is being accessed sequentially (e.g., when processing an array), by the time you try to read a second word from a bank, the first access has finished. Otherwise, it will return to the original bank before it has the next word ready.
Example with 8 banks, each 64-bit, access time 10 clock cycles:
- Clock cycle 1: Bank 0 starts access (result ready after 10 clock cycles)
- After 10 clock cycles: Bank 0 result available
- 7 banks are accessed sequentially till the 18th clock cycle
- 18th clock: Bank 0 is accessed again
- CPU cannot start fetching immediately; wait another 10 clock cycles
- Clock cycle 20: Results ready
π Definition β Bank Rule: Number of banks β₯ Number of clock cycles to access a word in a bank
Virtual Memory
Multiple processes or a single process may exceed physical memory available. The increasing gap between CPU speed and memory speed, plus the high cost of main memory, leads to treating physical DRAM as a cache for the disk (a single level store or single level storage concept). A Virtual Memory System manages two levels of the memory hierarchy: main memory and secondary storage. Memory is divided into segments, named as a page (a fixed-size block of contiguous pages).
Attributes: β Protection β Different processes operate in different address spaces with different permissions; they cannot access privileged information. β Relocation β Simplifies loading of programs and allows placing a program anywhere in memory, managed by hardware and software.
Cache vs. Virtual Memory
- Page or segment is used for block
- Page fault or address fault is used for miss
- CPU produces virtual address
- Virtual addresses are translated to main memory or physical addresses via address translation
- Mapping of virtual address to physical address uses a page table (contains the physical address of the segment or page)
- Replacement on cache miss = page fault
- The size of processor address is independent; cache size is independent of the processor address
- Secondary storage is the lower-level backing store for main memory
- File system occupies space on secondary storage
Issues of Virtual Memory Design
- Line size: Large, since disk is better at transferring large blocks
- Associativity: High (fully associative) to minimize miss rate
- Write Strategy: Write through or write back
- Miss rate: Extremely low (<< 1%)
- Hit time: Must match cache/main memory performance
- Miss latency: Very high (~20 ms)
- Tag storage overhead: Low, relative to block size
Typical System with Virtual Memory
The CPU generates the Virtual Address. The operating system manages a lookup table that records the location of the page or segment, translating virtual addresses to physical addresses.
Page Faults (like βCache Missesβ)
A page fault indicates the virtual address is not in memory. The OS exception handler is invoked. The current process suspends while the OS fetches the page from disk. The OS has full control over placement of pages in memory.
π‘ Why this matters: A page fault costs ~20 milliseconds (millions of clock cycles) compared to a cache miss which costs tens of cycles β this huge penalty drives the design decisions for virtual memory (large pages, high associativity, low miss rates).
β Key Takeaways
Main memory bandwidth can be improved by widening the memory bus (one wide memory fetch) or by interleaving memory across banks (parallel access to multiple narrower memories). The interleaved approach offers excellent bandwidth with standard memory chips, requiring that the number of banks at least equals the access latency to avoid stalls. Virtual memory treats DRAM as a cache for the disk, using pages as blocks, with page faults triggering OS exception handling. Unlike caches, virtual memory uses very large blocks, full associativity, and extremely low miss rates because the miss penalty is approximately 20 milliseconds. Protection and relocation are fundamental attributes of virtual memory systems, enabling multiple processes to share memory safely while allowing programs to be loaded anywhere in physical memory.
π§ Quick Revision Questions
- Formula: Write the miss penalty formula for a cache using a wider main memory, and calculate the penalty when transferring a 4-word block with 4-cycle address, 4-cycle data send, and 56-cycle access per word.
- Concept: Why does interleaved memory require that the number of memory banks be at least equal to the memory access time (in clock cycles)?
- Comparison: List three key differences between cache memory and virtual memory in terms of block size, associativity, and miss penalty.
- Definition: What is a page fault, and what role does the operating system play when one occurs?
- Problem: For a system with a 4-word block, compare the miss penalty and bandwidth for (a) 1-word wide memory, (b) 4-word wide memory, and (c) 4-bank interleaved memory.
π Lecture 33 β Memory Hierarchy Design (Virtual Memory System)
π Overview: This lecture completes the memory hierarchy discussion by focusing on virtual memory systems, including address translation mechanisms, page table operations, and protection schemes for multiple processes. It explains how virtual addresses are mapped to physical addresses, the role of the Translation Lookaside Buffer (TLB) for fast translation, and how memory protection is implemented to isolate processes.
ποΈ Topics Covered
The lecture covers virtual memory address translation concepts including page table operation in three steps, address translation via page tables with a simple memory system example, fast address translation using TLB with fully associative placement, an address translation example with 64-bit virtual addresses, and virtual memory protection mechanisms using base and bound registers and protection attribute bits in PTEs.
π Lecture Summary
Recap: Main Memory and Virtual Memory Design
The lecture begins by recapping main memory organization using banks of memory arrays called DIMMs (Dual Inline Memory Modules). Three types of DRAM are reviewed: Fast Page Mode, Synchronous DRAM (SDRAM), and Double Data Rate (DDR) DRAM, with focus on latency and bandwidth as key performance metrics. The performance concern for caches is bandwidth, especially for inputs/outputs and multiprocessors, which can be improved using Wider Main Memory, Simple Interleaved Memory, and Independent Memory Banks.
Virtual memory allows multiple processes to each have a dedicated full address space. Memory is divided into fix-sized fragments (pages) or variable-sized fragments (segments). Contiguous pages in virtual memory may be physically available on the main memory in non-contiguous locations. Virtual memory provides protection (processes cannot access each other's memory) and relocation (programs can be loaded anywhere in physical memory).
Recap: Cache vs Virtual Memory
In virtual memory, a page fault or address fault occurs when a referenced page is not in main memory. The CPU produces a virtual address which must be mapped to a physical address. Replacement strategies determine which page to evict when memory is full. The size of the processor address determines the virtual address space. Secondary storage (disk) holds pages not currently in main memory.
The page replacement strategies discussed include FIFO (First-In-First-Out), LRU (Least Recently Used), and Approximation to LRU, which uses a reference bit that is periodically reset; a page with a reference bit set indicates recent use. VM write strategies may be Write Back or Write Through, but Write Through is impossible because of: too long access to disk, the write buffer limitations, and the I/O system constraints.
π‘ Why this matters: Write-back is the only practical strategy for virtual memory because writing through to disk for every write would be catastrophically slow.
Recap: Virtual Memory Operation
The CPU generates the Virtual Address, then a lookup table (page table) is consulted to find the location of the page or segment. This translates virtual addresses to physical addresses. On a page fault, the OS has full control over placement, the OS exception handler is invoked, and the current process suspends while data is brought to main memory by the OS. The contents of the page table are updated after the transfer completes.
VM Address Translation Concept
Assume Virtual Address space V comprises a set of N pages V = {0, 1, ..., Nβ1}, and Physical Address space P comprises a set of M pages P = {0, 1, ..., Mβ1} where M < N (physical memory is smaller than virtual address space).
Assuming n-bit virtual address, m-bit physical address, and p-bit page offset, the limits are:
- Virtual address limit = N = 2βΏ
- Physical address limit = M = 2α΅
- Page size (bytes) = PS = 2α΅
The virtual address is divided into page offset (lower p bits) and page number (upper n-p bits). The page number indexes into a Page Table to find the corresponding physical page frame number.
π Definition β Page Table: A data structure maintained by the OS that stores the mapping from virtual page numbers to physical page frame numbers, along with protection and status bits.
Page Table Operation: 3 Steps
The page table operates in three distinct steps:
- Translation: Use the virtual page number to index into the page table and find the corresponding page table entry (PTE)
- Computing Physical Address: Combine the physical page frame number from the PTE with the page offset from the virtual address to form the complete physical address
- Checking Protection: Verify that the access type (read, write, execute) is permitted according to the protection bits in the PTE
π Formula: Physical Address = (Physical Page Frame Number Γ Page Size) + Page Offset
Simple Memory System Example
The example uses:
- 14-bit virtual addresses (16,384 byte address space)
- 12-bit physical address (4,096 byte address space)
- Page size = 64 bytes (2βΆ, so 6-bit page offset)
π Example: With 14-bit virtual addresses and 64-byte pages:
- Virtual page number uses 14 - 6 = 8 bits (256 virtual pages)
- Page offset uses 6 bits
- With 12-bit physical addresses: physical page frame number uses 12 - 6 = 6 bits (64 physical frames)
- The page table contains 256 entries, each mapping a virtual page to a physical frame (or indicating invalid)
Fast Address Translation
The page table is large and in the main memory, causing a high miss penalty if every translation required two memory accesses: one memory access to obtain the physical address and a second to get the data. This miss penalty can be reduced using specialized hardware.
Fast Translation with a TLB
The Translation Lookaside Buffer (TLB) is a small, fast cache that stores recently used virtual-to-physical address translations. It uses fully associative placement policy and includes protection information in the TLB entries. The TLB outputs the physical address while the page offset passes through unchanged to form a full physical address.
Merits of TLB:
- Can be fully associative, set associative, or direct mapped
- Typically contains 128-256 entries
- Mid-range machines use small n-way set associative organizations for good performance with reasonable complexity
Address Translation Example
The example uses a 64-bit virtual address with:
- Physical Address: 41 bits
- TLB β direct mapped with 256 entries
- First Level Caches: direct mapped with 8KB entries, block size 64 bytes
- Second Level Cache: direct mapped 4MB direct mapped; block size 64 bytes
This example shows how virtual addresses traverse through TLB and multiple cache levels to reach main memory.
VM Protection Process
The address translation mechanism is the foundation of memory protection. Protection attribute bits in the PTE (Page Table Entry) and TLB enforce access control. If a process attempts an access it does not have permission for, an exception is raised.
The protection mechanism ensures that an address is said to be valid if Base <= address <= Bound, using Base and Bound registers to define the valid address range for a process. Multiple processes have separate page tables each pointing to distinct pages of memory, and processes are prevented from modifying these tables to maintain security.
β Key Takeaways
The critical concepts from this lecture are the three-step page table operation (translation, physical address computation, protection checking) that forms the core of virtual memory address translation. The TLB is essential for performance because it eliminates the need for two memory accesses per translation, reducing the miss penalty. Protection is built into the virtual memory system through PTE protection bits and base/bound registers, ensuring that processes cannot access each other's memory space. The mathematical relationship between virtual address bits, physical address bits, and page size determines the structure of the page table and the size of the virtual and physical address spaces. Replacement strategies, particularly the distinction between FIFO and LRU (with its approximation using reference bits), determine how virtual memory systems handle page faults efficiently.
π§ Quick Revision Questions
- What are the three steps of page table operation during virtual address translation?
- How does a TLB improve the performance of address translation compared to accessing the page table in main memory?
- In a system with 14-bit virtual addresses, 12-bit physical addresses, and 64-byte pages, how many entries does the page table have?
- Why is write-through strategy impossible for virtual memory, and what alternative is used instead?
- How do base and bound registers work together with page tables to provide memory protection for multiple processes?
π Lecture 34 β Multiprocessors (Shared Memory Architectures)
π Overview: This lecture introduces parallel processing architectures as a means to further improve computer performance beyond instruction-level parallelism. It covers the fundamental classification of parallel computers, the distinction between centralized and distributed memory architectures for MIMD machines, and explores shared address space and message passing programming models. The lecture emphasizes performance limitations like Amdahlβs Law and communication latency.
ποΈ Topics Covered
Today's topics include a recap of ILP exploitation techniques, an introduction to parallel processing and the limitations outlined by Amdahl's Law and communication costs, Flynn's taxonomy of parallel computers (SISD, SIMD, MISD, MIMD), a detailed look at MIMD classification into centralized shared-memory (SMP/UMA) and distributed memory (NUMA) architectures, the fundamental issues of naming, synchronization, and latency/bandwidth in parallel machines, and the framework for parallel processing using shared address space and message passing models, concluding with examples of each architectural type.
π Lecture Summary
Recap
So far, our focus has been on studying the performance of a single instruction stream computer and methodologies to enhance its performance. We studied how Instruction Level Parallelism (ILP) is exploited among the instructions of a stream and how control, data, and memory dependencies are resolved. These characteristics are realized through pipelining the datapath, Superscalar Architecture, Very Long Instruction Word (VLIW) Architecture, and out-of-order execution.
Parallel Processing and Parallel Architecture
Further improvements in performance may be achieved by exploiting parallelism among multiple instruction streams, which uses multithreading (number of instruction streams running on one CPU) and multiprocessing (streams running on multiple CPUs where each CPU can itself be multithreaded). While evaluating performance enhancement due to parallel processing, two important challenges are the limited parallelism available in a program and the high cost of communication. These limitations make it difficult to achieve good speedup in any parallel processor. Parallel Architecture is a collection of processing elements that cooperate and communicate to solve larger problems fast. Parallel computers extend traditional computer architecture with a communication architecture to achieve synchronization between threads and consistency of data in cache.
Parallel Computers Performance: Amdahlβs Law
If a portion of the program is sequential, it limits the speedup. Amdahl's Law quantifies this limitation.
π Example: What fraction of original computation can be sequential to achieve speedup of 80 with 100 processors?
- Given: Target Speedup = 80, Number of Processors (Speedup of parallel part) = 100.
- Formula: Speedup = 1 / [ (Fraction_parallel / Speedup_parallel) + (1 - Fraction_parallel) ]
- 80 = 1 / [ (Fraction_parallel / 100) + (1 - Fraction_parallel) ]
- Calculation:
- 0.8 * Fraction_parallel + 80 * (1 - Fraction_parallel) = 1
- 80 - 79.2 * Fraction_parallel = 1
- Fraction_parallel = (80 - 1) / 79.2 = 0.9975
- Result: To achieve a speedup of 80 with 100 processors, only 0.25% of the computation can be sequential!
Another major challenge is the communication cost involving the latency of remote access. π Example: Consider an application running on a 32-processor multiprocessor, with a 40 nsec time to handle a remote memory reference. Assume base IPC (Instructions Per Cycle) for all memory reference hits is 2 and the processor clock rate is 1GHz.
- Given: Base IPC = 2, Remote access time = 40 nsec, Clock rate = 1 GHz (1 cycle = 1 nsec). Remote access cost = 40 nsec / 1 nsec/cycle = 400 cycles.
- Formula: CPI = Base CPI + (Remote request rate x Remote access cost)
- Base CPI = 1 / Base IPC = 1 / 2 = 0.5
- Scenario 1: 0.2% of instructions involve remote access.
- CPI = 0.5 + (0.002 x 400 cycles) = 0.5 + 0.8 = 1.3
- Scenario 2: No remote references (all local).
- CPI = 0.5
- Result: The multiprocessor with all local references is 1.3 / 0.5 = 2.6 times faster than one with 0.2% remote references.
Parallel Computer Categories
In 1966, Flynn proposed a categorization based on parallelism in the instruction and data streams.
- SISD (Single Instruction Single Data): This is a Uniprocessor.
- SIMD (Single Instruction Multiple Data): The same instruction is executed by multiple processors using different data streams. It has a single instruction memory and control processor but multiple data memories. Examples: Illiac-IV and CM-2. It offers a simple programming model, low overhead, and flexibility.
- MISD (Multiple Instruction Single Data): Multiple processors or functional units work on a single data stream. No commercial multiprocessor of this type is available.
- MIMD (Multiple Instruction Multiple Data): Each processor fetches its own instructions and operates on its own data. Examples: Sun Enterprise 5000, Cray T3D, SGI Origin. Its characteristics include flexibility to function as a single-user or multi-programmed multiprocessor and the use of off-the-shelf microprocessors.
MIMD and Thread Level Parallelism
MIMD machines can be used with each processor executing different processes in a multi-program environment, or multiple processors executing a single program sharing code and address space. In the latter case, these processes are referred to as threads. Threads may be large-scale independent processes or parallel iterations of loops. This parallelism is called Thread Level Parallelism.
MIMD Classification
Based on memory organization and interconnect strategy, MIMD machines are classified as:
- Centralized Shared Memory Architecture
- Distributed Memory Architecture
Centralized Shared-Memory
In small-level designs, processor-cache subsystems share the same physical centralized memory connected by a bus. In larger designs, the single bus is replaced with multiple buses or a switch. The key architectural property is Uniform Memory Access (UMA) , meaning the access time to all memory from all processors is the same. These multiprocessors are referred to as Symmetric (Shared Memory) Multi-Processors (SMP) .
π Definition β UMA (Uniform Memory Access): A memory architecture where the access time to all memory locations is the same for all processors. π Definition β SMP (Symmetric Multi-Processor): A multiprocessor architecture based on UMA where the single main memory has a symmetric relationship to all processors.
Decentralized or Distributed Memory
This design consists of individual nodes containing a processor, some memory, I/O, and an interface to an interconnection network. It is a cost-effective way to scale memory bandwidth if most accesses are to local memory. Distributed memory provides more memory bandwidth and lower memory latency for local accesses. The disadvantage is that data communication between processors is more complex as there is no direct connection.
Parallel Architecture Issues
The fundamental issues that characterize parallel machines are: how large is the collection of processors, how powerful are they, how do they cooperate and communicate, how are data transmitted, and what is the interconnection type. These issues can be classified into three main categories:
Fundamental Issue #1: Naming
Naming deals with how to solve large problems fast, what data is shared, how it is addressed, and how processes refer to each other. The segmented shared address space is named uniformly as <process number, address>. The choice of naming affects the code produced by a compiler, data replication in cache, and the global physical and virtual address space.
Fundamental Issue #2: Synchronization Processes must coordinate. In message passing, coordination is implicit with the transmission or arrival of data. In shared address space, processes must explicitly coordinate through additional operations like writing a flag, awakening a thread, or interrupting a processor.
Fundamental Issue #3: Latency and Bandwidth
- Bandwidth: High bandwidth is needed in parallel communication, but it cannot always be scaled and must match limits in network, memory, and processor. Overhead to communicate is a problem.
- Latency: It affects performance because a processor may have to wait, and it affects ease of programming because it requires more thought to overlap communication and computation.
- Latency Hiding: As latency increases the programming system burden, mechanisms are found to help hide latency. Examples include overlapping message send with computation, prefetching data, and switching to other tasks.
π‘ Why this matters: A high-bandwidth, low-latency interconnect is critical for performance. The ability to hide this latency through techniques like prefetching is what allows many parallel programs to scale efficiently.
Framework for Parallel processing
The framework for parallel architecture is defined as a two-layer representation: Programming and Communication Models. These models present sharing of address space and message passing.
- Shared Address Space Model: At the communication layer, it defines communication via memory (load, store). At the programming layer, it defines handling several processors operating on several data sets simultaneously to exchange information globally.
- Message Passing Model: At the communication layer, it defines sending and receiving messages via library calls. At the programming layer, it provides a multiprogramming model to conduct jobs without I/O communication.
Shared Address Space Architecture (for Decentralized Memory Architecture)
Shared address space is referred to as Distributed Shared Memory (DSM) . Each processor can name every physical location, and each process can name all data it shares with other processes. Data transfer takes place via load and store. It often uses virtual memory to map to local or remote physical memory.
Programming Model: A process is defined as a virtual address space plus one or more threads of control. All threads share a process address space. Writes to the shared address space by one thread are visible to reads of all threads in other processes. Popular shared address space architectures include:
- Main Frame Computers: Motivated by multiprogramming, it uses a crossbar for processor interface to memory modules. It was initially limited by processor cost, then by the cost of the crossbar (e.g., IBM S/390).
- Minicomputers β Symmetric Multi Processors (SMP): Motivated by multiprogramming and multi-transaction processing. All components are on a shared bus (e.g., Intel Pentium Pro Quad). The bus is a bandwidth bottleneck, and caching is key to the coherence problem.
- Dance Hall: All processors are on one side of the network, all memories on the other. It offers a scalable interconnect network where bandwidth is scalable, but has larger access latency.
- Distributed Memory β Non-Uniform Multiprocessor Architecture (NUMA): A large-scale multiprocessor where memory is distributed with non-uniform access time (e.g., Cray T3E). Non-local references are accessed using communication requests generated by the memory controller, and directory-based cache-coherence protocols are used.
Message Passing Architecture
In this model, whole computers (CPU, memory, I/O devices) communicate as explicit I/O operations. It is essentially NUMA but integrated at the I/O devices vs. the memory system.
Programming Model:
Local memory is directly accessed via private address space. Communication takes place via explicit send and receive. Send specifies a local buffer and the receiving process on a remote computer. Receive specifies the sending process on a remote computer and a local buffer to place data. Send and receive is a memory-memory copy where each supplies a local address, and it performs pair-wise synchronization.
- Synchronization:
receivewaits forsendwhen the send completes, the buffer is free, and the request is accepted.
Communication Model: Communication is integrated at the I/O level, not into the memory system. It has networks of workstations (clusters) but with tighter integration. It is easier to build than scalable shared address space machines. A typical example is the IBM SP, made from RS6000 workstations with network interface integrated in the I/O bus, where bandwidth is limited by the I/O bus.
β Key Takeaways
Amdahl's Law is critical for understanding the practical limits of parallel speedup, showing that even a tiny sequential fraction drastically limits performance, especially as processor count increases. Communication cost, measured in latency for remote memory accesses, significantly impacts effective CPI and overall performance. Flynn's taxonomy (SISD, SIMD, MISD, MIMD) provides the fundamental classification for parallel architectures, with MIMD being dominant. The key architectural distinction in MIMD is memory organization: UMA (SMP) offers uniform access times but limited scaling, while NUMA (distributed memory) offers better scalability but complex data communication. The two primary programming models are shared address space, which is simpler for complex communication patterns, and message passing, which makes communication explicit and is easier for sender-initiated operations.
π§ Quick Revision Questions
- According to Amdahl's Law, what fraction of a program must be parallelizable to achieve a speedup of 10 with 20 processors? Show your calculation.
- What is the key architectural difference between a Symmetric Multi-Processor (SMP) and a Non-Uniform Memory Access (NUMA) machine?
- In Flynn's taxonomy, what is the key difference in data handling between an SIMD and an MIMD machine?
- Explain the three fundamental issues (Naming, Synchronization, Latency/Bandwidth) in the context of designing a parallel machine.
- How does the communication abstraction differ in the programming layer between a shared address space architecture and a message-passing architecture?