π Lecture 27 β Overview of todayβs lecture
π Overview: This lecture addresses critical problems in virtual memory management, particularly when the system runs out of physical memory. It explores page replacement policies (global vs. local), the devastating problem of thrashing, and two key solutions: the working set model and page fault frequency algorithm. Understanding these concepts is essential for designing efficient multitasking operating systems.
ποΈ Topics Covered
The lecture covers page replacement strategies (global vs. per-process), the phenomenon of thrashing and why it exposes the lie of virtual memory, the working set model of program behavior and its implementation, scheduling details including the balance set concept, page fault frequency as a variable-space algorithm, and fault resumption techniques that allow OS emulation and virtualization.
π Lecture Summary
Page replacement: Global or local?
So far, weβve implicitly assumed memory comes from a single global pool (βGlobal replacementβ). When process P faults and needs a page, take the oldest page on the entire system. This is good for adaptable memory sharing β for example, if P1 needs 20% of memory and P2 needs 70%, they will be happy. However, it is bad because it is too adaptable and provides little protection. A critical question arises: what happens to P1 if P2 sequentially reads an array about the size of memory? P2 will steal all of P1's pages, causing P1 to thrash.
Per-process page replacement
With per-process page replacement, each process has a separate pool of pages. A page fault in one process can only replace one of that processβs frames. This isolates the process and therefore relieves interference from other processes. However, it also isolates the process and therefore prevents the process from using otherβs (comparatively) idle resources. Efficient memory usage requires a mechanism for (slowly) changing the allocations to each pool. Key questions arise: What is βslowlyβ? How big a pool? When to migrate?
Thrashing
Thrashing is when the system spends most of its time servicing page faults, leaving little time for doing useful work. This could be because there is enough memory but a bad replacement algorithm (one incompatible with program behavior). Alternatively, it could be that memory is over-committed β too many active processes.
Thrashing: exposing the lie of VM
Thrashing occurs when processes on the system require more memory than it has. Each time one page is brought in, another page, whose contents will soon be referenced, is thrown out. Processes will spend all of their time blocked, waiting for pages to be fetched from disk. I/O devices are at 100% utilization but the system is not getting much useful work done. What we wanted was virtual memory the size of disk with the access time of physical memory. What we have is memory with access time equal to disk access.
Making the best of a bad situation
For a single process thrashing, if the process does not fit or does not reuse memory, the OS can do nothing except contain the damage. For system thrashing, if thrashing arises because of the sum of several processes, the OS must adapt: figure out how much memory each process needs, change scheduling priorities to run processes in groups whose memory needs can be satisfied (shedding load), and if new processes try to start, refuse them (admission control). A careful consideration: this is an example of technical vs social β the OS is not the only way to solve this problem. The solution could simply be to go and buy more memory.
π Definition β Thrashing: a condition where the system spends most of its time servicing page faults and little time doing useful work, often because memory is over-committed.
The working set model of program behavior
The working set of a process is used to model the dynamic locality of its memory usage. The working set is the set of pages the process currently βneedsβ, formally defined by Peter Denning in the 1960βs. A page is in the working set (WS) only if it was referenced in the last w references. Obviously, the working set (the particular pages) varies over the life of the program, as does the working set size (the number of pages in the WS).
π Definition β Working Set: the set of pages a process currently needs, defined as those pages referenced in the last w memory references.
Working set size
The working set size changes with program locality. During periods of poor locality, more pages are referenced, and within that period of time, the working set size is larger. Intuitively, the working set must be in memory, otherwise youβll experience heavy faulting (thrashing). When people ask βHow much memory does Internet Explorer need?β, really theyβre asking βwhat is IEβs average (or worst case) working set size?β
Hypothetical Working Set algorithm
The algorithm estimates memory needs for a process. Allow that process to start only if you can allocate it that many page frames. Use a local replacement algorithm (e.g. LRU Clock) to make sure that βthe right pagesβ (the working set) are occupying the processβs frames. Track each processβs working set size, and re-allocate page frames among processes dynamically. A key challenge remains: how do we choose w?
How to implement working set?
Associate an idle time with each page frame. Idle time equals the amount of CPU time received by process since last access to the page. If a pageβs idle time is greater than T, the page is not part of the working set. To calculate: scan all resident pages of a process. If the reference bit is on, clear the pageβs idle time and clear the use bit. If the reference bit is off, add the process CPU time (since last scan) to the idle time. In Unix: the scan happens every few seconds, and T is on the order of a minute or more.
Scheduling details: The balance set
If the sum of working sets of all runnable processes fits in memory, scheduling is the same as before. If they do not fit, then refuse to run some and divide into two groups: active (working set loaded) and inactive (working set intentionally not loaded). The balance set is the sum of working sets of all active processes. The long term scheduler keeps moving processes from active to inactive until the balance set is less than memory size. It must also allow inactive processes to become active (if changes too frequently?). As the working set changes, the balance set must be updated.
π Definition β Balance Set: the sum of working sets of all active (currently running) processes.
Some problems
T is magic β what if T is too small? Too large? How did we pick it? Usually by βtry and seeβ. Fortunately, systems arenβt too sensitive. What processes should be in the balance set? Large ones so that they exit faster? Small ones since more can run at once? How do we compute the working set for shared pages?
Working sets of real programs
Typical programs have phases. The concept is a good perspective on system behavior. As an optimization trick, itβs less important: early systems thrashed a lot, current systems not so much. Have OS designers gotten smarter? No. Itβs the hardware (Mooreβs law): memory is much larger (more available for processes), and less obviously, CPUs are faster so jobs exit quicker, returning memory to the free-list faster. Some apps can eat as much as you give them, but the percentage of them that have βenoughβ seems to be increasing. This was a very important OS research topic in the 80s-90s, but less so now.
Page Fault Frequency (PFF)
Page Fault Frequency (PFF) is a variable-space algorithm that uses a more ad hoc approach. It attempts to equalize the fault rate among all processes and to have a βtolerableβ system-wide fault rate. It monitors the fault rate for each process. If the fault rate is above a given threshold, give it more memory so that it faults less. If the fault rate is below the threshold, take away memory so it should fault more, allowing someone else to fault less.
Fault resumption: lets us lie about many things
Fault handling enables many powerful OS techniques. To emulate reference bits: set page permissions to βinvalidβ. On any access, a fault will occur and the handler marks the page as referenced. To emulate non-existent instructions: give the instruction an illegal opcode. When executed, it will cause an βillegal instructionβ fault. The handler checks the opcode: if it's for a fake instruction, execute it; otherwise, kill the process. To run an OS on top of another OS: make the OS into a normal process. When it does something βprivilegedβ, the real OS will get woken up with a fault. If the operation is allowed, do it; otherwise, kill. Examples include User-mode Linux and VMware.
π‘ Why this matters: Fault resumption is the fundamental mechanism that enables virtualization β allowing one operating system to run inside another, which is the basis for modern cloud computing and server consolidation.
β Key Takeaways
A student must understand the critical difference between global and local page replacement: global is adaptable but provides no protection (P2 can steal all of P1's pages), while local isolates processes but wastes idle memory. Thrashing occurs when total working sets exceed physical memory, causing the system to spend all time on page faults with I/O at 100% but no useful work. The working set model provides a theoretical framework: a page is in the working set if referenced in the last w references, and the process must have its working set in memory to avoid thrashing. The page fault frequency (PFF) algorithm provides a practical alternative by monitoring fault rates and adjusting memory allocation dynamically. Finally, fault resumption is the mechanism that enables reference bit emulation, instruction emulation, and OS virtualization.
π§ Quick Revision Questions
- What is the fundamental problem with global page replacement when two processes have significantly different memory needs?
- Explain why thrashing is described as "exposing the lie of virtual memory" β what does the system actually deliver instead of promised performance?
- In the working set model, how is "idle time" calculated for a page, and what condition determines that a page is NOT part of the working set?
- What is the balance set, and how does the long-term scheduler manage it when the sum of working sets exceeds physical memory?
- How does the Page Fault Frequency (PFF) algorithm decide whether to give a process more memory or take memory away?
π Lecture 28 β Page Fault Frequency (PFF)
π Overview: This lecture examines Page Fault Frequency (PFF) as a variable-space memory allocation algorithm that monitors and adjusts page fault rates across processes. It also covers critical virtual memory concepts including fault resumption, copy-on-write, shared memory, memory-mapped files, and the Intel P6 memory system architecture.
ποΈ Topics Covered
The lecture covers Page Fault Frequency as a variable-space algorithm that equalizes fault rates among processes, fault resumption enabling emulation of reference bits and non-existent instructions, sharing through private virtual address spaces and shared memory, copy-on-write to defer large copies, memory-mapped files for file I/O using loads and stores, the P6 memory system with its TLBs and caches, P6 2-level page table structure, and address translation using the P6 TLB.
π Lecture Summary
Page Fault Frequency (PFF)
Page Fault Frequency (PFF) is a variable-space algorithm that uses a more ad hoc approach compared to the Working Set Model. It attempts to equalize the fault rate among all processes and maintain a "tolerable" system-wide fault rate. The algorithm monitors the fault rate for each process: if the fault rate is above a given threshold, the system gives the process more memory so that it faults less; if the fault rate is below threshold, the system takes away memory so the process should fault more, allowing someone else to fault less.
π‘ Why this matters: PFF dynamically balances memory allocation based on actual process behavior rather than fixed allocations, improving overall system performance.
Fault Resumption
Fault resumption allows the operating system to "lie about many things" by leveraging the processor's fault handling mechanism. It can emulate reference bits by setting page permissions to "invalid" so that on any access a fault occurs, and the handler marks the page as referenced. It can emulate non-existent instructions by giving an instruction an illegal opcode; when executed, it causes an "illegal instruction" fault, and the handler checks the opcode β if it matches the fake instruction, it performs the operation; otherwise, it kills the process. Fault resumption also enables running an OS on top of another OS by making the OS into a normal process; when it does something "privileged," the real OS gets woken up with a fault.
π Definition β Fault Resumption: The ability to handle faults (page faults, illegal instruction faults, privilege faults) by intercepting them in a handler that performs the desired operation and then resumes execution, allowing the system to emulate hardware features or virtualize the OS.
Sharing
Private virtual address spaces protect applications from each other, but this makes it difficult to share data (requiring copying). Shared memory allows processes to share data using direct memory references, avoiding the overhead of copying data between address spaces.
Copy on Write
Operating systems spend a significant amount of time copying data. Copy on Write (CoW) defers large copies as long as possible, hoping to avoid them altogether. In this scheme, shared pages are protected as read-only in the child process. This is how fork() is implemented as Unix vfork() β the parent and child share pages until one writes, at which point a copy is made.
π Definition β Copy on Write (CoW): A memory management technique where shared pages are marked read-only; if either process tries to write, a fault occurs, and the system creates a private copy of the page for the writing process.
Mapped Files
Mapped files enable processes to do file I/O using loads and stores instead of the traditional "open, read into buffer, operate on buffer" sequence. The mmap() system call in Unix binds a file to a virtual memory region. Page Table Entries (PTEs) map virtual addresses to physical frames holding file data, so virtual address base + N refers to offset N in the file. Initially, all the pages mapped to the file are invalid.
π Definition β Memory-Mapped Files: A mechanism that maps a file's contents into a process's virtual address space, allowing file I/O through normal memory access instructions.
P6 Memory System
The Intel P6 (Pentium Pro) processor features a 32-bit address space with a 4 KB page size. It has L1, L2, and TLBs that are 4-way set associative. The instruction TLB has 32 entries across 8 sets. The data TLB has 64 entries across 16 sets. The L1 i-cache and d-cache are each 16 KB with 32 B line size and 128 sets. The L2 cache is unified (holds both instructions and data) and ranges from 128 KB to 2 MB.
P6 2-level Page Table Structure
The page directory contains 1024 4-byte page directory entries (PDEs) that point to page tables. Each page table contains 1024 4-byte page table entries (PTEs) that point to physical pages.
Translating with the P6 TLB
The translation process follows these steps: 1) Partition the Virtual Page Number (VPN) into TLB Tag (TLBT) and TLB Index (TLBI). 2) Check if the PTE for the VPN is cached in set TLBI. 3) Yes: then build the physical address from the cached PTE. 4) No: then read the PTE (and PDE if not cached) from memory and build the physical address.
β Key Takeaways
Page Fault Frequency dynamically adjusts per-process memory allocation by monitoring fault rates against thresholds, balancing memory among processes. Fault resumption enables emulation of reference bits and non-existent instructions through careful fault handling. Copy-on-Write defers expensive memory copies by sharing read-only pages until writes occur, which is fundamental to efficient process creation with vfork(). Memory-mapped files allow file I/O through memory loads/stores using mmap(). The Intel P6 uses a 2-level page table (page directory with 1024 entries pointing to page tables, each with 1024 entries) with 4 KB pages, and address translation first checks the set-associative TLBs before walking the page table hierarchy.
π§ Quick Revision Questions
- What is the key difference between PFF (variable-space) and the Working Set Model in managing process memory allocation?
- How does fault resumption enable emulation of non-existent instructions on real hardware?
- Explain the copy-on-write mechanism: when does the actual copy occur, and how is it triggered?
- In the P6 2-level page table structure, how many PDEs are in the page directory, and how many PTEs are in each page table?
- Describe the steps involved in translating a virtual address using the P6 TLB, including what happens on a TLB miss.
π Lecture 29 β File Systems
π Overview: This lecture covers the implementation of file systems as an abstraction for secondary storage, including file organization, directory structures, and disk layout strategies. It explains how operating systems manage files, directories, and protection mechanisms, and compares different allocation methods like contiguous and linked structures.
ποΈ Topics Covered
The lecture introduces the concept of files and their properties, basic file operations in Unix and NT, file access methods (sequential, direct, record, indexed), multi-level directory hierarchies with path name translation, file protection mechanisms, and disk layout strategies including contiguous allocation, linked structures, and indexed structures. It also covers the DOS file system as a practical example.
π Lecture Summary
Files
A file is a collection of data with some properties including contents, size, owner, last read/write time, protection, and others. Files serve as the fundamental unit of storage in operating systems.
Basic operations
The lecture compares basic file operations between Unix and NT (Windows NT) systems:
| Unix | NT |
|---|---|
create(name) | CreateFile(name, CREATE) |
open(name, mode) | CreateFile(name, OPEN) |
read(fd, buf, len) | ReadFile(handle, ...) |
write(fd, buf, len) | WriteFile(handle, ...) |
sync(fd) | FlushFileBuffers(handle, ...) |
seek(fd, pos) | SetFilePointer(handle, ...) |
close(fd) | CloseHandle(handle, ...) |
unlink(name) | DeleteFile(name) |
rename(old, new) | CopyFile(name) |
MoveFile(name) |
π‘ Why this matters: Understanding the mapping between Unix and NT system calls helps developers write portable code across different operating systems.
File access methods
- Sequential access: read bytes one at a time, in order
- Direct access: random access given a block/byte number
- Record access: file is array of fixed- or variable-sized records
- Indexed access: FS contains an index to a particular field of each record in a file; apps can find a file based on value in that record (similar to DB)
Directories
Most file systems support multi-level directories β naming hierarchies (e.g., /, /usr, /usr/local, /usr/local/bin). Most file systems support the notion of current directory:
- absolute names: fully-qualified starting from root of FS
bash$ cd /usr/local
- relative names: specified with respect to current directory
bash$ cd /usr/local(absolute)bash$ cd bin(relative, equivalent tocd /usr/local/bin)
Path name translation
When you want to open /one/two/three:
fd = open("/one/two/three", O_RDWR);
File protection
FS must implement some kind of protection system:
- to control who can access a file (user)
- to control how they can access it (e.g., read, write, or exec)
File System Layout
How do file systems use the disk to store files?
- File systems define a block size (e.g., 4kb)
- A "Master Block" determines location of root directory
- A free map determines which blocks are free, allocated
Disk Layout Strategies
- Contiguous allocation: Like memory β Fast, simplifies directory access, but Inflexible, causes fragmentation, needs compaction
- Linked structure: Each block points to the next
- Indexed structure: Uses indirection and hierarchy (e.g., inodes)
Simple mechanism: contiguous allocation
Files are stored as contiguous blocks on disk, similar to how memory is allocated contiguously in early systems.
Linked files
- pro: easy dynamic growth & sequential access, no fragmentation
- con: slow random access (must traverse linked list)
- Examples (sort-of): Alto, TOPS-10, DOS FAT
Example: DOS FS (simplified)
The DOS file system uses a FAT (File Allocation Table) which is essentially a linked allocation scheme stored in a table. The directory entry points to the first block, and the FAT contains pointers to subsequent blocks in the chain.
β Key Takeaways
File systems provide abstractions for secondary storage through files with defined properties and operations. Different operating systems implement similar file operations with different API calls (Unix vs NT). File access methods vary from simple sequential to complex indexed access. Multi-level directories with absolute and relative path names enable hierarchical organization. File protection controls user access and permissions. Disk layout strategies include contiguous allocation (fast but inflexible), linked structures (flexible but poor random access), and indexed structures. The DOS FAT system exemplifies linked allocation with a table-based approach.
π§ Quick Revision Questions
- What are the four file access methods described in this lecture?
- What is the difference between absolute and relative path names?
- What are the advantages and disadvantages of contiguous allocation?
- Which file system operations differ between Unix and NT naming conventions?
- What is the role of the "Master Block" and "free map" in file system layout?
π Lecture 30 β Consistency Problem
π Overview: This lecture addresses the fundamental challenge of maintaining data persistence in file systems despite system crashes. It explores Unix file system architecture including indexed allocation, i-node structures, and caching strategies, while also covering file linking mechanisms.
ποΈ Topics Covered
The lecture covers the consistency problem in file systems, indexed allocation with Unix's five-part disk structure, i-node format and block pointers (including direct, single, double, and triple indirect pointers), file buffer cache with write-behind and read-ahead techniques, and the creation of hard and soft links for file synonyms.
π Lecture Summary
Consistency problem
The Big File System Promise is persistence β it will hold your data until you explicitly delete it, and sometimes even beyond that through backup/restore. What makes this hard is crashes. If your data is in main memory, a crash destroys it. There is a performance tension: we need to cache everything for speed, but if we do, then crash equals lose everything. More fundamentally, interesting operations involve multiple block modifications, but we can only atomically modify a disk a sector at a time.
Indexed Allocation
Indexed allocation requires an index table and supports random access. It brings all pointers together into the index block.
UNIX: All disks are divided into five parts
All Unix disks have five distinct areas:
- Boot block: can boot the system by loading from this block
- Superblock: specifies boundaries of the next three areas, and contains head of freelists of i-nodes and file blocks
- i-node area: contains descriptors (i-nodes) for each file on the disk; all i-nodes are the same size; head of freelist is in the superblock
- File contents area: fixed-size blocks; head of freelist is in the superblock
- Swap area: holds processes that have been swapped out of memory
π‘ Why this matters: This five-part structure is foundational to understanding how Unix manages storage and recovers from crashes.
The βblock listβ portion of the i-node (Unix Version 7)
Each i-node contains 13 block pointers. The first 10 are "direct pointers" (pointers to 512B blocks of file data). Then there are single, double, and triple indirect pointers.
A later version of Bell Labs Unix utilized 12 direct pointers rather than 10. Berkeley Unix went to 1KB block sizes.
π Definition β Maximum file size calculation: For Berkeley Unix with 1KB blocks and 1KB pointers:
- Single indirect: 256 Γ 1KB = 256KB
- Double indirect: 256 Γ 256 Γ 1KB = 64MB
- Triple indirect: 256 Γ 256 Γ 256 Γ 1KB = 17GB
Suppose you went 4KB blocks: 1K Γ 1K Γ 1K Γ 4KB = 4TB
i-node Format
Each i-node contains:
- User number
- Group number
- Protection bits
- Times: file last read, file last written, inode last written
- File code: specifies if the i-node represents a directory, an ordinary user file, or a "special file" (typically an I/O device)
- Size: length of file in bytes
- Block list: locates contents of file (in the file contents area)
- Link count: number of directories referencing this i-node
File Buffer Cache
Applications exhibit significant locality for reading and writing files. The idea is to cache file blocks in memory to capture this locality.
Caching Writes
On a write, some applications assume that data makes it through the buffer cache and onto the disk. Several ways to compensate for this:
- "write-behind"
- Battery backed-up RAM (NVRAM)
Read Ahead
Many file systems implement "read ahead". For sequentially accessed files, this can be a big win β unless blocks for the file are scattered across the disk. File systems try to prevent that during allocation.
Creating synonyms: Hard and soft links
More than one directory entry can refer to a given file. Unix stores a count of pointers ("hard links") to the inode. To make: ln foo bar creates a synonym ('bar') for 'foo'.
Soft links: also point to a file (or directory), but the object can be deleted from underneath it (or never even exist).
β Key Takeaways
The lecture's core message is that file system persistence requires careful handling of crashes through structured storage allocation (the five-part Unix disk layout), while i-nodes with direct and indirect pointers enable efficient random access to files of varying sizes. The file buffer cache with write-behind and read-ahead improves performance but requires NVRAM or other mechanisms to ensure data integrity during crashes. Hard links share the same inode and persist as long as any link exists, while soft links are independent pointers that can become dangling. Understanding the maximum file size calculations (17GB for 1KB blocks, 4TB for 4KB blocks) is critical for system design decisions.
π§ Quick Revision Questions
- What are the five parts of a Unix disk and what is the purpose of each?
- How does the i-node block pointer scheme (10 direct + single/double/triple indirect) allow Unix to support very large files?
- What is the maximum file size for Berkeley Unix with 1KB block sizes?
- What is the difference between a hard link and a soft link in Unix file systems?
- How do write-behind and NVRAM help with the consistency problem when caching writes?
π Lecture 31 β Consistency problem
π Overview: This lecture addresses the fundamental challenge of maintaining data persistence in file systems despite system crashes. It explores three main approaches to handling crashes, with particular focus on building atomic disk operations from smaller atomic units and the use of state duplication for recovery.
ποΈ Topics Covered
The lecture covers the Big File System Promise of persistence and the problem of crashes, three main approaches to handling consistency (restart, atomic updates, reconstruction), building arbitrary-sized atomic disk operations using the SABRE approach, crash recovery through state duplication, and Unix file system invariants that can be violated by crashes.
π Lecture Summary
Consistency problem
The Big File System Promise is persistence β it will hold your data until you explicitly delete it, and sometimes even beyond that through backup/restore. What's hard about this is crashes. If your data is in main memory, a crash destroys it. There's a performance tension: you need to cache everything, but if you do, crash means lose everything. More fundamentally, interesting operations require multiple block modifications, but you can only atomically modify disk a sector at a time.
π‘ Why this matters: The basic mismatch between what file systems need to do (update multiple blocks atomically) and what disks provide (single sector atomic writes) creates the core consistency problem.
Three main approaches to handling crashes:
- Solution 1: Throw everything away and start over β Done for most things (e.g., interrupted compiles), but probably not what you want to happen to your email
- Solution 2: Make updates seem indivisible (atomic) β Build arbitrary sized atomic units from smaller atomic ones (e.g., a sector write), similar to how we built critical sections from locks, and locks from atomic instructions
- Solution 3: Reconstruction β Try to fix things after crash (many file systems do this β "fsck"). Usually do changes in stylized way so that if crash happens, can look at entire state and figure out where you left off
Arbitrary-sized atomic disk ops
For disk, construct a pair of operations:
- put(blk, address): writes data in blk on disk at address
- get(address) -> blk: returns blk at given disk address
Such that "put" appears to place data on disk in its entirety or not at all, and "get" returns the latest version. What we have to guard against: a system crash during a call to "put", which results in a partial write.
SABRE atomic disk operations
The SABRE approach uses state duplication β maintaining two copies of state information. The atomic-put operation:
void atomic-put(data)
version++; # unique integer
put(version, V1);
put(data, D1);
put(version, V2);
put(data, D2);
The atomic-get operation:
blk atomic-get()
V1data := get(V1);
D1data := get(D1);
V2data := get(V2);
D2data := get(D2);
if(V1 == V2)
return D1data;
else
return D2data;
Does it work?
Assume we have correctly written to disk: { #2, "seat 25", #2, "seat 25" } and want to change seat 25 to seat 31. The system crashes during atomic-put("seat 31"). There are 6 cases depending on where failure occurs:
| Where put fails | Possible disk contents | atomic-get returns? |
|---|---|---|
| before | {#2, "seat 25", #2, "seat 25"} | Seat 25 |
| the first | {#2.5, "seat 25", #2, "seat 25"} | Seat 25 |
| the second | {#3, "seat 35", #2, "seat 25"} | Seat 25 |
| the third | {#3, "seat 31", #2.5, "seat 25"} | Seat 25 |
| the fourth | {#3, "seat 31", #3, "seat 35"} | Seat 31 |
| after | {#3, "seat 31", #3, "seat 31"} | Seat 31 |
Two assumptions:
- Once data written, the disk returns it correctly
- Disk is in a correct state when atomic-put starts
Recovery
The recovery function uses the duplicated state to restore consistency:
void recover(void) {
V1data = get(V1); # following 4 ops same as in a-get
D1data = get(D1);
V2data = get(V2);
D2data = get(D2);
if (V1data == V2data)
if(D1data != D2data)
# if we crash & corrupt D2, will get here again.
put(D1data, D2);
else
# if we crash and corrupt D1, will get back here
put(D2data, D1);
# if we crash and corrupt V1, will get back here
put(V2data, V1);
The power of state duplication
Most approaches to tolerating failure have at their core a similar notion of state duplication:
- Want a reliable tire? Have a spare.
- Want a reliable disk? Keep a tape backup (not in same building).
- Want a reliable server? Have two with identical copies; primary fails? Switch.
Fighting failure
In general, coping with failure consists of first defining a failure model composed of:
- Acceptable failures: E.g., earth destroyed by aliens from Mars β loss of file viewed as unavoidable
- Unacceptable failures: E.g., power outage β lost file not ok
Unix file system invariants
These are properties that must hold in a consistent file system:
- File and directory names are unique
- All free objects are on free list, and free list only holds free objects
- Data blocks have exactly one pointer to them
- Inode's ref count = the number of pointers to it
- All objects are initialized (new file should have no data blocks, just allocated block should contain all zeros)
A crash can violate every one of these invariants.
Unused resources marked as "allocated"
Rule: Never persistently record a pointer to any object still on the free list
The dual of allocation is deallocation. The problem happens there as well. With Truncate:
- Set pointer to block to 0
- Put block on free list
If the writes for steps 1 and 2 get reversed, can falsely think something is freed.
Dual rule: Never reuse a resource before persistently nullifying all pointers to it.
β Key Takeaways
The fundamental challenge in file system consistency is that interesting operations require multiple block modifications, but disks only provide atomic writes at the sector level. The SABRE approach solves this through state duplication β maintaining two copies of version numbers and data so that atomic-get can always return a consistent previous or new state regardless of when a crash occurs during atomic-put. Crash recovery leverages this same duplication to restore consistency. Unix file system invariants (unique names, correct free lists, proper reference counts) can all be violated by crashes, so rules about never pointing to freed objects and never reusing resources before nullifying pointers are essential for maintaining consistency.
π§ Quick Revision Questions
- What are the three main approaches to handling crashes in file systems?
- In the SABRE atomic disk operations, why is it necessary to store two copies of the version number and data?
- When a crash occurs during atomic-put, what determines whether atomic-get returns the old data or the new data?
- What is the dual rule for deallocation that corresponds to "never persistently record a pointer to any object still on the free list"?
- What five invariants must a consistent Unix file system maintain, and how can a crash violate them?
π Lecture 32 β Reactive: reconstruct freelist on crash
π Overview: This lecture focuses on crash recovery and consistency mechanisms in file systems. It covers both reactive approaches (like mark-and-sweep garbage collection) and proactive rules for maintaining file system integrity, along with the detailed process of fsck (file system check).
ποΈ Topics Covered
The lecture covers reactive reconstruction of freelists using mark-and-sweep garbage collection, file deletion with reference count management, issues with bogus reference counts, file creation and growth procedures, conservative file moving techniques, the two fixable cases for file system corruption, fsck reconstruction algorithm, and write ordering with dependencies.
π Lecture Summary
Reactive: reconstruct freelist on crash
When a crash occurs, the free list may become corrupted. A reactive approach uses mark-and-sweep garbage collection to reconstruct it. Starting from the root directory, the system recursively traverses all live objects and removes them from the free list.
π Definition β Mark and sweep: A garbage collection algorithm that marks all live objects by traversing from known roots, then sweeps by reclaiming unmarked objects.
π Example: If the free list incorrectly marks an allocated inode as free, the mark-and-sweep traversal will discover the inode by following pointers from the root directory and remove it from the free list.
π‘ Why this matters: This approach fixes the case where allocated objects are mistakenly marked as free, but it's expensive because it requires traversing all live objects, making reboot slow.
Deleting a file
The unlink operation removes a file by name. The system traverses the current working directory looking for the file name; if not found or permissions are wrong, it returns an error. It then clears the directory entry, decrements the inode's reference count, and if the count reaches zero, frees the inode and all blocks it points to.
π Formula: Unlink process β (1) traverse directory, (2) clear entry, (3) decrement refcount, (4) if refcount=0, free inode and blocks
π Example: Calling unlink("foo") in a directory that contains entry ("foo", inode 41) with refcount 1 will: clear the entry, set refcount to 0, then free inode 41 and all its data blocks.
Bogus reference count
A reference count too high means the inode and its blocks will never be reclaimed (e.g., a 2GB file becomes permanently allocated). A reference count too low is dangerous because blocks will be marked free while still in use, creating a major security hole (e.g., password file stored in "freed" blocks).
π Definition β Reference count: A counter indicating how many directory entries point to a given inode.
π Example: If reference count for an open file is 2 but should be 1, the file remains allocated after unlink, wasting space. If count is 0 when it should be 1, blocks may be reallocated while the file still uses them.
The proactive solution: Never decrement reference counter before removing the pointer to an object. Do synchronous writes. The reactive solution: Fix with mark and sweep.
Creating a new file
The golden rule: never (persistently) point to a resource before it has been initialized. This means file creation requires 2 or 3 synchronous writes.
π Example: Creating a file requires: Write 1 β write out the modified free list to disk and wait; Write 2 β write out zeros to initialize the inode and wait; Write 3 β write out directory block containing pointer to the inode.
Growing a file
The write(fd, &c, 1) operation translates current file position (byte offset) into location in inode (or indirect block). If the inode already points to a block, modify and write back. Otherwise: allocate a free block, write modified free list to disk and wait, write the newly allocated block to disk and wait, write the pointer (inode) to the new block to disk and wait.
π Example: Appending 1 byte to a 4096-byte file (block size 4096) where the file currently has one full block: allocate block #500, write free list (block #500 removed), write byte to block #500, update inode to point to block #500 β each step requiring synchronous write.
Conservatively moving a file
The rule: never reset an old pointer to an object before a new pointer has been set. Moving foo to bar (where foo β inode #41) requires: increment inode 41's reference count and write inode to disk; insert ("bar", inode 41) and write bar's directory block; destroy ("foo", inode 41) and write foo's directory block; decrement inode 41's reference count and write inode to disk.
π Example: mv foo bar where foo points to inode 41 with refcount 1: After step 0, refcount=2; after step 2, refcount=2 with two directory entries; after step 3, refcount=1 with only "bar" pointing to inode 41. Cost: 3 synchronous writes.
Summary: the two fixable cases
Case 1: Free list holds pointer to allocated block. Cause: crash during allocation or deallocation. Rule: make free list conservative β free by nullifying pointer before putting on free list; allocate by removing from free list before adding pointer.
Case 2: Wrong reference count. Too high = lost memory (safe); Too low = reuse object still in use (very unsafe). Cause: crash while forming or removing a link. Rule: conservatively set reference count to be high β unlink by nullifying pointer before reference count decrement; link by incrementing reference count before adding pointer.
Alternative: Ignore all rules and fix on reboot using fsck.
FSCK: Reconstructing File System
The fsck algorithm performs mark and sweep and fixes reference counts. It maintains a worklist starting from the root directory, traverses all reachable objects, marks them as allocated, and counts references to each object. After traversal, it compares the actual reference count (seen) with the stored reference count (refs) and fixes any discrepancies.
π Algorithm:
worklist := { root directory }
while e := pop(worklist) # sweep down from roots
foreach pointer p in e
if p.type != dataBlock and !seen{p}
push(worklist, p)
refs{p} = p.refcnt # p's notion of pointers to it
seen{p} += 1 # count references to p
freelist[p] = ALLOCATED # mark not free
foreach e in refs # fix reference counts
if(seen{e} != refs{e})
e.refcnt = seen{e}
e.dirty = true
Write ordering
Synchronous writes are expensive. Solution: have the buffer cache provide ordering support. Whenever block "a" must be written before block "b", insert a dependency. Before writing any block, check for dependencies. To eliminate dependency, synchronously write out each block in chain until done.
π Example: If block B and C have no dependencies, they can be written immediately. Block A requires block B to be synchronously written first. This creates a chain: write B, then A, then C.
β Key Takeaways
The most critical concepts from this lecture are: (1) File system consistency can be maintained either proactively through careful write ordering and conservative pointer manipulation, or reactively through mark-and-sweep garbage collection and fsck. (2) The two primary fixable corruption cases are incorrect freelist entries and wrong reference counts, each with specific rules to prevent them. (3) Synchronous writes ensure consistency but are expensive, leading to write ordering techniques as an optimization. (4) Reference counts must be incremented before adding pointers and decremented after removing pointers to prevent dangling references or memory leaks. (5) Fsck provides a comprehensive recovery mechanism by traversing the entire file system from roots, fixing both freelist and reference count issues.
π§ Quick Revision Questions
-
What are the two types of file system corruption cases that are considered "fixable" and what causes each?
-
Why is a reference count that is too low more dangerous than one that is too high?
-
How many synchronous writes are required for creating a new file, and what does each write accomplish?
-
What is the basic algorithm used by fsck to reconstruct a consistent file system state after a crash?
-
How does write ordering with dependencies reduce the cost of maintaining file system consistency compared to synchronous writes?