π Lecture 39 β Input Output Systems (Bus Structures Connecting I/O Devices)
π Overview: This lecture explores the critical role of bus structures in connecting I/O devices to computer systems. It explains the trade-offs between different bus designs, the protocols governing bus transactions and arbitration, and how these choices impact overall system performance. Understanding bus architectures is essential for designing efficient I/O subsystems that do not become performance bottlenecks.
ποΈ Topics Covered
The lecture begins with a recap of I/O system performance, including the producer-server model and metrics like response time and throughput, illustrated with a comparison of flash memory vs. disk. It then covers I/O interconnect trends and the fundamental concept of bus-based interconnects, including their advantages and disadvantages. The bulk of the lecture details bus transactions (read/write), transition protocols (synchronous and asynchronous), and bus arbitration protocols (daisy chain, centralized parallel, and distributed). It concludes with bus design decisions, the SCSI standard, and a survey of historical bus standards.
π Lecture Summary
Recap: I/O System
The overall performance of a computer is measured by its throughput, which is strongly influenced by the I/O system. Amdahl's Law highlights that system speed-up is limited by the slowest component, making I/O a critical factor. An I/O system comprises storage I/Os (secondary and tertiary storage) and communication I/Os (the I/O bus system interconnecting the microprocessor, memory, and I/O devices). Key performance parameters for I/O are diversity, capacity, latency, and bandwidth. The I/O system operates on a producer-server model, which includes a queue where tasks accumulate while waiting to be serviced. The metrics for disk I/O performance are Response Time (Queue time + Device Service time) and Throughput (percentage of total bandwidth).
π‘ Why this matters: This recap establishes that neglecting I/O performance can severely limit overall system speed, and it introduces the fundamental models and metrics used to analyze and compare I/O systems.
π Example: Comparing the time to read and write a 64Kbyte block to flash memory and disk.
- Flash memory: read 1 byte in 65 ns, write 1 byte in 1.5 ΞΌsec, erase 4KB in 5 msec.
- Disk Storage: average seek time = 4.0 msec, average rotational delay = 8.3 msec, transfer time = 4.2 MB/sec, controller overhead = 0.1 msec.
- Average read/write time for disk: 4.0 ms + 8.3 ms + 64KB/4.2 MB/sec + 0.1 ms = 27.3 msec.
- Read time for flash: 64KB / 1B/65ns = 4.3 ms. Flash is about 6 times faster than the disk for reading.
- Write time for flash: (64KB/4KB/5ms) + (64KB/1B/1.5ΞΌs) = 80 ms + 96 ms = 178.3 ms. The disk is about 6 times faster than the flash for writing.
Interconnect Trends
The I/O interconnect is the glue that interfaces computer system components. It is facilitated using high-speed hardware interfaces and logical protocols. Based on distance, bandwidth, latency, and reliability, interconnects are classified as backplanes, channels, and networks.
Bus-Based Interconnect
Communication on different interconnects is done via buses, which are shared communication links between subsystems.
- Advantages: Low cost (a single set of wires is shared) and versatility (easy to add new devices).
- Disadvantages: Creates a communication bottleneck, limiting maximum I/O throughput, especially in server systems where designing a bus to meet processor demand is a challenge.
- Limitations: Bus speed is limited by physical factors like bus length and bus loading (number of connected devices).
- Classification:
- I/O busses: Long, connect many types of devices, offer a wide range of bandwidth, and follow a bus standard (also called a channel).
- CPUβmemory buses: High speed, matched to the memory system, and connect to a single device (also called a backplane).
Bus Transactions
Bus transactions are defined with reference to memory (memory read or memory write). A transaction includes two parts: sending the address and receiving the data.
- Read Transaction: The address and read signal are sent to the memory. The memory responds by sending the data and de-asserting the wait signal.
- Write Transaction: The address and data are sent to the memory with the write signal. The memory stores the data and de-asserts the wait signal.
Bus Transition Protocols
Bus transition or bus communication protocols specify the sequence of events and timing for information transfer.
- Synchronous Bus Transfers: Follow a sequence of operations relative to a common clock.
- Asynchronous Bus Transfers: Not clocked; uses control lines (req, ack) for handshaking among devices.
Synchronous Bus Protocols: The address is transmitted in the 1st clock, using control lines to indicate the request type. The read begins when NOT READ is asserted. The data is not ready until the wait signal is reasserted.
Asynchronous Handshake: The bus is self-timed. The protocol for a write transaction is:
- t0: Master asserts address, direction, data; waits for slaves to decode.
- t1: Master asserts the request line.
- t2: Slave asserts ack, indicating data received.
- t3: Master releases req.
- t4: Slave releases ack.
For a read transaction:
- t0: Master asserts address, direction, data; waits for slaves to decode.
- t1: Master asserts request line.
- t2: Slave asserts ack, indicating ready to transmit data.
- t3: Master releases req, data received.
- t4: Slave releases ack.
Bus Arbitration Protocols
To manage multiple devices needing bus access, bus masters are introduced. A Bus Master can control bus requests and initiate a transaction. A Bus Slave is activated by the master. The protocol to manage transactions by more than one master is the Bus Arbitration Protocol. It provides a mechanism for arbitrating access to the bus so it is used cooperatively. The arbitration schemes balance two factors: bus priority (highest priority device serviced first) and fairness (every device wanting the bus is eventually guaranteed access).
Bus Arbitration Schemes:
- Daisy Chain Arbitration: The bus-grant line runs from highest to lowest priority device. A device intercepts the grant signal, preventing lower-priority devices from seeing it if it wants the bus. The sequence is: request, wait for grant, intercept grant, use bus, signal release.
- Centralized Parallel Arbitration: Uses multiple request lines. Devices independently request the bus, and a centralized arbiter chooses which device becomes the bus master.
- Distributed Arbitration:
- Self-selection: Each device wanting access places its identity code on the bus. Devices then examine the code to determine the highest-priority requester.
- Collision Detection: Devices independently request the bus. Multiple simultaneous requests result in a collision, and a device is selected based on priority.
Bus Options: Design Decisions
The design of a bus system depends on:
- Bus Bandwidth: High performance uses separate address & data lines; low cost uses multiplexed lines.
- Data width: High performance uses wider (e.g., 64-bit) data buses; low cost uses narrower (e.g., 8-bit) ones.
- Transfer size: High performance uses multiple-word transfers for less overhead; low cost uses single-word transfers for simplicity.
- Bus masters: High performance uses multiple masters (requires arbitration); low cost uses a single master.
- Split transaction: High performance uses separate request and reply packets (needs multiple masters); low-cost has a continuous connection.
- Clocking: High performance uses synchronous; low cost uses asynchronous.
Synchronous Bus Protocols β Multiple Masters
With multiple masters, the bus can offer higher bandwidth using split transaction or pipelined buses. In this technique, bus events are divided into requests and replies, making the bus available for other masters while memory reads the requested data.
Bus Standards
The SCSI (Small Computer System Interface) is a standard bus.
- Clock rate: 5 MHz, 10 MHz (fast), 20 MHz (ultra).
- Width: 8 or 16 bits.
- Devices can be slave (target) or master (initiator).
- SCSI protocol phases:
- Bus Free: No device is accessing the bus.
- Arbitration: Devices may request the bus; fixed priority by address.
- Selection: Informs the target it will participate.
- Command: The initiator reads commands from host memory and sends them to the target.
- Data Transfer: Data in or out, initiator to target.
- Message Phase: Message in or out, initiator to target.
- Status Phase: Target sends status, just before command complete.
The lecture also notes a 1993 I/O Bus Survey and a 1993 MP Server Memory Bus Survey, showing the characteristics of historical standards like SCSI, PCI, and various memory buses.
β Key Takeaways
The most critical takeaway is that the bus is the central communication link for I/O, and its design is a fundamental trade-off between performance and cost. Key design decisions like bus width, transfer size, and clocking (synchronous vs. asynchronous) directly impact throughput and latency. To manage multiple devices, understanding bus arbitration schemesβdaisy chain, centralized parallel, and distributedβis essential, as they determine how bus access is prioritized and shared, balancing priority with fairness. Finally, protocols like the synchronous and asynchronous handshake define the precise sequence of events for read and write transactions, while split transactions allow for higher bandwidth by not holding the bus for the entire transaction.
π§ Quick Revision Questions
- What are the two main components of a bus transaction, and how do they differ for a read operation vs. a write operation?
- Explain the primary difference between synchronous and asynchronous bus protocols, and give one advantage of each.
- Describe the daisy chain arbitration scheme, including how priority is established and how a device signals it wants the bus.
- List three key design decisions for a bus and state the typical choice for a high-performance system versus a low-cost system for each.
- In the context of SCSI, what are the roles of an "initiator" and a "target," and what occurs during the "Arbitration" phase?
π Lecture 40 β Input Output Systems (RAID and I/O System Design)
π Overview: This lecture covers the evaluation and improvement of storage I/O system performance through reliability, availability, and dependability metrics. It introduces RAID (Redundant Array of Inexpensive Disks) levels as a solution to improve both availability and performance of storage systems, explaining the trade-offs between redundancy, capacity, and performance for each RAID level.
ποΈ Topics Covered
The lecture begins with a recap of I/O device performance, interconnects, and bus arbitration protocols. It then introduces storage I/O performance evaluation through reliability, availability, and dependability metrics. The core content covers RAID architectures including RAID 0, RAID 1, RAID 3, RAID 4, and RAID 5, with detailed comparisons of their fault tolerance, capacity overhead, and performance characteristics for small writes and reads.
π Lecture Summary
Recap: I/O deviceβs performance
Last time we compared the performance of disk storage and flash memory. We noticed that flash is six times faster than the disk for read and the disk is six times faster than the flash for data write. Then we discussed the trends in I/O interconnects as: the networks, channels and backplanes. The networks offer message-based narrow-pathway for distributed processors over long distance.
Recap: I/O Interconnects
The backplanes offer memory-mapped wide pathway for centralized processing over short distance. The interconnects are implemented via buses. The buses are classified in two major categories as the I/O bus and CPU-Memory bus. The channels are implemented using I/O buses and backplanes using CPU-Memory buses.
Recap: I/O buses
Then we discussed the bus transition protocols which specify the sequence of events and timing requirements in transferring information as synchronous or asynchronous communication. We also discussed bus arbitration protocols β the protocols to reserve the bus by a device that wishes to communicate when multiple devices need the bus access. Here, we noticed that the bus arbitration schemes usually try to balance two factors.
Recap: I/O System
- Bus-priority: the device with highest priority should be serviced first
- Fairness: every device that wants to use the bus is guaranteed to get the bus eventually
- The three bus arbitration schemes are: β Daisy Chain Arbitration β Centralized Parallel Arbitration β Distributed Arbitration
Storage I/O Performance
Now having discussed the basic types of storage devices and the ways to interconnect them to the CPU, we are going to look into the ways to evaluate the performance of storage I/O systems. We know that if a storage device crashes then the prime objective of a storage device should be to remember the original information to make the storage device reliable.
Reliability Improvement
The reliability of a system can be improved by using the following four methods:
- Fault Avoidance β prevent fault occurrence by construction
- Fault Tolerance β providing service complying with the service specification by redundancy
- Error Removal β minimizing the presence of errors by verification
- Error Forecasting β to estimate the presence, creation and consequence of errors by evaluation
Reliability, availability and dependability
The performance of storage I/Os is measured in terms of its reliability, availability and dependability. These terminologies have been defined by Laprie in the paper entitled 'Dependable Computing and Fault Tolerance: Concepts and Terminology', published in the Digest of papers of 15th Annual Symposium on Fault Tolerant Computing (1985).
Dependability
Laprie defined dependability as the quality of delivered service such that reliance can justifiably be placed on this service. The service delivered by a system is its observed actual behavior and the system failure occurs when actual behavior deviates from the specified behavior. A user perceives a system alternating between two states of delivered service:
- Service Accomplishment β service is delivered as specified
- Service Interruption β delivered service is different from the specified service
Quantifying the transitions between service accomplishment and service interruption is the measure of the dependability. The dependability is measured in terms of:
- module reliability, which is the measure of the continuous service accomplishment
- module availability, which is the measure of the swinging between the accomplishment and interruption states of delivered service
π‘ Why this matters: Dependability is the overarching concept that combines reliability and availability. Understanding these terms precisely is critical for designing fault-tolerant storage systems.
Measuring Reliability
The reliability of a module is the measure of the time to failure from a reference initial instant. The Mean Time To Failure (MTTF) of a storage module, a disk, is the measure of reliability. The reciprocal of the MTTF is the rate of failure. The service interruption is measured as the Mean Time To Repair (MTTR).
π Definition β MTTF: Mean Time To Failure β the measure of reliability, representing the average time a device operates before failing. π Definition β MTTR: Mean Time To Repair β the measure of service interruption, representing the average time to repair a failed device. π Formula: Failure Rate = 1 / MTTF π Formula: System Failure Rate = Sum of individual component failure rates
π Example: Consider a disk subsystem comprising the following components:
- 10 disks, each with MTTF = 1,000,000 Hrs
- 1 SCSI controller with MTTF = 500,000 Hrs
- 1 SCSI cable with MTTF = 1,000,000 Hrs
- 1 power supply with MTTF = 200,000 Hrs
- 1 fan with MTTF = 200,000 Hrs
Solution: System Failure Rate = 10(1/1,000,000) + 1/500,000 + 1/1,000,000 + 1/200,000 + 1/200,000 = 23/1,000,000 Hrs System MTTF = 1/Failure Rate = 1,000,000/23 = 43,500 Hrs = 5 years
Availability
The availability of a module is the measure of the service accomplishment with respect to the swinging between the two states of accomplishment and interruption. The module availability is quantified as the ratio of the MTTF and Mean Time Between Failure β MTBF (which is equal to the sum of MTTF and MTTR).
π Formula: Availability = MTTF / (MTTF + MTTR) = MTTF / MTBF
Network Attached Storages and Reliability
A network provides well-defined physical and logical interfaces to interconnect separate CPU and storage systems at long distances. The networks are capable of sustaining high bandwidth transfer and their file-server Operating system supports remote file access. Hence, the network attached storages are more vulnerable to reliability issues and their dependability requirement is very high.
To improve both the availability and performance of storage systems, disk arrays are introduced, which contain many low-cost disks. The throughput of disk arrays is improved by having high bandwidth disk systems which employ many small disk drives. The throughput is increased by having many small arms on small (3.00" β 1.8") disk drives rather than one long arm on a larger disk (14" β 24").
π Definition β Disk Array: A storage system containing many low-cost disks to improve both availability and performance.
Array Reliability: Example
π Example: Reliability of N disks = Reliability of 1 Disk Γ· N Disk system MTTF = 50,000 Hours Γ· 70 disks = 700 hours (drops from 6 years to 1 month!)
Arrays without redundancy are too unreliable to be useful. However, the dependability can be improved by adding redundant disks to the array to tolerate faults.
Redundant Arrays of Disks
In a disk array, files are "striped" across multiple spindles. Adding a redundant disk to achieve high fault tolerance yields high data availability. If a disk fails, the contents are reconstructed from data redundantly stored in the array. The drawbacks of redundant disks are:
- Capacity penalty to store it
- Bandwidth penalty to update
These systems are known as RAID: Redundant Array of Inexpensive Disks or Redundant Array of Independent Disks. There exist several different approaches to include redundant disks in the disk array, classified by a numerical value which identifies the RAID level.
RAID Level Characteristics (for 8 user data disks):
| RAID Level | No. of disk faults survived | Corresponding check disks |
|---|---|---|
| 0. No Redundancy | 0 | 0 |
| 1. Mirrored | 1 | 8 |
| 2. Memory-Style ECC | 1 | 4 |
| 3. Bit Interleaved Parity | 1 | 1 |
| 4. Block Interleaved Parity | 1 | 1 |
| 5. Block interleaved distributed parity | 1 | 1 |
| 7. P+Q Redundancy | 2 | 2 |
RAID 0 β Non Redundant Striped
RAID 0 is the disk array without any redundant disk. The data is stripped across a set of disks, making the collection appear to the software as a single large disk. Note that the taxonomy RAID 0 is a misnomer as there is no redundant disk, but it is still referred to as RAID due to data striping.
RAID 1: Disk Mirroring/Shadowing
Each disk is fully duplicated onto its "shadow". Targeted for high I/O rate. Whenever data are written to one disk, those data are also written to the redundant disk. If a disk fails, the system goes to the mirror, so there are 8 survivals in this example (provided one disk of mirrored pair fails). It is the most expensive solution: 100% capacity overhead. One logical write = two physical writes.
There are two ways to stripe data with RAID 1:
- RAID 1+0: Create 4 pairs of disks, each organized as RAID 1, then strip data across the 4 RAID pairs
- RAID 0+1: Create two sets of 4-disks, each organized as RAID 0, and mirror write to both RAID 0
RAID 3: Bit-Interleaved Parity Disk
Rather than having a complete copy of the original disk, RAID 3 achieves desired dependability by adding enough redundant information to restore the lost information on failure. RAID 3 uses one extra disk, called Parity disk, that holds the check information in case of failure. RAID 3 acts logically as a single high-capacity, high-transfer-rate disk. The arms are synchronized logically and spindles rotationally.
Every read or write access goes to all the disks. For every read access, the parity is computed across the recovery group to protect against hard disk failures. For the RAID 3 shown, there is 33% capacity cost for parity. Wider arrays reduce capacity costs but decrease expected availability and increase reconstruction time.
RAID 4: Block-Interleaved Parity and RAID 5: Distributed Block-Interleaved Parity
Both RAID 4 and RAID 5 use the same ratio of data disk to parity disk as RAID 3, but they access data differently. In the Block-Interleaved Parity RAID 4, the parity disk is associated with each data block, identical to RAID 3. It supports a mixture of small reads, small writes, large reads, and large writes. However, a drawback is that the parity disk must be updated on every write, which is a bottleneck for back-to-back writes.
This bottleneck is resolved in Block interleaved parity RAID 5, where the parity disk is distributed among the blocks. The parity associated with each row of the data block is no longer restricted to a single disk. This allows multiple writes to occur simultaneously as long as the stripe-units are not located in the same disk.
π Example β Parallel writes in RAID 5:
- 1st write to block 8 must also access its parity block P2 (two reads from two disks β the 1st and 3rd disks)
- 2nd write to block 5 implies an update in P1 (two reads from two disks β the 2nd and 4th disks)
- Thus, the two writes could occur at the same time in parallel
In RAID 4, both P1 and P2 are on the same disk (5th disk), so it would be a bottleneck and could not be written simultaneously.
RAID 3 vs. RAID 4 and RAID 5
In RAID 3, every access goes to all the disks, while levels 4 and 5 use smaller accesses which allow independent access to occur in parallel. In RAID 4 and RAID 5, error detection information in each sector is checked independently for 'small reads' to see if the data are correct in one sector.
For small writes, RAID 3 reads blocks D1, D2, and D3 before adding Block D0' to calculate the new parity P'. Note that the new data D0 comes directly from CPU, so disks are not involved in reading it.
For small writes in RAID 4/5, the old value of D0 is read (1: Read) and compared with new value D0' to see which bit will change. Once checked, the old parity P is read and corresponding bits are changed to form P' using logical EX-ORs.
π Example β RAID 4/5 Small Write Efficiency:
- RAID 3: 3 disk reads (D1, D2, D3) and 2 disk writes (D0', P') involving all disks
- RAID 4/5: 2 disk reads (D0, P) and 2 disk writes (D0', P'), each involving just 2 disks
- One (1) Logical Write in RAID 4 and RAID 5 is equivalent to 2 Physical Reads and 2 Physical Writes
β Key Takeaways
The most critical concept is the trade-off between reliability and redundancy in storage systems β adding more disks increases throughput but decreases reliability, requiring redundant disks to restore dependability. Understanding the distinction between MTTF (reliability measure), MTTR (repair measure), and availability (MTTF/MTBF) is essential for quantifying storage system performance. RAID levels represent different approaches to balancing capacity overhead, fault tolerance, and performance β RAID 1 offers 100% overhead with mirroring, while RAID 3/4/5 use parity disks with lower overhead but different write performance characteristics. The key difference between RAID 4 and RAID 5 is that RAID 5 distributes parity across disks to eliminate the parity disk bottleneck, enabling parallel writes. For exams, remember the system reliability calculation formula (sum of failure rates = 1/MTTF), the relationship between disk count and reliability, and the specific read/write characteristics of each RAID level.
π§ Quick Revision Questions
- A disk subsystem has 10 disks (MTTF = 500,000 Hrs each), 1 controller (MTTF = 250,000 Hrs), and 1 power supply (MTTF = 100,000 Hrs). Calculate the system MTTF.
- If a system has MTTF = 10,000 hours and MTTR = 100 hours, what is its availability?
- Why is RAID 0 considered a misnomer, and what is the primary benefit of using it?
- In RAID 4/5, why does one logical write require only 2 disk reads and 2 disk writes instead of accessing all disks like in RAID 3?
- What is the fundamental difference between RAID 4 and RAID 5 that allows RAID 5 to perform multiple small writes simultaneously?
π Lecture 41 β Networks and Clusters (Networks: Interconnection and Topology)
π Overview: This lecture shifts focus from single computer architecture to interconnection networks, explaining how computers are connected to form networks. It covers fundamental network concepts, communication models, performance parameters, physical media, and various network topologies, establishing the foundation for understanding clusters and internetworking.
ποΈ Topics Covered
The lecture begins with a recap of I/O systems, storage dependability, and RAID levels before introducing interconnection networks. It covers network communication models, message formats, software protocols, performance parameters (bandwidth, latency), physical media (twisted pair, coaxial cable, fiber optics), bus-based networks, and both centralized (crossbar, multistage Omega) and distributed (ring, 2D grid/torus) switch topologies.
π Lecture Summary
Recap: I/O Systems and Storages
The lecture begins by concluding the discussion on storage I/Os and communication I/Os. The dependability, reliability, and availability of storage I/Os significantly influence overall computer system performance. Dependability is the quality of delivered service such that confidence can be placed on this service, measured by quantifying transitions between service accomplishment and service interruption. Reliability measures continuous service accomplishment or time to failure from a reference initial instant. Availability measures service accomplishment with respect to swinging between accomplishment and interruption states.
Recap: I/O and Storage Systems & Network Attached Storages
Storages interface with processors using channel, backplane, and network interconnects. Networks sustain high bandwidth transfers, and their file-server operating system supports remote file access. Network Attached Storages (NAS) have high dependability but are vulnerable to reliability issues. To improve availability and performance, disk arrays are introduced where data is striped across a set of disks, making the collection appear as a single large disk. Throughput improves due to many small disk drives having high bandwidth. The drawback is that with N devices, reliability decreases to 1/N of a single device.
Recap: Redundant Arrays of Disks
RAID (Redundant Array of Inexpensive Disks) improves disk array dependability by adding redundant disks to tolerate faults. Different RAID levels exist:
- RAID 0: Disk array without redundant disks, but employs data striping
- RAID 1: Disk Mirroring β each disk is fully duplicated onto its "shadow"
- RAID 3: Bit-Interleaved Parity Disk β uses one parity disk per group of data, parity computed across recovery group to protect against hard disk failures
- RAID 4: Block Interleaved Parity β parity disk associated to each data block, supports both small and large reads/writes
- RAID 5: Block Interleaved Distributed Parity β parity associated to each data block, data blocks distributed among different disks in each row, allowing simultaneous read/write of multiple blocks
π Definition β RAID: A disk array with redundant disks added to tolerate faults, classified by numerical levels.
Interconnection Networks
The lecture shifts focus from single computer architecture to connecting computers into networks. Standard components include:
- Computer nodes (host or end system)
- H/W and S/W interface
- Links to the interconnection network
- Interconnection networks (network or communication subnet)
The coordinated use of interconnected computers in a machine room is a cluster. Connecting two or more interconnection networks is called Internetworking, with the Internet being the prime example. Internetworking relies on communication standards to convert information between network types.
Based on nodes and proximity, interconnections are designated as:
- Local Area Network (LAN): Hundreds of computers in a building, up to a few kilometers
- Wide Area Network (WAN): Thousands of computers worldwide, thousands of kilometers β ATM (Automatic Teller Machine) is a typical example
- System Area Network (SAN): Hundreds of nodes within a machine room, less than 100 meters β this is basically the cluster
Moore's Law has contracted the network definition to include interconnection of components within a single computer.
Networks Communication Model
A simple model shows two machines connected via two unidirectional wires with a FIFO (queue) at each end to hold data. Machine A sends a request to B to get data; B responds with a reply containing the data. Messages contain extra information beyond data.
Networks Message Format
A basic message format uses:
- A 1-bit header specifying request (header=0) or reply (header=1)
- The request carries the address of the data word
- The reply carries the data word
Networks Interconnection Software
Interconnection networks involve software to establish communication. The network software:
- Cooperates with the OS to distinguish between processes on other networks
- Protects processes running on networks
- Ensures reliable delivery β message is neither distorted nor lost in transit
For reliability, the message format is modified by adding an error detection code (checksum or CRC) and using a 2-bit header. The sender calculates and adds this information; the receiver checks it and sends an acknowledgment if the test passes. The sender activates a timer each time a message is sent and copies data into an operating system buffer to resend if acknowledgment doesn't arrive before timer expiry.
Networks Interconnection Protocol
The acknowledgment protocol ensures reliable communication, but additional issues include:
- Different byte-order conventions (Big Endian or Little Endian) between manufacturers
- Duplicate delivery prevention when the original message is delayed in the network
- Sequence numbers to maintain message order
- Feedback mechanism when receiver's FIFO is full
Networks Performance Model
Network performance can be modeled at any level (chip, PCB, cluster) through interconnection performance parameters:
- Bandwidth: Maximum rate at which the network can propagate information
- Time of Flight: Time for the first bit from departure to arrival at receiver
- Transmission Time: Time for message to pass through network, excluding time of flight β time between first and last bit arriving
- Transport Latency: Sum of time of flight and transmission time
- Sender Overhead: Time for processor to inject message into network (hardware + software)
- Receiver Overhead: Time for receiver to pull message from network (hardware + software)
π Formula: Total Latency = Sender Overhead + Time of Flight + (Message Size / Bandwidth) + Receiver Overhead
Interconnection Network Media Hierarchy
Similar to memory hierarchy, interconnect media varies in cost, performance, and reliability based on maximum distance between nodes.
Twisted Pair (of Copper Wire): Two insulated copper wires (~1mm thick) twisted together to reduce electrical interference. Original telephone lines give a few megabits/sec (Level-1 or Category-1 UTP). Cat-3 UTP supports 10M bits/sec Ethernet; Cat-5 supports 100M bits/sec up to 1000M bits/sec over 100 meters.
Coaxial Cable: A stiff single copper wire surrounded by insulating material covered by cylindrical woven sheath. A 50 ohm base-band coaxial cable delivers 10M bits/sec over a kilometer. Offers high bandwidth and good noise immunity over several kilometers.
Fiber Optics: One-way (simplex) media β two fibers used for full-duplex. Contains a glass fiber core surrounded by cladding to confine light, covered by a protective buffer. Uses LED or laser as transmitter and photo diode as detector. Two forms:
- Multimode Fiber: Inexpensive light source, wavelength larger than light, wider dispersion, limited to ~1000M bit/sec over hundreds of meters or 100M bit/sec over a few kilometers
- Single-mode Fiber: More expensive lasers, single wavelength, transmits G bits/sec for hundreds of kilometers. Drawbacks: difficult connectors, less reliable, more expensive, bending restrictions
Interconnection Networks (Bus-Based)
Interconnecting hundreds of computers is more challenging. The bus-based LAN or Ethernet is the simplest way to interconnect more than two computers sharing a single media. Processors and memory connect through a "bus." It is simple and cost-effective for small-scale multiprocessors, but bus bandwidth limits the number of processors. Coordination and arbitration are required as multiple computers may need the same media simultaneously.
For small networks (a few hundred meters), centralized arbitration may be used. For networks spanning kilometers, distributed arbitration is needed. Arbitration works on "look before you leap," but looking first doesn't guarantee success β if two nodes transmit simultaneously, collision occurs. Techniques to avoid collision include collision detection and token passing. An alternative to sharing media is switching, where switches provide dedicated lines to all destinations, enabling faster point-to-point communication. Switches are also called data switching exchanges, multistage interconnection networks, or interface message processors (IMPs).
Network Topology
The most popular switch-based topologies are classified as:
- Centralized Switch Topologies: Crossbar, Multistage
- Distributed Switch Topologies: 2D Grid/Mesh, 2D Torus, Hypercube Tree
Crossbar Switch Topology
A crossbar switch is a non-blocking switch that facilitates unidirectional interconnection of all inputs to any output. It uses nΒ² switches where n is the number of processors. Links are unidirectional. Routing depends on addressing style:
- Source-based routing: Message specifies the path to destination; includes sequence of out-bound arcs; once an arc is picked, that portion is dropped from the packet
- Destination-based routing: Message contains destination address; a program in the switch decides from a routing table which port to take
A crossbar switch offers low latency and high throughput.
Multistage Interconnection Topology
An intermediate class between crossbar and bus-based networks β more scalable than bus in performance, more scalable than crossbar in cost. Built from small (e.g., 2Γ2 crossbar) switch nodes with regular interconnection patterns. The Omega Topology is a typical implementation with:
- Each switch is a 2Γ2 crossbar
- Has logβ n identical stages
- Uses n/2 Γ logβ n switches versus nΒ² in crossbar
Omega Interconnection Topology: Connections depend on communication patterns and may cause blocking. For example, a message from P1 to P7 blocks while waiting for a message from P0 to P6 as they follow the same path.
Fully Connected Switching Network
A distributed switching network distributes switching throughout the network. A fully connected network interconnects all nodes to each other.
Terminology of Distributed Network
- Degree: Number of links to each node
- Diameter: Number of nodes between source and destination
For a fully connected network:
- Diameter = 1
- Degree = K - 1 (K = number of nodes)
- Links = K Γ (K-1) / 2
- Bisect = K Γ K / 5
Distributed Switch Topologies β Ring Network
The simplest low-cost alternative to fully interconnected networks is a ring network. A small switch is placed at every computer. Only two nodes connect to a particular node, so messages must hop through intermediate nodes until they reach the destination. For example, connecting the 1st node to the 4th node requires hopping through the 2nd and 3rd nodes. A variation is the Token Ring, where a single token goes around the ring to determine which node can send β a node can send only when it gets the token.
Ring network measures:
- Degree: 2
- Diameter: N/2
- Bisect: 2
- Bandwidth: N
- Latency: N/2
2D Grid and 2D Torus Mesh
Connecting switches associated with each node to switches on the left, right, up, and down, and connecting top and bottom rows gives a grid structure. Connecting the switches of left and right columns gives 2D Torus Mesh.
β Key Takeaways
- Network performance is quantified by total latency = Sender Overhead + Time of Flight + (Message Size / Bandwidth) + Receiver Overhead β this formula is essential for comparing network designs.
- Three levels of networks exist based on scale: LAN (building), WAN (world), and SAN (machine room/cluster), each with different distance and node constraints.
- Crossbar switches offer non-blocking, low-latency interconnects but scale poorly (nΒ² switches), while multistage networks like Omega offer better cost scaling (n/2 Γ logβ n switches) but can suffer from blocking.
- Physical media (twisted pair, coaxial, fiber optics) trade off cost, bandwidth, distance, and reliability, with single-mode fiber offering the highest performance at highest cost.
- Distributed topologies (ring, 2D grid/torus) balance cost and performance β ring has degree 2 but diameter N/2, while fully connected has diameter 1 but K-1 links per node.
π§ Quick Revision Questions
- What is the formula for total latency of a message in a network, and what does each term represent?
- How does RAID 5 differ from RAID 4 in terms of parity disk placement and data distribution?
- What is the difference between source-based routing and destination-based routing in a crossbar switch?
- For a ring network with N nodes, what are the degree, diameter, and bisect width?
- Why does the Omega multistage topology use fewer switches than a crossbar, and what trade-off does this create?