Distributed Operating Systems - Design and Concepts

Introduction

A Distributed Operating System manages a collection of independent computers (nodes) and makes them appear as a single coherent system to users. Unlike a network OS where users explicitly access remote machines, a distributed OS provides transparency — users do not know (or need to know) where their programs execute or where their files are stored.

Imagine a university library with multiple branches across campus. In a "network library" model, you must physically go to the specific branch that has your book. In a "distributed library" model, you request any book from any desk, and the system automatically retrieves it from whichever branch has it — you never need to know where it was stored. That is the essence of a distributed OS.

Design Goals

Transparency

The most important goal. A distributed OS should hide the complexity of distribution:

Scalability

The system must handle growth in three dimensions:

• Size: More users, more resources
• Geography: Nodes spread across cities, countries, continents
• Administration: Multiple organizations managing different nodes

Fault Tolerance

In a distributed system, partial failures are the norm — individual nodes crash, networks partition, messages get lost. The system must continue operating correctly despite these failures.

Communication Mechanisms

Processes on different nodes need to communicate. The fundamental mechanisms are:

Remote Procedure Call (RPC)

RPC makes remote communication look like a local function call:

Challenge: Unlike local calls, RPC can fail due to network issues. What if the server crashes after executing the function but before sending the response? The client does not know if the operation was performed.

Message Passing

Lower-level than RPC. Processes explicitly send and receive messages:

Variants:

• Synchronous: Sender blocks until receiver gets the message
• Asynchronous: Sender continues immediately (message buffered)
• Multicast: Send to a group of receivers simultaneously

Distributed File Systems

A distributed file system allows files to be stored across multiple machines but accessed uniformly from any node.

NFS (Network File System)

The classic Unix distributed file system. A client mounts a remote directory as if it were local:

• Stateless server design: Server does not track which clients have files open (simplifies crash recovery)
• Client-side caching: Improves performance but raises consistency issues
• VFS layer: Integrates transparently with local file system calls

Google File System (GFS) / HDFS

Designed for massive data at scale:

• Files split into large chunks (64-128 MB)
• Each chunk replicated across 3+ nodes
• Single master tracks chunk locations (metadata)
• Optimized for sequential reads and appends (not random writes)

Consistency and Consensus

The fundamental challenge of distributed systems: how do multiple nodes agree on the current state of shared data?

CAP Theorem

A distributed system can guarantee at most two of three properties simultaneously:

• Consistency: All nodes see the same data at the same time
• Availability: Every request receives a response (success or failure)
• Partition Tolerance: System continues operating despite network partitions

Since network partitions are unavoidable in practice, real systems must choose between consistency (CP systems like ZooKeeper) and availability (AP systems like Cassandra).

Consensus Algorithms

When nodes must agree on a value (who is the leader? what is the latest write?), they use consensus algorithms:

Paxos: Theoretical foundation for consensus. A proposer suggests a value; acceptors vote; once a majority agrees, the value is decided. Correct but notoriously difficult to implement.

Raft: Designed as an understandable alternative to Paxos. Uses a clear leader election mechanism and log replication:

1. One node is elected leader
2. All writes go through the leader
3. Leader replicates log entries to followers
4. Entry committed once majority acknowledges
5. If leader fails, new election occurs

Clock Synchronization

In a distributed system, there is no global clock. Each node has its own local clock that drifts independently. This creates ordering problems — how do you determine which event happened first?

Lamport Logical Clocks

Instead of real time, assign logical timestamps that preserve causality:

• Each process maintains a counter
• On local event: increment counter
• On send: attach current counter to message
• On receive: set counter to max(local, received) + 1

This guarantees: if event A caused event B, then timestamp(A) < timestamp(B).

Vector Clocks

An extension that can also detect concurrent events (events with no causal relationship). Each node maintains a vector of counters (one per node in the system).

Distributed Mutual Exclusion

When processes on different machines share a resource, you need distributed locking:

Centralized approach: One designated coordinator grants locks. Simple but single point of failure.

Token-based: A token circulates; only the token holder can enter the critical section. Fair but token loss is problematic.

Ricart-Agrawala algorithm: Request-based. A process wanting the CS multicasts a request; enters when all processes grant permission. No single point of failure.

Challenges of Distributed Systems

1. Partial failure: Some nodes fail while others continue (must detect and handle)
2. Network unreliability: Messages can be lost, duplicated, reordered, or delayed
3. No global state: No single node knows the complete system state at any instant
4. Clock skew: Cannot rely on synchronized timestamps for ordering events
5. Split brain: Network partition causes two groups each believing they are the "correct" system
6. Debugging difficulty: Non-deterministic behavior makes bugs hard to reproduce

Real-World Examples

• Google Spanner: Globally distributed database using GPS and atomic clocks for consistency
• Apache Kafka: Distributed message queue with replicated logs
• Amazon DynamoDB: Distributed key-value store (AP system, eventual consistency)
• etcd/ZooKeeper: Distributed coordination services using consensus

Key Takeaways

• A distributed OS hides distribution complexity behind transparency
• Communication happens via RPC or message passing
• CAP theorem forces trade-offs between consistency and availability
• Consensus algorithms (Paxos, Raft) enable agreement despite failures
• Logical clocks solve event ordering without synchronized physical clocks
• Partial failure handling distinguishes distributed systems from centralized ones