Protection Mechanisms - Security Implementation

Introduction

Protection mechanisms are the enforcement tools the operating system uses to implement its security policies. While authentication verifies identity and authorization defines permissions, protection mechanisms actually prevent unauthorized operations from succeeding at the hardware and software level. If a process tries to access memory it does not own or execute a privileged instruction it should not use, protection mechanisms physically stop it.

Think of it like a bank vault: authentication is the guard checking your ID, authorization is your account permissions, and protection mechanisms are the steel walls, locked doors, and reinforced glass that physically prevent unauthorized access.

Protection Domains

A protection domain defines what resources a process can access and what operations it can perform. Every process executes within a domain that limits its capabilities.

Domain = set of (object, rights) pairs

For example, a web server process might execute in a domain with:

• (web_files_directory, read)
• (log_file, write)
• (port_80, bind)
• (temporary_dir, read+write)

It explicitly does NOT have rights to read /etc/shadow, write to system binaries, or access other users' files.

Domain Switching

Processes often need to temporarily change domains — for example, a user process making a system call enters the kernel domain temporarily:

In Unix, the setuid bit enables controlled domain switching: when a user executes a setuid program, it runs with the file owner's permissions rather than the user's.

Hardware Protection Mechanisms

CPU Protection Rings

Modern CPUs provide privilege levels (rings) enforced in hardware:

In practice, most systems use only Ring 0 (kernel) and Ring 3 (user):

• Ring 0 can execute ANY instruction, access ANY memory address, control interrupts
• Ring 3 cannot execute privileged instructions (I/O, interrupt control, TLB manipulation) — attempting them triggers a hardware trap

The mode bit: A single bit in the CPU's status register tracks current privilege level. It is changed only by hardware-defined transitions (interrupts, system calls, returns).

Memory Protection

The MMU (Memory Management Unit) enforces memory isolation:

1. Base and Limit Registers: Simplest form — each process has a base address and limit. Any access outside this range triggers a hardware fault.

1. Page Table Permissions: Each page table entry contains protection bits:

• Read/Write bit: Controls whether writes are allowed
• User/Supervisor bit: Controls whether user-mode code can access
• Execute-Disable (NX) bit: Prevents execution of data pages (prevents code injection attacks)

1. Segmentation Protection: Segments have privilege levels (DPL). A process can only access segments with DPL ≥ its current privilege level.

I/O Protection

All I/O operations are privileged — user processes cannot directly access hardware devices:

• I/O instructions (in, out on x86) trap to kernel if executed in Ring 3
• Memory-mapped I/O regions are marked as supervisor-only in page tables
• DMA configuration is kernel-only (otherwise a device could read arbitrary memory)

User processes access devices only through system calls → kernel validates the request → kernel performs the I/O → returns result to user.

Software Protection Mechanisms

Address Space Layout Randomization (ASLR)

Randomizes the memory layout of a process each time it runs:

• Stack at a random address
• Heap at a random address
• Libraries loaded at random positions

This makes it extremely difficult for attackers to predict where code or data resides, defeating exploits that rely on known memory addresses.

Stack Canaries

A random value placed between local variables and the return address on the stack. Before a function returns, the canary is checked — if it has been modified (indicating a buffer overflow), the program is terminated.

Sandboxing

Running untrusted code in a restricted environment that limits what damage it can do:

chroot: Restricts a process's view of the file system to a subtree:

seccomp: Restricts which system calls a process can make:

Containers (namespaces + cgroups): Complete isolation of process view (PID, network, filesystem, users) combined with resource limits.

Mandatory Access Control (MAC)

SELinux and AppArmor enforce access policies that even root cannot bypass:

Capability-Based Systems

An alternative to ACL-based protection. Instead of checking "does this user have permission for this object," the system uses unforgeable tokens (capabilities) that grant specific rights:

Advantages: No ambient authority (a process can only access what it has been explicitly given capabilities for), easy to delegate (pass capability to another process), natural least privilege.

Modern examples: Linux capabilities (fine-grained root privileges), WASM capabilities, Capsicum (FreeBSD).

Defense in Depth

No single protection mechanism is sufficient. Modern systems layer multiple mechanisms:

Key Takeaways

• Protection mechanisms physically enforce security policies at hardware and software levels
• CPU rings separate kernel (privileged) from user (restricted) code
• MMU enforces memory isolation between processes using page table permissions
• I/O protection ensures user processes cannot directly access hardware
• ASLR, NX bits, and stack canaries make exploitation difficult even when vulnerabilities exist
• Sandboxing (chroot, seccomp, containers) limits the damage compromised processes can cause
• Defense in depth: layer multiple mechanisms because no single one is unbreakable