OS Security Basics - Threats and Controls

Introduction

Operating system security is the foundation upon which all other software security rests. If the OS is compromised, no application-level security measure can save you — the attacker has access to everything. The OS is the trusted layer that enforces isolation between users, protects memory between processes, controls access to files, and mediates all hardware interaction. Understanding security at the OS level is understanding security at its most fundamental.

Every application, database, and network service depends on the OS to honestly enforce access control, faithfully deliver data, and reliably isolate processes. A security flaw in the kernel is a security flaw in everything running on that system.

Security Goals: The CIA Triad

All security mechanisms aim to protect three properties:

Confidentiality

Preventing unauthorized access to information. Only those with proper clearance and need-to-know should see sensitive data.

OS mechanisms for confidentiality:

• File permissions (restrict who can read files)
• Process isolation (one process cannot read another's memory)
• Encryption (protect data even if physical media is stolen)
• Access control lists (fine-grained read restrictions)

Threats to confidentiality: Unauthorized data access, packet sniffing, shoulder surfing, social engineering.

Integrity

Ensuring data is not modified by unauthorized parties. When you read a file, you must be confident it has not been tampered with.

OS mechanisms for integrity:

• Write permissions (restrict who can modify files)
• Memory protection (prevent code injection)
• Checksums and digital signatures (detect modifications)
• Journaling file systems (prevent corruption from crashes)

Threats to integrity: Data corruption, malware modifying files, man-in-the-middle attacks, buffer overflows.

Availability

Ensuring systems and data remain accessible to authorized users when needed.

OS mechanisms for availability:

• Process scheduling fairness (prevent one process from starving others)
• Resource limits (prevent denial-of-service by resource exhaustion)
• Redundancy (RAID, clustering)
• Rate limiting (prevent abuse of shared resources)

Threats to availability: Denial of Service (DoS) attacks, resource exhaustion, hardware failures, ransomware.

Threat Categories

External Threats

Attacks originating outside the system:

• Network attacks: Port scanning, exploitation of network services, packet injection
• Physical access: Booting from USB, removing hard drive, hardware keyloggers
• Social engineering: Phishing, pretexting, baiting

Internal Threats

Attacks from within (authorized users or compromised accounts):

• Privilege escalation: Normal user gaining root access
• Data theft: Employee copying confidential data
• Sabotage: Disgruntled employee deleting or modifying data

Software Vulnerabilities

Bugs that create security holes:

• Buffer overflow: Writing beyond allocated buffer to corrupt memory
• Race conditions: Exploiting timing windows between security check and operation
• Input validation failures: SQL injection, command injection, path traversal
• Use-after-free: Accessing memory after it has been deallocated

Security Principles for OS Design

1. Principle of Least Privilege

Every process and user gets only the minimum permissions needed for their task:

2. Defense in Depth

Multiple independent security layers — if one fails, others still protect:

3. Complete Mediation

EVERY access to EVERY resource must be checked against access policy — no shortcuts, no caching of security decisions that could become stale.

4. Fail-Safe Defaults

The default should be "deny access" — permission must be explicitly granted. If something goes wrong, fail to a secure state rather than an open one.

5. Economy of Mechanism

Security mechanisms should be simple and small. Complex systems have more bugs — and in security code, bugs mean vulnerabilities. The Linux kernel's 28+ million lines of code are a security liability compared to a microkernel's 10,000 lines.

6. Open Design

Security should not depend on secrecy of the implementation ("security through obscurity"). The system should be secure even if attackers know exactly how it works. Cryptographic algorithms are public — their security comes from mathematical hardness, not secrecy.

7. Separation of Privilege

Require multiple conditions for access. Like requiring both a key card AND a PIN to enter a secure area:

• Two-factor authentication
• Requiring two administrators to authorize critical operations
• Unix: must be file owner AND have write permission

The Trusted Computing Base (TCB)

The TCB is the set of hardware, firmware, and software components critical to system security. If ANY component of the TCB has a bug, the entire system's security may be compromised.

Design goal: Minimize the TCB. A smaller TCB means:

• Less code to audit for vulnerabilities
• Fewer bugs (less code = fewer defects)
• Easier to formally verify correctness

This is why microkernels (small TCB) are preferred for high-security systems, despite monolithic kernels (large TCB) having better raw performance.

Protection Rings

Hardware-enforced privilege levels that limit what code at each level can do:

Transition from Ring 3 to Ring 0 only through controlled gates (system calls, interrupts). A user process attempting to execute a privileged instruction triggers a hardware exception — the CPU refuses to execute it.

Security Mechanisms in Modern OS

Common Attack Surfaces

The OS presents several attack surfaces:

1. System calls: The interface between user and kernel — any bug in syscall handling is exploitable
2. Device drivers: Run in kernel mode but often written by third parties — largest source of kernel bugs
3. Network stack: Processes untrusted external data — parsing bugs enable remote exploitation
4. File systems: Mount a maliciously crafted filesystem → trigger kernel bugs in FS code
5. User-facing services: SSH, login, cron — any service accepting input is a potential target

Key Takeaways

• OS security protects Confidentiality, Integrity, and Availability (CIA triad)
• The Trusted Computing Base should be as small as possible (less code = fewer vulnerabilities)
• Least Privilege and Defense in Depth are the two most important security design principles
• Hardware protection rings physically enforce the user/kernel privilege separation
• Security is not a feature you add — it is a property that must be designed in from the start
• Every additional line of kernel code is a potential security vulnerability
• Modern OS security combines hardware mechanisms (MMU, rings) with software policies (ACLs, MAC)