Malware and Threats - Security Threats

Introduction

Malware (malicious software) is any software intentionally designed to cause harm to computer systems, steal data, or gain unauthorized access. Understanding different types of malware and how they exploit operating system mechanisms is essential for building secure systems. As an OS developer or administrator, you need to know what you are defending against.

The relationship between malware and operating systems is adversarial: malware exploits OS vulnerabilities, and OS designers build protection mechanisms in response. This arms race has driven many security innovations — memory protection, address space randomization, secure boot, and sandboxing all emerged as responses to evolving threats.

Types of Malware

Viruses

A virus attaches itself to a legitimate program and activates when the host program runs. It cannot spread on its own — it requires human action (running an infected program, opening an infected document).

How a virus works:

Types of viruses:

• File infector: Attaches to executable files (.exe, .com)
• Boot sector virus: Infects the Master Boot Record; runs before the OS loads
• Macro virus: Embedded in documents (Word macros, Excel VBA)
• Polymorphic virus: Changes its code signature on each infection (evades signature-based detection)
• Metamorphic virus: Completely rewrites itself while maintaining functionality

Worms

Worms are self-replicating programs that spread across networks without human intervention. Unlike viruses, they do not need to attach to host programs.

Famous worms:

• Morris Worm (1988): First internet worm; exploited Unix finger and sendmail vulnerabilities
• Code Red (2001): Exploited IIS buffer overflow; infected 359,000 servers in 14 hours
• WannaCry (2017): Ransomware worm; exploited Windows SMB vulnerability; hit 200,000+ systems globally

Worm propagation:

Trojans (Trojan Horses)

Software that appears legitimate but contains hidden malicious functionality. Named after the Greek myth — the gift conceals a threat.

Examples:

• A "free game" that installs a keylogger
• A "system optimizer" that actually installs a backdoor
• A "PDF viewer" that creates a reverse shell to the attacker

Trojans rely on social engineering — tricking users into voluntarily installing them.

Ransomware

Encrypts the victim's files and demands payment (usually cryptocurrency) for the decryption key.

Attack flow:

1. Initial infection (phishing email, exploit kit, compromised website)
2. Privilege escalation (gain admin access)
3. Lateral movement (spread to other systems on the network)
4. Encryption of files (AES-256, RSA key pair)
5. Ransom note displayed (pay within 72 hours or data deleted)

Notable ransomware:

• WannaCry: Used EternalBlue exploit (NSA tool leaked)
• NotPetya: Wiped data under the guise of ransomware (actually cyberweapon)
• REvil: Targeted enterprises with multi-million dollar ransoms

Rootkits

Malware that hides its presence by modifying the operating system itself. A rootkit operates at the OS level, making detection extremely difficult.

Types by level:

• User-mode rootkit: Replaces system utilities (ls, ps, netstat) with modified versions that hide malicious files/processes
• Kernel-mode rootkit: Modifies kernel data structures or hooks system calls; invisible to user-space tools
• Bootkit: Infects the boot process; loads before the OS, intercepting all subsequent operations
• Firmware rootkit: Hides in device firmware (BIOS/UEFI); survives OS reinstallation

Spyware and Keyloggers

Silently monitors user activity:

• Keylogger: Records every keystroke (captures passwords, messages, credit card numbers)
• Screen capture: Periodically screenshots the desktop
• Clipboard monitor: Captures copied data (cryptocurrency addresses, passwords)

Attack Vectors and Exploitation Techniques

Buffer Overflow

The most classic OS-level exploit. When a program writes beyond a buffer's boundary, it can overwrite the return address on the stack, redirecting execution to attacker-controlled code.

OS defenses: Stack canaries, ASLR, NX (non-executable stack), bounds checking

Privilege Escalation

Gaining higher privileges than authorized:

• Exploiting setuid programs with vulnerabilities
• Kernel exploits that escape user mode
• Misconfigured sudo rules
• Unpatched local privilege escalation CVEs

Social Engineering

Manipulating humans rather than software:

• Phishing emails with malicious attachments
• Pretexting (impersonating IT support to get passwords)
• Baiting (leaving infected USB drives in parking lots)

How Operating Systems Defend Against Threats

The Security Mindset

As a future systems professional, adopt these principles:

1. Assume breach: Design systems to limit damage even after compromise
2. Defense in depth: Multiple layers — if one fails, others still protect
3. Least privilege: Every process gets minimum necessary permissions
4. Patch promptly: Most successful attacks exploit known, already-patched vulnerabilities
5. Monitor and audit: Detect intrusions quickly to minimize damage

Key Takeaways

• Viruses require human action to spread; worms spread automatically via network
• Trojans deceive users into voluntary installation of malicious software
• Ransomware encrypts data for extortion; backup is the primary defense
• Rootkits hide at the OS level, making detection extremely difficult
• Buffer overflows remain a critical vulnerability; modern OS uses ASLR, NX, and canaries
• No single defense is sufficient — security requires layered mechanisms and constant vigilance