DBMS Topics
HubUnit 1 — Introduction to DBMS & Entity-Relationship ModelAdvantages and Disadvantages of DBMSAggregationCharacteristics of DBMSData IndependenceData ModelsDatabase Users and AdministratorsDBMS LanguagesER DiagramsEntity-Relationship ER ModelIntroduction to DBMSKeys in DBMSRelationships and CardinalitySchema and InstanceSpecialization and GeneralizationUnit 1 Summary — Introduction to DBMS & ER ModelThree-Level Architecture ANSI/SPARC ArchitectureWeak Entity SetUnit 2 — Relational Model & SQLConstraints in SQLCursorsDCL and TCL CommandsDDL CommandsDML CommandsDomain Relational Calculus DRCFunctions and Stored ProceduresJoins in SQLNested Queries SubqueriesRelational AlgebraRelational CalculusRelational ModelSQL IntroductionUnit 2 Summary — Relational Model & SQLTriggers in SQLTuple Relational Calculus TRCViews in SQLArmstrong's Axioms & Closure of AttributesBoyce-Codd Normal Form BCNFClosure of AttributesDependency PreservationFifth Normal Form 5NFFirst Normal Form 1NFFourth Normal Form 4NFFunctional DependencyLossless DecompositionMultivalued DependencyNormal Forms — OverviewNormalization — IntroductionSecond Normal Form 2NFUnit 3 Summary — NormalizationThird Normal Form 3NFUnit 4 — Transaction Management & Concurrency ControlACID PropertiesCheckpointsConcurrency Control & SerializabilityDeadlock in DBMSLock-Based Protocols & Two-Phase LockingLog-Based RecoveryRecovery SystemSerializabilityShadow PagingUnit 4 Summary — Transaction Management & Concurrency ControlTimestamp-Based ProtocolTransaction ManagementTransaction StatesTwo-Phase Locking 2PLUnit 5 — Storage & IndexingB+ TreeB-TreeCost EstimationDynamic HashingFile OrganizationHashing TechniquesIndexing — IntroductionMulti-Level IndexingQuery OptimizationQuery ProcessingSingle-Level IndexingStatic HashingUnit 5 Summary — Storage & IndexingUnit 6 — Advanced Topics in DBMSBig Data and DBMSCloud DatabaseData MiningData WarehouseDatabase SecurityDistributed DatabaseDistributed Query ProcessingDistributed DBMS Architecture, Fragmentation & ReplicationMongoDB — IntroductionNoSQL DatabaseReplication in Distributed DatabasesUnit 6 Summary — Advanced Topics in DBMSViva Voce — DBMSImportant Viva Questions — DBMSInterview Questions — DBMSViva Answers — DBMSDBMS Multiple Choice Questions MCQsDBMS Lab Programs
Big Data and DBMS
Last Updated : 21 May, 2026
Big Data refers to datasets that are so large, fast-moving, or complex that traditional database systems cannot process them efficiently. It requires new technologies and
Unit 6206 words13 headingsTables includedExamples included
What is Big Data?
Big Data refers to datasets that are so large, fast-moving, or complex that traditional database systems cannot process them efficiently. It requires new technologies and approaches for storage, processing, and analysis.
The 5 V's of Big Data
| │ Volume | Terabytes to Petabytes of data │ |
| 4 PB/day; Twitter: 500M tweets/day)│ | |
| │ Velocity | Speed at which data is generated │ |
| │ Variety | Different types and formats of data │ |
| │ Veracity | Uncertainty and quality of data │ |
| │ Value | Extracting useful insights from raw data │ |
Why Traditional DBMS Fails for Big Data
| 1. Vertical scaling only | one powerful server has hardware limits |
| 2. Fixed schema | cannot handle unstructured/semi-structured data |
| 3. ACID overhead | slows throughput for write-heavy workloads |
| 4. Row-oriented storage | inefficient for analytical queries |
| 6. Single point of failure | RDBMS needs HA solutions |
Big Data Ecosystem
Apache Hadoop
Hadoop is an open-source framework for distributed storage and processing of Big Data.
| HDFS (Hadoop Distributed File System) | ||
|---|---|---|
| → Stores data across many nodes | ||
| → Replicates each block 3 times (fault | ||
| tolerant) | ||
| → Block size: 128 MB default | ||
| MapReduce | ||
| → Distributed processing framework | ||
| → Map phase: process each record | ||
| → Shuffle: group by key | ||
| → Reduce phase: aggregate per key | ||
| YARN (Resource Manager) | ||
| → Manages cluster resources (CPU, RAM) |
MapReduce Example — Word Count
| Input | "apple banana apple cherry banana apple" |
| apple | 1 |
| banana | 1 |
| apple | 1 |
| cherry | 1 |
| banana | 1 |
| apple | 1 |
| apple | [1, 1, 1] |
| banana | [1, 1] |
| cherry | [1] |
| apple | 3 |
| banana | 2 |
| cherry | 1 |
Apache Spark
Apache Spark is a faster alternative to Hadoop MapReduce that processes data in-memory.
| Spark SQL | Spark | Spark | GraphX | ||
|---|---|---|---|---|---|
| (SQL on | Streaming | MLlib | (Graph | ||
| big data) | (real-time | (machine | analytics) | ||
| analytics) | learning) |
Big Data Storage Technologies
| Technology | Type | Best For |
|---|---|---|
| HDFS | Distributed file system | Batch processing, large files |
| HBase | Column-family NoSQL on HDFS | Random read/write on big data |
| Apache Cassandra | Column-family NoSQL | High write throughput, time-series |
| Apache Kafka | Message streaming | Real-time data pipelines |
| Amazon S3 | Object storage | Data lake, unstructured data |
| Elasticsearch | Document search engine | Full-text search, log analytics |
Big Data Architecture Patterns
Lambda Architecture
| │ | Processes all historical data |
| │ | Accurate but slow (hours) |
| Serving Layer | Combines batch + speed layer results for queries |
Kappa Architecture
Simpler — uses only a streaming layer (Kafka + Spark Streaming) for both real-time and historical data.
Big Data and DBMS Integration
Traditional RDBMS still used alongside Big Data tools:
Data Flow:
Source Systems (Apps, IoT, Web logs)
│
▼
Big Data Ingestion (Kafka, Flume)
│
▼
Big Data Storage (HDFS, S3, Cassandra)
│
▼
Big Data Processing (Spark, Hive)
│
▼
Data Warehouse / RDBMS (for reporting & analytics)
│
▼
BI Tools (Tableau, Power BI, Grafana)
Hive — SQL on Big Data
Apache Hive allows writing SQL-like queries (HiveQL) on data stored in HDFS:
sql exampleWoHoTech
-- HiveQL (looks like SQL, runs on Hadoop cluster)
SELECT department, AVG(salary) AS avg_sal
FROM employee_data
WHERE year = 2024
GROUP BY department
HAVING AVG(salary) > 60000;
-- Compiles to MapReduce or Spark jobs behind the scenesExam Focus
Revise definitions, diagrams, examples, and short-answer points for Big Data and DBMS.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this DBMS topic.
Search Terms
dbms, database management system, database notes, sql, unit, big, data, and
Related DBMS Topics
Continue learning this concept
Unit 6Data MiningData Mining is the process of discovering interesting, previously unknown patterns, correlations, anomalies, and insights from large amounts of data stored in databases aUnit 6Data WarehouseA Data Warehouse DW is a large, centralized repository of integrated data from multiple sources, optimized for analytical querying and business intelligence BI — not for Unit 6Unit 6 — Advanced Topics in DBMSThis unit covers modern and advanced database topics including distributed databases, NoSQL technologies, Big Data, data warehousing, and database security.Unit 6Cloud DatabaseA cloud database is a database service built, deployed, and accessed through a cloud computing platform. It provides database capabilities as a service DBaaS — Database aUnit 6Database SecurityDatabase security refers to the collective measures used to protect and secure a database from unauthorized access, misuse, corruption, or theft of data. It encompasses bUnit 6Distributed DatabaseA distributed database is a collection of multiple, logically interrelated databases distributed over a computer network. The data is stored at different sites nodes/loca