HookA petabyte a second, and nowhere to put it
Deep under the French-Swiss border, the Large Hadron Collider at CERN smashes bunches of protons together roughly forty million times a second. When those collisions happen inside detectors such as ATLAS and CMS, the raw signals amount to something on the order of a petabyte of data per second — a quantity so large that no storage system on Earth could keep even a fraction of it. CERN's answer is brutal: a multi-stage trigger system, part hardware and part software, throws away more than 99.99% of the collisions within microseconds, keeping only the few hundred to roughly a thousand events per second that might contain something new. Even after that ruthless filtering, the collider records tens of petabytes a year, and the only way to analyse it is to spread the work across the Worldwide LHC Computing Grid — around 170 computing sites in more than 40 countries, working on pieces of the same dataset at once.
That is Big Data in one machine: too big to store on one disk, arriving too fast to process in real time, and mixed in form. The label Big Data describes datasets whose size, speed or messiness defeats the tools built for ordinary databases — and the interesting part of this section is not the adjectives but the consequence. Once data no longer fits on a single computer, you are forced to process it across a cluster of many machines at once, and that in turn forces a particular style of programming and a particular way of modelling the data. This is why an A-level on relational databases suddenly turns to functional programming, immutable facts and graphs: they are the tools that survive when the data outgrows the machine.
ModelThe three Vs — and why relational databases hit a wall
Big Data is characterised by three properties, conventionally the three Vs. Volume is the sheer quantity — measured in terabytes, petabytes and beyond — so large it cannot sit on a single machine. Velocity is the speed at which new data arrives and must be handled, sometimes as a continuous real-time stream rather than a fixed batch, like the LHC's forty-million-collisions-a-second or the firehose of posts on a social network. Variety is the range of forms the data takes: neatly structured rows, but also free text, images, sensor readings, video and log files, arriving with no consistent schema.
A traditional relational database is superb for structured, moderate-sized data, but each of the three Vs breaks it. Relational systems are designed to scale vertically — put the database on a bigger, more powerful single machine — and there is a hard ceiling on how big one machine can get, whereas Big Data demands horizontal scaling across hundreds of cheap machines. The rigid, fixed schema that gives a relational database its integrity becomes a straitjacket when the variety of data will not sit in tidy columns. And the strong consistency guarantees a relational database enforces on every transaction become extraordinarily expensive to maintain when the data is spread across a distributed cluster. Big Data techniques exist precisely because the relational model, for all its strengths in this very course, does not survive contact with data at this scale.
ModelFunctional programming and distributed processing
If a dataset is spread across a thousand machines, the only way to process it in reasonable time is to have all thousand work at once — and that is exactly where functional programming earns its place. A pure function has no side effects: it reads its inputs, returns a result, and changes nothing else in the system — no shared variable is written, no global state is touched. This property, called referential transparency, means a function's result depends only on its arguments, so the same call always gives the same answer. The consequence for Big Data is decisive: because pure functions do not interfere with one another through shared state, they can be run in parallel across many machines with no risk of the race conditions that plague ordinary concurrent code. There is no lost update to worry about when nothing is ever updated.
The workhorses are the higher-order list functions. Map applies a function to every element of a dataset independently — perfectly parallel, since each element is handled in isolation. Filter keeps only the elements that satisfy a condition. Reduce (also called fold) combines a whole collection down to a single value, such as a sum or a count. Google's MapReduce framework, published in 2004, built an entire industry on exactly this pattern: split the data across the cluster, map a function over each shard in parallel, shuffle the intermediate results together by key, and reduce each group to a final answer. Immutability plus these three operations is what makes distributed processing tractable.
Count how often each hashtag appears across a dataset too big for one machine, split over three shards. Shard 1 holds the tags [#cs, #exam, #cs]; shard 2 holds [#exam, #cs]; shard 3 holds [#exam, #revision]. Each machine runs map independently, turning every tag into a key-value pair with count 1. Shard 1 emits (#cs,1),(#exam,1),(#cs,1); shard 2 emits (#exam,1),(#cs,1); shard 3 emits (#exam,1),(#revision,1) — all three maps run at the same time because none depends on the others.
The framework then shuffles the pairs so that all values for the same key land together: #cs gets [1,1,1], #exam gets [1,1,1], #revision gets [1]. Finally reduce sums each group: #cs → 3, #exam → 3, #revision → 1. Notice the arithmetic is associative — it does not matter which machine adds which 1s first, so the reduce step can itself be split across machines and combined at the end. This indifference to order is a direct gift of having no side effects, and it is why the same job runs on three machines or three thousand with no change to the logic.
ModelThe fact-based model — never overwrite the truth
A relational database mutates data in place: when a customer changes address, the old address is overwritten and gone. That is efficient, but at Big Data scale it is a liability, because you have destroyed history and created a single value that many machines must agree on and lock. The fact-based model takes the opposite approach: the master dataset is a growing collection of immutable, atomic facts, each tagged with a timestamp recording when it was true. You never update and never delete — you only append new facts. 'Alex lived at 4 Elm Road as of 2019' is not overwritten when Alex moves; a new fact, 'Alex lived at 9 Oak Lane as of 2023', is simply added alongside it.
The benefits map straight onto the three Vs. Immutable, append-only data is far easier to distribute, because facts never change and so never need to be locked or kept consistent across machines — every machine can safely hold a copy. It is robust to error: a bad piece of processing can be re-run because the raw facts were never destroyed, whereas an overwrite in a relational database is unrecoverable. And it preserves the full history, so you can reconstruct what the data looked like at any past moment by considering only the facts whose timestamps precede it. The trade-off is that the master dataset grows without bound and current values must be computed from the accumulated facts rather than simply read — which is precisely the job the map-reduce machinery of the previous block exists to do.
ModelGraph schemas — modelling connection itself
The final piece is how those facts are shaped, and for highly connected data the answer is a graph schema rather than rigid tables. A graph is built from nodes (the entities — a person, a post, a place), edges (the relationships between them — 'follows', 'liked', 'lives in') and properties (data attached to a node or edge — a name, a timestamp, a weight). Where a relational schema fixes its structure in advance and forces every relationship through foreign keys and JOINs, a graph schema is flexible: new kinds of node, edge and property can be added without redesigning the whole database, which suits the ever-changing variety of Big Data.
Graphs are the natural home for questions that are really about connections — social networks, recommendation engines, fraud rings, the web of links between pages. Asking 'which of my followers also follow someone I follow?' is an awkward multi-way JOIN in a relational database but a direct walk across edges in a graph. Crucially, a graph schema pairs beautifully with the fact-based model: each immutable fact can itself be expressed as a small piece of graph — two nodes and an edge — so the master dataset becomes an ever-growing graph of timestamped, atomic relationships. Volume is handled by distributing the nodes and edges across the cluster; velocity by appending new ones; and variety by the schema's willingness to accept new shapes of connection without a rebuild.
Represent the fact Sam followed the account @vofti on 6 July 2026 as a graph. Create two nodes — one for the person Sam, one for the account @vofti — then join them with a directed edge labelled 'follows' pointing from Sam to @vofti, and attach the property date = 2026-07-06 to that edge. A second fact, Sam liked post P57, adds a node for P57 and a 'liked' edge from Sam to P57. Nothing existing is altered; the graph simply grows.
Now the power of the model shows itself. To find everyone who both follows @vofti and has liked post P57, you do not write a three-table JOIN — you walk the graph: gather every node with a 'follows' edge into @vofti, gather every node with a 'liked' edge into P57, and take the overlap. Because each fact was stored as immutable nodes and edges with timestamps, the same graph can answer 'who followed @vofti before July 2026?' just by ignoring edges with a later date — history is queryable, not overwritten.
VocabularyKey terms the mark scheme pays for
TrapsMisconceptions that cost marks
ExamWhat examiners want
Big Data is assessed on the written Paper 1 and Paper 2 and is more conceptual than computational, so marks come from explaining why, not from recitation. When you name the three Vs, do not just list them — tie each to a consequence: variety is why a fixed relational schema fails, velocity is why real-time streams cannot be batch-loaded, volume is why the data must be spread across a cluster. An answer that only defines volume, velocity and variety sits at the bottom of the mark scheme; one that connects them to distributed processing reaches the top.
The most reliably examined link is functional programming to distributed processing: state clearly that pure functions have no side effects, that this means computations do not interfere through shared state, and that this is exactly what lets map and reduce run in parallel across machines without race conditions. Learn to describe a small MapReduce by hand — map produces key-value pairs, the framework shuffles by key, reduce combines each group — because 'trace or explain the processing' questions want that pipeline. For the fact-based model, stress immutability and append-only storage and the three payoffs it brings (easy distribution, recoverability from error, full history), and contrast it explicitly with a relational database overwriting data in place. For graph schemas, describe nodes, edges and properties and give a connection question — such as mutual followers — that a graph answers by walking edges where a relational database would need multiple JOINs. Precise cause-and-effect, anchored to a concrete example, is what turns a definition into a full-mark answer.