How Distributed Systems Run Millions of Tasks at Once: Concurrency, Consistency, and Coordination

The whole point is doing things at the same time

A distributed system is a collection of independent computers that work together so well they look like one machine. The reason we build them is concurrency: the ability to do many things at the same time, on many machines, so the system serves a million users as comfortably as it serves one. Amazon processing orders, searches, and recommendations all at once, or Nepal's PhonePay clearing thousands of transactions during Dashain, are concurrency in action.

But concurrency is a double-edged sword. The instant two machines can touch the same data at the same time, you have to answer a hard question: what does "correct" even mean when there is no single clock and no single copy of the truth? This post is about the machinery that answers it.

Concurrency versus parallelism

People use these words interchangeably, but the distinction is useful:

• Concurrency is dealing with many things at once, structuring the system so tasks can make progress independently.
• Parallelism is doing many things at once, literally running on multiple cores or machines simultaneously.

A distributed system is concurrent by nature (independent nodes) and parallel by design (real hardware running side by side). PhonePay does not process your payment after everyone else's; it runs many payments in parallel across many servers, which is the only way the numbers work at national scale.

The hard part: consistency models

Once data is replicated across nodes, "is this value up to date?" stops having a single answer. A consistency model is the contract the system makes about what a read can see. The two ends of the spectrum:

• Strong consistency: every read returns the most recent write, always. Simple to reason about, but slow and fragile, because every node must agree before you proceed. A bank ledger wants this.
• Eventual consistency: reads may briefly return stale data, but all replicas converge given enough time. Fast and highly available. A product catalogue or a "likes" counter is fine with this.

Amazon's own Dynamo paper made eventual consistency famous precisely because, for a shopping cart, staying available during a failure matters more than every read being perfectly fresh. The engineering skill is matching the model to the data: strong for money, eventual for "customers also viewed".

Coordination trick 1: load balancing

Concurrency only helps if work is spread evenly. A load balancer sits in front of your servers and decides where each request goes. Common strategies:

• Round robin: hand out requests in rotation. Simple, ignores how busy each server is.
• Least connections: send to the server currently handling the fewest requests. Adapts to uneven work.
• Consistent hashing: map each key to a server so the same user or key keeps landing on the same node, which is vital for caches and sharded data.

PhonePay redistributes transaction load dynamically so no single node becomes the bottleneck during a festival spike.

Coordination trick 2: consensus

Sometimes nodes genuinely must agree on one value: who is the leader, what order did events happen in, is this transaction committed? Getting independent machines to agree despite failures is the consensus problem, solved by algorithms like Raft and Paxos. They work by requiring a majority (a quorum) to accept a value before it counts, so the system can lose a minority of nodes and still make a single, correct decision. Every system that offers strong consistency has a consensus protocol beating at its heart.

Coordination trick 3: idempotency

In a network, messages get lost and retried. If PhonePay's "transfer 500 rupees" request times out, the client resends it. Without care, you just sent 1000 rupees. The fix is idempotency: design operations so that doing them twice has the same effect as doing them once, usually by tagging each request with a unique ID and ignoring duplicates. Idempotency is what makes "just retry on failure", the backbone of reliability, safe.

Seeing concurrency in code

Here is the shape of concurrent processing: fire off many tasks and let them run together instead of one after another.

const tasks = [1, 2, 3, 4];

const process = task =>
  new Promise(resolve =>
    setTimeout(() => resolve(`Processed task ${task}`), 1000)
  );

// All four run together and finish in ~1s, not ~4s
Promise.all(tasks.map(process)).then(results => console.log(results));
// ["Processed task 1", "Processed task 2", "Processed task 3", "Processed task 4"]

Four one-second tasks finish in about one second, not four, because they overlap. Scale that idea from four tasks on one machine to millions across a cluster and you have the engine behind Amazon and PhonePay.

The challenges you sign up for

Concurrency and replication buy speed and resilience, but the bill comes due as:

1. Network latency and partitions: messages are slow and sometimes never arrive, forcing consistency-versus-availability choices.
2. Data consistency: keeping replicas in agreement is the central difficulty, which is why consistency models exist.
3. Partial failure: some nodes are down while others are up, and the system must behave sensibly in that in-between state.
4. Security: more nodes and more network mean a larger attack surface, critical for a payment system like PhonePay.

Takeaway

Distributed systems are fast because they do enormous amounts of work concurrently. They are hard for exactly the same reason: concurrency plus replication forces you to define what "consistent" means and to coordinate independent machines with load balancing, consensus, and idempotency. Master those ideas and the behaviour of systems like Amazon and PhonePay stops looking like magic and starts looking like a set of deliberate, understandable choices.