by Learn what database sharding is, how shard keys and routing work, and when sharding makes sense for scaling data, storage, and write throughput.
Updated August 2026 — full tutorial restored for this URL.
TL;DR
-
Sharding splits one logical dataset across multiple databases so each shard stores only part of the data.
-
Partitioning and sharding differ: partitioning typically happens within one database, while sharding distributes data across separate database servers.
-
Shard keys matter because a poor key can create hotspots and uneven workloads across shards.
-
Sharding can scale writes and storage, but it makes joins, transactions, indexing, reporting, and operations more complex.
-
Sharding is not always necessary. Try vertical scaling, read replicas, caching, and archival before introducing it.
Sharding splits one logical dataset across many databases (shards) so each node holds only a slice of the rows. It is horizontal scaling for data volume and write throughput.
1. Partitioning vs sharding
- Partitioning often means splitting tables inside one database engine.
- Sharding usually means separate database servers, each owning a key range or hash bucket.
2. Shard keys
Pick a key that appears in most queries (e.g. customer_id). Bad keys create hotspots (everything lands on one shard).
- Hash(
user_id) % N - Range: user_id 1–1M → shard A, 1M–2M → shard B
- Directory/lookup service maps key → shard
3. Routing example
def shard_for(user_id: int, n: int = 4) -> int:
return user_id % n
# app connects to shard DSN[shard_for(user_id)]
Cross-shard joins and transactions become hard — design aggregates to live on one shard when possible.
4. Benefits and costs
- + scale writes/storage beyond one machine
- + isolate noisy tenants
- − operational complexity, rebalancing, uneven load
- − global secondary indexes and reporting get harder
5. When not to shard
Exhaust vertical scaling, read replicas, caching, and archival first. Shard when a single primary cannot meet growth and you have a clean shard key. Many teams never need it.
Final Thoughts
Database sharding can be a powerful way to scale when a single database can no longer handle growing data volumes or write workloads. By distributing data across multiple databases, teams can push beyond the limits of one machine while improving workload isolation.
But sharding also introduces significant complexity. Choosing the right shard key, routing requests correctly, handling cross-shard operations, and eventually rebalancing data all become part of the architecture.
For that reason, sharding should be a deliberate scaling decision, not a default design choice. Exhaust simpler options first, then shard when the growth problem is clear and you have a shard key that matches how your application accesses its data.
