Chapter 3 – Locality and the Edge
When Data Travels With Your Application
In Chapter 2, we established that physics is undefeated. The speed of light isn’t negotiable, cross-continental round trips cost 100+ milliseconds, and no amount of clever engineering can eliminate the latency tax of distance.
So here’s a radical thought: what if we just... don’t go over the network?
What if the database isn’t a separate service you call across the network, but rather a library you link into your application? What if every query is a function call, not an RPC? What if the “distance” between your business logic and your data is measured in nanoseconds instead of milliseconds?
This is the architecture of extreme locality—where data lives so close to computation that the network might as well not exist. And it’s not a thought experiment. It’s running in production at massive scale, from retail stores in rural areas to oil rigs in the North Sea to satellites in orbit.
The Embedded Database Renaissance
The concept of embedded databases is old. SQLite, the most-deployed database in history, has been around since 2000[1]. Berkeley DB predates it by nearly a decade[2]. But something shifted in the last five years: embedded databases stopped being a niche solution for mobile apps and desktop software and became a legitimate architectural pattern for distributed systems.
Several forces converged:
Edge computing maturity: CDN providers evolved from serving static files to running compute at the edge. Cloudflare Workers, Fastly Compute@Edge, and AWS Lambda@Edge created environments where you could run application logic in hundreds of locations worldwide[3].
Serverless evolution: Serverless functions went from stateless, ephemeral containers to environments with persistent local storage and longer execution windows. Suddenly, you could attach storage to your function and have it survive across invocations.
IoT proliferation: Billions of devices deployed in environments with unreliable or expensive connectivity. These devices needed to operate autonomously, storing and processing data locally, syncing when possible.
Composable application platforms: Systems like HarperDB pioneered the model where the application and database are tightly coupled—not separate services, but a unified runtime where your API endpoints have native, in-process access to a full-featured database[4].
The result is a spectrum of “data travels with code” architectures:
The Architecture Patterns
Pattern 1: Process-Embedded Database (SQLite, HarperDB Embedded)
The database engine runs in the same operating system process as your application. Queries are function calls. Data lives on locally attached storage.
┌─────────────────────────────────┐
│ Application Process │
│ ┌──────────────────────────┐ │
│ │ Application Logic │ │
│ │ (API, Business Rules) │ │
│ └────────────┬─────────────┘ │
│ │ (function call) │
│ ┌────────────▼─────────────┐ │
│ │ Database Engine │ │
│ │ (SQL parser, query │ │
│ │ executor, storage) │ │
│ └────────────┬─────────────┘ │
│ │ │
└───────────────┼─────────────────┘
│
┌──────▼───────┐
│ Local Disk │
└──────────────┘
Query path: ~1-10 microseconds
Network hops: 0
Failure modes: Local disk, process crash
This is HarperDB’s composable application model in its purest form. Your API endpoint can execute SQL queries, NoSQL operations, or vector searches without leaving the process. There’s no network serialization, no connection pooling, no authentication handshake—just native function calls[4].
Pattern 2: Container-Attached Database (Fly.io Volumes, Railway Volumes)
Each container instance gets its own attached volume with a full database. The database is technically a separate process, but it’s on the same machine and network namespace as your application.
┌─────────────────────────────────┐
│ Container / VM │
│ ┌──────────────────────────┐ │
│ │ App Process │ │
│ └────────────┬─────────────┘ │
│ │ (localhost:5432)│
│ ┌────────────▼─────────────┐ │
│ │ PostgreSQL Process │ │
│ └────────────┬─────────────┘ │
│ │ │
└───────────────┼─────────────────┘
│
┌──────▼───────┐
│ Attached Vol │
└──────────────┘
Query path: ~0.1-1 millisecond
Network hops: 0 (localhost)
Failure modes: Volume failure, container crash
Fly.io pioneered this model for globally distributed apps. Deploy your app container to 20 regions, each gets its own PostgreSQL instance on an attached volume. Every region is autonomous[5].
Pattern 3: Distributed Objects (Cloudflare Durable Objects)
Each “object” is a singleton instance with exclusive access to its own persistent storage. The runtime guarantees only one instance exists globally at any time.
User Request (Tokyo)
│
▼
┌──────────────────────┐
│ CF Edge (Tokyo) │
│ ┌────────────────┐ │
│ │ Durable Object │ │
│ │ Instance │ │
│ │ (Shopping Cart)│ │
│ └───────┬────────┘ │
│ │ │
│ ┌─────▼──────┐ │
│ │ Local KV │ │
│ │ Storage │ │
│ └────────────┘ │
└──────────────────────┘
Query path: ~0.01-1 millisecond
Network hops: 0 (same isolate)
Failure modes: Object migration, storage failure
Constraint: Single instance per object ID
Cloudflare Durable Objects take this further—each object is strongly consistent because there’s only ever one instance. Need to increment a counter? The object handling that counter is the only one that can modify it[6].
Where This Architecture Wins
There are environments where extreme locality isn’t just optimal—it’s the only viable option.
Scenario 1: Intermittent Connectivity (Retail, Field Operations)
A retail store in rural Montana loses internet for three hours. With a traditional client-server architecture, the point-of-sale system is down. Transactions halt. Customers leave. Revenue is lost.
With an embedded database, the store operates normally. Transactions are recorded locally. Inventory is updated locally. When connectivity returns, the local state syncs to central systems. The network is an optimization, not a requirement.
This pattern is common in:
Retail point-of-sale systems (Shopify POS, Square)
Field service applications (utility workers, medical devices)
Maritime and aviation systems (ships, aircraft)
Military and emergency response (where connectivity is never guaranteed)
Scenario 2: Extreme Scale-Out (IoT, Sensors, Edge Inference)
You’re running inference models on 50,000 security cameras. Each camera generates 10 predictions per second—500,000 predictions/second globally. Sending all of this to a central database would require:
500k writes/second to handle
Massive bandwidth costs (assuming 1KB per prediction: 500 MB/second = 1.3 PB/month)
Central database that can handle write amplification across regions
Complex failure handling when network partitions
Or: each camera has an embedded database. It stores predictions locally. It runs local aggregations and anomaly detection. Only interesting events (threats detected, system failures) are sent to central systems. You’ve reduced your central database load by 99% and eliminated the network as a bottleneck.
This works for:
IoT sensor networks (temperature, humidity, vibration monitoring)
Edge ML inference (computer vision, anomaly detection)
Autonomous vehicles (must operate without connectivity)
Industrial control systems (manufacturing, utilities)
Scenario 3: Geographic Compliance (Data Residency Requirements)
You have customers in the EU who demand that their data never leaves the EU. Traditional approach: deploy a full multi-region database cluster in EU regions, replicate between them, and ensure routing keeps EU customers’ requests in EU regions.
Embedded approach: each EU customer’s data lives only on EU-deployed application instances. The data physically cannot leave the EU because it’s not networked to non-EU systems. Compliance is architectural, not procedural.
The Operational Challenges
Extreme locality eliminates network latency, but it doesn’t eliminate complexity—it relocates it. Here are the problems you’re trading for:
Challenge 1: Write Amplification
Let’s model a simple scenario. You have a composable application platform with 10 nodes, each with an embedded database. You want all nodes to have access to all data so any node can serve any request.
Write amplification factor: 1 write becomes 10 writes (one per node).
Now scale to 100 nodes. Your write amplification factor is 100×. A system handling 10,000 writes/second at the application level is actually handling 1,000,000 writes/second at the storage layer.
This has cascading effects:
Storage throughput: Modern SSDs can handle ~100k IOPS. With 100× write amplification, your effective write capacity drops to ~1k application-level writes/second per node.
Storage wear: SSDs have finite write endurance. Write amplification accelerates wear. A drive rated for 5 years of life at 10k writes/second will last 18 days at 1M writes/second.
Bandwidth cost: Each write must be replicated to all other nodes. For 100 nodes with 10KB writes at 10k writes/second:
Per-node bandwidth: 100 MB/second outbound
Cluster bandwidth: 10 GB/second intra-cluster
Cost: ~$50k/month just for inter-node replication traffic
The naive solution—replicate everything everywhere—doesn’t scale past a certain cluster size. You need intelligent sharding.
Challenge 2: State Reconciliation
If every node has local state and can accept writes independently, you have a distributed consensus problem. Two nodes modify the same record simultaneously—which write wins?
Last-write-wins (LWW): Simple but loses data. If Node A sets inventory = 100 and Node B sets inventory = 95, one of those writes is discarded.
Conflict-free Replicated Data Types (CRDTs): Mathematically provable eventual consistency. Works well for counters, sets, and registers. Falls apart for complex transactions[7].
Vector clocks: Track causal relationships between updates. Can detect conflicts but can’t resolve them automatically. You need application-level conflict resolution logic[8].
Consensus protocols (Raft, Paxos): Provide strong consistency but require synchronous coordination across nodes—which reintroduces network latency and violates the “local-first” principle[9].
Real-world example: HarperDB’s clustering uses a gossip protocol for schema synchronization and supports both eventually consistent and transactional consistency models depending on the operation. But this requires careful design—some operations can be local, others must coordinate[10].
Challenge 3: Schema Evolution
You have 1,000 embedded database instances running in the field. You need to add a new column to a table. How do you roll out the schema change?
Synchronous migration: Take the whole fleet offline, update all schemas, bring it back online. Not viable for always-on systems.
Rolling migration: Update instances gradually. But now you have mixed schema versions. Your replication protocol must handle records with different shapes. Your application code must handle both old and new schemas simultaneously.
Backward-compatible migrations only: Only add nullable columns, never remove or rename. This works but constrains your data model evolution forever.
Schema drift is insidious. A node goes offline for a week. It comes back. It’s seven schema versions behind. Does it:
Refuse to participate until manually updated? (safe but operationally painful)
Automatically migrate its local schema? (risky—what if migration fails?)
Participate with the old schema and drop fields it doesn’t understand? (data loss)
There’s no perfect answer. Each system makes different trade-offs.
Challenge 4: Observability and Debugging
With a centralized database, debugging is straightforward. Something went wrong? Query the database. Check the logs. Examine the replication lag.
With 1,000 embedded instances:
Which node has the canonical version of this record?
Which nodes have stale replicas?
Why did replication fail between Node A and Node B?
What’s the cluster-wide query performance?
You need distributed tracing across all nodes, consensus on cluster health, and tooling to aggregate logs and metrics from a fleet of autonomous instances. This is Kubernetes-level orchestration complexity, but for databases.
The Intelligence Problem
Here’s the fundamental tension: extreme locality is powerful when you can predict which data should live on which nodes. But prediction is hard.
Consider an e-commerce application:
User sessions should be local to the user’s region (predictable)
Product catalog should be everywhere (predictable)
Inventory counts should be... where?
If you replicate inventory everywhere, you have write amplification. If you shard by product ID, cross-shard queries (showing multi-product carts) require network calls. If you shard by warehouse, you’ve just re-created a distributed database.
The “right” answer depends on access patterns:
If most queries are “show me inventory near me,” shard by geography
If most queries are “show me all inventory for product X,” shard by product
If access patterns change dynamically (flash sale on a product), static sharding fails
This is why HarperDB introduced sub-databases and component-level data placement—letting developers specify which data lives where, rather than forcing a single clustering strategy[11]. But this pushes complexity onto the developer.
When Locality Alone Isn’t Enough
There are workloads where extreme locality fundamentally doesn’t work:
True global state: You’re building a multiplayer game. Players in Tokyo and London are interacting in real-time. They need to see each other’s actions immediately. No amount of local-first design can eliminate the need for cross-region coordination.
Regulatory global access: Your EU customer’s data must stay in the EU, but your US compliance team needs read access for audit purposes. You can’t keep the data purely local—you need controlled, auditable replication across regions.
Cross-entity transactions: User A in Tokyo transfers money to User B in London. This is a transaction spanning two geographic regions. If both users’ data is local to their regions, you need distributed transaction coordination—which reintroduces all the latency and consistency challenges you were trying to avoid.
Scale beyond local capacity: Each node’s local storage is finite. If your dataset grows beyond a single node’s capacity, you must shard. And once you’re sharding, you’re no longer purely local—some queries will span shards and require network calls.
The Synthesis Ahead
Extreme locality is not a panacea. It’s a point on the spectrum with clear advantages and clear limitations.
What it proves, though, is that the network is optional for many workloads. The ~100ms tax of cross-region queries isn’t inevitable—it’s a choice. If you’re willing to accept the operational complexity of distributed local state, you can eliminate network latency entirely for reads and reduce it dramatically for writes.
But “eliminate network latency” and “accept operational complexity” are two halves of a trade-off. The question is: can we get the benefits of locality without the operational burden? Can we build systems that intelligently place data—sometimes local, sometimes distributed—based on actual access patterns rather than upfront architectural decisions?
In Chapter 4, we’ll examine the opposite end of the spectrum: global distributed databases that explicitly embrace distance and coordination. Systems like Google Spanner and CockroachDB that say “yes, we’re paying the physics tax, but in exchange we get global strong consistency.”
Then, in later chapters, we’ll explore the middle ground: systems that dynamically migrate data based on where it’s being accessed, that optimize placement continuously, that try to give you local latency where possible and coordinated consistency where necessary.
Because ultimately, the goal isn’t to eliminate distance—it’s to make distance matter less.
References
[1] D. R. Hipp, “SQLite: A Self-contained, Serverless, Zero-configuration, Transactional SQL Database Engine,” 2000. [Online]. Available: https://www.sqlite.org/
[2] M. A. Olson et al., “Berkeley DB: A Retrospective,” Proc. 25th International Conference on Data Engineering, pp. 1-10, 1999.
[3] Cloudflare, “Cloudflare Workers: Deploy Serverless Code Instantly Across the Globe,” Technical Documentation, 2024. [Online]. Available: https://workers.cloudflare.com/
[4] HarperDB, “Composable Application Architecture,” Technical Whitepaper, 2023. [Online]. Available: https://www.harperdb.io/
[5] Fly.io, “Fly Volumes: Persistent Storage for Distributed Applications,” Technical Documentation, 2024. [Online]. Available: https://fly.io/docs/volumes/
[6] Cloudflare, “Durable Objects: Strongly Consistent Coordination at the Edge,” Blog Post, 2020. [Online]. Available: https://blog.cloudflare.com/durable-objects-easy-fast-correct-choose-three/
[7] M. Shapiro et al., “Conflict-free Replicated Data Types,” Proc. 13th International Symposium on Stabilization, Safety, and Security of Distributed Systems, pp. 386-400, 2011.
[8] D. S. Parker et al., “Detection of Mutual Inconsistency in Distributed Systems,” IEEE Transactions on Software Engineering, vol. SE-9, no. 3, pp. 240-247, 1983.
[9] D. Ongaro and J. Ousterhout, “In Search of an Understandable Consensus Algorithm,” Proc. 2014 USENIX Annual Technical Conference, pp. 305-319, 2014.
[10] HarperDB, “Clustering and High Availability Architecture,” Technical Documentation, 2023. [Online]. Available: https://docs.harperdb.io/
[11] HarperDB, “Sub-databases and Component Architecture,” Product Documentation, 2024. [Online]. Available: https://www.harperdb.io/product/
Next in this series: Chapter 4 - The Global Cluster Paradigm, where we’ll examine systems that explicitly embrace distance and coordination—and discover what strong global consistency actually costs in the real world.