(updated)
|
4
min. read

Escaping the Network Tarpit

Conrad Hofmeyr

Modern software applications are distributed applications. This is a trend that has accelerated over the past decade. Users want their software and data to be accessible from anywhere, and therefore developers build software for the cloud. And because so much data lives in the cloud, modern software involves constantly moving app state over the network.

Yet networks are imperfect. Even the best modern networks (5G, fiber, Starlink, etc.) run into connectivity lapses, latency spikes and temporary throughput bottlenecks. This is especially unavoidable if you consider that the Internet is a network of networks. Nevertheless, users want it all: they want their software to be fast, reliable, collaborative, and feature-rich. 

Developers therefore spend an inordinate amount of time dealing with the complexities of moving state over the network: Data fetching. Loading state. Caching. Client-side indexing. Cache invalidation. State management frameworks. Optimistic updates. The list goes on.

Carl Sverre has done a great job of describing this as reinventing databases on the front-end:

“In any frontend application of sufficient complexity, engineers will necessarily end up building so many data management features that they are essentially creating a domain specific database. This added complexity is duplicated in each project we work on and takes away from spending time on delighting users and solving business problems.”

We’ve started calling it the network tarpit: It seems simple at first, but quickly becomes endlessly complex. And the more you struggle, the more entangled you get…

Even the top software companies in the world struggle with the problem. If I use Facebook or Google Maps in the elevator at the office or even in certain sections of the city street where I live, I often just get stuck in a loading state.

Enter sync engines

The good news is that sync engines and the local-first movement are changing the paradigm to something better: Instead of reinventing a database in the front-end, what if we put an actual database in the front-end and asynchronously keep it in sync with a database in the cloud?

In the year since the first edition of Local-First Conf, we’ve seen an explosion of developer tools aiming to solve this problem. We believe that sync engines are ushering in a new golden era for app development that’s more productive and more fun.

Different architectural philosophies

With this explosion of tools, there’s also been some patterns emerging in different types of architectures and approaches. 

Some tools merely act as a sync layer between the backend and frontend — like ElectricSQL, Zero and PowerSync. Others replace your entire stack, from backend database to frontend database — like Instant, Triplit, Firebase and arguably Ditto too. Aaron Boodman has described this approach and the trade-offs well, contrasting sync engines with syncing datastores.

We are skeptical that entire-stack products (“syncing datastores”) will win out. For one, it’s not clear that most developers want this. These products generally expect developers to entrust their system-of-record production data to an unproven new database, rather than a mature system like Postgres or MySQL. Entire-stack products also require a mammoth engineering effort. It will take a lot of time and investment for the new upstarts to reach parity with mature databases. 

Some tools allow you to bring your own backend database, but they provide their own vendor-specific frontend data layer. This approach also didn’t resonate with us, because we don’t think that most developers would prefer a nonstandard API surface over a standards-based one (e.g. SQL). A standards-based interface also allows you to bring your own favorite ORM to work with the data, if that’s what you prefer.

For that reason, we designed PowerSync to act as a sync layer between powerful open database technologies on both the backend (Postgres, MongoDB and MySQL) and the client-side (SQLite). As we’ve written about in our 2025 roadmap announcement, using SQLite on the client-side means that developers can benefit from its familiar syntax, powerful functionality, ecosystem (ORMs and extensions), high performance, and maturity. This overall philosophy also led us to make PowerSync itself open-source and source-available. 

We have a strong conviction that the philosophy behind PowerSync is the right one for sync engines to achieve wide adoption. We’re doubling down on our architecture and expanding the product with new major features this year — to help developers free themselves from the network tarpit.