SpacetimeDB—A Yak Shaving Success Story?

SpacetimeDB is a unique "database", written in Rust, that was developed by Clockwork Labs, and a month ago made a bit of a splash with their promotional video.

It's available on GitHub under a BSL 1.1 license. (This is notably not open-source, but instead source-available. More on that later.)

But it's much more than just a database.

In a press release, Clockwork Labs referred to SpacetimeDB as a, "production-ready, all-in-one backend solution for your multiplayer games".

But we find out later in the video that it's not just for video games. Pogly is a "real-time collaborative stream overlay", which lets Twitch streamers offload the management of parts of their stream to their moderators. And it's "powered by SpacetimeDB". It's already being used by some mega streamers, like Shroud.

So who the hell is Clockwork Labs? And what is SpacetimeDB?

Who Is Clockwork Labs

Clockwork Labs is a venture-backed, 40-person game studio that has never released a video game. They've been around since 2019.

Their first release is going to be an MMORPG known as BitCraft, which is still in its development and testing phase, but has millions of players signed up.

As they noted in their launch, this is sort of a crazy thing to do. MMORPGs have historically had massive budgets, enormous dev teams, and long timelines (5-7 years or more).

It might be crazy, but it's working? Sort of?

Here's a blurb from their press release from 2021:

Clockwork Labs has raised $4.3 million in funding for its upcoming unannounced community sandbox MMORPG and is venture-backed by Supercell, Firstminute Capital, 1Up Ventures, Supernode, and Skycatcher. The company is dedicated to building massively-multiplayer “societal games” that are less focused on pure combat and instead encourage social interaction and cooperation between players. This focus has also attracted angel investors such as Hilmar Pétursson, the CEO of CCP Games, the makers of EVE Online, and Unity founder, David Helgason.

We now know that this unannounced MMORPG is BitCraft.

And in June 2022, Clockwork Labs raised another $22 million, in a Series A led by a16z crypto and Andreessen Horowitz.

Despite the fact that they've never released a game as a studio, they do bring a lot of talent from the industry. They've attracted a star-studded team which has tons of experience building distributed systems and blockbuster video-games. This team, at the very least, has convinced investors that their ideas are worth investing in.

And when you get the right team of people together, good things happen.

SpacetimeDB's Origin Story

Honestly, Clockwork Labs was not on my radar whatsoever until a week ago. I'm just a software consultant and I don't really pay attention to the game development world. But the SpacetimeDB launch video really caught my attention.

Tyler Cloutier (founder @ Clockwork Labs), was just hitting home run after home run throughout the whole video. It was a freakin' home run derby.

Tyler's big reveal was that, after 5 years, Clockwork Labs was finally announcing their internal tool which they had been using to beat the odds of game development, and develop their MMORPG without having to spin up their own cloud-server-farm-microservice-hellscape and sell their soul to pay off AWS bills. This is the silver bullet. It lets their devs work on game logic instead of infrastructure logic, saves them money, and is faster than Postgres.

So, in Clockwork Labs' efforts to build this ambitious MMORPG known as BitCraft, they developed SpacetimeDB.

The launch video does a fantastic job of setting up exactly why they needed to build new tooling.

If you've ever tried game development, then you understand that the landscape is full of game engines, visual programming tools, and other programming abstractions. It's difficult to navigate that landscape, and balance the flexibility that custom programming provides vs. the time and complexity of game development vs. out-of-the-box solutions.

At one point in the video, they lay out a cartesian plane and plot tools based on ease of use and how "complete" of a solution they are.

They put SpacetimeDB up there in the top right corner in the easy-to-use, complete solution category. Sounds nice, right?

Then they go into how a database that runs an MMORPG needs to serve three main domains:

1) Server Compute
2) Transactional Storage
3) Real-time Networking

Right there in the middle of the triple venn diagram of video game infrastructure is SpacetimeDB.

These are some big promises they're making!

It was at this point in the video that I started asking myself questions like:

• "Wow, this does not seem like a game studio's video."
• "I wonder how much venture capital has gone into SpacetimeDB vs. their game?"

I don't mean this disparagingly, but Clockwork Labs seems like they've formed a fantastic software development company that's dressed up like a game studio.

But before I fully unpack that thought, let's keep looking at SpacetimeDB and how it actually accomplishes what they say it can.

SpacetimeDB: How Do They Do It?

Clockwork Labs breaks SpacetimeDB into three, main abstractions:

So, first, you write your game backend as a SpacetimeDB module. Right now this has to be Rust or C#.

Tables are relational database tables, but they're not using PostgreSQL, or MySQL, or anything. They have their own, custom-built relational database logic.

You define your tables using the SpacetimeDB ORM (Object Relational Mapping) system.

And then we have the reducers. A reducer is a "stateful, transactional cloud function".

These reducers do pretty much everything—authorization code, game physics logic, rendering, etc.

They also have "subscription queries", which are the real-time networking piece of the puzzle. These are normal SQL, but you subscribe to the data, via some sort of websocket or webhook, and get transactional updates to the data, so you don't need to keep re-querying.

Okay, but we still have to address the main trick that SpacetimeDB employs, and that is by storing your code, alongside your data, in memory.

Usually, your data is separated from your server by the network. So, in order for your server (aka game logic) to access data (aka your database), you need to pass through some flavor of networking. But SpacetimeDB just cuts this out and combines the database and the server.

Combining the Server and Database—Is This The Way?

Okay, so to summarize, SpacetimeDB is a Rust codebase which implements a database and server in one service, and has three main abstractions:

• Relational tables (normal database stuff, which you can interact with via SQL)
• Reducers (which are basically cloud functions / lambdas that live alongside your data)
• Subscription queries (to achieve efficient real-time networking—required for online video games and many other services)

My first question is, well, why do we usually separate the server and the database?

I've worked on systems where the server and the database were closely coupled, and it wasn't very fun. One such application was an image library, which was running an Apache web server and a MySQL database on the same EC2 instance. It wasn't an ideal setup for many reasons, but narrowing in on the server + database combo—they were competing for resources. Long, expensive SQL query? Uh oh, now the web server has no resources. Large influx of image viewers? Now the database is starved for resources. This made resource monitoring very difficult. Also, trying to do careful maintenance, database backups, doing anything that touched the web server—it wasn't impossible, but it was a headache. I've been spoiled by modern cloud architecture and a simple separation of concerns.

So, is SpacetimeDB combining the server and database really a step backwards?

Or maybe the separation of server and database is not as big of a deal in the video game world? How do they make sure that upgrading compute doesn't break the storage layer, or vice versa? Could a bug in the game logic break the persistent state storage?

That is my biggest concern with SpacetimeDB, and I'm very interested to see how real usage of this architecture manifests. It's hard to tell how this would feel, from a DevEx point of view, without having to actually add more compute, or upgrade the OS version.

My other concerns with SpacetimeDB is that everything is custom. Will it miss out on the benefits of the open-source economies of scale? There are so many tools for Postgres, or MySQL, or SQLite, and developers are used to these technologies.

Furthermore, what is the state of vendor lock-in? How difficult will it be to add another technology to the stack? What if you need something like DynamoDB? Do you need to wait until Clockwork Labs adds their version of a key-value store?

Clockwork Labs—a New Cloud Provider?

The end of the promotional video highlights their new "Maincloud" product—a multi-tenant, managed cloud service for running SpacetimeDB applications.

So, is Clockwork Labs still a game development studio? If so, they might be in the running for biggest yak-shaving project ever.

They've made a very, very interesting pivot. After their original mission of building an MMORPG and community-oriented games, 5 years and over $26 million of venture capital later, they've released a cloud platform.

They have their hands in a couple of different buckets. They're developing SpacetimeDB, they have their "Maincloud" platform, and they're still developing BitCraft.

I, for one, am excited to see what these different avenues lead to.

Open Source Status

SpacetimeDB was released under a BSL 1.1. You can see the license here.

BSL == Business Source License. It's relatively new, and somewhat controversial. Although they didn't create the license, you may recognize it from HashiCorp's de-open sourcing of Terraform in August of 2023.

The MariaDB created the license in 2017, and it's pretty common for new databases to adopt (i.e., CockroachDB).

But I am highlighting SpacetimeDB's license because it's likely to lead to some confusion down the road.

For example, the OpenTofu team, which forked Terraform when HashiCorp switched licenses, noted that the BSL is "ambiguous" and "makes it challenging for companies, vendors, and developers using Terraform to decide whether their actions could be interpreted as being outside the permitted scope of use".

This puts users of SpacetimeDB in a similarly ambiguous situation.

Conclusion

This was a fun one to write, and I'm looking forward to seeing how the SpacetimeDB and Maincloud launch continues to shape up.

There's more that I didn't talk about, like how they're working on arbitrary point-in-time rollbacks (aka time-travel from a database perspective?), and how that's part of the origin story of name SpacetimeDB.

If you're employed at Clockwork Labs and want to weigh in, or if you have stories about SpacetimeDB, using it, exposure to it, or coupling of a server + database, I'd love to hear from you.

Thank you for reading,

Nick