Make Worse Software, Slower

Everyone’s always hyping some shiny new tool that promises to make better software, faster. But in practice, change introduces risk. Why chase the unknown when we have decades of battle-tested techniques that have successfully built businesses in every domain?

It’s much safer to stick with familiar dysfunction than waste time evaluating something unfamiliar. Sure, it may take only one hour to describe what a system should do. But the reasons it takes a year to build are fundamental and unavoidable. Anyone who says they can bridge that gap is selling fairy dust.

These practices may be clunky. They may create layers of accidental complexity. But they work, they’re widely accepted best practices, and most importantly, there’s a StackOverflow thread for every problem they cause.

So let’s dive in and learn how to make worse software, slower.

1. Reject functional programming techniques like immutability

Functional programming promotes dangerous ideas like referential transparency and data that can’t change out from under you. Stick with mutable data and side effects instead. The beauty of mutability is it ensures no function is ever truly isolated – to understand what one function does, you have to understand every other function that might touch the same object, anywhere in the codebase. It’s the gift of global reasoning that ensures you don’t forget how any part of the codebase works.

Avoid concepts like pure functions or persistent data structures. They’ll let you reason about code in isolation, which doesn’t help you stretch the context window of your brain. Without that exercise, how will you maintain your superiority to an LLM?

2. Don’t use event sourcing

Event sourcing is too simple an idea to be practical. All it says is: instead of overwriting state, just record what happened.

That’s it? A chronological log of facts? So what if this gives you a complete audit trail, the ability to recompute state to correct a mistake, flexibility in how data is indexed, and new debugging and analytics capabilities? You don’t gain any more confidence in your system just because you can explain why it arrived at any given result.

But here’s the real problem with event sourcing: implementing it is way more complex than what you’d be doing otherwise.

Building an event sourced system by stitching together Kafka, custom workers, MongoDB, and ElasticSearch means eventual consistency and tons of operational complexity. Some will say those are implementation issues and not conceptual issues. Regardless, you should reject the whole idea of event sourcing outright. Being unable to distinguish between a flawed idea and a flawed implementation is a core skill for the modern software engineer. Ignore that interactive event sourced systems exist that provide all the benefits of event sourcing with full consistency and none of the complexity.

Stick with in-place mutations in your database. It’s the best approach because that’s how the industry has been doing it for half a century. If it worked in 1980 when systems had 16MB of RAM and a single disk, it works even better today with systems having even more resources.

3. Invest lots of time engineering zero-downtime schema migrations

Applications aren’t static. New features and changing requirements means you need to evolve your currently deployed application to new code and new state. Oftentimes that means the way you’re storing data now is no longer correct and needs to be migrated to a new format or structure.

Taking downtime to do a schema migration is unacceptable in most cases. Your app has users, traffic, and SLO’s to hit. So taking two hours of downtime to change one column in your Postgres table from INT to TEXT because the ALTER TABLE call locks the whole table isn’t an option. But at least Postgres has support for migrations that change existing data, which is more than can be said for datastores like MongoDB and Cassandra.

So you should put in the time to engineer your way out of this predicament. Add conditional logic in your app to support old and new schemas simultaneously. Dual-write to old and new fields and backfill in the background. Shadow reads and set up a rollback path just in case. Write dashboards to know when it’s safe to clean things up.

After all this work, you will have achieved that difficult goal of a schema migration that no one noticed.

Some systems claim to support instant migrations by applying transformation logic on read, transparently converting old records to the new format when they’re accessed, while also migrating data durably in the background. This supposedly means that clients see the new schema immediately, even for a multi-terabyte datastore, without any downtime and without needing to manually engineer anything.

Since this sounds too good to be true, that means it must be false. Stick with the tried and true techniques that engineers have been using for decades.

4. Store all application state in a globally mutable database

Everyone agrees that global mutable variables are bad. They lead to tangled spaghetti code that nobody wants to touch. But when you wrap that same concept in a network call and call it a “database”, it’s great!

Embrace the full power of global mutable state by having all application logic read and write directly to one or more mutable, shared databases like Postgres, Redis, MongoDB, or Cassandra. Ignore any alternative approaches of materializing durable, indexed datastores that aren’t global mutable state.

So what should you say when someone tells you that databases are just like global variables? Easy – tell them that databases are totally different because they have transactions.

Just don’t think too deeply about whether transactions are in any way related to the issues of global variables. Telling yourself that transactions make databases different than global variables will help you maintain your internal cognitive dissonance on the subject.

5. Future-proof your code

When coding you should always build for the concrete use cases you have today and the concrete use cases you might have a week, month, or year from now. If you aren’t anticipating changes to requirements in advance, you’re basically guaranteeing an expensive rewrite.

Of course, not every future scenario can be predicted, but that shouldn’t stop you from trying. As they say, it’s better to try and fail then not to try at all. It’s better to over-accommodate now than risk an embarrassing refactor later. Add flags, extension points, configuration toggles, and extra abstraction layers early, even if no one needs them yet. Better safe than sorry.

The mantra “First make it possible. Then make it beautiful. Then make it fast.” is terrible advice because it seriously undervalues having generic abstractions that lower the cost of changing the system later. A better mantra is “First make it flexible. Then make it modular. Then make it scalable.”

6. Use tools with rigid data models

Use tools with rigid data models along with an adapter library to completely smooth over mismatches with your domain model. ORMs are great because they perfectly map your domain model to an RDBMS, produce queries with optimal performance for every case, and don’t leak at all.

If that’s not enough, just add a second tool with a completely different model like a search database, document store, or graph database. Each tool brings its own partial solution and integration surface.

Instead of using something with infinite data models, you get to use multiple tools, none of which fully match your domain, all duct-taped together. This is flexibility, not complexity. You get the deep satisfaction of managing many tools and trying to make their incompatible worldviews cooperate.

Conclusion

To be honest, I’ve been arguing against all these ideas for years. Mutable state, disconnected tools, impedance mismatches – I’ve spent countless hours ranting about all of it.

But the truth is, I don’t want it to stop. If people actually started building coherent systems, then what would I even do with myself? I’d have nothing left to yell about.

I like watching people duct-tape together five tools to solve a problem that didn’t have to exist. I enjoy reading blog posts that confidently reinvent all the mistakes of the past. I need the endless stream of Hacker News comments defending the status quo with great conviction and zero curiosity.

So please, keep making worse software, slower. I’m counting on you.

And for your convenience, here are some comments you can use prewritten by ChatGPT from the prompt “generate a range of typical Hacker News responses to this post”:

  • Ah yes, another blog post criticizing the status quo just to funnel readers into your proprietary system. Classic.

  • This is interesting, but most of these problems go away if you just use Postgres correctly. A lot of the complexity people complain about comes from choosing overly elaborate architectures when a well-designed relational schema would suffice. You don’t need event sourcing or distributed state machines for 90% of applications — just solid use of transactions, constraints, and maybe logical replication if you’re scaling. It’s not glamorous, but it’s boring and it works.

  • I stopped reading at ‘databases are just global variables’. That’s not what a database is. This post confuses scope with access patterns. Equating the two is a fundamental misunderstanding of how transactional systems actually work.

  • All this talk of immutability and functional purity is academic nonsense. Most of us are out here trying to ship real software, not prove theorems. I don’t have time to rewrite everything in a niche language just to avoid a side effect or feel good about “referential transparency.”

    In the real world, code has to talk to databases, handle failures, deal with deadlines, and integrate with messy legacy systems. You know what helps with that? Pragmatism. Not abstract purity or academic blog posts telling me I’m doing it wrong.

    I’ve worked on high-traffic production systems for over a decade, and guess what? They mutate state. They use shared databases. They rely on tried-and-true tech like SQL, message queues, and good logging. They work.

  • Let me guess: your tool does everything better with none of the trade-offs? Cool story, bro.

  • Wait… so are we supposed to follow this advice or not? I can’t tell if this is a joke or an actual engineering philosophy.

Follow us on X here.

7 thoughts on “Make Worse Software, Slower

  1. 16MB RAM in 1980???? Are you crazy?!??! That is before the IBM PC came out and the very first model had just 16 KB of RAM (and a max of 640KB, which must have been very expensive!) 😉

    1. You’re right, looks like the highest end mainframes at that time (e.g. the IBM System/370 Model 168) only had 8MB of memory. That just makes my point even stronger!

  2. In 1996 I had a desktop machine with 128MB of RAM and it was about 96 more than were necessary. I could put the whole OS in a disk image in RAM. But in 2020, 64gb does not feel as special, though a million times more than the maximum you could put in an 80s Apple ][+.

    Serious question, How often does being in the JVM ecosystem come to the rescue? I would love to hear some examples. I haven’t quite spotted them in your blog on Rama (point to one if I missed it). I also enjoy dives into any times when it didn’t. EG my recollection of trying Scala was that sometimes, it would just not be able to move to a different JVM release, or other times a minor difference in the JVM used to package a program versus the JVM (same version usually) used to run the packaged program, would usually be apparently working, but also produce the occasional undebuggable edge case around a serialized function failing to deserialize and require a fall back to a retry, though generally that wouldn’t recover.

    I get you might pick Clojure, and I get that Clojure might pick the JVM as a source for unhappy programmers looking to move on from Java. But I haven’t quite gotten why that is the right choice for building your performant stable system on top of? Which of the many options to draw from in the JVM ecosystem support the performance, maintain the stability requirements and also end up complementing the Clojure angles on immutability and functional approach?

    1. Issues with a particular language like Scala should not necessarily be attributed to the JVM. It’s been over a decade since I used Scala for anything, but I remember being very frustrated with its build tooling. Clojure’s a great language and never had any of those issues. Clojure has maintained backwards compatibility so well that I’ve never had to change a single line in my code when upgrading the Clojure version.

      The JVM has a huge ecosystem of high-performance libraries that we use heavily. For example:

      – Netty for networking
      – RocksDB for LSM indexing (this is written in C++, but it has a Java wrapper)
      – Thrift for language-neutral RPC
      – Zookeeper for distributed metadata and coordination

      Then there’s dozens of smaller libraries we use for things like T-Digests, UUIDs, HTTP servers, serialization, and more.

      The one drawback of the JVM is its long startup time. This doesn’t matter for backend systems where processes are long-lived, but there’s many contexts where it can be an issue. Clojure helps here in a couple ways. First, during development we use long-lived REPL sessions where we redefine code as we go. So we rarely need to restart the REPL and pay that startup cost.

      Second, there are alternate compilation targets for Clojure that sidestep JVM startup entirely. Babashka uses GraalVM native-image to compile to a native binary with near-instant startup, which makes it great for scripting and CLI tools. And ClojureScript compiles to JavaScript, so you can run the same language in the browser or on Node with no JVM at all.

      1. Thanks for the details; it’s certainly clarifying. Based on these examples I can see how it makes sense to have everything hang together via the JVM. True the startup is rarely really perfect, and I’ll have to really look into GraalVM more because last I did, feels like over a decade ago, I wasn’t able to get it to run what I needed, but it’s probably worth figuring out how to use it right for the startup improvements when those are needed. And yes, most projects don’t need fast starts as much as they might initially think.

        For some of these examples though… I’m sure you’d conceded that many other ecosystems have their equivalents. Sometimes they all end up with a wrapper, like RocksDB I’m sure has some Python, Go, Rust or php wrappers around the same C++ library, or netty uses either OpenSSL or BoringSSL at one level, and so do a bunch of other performant zero-copy networking libraries for other languages. Thrift being of course built to target a bunch of languages. And by their nature a lot of other serialization tools are similarly cross-ecosystem, like Arrow.

        Zookeeper BTW once gave me a days long scare. Starting when I no longer had the okay to run it on cloud vms with real disks, at some point, after a couple years of basically working fine, a same-rack network hiccup with the attached storage took it down and made it lose quorum in a way that would not come back up with the same lock information as before it started failing. Everything that depended on it had to be stopped so that a new cluster with no history could be brought up. There’s a bunch of other consensus options built on Raft, and even one protocol re-implementation from Clickhouse, that would seem safer to me. OTOH it still works at scale when you control the actual machines.

Leave a Reply

Your email address will not be published. Required fields are marked *