Complexity I: The Inner Platform Effect

Software engineering isn’t really about code. It’s about managing complexity - deciding what control belongs with the system, what belongs with the user, and how much friction sits on either side.

The uncomfortable truth is that a lot of software fails to manage complexity well. Too often, we rebuild the tools we already have, just with shinier UIs and higher maintenance costs. The result is layers of incidental complexity for users and engineers alike, that really belie the true power of great software.

In this series, I want to explore a simple but potentially confronting idea: much of the software we build doesn’t actually add much, or sometimes even any, value beyond the underlying tools it’s been built on.

This may not be surprising to people who worked in the world of tech startups during the height of the SaaS era (2010s, give or take). Many tech startups began with “xyz industry is still using spreadsheets to manage things” as their key opportunity. Many tech startups were kicked off by less experienced engineers and designers (in fact, often no designer). This leads to a world where a lot of software simply reimplements those spreadsheets, instead of really leveraging the power to wrangle complexity that well designed software can have.

This is the basis for my oft-quoted, only-half-tongue-in-cheek, rather curmudgeonly remark that “most software should have remained a spreadsheet”.

But this isn’t just a rant about bad products. It’s about the choices we make as engineers, and how we can do better. In this multi-part series, we’ll explore:

• What do we mean by complexity?
• What are some ways we can identify complexity?
• How do we decide who should wear the complexity? Us, or users?
• How do we ensure we are designing software that actually adds value, instead of just adding layers?

First, we’ll look at one simple example, and one complex example, of the Inner Platform Effect. This will introduce us to the idea of complexity and how it might be either managed or “passed on” by software.

For a gentler introduction with some common, concrete examples, consider pre-reading another article of mine Solve Your Users’ Problems.

After I share these real world examples, it’s expected there may still be a bit of confusion about what exactly is meant by complexity. Rest assured, the next article will pick up where we left off, and hopefully help answer this question.

Simple: Reinventing Spreadsheets and Databases

I’d like to start with something I think is very important: spreadsheets and databases are some of the most powerful tools ever created. The people who design and work on these are incredibly talented, and undoubtedly have brought more value to the world than almost every other software engineer in existence, including me.

Why is it that they produce so much value? In short, it’s about the complexity they effectively manage:

1. Representational Complexity

They effectively bridge the gap between how humans think about data, and how machines store and operate on that data.

• Human: I have a table of customers, and their details
• Machine: I have a huge string of binary data
• Spreadsheet/Database: I organise blobs of binary data into tables, rows and columns

Spreadsheets and relational databases both give people an effective, intuitive cognitive model of structured data that mirrors how they reason - lists, hierarchies, relationships - without exposing memory layout, serialization, or file structures.

2. Storage and Retrieval Complexity

They hide the pain of persistence, indexing, and retrieval in a way that’s performant and efficient.

• users (usually) don’t think about file offsets, pages, or memory boundaries
• users (usually) don’t think about indexing, caching, transactions, and recovery

Up to a limit that most users will never find, they turn the enormous complexity of persisting large amounts of structured data into “just there”.

3. Computation and Dependencies

They can manage complex relationships between pieces of data automatically, including computationally derived data.

• Spreadsheets: maintain a dependency graph of formulas, recomputing them in order when necessary
• Databases: plan queries, enforce constraints, maintain transactional consistency, etc.

The user says =SUM(A1:A10) or SELECT SUM(...) FROM ... GROUP BY ... and the system decides how to compute this.

Both absorb the algorithmic complexity of computing dependencies, propagating changes, atomicity, etc. by turning declarative intent into executable logic.

4. And so on

There are many other things we could say about specific spreadsheets and databases here - I’ll stop there in the interest of focus.

But this is why I think that spreadsheets and databases are some of the most powerful software ever built - because they very effectively abstract away a tonne of complexity.

Barring the extreme scale where these abstractions start to break down, which few users will ever genuinely meet, they manage nearly every kind of complexity between a human’s idea (“I have some data”) and a computer’s reality (“I have bytes on a disk”).

And that is why, so often, many pieces of business software end up orbiting them - almost always built on top of them, and often secretly reinventing them.

Jira is an expensive spreadsheet

Ok, let’s take a look at Jira - I will note here I have a deep grudge against Jira, but I think my point still stands. Jira is the perfect example of a piece of software modelling its problem domain rather than providing solid solutions, and this is why it’s often that people who don’t like it will revert to a spreadsheet. Because, secretly, that’s all Jira really is.

The problem Jira claims to solve is “project and task tracking”. So what are its entities? Well, projects and tasks… and we have ways to create these, edit them, update them, link them. When you create a project, you create a row in a database. When you create a task, it’s a row in a spreadsheet. Assigning work? Well, ok, here - have a table of users, who can be linked to the task rows.

There are two major “sins” here:

• modelling the problem domain, instead of providing solutions
• providing too much customisation, which I will leave aside for the moment ¹

Let’s focus on the first issue here, which is modelling the problem domain instead of providing solutions. This is a very natural way for engineers to think, especially when we don’t start at the end and focus on building value, rather than code. Unfortunately, it leads to reimplementing the tools we built the software on top of, which is more-or-less what the Inner Platform Effect describes.

In this case, our problem is “task tracking”, and Jira’s core models are a “project” and a “task” - this is what I mean when I say it simply models its problem domain.

You might be thinking “well how else do you do it? why is this such a big deal?” - I understand that why I call this is a “sin” is perhaps not clear yet. I expect it to take the remainder of this series to elucidate my thinking here, because it’s not common, and it’s not straightforward. So you’ll have to bear with me!

But anyway, here’s the two major pieces of evidence that Jira is just reimplementing a database:

1. It exposes a sheet view
2. It implements its own query language, JQL

So, according to their own marketing material, the spreadsheet view “is the power of Jira blended with the simplicity and usability of [a] spreadsheet” and JQL is “the most powerful and flexible way to search for your issues in Jira”.

By their own admission, the key concepts forwarded by spreadsheet software and relational databases are more simple and usable, and more powerful, respectively, than anything else Jira offers… And so Jira is simply “passing on” the complexity of working with a database, instead of effectively managing it.

If, at this point, you still think there’s no chance that Jira is simply a database reimplemented in a very expensive, very clumsy way, with a slapdash coat of paint and a huge marketing and sales engine behind it, then I’m not sure you’ll find anything useful in the rest of this discussion. Best click away now.

Complex: Reinventing Operating Systems

When React 16 introduced fibers, it was explicitly a rewrite of its reconciliation algorithm to support incremental rendering - that means breaking rendering work into small, interruptible chunks.

In essence:

• the original React render was synchronous: once it started, it blocked the main thread until finished
• the fiber architecture made rendering cooperative: each “unit of work” could yield back to the browser, allowing updates to stay responsive

Conceptually, React reimplemented a lightweight task scheduler inside its framework. Each fiber stores its own continuation (what work to do next), parent and sibling pointers, and a notion of priority. That’s threading (just without preemption).

The browser already implements its own complex multitasking environment, so React is now effectively a simulation of multitasking inside an environment that’s already multitasking.

Underneath, the OS kernel is already managing thread scheduling, preemption, task priority, context switching - each layer reimplements this concept of cooperative scheduling to achieve multitasking in a way that prioritizes its own goals, because the layer underneath isn’t quite doing the job the way they want.

This is the Inner Platform Effect in action:

• React builds a “mini operating system” inside the browser
• the browser is already a “mini operating system” inside the OS

It’s task schedulers all the way down… 🐢

Is this bad?

I would say no, it’s not bad engineering. You may disagree - like I say, this one is complex.

There may be a case to consider that the requirements of web applications have changed so much so fast that we’re being forced into building layers on top of existing layers that should ideally be performing these functions already. But I think this actually points to a problem of human coordination and collaboration at immense scale, rather than specifically an engineering problem - is the alternative for the React team to rethink the Internet? Maybe!

It may highlight a fundamental mismatch between React’s architecture and the runtime environments it’s expected to run in - and that perhaps this is working around issues with React, specifically, and not more generally with the ever growing complexity of web applications. I’m not enough of an expert to say whether this is the case, but the Inner Platform Effect visible here is a “smell” we shouldn’t ignore.

The key for me is in the intent - the React team decided to take on additional complexity consciously and with boundaries, in order to solve for genuine problems their users were facing. That, to me, is good engineering - they accepted complexity on behalf of their users. As long as this decision was made well, and with benevolent and realistic intent to keep managing that complexity indefinitely, then the React team have solved a user problem, and have added value through their software.

Even if there is some ideal world where the browser is the OS, or we are otherwise able to collapse these layers effectively and reduce total complexity over the entire hierarchy, in the domain that the React team controlled, in the local complexity basin they were confined to, this arguably drove down total complexity for their users.

What have we learned?

Recall the goal of this series is to explain the value of software primarily as a measure of the complexity that software effectively manages on behalf of its users.

We’ve started on this journey by exploring the Inner Platform Effect in two ways:

1. when we model our problem domain instead of focusing on solutions, we often simply re-expose our underlying tools (usually a database)
2. sometimes, “reimplementing” parts of our underlying tools can actually reduce complexity for users

In the next part of the series we’ll dive a bit deeper into what complexity actually is, how we can define it, and some ways we can identify it, which will lead us to thinking about how we decide where that complexity should live, and ultimately to how we can ensure we are building software that produces genuine value for our users.

---

Footnotes

1. It may seem like I’m underselling Jira, I don’t think I am. I am deliberately leaving out automation capabilities which is where too much customisation causes problems, however I think this distracts from the point I’m trying to make in this article. I may follow up later. ↩