November 25, 2022
Over time, as I have progressed through the ranks from software engineer, to senior software engineer, and finally to the tech lead position I currently occupy at Messari, I've accumulated some principles that have given me a particular reputation at work. I'm interested in explaining myself a little bit below, both for the general interest and so I have somewhere to point people to when they ask for my software engineering philosophy or similar.
At the outset, all the usual disclaimers—these principles work well for me and I'm not interested in arguing about them. All I can present to the world is the synthesis of my own experience.
The Unix philosophy—namely, “do one thing well” is the cornerstone of genuine extensibility in programming. Without it, you cannot truly stack modular building blocks together, no matter how many fancy frameworks come to the table. The frameworks of stdout and stdin and the POSIX-compliant shell really work wonders at the small scale.
There is a troubling tendency to try to make life easier in software engineering by relying on bulky libraries with their own unique interfaces that take a while to learn. Sometimes this makes sense—please do not try to reimplement something like React when lots of people have already made React (and it's much easier to hire for the framework than to hire for building a framework)—but often I find it silly. There is a line somewhere that shouldn't be crossed. In Golang, for example, I have always tested via the standard library testing package—I find testify profoundly pointless because testing already does what you want! I am not sure what the point is in making Golang tests “look more like” tests in other languages, as it were, when one can expect any competent Golang programmer to know testing but not necessarily testify and nobody ought to bother with the more niche implementations in testify anyway.
It is much easier to trust one's own code than an import. Just today I was fighting gorilla/mux's built-in CORS handling middleware and decided to reimplement the necessary handler myself, properly, rather than sift through someone else's source to do something simple. Now I have a simple CORS-handling library built solely from the Golang standard library, and it can be reused anywhere!
The point is this: just as you should stitch simple programs together, you should write your own simple libraries when the need arises, or rely on trusted, simple, single-use ones that can be used elsewhere—in other words, don't import something bulky to accomplish something simple. Put another way: do not write or use code that tries to be too multi-purpose, because it will become bulky and hard to read very quickly. Software engineers like to feel smart, but most of the time being clever is a mistake and you're better off writing the simplest, shortest possible solution. This is one of the things people mean when they say not to overoptimize until the need arises, and to “keep it simple, stupid!“
The latest project I've been doing at work is a new user authentication and billing service. I spent a couple weeks hammering out the proper data model—users, teams, subscriptions, audit trails, and so on—and then I wrote a new microservice composed only of an API server, a database operator, and about three clients for the related upstream authentication and billing services. The API server is simple and RESTful, with all its endpoints listed at the top of its first file. The database operator similarly categorizes and lists all possible database operations alongside simple unique integer error codes that attach to each failed query (so it can be obvious which code path failed whenever one does).
And so it is immediately clear to the reader that there is one API server that has these methods, and one database operator that has these methods. One is not lost in excessive abstraction in advance of it becoming necessary, nor must one dig to find a relevant functionality. Occam's razor does wonders for application design.
When I started at Messari early last year, our engineering team was six people. Now we are approaching 40! Most of our original knowledge-sharing practices did not scale and it is thus much harder to keep up with what everyone is doing than it used to be. We still write microservices in a variety of Kubernetes-managed namespaces, and we've formed something of a web of dependencies between them. It's likely individual person sees them all. Add to this the fact that bugs are inevitable, especially in our legacy code, and it's possible to create a cascading failure by breaking a service rather deeper in the dependency chain.
This is bad and something I have started some initiatives to change. Most companies impose heavy structure and force more decisions to receive more types of approval in advance, so this or that director with this or that architectural vision can stop any bad ideas before they get implemented. Messari has taken an alternative approach: emergent order.
The idea is this, using my new user service as an example: say there are 20 services that need to be able to fetch a user's email based on their ID. Naturally, the first thing to do is to give each of these 20 services an environment variable that tells them where the user service is located and let them make requests via HTTP or gRPC or whatever. But in a fast-paced working environment, this sucks when there is dependency downtime or when changes are made that break the contract formed by the environment variable (in this example). It's also a lot for people to remember: who wants to litter one's environment with 20 different service URLs just to interface with them once in a blue moon on an uncommon code path?
One level of abstraction is necessary. In our case, user service maintains a more generic request-response option via NATS—instead of knowing how to ask the user service a question based on some specific fully-qualified URL, other services only need to know the topic on which to ask a general-use message bus a question. The message bus relays the request to the user service, and likewise with the response; at the cost of negligibly higher latency (scaled NATS can handle millions of messages per second) you have eliminated the need for services to talk directly to each other. Redis is another great tool here—perhaps the user service's job is to maintain some keys in a general-purpose Redis cache that everyone already knows how to contact. Of course these solutions will differ based on the use case, but these sorts of solutions create a bit of resiliency, and a ton of comprehensibility, at very little cost.
To the point: the idea is to mentally shift your team's understanding of some service such that it's not necessary to remember precisely what it's called, or what a request's URL path should look like, or how to determine when it's down (to speak of the classical problem). Your colleagues do not need to be plagued by this level of detail. Instead, colleagues should be able to take as given that if you pass a user ID to request:user:email, either a valid email address comes back or nothing does, and then they can just handle those cases intuitively without switching contexts.1
See the relevance to the Unix philosophy as well? We want doing some lookup to be as simple as piping a string to grep in the terminal. We want to create an abstract interface that unburdens the rest of our team from needing to think about the complexity that we're dealing with ourselves. And the best way to do this was decided by programmers who pre-date us all! No fancy frameworks anyone has to learn—just the simple request-response paradigm at scale. One layer of abstraction and no more than that.
Rust is the greatest programming language. I'm being glib because I love Rust, but what I mean to say is this: even if it takes a while to write (or compile!) your code, it's better that it never crashes. It is extremely easy and pleasant to write Rust that cannot crash, to know you won't have to fight with a GC to avoid OOMKills, and to go about your day. The greatest successes I've ever had with composable microservices are NATS (or event) based programs that listen to some topic on a message bus, fulfill one simple function, and emit output to another topic on the same bus.
Go is very verbose. It takes a while to read and there are a lot of simple things that cannot be made smaller because the language lacks macros entirely (looking at you, error handling). Go is also bad at shielding you from making mistakes: it's great that it doesn't let a named variable go unused, but it sure would be nice if Go would warn you of an uninitialized map or bad pointer dereference at compile-time, at least in simple cases. I love Go, don't get me wrong, but it lures you into a false sense of security with some very good types of compile-time checking even though it completely lacks other types.
Rust is the opposite of verbose. It packs maximum expression into every statement so you have to think very precisely what you are doing at every step. It forces you to address every possible state any Result or Option could be in during every value check. More than any other language in widespread industry use, Rust forces you to be precise at every level and crams as much validation as possible into your workflow prior to a successful compile.
You may not think this is what you want, but this is what you want. I have written Rust professionally and I have yelled at the compiler at midnight trying to understand why it doesn't like my &Result<Option<Result>> being passed between threads in this way but not that other way. Sure. I grant you that. But you can surely recognize that it's better than unplanned production downtime that pages you at 4am and costs your company big money. None of us are without sin and we have all broken production sometime, but why not reduce the risk?
My last point is more brief. If I can understand the code, that's great, but it's much more important that any given coworker can quickly read and understand the code in the future. I don't have permission to share any source snippets from Messari, but suffice it to say that our rate limiting middleware is the single most disgusting codebase I have ever read in my professional life—and I started out as a PHP developer. It works great, but I hate it and want it to go away.
Do you want your coworkers to feel that way about your code? Ever? I didn't think so. So make sure what you're doing makes sense to other people!
One of my greatest mentors in the industry once told me that whether a given piece of code actually works is transitory. Necessarily so, because business requirements change and technical debt is inevitable to address these on the fly. Dependencies break. Treasured open source packages stop being maintained or develop security flaws. No way around it. In other words, Every bit of code that you write is eventually going to be rewritten, full stop. If you're a sociopath, maybe you can justify not caring about the poor bastard who replaces you at this job who has to deal with your mess, but even the sociopath doesn't want to be stuck babysitting a codebase no one else wants to work with every three months when the business decides it needs to support a new kind of coupon code. Better to make sure a coworker can quickly spin up so you can delegate or avoid getting pulled back in every time. This is also why it's not good to be too clever unless the use case truly demands it.
I hope this brief document suffices as a summary of the sorts of things I think about as I write software. In short: readability; composability; at most one layer of abstraction; standard libraries. Cool? Cool.
Of course, I have also described the fundamental benefits of an event-driven system—it should be easy for any service to subscribe to any given stream of events rather than having to think about the potentially wide variety of services that may be deployed that can contribute to that stream. Complexity can remain hidden if the interface to it is easy. ↩
Licensed under CC-0 and of the public domain. Hosted with Cloudflare.
Created with ssg5 by Roman Zolotarev with a slightly modified modification based on one by Wolfgang.