Hi friends! It has been utter chaos here at ACRL and I haven’t had a moment to come up for air in several weeks. However, I think1 things are calming down a bit on my end, and plus I had something so utterly unpleasant happen to me that I just had to drop everything and write a blog post about it. The name of that unpleasant thing? Bazel.
Huh? What’s a bazel?
So Google has this questionable habit of building something internally, and then releasing a worse version of it as an open-source project, and then throwing a lot of marketing money at the open-source version until it takes over the world. In this post, I’m not talking about the most famous example of this2, but instead a lesser-known project called “bazel”, which is based off an internal Google project called “blaze”.
Bazel, if you’re lucky enough to have never heard of it before, is a tool for building software. It’s kinda like Make, or CMake, or Ninja, or any number of other build systems. The singular task of a build system is: take a bunch of source files written in text, and produce one or more executables at the end. Bazel adds onto this goal, in that it creates “reproducible builds”, which means essentially, if I give you the exact same set of inputs to the build system that I used on my computer, and you build it on your computer, you will get a bit-for-bit identical binary executable at the end3.
There’s a variety of reasons why you might want that, ranging from software provenance and supply-chain security to caching and build timing, and I’d just like to acknowledge that Bazel (probably) is solving a really hard problem that some people in the industry need solved. But, and this is my central thesis for this blog post, the user experience as someone who just wants to compile some code and run some tests is terrible. Here’s ten reasons why.
Reason 1: You have the wrong bazel installed
When I first started working with bazel, I was on a Mac, so I did the natural and normal thing that you do on MacOS: brew install bazel. Then, later, I needed to upgrade bazel and was greeted with an arcane message that said I needed to
cd "/opt/homebrew/Cellar/bazel/9.0.0/libexec/bin" && \
curl -fLO https://releases.bazel.build/9.1.1/release/bazel-9.1.1-darwin-arm64 && \
chmod +x bazel-9.1.1-darwin-arm64 && \
cd -What? What year is this? I thought homebrew was just supposed to handle upgrades for you? Why are we downloading release targets by hand and installing them in homebrew’s directory tree???
Anyways, not to spoil the punchline, but apparently the actual tool I needed to install is called bazelisk. Now to be fully transparent: if you read the Bazel docs4, this is their recommended approach, but I’m placing this complaint at the top of my post because it is emblematic of Bazel’s apparent disdain for the people who are actually using the software.
If Bazelisk is the tool that developers are supposed to use, why is there a separate homebrew target for bazel? Why does nothing in the tool itself ever tell you, “Hey, you might have installed the wrong thing, here’s the tool you actually want”? Why is the only message it provides when you try to upgrade “Download the latest thing from the releases page” instead of either “just doing the upgrade” or, again, telling you that you have the wrong tool installed???
This is a theme we will return to throughout this blog post: Bazel itself clearly knows that something is wrong, and it clearly should (or at least could) either resolve the problem automatically or tell the user how to resolve it, but instead it does neither of these and expects the user to flounder through extremely arcane and indecipherable error messages. This is not good UX design.
Reason 2: Why are there all these BUILD files everywhere?
One of the ways that Bazel accomplishes its tasks is by building a definitive list of dependencies for every single package, library, or module for the system it is building. This is fine and necessary to ensure reproducible builds. The way it accomplishes this task is by putting a BUILD file in every single subdirectory of your project. Again, also fine.
Now, naturally, any time your build changes (by including a new dependency or source file), you have to update the BUILD file, and thankfully, bazel doesn’t force you to do this by hand. Instead, it provides a tool called gazelle5 to update your BUILD files. BUT—if you, say, happen to forget to run gazelle, Bazel just refuses to compile. It just will complain that “some symbol is undefined”. It doesn’t try to resolve it automatically (which would be the nicest approach); it also doesn’t tell you how to resolve it, which is extra confusing if you’re a newcomer, because clearly the symbol is defined, it’s right &$*#ing there, I’m looking at it in my text editor. All Bazel says, in the middle of a stack of other output6, is “symbol is undefined”.
Again, Bazel knows that something is wrong. It knows how to fix it. And yet, it does nothing, provides no helpful information to the user, and leaves you to scrounge around for feedback until you’ve fully internalized the fact that you just need to run gazelle after every single change to any file in the source tree, just in case.
(A weird aside: if you use various non-code files—e.g., text files for snapshot testing, those files also need to be contained in bazel’s BUILD files, even though they have nothing to do with the generated code. I think this makes sense if you think about it long enough, but it was still weird to me when I first encountered it).
Reason 3: What is up with this syntax?
When you run any bazel command, you have to specify what you’re operating on: e.g., bazel build //some/package. The thing following the command “looks like” a directory path, except it is not. Sometimes you include the beginning forward double-slash. Sometimes you do not. Sometimes you include a trailing ellipsis (...). Sometimes you do not. Sometimes the path goes after a double-hyphen (presumably to separate this from other command-line arguments, which is a fairly standard convention), sometimes it does not.
The point here is: interacting with Bazel requires you to learn a new syntax that looks and acts sort of like a directory structure, but is not actually a directory structure, and if you do it wrong, Bazel just complains about missing files (and/or symlinks, which is extra confusing, what symlinks are we even talking about?). It clearly knows that the command is structured incorrectly, but it doesn’t give you any indication for how to structure things actually correctly, you’re just left to flounder.
Reason 4: How do I even build something successfully?
The whole point—the raison d’être, as it were—of Bazel is to build things. So then, why is it so incredibly difficult to actually build something resembling an executable with the tool? The first time I tried to build things I had my paths structured wrong and it complained about missing symlinks (see Reason #3, also note that I was just following the command that was listed in the documentation, which, sigh. I get that documentation goes out of date, but come on).
Once I fixed my paths, it then failed with the following error:
curl: (22) The requested URL returned error: 401
WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: no such package '@@some_stuff++whatever+blah_blah_blah_linux_amd64//': oauth failed:(Note, again, this error is buried underneath a wall of other text).
But OK, clearly it’s trying to download “something” from “somewhere” and it’s either getting a 401 Unauthorized response OR a 404 Not Found response, and it’s not immediately clear which, but more to the point: this error was indicating that I hadn’t logged in to a private docker registry that was NOT hosted at some-url.com. In fact, nowhere in the entire error message does it say anything about Docker.
Also, what on earth is this nonsense?
@@some_stuff++whatever+blah_blah_blah_linux_amd64//I have no idea what any of that means, except that it’s clearly some dependency that it’s trying to download. Bazel has to know where it’s trying to download from, and it has to know why it’s failing, but it doesn’t actually tell you any of that information, it just gives you an unrelated error message in arcane syntax and leaves you to flounder.
Anyways, I logged into the private registry and tried to build again, and was greeted with the following error:
ERROR: no such package '@@+http_io+whatever_v5//file` java.io.IOException: Error downloading [...] GET returned 404 Not Found
ERROR: /path/to/BUILD:67:42 //thing/im/building depends on @@+http_io+whatever_v5//file in repository @@+http_io+whatever_v5 which failed to fetch. no such packageOK again so clearly again it’s depending on some package that… doesn’t exist? This makes no sense. Clearly it exists for other people, or everyone would be complaining that everything is broken. Why doesn’t it exist for me?
For quite a while I was stuck at this stage in the process, and resorted to just pushing my code to GitHub and letting CI run the build and the tests, because CI was obviously configured correctly and my system was not, and that was easier than trying to figure out and resolve whatever was broken for me locally. Man it sure would have been nice if Bazel had given me a clue!
Finally I got annoyed enough at the CI delays that I decided to revisit, and discovered (after asking some other folks for help) that Bazel was failing because I hadn’t logged into GitHub with the gh CLI app! WTF??? I thought Bazel was supposed to be hermetic? I mean, OK, look, I’m not naïve, sometimes there’s code in a private GitHub repo, and clearly you’re going to have to log in to access it. But again, Bazel doesn’t tell you what code it’s trying to download or why it failed (or even better, say “hey I need some creds can you please provide them”). It just spits out a page of errors in an arcane syntax that refers to packages and modules that you’ve never even heard of before, and expects you to figure it out.
Reason 5: That error handling, though.
While we’re on the subject, let’s talk about that output. Back in grad school, I cut my teeth on C++/Boost template errors, which were infamous for producing inhumane amounts of error messaging because you forgot a semi-colon. I got really, really good at parsing through C++ errors to figure out what the actual underlying issue was7. The error messages were atrocious, no doubt, but usually the right answer was “scroll to the top and resolve the first issue, and then work your way down”. Sometimes there were so many errors it would overflow your terminal scrollback, so I learned to always redirect compiler output to a file so I could get back to the top.
Fast forward a decade, and now most of my development is either in Rust or in Go. Rust has famously set the high waterline for “good error output”; it (generally speaking) distills the error down to the actual underlying cause, often includes a hyperlink to the official docs with more information, and (9 times out of 10) will give you the actual fix inline. Go has a much more barebones approach to compiler output, but honestly it’s still pretty good: it’s not going to hold your hand through how to fix it, but it will generally tell you what’s wrong in a single line. I’ve never seen pages of error output from the Go compiler, and sure, sometimes I have to Google the answer myself instead of clicking a link or taking the compiler’s recommendation, but it’s fine. I don’t hate it8.
The thing is, with all of these systems, it was easy to identify the actual error message itself. Even with C++, even if the compiler output was thousands of lines long, I always knew I could scroll to the top to figure out what broke.
Bazel, on the other hand, both produces pages and pages of error messages, and puts the actual error somewhere in the middle. I can know that there is a Go compiler error, and even have a rough idea of what the error probably is, but I still have to spend a significant amount of time looking for the actual message. Worse, the actual error message is not singled out or highlighted in any way, while all of the content both above and below is covered with ERROR labels and other scary-looking output. So my experience with Bazel thus far is, if I want to identify a problem, I need to find the least-obvious line in the entire output to determine what to fix.
Reason 6: Compile times are whack
My next complaint is about Bazel’s respect for your (as a developer’s) time. The compile times are through the roof! And I say that as a Rust dev, the language which has as its primary complaint “things take too long to compile”. In some ways I don’t think there’s a lot you can do about this; if you’re trying to make an entirely hermetic build with bit-for-bit reproducibility, that’s going to take more time than “just” running go build. But still. I’ve got stuff to do, I can’t wait 10 minutes for you to finish building my app!
To be fair to Bazel, most builds don’t take 10 minutes, because of Bazel’s local on-disk cache. Congratulations, you’ve identified my next complaint!
Reason 7: Ouch, you punched me in the disk space!
To try to alleviate some of the compile-time issues and need to download every single dependency on the planet for every build, Bazel maintains an absolutely immense local cache on your machine. One of my colleagues was bragging about their 1TB local Bazel cache. WTF??? Look I know we all grew up in an era where disk space was cheap, but it sure isn’t anymore9. And even if it was, seriously? A terabyte of data just to compile some nonsense? I get mad when Cargo doesn’t clean up after itself. I get mad because Docker just likes to vomit all its stuff all over your hard drive indefinitely10. A freaking terabyte???? My god, what have we done.
Reasons 8, 9, and 10: To be continued.
I am just on the very beginning of my Bazel journey, and have captured in this blog post all (or at least most) of my complaints with Bazel’s user experience. But I’m a fairly new Bazel user, and I’m sure that given my experience thus far, there are going to be many more reasons in the future that I run into with this software, so I’ve left space in this blog post to account for those as well.
And look, this is the Internet. I’m sure some of you will come along and say “Well if you just understood the system” or “This question is answered in the docs” or some other variant of “Git Gud, Noob”. My rejoinder to that is: I am a pretty smart person in general. I am able to reason about some pretty arcane shit. I like to work on complicated, hard-to-understand systems11. And I’m sure that, given enough time, I could “git gud” at bazel.
But here’s the thing: I don’t want to. All I want is to build some source code and produce a binary. I don’t want to have to learn an entirely new language, parse through pages and pages of errors that don’t actually even tell you what the problem is, spend hundreds of dollars on a new disk drive, and waste hours of my life waiting for something to build that used to “just work”.
I understand that Bazel solves some real problems, for some real organizations, and that if you have those problems that need to be solved, Bazel might be your best option12. I also want to be clear: I’m sure that whatever Bazel is doing in the backend is probably actually solving those problems moderately well! But I am extremely unconvinced, unless your name starts with a G and rhymes with Froogal, that you have any of the problems that Bazel solves bad enough to subject every single one of your developers at your organization to a build system that treats its users with such disdain13.
Anyways, just remember, the next time you’re reading through hundreds of lines of error messages just to discover that the real error isn’t mentioned anywhere, and you have to go beg one of your more-enlightened coworkers for help: we did this to ourselves. You’re welcome.
As always, thanks for reading 🤣
~drmorr
/me frantically knocks on wood
AKA, Kubernetes.
If you don’t work in software—or heck, even if you do!—you might be surprised to know that a) this is not the case for nearly every other build system, and b) this is an incredibly challenging problem to solve.
Which, of course, I did not.
No, I don’t know what cute deer-like animals have to do with build systems either, but whatever.
Don’t worry, we’ll get to this.
And to this day, I’m convinced that one of my superpowers as a developer is “being able to parse pages and pages of errors or log messages to identify problems”. It’s like, I dunno, I’m a hipster LLM or something—I could understand errors and logs before it was cool.
Is that one of the nicest things I’ve ever said about Golang? It’s possible!
Thanks, AI.
Seriously. I set up a systemd timer that runs weekly and clears out all the old images and container builds on my hard disk because I got sick of running out of disk space all the time.
Case in point: I run Kubernetes, at home, for fun! It is well-established at this point that I am a sucker for pain.
The only other player I’m even aware of in this space that I’m aware of is Nix, which, lol.
And if your name does start with G and rhymes with Froogal, you aren’t going to use bazel anyways, you’re going to use blaze, which by all accounts is a significantly nicer piece of software.