Plumbing the depths of the Rust mines

We’ve delved too greedily and too deep into the mines of Rust. Generated by stable diffusion with the prompt “a giant flaming crab across from a wizard on an underground bridge”.

Welcome back! The SimKube series is on hold for the time being, since I still don’t have any good answers for the “data analysis and visualization” part of the story. To be fair, I haven’t put much time into it, because I’ve been deep in the mines of Rust, and it’s more-or-less been consuming my every waking moment. So over the next couple of weeks, I’m going to do a scathing takedown well-reasoned and informed discussion of the two languages I use the most these days: namely, Rust and Go. I’m sure this will go well¹!

Come on, not a whole post about Rust. Seriously?

Oh get over it. Next week I’ll talk about Golang. It’ll be great!

But I do want to frame the conversation for both of these posts before I get into the weeds: there are lots of posts out there comparing Rust to Go, and most of them tend to be pretty surface-level: “Rust has a borrow checker and Go does coroutines well!” And, I mean, that’s great and probably helpful to people who are just getting started, but what I haven’t seen as much of are posts discussing “what it’s like to actually program in one of these languages.” So that’s what I’m aiming for in these couple of posts.

Some background: I’ve written code in a lot of different languages, starting in assembly and C, moving on to JavaScript and Java (long before either of these languages looked anything at all like they do today), then C++, and for the last 8 years or so, Python, Golang, and Rust, in approximately that order. What that means is that my Go code tends to look a lot like Python, and my Rust code tends to look a lot like Go. I definitely have some opinions and styles and coding conventions that go against the grain, but I think I have enough experience in these languages to speak semi-confidently about them.

Also, I’ve touched on this a bit in a few previous posts, but one question that I want to get out of the way is “Why are you doing so much in Rust when you’re trying to work in the Kubernetes ecosystem?”

This is a very fair point: the rest of the community is pretty solidly entrenched in Go at this point, and Go’s concurrency primitives arguably make it a better language for interacting with Kubernetes (conversely, you could argue that Kubernetes is designed the way it is—i.e., requiring strong concurrency primitives—because it was written in Go). However, I have three main reasons for wanting to experiment with Rust:

1. Performance: Rust is a “systems-level” language. It uses llvm to compile down to native assembly, and is incredibly efficient. If I do end up writing any sort of scheduler in the future, I’m going to need a language that’s blazing fast, and Rust fits that bill better than Go.

2. Type safety: I have spent waaaaay too long in SEGFAULT and NullPointerException debugging hell to ever want to deal with that again. So if there’s a language that promises to make that a non-issue, I’m sold.

3. Curiosity: I’ve been really curious about Rust for a long time, and have been looking for a project to get better at the language with, and now that I’m my own boss, I’m doing it!

Overall Impressions

I think I can sum up my feelings about Rust and Go thusly: “Rust is a far more tedious and frustrating development experience, but when I’m finally done I am very happy with the outcome. Go is a simple and straightforward development experience, but I always feel a little gross after I finish the code.”

We’ll unpack the Go side of things more next week, because this week I want to unpack Rust. Let’s get the common/obvious stuff out of the way first:

1. Type safety and memory safety is great. It is the bees knees. 100% 12 stars out of 10 would purchase again²³. I don’t even mind the borrow checker errors, because I know what lies on the other side of that abyss.

2. The compile time for my Rust code is definitely “slow”, but (at least thus far) I haven’t found it to be frustratingly-so.

3. I find the macro system to be somewhat mind-boggling: the language isn’t expressive enough on its own, so they bolted a whole second language with some seriously arcane syntax on top of it. And then that language still wasn’t expressive enough, so they went and built a whole ‘nother system to parse ASTs so that you could more-easily write Rust code to generate other Rust code. It’s wild, and the only credible solution I’ve seen to address it has been essentially abandoned because of drama⁴.

With those out of the way, let’s talk about some of the stuff that I don’t see get as much attention online.

Testing

Generally speaking, software engineers agree that writing tests is important. They get into endless debates and arguments about “how much” testing, and “what kind” of testing, and “when” you should write your tests, but everybody agrees that you should test your code. So what’s the testing story in Rust?

I would say it’s pretty bare-bones⁵. Tl;dr, you annotate your tests with the #[test] attribute, you put an assert! macro at the end, and that’s pretty much all Rust natively gives you for testing. No fixtures, no mocking, no parameterized or table-driven tests, nothing. Fortunately there are a couple crates⁶ that can help you out here: the rstest crate adds fixtures and parameterization, and the mockall crate is able to do mocking in a surprisingly large number of circumstances⁷. The one thing I find a bit lacking still is the set of assertions you can make in your tests. The assertables crate is a good start, but it’s missing a bunch of “obvious” assertions like assert_len for sequences.

The other thing I find frustrating about testing in Rust is that it’s extremely rigid in the organization and placement of your tests. By default all your unit tests live in the same file as your code, which I generally find to be untenable because I write many more lines of code in my tests than I do in my code, and it just gets confusing and bloated to stuff all the testing code in the same file. So instead you have to put your tests in a submodule and include a

#[cfg(test)]
mod module_test;

block somewhere in your code. I really wish that Rust would just auto-discover tests wherever you happen to put them, instead of having to jump through these shenanigans.

And speaking of frustrating bits, there is exactly one location you can put your integration tests in: in a tests directory in the root of your project. As best as I can tell, this is hard-coded into cargo and there’s no way to change it. It would be fantastic if I could rename this to itests or something else, especially since (historically) I’ve put a lot of “test utility functions” in the root tests directory, and it’s annoying to conflate those two things.

Modules and Code Organization

While we’re on the subject of organizing things, let’s talk a bit about modules. Modules are Rust’s way of scoping or namespacing things. As the old C++ adage says, “Namespaces are great! we should do more of those.” And in general, I think Rust mostly gets this right, but I have some quibbles.

My first complaint is that there are four ways to put code into a submodule, and none of them seem to offer clear benefits over the other. You can declare submodules inline in a file like this:

// main.rs
fn main() { ... }

mod foo {
  fn bar() { ... }
}

Secondly, you can put your submodule code in a separate file:

// main.rs 
mod foo;

fn main() { ... }

// foo.rs 
fn bar() { ... }  

And lastly, you can put your module into a separate directory, and either declare it as a module by including a “sidecar” file next to the directory, or sticking a mod.rs file inside the directory. The Rust documentation says that the former method is the “recommended” way going forward, but I can’t honestly see why that’s “better”? Searching online seems to say that it’s to prevent the proliferation of mod.rs files that you can’t easily distinguish in your IDE, but frankly I feel like using the mod.rs approach provides a much cleaner directory structure, and there are enough crates out there using it that I don’t think it’s going to go away anytime soon, even if it’s not recommended. So that’s what I’m doing in my code⁸.

The other thing I find frustrating are the Rust “visibility” controls. There are (essentially) four quantifiers that you can stick in front of any definition: pub, pub(crate), pub(super), or nothing. The first and last are self-explanatory: pub means “anyone can reference this thing” and nothing means “I am only accessible within my current module”. The middle two are a little trickier: pub(crate) means that “this isn’t part of the public API but anyone within this package can reference me”, and pub(super) means “only my parent module can access this thing”.

Where this gets confusing is that modules themselves can have any of the same visibility specifications. So if you have module foo that contains function bar, and bar is marked pub but foo is not, bar is still not a part of your public API! The other place this gets confusing is with re-exports: it’s a reasonably common practice to “re-export” definitions from a submodule in a parent module. This is so that, if you have SomeStruct declared in foo::bar, instead of typing use foo::bar::SomeStruct everywhere, you can “re-export” SomeStruct into foo and just reference it as foo::SomeStruct. The last place this gets confusing is with tests: because of the way I’m structuring my tests, if I want to write a unit test for a function that is private, I must declare it as pub(super), even if I don’t actually want to expose it to the enclosing module.

The net effect of all these interactions means that I’m constantly trying to figure out what visibility each of my type definitions are at, and what visibility they “should be at”, and I find that this consumes way more of my mental energy when I’m coding than I would prefer.

Debugging

While we’re still vaguely on the subject, let’s talk about a related topic: debugging code. This is the one area that Python really falls short for me: I should never ever ever have to modify my code in order to debug something. I should be able to attach to a running process, stick breakpoints wherever I want all willy-nilly, inspect the state of anything and everything, evaluate arbitrary expressions, and — did I mention this already — not have to change my code to do so. It’s completely nuts to me that the Python community still thinks that import pdb; pdb.set_trace() is an acceptable way to invoke the (already rather lackluster) debugger.

This is actually one of the bits of the Go ecosystem that really shines: delve is a top-notch debugger for Go code. Rust’s debugging support, on the other hand, is sortof middling. It’s not Python levels of bad, but it’s not delve either. Debugging with Rust uses either gdb or lldb that have been rebuilt with some “Rust-y” extensions. I used gdb a lot back in my C++ days, so I’m pretty familiar with how these tools work, and “most” of the time they work pretty well for Rust code, but there’s a bunch of areas where they fall short:

1. For reasons I haven’t fully sussed out⁹ sometimes even with a debug binary, symbols, variables, and definitions are just “not available” inside the debugger.

2. If I want to evaluate an expression that includes a trait function call, that eval always fails, saying that it can’t find the associated function for the object.

3. I haven’t figured out a good way to set breakpoints inside closures: this is especially obnoxious because rustfmt somewhat aggressively collapses closures into a single line, which means I can’t even set a breakpoint on the line number inside the closure.

4. Also when setting breakpoints, sometimes it just… sets the breakpoint in the wrong place? I don’t quite understand why this happens either.

Anyways, all of this is to say: the debugger for Rust binaries does work, and it’s sometimes useful, but I find myself falling back to print-statement debugging way more often than I’d like.

The Standard Library

Moving on, let’s talk about the standard library: Rust takes a very “batteries included” approach to its stdlib, which in general I prefer. It means I have a consistent, hypothetically-well-maintained set of tools and functions I can reach for when I’m trying to solve a problem in my code. Even so, the Rust stdlib is… overwhelming, to say the least. Each module has dozens upon dozens of associated methods, structs, traits, etc., and finding the one function that does what you want is like looking for a needle in a haystack.

It doesn’t help that the nomenclature for some of these things leaves a bit to be desired. Quick, without looking it up, what’s the different between the .iter() function and the .into_iter() function? Could you tell me the difference just based on the names alone? What about the difference between .ok(), .ok_or(), .ok_or_else(), .or(), and .or_else()?

I posted this toot on Mastodon the other day:

Another day, another inscrutable function from the #rust stdlib that I learned the purpose of.

I’m constantly learning about new functionality or nicer utility functions that I can use to make my code cleaner, but discovering this stuff from the get-go is really hard! This is also exacerbated by the structure of the Rust docs — functions that are associated with traits that a struct implements are not shown in the “quick list” of functions in the left-hand-column of the docs. Instead it just shows which methods are “directly” associated with the struct, and a list of traits that are implemented for the struct (and the list of implemented Traits is waaaayyyy down at the bottom). So if you don’t know that (for example) a Vec<T> implements the Extend trait (or you don’t know what the Extend trait does), you might never know that you could call vec.extend() in your code, and unnecessarily reimplement that functionality!

The Pink Bugbear in the Room: async

Whew, this post is getting long! I think we need another image to break things up a bit. Image generated on stable diffusion with the prompt “a pink bugbear in a room”. Inn’t he cute???

Alright, I saved the most fun topic for last: asynchronicity. There’s been a lot of ink spilled on this topic already, so I’ll try not to rehash old ground too much, but I will say that async coding in Rust is frustrating. I have a reasonably good understanding of a) asynchronous code in general, b) asynchronous code in Rust¹⁰, and c) formal methods and type theory, and even so I still find this to be the most difficult and rage-inducing part of the language.

At an intellectual level, I understand that “zero-cost async” and the borrow checker and etc all are really hard problems to solve. I’ve read a bunch of articles about why these are hard, and to be honest, they’re mostly above my head. I’m sure I could figure them out, but—this shit is hard, yo. No question. But, I can understand all that and still be frustrated when the language I’m using doesn’t just do the things that I want it to do.

Instead of going into details I’m just gonna share a bunch of screenshots from a thing I was trying to do yesterday, which I spent most of two days trying to decipher:

In summary: I’m trying to watch a stream of pod events, and modify those events as they come in to attach the “chain” of owner references to those pod objects. Here it’s complaining because I’m calling .await inside a closure, and closures aren’t asynchronous:

The error reads: `await` is only allowed inside `async` functions and blocks

Ok, fine, let’s try to make this an async function:

The error reads: `async` non-`move` closures with parameters are not currently supported. Consider using `let` statements to manually capture variables by reference before entering an `async move` closure.

Well uh, that’s a remarkably specific level of tired, er, unsupported feature. But OK, it sounds like if I make this an async move closure it’ll wor—

The error reads: async closures are unstable; see issue #62290 <https://github.com/rust-lang/rust/issues/62290> for more information. To use an async block, remove the `||`

Oh. Well, I guess this is unstable, but fine, I can install a nightly version of Rust and turn on the feature, but just for fun let’s go read that GitHub issue first:

As @Nemo157 said in their comment

Despite "Implement the RFC" being checked this feature is still largely unimplemented

Why is this still incorrectly marked as implemented, 4 years later?

ARGGGGHHHH! What the hell, people. Am I just crazy? Have other people not tried to do this kind of stuff? What is going on? :tableflip:

This is just one example, but I feel like I’m constantly running into issues like this. I’ve had to figure out weird lifetime issues, weird ownership issues, weird issues with closures, and when I try to search for documentation or help, I get almost nowhere. I do not think I am an incompetent developer or dumb or anything like that, and yet these issues are a constant struggle. It is getting to the point where I feel like I’m just fighting weird language idiosyncrasies instead of, you know, doing what I want to do, which is write code and solve problems.

In Conclusion

Whew! 3400 words! This is definitely my longest blog post on here to date! If you made it this far, congratulations. Just to end on a positive note: I know I did a lot of complaining in this post, but I do generally still like Rust as a language. It’s a lot of work to get code to run, but when you get there, you feel really proud of what you’ve written and I think it makes for some incredibly elegant code. I’m not giving up on Rust yet! I’m at least going to keep using it for (part of) my SimKube project. Whether I do other stuff with it (like, write a scheduler) remains to be seen…

As always, thanks for reading! Come on back next week for 3400 words kvetching about Golang :D

~drmorr

1. Narrator: It will not, in fact, “go well”.

2. Also, for my Python devs out there, if you are not using type annotations and mypy, I will not take you seriously.

3. Just kidding. I will still take you seriously and love you. I just might judge you a little bit.

4. And hoooooo boy is there a lot of drama in the Rust community. I really don’t want this to become a post about (any of) the drama, so I’ll leave it at this: it is disappointing to note that a language and ecosystem that claims to prioritize “providing a friendly, safe and welcoming environment for all” has managed to make so many people feel unwelcome.

5. And for me, at least, the Gold Standard here is Python. The combination of the pytest and mock libraries, combined with the ability to inspect the internal state of, well, everything, means that you can write some seriously powerful tests. Now understandably, the fact that Python is interpreted means that there will always be more you can do, but that’s always the standard that I’m looking to replicate.

6. “Crate” is the Rust word for “package”

7. It is a bit disappointing that generating mocks in Rust requires you to change/modify your production code, but again: compiled language means I don’t think you can get around that. At least most of the time, none of the mockall code will actually ship in your production binary, so that’s good at least.

8. This feels reasonably close to Python’s __init__.py requirement, too.

9. I assume it has something to do with the translation from Rust into LLVM IR? But I’m not sure.

10. Heck, as an early pandemic project I wrote my own interrupt-driven asynchronous runtime in Rust for an embedded system! I found a bug and got a PR submitted to rustc fixing a different bug around this! I think I know more about this than your average programmer.