70,000 Lines of Code, 7 Languages: How to Scale via Rust Core Transcript, AI Summary & Key Points

00:01 [music] [music] >> Spencer as Eric just mentioned, I manage the SDK team at Temporal. Um today I'm going to share with you some like pretty practical advice about how you can potentially adopt this particular architectural pattern that I implemented at Temporal that has really served us very well where you have like a shared core um written in Rust.

00:28 Uh and then like a bunch of language specific layers built on top of that. So, that's what we're going to talk about today. Uh [snorts] so, this first slide, you know, don't worry that this is illegible. Um the the point is actually that it's messy. Uh you're not you don't need to understand it. >> [laughter] >> Um this represents something called like a a local activity in Temporal.

00:47 Um And >> [clears throat] >> the important thing to take away from this slide is that it's like complicated, right? Um this state machine diagram is backed by about like 800 lines of code. Uh so, who would want to write this seven times in like seven different languages? Any any takers? No. No. I was hoping at least one of you would raise your hand cuz then I'd be like, "Good.

01:12 You like suffering." Like, "I too enjoy suffering." Um but no, I mean realistically you obviously don't want to do that, right? Uh to put it into context here is the rest of the state machines. Uh that's is about 7,000 lines of code that this whole thing represents. Um And an overall repository that's about 10 times that big, about 70,000 lines of code, right?

01:36 So, this logic um can't live on the server side. Uh it has to be in the SDKs that are users take as dependencies. Um so, again, you know, we want to provide languages SDKs in a bunch of different languages. You don't want to write that seven times, right? That's just like, that's not going to work. So, [snorts] why does Temporal have this problem? Like, why am I showing you a bunch of messy state machines?

02:05 Well, like I just mentioned, we want to offer SDKs in a bunch of different languages. We need to meet our users where they are. We can't really expect them to like adopt some, you know, unique language that we would make up just for the purpose of writing workflows like you probably are familiar with like there's a lot of products on the market that kind of do exactly that.

02:26 They have some kind of like graph-based DSL or whatever and you like inevitably run into limitations with that and like no one wants to learn that and blah blah blah. So, you know, we offer languages, SDKs rather, in a bunch of different languages to meet people where they are. Um So, you know, what is Temporal like in very briefly? We're a platform for durable execution, which is to say like you write some code, we take it, we make it resilient in a bunch of different ways.

02:53 Um And in order to like enable that capability, you know, our users, they write some code in their language of choice and language of choice, they write it against our various APIs and we, you know, do magic and make that durable. So, like providing all of these durability guarantees is the reason that there is like so much code in those last slides and like all the complicated things is like uh a bunch of logic has to exist on the client side.

03:22 They have to be super fat cuz they need to do a bunch of complicated things that wouldn't scale if we made like, you know, an RPC for like every single little, you know, bit of logic that has to happen in on the client. So, with all those kinds of restrictions in mind, um you know, what did we need to solve for as a business when we decided we wanted to offer SDKs in a in a bunch of languages?

03:46 So, the first uh and most important thing, certainly for our business and probably for most of you, is reliability, right? Like, these things need to work. There can't be a bunch of bugs. Uh, you know, the minute we violate that guarantee in any kind of way, like you immediately lose customer trust, like game over, right? Like, that's that's priority zero.

04:10 We needed the SDKs to be consistent. Um, like, we can't have divergence among different languages, and sort of have them doing slightly different behavior. That would also cause all sorts of really bad problems. So, like, that's another huge priority. They needed to be maintainable, right? Like, at the time when I started this project, we had like five guys, four guys on the team.

04:36 Um, now I've got 10 guys on my team. Um, you know, we support currently seven languages. Eric mentioned eight, cuz I wrote that. I was really hoping I already would have released the next one before I got here, but that didn't happen. Um, but, you know, we've got 10 guys working across seven different languages. Like, uh, I'm super impressed with my team.

04:57 I think that's mind-blowing. Um, so, we need something that was going to scale with like another ton of people. Another very important concern for us, we wanted to provide like an idiomatic experience for our users in the language that they're used to, right? So, like, that doesn't mean, okay, we just kind of write everything in Rust, and we like pipe it through in like the most obvious way possible, and like what you get is what you get in the language layer, and too bad.

05:20 No, you know, we want to provide SDKs where like you use the things that you're used to using in that language, and it feels natural. Like, it's a good experience. Uh, last, but probably not least, uh, thing had to be performant. It's actually not like a like a particular concern of ours. Like, um, we have relatively good budgets client-side for performant stuff.

05:45 But, at the same time, like, if you're asking your users to run like library code that you ship like every incremental extra bit of resources that you use is like shows up in their cloud bill somewhere, right? So like, you know, we still want to be reasonably efficient. >> [snorts] >> So, obvious conclusion here, writing this multiple times in all these different languages like not a viable choice.

06:09 We needed to pick some kind of shared code solution. Um we needed to pick the language to write that in. What language did we pick? No surprise to any of you here. It was Rust. That's in the title. So, excuse me, why did we pick Rust? I mean, I think for the typical reasons people choose it these days, it's safe, it's fast, it's expressive, it's portable.

06:35 This one is sort of quite important for us, right? Like our users are using Linux, you know, OSX, I guess it's called macOS now, um Windows, they're on ARM, they're on x64. Like we got to be able to like support all these different platforms. Very importantly, it is not C. Uh no one really wants to be writing C. Uh but it is C compatible, which is very very important because there's basically no other way to call, you know, across a language boundary in memory these days.

07:11 I mean, you could you could do everything over like IPC, but maybe, you know, that's not ideal. Like the C, you know, FFI interface is a well-established standard that people have been using forever. Rust in fact doesn't even have a like stable binary interface. There is no way to export Rust functions and call them directly. So, you guys knew that though, right?

07:32 Like Rust was in the title of the talk. I'm not necessarily here to sell you on the virtues of Rust today, although I do think it's a great language and I really like it personally. Um but in this case, I think it was just sort of like the right tool for our job here, right? Like we could have just written everything in C. I mean, I think like probably pretty clear why we didn't really want to do that.

07:54 Um we could have picked Zig, which is like another sort of modern, you know, C replacement kind of language that is also like cool. Uh but at the end of the day, we chose Rust because of its focus on safety, memory safety specifically, and for its expressiveness, and for the fact that it has like a pretty good community behind it still, like certainly compared to something like Zig.

08:18 So, that's kind of the setup. What's the architecture that we landed on? It looks like this. Um so, over here on the left, we have like the Temporal service. It's in the cloud, whatever. We don't care about that for the purposes of this talk. Over here on the right, we have some users' worker, um which is, you know, what they kind of run to interact with Temporal.

08:40 The fact that it's a worker, for the purpose of my talk to you guys, is not really relevant. This could be anyone's code using something that you've built using this pattern, right? Like here, it happens to be a Temporal worker. For you guys, if you are interested in adopting this kind of architecture, it's, you know, whatever thing needs to be deployed that way.

08:57 So, there's all this code over here on the right, and all this stuff that's in kind of the light purple is like code that I've authored, my team has authored. And then there's this user code at the top, right? So, the stuff I'm talking about today is really the two layers at the bottom. We have like the Rust core, which is the core shared logic. And then we have this like bridge layer above it, right?

09:20 Um and that there's kind of like one of these bridges for each of these different language layers, although in the case of Swift and C#, they share one. I'll talk a little bit about that later. Um and a lot of what I'm going to focus on goes on in that bridge layer. And then above that, you have like the language specific SDKs, right? Which expose the bits that people are going to like actually program against, right?

09:40 They kind of don't see anything below this line, and ideally they should never have to know that it even exists. In practice, maybe they might, you know, for various reasons, but like the the third thing, the language layer, is like what they really interact with. >> [snorts] >> So, again, this layer is the part most of my talk will be focused on, um this like bridge layer.

09:59 It forms the glue between like the language specific layer and the rest core. So, let's talk about them. Um Rust and language bridges. So, as I just mentioned, each language kind of has their own one, notable exception in one specific case, um and these are also written in Rust. They're defined as Rust crates that live in the language specific repo.

10:25 So, we have all these language repos that have like the language that they're written in, and then also some Rust in there, which is this bridge. Uh ultimately, all of them are calling into the core through CFFI somehow. Um but like depending on what language we're talking about, that fact may be more or less obscured. Um some languages have these like pretty nice helper libraries for specifically for like this kind of use case where you want to use some you want to write some stuff in Rust and call into it from the

10:57 language layer. Um and so, in a lot of those you you can you kind of like can paper over the fact that there's like C calls going on. Uh but not all of them have that. Uh you know, some of them don't have anything at all, in which case you're directly exposing C, you know, C compatible functions, and you're calling them like the hardcore way, right?

11:17 Um so, like in an ideal world, these layers are thin, right? Like you don't want a ton of stuff in here because at the end of the day, this is just distraction. Like you have like the core logic in the in the Rust library, you have like the user-facing stuff in the language layer, this is pure mechanics, right? Like there's no other reason for it to exist.

11:37 Um so you want to keep them small, but like they're not trivial. Like there's a lot of stuff we got to do in them as we will see. So, speaking of those bridge helpers, our first bit of sort of practical advice in the talk, here's like the current ecosystem of this kind of thing at least as far as the language layers we're using are concerned. Um Python and TypeScript both have these very nice helper libraries called PyO3 and Neon respectively.

12:04 The TypeScript JavaScript one, that's for Node specifically, for to to be clear. This is if you are writing an application that like runs inside of Node and uses Rust. If you're not using Node, this can't help you. Um for Ruby, there's also a pretty decent helper library called Magnus. Doesn't do nearly as many things as PyO3 and Neon do, but it helps a little bit.

12:24 For our .NET and our Swift ones, the Swift one is actually recently authored by Apple. They they published it. We didn't write that directly, but they used like the core that, you know, my team and I have written, which is really really cool to have like an external contributor come along and do that. Um but in those cases, uh for Swift I don't think there's anything.

12:41 For .NET, there was some stuff, but it it didn't have the maturity level that was going to kind of be necessary for us. And so in those cases, we just had to do it all by hand. Like it's just like handwritten representation C exported stuff. So, that's the helpers. Moving on to the concerns, things we have to deal with in the bridge. Uh the first and perhaps most obvious is just like type conversion, right?

13:07 Like you have things you need to represent and you need some representation of them both in Rust and in the language layer, and then, you know, they need to pass through the bridge somehow. Um So, there's like a lot of lines of code here. Uh and we're going to talk about each of the things I'm about to list like specifically, so we'll talk more about this later, but um this is can be quite verbose.

13:33 Uh the second concern I want to talk about is sort of like async stuff. So, in Rust, you have futures, right? Um in TypeScript, you have promises, type in your TypeScript you have promises. And um Python, you also have futures. In Ruby, you have fibers. Like so conceptually, these are all kind of basically the same thing, but they don't like exactly map to one another, and so you have to figure out ways to bridge this stuff together.

14:02 Lastly, uh memory management. Uh this might not really be that hard if you're get to use one of the nice helpers. Like this probably just kind of gets taken care of for you of taken care of for you. But if you have to write the C stuff by hand, like now we're in a situation where you have to be writing unsafe Rust. This is kind of very much like writing C.

14:21 It's going to be tricky. Um but, you know, the advantage compared to if you had done everything in C as well, it's just restricted to like this layer. So, at least you have, you know, at least you have that benefit. So, let's talk about type conversion. Um as I mentioned, kind of much blood is spilt dealing with this. Um my first practical piece of advice here is that you can use an IDL to do a lot of code gen for you.

14:48 For example, Protobuf, um which we use within Temporal. I'll talk about why that's the case in a little bit. So, these can help, um but they can't do like everything for you. And part of the reason for that is that there are many representations of types that you might want to have. Um so, starting from like the top and the bottom of this list are like ergonomic types, like what I would call the types that you actually want to have to work with when you're like writing logic.

15:18 So, at the bottom, your users, you want to give them types that are like nice to use and work with. Um at the top, types you're using like just in your own core like in your own work, you want these to be nice. And the thing is, most IDLs don't actually generate very nice stuff to work with, you know? Like um they are great for putting stuff on the wire and moving it between computers and blah blah blah, but they're not necessarily like great hands-on experience.

15:46 Um then, you know, say you do use an IDL for some of the reasons I'll talk about, you might have those types. Uh and then also perhaps you have some just like handwritten types in the bridge for like certain situations that didn't match the other, you know, ones. So there's like a whole lot of a bunch of different things you might have. You've got to write all this code to like convert between them.

16:04 It just like takes up a lot of time and energy. Um so there's there's a lot of code going on here, but uh you know, how do we how do we deal with that problem, right? So like first let me talk about why I happen to use Protobuf. This is really just because if you remember in that architecture diagram, like the Temporal service, we talked with it via gRPC.

16:24 There's a bunch of stuff already defined in Protobuf that I needed to reuse. It just like wasn't going to be practical for me to like rewrite all of that, right? So that's why we use Protobuf. I might have made a different choice if I had the opportunity to and and also kind of talk about that a little bit later in the talk. Um so, you know, there's a lot of code here.

16:44 Like I mentioned, a good technique for addressing that, particularly in Rust, is has a very nice macro system, right? So like a lot of this stuff is like fairly obvious to the point where you can kind of just write a macro and like do a bunch of these type conversions like that way, right? And instead of writing thousands of lines of like this field goes to this field, this field goes to this field, this field goes to this field, you can like write a couple macros and just, you know, generate that code for every type

17:09 everything. Uh another very practical thing I would do here is like avoid complicated types in the bridge interface. It's like tempting when you're in Rust to like make some fancy enumeration with like generics and like complicated type bounds and stuff. And then when you try to pass that over the C bridge, what you'll find is that like some very non-obvious things might happen.

17:34 And >> [laughter] >> you get in the situation where like your fancy type, while it was nice to use in the core, is like not nice to pass across the bridge, hence why you might want to make one of these sort of simpler representations and make some conversions on either side. So, fun little pop quiz section of the talk. Speaking of IDLs, right? Like Protobuf, for example, is what you would call like a serializing IDL, right?

17:58 Like it has to serialize to like a wire format, right? So, that has some kind of performance cost. So, which do you think of these two options would be faster? Um make a JavaScript object from Rust, like using Node's built-in C APIs for doing that kind of thing, or do you think it would be faster to take that Rust object, serialize it to JSON, read it on the Node side, and turn it back into an object?

18:25 Seems obvious, right? Like it's got to be the first one. There's no possible way that like serializing to JSON and deserializing again would actually be faster than just like directly moving memory and making the object. But wait, like not what you thought? Like the guy who wrote Neon, this this helper library, here in some Slack that is like you know, helping someone, and this person's like, "Why is this so slow?"

18:49 And the guy's like, "Oh, actually the like JavaScript object creation APIs in Node are super slow for some reason." And like it actually might be faster for you to like serialize to JSON and serialize back. Um which I was like, "That's nuts. Like that seems crazy." But I mean, it's kind of funny cuz like if you think about it, it sort of makes some sense, right?

19:05 Like JSON serialization in V8 is probably one of the most like optimized pieces of code on the planet, right? Like that is the hot path for like a like a double-digit percentage of software that exists in the world, right? Like it's it's going to be optimized. Um and so we're like, "Oh, you know, cool cool cool." And then and then but then I went and ran some benchmarks and actually it was the thing you would assume.

19:28 It turns out that serializing the JSON and back wasn't actually faster, at least in our very specific use case. Uh the direct object creation thing, which you would think was faster, was faster. But by like like not a lot, right? Like these are pretty small numbers here. Um like I did two different tests where I made just like kind of a simple object and then I made one with a bunch more fields.

19:49 You can see the advantage kind of started to disappear. So, I think if you have some like very complicated big object with like a whole bunch of fields, the JSON thing actually like starts to become more true. Um which, you know, I'm not going to get into that. It's like no details. I don't really know. But the relevant part for the talk here, which is advice that is not new, is like, you know, measure stuff, right?

20:12 Like in particular in the context of what I was talking about with IDLs, if you come along and you're like, "Wow, like I don't want to use one of these because like these two things are in the same process. Like why should I be serializing? That seems so pointless and wasteful." Um on a like principle level, that's true. On like a practical level, it just might not matter at all.

20:31 And like the savings you get from not having to like write a whole bunch of types by hand might be totally worth the like, you know, couple extra CPU cycles that it costs to serialize back back and forth in the same process, even though it's pointless. So, and we'll we'll come back to that choice in a little bit later. But onto the next topic for now, um the async code stuff.

20:53 So, you're listening to me, you're thinking, "Okay, maybe I want to adopt this architecture." Um if what you need to do inside of that res core is just like pure logic, this might not apply to you, right? Um but if you do need to do things like make network calls, write to disk, whatever, do async stuff, you're almost certainly going to have things that exist in the Rust core that, you know, are going to be taking advantage of futures and you might need to wait on those things from the from the language side.

21:21 And like each language, like I sort of mentioned, has its own kind of different representation of these things. Um, and you might need to kind of wait in both directions, right? Like you might have a future in Rust that you need to wait on from the language side, and you might have a promise or a task or whatever on the language side that you need to wait on from Rust.

21:43 So, you need some way to like connect these two concepts. That's not always easy. Uh, in the cases with the helper libraries I mentioned, PyO3 and Neon, they they have some like pretty nice stuff that'll kind of do a lot of this for you. Um, if you can't use those, you're going to need to do it by hand. So, I'll kind of show you a couple options there.

22:04 >> [snorts] >> Uh, the first one is to just use our old friend callbacks, right? So, um, you know, maybe this seems obvious, but I I think it sometimes cannot be when you have all the sort of noise of everything else that's going on in the bridge here. If you need to do something like I was talking about where you need to wait on some Rust thing from the language side, you can end up doing something like this, right?

22:26 Like you define some function in Rust. It accepts a callback that the language side passes in, and this is like ultimately, this is some like pointer, right? This is like a pointer to some function address or whatever, you know, cuz it's coming across the the language boundary. And then inside of here, you can, you know, do your do your Rust future thing.

22:44 You know, wait on it, invoke the callback, pass the result. You're good to go. Um, but this actually doesn't always work. Um, you might need to deal with some other like weird concerns. Um, so there's that a different technique that involves using event loops, but like first a little bit about why you might not be able to use the callbacks thing. So, some some languages like, you know, can only kind of do things single-threaded.

23:11 And in fact, even the, you know, um, when I was talking about neon just a second ago in the last slide, that's sort of true for for V8, right? Like this is the async event loop in JavaScript is serialized on one thread. Uh, the same is true for like Python cuz it has a global interpreter lock. And the example I'm going to give here specifically is Ruby, which also has a global interpreter lock.

23:32 It's called the GVL there. So, all execution is serialized, but in Ruby's case, this is like particularly annoying cuz I can't just I can't do the callback thing I showed in the last slide because in Ruby all invocations that are going to go back and touch Ruby code need to happen on a thread that Ruby itself created. Um, so like, you know, let me just go back a bit.

23:55 So, like if you have this code, and here you see it's inside of a Tokio spawning a task. I don't know how familiar you guys may or may not be with like Rust async ecosystem, but what this means is this is a little bit of code inside the spawn thing that could run on some like arbitrary thread that the Tokio executor spawned, right? Like it some thread somewhere created by Rust.

24:16 But I can't I can't invoke this call this Ruby callback here because Ruby doesn't own that thread. It doesn't have all the right and stuff doesn't line up. So, in that case, we have to do something totally different. Um, these callbacks every time you would sort of pass a callback and you instead kind of wrap that function, you turn it into some version of that that like pushes a little events into a queue, and you instead do something like this when you define a function that's like, you know, run this loop or

24:44 whatever. Ruby is expected to call this at the beginning of, you know, program setup. And then it you just have this single thread here that just spins and pulls like callback fulfillment requests off of this queue, and that way they always get executed inside of Ruby land. Um So, like quite a bit of hoop jumping. Like, I don't have code to show you compared to like the PiO3 thing.

25:05 In the PiO3 thing, it's just like, this is a Python future now, and you're done. Um you know, here it's a much bigger pain. So, just something to be aware of like depending on what language you're doing this stuff in, it's like more or less difficult. Uh last of the bridge topics here, memory management. Um Again theme here, uh those helper libraries are like quite nice.

25:30 You know, you use them in the languages where you can. Um if you can't use them you're going to need to be writing unsafe code. And my advice would be to be careful with that. Um you know, I think like the premise of Rust is that even very smart people screw up memory management code. Uh that premise is true. And I can tell you as a manager that you will have people on your team if you do something like this where like, it's cool, like I got it, you know?

25:59 Like, I'm not going to mess up the memory management. Like, I'm a super good engineer. Um but they they will. They will mess it up. Like, it's just really, really tricky to get it right. Um so, don't pretend that you're going to nail that. Um you know, take the time to really make sure that you're doing this type of thing properly. And it goes back to an earlier principle I mentioned, which is like the thinner that you keep the stuff, the easier it is to not screw up, right?

26:21 So, like keep the absolute minimum amount of like code and logic that you can in these bridges cuz like then you have less opportunity to screw these things up. Um but anyways, you know, what is what does that look like? I mean, to some of you this is going to sound really obvious. To some of you it might not at all. So, like you know, memory it must be sort of created and destroyed in the in the same place, right?

26:42 So, if you're on the Rust side, you're allocating some memory and then it needs to go over to the language side. What that, you know, in practice looks like is you need to turn it into a raw pointer, pass it to the language side, stuff happens there. They pass that raw pointer back to Rust, you like reconstitute it into something that you made the raw pointer from, which might be like a box or an arc or whatever, and then you you free that memory.

27:07 Um and vice versa on the language side, um you might allocate some memory and then, you know, a lot of these are garbage collected languages, so you need to tell the garbage collector like don't garbage collect this, pass it over to the Rust side, do stuff you want to do with it, um pass that thing back, and then tell the um host language like, okay, turn the garbage collector back on and, you know, maybe you do or don't free this.

27:32 And life proceeds as normal. So, [snorts] that's kind of what I wanted to talk about as far as the bridges are concerned. Uh change gears a little bit, talk about kind of some of the general limitations of this architecture, lessons that we've learned. Um one of them is like injecting behavior into the into the core layer, right? So, for example, you know, we we do all that gRPC stuff that I mentioned earlier.

28:01 Um if you have some situation where people in the host language want to, you know, introspect that, right? Like or even modify the behavior in some way. So, say they want to do something like every time you make this specific RPC call with like this specific parameter, I want to like go tick some metric, right? Um if you don't have some like totally generic mechanism from that from the very beginning, you're you end up in like kind of a tough place, right?

28:32 Like, what do I do when they come along with that ask? Like, do I add some really specific option that's like enable specific metric that you asked for and tick you know, that only applies to like one user and you know, 99% of people are getting no value out of that. Um, so that's kind of like a difficult position to be put in. Um, so the the lesson here is like plan ahead for that.

28:57 And if there are places where you imagine that users might want to do this kind of like introspection or behavior modification, like you know, try and incorporate that into your design so you can have some kind of like very generic callback. So, in this case, what I ideally would have done from the very beginning is every time core does anything with GRPC, it would go through some kind of like make a GRPC call callback that would go all the way back through the language layer.

29:24 Language can do whatever they want. And potentially I'll touch on this a little bit more in a minute, too. Um, potentially even actually have the language layer be responsible for executing the GRPC call and kind of like not have the side effect be in the core layer. But I'll I'll touch on that again later. So, main idea here is like try and plan ahead for these spots.

29:45 Cuz otherwise you get in this position where you have to do very specific like fiddly knobs, which is not fun. Another challenge like shipping native code in all these languages is not always pleasant. Um, for some of them it's easy. Like the Python ecosystem is very used to this idea, right? Cuz they do all sorts of data science, but Python's super slow, so they like need to have native extensions.

30:07 Um, so like the PyPy package management ecosystem's like good at this, right? Like you you can upload for all sorts of different platforms and architecture. And like when the user takes your dependency, they only get the right one and like cool. Um, some of them it's not so good. NPM doesn't work that way. Like for my NPM package, I just every time a user takes a dep, they download all the binaries for every architecture because NPM just like doesn't support this.

30:33 Too bad. Um, that stuff's actually kind of relatively minor though compared to like some of the real problem situations. Um so we have Java and Go SDKs. These are actually the only SDKs that aren't based on top of the Rust core I'm talking about and that's just because they they predated it. They existed before we started this project. Um but one of the things that kind of prevents me from aside from the fact that it would just be a crazy investment from porting these to the Rust core is that uh people really don't

31:04 like to run native extensions in these languages a lot of the time, right? Like Java with the JNI and to an even greater extent Go with Cgo um cause a bunch of like operational concerns for people cuz they're really not used to writing stuff that uses native extensions. Like Go with Cgo in particular, you literally have to like set different build flags.

31:25 They have to like they have to change their build process. Like that just sucks. Like you don't want to force that on your users. Um so this is like a real limitation, but I'm going to talk about a way that we can maybe address that. Um which is coming up now. So enough about the downsides, you know, the upside of downsides is that you learn something from them.

31:42 Um so let's talk about some of the things like I would do differently if I had the chance to do this all over again. And that maybe my team will still do in the future. First is uh WebAssembly. Um I don't know if you guys have heard about WebAssembly or not. Uh I think it's super cool. It's really promising technology out there if you're in the business of needing to ship portable code, which is like sort of exactly what we're doing here, right?

32:05 Um when I looked at the schedule for the talks here, I was like kind of surprised that it just didn't show up anywhere. I figured like someone would be talking about this and and maybe they are and I just didn't find it, but as far as I could tell, I'm the only one. Um so I'll give you kind of like a brief overview of what WebAssembly is cuz I think it's I think it's super cool if you guys haven't heard about it before.

32:25 So WebAssembly is a bytecode. Like the JVM. JVM's bytecode, right? Um in many ways it has some similar goals. Uh it's very fast. It's portable, which is to say there's no, you know, platform specific compilation step. Um [snorts] it's constrainable, right? Like it's very good for running like untrusted code, which doesn't really matter for kind of the use case I'm talking about today, but is is a cool point.

32:51 Um you can kind of restrict all the side effects that might ever happen, you can restrict the compute, like cool properties that way. And it's not only it's not only for the web, right? Like web is in the name and sort of the original conceptualization of WebAssembly is that it's like faster JavaScript, basically. Um but you can run it outside of a browser context and it's useful for a lot more stuff than just stuff that runs in a browser.

33:17 So, how could WebAssembly help the whole like architecture I'm talking about? Well, one, I mean, it can solve the problem I mentioned earlier, like you don't ship a bunch of blobs for a whole bunch of different platforms and architectures, you just ship one, you know, bit of bytecode. So, that's cool. Um Rust is probably the best language for targeting WebAssembly from.

33:38 It has very good support for it. You just pick a back end in the compiler and you're good to go. There are some caveats there, like if you need to use, you know, sort of operating system stuff, like disk access and yada yada, that's going to work or not depending on exactly how you're executing the WebAssembly VM. So, you know, some some nuance there.

33:58 Um you can possibly avoid native extensions entirely, right? So, like I said I was going to give you a potential solution to that Java and Go thing. Here it maybe is. There are a couple implementations that are, you know, what I would call like pure, I guess. They're WebAssembly VM interpreters that are written in these respective languages. So, Java has one called Chickory, Go has one called Wasmtime.

34:25 These don't have native extensions because they're written in Java and Go, respectively. Right? Um and they appear to be like quite complete. I haven't had a chance to really use them like in anger yet. Um, but if you're doing just pure compute stuff, they will work. If you need to do things like the operating system level like file system network access things, they may or may not work, but I'll I'll talk about that again in a second.

34:52 And there's other like really cool stuff you could potentially do here, right? Um, like say you don't want to have to have your users like get a new version of your stuff and redeploy it just to get an update. You could potentially push, you know, dynamic updates uh of the core logic out, you know, and just push a new web assembly bottom and like start executing from there.

35:10 You know, depending on your use case that might be a really bad idea, but like if that's something that suits what you're doing, that's like I think that's really cool. Um, different topic entirely. Uh another thing I would do if I could kind of do it over again is use an ideal that doesn't involve a serialization step. Right? So, I I mentioned I kind of had to use Protobuf because like that's what the service uses.

35:32 Um, but like the whole thing I was talking about with the performance, you know, the little graphs and things, uh you could just not have that problem. There are ideals that don't involve serialization at all. Um, the two most popular ones are FlatBuffers and Cap'n Proto. These ideals don't have a serialization step like which is to say that they generate code in, you know, multiple languages that uses like the same physical memory layout, right?

35:56 So, you don't take some memory, turn it into a representation that's going to work in both places and then, you know, do that again. You just like you just both of them know how to read the same thing, right? So, it's I think on Cap'n Proto's site they say "Infinity faster" at the top of the thing. It's like, yeah, it's just it doesn't There is no step.

36:16 And this is something I would do if I could because if you profile Temporal's core, it spends um 90% of its time serializing. So, that's that's quite a lot, right? Um Last thing I would do is route side effects back through laying. This is kind of connects with a bunch of stuff I said earlier, right? Like I would just take everything that does side effects and probably pipe it back through the language layer.

36:40 Um which means, you know, network calls, logging, metrics, plus to a bunch of stuff, anything users might need to customize. And this serves like a dual purpose, right? Like one, it gives users all those customization hooks that might want to have. And two, if you like are doing this if you're like tomorrow you're like I'm going to do everything Spencer said, and you're doing this with web assembly, um now you don't have those concerns about like does it can it do the right OS calls or whatever cuz you just you just

37:10 don't make OS calls in your, you know, they all go back out to like language layer and the language is responsible for doing that. So, you can kind of potentially kill a bunch of birds with one stone here. Um so, let's wrap up a little bit. That's kind of like about all the advice I got. I do want to come back to this slide, you know. Um I can confidently say that engaging in this project like writing Rust core was worth it.

37:34 Not having to write all that crap over and over again has been like a huge win for my team and I. Um specifically uh in my estimation this cut the new development time for new languages in half. And probably more importantly, it has like ongoing payoff when we need to add new features, right? Like everything involves like we still have to touch the language layers, right?

37:56 Cuz we need to provide that like nice white glove experience to our users. We need to write like good APIs that they want to use. But like the logic part can be in the Rust core and you just write that one time. Also, there's definitely fewer bugs. Like this is difficult to quantify, but like it is obvious that like there's just no way that we have like more bugs with this architecture than we would with the other one.

38:17 There's just dramatically less code. Like therefore, there's less bugs. So, to summarize, some key takeaways for today, Rust is a cool language and it's particularly nice for if you need to write a bunch of shared logic. Um and if you're adopting this architecture, I would use those nice helper libraries I mentioned cuz they do quite a lot of heavy lifting for you.

38:40 Uh additionally, I would use some kind of code generation. It's not going to do everything for you, but it'll certainly help. And you have a lot of options to choose from here and if you're free of some of the constraints that I had, use one of those no serialization ones. Um this is more of philosophy than practical advice, but when you're designing stuff like this, like take the time to make like a very nice idiomatic experience for your users.

39:04 Like it can be tempting when you're doing this kind of thing to just auto-generate like everything and just like plumb the whole thing through and you have this like kind of crappy experience at the language layer that just doesn't really feel tailored or like cared for. And then lastly, um plan for hooks. Uh like I mentioned a couple slides ago, right?

39:24 Like kind of planning for these places where behavior might need be injected and kind of threading that through. I think this one's like quite hard to get right. You have to have a lot of forethought here, but like it's worth trying to think through the options. >> [music] [music]