Running Payments on AWS, GCP and Azure at Once Transcript, AI Summary & Key Points

00:10 Good morning. Um so in 2021 the UK's banking regulator published this statement in which they expressed their concerns that with banks moving to the cloud and a limited number of cloud providers available there were growing concentration risks that if any one cloud provider had an outage they were going to take with them a big chunk of the UK's financial services.

00:32 The key message from this document is that banks must be able to exit a cloud. So one of the largest banks in the UK, one of our customers heard this and said right we need a multi cloud strategy and in turn we looked at this and thought we need to be running on multiple clouds. So who are we? Well form three we power accounttocount payments. We connect banks and financial institutions to payment schemes in the UK, Europe and the US.

01:04 So if you've made a bank transfer recently, there's a pretty good chance that we handled it. You're welcome. Um, my name is Ross. Um, you're going to hear a bit more from me later on, but for now I'm going to hand over to Kev, our VP of engineering, who's going to tell you how we got our platform running on three clouds at once. >> Thank you very much, Ross.

01:26 Okay, before we move on to the multiloud architecture, I just want to take you on a little bit of a journey through time. Back around nine or 10 years ago when Formfree were just starting out. The world was a very different place back then. We only had between four and 10 engineers over the one to two two-year period. So we built an architecture that enabled us to move quickly.

01:47 We knowingly coupled ourselves to AWS as tightly as we could because we wanted to offload the responsibility of running uh technology to a large cloud provider so we wouldn't have to run it ourselves. So a high level architecture was because we're an API, we have a load balancer where traffic comes in. We use Java um to write microservices running in Docker on AWS ECS which obviously their pri proprietary Docker container runtime.

02:19 Because we have asynchronous workflows, we needed some kind of message bus to coordinate those those workflows. And for that we used AWS SQS. And to keep all of our data safe, we stored our data in uh a Postgress database running on RDS. And this was great for us. It enabled us to innovate, go to market quickly, and offload a lot of those responsibilities such as backing up our database, running it multi-AZ to AWS, so we didn't have to worry about it.

02:46 So what changed? So Ross obviously introduced this talk by telling you what changed where our customers started to need the multicloud strategy. we were successful as we were growing and those requirements were pushed down onto us. So that caused us to sort of take a step back and think about building a v2 platform that would be able to run across multiple cloud providers.

03:13 So what are some of the design goals we had for this platform? The first was we wanted it to be active active active. So on the V1 architecture I just showed you a minute ago, we got really good at running workloads where they could fail and a request was retrieded in another availability zone. So we got really really proficient at handling availability zone failure.

03:34 So we thought we could take that paradigm and as crazy as it sounds move it up to the cloud level. So what if we could treat a cloud provider like it was an availability zone. So could we use some of those learnings from our v1 platform and apply them to multiple clouds? So if any cloud failed, the world would carry on running like nothing had happened.

03:56 We also took a long hard look at the technologies that we'd use that had turbo boosted us for our first one to two years of form free and as I mentioned they all completely locked into AWS. So we needed to change that because we wanted to run across multiple cloud providers. So we picked some technologies that were cloud agnostic and we really moved the paradigm from allowing the clouds to run our technologies for us to running them ourselves um with the form free platform team.

04:31 And then that touches into the third design goal in that we now were almost going the complete opposite where we wanted to avoid cloud pacific technologies. And the reason for this is because we wanted the form free engineers who are building products um to run payments and allow account-to-account payments. We wanted to run that same software on any cloud provider.

04:50 We didn't want to be in a world where we needed one version of the software for AWS, a different version for Google and a different version for Azure because that would just be a maintenance nightmare. So this is the V2 architecture that we landed on. So how it looks is we have a Kubernetes cluster in Google, a Kubernetes cluster in Azure and a Kubernetes cluster in AWS.

05:15 So we have a load balancer which provides our API endpoint into any cloud. So our customer is able to send their their request to any cloud provider and they're able to do client side load balancing. So if for example they try to send a request to GCP and they don't hear back they can send the request to Azure um to make sure the request goes through.

05:39 We moved away from Java and over to Go for our microservices. There are a few reasons for that is that we found Go to be much more lightweight to fit much more um fit much better in the sort of microser architecture we're using and provided a much smaller deployment footprint and the biggest thing I think Go has going for it is that you can't write magic code.

06:00 Um, so when you, especially in the microservices world, when you're moving from repository to repository, you can just read through all of the code, including the the dependencies and it's really easy to understand. And then the secret source for this architecture was that we offloaded the data storage um challenges to two vendors and that is Nat Jetstream.

06:23 So Natsjet stream is a message broker written in go very high performance that's designed to run across multiple nodes and we actually run a single logical nature cluster across the free kubernetes clusters. So if you send a message in GCP you could receive that message back in Azure or vice versa. And for data storage we chose cockroach database. So cockroach database uses a Postgress um compatible dialect.

06:53 So it allowed for us to do an easy migration from Postgress we were using in V1 over to Cockroach in V2. And the real sort of um key key challenge that we're offloading to these vendors is keeping our data safe. So running a right consist right writing a right consistent data store that runs across most multiple nodes is a is a real challenge. But luckily it's not a challenge that a form free we had to solve because these two vendors could solve it for us.

07:25 So with all of this in place our applications could just run in a Kubernetes cluster locally to the cloud and just use u nats and cockroach to send messages and save their data and they almost didn't have to worry about how it was working under the hood. To connect all of this together, we had to privately network our Kubernetes clusters together using highly available redundant links.

07:50 Um, so just to say for the challenges coming up, any pod in any Kubernetes cluster can talk to any pod in any other Kubernetes cluster via pod IP address. So that was an underlying network that we put in place. So I've almost done a Quinton Tarantino and sort of shown you the finished solution. Um so I just want to rewind a bit now and explain to you some of the challenges that we faced in building this architecture.

08:19 There were many but this is like our three peaks challenge if you like the the biggest challenges that we had. The first one was bootstrapping this cockroach cluster across the three Kubernetes clusters. So as mentioned each Kubernetes cluster is completely independent but in order to run cockroach across the three we wanted a nine pod setup that is free cockroach pods in each cloud.

08:46 Now for this to work you need to tell each cockroach pod where all of the other pods are so they can connect together and then bootstrap the database and then become a single unified database. Now within a single cloud, this is actually quite straightforward because Kubernetes has a really good primitive for this called a headless service where it basically gives you like a DNS name that tracks the pod IP address.

09:14 So within cluster you can just talk to each other on like cockroach-0.svc.cluster.local as an example. But the challenge really comes in how do we address a pod on another cloud. So we can't hardcode an IP address because the IP addresses come and go as the pods recycle. So um our solution to this was that we decided to use DNS. Now another challenge to this might be well how do you know what the DNS um the DNS addresses of another cluster?

09:52 So our setup for this was it within each um Kubernetes cluster you have a built-in core DNS server that can resolve the sort of local names the svc.cluster.local names that are built in out of the box. Then the secret source the real engineering revelation we had was that well why not just introduce a new like pretend suffix where we add the cloud into the address.

10:18 So to walk through an example, we can see that if cockroach pod one, sorry, cockroach pod two on on Google cloud wants to talk to cockroach pod one on Azure, it can address it using this special address where Azure is inserted into the the DNS name. Now to make this work, the core DNS within each cluster has some special rules in it where if it sees the cloud in the DNS name, it forwards the request over to the other cloud.

10:49 And and why it can do that is because we've we've installed a network load balancer in each cloud on a static IP address. And that network load balancer just points to the core DNS uh server running in the the respective Kubernetes cluster. So with all of that in place, you can see that if I just walk through an example, the cockroach pod 2 can address uh cockroach pod one in Azure using the name you see on the screen.

11:18 When it passes it across, we have to do some um funky rewrite rules within the cloud to basically strip off the Azure bit because we haven't really done anything special in the in Kubernetes itself. We still use the built-in DNS names. So we rip off the Azure um part from the DNS name and then we just resolve using the normal headless service name get the IP address and then rewrite it back to the caller.

11:43 So with all of that in place that enable now enables us as if by magic just just to refer to any pod in any other cluster using the normal DNS name just with the cloud sort of inserted in it. I thought that was a really neat solution where we're sort of building on top of the the primitives that Kubernetes already provides. Okay, the next the next peak or the next challenge we had is that each cluster thinks it's alone and that's because it is alone.

12:14 So we made an a sort of decision to run three separate Kubernetes clusters. The reason for that is we wanted to increase our fault tolerance. So if you run a single logical Kubernetes cluster across the three clouds, you would have had a single point of failure where if that control plane went down, you would have lost the whole cluster. So we didn't want that.

12:37 So instead, we we knowingly took the decision to have three separate clusters. But the challenge with that now is that some of the built-in primitives obviously only know about their local cluster. So with this cockroach database that we've just been speaking about that we want to run across the three clouds that has nine cockroach pods underneath the bonnet.

12:59 Cockroach uses an algorithm called raft um for cons for data consensus. So the raft algorithm says that for a nine node cluster you need five pods to be alive otherwise you have a database outage which is obviously very bad. Now, what you want to do with a sort of um with a workload such as this that you need to keep running, a high resiliency workload is you want to be able to tell you want to nudge Kubernetes to tell it what it can and can't do when it's doing node rollouts and rescheduling pods.

13:37 Now, in order to do this, Kubernetes actually provides a primitive out of the box called a pod disruption budget. Now, what a pod disruption budget really is is you can tell each cloud, okay, at maximum you can only take down one of my pods if you're rolling a node or moving a pod from one node to another node. But remember, this is actually a single database.

13:57 So, the best we can do with the built-in primitives is the setup you see on the screen to say per cloud you could have a max unavailable of one pod. Now that would mean that Kubernetes would be well in within its rights to take one pod offline in every cloud and that would be a valid configuration. But now we're running very close to our resiliency um tipping point where we need five nodes running otherwise we're going to have an outage.

14:26 And in fact if we got ourselves into this state if we lost the cloud from here we'd actually drop down to four nodes and we'd have a total data space outage. So that was obviously not going to work for us. So our solution to this problem was we built a a new primitive called a crosscluster pod disruption budget. We termed it XPDB. And with this in place, it allows you to span a pod disruption budget across n clusters.

14:55 So in our situation, we spann it across three clusters. But with this in place, we can now say only allow one pod to be taken down at any one time across the three clusters. So if a pod in Azure is is sorry, if the Kubernetes in Azure is trying to sort of move a pod, it won't be able to if there's one been taken down in Google as an example. So this gives you much higher resiliency and allows you to run with a lot more nodes online at any one time.

15:22 But again, it was just building on the the sort of primitives that Kubernetes already provided and extending them across multiple clouds. If you're if you're interested in this and want to dive a bit deeper, my colleague Maritz actually did a YouTube did a talk on this. You can go to that link to watch the YouTube video and we've also open sourced this um operator on GitHub.

15:48 So the last peak, this peak is actually a day two challenge that we had which was after we built the platform and we were running it for a little while. We realized that some of the decisions we had made were coming back to bite us in terms of trying to maintain it and keep it online. So you can imagine the world we're in. We are a payments provider.

16:11 We have to keep everything updated all of the time which basically means always rolling nodes every time they've got a CVE with the latest operating system. Now our initial setup that you can see on the screen per kubernetes cluster we had multiple node pools um where we grouped different technologies like NAT cockroach go etc. Because we're an infrastructure as code shop every one of those nooles was defined in code.

16:38 So you now imagine if you wanted to do a operating system update, you'd have to do a PR per node pool per cloud and that would be a lot of pull requests. You could say you could change them all in one go, but that would obviously be very disruptive because all the node pools be rolling at the same time, which you probably wouldn't want to do um because you want to try and keep things more stable than that.

17:06 So this was a problem that was really killing us in terms of trying to maintain the system but it actually like I haven't even explained the problem actually was a lot worse than this because every environment doesn't live on its own we have development we have staging we have production where we promote changes through so you can imagine all the the PRs have just multiplied but actually gets even worse because we run platforms in different geographies for different product sets for different customers so you can

17:32 basically think keeping all these no pools up to date was basically a full-time job to create all these PRs and merge them and make sure everything's um merged. Okay. So, this is uh what I'd like you to consider is what is the right level of abstraction when you're modeling something as infrastructure as code because I think infrastructure as code is great but the challenge we made is we were too granular with how we're modeling it and it put ourselves into this bind where trying to keep things up to date was just a

18:03 just a nightmare. So the way we got out of this or our solution to this was we wrote our own operator to manage our npools. The operator is called the cluster life cycle operator or CLO. And basically what it does is it manages all the nools and then we can just define our own interface with the CLO to tell it how to operate. So we can build in lots of rules like how many nles you can roll simultaneously, how quickly to roll them out and all of that good stuff.

18:35 And then at the top level we can just tell CLO what image we want it to be running across all of the n pools and then it can just manage that for us. So what was hundreds and hundreds of PRs becomes one PR per platform now which is obviously much more manageable. So that was a real gamecher for us. So all of this so far, all this theory, that's all great and all, but you're probably asking yourself, well, does this multicloud thing really work?

19:04 Looks quite um, you know, far-fetched. The story I have for you was back in the summer of last year. Um, I was on the train on the way home and the football team I support had actually just hired a new manager. So the sucker I am really excited about this. So, I've got my phone out cuz I love watching the YouTube channel where they analyze the tactics and stuff.

19:26 And I'm sure you've seen it, but these YouTube channels today, they're so high tech, it's not even, it's almost like watching a TV channel. There's like a bar across the bottom scrolling with the news. There's multiple people on the screen. There's music, there's a background. It's all very impressive. And I got my phone out and opened the channel. And there was there was no there was no scrolling bar across the bottom.

19:52 There was no multiple people on the screen. There was no background. There was just one guy like glitching in and out. And I was like, "What's going on here? It's like amateur hour." And then he was like, "Oh, you have to bear with me. The streaming um software I'm using is not not working at the moment." I was like, "Oh, okay. That explains it." And then I read in the comments what was going on and it was this.

20:12 Um, Google Cloud was having one of their largest outages um, which is around about three or four hours. So then it's that feeling that hits you. You're like, "Oh, so close my phone, work laptop out. I thought I'm just have to go in and check check the system and see what's happening." And it was one of those really proud career moments where I could open my laptop, look at our alerts, and all that we had was like a low-level alert where we had some um pods crash looping in GCP, but all of the payments were continuing

20:48 to flow and they were picked up by the other clouds. And that's where our decision to treat a cloud provider as an availability zone really got proved out. Okay. gonna hand over to Ross. >> In 2022, having kind of established ourselves in the UK and in Europe, Form 3 decided it was time to make it in America. So when I joined the team later that year, the they were busily building out our triple active multicloud setup in the image of our UK environments.

21:30 But our sales team in the States was having a hard time pitching this to our potential customers. And what we came to realize is everything's bigger in America. And if you look at it from a US perspective, our state-of-the-art solution almost starts to look a little bit quaint. And you can sort of imagine how our customers in the States might look at this and see and think that we're kind of putting all of our eggs in one geographical basket.

22:00 Um our initial reaction was to push back on this and my predecessor is architect in the US Sam Tavakoli did some great work looking at all the different types of disasters that might befall us in the US the natural man-made the far-fetched and he concluded that really there was no set of circumstances that would cause us to lose more than one cloud or data center at a time.

22:22 So really we could just lift and shift our architecture from the UK to the US and run it without a hitch. Um, and it's tempting to get a bit cynical about this to imagine that our prospective customers were misguided in some way or that the disasters they were preparing for were just sort of asteroids of the mind. Um, but really we weren't telling the right resilience story for the US market.

22:50 And I think any market or industry you look at has got its own sub set of resilience stories, its own kind of mantras and metaphors that they live by. And our customers in the US were used to talking about geographical resilience. They were used to running things on east coast with a backup on the west coast. And so as a new entrant to this market with no proven experience in the US, we didn't want to give our sales team an extra hurdle to clear by trying to pitch this novel solution.

23:19 Now we could conceivably take our triple active multiloud and try and spread it out across the US. But this is not an option for us because everything's bigger in America including latency. Um, every time we process an inbound payment from Scheme, we have to validate it, store it, orchestrate it across a bunch of services, send it to the customer for approval, and then once it's approved, we'll send the message back to Scheme and then we await one more message so that we can set the final status on that transaction.

23:52 And every time we write to the database and every time we put a message on that, we have to get corum across the clouds. So if we're spread out across the US, we could easily burn through our SLAs's just waiting for electrons to cross the continent. Now thankfully although our customers wanted to have geographical resilience, they also understood the associated trade-offs.

24:18 They were used to seeing a bit of downtime and accepting the possible loss of some data in the case of a disaster. So working with a non-zero RTO and RPO, we could go back to first principles and find a solution that fit the market and kept our cost sensible in the US. So we evaluated a bunch of options, ruled most of them out for various reasons and in the end settled on an architecture that was probably more familiar to our customers in the US, just a good old active standby.

24:48 So, we were going to run in the AWS on the east coast and if the proverbial hits the fan, we'll flip over to GCP in the west coast. Now, in order to move a running payments platform between clouds, we've got to move three things: services, data, and connectivity. Now, services are the easy bit. So, let's talk about data. We've got three data stores to worry about.

25:11 As Kev mentioned, we're using Cockroach for most things. We've got to keep that consistent because we're doing a lot of things like duplicate checks. Um, open search powers our read model for payments. So, this is what you hit if you do a get request to slash payments. And then Postgress is a little bit of a vestage from our v1 platform. This is used by our audit service at least for now.

25:38 Um, annoyingly each of these has at least one replication mechanism. um each of which requires its own setup and management. Um and it's sort of exacerbated by open search and postgress because we're using hosted cloud services to run these where we don't have exact control of which software version they're running or we can't install plugins in some of them.

26:00 So this was going to be a bit of a pain to run. Um and we with limited time got ruthless. We decided first of all to do away with Postgress and the associated service on the disaster recovery site. Things happening over there would still be auditable but only once you got back to AWS. Um for the other two we decided just to lean into the RTO and RPO that we had available and do the most basic thing uh which is to rely on backups.

26:30 So Cockroach and Open Search write their backups to S3. Kev's team provided us with a tool that pushes those over into GCS and then if we have to fail over we'll restore those um from GCS uh into those databases and for the amount of data that we've got to shift and the frequency of backups that we can take this gives us an acceptable RTO and RPO but of course after a disaster although our customers are happy to you know accept some possible data loss once we get them back up and running, they're understandably going

27:05 to ask us what's happened to the payments they might have missed. Uh, and we didn't want to have teams trying to deal with this manually or poke around in the data. So, we've set up a process whereby we can ingest reports from the scheme to fill in some of the the missing payment data. But this gave us a new problem. Our payments are identified by UYU IDs.

27:25 But in the scheme, the ID space is much smaller. So when we send the payment to scheme, we've got a process by which we um map our ids onto the scheme ids. That process is deterministic, but it is lossy. So we store that mapping in the database, send payment to scheme. Now if after a disaster, we've forgotten about that mapping, then when we ingest a report from Scheme, we're not going to know what Scheme is talking about, and we can't inform the customer of what's happened to their payment.

27:57 So what we decided to do here is to meet our customer halfway. They'll send us IDs of their recently submitted payments which allows us to regenerate that mapping and inform the customer of the outcome. Okay, that's data. Let's talk connectivity. So the first payment scheme that we put live in the US is TCHRTP. We connect into that using IBMMQ over VPN.

28:24 the the scheme has two physical locations and we as participants have two logical sites each with a an MQ broker and we crossconnect them into scheme. So the naive implementation of that would just be to say let's put one on each cloud except we're not going to be running in GCP very often um which leaves us with a single point of failure in AWS. Um in practice we found that running MQ in Kubernetes needs a bit of effort.

28:53 losses of connectivity can be quite slow to recover. So we have to be really careful about rolling nodes software updates of MQ. So what we decided to do instead was to run both of our MQs in AWS and if the worst happens we'll flip them over into GCP. Um the process for flipping those over at Scheme is that you um fill out a word document with the IP addresses of your MQ brokers and then email that to them.

29:21 they'll pick that up in business hours and they'll process it when you know four hours later. So, it's an admin process. It's not really intended to be used for for emergencies. So, what we've done in instead is have each of these pair of MQ brokers present the same IP addresses and then we flip between them using BGP. So, when we want to get from AWS to to GCP, we stop publishing the roots to AWS, start publishing routes into um into GCP.

29:52 Okay, so now we move from the technical to the tactical. Under what circumstances are we actually going to initiate a failover from one site to the other? Well, we settled on this question which is now at the top of our failover runbook, which is is it likely that AWS will take significantly longer to recover than our predicted failover time. So significantly longer because when we fail a customer over, they're going to want to get back to AWS.

30:17 it's physically closer to them. As I mentioned, we're not running every service in the DR site. Our predicted failover time, this couldn't be a guess. This had to be empirical from measured rehearsals. And finally, likely based on what we know about this incident and what we've seen in previous AWS incidents. So my friend chat GPT and I went through the complete history of AWS incidents and looked at possible um you know look at the outage duration and the affected services and we concluded that based on our predicted

30:51 failover time there was one of those incidents where we might have benefited from failing over. So a lot of the chat prior to this service going live was we are never going to do this for real. We'll demonstrate it the to the customer, but surely we'll never actually use this thing. The service went live in October of 2025 and enjoyed two silky smooth weeks of operation before this happened.

31:20 So, um AWS were having a bad day. We could see the list of affected services growing minute by minute. Um we could see impacts across the industry, but we were getting away with it. Um, I think our choice to be cloud agnostic meant that things affecting AWS's control plane didn't really hit us until it did. Um, our VPN went down. We lost our connection to Scheme.

31:44 Payment stopped flowing. So, we had a decision to make. Now, we could sit on our hands and wait for AWS to fix the issue or we could initiate a failover and get our customer back up and running. Yay, save the customer some time. But as I mentioned, we will eventually have to get the customer back to AWS. So we got to price that in, which reduces the benefit of the whole failover thing.

32:08 And of course, we don't know how long this underlying incident is going to last. So if we take too long to act or if it's shorter than we thought, we could actually be making the situation worse for our customer. So this kind of left us stuck between two bad options. Now, over the years, we've become pretty good at handling incidents. Our engineers take their own call duties really seriously.

32:30 We've got an excellent team of incident managers. They're all biased to action, want to fix stuff for the customer, but in this situation, they kind of felt trapped between two bad options. In the end, AWS recovered. We didn't fail over. If we had failed over, the customer would have experienced a longer outage overall. So, we probably made the right decision, but it really didn't feel like it.

32:56 And if we've been running our triple active UK style multiclouds, we could have just laughed this off and spent our time writing blog posts about how resilient we are. But instead, we had to go back and look at our options and say like the only way that we're going to retain any control in this situation is if our whole fillover process is faster. So we realize that data restore is a real bottleneck for us.

33:21 So um we've set a whole bunch of things in motion. I'm going to give you a kind of speedrun through some work in progress that we're looking at now to make this better. So the first thing is we're adding logical data replication in cockroach between our clouds. Um logical data replication is biirectional asynchronous SQL level. Um it doesn't guarantee consistency but we'll be safe so long as we're only writing to one cloud at a time.

33:51 Um it also has some uh some good conflict detection so that we've got a bit of peace of mind there. Um the second thing then is rather than try to deal with all the other sort of aggregating services uh and replicate each one individually, we're just going to replicate our events bus. So we'll replicate our NATS streams over to into the other cloud and then those services can consume and write to their data stores locally.

34:18 These two changes will vastly reduce our RTO and RPO. So we're really just limited then by how long it takes to do that BGP switch and reconnect to MQ on the other side. So our customers should be pretty satisfied by this, but in turn they need to satisfy their regulators. And one of the things that they'll need to do there is demonstrate their disaster recovery capac capacity in production annually.

34:43 That's fine for them, but we run a multi-tenented platform. So we need a way for them to do that without disrupting other customers on the platform. So we're putting this change in place now to allow customers to shift their processing from one cloud to the other while maintaining their connection to Scheme through AWS. We're doing that by splitting the NAT subjects between our MQ adapter component and the the scheme gateways by customer so that the gateways in each cloud can be configured for which customers they're

35:16 interested in and just read those messages. Again, we're relying here on the fact that our data is segregated by customer and so long as the customer is only active on one site at a time, we're safe in terms of consistency. Behind the scenes, we've made some NATS changes. So services that act on individual transactions will only see events from the same cloud.

35:39 Whereas aggregating services will get events from everywhere. And what this means is that individual transactions will always be processed to their conclusion on one cloud, but then after replication will be visible everywhere. So when the customer flips back from AWS to GCP, they can see the payments that they created in GCP. Now there are a couple of um trip hazards for doing this sort of thing particularly the asynchronous database replication and one of those is outboxes.

36:08 We use the outbox pattern in a load of services where if you've got something like an API that writes a change to payment database say it will write that change to the payment table and then in the same transaction it will write a row into an outbox table containing all of the events that need to be emitted relating to that change. And periodically, we run an outbox processor that reads from the outbox table and emits an event overn.

36:36 Finally, then it'll delete that row from the outbox table. So that's fine. But if you do it in duplicate with asynchronous replication, there's a danger that events that are written on one cloud get replicated and then get processed on the wrong cloud. Meaning that event ends up in a service where that customer is configured off and nothing will happen.

36:58 So what we decided to do here is to keep the replication for the the resources themselves for the payments but break the replication for the outboxes meaning that the events will only be emitted on the clouds where the things have happened. And we've got a similar issue here with timers. We do a number of things where we have longunning timers. We record the fact that something needs to happen maybe days hence.

37:19 Um and so we want to make sure that those things get replicated so that they follow the customer around. Um, but we then need to configure the the processing part to make sure that they only act on um timers for in the cloud where that customer is configured on. But of course, this is all cool, but most of the time we're just going to have all of our customers running on AWS with just the peace of mind that we've got this really quick replication going on.

37:50 We've got a really warm standby site and a much faster failover time. Um, one last thing I should mention here is that this is really, this step, this last step I mentioned is the only time that we've had to make significant changes to our services. Everything else we talked about so far today has kind of happened below the surf below the surface. It's been handled by our platform team.

38:13 Speaking of whom, let's go invite Kev back to wrap us up. Okay, so how did we run on three clouds at once? The first one was that we picked cloud agnostic technologies such as Kubernetes, NATS, and Cockroach. So we could run the same code bases everywhere and not have to run separate code bases on each cloud provider. The second pillar was the fact that we use single data stores across the three clouds.

38:45 that really made writing the software and thinking about the software even though it is active active active across three clouds much easier to reason about. And the third one was that we treated each cloud provider like it was an availability zone. Reasons potentially to not go down this path are if what you're building doesn't fit the resilience stories for the the market that you're selling into.

39:10 If your state-of-the-art solution doesn't impress your customers, then it's maybe not worth your effort or your money for that matter. Running triple active multicloud is an expensive business. Uh, and bankruptcy is kind of incompatible with uptime. Um, and then finally, I wouldn't go down this road unless you've got a really robust platform engineering practice and a really capable and trusted platform team who can make this happen and let everybody else keep cracking on above the surface.

39:40 So with that, thank you very much uh for listening and we'll happily happily take any questions you've got. >> How did you get customers comfortable talking about Z nonzero RPO RTO? How did they Yeah. is actually the default in the market. So in in the US the the expectation is that you run on the east coast and the west coast. That that's kind of what most people seem to be doing.

40:18 Um multicloud is a bit seems to be quite a new concept there. We're only seeing some some of our competitors starting to catch up to the idea of being multiloud now but there is an expectation of having to sort of shift from one to the other and not run active active across the two. So you come to customers and they understand east west and they understand that means data loss.

40:40 >> Yep. Yeah. >> Yeah. That was weird for us too. I have to say that's the thing. So we Yeah. It felt like an odd step backwards but yeah that's that's how we got to it. All right. So I actually have a question too. So um it's a so how do you how does the company generally feel about a potential outage in one of the cloud providers and everybody having to go somewhere else and then everybody's going somewhere else at the same time.

41:19 So what what are your thoughts there? >> Do you mean on a multiloud side on the active active active platform or on the on the US because I think the answer is probably different. >> Uh multiloud cloud to cloud. >> Okay. So in the on the multicloud platform we've gotten to the position I wouldn't say we're blas about it but we don't really consider it.

41:39 So I'd say over the last the back half of 2025 we saw each of the three major cloud providers have decent size outages the Google one that I spoke about the one that Ross showed and Azour had quite a large one as well. I think it was in October and none of those impacted our platform. So we're actually gotten to the position now where the platform it really is like treating a cloud provider like an availability zone.

42:03 So I think the key is we don't we don't switch people across the clouds. They just use all of them at the same time. >> Thank you. >> Hi. Uh how elastic are you with what you're doing in those Kubernetes clusters? Is this stuff scaling up and scaling down or are you looking at more kind of like fixed processing and so that sort of gives you more comfort in that scenario that we're talking about where oh suddenly everybody wants to be on AWS because Google and the other one have gone away.

42:38 >> No. So the answer today is we are not very elastic at all. We are more pre-scaled for the for the volume that we know the customers can give us. So customers give us sort of a six month projected volume and we'll test that on an early environment to make sure we can cope with it and we add some sort of um tolerance. So if one cloud's down the other two clouds can handle the load but we don't do anything in the sort of dynamic scaling space at the moment.

43:05 >> Works nicely for that scenario. So yeah makes sense. Thank you. Yeah, just a just a quick sort of high level question around trade-offs actually. So obviously you're getting a lot of resilience across the three clouds in in the UK example, but what are the trade-offs in terms of cost? because that's usually a big tradeoff is you know we we can have everything all singing all dancing but normally there's limits and constraints so how did you go about analyzing those tradeoffs and especially the cost side of things

43:50 and the complexity as well >> yeah I think the um the cost question is an interesting one we've made sort of a number of strides forward to reduce the cost of running the platform we're going to continue to do that it's definitely more expensive to run across free clouds there's no getting away from that. Um, but I I do think there's things you can do to make it sort of a bit cheaper than what it may seem.

44:12 Um, but yeah, it's definitely an interesting tradeoff. I guess the number one sort of driving factor for us is that this is what the market demanded, what our customers wanted and needed. Um, so it's the sort of platform that we built to satisfy that market. So I guess cost, yes, it's it's more expensive than single cloud, but it was like the the product that fit the market.

44:35 So that's what we went for that solution. >> There there's an interesting trade-off we made with um cost recently that sort of surprised me. So one of the when Kev was showing like the number of different environments were having to look after particularly those pre-production environments that need to be scaled to least production size or production plus 6 months.

44:53 Um, one of the things we've done is have multiple production environments because otherwise you have to if we had one enormous production environment, we would have to reflect that scale in the pre-production environment and so that you then end up carrying more cost overall. So what we've ended up with is some multiple smaller production environments and then match that with its with the pre-production environment.