Relate is an open-source embedded graph database written in Rust, designed for agent use cases. Nicknamed “Archie,” it stores relationships among code, checks, and evidence so applications can assemble connected briefs and evidence packets before generating written output. The evidence can be preserved outside an LLM conversation, supporting provenance-aware review without additional LLM calls and reducing repeated searches through session logs.
Tracetoon is an open-source tool that generates pull-request descriptions from AI-agent session logs associated with a branch. It identifies the sessions that produced the branch, creates a redacted evidence digest covering the requested work, approved plan, decisions, and recorded tests, builds, and lint runs, and has an agent write the description from that evidence. An advisory claim check flags statements that the evidence contradicts or does not establish; uncaptured activity is reported as “not captured” rather than inferred. Users preview and approve the text before posting, and edits made in the pull request are preserved on later updates. It can optionally generate a one-off comic for the pull request. The project also describes a Relate-backed evidence graph and connected briefs for assembling provenance and relationships before writing and review.
Searchable transcript of Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin — AI Engineer (23:01). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 Hello and welcome to our ag engineer San Francisco talk. My name is Steven Chin. I'm joined by Cassandra Chin who's the creator of the t trace tune framework. >> Hi, I'm Cassandra Chin. I built trace tune because I wanted to improve PRs. I see a lot of developers doing great things. The speed of innovation is as fast as it has ever been with AI technologies.
00:24 But a lot of poll requests are getting rejected. And it's not because the code is bad. It's because the PRs often contain incorrect information or are written poorly. So hopefully with Tune, we can improve that. It also has the option to add a fun comic to help explain your pull requests in a funny way. >> Cool. And um >> what we're going to do today is we're going to go through a little bit about kind of the common issues of PRs, how to improve the quality, how to improve token efficiency on this, and hopefully help
00:58 everyone to be a better AI engineer for your organization. Like our main character, Alex, who all he wants in life is to get an AI engineer hoodie. >> That's a really neat hoodie. I have a lot of AI engineer t-shirts, but I don't have the hoodie yet. >> Get switched to make this AI engineer hoodie. Everyone >> everyone needs an >> ready. And um this is enough motivation that hopefully he's he's going to do an amazing talk at the conference about his use case study on what he's doing in his company to improve the
01:35 quality of PRs. Here is a picture of the token factory where Alex works. So, um, Alex works together with a bunch of agents. He has, um, of course, finance looking over his shoulder because we we all spend too much tokens and they want to make sure we're keeping up with costs. Um, his buddy there, Pip, he's walking with carries all the evidence and looks into the the archives and what's actually happening.
02:05 And then, u Meera does all the reviews. So, she's she's keeping track of things. >> It looks like Pip left his station empty >> part of the factory until until he sees what Alex produced for a PR. Oh, no. >> That's a very long ledger. A lot to review. >> A lot of code. So, it's it's hard. And Merror's Merror's going through everything for the fine tune comb.
02:38 She actually is red lines. Found some issues with the PR. And this is a a real PR. I think this is actually one of your trace tune PRs where it needs to have exact preview links for the URLs, command attribution, make sure the earlier work is um qualified as well and explain all of this in detail in the PR record to be accepted by Mera's very critical review.
03:07 >> This was interesting cuz as I was building trace tune, I was actually like as I was building it, I was using trace tune to test it. So I would have trace tune build the PRs and as I improved the PRs it would like get better. >> You got amazing comics along the way too just to have fun while you were building stuff. >> Addictive. [laughter] So um you can see that the repeated PRs which Alex is doing to try to get good quality results are not working out well.
03:41 is working out for his token billionaire card. Do do you know anything about the token billionaire program? >> Yeah, so actually I am a token billionaire. I spent lots and lots of tokens to build trace tune. >> That's kind of unfortunate. Was that was that all my money? >> Uh don't sweat the details. >> Okay. So, um I I think there's a lot of good uses for for tokens and for spending more AI work.
04:12 Um maybe we're doing repeated PRs is not where you want to spend your token >> and definitely the finance department's not happy with this. So, he wants a results and he wants to make sure that the PRs are accurate, are working and he has they don't have need to keep getting repeated again and again by reviewing the logs and reconstructing the evidence repeatedly.
04:36 And it's interesting cuz normally when you're just like uh coding on your agent, you're in your session and you ask it to write a PR, it just reads your session log to try to figure out what it should have done using its inflated context window. It actually often like as it's churning it compresses its context window. So it gets more and more inaccurate as it goes along.
05:05 So instead, what we're doing here is with trace tune, we're having it read through uh different sessions on your computer to try to really figure out what you built and what happened. >> Yeah. And I think like like that concept of um looking through the audit trails, looking through like what the agents have actually done is a lot of what you were doing with the Showtale project you were doing for education.
05:31 >> Yeah. I actually started this project with this showtail project because there was this like academic research study I was supporting. But basically the issue they hit is students were creating programs but there was no way to trace back the actual work and effort they did cuz with AI coding the what you submit is the final work the program but most of the work is the prompting and everything that came before and that doesn't get submitted with the program and I use the technology and theory behind that to be like
06:07 well I can also improve poll requests with all this data collection. >> Cool. And I think that improvement in pull requests is something that benefits everybody because now you can find the evidence like um logging the change directory before the or the CWD before the CD runs. explaining why maybe other test data is leaking into other repos and explaining what's happening behind the scenes on the PR rather than what most AIPRs read like which is a line by line explanation of the diffs since that's what they build the
06:42 PRs off of but incorrect facts can still pollute the information um can cause a problem with conflicting PRs and then still give you back inaccurate results. So maybe we have a solution for this. What What do you What do you think? Do you have a solution, Cassandra? >> Yeah, everyone should use Trace June because of the comics. >> So because of the comics, not because it gives you better PRs, right?
07:15 >> Well, I started with the comics and I kind of went backwards and like, oh, I need to actually make the PRs better. Well, hopefully better PRs, better comics, and maybe better token savings if by the end of this. Maybe >> it should be. >> So, um, trade students an open source project. It lets you improve the quality of your PRs by pulling in agent context by pulling in information about the the code and then having an independent writer independent reviewer build better quality PRs plus optionally giving you an
07:54 amazing comic which what do you think the best feature is here Cassandra? >> I think it's obviously the comics. This is one of the earlier comics from when I was using traune to build it. That's good. And then you you you made sure that there's new characters every episode, not none of this repeating characters. >> Yeah, that's what this PR was about.
08:22 >> Okay. And then this is a description of the original trace tune flow. So it finds all the traces for a particular branch. It researches them, reviews the description, adds more context, and kind of goes in a loop until it gets all of the evidence it needs to make a a quality PR. And um subjectively, but also through um PRB bench, which is testing framework, um gives you better quality, better research, lower um rate of inaccuracies in the PRs.
08:51 I think at this point the PRs were already better than if you don't use trace dune, but I found that it still makes some mistakes occasionally. You can see at the end that it tries to revise with more context. I think that's where the mistakes come in. [snorts] Okay. Well, Alex Alex thinks it's good enough. He's going to try to solve the token crisis the factory with trace tune and see if this will solve all of his problems.
09:17 Polish checked. Alex looks very very happy but Mera doesn't look that happy. >> It looks like we still hit a mistake here like I was saying earlier. So the original PR claim was no later passing run is recorded. And the problem here is the in the actual trace it shows it running a bun test at the same time as the bun xtsc and you can't run those two at the same time.
09:50 They have to be run sequentially for the test to pass. >> Yeah, exactly. So he claims test passing and all the checks were run and actually that's not what the code did. So poor Alex. Another failed PR, a very angry badger with all the token cost. Um kind of kind of makes me feel like Monday morning with the Neo Forj finance department looking at my token spend each week.
10:17 But they're they're very forgiving, [laughter] especially when it's a important project like like this. So maybe Alex can save some cost by using cheaper writers is the number one thing finance asked me to do. They're like, "Hey, hey, no more opus on it. Haiku, do something cheaper, please. Does this work?" >> I feel like the quality is a bit lower when you do it like that.
10:42 Just reducing the cost doesn't quite help. >> Yeah. So, we were trying to get better quality and we we ended up getting worse quality by just using cheaper writers. And obviously, Alex is in a bind here, right? He's he's spent too much on revisions. He doesn't have a solution for reducing token costs. He's not getting the results he wants out of his PRs.
11:04 But maybe there's a there's something Pip can help us with. Pip Pip has a friend. >> I wonder who this friend could be. >> Oh, well, I I think I think you know from my previous talks that that this friend is definitely going to be a graph. >> You're the graph guy. We're doing another graph again. >> Yeah. Yeah. There's always a new graph, but this is not the same old graph.
11:30 This is not the the Neo Forj database. This is our new friend Archie, who is a embedded database written in Rust called relate that actually is being launched this week at the nodes conference and also this talk at a engineer in San Francisco. And it's an open- source um database which allows us to get a lot of the benefits of more accuracy, multihop queries, more explainable traces without having the heavyweight approach.
12:07 So it's kind of an agent focused graph database specifically for these sort of embedded use cases. Okay, so let's see how Archie can help us out. So before we were relying a lot on on agents and loops of agents to resolve the problem. But graph databases can store relationships can find like like different things which are clustered build a brief which connects all this information together and then give the agents even a cheaper agent better information and more context to give a good drafting decision.
12:43 And the good thing about Archie is he has zero LLM costs. It's all local compute cost and storage and it's a fully immemory database and rest so it's highly optimized. So we get this providence about all these submitted changes and this evidence graph basically for free compared to all the LLM and token costs which we were doing before. And this means our our buddy Alex doesn't need to research.
13:10 She doesn't need to go through and find all the facts again and go through the sources because relate is going to do all that hard work for him and build an evidence packet that he can use for trace tune. >> So all sounds really great. Maybe we do need more graphs. >> That's I'm surprised to hear that from you after all the graph talks you've had to put up with.
13:31 Okay, but we can do this and make it even better with the neighborhood brief. So basically taking all of the the Y the code and the checks putting attributions and relationships between those and then returning back a connected set of information as a packet. And by doing this then when the agent is drafting up the PR it has all the connected facts with their sources together in a single source.
13:57 And that makes the writer draft more accurate with fewer passes and it also makes the reviewer's job much easier. So we don't need an expensive reviewer. We can save by using a few haiku reviewers instead of a um sonnet reviewer or or a luna in the case of codeex to find come up with the findings check those against the draft redo the writing and then we retain the higher quality writer a a sonnet or terror class writer to make sure we have a high quality brief that will pass human approval.
14:33 When we were building this, it felt very counterintuitive to have the writer be high higher quality than the reviewers. >> It actually it it's common that people always want to have the planner, the reviewer, the checker to be the highest quality model. But if you're doing high quality writing, a lower model, no matter how many iterations and how much feedback you give it, can't do it.
14:54 It just turns in things into a checklist and makes it very mechanical. So to give writing which feels humanlike, you do need a higher quality model. And it's it's actually the the reviewing and the cross-checking step which is more mechanical. So it's a little counterintuitive, but that works out quite well. It's also a little counterintuitive that um our badger has a thumbs up coming out of his chest, but let's let's ignore the high quality AI artwork.
15:25 Okay, so we've checked all the claims against the evidence. We've looked for missing findings with our cheaper reviewer. Did the focus work and then found the things which were connected information to give a better draft and review for mirror. And here's the output. So it has the the Y. So log CWD and that describes the shell before the CD runs. What's the exception?
15:54 So the CD targets in repo or the command mentions a route. What passed? It has type check plus 71 test checked on submitted content not before. And then it starts in correlate.ts inspecting the leading CD exception. And Meera looks pretty happy with these results, right? >> Yeah. This might be the first scene where she's genuinely happy. >> I think she was kind of happy with the the AI engineer sweatshirt before she realized that Alex actually was a was a goof.
16:25 Okay, so here are some of the actual data and results from the sideby-side analysis using PRBench, which is a um pull request benchmark, which uses audit traces plus code and um LM judges to review the quality of the work. Of course, reviewed by a human as well for accuracy. And you can see that on both Claude and Codeex, you get a significant quality improvements on a scale of 1 to 100.
16:54 The judges by and large almost unilaterally liked the trace tune plus relate quality better. Um on a 1 to 100 scale it was like 74 to 76 82 to 86. So plus two plus 4 point. >> These tests contributed to my token billionaire card because we were running lots and lots of tests. We created this side project called PRB bench just for testing and it we would have different agents test over different poll requests and compare the results.
17:24 Initially I had it like running like sequentially. It'll do one test and finish it then do the next but then we decided to run them all in parallel. It's a very quick way to reach high spend. >> Yeah. So that's kind of funny. I I at at at at the current testing rate with the number of use cases and tests and the benchmark um we can fully saturate my my high-end gaming computer with as many threads as Linux can handle simultaneously and and of course the token spend is is slight slightly higher than what we'd like.
18:02 But it's all for the good of improving the quality of pull requests for the world. So you all get the benefit of all these test runs and having the highest quality improvements and you save on tokens. So we've spent the tokens for you in advance because trace tune with the graph solution takes about half as many tokens as it did previously and which is an amazing result from the graph because now you're premputing you're getting all this evidence graph construct in advance.
18:34 The writing's cheaper, the reviewing's cheaper with a higher quality result. >> I wasn't sure about the whole graph thing initially, but after seeing the cost tank almost in half, like it's not even a question, it's making this program work. >> And this is something we did for the Trey Stone project, but you can do this for pretty much any project where you have data, you have relationships, you have evidence, you have things which you're trying to to compare because agents are not that good at doing the data analysis
19:08 at doing all of the evaluation and the curation. So if you give them a help at the front end by creating a a custom graph layer with an in-memory database like relate suddenly your costs go down your quality goes up and you too can be a token billionaire like our friend Alex. Well I guess it's a now he's using it as a bookmark. I think that's a good use for it.
19:33 It's a really high quality bookmark. It's heavier than a credit card. >> It is. It is a nice surprise for um spending all those tokens on PR Bench. So, finance is happy. Lower drafting costs that they can reasonably fund. Alex is happy. He has his new bookmark. key is now has a great case study with the connections doing neighborhood retrievable of facts so that it's a better brief focusing the review so the reviewers get the same compact brief ma gets exactly the evidence she needs and then here's Alex on stage at um
20:09 AI engineer somewhere in the world better PR's lower drafting cost with a graph plus open-source trace to builds, which also gives you amazing comics. Okay, so big round of applause for our friend Alex, who's now earned the AI engineer hoodie. >> I think that's definitely a wardrobe upgrade for him. >> Yeah, I don't think the pocket protector look was that great for him.
20:44 So, good for him. I think we'll see Swixs on stage with one of these hoodies soon. He he can't he can't not do it now that we shamed him into doing it at his own conference. The token billion program is amazing. A big kudos to the team for coming up with ways to incentivize and reward people who are on the bleeding edge of AI usage and learning how to make it really effective for the rest of us.
21:09 And um now you too can have better PRs. Impress your co-workers by having wellressearched effectively written and lowerc cost PRs >> with a comic >> with a comic. So for more information go to trace tune.com learn about trace tune. Give it a try. It works with cloud. It works with codecs. It works with all your favorite AI tools. And if you want to do these same sort of amazing results on your own project, try out Relate, our new Rustbased embedded open-source graph database, which brings context to your agents and
21:47 helps you build more efficient applications that just work better, are cheaper, and give you better results. So, thank you all for joining our AI engineer San Francisco talk. And if you're watching this talk this week at San Francisco, what should you do this weekend with your kids on Saturday? Cassandra, >> we are running an AI engineer kids day. So, anyone who's available, just sign your kids up.
22:14 We're teaching kids workshops. Kids will be learning how to do AI programming to make their own games. And I think we're also doing a No, it's just the gaming one. No, we'll do we'll do the robots, too. >> Okay. Yeah, we'll also be doing robots. So, kids will get to control these little soda can robots using AI. >> Yeah. So, a lot of fun. Physical AI gaming AI all in one place.
22:38 Um, sign up at AI engineer kids day and um, you know, bring your kids, bring your nieces, bring your nephews, and let's train the next generation of AI engineers. All right. Thanks so much and enjoy the rest of the conference.