Test in three layers — unit, integration and end-to-end — stacked as a testing pyramid, and when AI writes the code, make sure expected behavior comes from a spec, API contract or acceptance criteria rather than from the same agent that wrote both the code and its tests; use a separate review step or agent to verify important changes.
Searchable transcript of How to Test Your Code (Even When AI Writes It) — ByteByteGo (04:12). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by ByteByteGo. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 A program can pass every unit test and the product can still be broken. Maybe the database cost is wrong. Maybe two services disagree on an API. Or maybe everything works individually, but the checkout flow still fails for the user. That's because different tests catch different kinds of bugs. So let's look at the three layers of software testing and what each one is good for.
00:19 Let's start with unit tests. In this example, we're building an online store. We have a function that calculates the final price of an order, including discounts, taxes, and shipping. A unit test might check whether a 10% discount is calculated correctly, or whether an order over $50 gets free shipping. We're testing one small piece of logic on its own with no database and no other services.
00:45 That's why unit tests tend to be fast, and we can run a lot of them every time the code changes. But testing pieces in isolation isn't enough. Our checkout service for example needs to save the order to a database. It also needs to call the payment service. This is where integration tests come in. An integration test checks whether multiple parts of the system actually work together.
01:07 We might call the checkout API and verify that it writes the right data to the database. Or we might verify that checkout sends the payment service the fields it expects. The check out logic could pass every unit test and still fail here because the SQL is wrong or because the two services disagree on an API format. Today's video is sponsored by Code Rabbit, the most installed AI app on GitHub and GitLab.
01:31 These days, one AI generated pull request can touch a dozen unrelated things and reviewing it means bouncing between tabs. Code Rabbit's new review UI fixes that. It splits the diffs into small related chunks. You can take one at a time. Click any function and its definition pops up right there. Got a question about the change? Just ask and you get the answer without leaving the page and it flags the critical stuff first so you cut review time and bugs in half.
01:56 Start a 14-day trial. Link in the description. Now compare that with an end-to-end test. Instead of calling the checkout API directly, we open the website like a customer would. We add something to the cart, go through checkout, and make sure the confirmation page appears. Now we are testing the whole path. the front end, back end, database, and the services behind them.
02:19 That's the key difference. An end-to-end test checks whether the complete user workflow works. The broader the test becomes, the more the real system it exercises. But it also becomes more difficult to set up, slower to run, and harder to debug. If a unit test fail, there's usually a small amount of code to investigate. If an end toend test fails, the problem could be almost anywhere along the path.
02:45 This is where the idea of a testing pyramid comes from. We generally want lots of small fast tests near the bottom, fewer integration tests above them and a smaller set of end-to-end tests covering the most important workflows. The idea is to test something at the lowest layer that can give us the confidence we need. So what changes when AI writes the code?
03:05 Mostly the speed and the volume of change. An AI agent can make a large change, generate tests for it, run those tests, fix the failures, and try again. But there's one failure mode we need to watch for. If the agent writes the implementation, then writes tests based on the same implementation, it can carry the same wrong assumption into both. The code is wrong, the tests agree with the code, everything passes.
03:33 The expected behavior has to come from somewhere else. A specification, an API contract, or a set of acceptance criteria. For important changes, we can also have a separate review step or even a separate agent to verify the result. Don't let the agent that build it gray its own work. The same basic testing pyramid still applies. Unit tests give us fast feedback.
03:55 Integration tests catch problems at the boundary between systems. End to end tests tell us whether the whole workflow actually works. AI can make it much cheaper to build and use these feedback loops, but it doesn't define what correct means. We still have to do that.