Is Jenkins Dead? No, And That's Much Worse Transcript, AI Summary & Key Points

00:00 Gather around. Get comfortable, kids. Tonight, I'm going to read you a bedtime story. This is the tale of Jenkins, the butler who never once said no. Now, this is not one of those stories where the hero loses. Jenkins won. Jenkins won everything. For the better part of a decade, if you wrote software for a living, Jenkins stood between your keyboard and your customers.

00:26 and it built your code and it tested it and it shipped it every single night without ever once being thanked. And here's the thing, almost nobody chooses Jenkins anymore. Ask around your office and you will find people who would rip it out tomorrow morning. They can't. 20 odd years after a young engineer at Sun Microsystems wrote the first version of it, it is still there.

00:53 still running the thing that actually ships your product. And everybody has quietly agreed not to touch it. So tuck in because tonight's nightmare isn't that our hero dies at the end. It's that he doesn't. He's still running tonight and you cannot switch him off. There is one part of this story that I am not going to cover. What should happen to all the keys the butler needs?

01:20 Jenkins has to reach source control, cloud accounts, deployment targets, and production systems. Putting those credentials directly into jobs might be convenient, but it also turns the machine running them into a cupboard full of keys that nobody wants to open. That is where the sponsor of this video comes in. In physical, an open source platform for managing secrets across your stack.

01:44 in physical gives you one place to store, sync, and rotate API keys, database credentials, and certificates, then inject them into applications and CI pipelines without putting them in source code or spreading them across ENV files. It can also scan commands for leak secrets before they're pushed and handle certificate and key management from the same platform.

02:09 You can self-host in physical or use their cloud. Check it out at infysical.com or through the link in the description. Big thanks to Infysical for sponsoring this video. And now, let's go back to a young engineer at Sun Micro Systemystems who kept breaking things and decided to build himself a butler. Once upon a time in the summer of 2004, there was a young engineer at Sun Micros Systemystems and his name was Koske Kawaguchi.

02:38 And Koske kept breaking the build. His own machine was always fine, of course, but somebody else would update their workspace and suddenly nothing compiled and the whole thing went to and the phone would ring. And on the other end would be somebody who had just lost their entire morning being very very polite about it. Hey, I think you touched this the last time.

03:05 Can you look into it? And it usually was indeed him. Now Koska was a lazy man. He will happily tell you so himself. And a lazy man in that position does not resolve to be more careful. No. A lazy man thinks. What if something else checked? Something tireless. Something that takes the work the moment you hand it over, builds the whole thing from scratch, and tells you the truth about it before anybody else finds out.

03:35 What he wanted. when you get down to it was a butler. So he built one. He named it Hudson after a butler from a television series because that is exactly what it was. A servant. You ring and it comes and it does the thing and it does not complain about it. And this is the detail that matters. Everything that happens over the next 20 something years happens because of this one thing.

04:02 Cost did not build something that runs builds. He built something that runs any command you give it. Whenever you tell it to builds first, but any command realistically in most places that job had belonged to a shell script on a machine under somebody's desk. It went off at 2 in the morning. Exactly one person understood it. And when that person went to holiday, the whole team held its breath.

04:25 Hudson was released in February 2005 and Koska's colleagues loved it immediately. partly because it was good, mostly because they were all exactly as lazy as he was. What Kosk built there has a name, continuous integration, and he did not invent it. The idea was already a decade old and cruise control had been doing it in the open since 2001. What Hudson did was put it within reach of everybody and it managed that by saying yes to everybody, yes to you.

04:59 Cruise control wanted XML configuration files and Hudson gave you a web page and a button. And yes to everyone else because Costco built it so that anybody could bolt a new ability onto it without asking his permission. That's the plug-in system and it grew into one of the largest plug-in ecosystems in open source. 1,800 of them written by thousands of people.

05:24 Hudson did not stay at Sun Micros Systemystems for very long. It was free. It was open source. And it ran on more or less anything. So it got out of sun the way those things always do. One engineer at a time. Somebody used it at work, liked it, took it with them to the next job and set it up there in an afternoon. And that really was all it took. One afternoon.

05:51 You gave it a machine. You told it where your code lived. You clicked a few things. And by the end of the day, something was building your project every time anybody touched it. If you have ever done that for the first time, you remember how good it felt. Meanwhile, Costco kept shipping. Not every year as it was custom back then, not every quarter, a new release every week, a habit the project still has to this day.

06:19 Now, here is the thing about Hudson. On its own, it did not do very much at all. It scheduled work. It run things and it showed you the results on a web page and that's about it. Everything it could actually do. Talk to your version control, run your build tool, publish your test results, send the email when it all fell over, every single one of those was a plug-in.

06:41 So the plugins came, dozens at first and then hundreds written by people cost never met for tools he had never used. Whatever strange internal thing your company depended on, sooner or later somebody wrote a plug-in for it and then Hudson could talk to that as well. Which means the thing the entire industry was coming to rely on was mostly not written by the people who made it.

07:07 It was a large pile of other people's work held together by a small program whose real talent was holding things. In May of 2008 at Javan, Sun gave Hudson a Duke's choice award. Sun handing out the prize for a program that one of its own engineers had written because he kept breaking the build. And by the end of that decade, it simply won. Cruise control faded.

07:36 The others faded as well. If you were doing continuous integration at all, you were almost certainly doing it with Katsson. And notice how it won because this matters later. Almost nobody picked Hudson for its architecture. People picked it because it cost nothing because you could have it running before lunch and because somebody had already written a plug-in for whatever strange thing you needed.

07:58 That is how infrastructure actually gets adopted. Not by being the best design, but by being the pot of least resistance, which is wonderful, right up until the day you would like to change your mind. In January 2010, Oracle bought Sun Micros Systemystems and Hudson along with everything else that Sun owned went with it. Kosk left a few months later and went to work for a young company called Cloudbase.

08:23 Now the fight when it came did not start over money. He did not even start over the code. He started over where the project lived. Hudson's home was sold infrastructure. It was slow. It was cracking. and the developers wanted to move everything to GitHub where the rest of the world already was. And then in the autumn of 2010, a migration went wrong and the Hudson developers found themselves locked out of their own source code.

08:53 So they said, "Right, screw it. We're moving." And Oracle said, "No." And when they pushed, Oracle reached for the one thing it could actually hold on to, not the code. The code was open source. Anybody could take a copy. the name. On the 28th of October, while the developers were busy moving the project to GitHub, Oracle filed a trademark application for the word Hudson.

09:18 Nobody ever went to court. There was no lawsuit, no judge. There were meetings. And in those meetings, Oracle would not let go of the name. And the developers worked out that fighting Oracle over a single word was not something any of them could afford. So, they did the other thing. They held a vote. Keep the name or walk away and call the project something else.

09:40 214 people voted to walk. 14 wanted to stay. And then they had to choose a new name. They considered Alfred after Batman's butler and let it go because something else already had it. And in the end they settled on the name of a different battler altogether. Jenkins because that was the joke. Oracle had taken the name. So they went and got another one exactly like it.

10:10 Same code, same people, same plugins, same project. All that changed was the word on the front. Everything else changed later until it was a completely different project and still somehow the same. Oracle kept Hudson and in May of 2011, it handed the whole thing to the Eclipse Foundation. the code, the trademark, the domain name, all of it. Hudson's website was switched off in January of 2020, and the project was archived a year after that.

10:40 By then, almost nobody had used it in years. Now, it's tempting to make Oracle the villain of this story, and I don't think that's quite right. Oracle had just paid $7.4 billion for Sun, and securing what you bought is exactly what you do. The mistake was thinking that Hudson was one of the things it had bought. Sun did not own Hudson. Sun owned a word attached to a project built by thousands of people who worked somewhere else entirely.

11:13 Oracle secured the only part it could legally hold and lost the only part that was worth anything. And it wasn't a one-off. In the 18 months around that acquisition, my SQL became Maria DB. Open Solaris became Lumos, Open Office became Libra Office, and Hudson became Jenkins. Four for four. The plugins never stopped coming. A few hundred became a,000.

11:38 A thousand became 1,800, which is roughly where it stands today. And remember what Kosk actually built. Not something that runs builds, something that runs any command you give it whenever you tell it to. So people gave it everything, not just building and testing, deployments. database migrations, nightly backups, certificate renewals, the Friday release, the report the finals team needs on the first of every month, the little script that used to live on somebody's laptop, which meant that quietly over about 10 years,

12:13 Jenkins became the most privileged machine your company owned. It could reach production. It held the cloud credentials and the signing keys and the keys to every server you had. If you wanted to own a company, you did not attack the company. You attacked its Jenkins. All of those plugins run in a single process, sharing everything, which meant that upgrading one of them could break another one written by somebody else for an entirely different purpose.

12:39 So people stopped upgrading and you have met that one, right? the Jenkins nobody touches the one four years behind where somebody had effectively taped a note next to the upgrade button that says please don't. Some of those plugins had been abandoned altogether. The person who brought the one your release depends on stopped maintaining it in 2016, moved on and never came back.

13:05 It still works but nobody's watching it. In January of 2024, a vulnerability called CV and then something something let a stranger with no account at all start reading files of your Jenkins, not your application, your Jenkins, the machine holding all the keys and by then most of them could not be rebuilt anyway. 20 years of settings entered by clicking save to a disk.

13:31 If your junk is burned down tonight, you would not restore it from source control. you would try to remember. Now, here is the part I want you to sit with. Every single one of those plugins was the right call on the day somebody installed it. Nobody was lazy. Nobody was stupid. Somebody needed a thing to happen on a schedule. There was already a machine that did things on a schedule and handing it over took about 4 minutes.

13:59 That is what lockin actually is. It is never a decision. Nobody signs anything. It is an accumulation. 2,000 small correct yeses and then one morning you work out that leaving would cost more than anybody that your company is willing to spend. That is how you get politely for minutes at a time. Now the people who run Jenkis knew all of this. Every single thing I have just described they knew and they tried to fix it.

14:29 Start with the falling over. give Jenkis enough work and it runs out of memory and it stops answering and it somebody has to go and restart it and when it comes back every build that was running is simply gone. So they set out to fix that properly. A build ought to survive its own server restarting which means Jenkins has to be able to freeze a running script halfway through and pick it up again later which means it cannot simply run your script.

14:54 It has to rewrite it into something that can be paused at every single step. And that is why the language inside the Jenkeis file looks like Groovy but is not really groovy. Write an ordinary loop the ordinary way and it breaks. And the only way to find out is to commit it, push it and wait four minutes for a stack trace to explain in Java why you're an idiot.

15:20 And that is not carelessness. That is a fix working exactly as designed for a problem that should never have needed fixing. And they kept going. In 2016, Jenis 2.0 made pipelines a first class thing and put the Jenkeis file in your repository where it belonged. That was right. That worked. The same year they built Blue Ocean, a modern, clean, genuinely lovely new interface.

15:43 It went into maintenance in 2023, never received another feature, and is now being retired altogether. So the screen you look at today is still more or less the screen from 2009. In 2018 came configuration is code so you could finally describe the whole Jenkins in a file instead of clicking. 10 years after everybody needed it and it was of course a plug-in and that same year they sent an air out into the new world.

16:17 Jenkins X. Jenkins for Kubernetes, cloud native, the future of everything. To survive out there, it had to take Jenkins out of Jenkins. It runs on Tecton now. And today, it has one maintainer. Most of its releases are filed by a robot. It has quietly dropped Jenkins from its name as well. But back at Jenkins itself, none of those repairs could ever be rebuilt.

16:41 That was the whole problem. 1,800 plugins are written against the insides of Jenkins. So the inside of Jenkins cannot be changed. Everything had to go on top, never underneath, which left the repair that mattered most permanently out of reach. There can only ever be one controller. It can work out to as many machines as you like. That part works beautifully, but the controller itself cannot be copied.

17:10 You cannot run two of those. So large companies ended up running dozens of separate Jenkins installations that have never heard of one another. And nobody could ever turn Jenkins into a service you simply sign up for. And while all of that was going on, the world quietly stopped needing Jenkins at all. In 2019, GitHub put continuous integration directly into the place your code already lived.

17:33 And GitHub was not alone. Travis, CircleCI, Buildk Kite, GitLab, some of them at it for years already. No server to run, no plugins to install, nobody to restart it at 2 in the morning. And notice what did not happen in any of that. Pipelines, Blue Ocean configuration is code, Jenis X. Jenkins refused none of them. It accepted every single one gratefully.

18:01 The problem was never that it would not change. The problem is that it could only ever be added to, never replaced. And after 20 odd years of only ever adding, what you have is not something you can fix. The only repair left is to leave, which is the one thing almost nobody can afford to do. So here is where Jenkins is tonight. It did not die. It is not a museum piece.

18:26 It is running right now inside banks, inside telos, behind VPNs, in regulated industries, on machines you cannot reach from the internet doing exactly what it has always done. Jenis is not chosen anymore. Jenkins is inherited. And look at what losing actually looks like here. Around a third of companies run two different CI tools. Some run three, some more.

18:52 They did not replace Jenkins. They bought the new thing and kept Jenkins as well because when a company moves to GitHub actions, it moves the builds. The builds are the easy part. They are written down. Somebody understands them and they already live in the repository. Everything else stays. The nightly backup stays, the certificate renewal stays, the database migration stays, the report the finance team needs on the first of every month stays.

19:17 And the little script that used to live on somebody's laptop and moved to Jenkins and has run every night since 2014, that stays as well. So what is left at a very large number of companies is a machine that runs a pile of jobs nobody has ever made a list of. fired every night at two in the morning, understood by exactly one person, touched by nobody, which is of course exactly precisely the thing that Costco built it to replace.

19:49 And it will not stop. It cannot. It has never once refused anything anybody has asked of it. And it is not going to start now. It will run tonight and tomorrow night and the night after that. One person where you work knows what all of it is for. And that person is going to leave this year or next year or the year after that. And when they do, nobody's going to switch it off because switching it off would mean first understanding what it does.

20:19 Good night then. Sleep well. And in about 4 hours at 2:00 in the morning, your phone is going to ring and it will be somebody who has already been awake for an hour and who is not in the mood. Jenkins is down again. You brought this thing in here. Deal with it.