I read every major CS paper of the last 100 years... Transcript, AI Summary & Key Points

00:00 The year was 1936. Alan Turing asked a simple question, "Can machines think?" Actually, no, that's not right. What he really asked was something way more boring, "Can every mathematical problem be solved by an algorithm?" Surprisingly, he proved the answer is no, but in the process, he accidentally invented the computer. Then 12 years later in 1948, another legend shows up named Claude Shannon, and he reduced all human communication down to ones and zeros, casually inventing the bit like it was no big deal.

00:27 One thing led to another and now in 2026, we have 18-year-olds in hoodies typing import torch into Python files and cashing billion-dollar checks from venture capitalist boomers. But reaching this point has been underpinned by a century-long chain reaction of computer science papers written mostly by dead people much smarter than us. In today's video, we'll look at 10 of the most important scientific papers in the history of computer science and how they changed the world for better or worse.

00:54 Our story begins nearly a century ago when mathematician David Hilbert asked the field's biggest flexive of a question, "Is there a universal algorithm that can decide whether any mathematical statement is true?" Or in other words, "Can we automate math itself?" He called this the Entscheidungsproblem, which is German for decision problem. By 1936, Alan Turing comes around and gives a brutal answer to this question, "No."

01:15 But in order to prove it, he wrote this paper on computable numbers that had to define what an algorithm even is. And so, he imagined a hypothetical machine with an infinite tape, a read-write head, and a tiny table of rules. This Turing machine is the abstract blueprint for every computing device you've ever owned. Once created, he asked it to solve the halting problem, "Can you write a program that looks at any other program and tells you if it'll finish running or loop forever?"

01:43 Turing proved that it's impossible for a program like this to exist. It simply leads to a logical contradiction, which means math has problems that no algorithm can solve. That's annoying, but 12 years later, a guy named Claude Shannon would ask his own annoying question, what is information as a thing you can measure? In his paper, a mathematical theory of communication, he rips out the meaning from normal words entirely.

02:05 I love you and the cat is on fire carried the same information if they're equally surprising. And he measures that surprise in a unit called the bit. He proved that all information could ultimately be boiled down to a stream of ones and zeros. But here's the crazy part, to estimate how much information was needed to transmit a message, he borrowed work from thermodynamics nobody understands, entropy.

02:26 And to estimate entropy of English, Shannon made people guess the next letter in a sentence. When a letter is easy to guess, it has low entropy. When a letter is hard to guess, it has high entropy. But wait a minute, having humans guess the next token is exactly what AI does today, just on a much bigger scale. And Shannon wasn't trying to build artificial intelligence, but he gave us the math for uncertainty, prediction, and compression, and accidentally wrote the spiritual ancestor to the loss function.

02:54 And that's exactly why Anthropic named their AI model Claude. Then, 10 years later at Cornell, a psychologist, not a computer scientist, builds the first machine that actually learns. He gets inspired by the way neurons work in the brain, so he designs a thing called a perceptron that takes inputs, weighs them, and then adjusts those weights when it's wrong until it can classify patterns on its own.

03:14 It's the building block for modern neural networks and the hype is immediate and unhinged. The Navy funds it and the New York Times reports that the computer will soon be conscious, but 11 years later, the hype would die out completely thanks to two haters at MIT who published another paper with a completely different vibe. With basic math, they proved that a single-layer perceptron can't even learn exclusive or, which is just trivial logic that means this or that, but not both.

03:40 This paper, or technically a book, was essentially a death certificate for AI at the time. Funding evaporated and deep neural networks entered their first AI winter. But there is a twist buried in the fine print. They actually figured out that stacking layers of perceptrons fixes everything. The only problem is that back then, nobody knew how to train a stack of perceptrons.

03:59 It would take another 17 years to figure it out, but first we need to talk about times, clocks, and the ordering of events in a distributed system by Leslie Lamport, because neural networks are useless unless you can run them on a massive scale. This paper realized that separate computers with no shared clock can't really have a universal now time, and that's a big problem when you have multiple computers in a distributed system trying to do things in order.

04:23 Well, he figured out a way to fix this with the happen before relation. You stop trusting the wall clock time and order events by causality instead. If A could have caused B, A comes first. From that, he builds logical clocks, which allow an unlimited number of machines to stay in agreement without ever looking at a real clock. Eventually, this paper would become the bedrock for every database, blockchain, and every massive AI training run, because you need thousands of GPUs that constantly stay in sync and agree on

04:50 state without dissolving into chaos. That was a game-changer, but 17 years after neural networks were left for dead, the three researchers, including the godfather Geoffrey Hinton, answered the question that everyone gave up on: How do you train a stack of layers? But before we answer that, we need to quickly talk about Coder, who was cool enough to sponsor this 10-minute video on esoteric computer science papers.

05:12 They provide self-hosted development environments that let you work with multiple agents in parallel and with enterprise-level security. And they just launched Coder Agents, a chat interface and API for delegating coding jobs to agents running on your own infrastructure. It's the only architecture that lets organizations self-host both the agent workflow and the development environments where the code is actually executed.

05:34 This gives teams greater control over source code access, agent execution, governance, and security boundaries. It's also model agnostic, so you can connect any LLM you want and switch between them with just a config change. The Coder agents are designed for teams in regulated industries who need to self-host their AI workflows with complete control.

05:53 They're already used by dozens of financial institutions and government organizations, and you can check it out at the link below. Now, back to the question, how do you train a stack of layers? The answer is backpropagation. Run your data forward, measure how wrong the output is, and then push that error backward through every layer using the chain rule from calculus to nudge each weight in the direction that's a little less wrong.

06:14 Do that a few million times and the network teaches itself. The crazy discovery though is that the middle hidden layers started inventing their own features, edges, shapes, and concepts that nobody programmed in. That exclusive or problem that was impossible 17 years ago it just became trivial. Backpropagation is still essential to neural networks today, but back then they sucked because we didn't have enough data or compute.

06:37 Well, that was about to change in 1998 with the rise of the internet and this famous paper from Larry and Sergey about the anatomy of a large-scale web search engine. The paper describes the PageRank algorithm where instead of ranking a web page by how often a word appears, it treats every link as a vote. And each vote is weighted by how trustworthy the voter is.

06:58 They built a prototype in their dorm room which eventually became a company called Google that you may have heard of. Most importantly though, this algorithm helped assemble the largest structured pile of human text ever created. And that massive pile of text would eventually become the training data or feedstock for future AI models. We finally see this in action in 2012 with the legendary ImageNet paper created by a dream team of Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton.

07:23 Remember when I said backpropagation needs data and compute? Well, finally the stars aligned. The dataset is called ImageNet and it's a monster dataset of millions of hand-labeled photos while the compute is a couple of Nvidia consumer-grade gaming GPUs. A grad student named Alex wires up a deep convolutional neural network, names it AlexNet and trains it in his bedroom.

07:44 Then he walks it into the annual ImageNet contest and humiliates everyone. This is a contest where AI models try to classify objects in an image like hot dog or not hot dog. And while everyone was fighting over a fraction of a percent, AlexNet walked in and dropped the error rate by 10 points in a single year. And this freaked everyone out because it was suddenly clear that deep learning actually works.

08:06 It just needs more data, more compute, and the right architecture. Luckily, we would get that architecture a few years later thanks to Ashish Vaswani and Google in the paper "Attention is All You Need". Around this time, large language models had a huge problem. They would start a sentence and by the end they would forget what they were even talking about.

08:22 That's because they would read and predict tokens sequentially one after the other. This paper fixed that by introducing a new architecture called the transformer that throws out sequential reading entirely. Instead, it lets every word look at every other word at once and decide what's relevant. Not only does this make large language models feel more intelligent, but transformers also scale better as well.

08:44 Google made the big mistake of giving this architecture away for free and now every AI lab uses it and that's where you get the T in chat GPT. Speaking of which, that brings us to a paper released by OpenAI in 2020, "Language Models Are Few-Shot Learners". Basically, OpenAI takes the transformer and then asks the dumbest question possible. What if we just make it enormous, not two times bigger, but scale it to 175 billion parameters and feed it the entire internet as a data set.

09:11 They made a crazy bet that intelligence isn't some secret algorithm we're missing, but rather it simply emerges once you cross a threshold of scale. The end result was GPT-3, the model that ignited the current AI bubble that we're living through right now. What's crazy is that all of a sudden this model could translate, summarize, and write code without ever being specifically told how to do these things.

09:32 At such a large scale, it learned how to generalize these things on the fly. Two years later, this paper would evolve into ChatGPT, which today is now a trillion-dollar product. But, when you think about it, what is ChatGPT even doing? Well, it's just predicting the next word or token, just like Claude Shannon was doing in 1948. So, here's the TLDR for the last 100 years.

09:51 Alan Turing defined the machine, Claude Shannon gave it currency, Rosenblatt gave it a neuron, Geoffrey Hinton taught it how to learn, Google gave it data and an architecture, and OpenAI just turned the dial to the maximum. This has been the history of artificial intelligence in 10 scientific papers. Thanks for watching, and I will see you in the next one.