CPU timestamp measurements and wall-clock time serve different purposes: wall-clock time requires a calibrated frequency and an offset from an external time source, while benchmarking can use differences between timestamp-counter readings. Accurate measurements also require accounting for instruction reordering and serialization.
🔒 11 more in the full analysis
Searchable transcript of Why CPU Time ≠ Wall Clock Time! - Computerphile — Computerphile (19:41). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by Computerphile. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 So today I'd like to talk about time. What time is it and how does a computer know that? My day job I'm high performance computing and high performance computing when you're trying to like measure is something faster than something else you want to know what time it is so that you can say start a clock do some work stop the clock and that's fine. If it's tens of seconds you can just use regular time stuff but if it's like hundreds of nanoseconds how can how can you be sure you're getting the time right?
00:27 But in general the same process is used to find out what time it is like wall clock time like you know is it 3:30 in the afternoon right the same sort of principles applied so I want to start with that so first of all like my first computer the ZX Spectrum and then later the BBC Micro they both kept time well they didn't really keep time they didn't know what time it was outside in the real world they just knew how much time had passed because 50 times a second there was an interrupt we've talked about those before an
00:57 interrupt that says, "Hey, the video chip is about to start drawing the screen again, you know, in in time with the picture that was going out to the TV." That's a convenient time to just add one to a counter so that you know that one more 50th of a second has gone past. And so then in your basic program when you're saying print the time, it's just looking at whatever that counter was and dividing it by 50 and that's how many seconds have passed since the computer turned on.
01:19 And then, you know, they got fancy with batterybacked things that had a little clock, an actual clock that knew what the actual time was. and then when it started up it just read that clock time and then it started adding to it 50 times a second or whatever. So great stuff great stuff. Um modern computers need to know the time at a slightly better granularity than 50 times a second.
01:39 A 50th of a second is you know an eternity uh for the kind of things that we do these days. And so most CPUs since the early 2000s have a counter in them that ticks up that's automatically ticking up at a much higher rate than our 50 times a second. So on the computer that I'm recording this one over here, the counter goes up at 3 GHz, which is pretty fast, but that's kind of also the speed of the computer.
02:06 So you could think that maybe I could measure the speed of like individual instructions with that. And maybe we'll get to that at the end. But I want to talk about first of all how we take that number and we make it a useful number for us. Right at the beginning of the day, you turn your computer on and that counter starts at zero. So we need to first of all we we how we now know how long it is uh at any time.
02:27 We can ask how long it is since the computer turned on. And so somewhere around bootup time, you could imagine the operating system goes off to the network or goes off to the batterybacked clock that lives inside your BIOS or whatever and says, "Hey, what time do you think it is? Let me just remember that." Now, what I've got is a a sort of starting point, a base offset, and then I can add however many nanconds have passed.
02:50 But this counter doesn't measure nonds. It measures, as I say, my particular computer's case 3 GHz. How do I know what the frequency of my computer is when I don't have anything to measure it by until I've got it, you know, running like? So, so there are some tricks sometimes, you know, older machines would use an interrupt. So, back back into the old uh old days, they would sort of set an interrupt to say, "Call me back in 200 milliseconds from some other like part of the hardware."
03:19 And they would just literally measure it and say, "How many how many million ticks of this counter have gone past?" But modern ones actually tell you what they think their crystal is set to. So you can actually ask the CPU what what is your how fast is your clock going up and it'll say 3 GHz. Now the problem with the computer telling you that it goes up by 3 GHz.
03:39 That is what it thought it was happening when it was set at the factory. Now I you probably are well aware that like you know your Casio watch from 1980 although it was you know supposed to be ticking however many you know times that it was being counted to get you know how many seconds per day it would be off by you know a second or two a day and it doesn't take very much for your clock to be off by before suddenly you're losing time by the end of the day.
04:05 So something else is going on. So let's talk about what happens right. So concretely uh and I'm going to talk about the the x86 here because it's the bit I know but the arm machines have a similar instruction and they have a similar counter and they know about these things. So um the the x86 has an instruction called rdtsc. You know how um instructions they love to like squish letters together and make it completely unreadable rather than you know the nice lang highle languages stuff here.
04:33 This stands for read timestamp counter. This gets the number of uh of of whatever that counter currently has. It puts in a couple of registers. We can do some maths with it, which is what we're going to do. So, what we want to try and work out is how many nanose is it been since some epoch. And in Unix world, which is where I live, uh the epoch, that is the sort of start time that everyone agrees on is the 1st of January 1970.
04:57 So, that's what we're going to measure. We we just want to know how many nanconds have passed since that moment in time. And that's how we measure all times against each other. That's a lot of nanconds, but that's fine. We've got 64 bits. So, we've got a lot of numbers we can we can wedge in there. So, what I could do, for example, I could say that the nancond time is equal to however many nanconds since 1970 was at boot that we'll get from either NTP or we get it from uh the the the clock inside the computer itself.
05:24 So, we're going to say NS at boot. And I'm going to slightly wave my hands about that again because there are some nice computer files that explain how NTP works. But then we need to add the time stamp count which I'm going to write as TSC the time stamp count from this instruction and then we sort of need to multiply it by some kind of thing that turns the frequency of the time stamp counter into nanoseconds.
05:48 So in my case in my computer I need to essentially divide by three or multiply by a third. So I'm going to have multiply by some kind of factor over here that will give me the current time in nanoseconds. Fantastic. How do we get that factor? So at the beginning what we're going to do is we're going to say that factor is one over whatever the chip says it is 3 GHz because 3 GHz 1 ghahz would be counting in nanoseconds.
06:13 It's kind of bonkers to think about these things. But I don't want to use floatingoint maths. That's that's a pain. I don't want to actually use a third here and neither does the computer. We know that integer u numbers are faster and it turns out that when you want to know what time it is, you often want to know what time it is and you don't want to take a lot of time to find out what time it is.
06:31 So you don't want to do lots of complicated maths, right? It's kind of an irony here. >> When you asked me what time it was, it was 12:00, but it's not anymore. >> It's now Yeah. 12 because it took me so long to work out it was 12:00. Exactly. Exactly. And you know again with the kind of um uh you can imagine how often all the programs on your computer are saying what time it is um either to display it or for the own internal measurements about things.
06:52 So in especially in my world again I want to know what time it is very very often nanoseconds apart and kind of say hey how long did that take? So, we want this to be fast, right? What we're actually going to do, and this sort of harks back to the divide conversation we had, is that we're going to pre-calculate some kind of multiplier that's an integer multiplier, and then we're going to have a divisor.
07:12 And now, if you remember, if we were to do this in in decimal, what I might say is, for example, is why don't we multiply by 333 because multiplying is relatively cheap. And why don't we divide by 1,000? So that effectively we're saying that 1/3 is approximately 333 over 1,000. And dividing by a th00and for humans is easy. We just drop the last three digits.
07:37 So I multiply by 33, drop the last three digits, and then I've kind of done a divide by three pretty cheaply. Now computers work in base two. So there's some fraction in base two, and then there's a shift of by some power of two down to do the divide. So it might be I and I should have checked this before I we we started recording but um what is a third of 256 uh sort of 80ish maybe something like that uh >> 85 or something like that is it?
08:04 >> Yeah. So so you could imagine we could multiply by 85 and then divide by 256 which is to say shift down by 8. So that would give us you know and again a multiply is not the cheapest thing we can do but it's one of the cheaper integer things that we can do. So what we're actually going to do then is our NS at time is actually equal to the boot time plus the time stamp count times some factor and then all of this down by some shift value here.
08:33 That less than and less than there is is like a C term to say shift down some number of digits. That is throw away the bottom n bits. So that's cool. Um the problem with this particular equation and it's pretty simple right we've got a couple of things here is that um we knew that the the nanoseconds time at boot but to your point about you know the clocks moved on we kind of there's there's always going to be some kind of skew between those two numbers right and so what we really need to do is like well it was
09:04 actually thousands and thousands of cycles in that we got to the point in the kernel booting and all the you know the black screen with all the text coming up where it went Oh, uh, I think we should try and set up the clock now. And so obviously some amount of time has passed. So obviously there's some offset from boot when that we first got the time.
09:21 And this actually generalizes quite nicely because of two things are about to happen. One is once we've started up, we will probably discover the time wasn't quite right. Maybe we will talk to the internet. You know, maybe NTP comes along and says, you know what, actually uh your clock wasn't right. You need to start adjusting it. And so we always want to have a way of offsetting the time based off of two things.
09:43 The number of cycles at the point we last got a really accurate time and then what that accurate time was. So our final thing becomes NS is equal to and it's going to be some kind of time offset. That's like the last time we knew categorically what the real time was on the outside world plus and now we're going to do the time step count minus whatever the time stamp count was when we got this time.
10:07 So we're going to say some offset here. And now that is multiplied by the factor and shifted down by the shift. And now we can adjust every time that we get a new accurate time. We can replace the accurate time and the offset. And now we're in a good space. But one thing I'd like to talk about is this factor here because I just told you that like it might be off.
10:31 And so one of the things we might have to do is fiddle with this factor if for example I actually measured it this morning when I was here. So like my computer is 3 GHz but it's actually currently 299.99 MHz which is not 3 GHz. You may know it's very very close to it but if I left it like that I'd lose like 2 seconds a day. The other thing is that of course a lot of modern chips will speed up and slow down for power saving reasons and things won't weren't they?
10:59 So I mean >> that is >> are we putting that to one side for this anyway? >> No. So that's an excellent point. Absolutely brilliant point because it used to be in the very early incarnations of this RDTSC instruction. It did measure the clock frequency of the sort of prevailing core that you were on, which as you say goes up and down because CPUs go to sleep.
11:18 CPUs sometimes burst up, you know, like this is a 3 GHz machine, but it can burst up to 5 GHz I think at sometimes, you know, if there's nothing else going on and there's one core that needs to do. And so it used to be a real real problem and then fairly soon on um because it was so useful to have a stable number effectively there is a base clock that all of the cores use that doesn't change even if they all go to sleep or even if they uh burst up to 5 GHz or if it or whatever.
11:47 So that that makes life a lot easier. But what you can't guarantee is that that clock is actually genuinely stable because you know you heat stuff up and it starts to run faster or slower and all of that. So this waivers intraday. So it may be that it's just slightly wrong. Like when my machine I checked it this morning, it may be like just 299.99 is right.
12:05 But also it might be that it's been idle all night and it's cooled down and now it's slightly lower. So this the kernel certainly Linux anyway is forever going off to the internet and asking through NTP and other things what time do you think it is? and then adjusting its idea about how fast that crystal is running relative and so this factor will be slightly tweaked and knocked around just a tiny tiny bit you know similarly if it discovers the time is completely wrong it you could choose to update immediately jump
12:37 these offsets and times just to say oh no actually I was out by an hour or whatever right I'm not talking about like daylight savings time here because you know that that's usually an offset that's applied after this but NTP tends to try and just sort of waft things in the direction of correct so there isn't a sudden discontinuity in the time otherwise you know you you look at the date of your time and you're like wait a second that that time isn't hasn't even happened yet or it's like that's back in the past I just
13:01 made this what's going on so there's a lot of sub subtle things about this kind of stuff um uh so there's a couple of other things that are really interesting to me about this first thing is that as I said this is a calculation that I'd like to be able to do very very very quickly all of the time because I forever ask the computer what time is it on Linux systems you usually make a system call which is an expensive operation.
13:24 It's kind of like an interrupt. It kind of stops the whole thread of what's going on on in your process. The CPU changes into a privileged mode. We've done things on this before now. And it's kind of a heavyweight thing to go to an operating system routine that now has privileged access to all of these special values that you could otherwise not change.
13:42 Otherwise, you could imagine you can just change it and muck up the clock in unusual ways, right? That is what you would typically do. But it's so expensive that the Linux folks have come up with a clever trick where they actually map some code into your program and they map these offsets there. This the shift, the factor, the time and the offset. These four values get mapped into a read only area of memory that even an unprivileged program is allowed to read from which is super cool because now I can do this all of in
14:11 user space provided I can get this time stamp count and usually you can. And so you'll see that the libraries that say give you what the time is will just say get me the time stamp count and then do this very simple operation to get the actual wall clock time and that takes 35 nanos round about that amount of time. So it's not free free but it's pretty cheap.
14:31 Um, there's a whole other computer file video I think hiding in here as to how you can do that because while you're reading time, offset, factor, and shift. It could be that the colonel is changing them. There's some clever trick that it does to make sure that's as cheap as possible, but maybe I'll leave that as a for another time teaser. And then, so there's one another thing I'd like to just talk go.
14:56 So when I'm doing measuring of time, I don't care what wall clock time it is because I only care about the difference between two times. So I only care that like I start a clock and it has the number 1 2 3 4. And I don't care that that's 3:00 in the afternoon or 5:00 or 3 in the morning, right? It's just 1 2 3 4. I do some work and I get the clock again, it's now 1 2 3 8.
15:16 And I'm like, oh, four units of time passed. And that's the thing that I care about when I'm benchmarking, when I'm profiling, when I'm trying to work out what where my all my time's going. So effectively that would cancel out this offset this time here but not the frequency and shift if I actually want to turn it into nanoseconds. So typically what you'll do is something like read time stamp count some code here that I care about.
15:37 I care about measuring and then read time stamp count again and then I can subtract you know into say A and this is into B and then I do you know time elapsed B minus A and then this is now times that frequency. Whoops. and down by the shift. And frankly, if I'm doing lots of calculations, I don't even need to do this because if I just care about things getting faster or slower, I don't need to know if it's faster in nanconds or faster in clock cycles.
16:04 As long as they're something I can compare, that's the main thing. So, this is typically what um what we would do when we're like profiling code. And uh just a last thing on the this specific here because I think this sort of dovetales in with a whole bunch of other things we've talked about is um you you very much understand and and remember that computers don't execute the instructions in the sequence that I give them.
16:29 Although I've drawn it nicely down here, the pipeline's filled with instructions and we try and run the instructions that we can run because we know what their inputs are already and or their outputs uh um are are clear. Now register naming means the output's always clear. But like what does a read time stamp count instruction depend on? What does it have to wait for before it can start running?
16:52 If we see it ray in the future, it >> it doesn't need anything, does it? What does it need? >> It does not need anything. No. So, >> oh, so is it going to do those two at the same time? Is that what you're saying? >> It could easily choose to do that. Yes. Yes. So, what might happen is that like, you know, you've got your divides in here and your square roots or whatever you've got going on in here and the computer is like, well, this takes a long time.
17:15 I'll start I'll kick those off, but hey, I can start this rig read time stamp counter already because it doesn't depend on anything before it. And then it turns out that it takes, you know, like 33 cycles. You're like, that's amazing. That's super fast. Um, and then you're like, but when I do millions of these back to back, it definitely takes more like 300 cycles.
17:32 What's going on here? And it's exactly that. So, that is a problem. It's something you have to think about. And it's like where the computer is sort of lying to you. And it's not even really clear what you mean because, you know, there is a a variant of this instruction RDTSC P, which is like a patched version. It's not the P does not stand for patch.
17:47 If I'm remembering my my um computer history here, Intel came up with this instruction and then AMD added this one as a kind of version for them. And it does two things. It also reads another somewhat useful piece of information at the same time. But importantly, it says all of the instructions earlier than this have to have finished before this is allowed to start running.
18:10 And so you get a better fence post here. So this would be an RTSCP. So that helps me understand that things are going on. But you may notice that's there's still a problem with this. So let's say we just turn these both into RDTSPs. After this RDTSCP, I've got loads of other code. What's to stop the computer going, "Oh, I can get ahead of myself here and I'll pull that into the benchmarked area.
18:32 So it's not quite as bad, but code can leak into the benchmarked area and therefore cause um microarchitectural pressure on things that you wouldn't otherwise care about. So typically there are some other serializing instructions that we would put either side of this. There's something called CPU ID which is actually tells me what CPU am I running on.
18:54 That's what it really means. But it also has the side effect of being completely everything before me and everything after me has to have stopped. It stops the whole CPU to tell you the answer of who it is. There are some other things to do with fences to do with loading and storing memories that we put around these type of things. But it it turns out that measuring time is a lot more difficult than you might imagine on computers over here.
19:24 And we're going to have like the the memory or RAM robot over here, which you know, if I could draw a robot sheet, then it would be one of those. Obviously, all he does is he looks out and every time he sees an address in the window in the room where the CPU is,