How Image Watermarking Works - Computerphile Transcript

00:00 Last time we looked at embedding watermarks in text output from well, usually AI, right? So, any generative process. I thought it might be more interesting to look at the image side of things because watermarks are also being embedded in images. And given that I work in computer vision, that's inherently much more interesting for me. Let's think back many years to the video on steganography.

00:23 >> And then of course you can uncover the secret message. >> And then remind ourselves what is a watermark in an image? >> Steganography was that was how you hiding things in >> hiding messages. And now steganography was mostly about trying to prevent people from realizing there was even a message in there, which I didn't do a hugely good job of, but you know, in theory.

00:38 But also you were trying to make it robust to the image being manipulated and also not drastically affect the quality of the image. But steganography mostly is about being secret and maybe trying to maximize the amount of payload cuz you want to actually send actual files secretly. Um watermarking is not really about that. Watermarking is just saying did this person or this process create that image from some signature buried in the image.

01:05 So, we often have signatures buried in images. They just put them in the metadata, right? So, if you take a picture on your camera, you'll have loads of information on what shutter speed you're using and all this kind of stuff and how you've messed up the focus and things like this. >> You didn't talk about the wobbliness. >> No, no, the wobbliness.

01:18 That's not in the meta um you know, that's all in the metadata, but that can be easily edited, it can be easily removed. There's no kind of longevity for that. And also, if you upload that image onto the web, it will almost certainly, let's say convert turn it into a JPEG and remove all of that information. >> Well, that's the biggest problem any with any of this, isn't it?

01:35 Manipulation of images and every image pretty much gets manipulated. >> Yeah. So, suppose I create an image, it doesn't need to be about AI. So, maybe I'm a photographer and I take an image and I don't want people stealing my image. I want people paying for the license for it. I watermark that image in some way. What I don't want to happen is someone just crops off the last few pixels around the edge and now it's a free image, you know?

01:55 So, I have to think about how I might pull off something like that. So, could we use the steganography effect that we did before? So, could we just manipulate some of the least significant bits of all the pixel data? Let's think about that. So, if you remember about how I don't know how many years ago it was, but when I talked about digital images, please don't show the picture of a thinner, younger me.

02:17 Um the the um the idea is that the pixel data is usually stored in the red, green, and blue channels, and each of these are typically eight bits long. So, they have, you know, 1 2 3 4 5 6 7 8. And so, your maximum pixel value is 255 all ones, and your minimum is all zeros. >> Mhm. So, zeros is just black >> Black zeros is black, or at least no red, right?

02:40 And then it would depend on the other three channels. And we could do we could use the least significant bits to hold our data. So, we could use the least significant two bits, for example, right? So, maybe we could bury a message of 1 0 by just changing this to a 1 and keeping that as a 0. And then we go through in some process, and we read off the last two bits of every pixel, and there's our message.

03:02 And we could have a watermark that said Mike Pound took this picture, and you must pay for it, please. I'm terrible at photography, by the way, so no one's buying them. Um now, this doesn't work because there's a couple of good things. The first thing is this is very easy to do, and it's very, very quick. And you can have a huge amount of data stored in here.

03:19 So, you could have my bank details from where to where to send the license money, right? It's all great. But actually, it's unbelievably easy to change, because you can basically change the last two or three bits of an image, and you won't really notice any difference. >> One of the main reasons for doing it this way is that changing them to put the data in doesn't show much of a difference on the >> That Oh, yes, that's the benefit of it, right?

03:38 So, for the benefit of it, I guess, is that by changing this to a 1 or this to a 1, your your pixel data goes up and down up to three out of 255 levels, you're going to see no difference at all. At the cost of the watermark is very easily removed and quite easily detected because you can just read it off. So, you could get my nice paid image, you could take a look at it and go, "Well, there's a watermark in here.

04:00 I think I'll remove that. I'll just randomly flip the last couple of bits. Watermark's gone." So, this isn't going to work. Modern approaches, like most things, like to use AI, right? They use a neural network. They don't use anything like large language models or anything like this. The idea is we need to add something a little bit more substantive.

04:16 But, it also needs to not be pixel level because we need to kind of have good coverage. Otherwise, it's going to be very easy to fiddle around with pixels and remove it. And we need to think about how do we make it robust if someone's compressing it as a JPEG or changing the color or flipping the image. Any of this kind of information needs to preserve the watermark.

04:35 This has been a little bit in the news because Google are now they've been talking about SynthID, which is the same technique that's used for text, but there's the SynthID image version, which um is now used to produce to watermark any generative Google image. It will also typically watermark when you take a picture with a Pixel camera and things like this, right?

04:56 And we'll talk perhaps more about that in a different video when we talk about signatures. We're going to talk about Adobe's Trustmark as a base because it's very similar to the Google implementation. We can talk about some of the differences. And also, good for them, they released it open source, right? Which means that we have access to the models.

05:11 We can we can watermark something. We can look at the watermark. We can take it away again. And I think that's from an understanding point of view, obviously, a good thing. Google's system, we know roughly what it does. It's not released, right? And that's I I guess it's it's partly trade secrets, but it's mostly because in this case, knowing exactly how the watermark was added might allow you to try and circumvent it in some way, right?

05:34 A bit, maybe. So, let's think about this. We've got an image and I'm going to draw one of my good pictures now. It's going to be an image of some people. And this one's taller than that one. There we go. And we want to add something to this image to say this was created by AI or this was made by Mike and needs to not be copied or what have you. So some reason why I want to embed a watermark in here.

05:56 Let's think about how we might do that using a neural network. Well, the first thing we could do is we could just create one of our big encoder-decoder networks that we covered before, which are still very, very common in AI, which basically takes the image, it takes a watermark, and it just outputs another copy of the image with the watermark added.

06:15 So I'm just going to kind of do that a little bit to pretend there's a watermark in there. Now, there's a few things that are really important. First of all, this image should be extremely similar to this image. If you've changed the color of the person's jumper, people are not going to be happy about that or just remove the person. On the other hand, there has to be something in there because otherwise, there is no there is no watermark.

06:34 How would we then detect that this watermark exists? Well, what we would do is we would just have another network that was trained. Maybe we would say yes or no. So, we add a watermark and then later on, someone comes along and runs this network and says, "Is there a watermark in this image?" and it says yes or no. Now, that's already a bit of a problem because suppose I wanted to watermark my image as opposed to a Google AI image, I need some way of saying this is a a Mike watermark, not a Google watermark or an Adobe

07:03 watermark. How do I do that? Well, the answer is we have to add a little code here and we have to be able to recover this code. So, maybe we have a short string, so let's say Mike. This gets taken in by the network and now this watermark is in some sense influenced by this, right? In the same way that you might generate an image influenced by a text prompt.

07:20 Almost like a QR code embeds. A bit like that, yeah. But the amount of data you can embed is very, very small compared to steganography or any normal process of actually storing data, right? We're talking a little 32 bits, maybe 100 bits for a really advanced system. Not a lot of not a lot of data. So, So, changes now, so we don't do this, which I've now filled this box.

07:42 So, going to I'm going to I'm going to get rid of that. Draw an arrow over here. Problem solved. This is going to be the the the mic string. Right? Or I couldn't find a string. So, this will return a string or it will return no string because there was no watermark. So, will this process work? And the answer is, yeah, kind of. Right? So, let's suppose we train this.

08:03 How would we do it? Well, what we would do is we would train a bit like those classic sort of adversarial training processes that we saw a few years ago, where you just give these networks thousands of images, let's just say off internet, thousands of random watermark strings, and the job is for this to return the string. And so, you can imagine when you're training this, this gets a bit better at finding them, but also sort of sends a signal back here, and this has got to influence the generation process in some way

08:30 to make this more recoverable, and things like this. So, this is a kind of joint training process, where you're training both of these at the same time. And you could also add other losses, right? So, remember, when you're doing machine learning, a loss is the mathematical measure of how well you're doing. So, this will be a loss here of saying, have you managed to recover this word?

08:53 But, you might also have another network here that's detecting whether or not this is real or watermarked. Question mark. And so, what we're trying to do is make sure that this doesn't work. We're saying, okay, can you successfully recover the mic, but also, can you make sure that this network is not able to tell that you did something? And if that's the case, then we also have something that's perceptually pretty similar to this.

09:18 Otherwise, it will just draw a big circle, or it will literally write the word mic on it, and and it's it's not helpful, right? So, that would be rubbish. Um Good. So, this is almost there, right? But, actually, in a kind of the last few years, we've started to develop slightly better methods than this because this is this is the grounding of how they work.

09:41 But the problem is that this isn't robust. So, suppose I have this watermark and I just decide to delete that bit of image, right? Or maybe I subtly alter the the the color of like this person's sleeve or something like this. That's going to throw the watermark off. If we divide this image in two, we make it a lot smaller, the watermark is thrown off.

10:00 It's never seen any examples of this. >> Cuz cuz resizing, that happens all the time. You've taken a massive megapixel image, but then you want to send it via a messaging service on your phone. So, it's going to get shrunk. >> Yeah, exactly. So, in actual fact, we need to make this even a little bit more complicated. All right, so I'm going to redraw it.

10:17 Let's think about how could we make this better? What the main goal is to try and make this as as robust as possible to people messing around with the image because of course people are going to edit the image, not necessarily even to deliberately remove the watermark, but just because that's what happens to images. So, here's our input image. What we're going to do is put it through our nice encode decoder network.

10:37 Here's our more generic watermark. All right, this is our encode decoder. This is our our watermarked image, which I'm going to signal by just putting a watermark on it. Now, here is what where we would try and extract the watermark. We're not going to do that anymore. What we're going to do is do some kind of messing about. >> Is that a technical term?

11:01 >> That is not what was in the paper. All right, crops, flips, colors, resizing, deleting bits, rotations, as much as we can do at random to this image to create a noisy Why did I do a an orange border? A noisy version of this image, which is way way harder to extract the watermark from. So, we are doing a set of let's say 10, 20, 30 different random augmentations, all of which are saying what would have happened if this watermark was messed with?"

11:33 And then we have our network which is saying, "What's the watermark?" And we then we start training again. So, we train now thousands of images, thousands of random watermarks, thousands of different augmentations. Can we extract the watermark? And over time, your network starts to become a little bit better at doing this. Um now, there's a few other kind of technical details that are really interesting in the in the um TrustMark paper and subsequent implementations.

12:03 How do we deal with the fact that the image might change size? Right? So, for example, if you spread your watermark around as a nice kind of let's say let's say it gets darker as you go right in the image. It's going to get darker faster if you shrink the image. So, we need to deal with that. Now, we can do that a bit in the augmentations, but actually we also want to be able to watermark quickly any size of image.

12:22 So, the actual TrustMark system, which is essentially this bit here, this is the this is the watermarking, this is the detector. Right? This is the training process. What we want to do is we want to do this at a fixed size. So, this is actually done at 256 pixels. So, if you wanted to Let's suppose you now you know you you you you've done this, you've got this network here and this network here.

12:53 Let's call this network F and let's call this network G. Uh so, they they do actually do that a lot in the literature. Um so, F is your encoder that adds a watermark. G is your detector that detects the code of a watermark. What we want to do if we want to watermark an arbitrary image that we've generated or taken on a very high-resolution camera or something like that, it's not necessarily going to work if we haven't trained on high-resolution images.

13:13 So, we take our big image, let's say it's 10 megapixels. We shrink it down to 256 pixels across, and then we apply our F function to watermark it. So, that produces a very small watermark image that looks a bit like this. Okay? Then, we subtract our shrunk image from this to get the difference, the actual watermark pixels. This will be an image that's mostly gray with nothing much changed, and then a few small bright spots or dark spots as as subtle watermarks have been added.

13:48 So, this is our kind of So, we take this away and we get our kind of you know, our sort of weird watermark changes. I mean, I I can't draw this. We'll show it in a minute, right? And it'll make much more sense. So, we then can expand that back up and apply it to the original image. So, what we're doing is we're doing Yeah, look at that. I wish I'd drawn a more sensible watermark.

14:14 So, we can expand and now it can now we can watermark any size of image. So, then to detect, all we do is we shrink any image to 256 and then detect the watermark, and it should still be robust. And now our system handles any size. So, Trustmark was released open source. You can download the weights and you can add watermarks. So, I've coded that up and let's have a look at what it does.

14:35 So, we have here an image of I downloaded off the campus we're on, Jubilee, and it's a perfectly normal image, and we're going to try and add a watermark. So, watermark.py, we're going to add the keyword or the watermark computer. That's our string. That's about as long as we can cram in before it starts to break down. We're going to take it for this image and we're going to output a watermarked version, and we're also going to output the difference image so we can actually see the watermark just just sort of out of

15:01 interest. All right? And we're going to run this. Now, I haven't properly set up CUDA and PyTorch, so it's taking ages. But, I normally would have done that. So, the maximum change to a pixel intensity was about 25 points, which is actually not It's actually a lot bigger than it would have been if we'd have done a least significant bit change. But, if it's done in a subtle way on a on a bit that's already bright, maybe it doesn't make any difference.

15:26 Well, on a reflection or something like this. Let's have a look. So, if I click on the watermarked image, looks kind of normal, right? Can't really see anything wrong with it. You can see it if we flip between the two. So, if I flip between the two, you can see it's adding highlights. It's filling There's some Yeah, there's some artifacts in the sky.

15:48 There's some more reflections than there were before. So, it is adding some changes to the image. If you look at the difference, you can see that these changes are actually spread all the way across the image. There are fewer on the sky, which is a good thing because if it was a completely blue sky, that would that would be terrible, right? If you just put loads of edges in there.

16:05 But, you can see it's making some substantive changes to areas of tree and things like this. So, let's see if it works, if we can detect it. So, python uh detect. dot dot pi. Can't type. And we're going to detect uh Jubilee campus watermark. What it's going to try and do is determine first if there is a watermark and also what the code was that was added into this image.

16:29 There we go. It's done it. And it found it fine. So, how robust is it? Well, let's have a mess around. So, if I open an editing program, Paint.net, and I take my original image, what should we do to it? >> Shrink it. >> JPEG it? Bit of both? Okay. Well, okay. Let's let's do a couple. So, let's save as a low quality. I mean, not that low. Let's not get carried away.

16:50 And I'm going to drop the Apparently, it's on quality 90. I'm going to drop it down to about 75. Let's see if it works. I hope so. It's been trained with this in mind, but that's the idea. So, it should work. >> No watermark detected. >> Didn't work. Okay. >> You did edit the watermarked one and not the unwatermarked one, didn't you? Did you drag the wrong one in?

17:15 >> Yes. All right. So, what's happened here is I've manipulated the non-watermarked image and then been able to unable to retrieve a watermark from it, right? So uh clearly I have not had enough Costa coffee. Um Ah, okay. That may be up there in the top five mistakes of all time that I've done on this channel. Uh okay, let's try this again and do it properly.

17:33 So, let's take the watermarked image and let's save it as a JPEG. Let's say low quality and save it as um 75. Oh, I know. I'm going to 75. Okay, so that's saved at 75. Um and then I'm going to shrink it to 600 and I'm going to save as small. Put the quality back up again, maybe a bit. I don't know, let's say 93. Make it up. Uh okay. Right, let's see this.

18:07 So, let's see uh let's do this. Okay, so this is the JPEG version. So, what JPEG is doing, if you remember back to our older videos, is it's taking all of the high frequencies away. So, if you'd have done any kind of least significant bit approach, it wouldn't work. But it has worked, right? Watermark 8 the same text string has worked absolutely fine.

18:27 If we do a small one, it should also work, right? Now, I don't know to what extent it will continue to go if you keep making it smaller and smaller, but it's it's pretty robust. And the modern systems that have built upon Trustmark are even more robust. So, they can There we go. They can find watermarks when you've done quite a lot of substantial edits to an image, right?

18:47 And even if you pick up bits of image and put them in other images, the watermark remains there. >> Look, I'm guessing you probably can push this too far, though. >> I would have thought so, yes. >> you crop it or something? >> Okay, so let's let's take a >> Maybe you took a bit of the building off on the right or something. >> Yeah, let's let's let's take let's take like that bit, right?

19:06 Uh that's the wrong button. That's the wrong button. That's the right button, right? So, we've cropped it Let's maybe fiddle with the colors a bit. So, let's I don't know, just just just increase the saturation a little bit, make it a bit more sunny. Yeah, okay. Now, we'll save that as a watermark new. Okay. And we've also JPEG added a bit of JPEG compression.

19:27 And let's see if it detects it. I don't know is the answer to that. >> I think it'd be very difficult to detect a watermark. That's quite a lot of changes. >> It is quite a lot of changes, yes. Modern systems that built It's still got it. Right. Yeah. I mean, let's let's let's go further. So, let's let's crop down to kind of just just the just let's crop down to there.

19:46 Right. So, now we're only looking at the at the water really and we've got a few trees in there. Ah, it's gone. It's gone. Right. So, but I removed, you know, a half the image by area. So, you know, not too bad. Um So, I think yeah, it's not impervious. In actual fact, the the the more recent systems are more robust to this. They make the augmentations even stronger and they improve the training, the power of the actual backbone networks of doing this.

20:11 They've They're often using transformers rather than just convolutional networks. And so, there are lots of modernizations that make this even more robust. It's not flawless, but it's not too bad. We saw that work pretty well. It wasn't perfect. Once we cropped enough or changed the hue, things started to break down. Modern versions of this, so like Google Synth SynthID text, will go even stronger with the augmentations to try and make sure that those kind of things will still be picked up.

20:38 There are actually some where they've been trained to work even if you print the photo out on a piece of paper and take a picture of that, it will still flag as a watermark. Now, there are other attacks that have also been considered. Cuz you got to think about like if if this was a watermarking system, what kind of things might you want to do as an attacker?

20:57 You might want to remove a watermark from an image you are trying to hide. You might want to fictitiously add a valid watermark to another image. So, one thing you could do is if you had access to the original source image and you had access to the watermarked version, you could subtract them and calculate that watermark. And then you could add that or stamp that onto any image and watermark that image.

21:19 Let's suppose that the image is watermarked as having been created by AI, you could then stamp it on someone else's image and say, "Ah, it's AI." when it's not. So, how do we stop that from happening? Well, what they do is they build that into the training process. They say, "Can we reliably recover the watermark, but only when it was the actual image that that watermark was added to?"

21:38 So, they take that watermark and they put it on different images and try and train against that ability. >> Because when you showed us the gray image that was the difference in that particular one, it was specific to that image. >> It will always be a bit specific to the image because of the way the training process is trying to make it so you can't see it.

21:55 So, if you if you just completely disregarded the image and wrote the word computer, obviously that's not going to work. But, it's not robust enough. You could probably still take that watermark off, put it on a landscape, and it would be seen as being watermarked, and you might not notice. So, they're kind of building a more about a more rigorous method into the training.

22:13 So, they take incorrect watermarks, put them on different images, and try and remove the ability to detect that watermark while also creating the ability to detect the valid watermarks. So, in sense, this watermark process is now based not only on the input image, not only on the code we were trying to incorporate, but also tying those linking those two things together so that you can't swap them around.

22:39 >> Is the idea of this that it's mainly used for AI images, or is it for people, like you say, who are perhaps trying to protect their IP? What what's the main idea? >> I think So, Google's roll out of this is primarily about AI images because they produce a lot of them. Um but, in actual fact, watermarks have been around for many, many years. It's This is not something new.

22:58 A lot of the time it's about copyright or ownership or licensing. Often, the watermarks are made obvious because they want to aesthetically ruin an image so you don't use it. But often these more subtle watermarks, they're about licensing, they're about making sure something isn't leaked, these kind of things. Regardless of why you want to do it, we have to accept that this is not a perfect solution because what will happen a lot of the time is someone will find a way to remove it, they'll crop a bit here and there,

23:20 they'll change the colors in some way. And given how powerful image editing is now, you can imagine you could still make a quite pleasing image but it's no longer watermarked. Let's imagine for a second that we're in a courtroom. Right? And I show a picture of someone committing a crime and it has a watermark that says it was generated by Google AI, right?

23:39 That's not going to be good for my for my prosecution my prosecution case. But the reverse is not necessarily the case. I can't say and it's not watermarked and thus it is definitely not AI because I've got no evidence of this. So actually all that watermarking does is say this image went through this particular process. It doesn't say that the image is or not is is trustworthy or not or it is always not AI art necessarily.

24:07 And there may be a situation, let's say you're submitting a photo to a wildlife photography competition where you want to prove there was no AI used at all, right? This doesn't do that. You can't say I've watermarked it with no AI because anyone could do that. I've just used I've just shown you the code. So how do we do that? Well, the answer is we use it the same method we use to make sure we have trust on the internet which is we sign with digital certificates and digital signatures.

24:31 So we include in the metadata now more signatures that verify where something came from and we'll talk about that in the next video. Sofa versus Matt, zero versus one, Matt wins. And you can see that we've selected the winners completely at random because these ones and naughts are generated by a random function, a pseudo-random function. So, the distribution of words in our next round are going