1 00:00:01,550 --> 00:00:03,920 The following content is provided under a Creative 2 00:00:03,920 --> 00:00:05,310 Commons license. 3 00:00:05,310 --> 00:00:07,520 Your support will help MIT OpenCourseWare 4 00:00:07,520 --> 00:00:11,610 continue to offer high quality educational resources for free. 5 00:00:11,610 --> 00:00:14,180 To make a donation or to view additional materials 6 00:00:14,180 --> 00:00:18,140 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:18,140 --> 00:00:19,026 at ocw.mit.edu. 8 00:00:22,500 --> 00:00:24,690 GILBERT STRANG: I'm determined to tell you something 9 00:00:24,690 --> 00:00:27,000 about the convolution rule. 10 00:00:27,000 --> 00:00:31,290 I just get close to it, but haven't quite got there. 11 00:00:31,290 --> 00:00:34,380 And actually, I'd like to say something also 12 00:00:34,380 --> 00:00:39,000 about why convolution is so important. 13 00:00:39,000 --> 00:00:49,950 I mentioned here a paper about images in deep learning by-- 14 00:00:49,950 --> 00:00:52,280 it has three authors, and these are two of them. 15 00:00:52,280 --> 00:00:54,790 Maybe you recognize Hinton's name. 16 00:00:54,790 --> 00:00:56,790 He's originally English. 17 00:00:56,790 --> 00:01:01,590 He was in San Diego for quite a few years, 18 00:01:01,590 --> 00:01:03,530 and now he's in Canada. 19 00:01:03,530 --> 00:01:08,400 So Toronto and Montreal are big centers now for deep learning. 20 00:01:08,400 --> 00:01:15,710 And he's really one of the leaders, and so is Sutskever. 21 00:01:15,710 --> 00:01:21,810 So maybe you know that the sort of progress of deep learning 22 00:01:21,810 --> 00:01:24,750 can often be measured in these competitions that 23 00:01:24,750 --> 00:01:32,250 are held about every year for how well does-- 24 00:01:32,250 --> 00:01:42,390 people design and execute a whole neural net. 25 00:01:42,390 --> 00:01:47,250 And this was a competition about images. 26 00:01:47,250 --> 00:01:52,650 So that is really demanding, because, as I said last time, 27 00:01:52,650 --> 00:01:55,800 an image has so many samples, so many 28 00:01:55,800 --> 00:02:00,600 pixels that the computational problem is enormous. 29 00:02:00,600 --> 00:02:05,700 And that's when you would go to convolution neural nets, CNN, 30 00:02:05,700 --> 00:02:10,210 because a convolutional net takes fewer weights, because 31 00:02:10,210 --> 00:02:13,440 of the same weight as appearing along diagonals. 32 00:02:13,440 --> 00:02:16,140 It doesn't need a full matrix of weights, 33 00:02:16,140 --> 00:02:19,480 just one top row of weights. 34 00:02:19,480 --> 00:02:24,390 Anyway, so this is one of the historical papers 35 00:02:24,390 --> 00:02:26,890 in the history of deep learning. 36 00:02:26,890 --> 00:02:30,220 I'll just read a couple of sentences. 37 00:02:30,220 --> 00:02:33,100 We trained-- so this is the abstract. 38 00:02:33,100 --> 00:02:38,300 We trained a large deep convolutional neural network. 39 00:02:38,300 --> 00:02:45,760 I'll just say that it ran for five days on two GPUs. 40 00:02:45,760 --> 00:02:50,210 So it was an enormous problem, as we'll see. 41 00:02:50,210 --> 00:02:53,920 So we trained a large deep network, CNN, 42 00:02:53,920 --> 00:02:59,200 to classify 1.2 million high res images in ImageNet. 43 00:02:59,200 --> 00:03:04,540 So ImageNet is a source of millions of images. 44 00:03:04,540 --> 00:03:08,350 And on the test data, they-- 45 00:03:08,350 --> 00:03:12,520 well, the last sentence is maybe a key. 46 00:03:12,520 --> 00:03:14,620 We entered a variant of this model 47 00:03:14,620 --> 00:03:18,760 in the competition, 2012 competition, 48 00:03:18,760 --> 00:03:26,020 and we achieved a winning top five test error rate of 15% 49 00:03:26,020 --> 00:03:30,880 compared to 26% for the second place team. 50 00:03:30,880 --> 00:03:32,860 So 15% error. 51 00:03:32,860 --> 00:03:37,130 They got 26% was the best that the rest of the world did. 52 00:03:37,130 --> 00:03:43,890 And so that-- and when he shows the network, 53 00:03:43,890 --> 00:03:47,250 you realize what's gone into it. 54 00:03:47,250 --> 00:03:55,000 It has convolution layers, and it has some normal layers, 55 00:03:55,000 --> 00:04:00,580 and it has max pooling layers to cut the dimension down 56 00:04:00,580 --> 00:04:01,240 a little bit. 57 00:04:04,720 --> 00:04:09,970 And half the samples go on one GPU and half another. 58 00:04:09,970 --> 00:04:12,820 And at certain points, layers interconnect 59 00:04:12,820 --> 00:04:15,220 between the two GPUs. 60 00:04:15,220 --> 00:04:20,370 And so to reduce overfitting-- you remember that. 61 00:04:20,370 --> 00:04:23,170 It's a key problem is to reduce overfitting 62 00:04:23,170 --> 00:04:25,750 in the fully connected layers. 63 00:04:25,750 --> 00:04:31,480 Those are the ordinary layers with full weight matrices. 64 00:04:31,480 --> 00:04:36,390 We employed a recently developed regularization called dropout. 65 00:04:36,390 --> 00:04:40,615 So dropout is a tool which, if you're in this world, you-- 66 00:04:40,615 --> 00:04:43,900 I think Hinton proposed it, again, 67 00:04:43,900 --> 00:04:46,390 by seeing that it worked. 68 00:04:46,390 --> 00:04:51,760 It's just a careful dropout of some of the data. 69 00:04:51,760 --> 00:04:54,670 It reduces the amount of data, and it 70 00:04:54,670 --> 00:05:00,340 doesn't harm the problem. 71 00:05:00,340 --> 00:05:03,970 So the neural network has 60 million parameters. 72 00:05:03,970 --> 00:05:05,830 60 million. 73 00:05:05,830 --> 00:05:12,250 With 650,000 neurons, five convolutional layers, and three 74 00:05:12,250 --> 00:05:14,560 fully connected layers. 75 00:05:14,560 --> 00:05:18,490 I just mention this. 76 00:05:18,490 --> 00:05:23,560 If you just Google these two names on the web, 77 00:05:23,560 --> 00:05:24,820 this paper would come up. 78 00:05:27,816 --> 00:05:31,440 So we're talking about the real thing here. 79 00:05:31,440 --> 00:05:34,840 Convolution is something everybody wants to understand. 80 00:05:34,840 --> 00:05:36,190 And I'd like to-- 81 00:05:36,190 --> 00:05:41,170 since I've started several days ago, 82 00:05:41,170 --> 00:05:44,950 and I'd like to remember what convolution means. 83 00:05:44,950 --> 00:05:52,230 Let me-- so if I convolve two vectors 84 00:05:52,230 --> 00:05:57,040 and I look for the k-th component of the answer, 85 00:05:57,040 --> 00:06:04,930 the formula is I add up all the c's times d's where 86 00:06:04,930 --> 00:06:10,330 are index i plus j adds to k. 87 00:06:10,330 --> 00:06:11,710 Why do you do such a thing? 88 00:06:11,710 --> 00:06:17,620 Because c might be represented by a polynomial, 89 00:06:17,620 --> 00:06:23,950 say x plus c Nx to the N. And d might 90 00:06:23,950 --> 00:06:30,745 be represented by another 1d 1x plus dm x to the m, let's say. 91 00:06:34,980 --> 00:06:40,200 And convolution arises when I multiply those polynomials. 92 00:06:40,200 --> 00:06:42,690 Because for a typical-- 93 00:06:42,690 --> 00:06:44,490 and then collect terms. 94 00:06:44,490 --> 00:06:49,860 Because a typical power of x, say x to the k, 95 00:06:49,860 --> 00:06:51,560 the coefficients are-- 96 00:06:55,410 --> 00:06:58,970 well, how do we get x to the k in multiplying these? 97 00:06:58,970 --> 00:07:02,460 I multiply c0 times a dk. 98 00:07:02,460 --> 00:07:05,670 Somewhere in here would be a dk x to the k. 99 00:07:05,670 --> 00:07:13,050 So a c0 times a dk would give me an x to the k term. 100 00:07:13,050 --> 00:07:15,180 And a c1 times-- 101 00:07:15,180 --> 00:07:17,570 everybody sees this coming now? 102 00:07:17,570 --> 00:07:20,520 c1 has an x in it already. 103 00:07:20,520 --> 00:07:25,530 So over there, we would look at dk minus 1 with one less x. 104 00:07:25,530 --> 00:07:29,440 So it would be c1 dk minus 1. 105 00:07:29,440 --> 00:07:33,780 This is just what you do when you multiply a polynomial. 106 00:07:33,780 --> 00:07:37,980 And the point is that the way we recognize those terms 107 00:07:37,980 --> 00:07:45,900 is that the exponents 0 and k, the exponents 1 and k minus 1, 108 00:07:45,900 --> 00:07:47,550 always add to k. 109 00:07:47,550 --> 00:07:49,950 So that's where this formula comes from. 110 00:07:49,950 --> 00:07:57,240 We take a c times a d hiding behind our cx to the i and a dj 111 00:07:57,240 --> 00:08:04,830 x to the j and when i plus j is k, this is x to the k. 112 00:08:04,830 --> 00:08:07,560 And that's the term we're capturing. 113 00:08:07,560 --> 00:08:12,610 So this is the coefficient of that term. 114 00:08:12,610 --> 00:08:17,760 And let me write it as a slightly different way, where 115 00:08:17,760 --> 00:08:23,130 you actually see even more clearly convolution operating. 116 00:08:23,130 --> 00:08:26,730 So j is k minus i, right? 117 00:08:26,730 --> 00:08:35,700 So it's the sum of cidj, but the j has to be k minus i. 118 00:08:35,700 --> 00:08:42,059 So this is the way to remember the formula 119 00:08:42,059 --> 00:08:49,900 for the coefficients in c star d in the convolution. 120 00:08:49,900 --> 00:08:52,250 You look at c's times d's. 121 00:08:52,250 --> 00:08:54,710 It's a form of multiplication. 122 00:08:54,710 --> 00:08:59,660 It comes from ordinary multiplication of polynomials. 123 00:08:59,660 --> 00:09:03,290 And when you collect terms, you're collecting c, 124 00:09:03,290 --> 00:09:08,930 the i-th c and the k minus id, and you're taking all possible 125 00:09:08,930 --> 00:09:09,920 i's. 126 00:09:09,920 --> 00:09:12,260 So it's a sum over all possible i's 127 00:09:12,260 --> 00:09:16,880 there to give you the k-th answer. 128 00:09:16,880 --> 00:09:19,550 Well, just to see if you got the idea, 129 00:09:19,550 --> 00:09:22,820 what would be the convolution of two functions? 130 00:09:22,820 --> 00:09:26,880 Suppose I have a function f of x. 131 00:09:26,880 --> 00:09:30,410 And I want I convolve that with a function g of x. 132 00:09:33,590 --> 00:09:35,350 OK. 133 00:09:35,350 --> 00:09:39,680 And notice that I have not circled this symbol. 134 00:09:39,680 --> 00:09:42,110 So I'm not doing periodic convolution. 135 00:09:42,110 --> 00:09:44,610 I'm just doing straightforward convolution. 136 00:09:44,610 --> 00:09:48,740 So what are we going to have in the case of two functions? 137 00:09:48,740 --> 00:09:51,650 What would that mean, a convolution of functions? 138 00:09:51,650 --> 00:09:56,430 I'm in parallel here with a convolution of two vectors. 139 00:09:56,430 --> 00:10:00,300 So think of these now have become functions. 140 00:10:00,300 --> 00:10:02,690 The case component has become-- 141 00:10:02,690 --> 00:10:07,730 really, I should say f star g at x. 142 00:10:07,730 --> 00:10:12,260 That's really the parallel to this. 143 00:10:12,260 --> 00:10:12,920 So let me. 144 00:10:16,940 --> 00:10:20,900 So I'm telling you the answer at x. 145 00:10:20,900 --> 00:10:24,760 Here I told you the answer at k. 146 00:10:24,760 --> 00:10:29,270 The k-th component looks like that. 147 00:10:29,270 --> 00:10:33,500 What does the x value of the convolution look 148 00:10:33,500 --> 00:10:35,390 like for functions? 149 00:10:35,390 --> 00:10:37,910 OK, I'm just going to do this. 150 00:10:37,910 --> 00:10:39,800 I'm going to do the same as this. 151 00:10:39,800 --> 00:10:43,200 Instead of summing, what will I do? 152 00:10:43,200 --> 00:10:43,820 Integrate. 153 00:10:46,980 --> 00:10:52,200 Instead of c sub i, I'll have f of x. 154 00:10:56,380 --> 00:11:01,300 The index i is changing over to the continuous variable x. 155 00:11:01,300 --> 00:11:10,390 And now g instead of dk minus i, what do I have here? 156 00:11:10,390 --> 00:11:16,330 So it's the k minus i component. 157 00:11:16,330 --> 00:11:17,770 That will go to-- 158 00:11:17,770 --> 00:11:19,960 let me just write it down-- 159 00:11:19,960 --> 00:11:23,320 t minus x. 160 00:11:23,320 --> 00:11:29,380 So in this translation, f is being translated to c. 161 00:11:29,380 --> 00:11:34,100 Or sorry, f corresponds to c. 162 00:11:34,100 --> 00:11:36,540 g corresponds to d. 163 00:11:36,540 --> 00:11:40,830 k corresponds to x. 164 00:11:40,830 --> 00:11:45,160 Oh no, sorry. i corresponds to x. 165 00:11:45,160 --> 00:11:52,940 And k minus i corresponds to t minus x. 166 00:11:52,940 --> 00:11:56,160 So k corresponds to t. 167 00:11:56,160 --> 00:12:01,370 This would be the convolution of two functions. 168 00:12:01,370 --> 00:12:04,460 Oh, it's a function of t. 169 00:12:04,460 --> 00:12:06,470 Bad notation. 170 00:12:10,940 --> 00:12:14,630 The t is sort of the amount of shift. 171 00:12:14,630 --> 00:12:16,310 See, I've shifted g. 172 00:12:16,310 --> 00:12:18,190 I've reversed it. 173 00:12:18,190 --> 00:12:23,360 I've flipped it and shifting it by different amounts t. 174 00:12:23,360 --> 00:12:25,340 It's what you have in a filter. 175 00:12:25,340 --> 00:12:32,310 It's just also always present in signal processing. 176 00:12:32,310 --> 00:12:35,990 So that that would be a definition. 177 00:12:35,990 --> 00:12:40,850 Or I could, if you like, if you want an x variable to come out, 178 00:12:40,850 --> 00:12:46,580 let me make an x variable come out by exchanging t and x. 179 00:12:46,580 --> 00:12:52,600 So this would be x minus t dt. 180 00:12:52,600 --> 00:12:54,500 I like that, actually, a little better. 181 00:12:54,500 --> 00:12:59,480 And it's the integral over t minus infinity to infinity 182 00:12:59,480 --> 00:13:04,670 if our functions were on the whole line. 183 00:13:04,670 --> 00:13:07,475 So there will be a convolution rule for that. 184 00:13:10,280 --> 00:13:15,830 This will connect to the Fourier transform of the two functions. 185 00:13:15,830 --> 00:13:19,370 Over here, I'm connecting it to the discrete Fourier 186 00:13:19,370 --> 00:13:21,590 transform of the two functions. 187 00:13:21,590 --> 00:13:27,110 And I've been making the convolution cyclic. 188 00:13:27,110 --> 00:13:30,020 So what does-- can I add cyclic now? 189 00:13:30,020 --> 00:13:32,210 This is ordinary convolution. 190 00:13:32,210 --> 00:13:34,400 This is what you had in the first lab, 191 00:13:34,400 --> 00:13:38,600 I think, from Raj Rao. 192 00:13:38,600 --> 00:13:40,220 The first lab, you remember you had 193 00:13:40,220 --> 00:13:47,330 to figure out how many components the convolution 194 00:13:47,330 --> 00:13:48,860 would have? 195 00:13:48,860 --> 00:13:51,080 And you didn't make it cyclic. 196 00:13:51,080 --> 00:13:57,080 So a cyclic convolution, if this has n components 197 00:13:57,080 --> 00:14:00,050 and this has n components, then the convolution 198 00:14:00,050 --> 00:14:02,490 has n components. 199 00:14:02,490 --> 00:14:06,950 Because keeping n is the key number 200 00:14:06,950 --> 00:14:10,940 there, the length of the period. 201 00:14:10,940 --> 00:14:13,310 And similarly, over here, if f is 202 00:14:13,310 --> 00:14:16,940 2 pi periodic and g is 2 pi periodic, 203 00:14:16,940 --> 00:14:21,170 then we might want to do a periodic convolution 204 00:14:21,170 --> 00:14:24,860 and bring it-- 205 00:14:24,860 --> 00:14:31,590 get an answer that also has 2 pi period 2 pi. 206 00:14:31,590 --> 00:14:34,310 So you could compute the convolution 207 00:14:34,310 --> 00:14:38,850 of sine x with cos x, for example. 208 00:14:38,850 --> 00:14:43,280 OK, let's stick with vectors. 209 00:14:43,280 --> 00:14:49,510 So what's the deal when I make it cyclic? 210 00:14:49,510 --> 00:14:53,720 When I make it cyclic, then in this multiplication, 211 00:14:53,720 --> 00:14:55,940 I really should use-- 212 00:14:55,940 --> 00:14:59,840 I've introduced w as that instead of x. 213 00:14:59,840 --> 00:15:00,400 So cyclic. 214 00:15:04,900 --> 00:15:14,350 x becomes this number w, which is e to the 2 pi i over n 215 00:15:14,350 --> 00:15:17,500 and has the property then that w to the n-th 216 00:15:17,500 --> 00:15:22,120 is 1 so that all vectors of length greater than n 217 00:15:22,120 --> 00:15:27,640 can be folded back using this rule to a vector of length n. 218 00:15:27,640 --> 00:15:30,940 So we get a cyclic guy. 219 00:15:30,940 --> 00:15:36,100 So how does that change the answer? 220 00:15:36,100 --> 00:15:41,320 Well, I only want k going from 0 to n minus 1 221 00:15:41,320 --> 00:15:43,180 in the cyclic case. 222 00:15:43,180 --> 00:15:46,810 I don't want infinitely many components. 223 00:15:46,810 --> 00:15:49,990 I've got to bring them back again. 224 00:15:49,990 --> 00:15:56,380 And let me just say what the rule would be. 225 00:15:56,380 --> 00:16:03,460 You just ask, say, i plus j. 226 00:16:03,460 --> 00:16:07,720 You would look at that modulo n. 227 00:16:07,720 --> 00:16:12,790 That's what a number theory person would call it. 228 00:16:12,790 --> 00:16:18,400 We only look at the remainder when we divide by n. 229 00:16:18,400 --> 00:16:23,590 So now the sums go only from 0 to n minus 1, 230 00:16:23,590 --> 00:16:28,756 and I only get an answer from 0 to n minus 1. 231 00:16:28,756 --> 00:16:33,440 Well, I've done that pretty quickly. 232 00:16:33,440 --> 00:16:37,350 That's if I wanted to do justice to-- 233 00:16:37,350 --> 00:16:41,350 So the difference between non-periodic. 234 00:16:43,950 --> 00:16:54,790 So non-periodic and periodic will be the difference 235 00:16:54,790 --> 00:16:56,950 between-- 236 00:16:56,950 --> 00:17:05,150 so I have some number t0 on the diagonals. t1, t2, t minus 1, 237 00:17:05,150 --> 00:17:07,000 t minus 2, and so on. 238 00:17:07,000 --> 00:17:08,069 Constant diagonals. 239 00:17:08,069 --> 00:17:10,450 So the key name there is Toeplitz. 240 00:17:14,069 --> 00:17:20,930 And if it's periodic, then I have, I'll say, c, c, c. 241 00:17:20,930 --> 00:17:25,640 And then the next one will be c1, c1, coming around to c1. 242 00:17:25,640 --> 00:17:27,290 And c2 coming around. 243 00:17:27,290 --> 00:17:32,210 So it's n by n period n. 244 00:17:32,210 --> 00:17:34,485 So it's a circulant matrix. 245 00:17:38,090 --> 00:17:40,550 N by N. 246 00:17:40,550 --> 00:17:42,680 OK. 247 00:17:42,680 --> 00:17:45,490 That's the big picture. 248 00:17:45,490 --> 00:17:48,950 And I think in that first lab, you 249 00:17:48,950 --> 00:17:51,350 were asked to do the non-circulant case. 250 00:17:51,350 --> 00:17:54,560 Because that's the one where you have to do a little patience. 251 00:17:54,560 --> 00:17:55,750 What will be the length? 252 00:17:55,750 --> 00:18:00,740 Yeah, what would be the length of a non-circulant? 253 00:18:00,740 --> 00:18:02,420 So not circulant. 254 00:18:02,420 --> 00:18:10,310 Now, suppose the c vector has p components 255 00:18:10,310 --> 00:18:13,580 and the d vector has q components. 256 00:18:13,580 --> 00:18:16,600 How many components in their convolution? 257 00:18:16,600 --> 00:18:18,080 Shall I write that question down? 258 00:18:18,080 --> 00:18:22,580 Because that brings out the difference here. 259 00:18:22,580 --> 00:18:39,150 So if I have p, if c has P components, d has q components, 260 00:18:39,150 --> 00:18:43,330 then the convolution of c and d has how many? 261 00:18:46,300 --> 00:18:47,630 So I'm multiplying. 262 00:18:47,630 --> 00:18:52,060 So it's really this corresponds to a polynomial of degree 263 00:18:52,060 --> 00:18:54,460 p minus 1, right? 264 00:18:54,460 --> 00:19:01,040 Polynomials of degree p minus 1. 265 00:19:01,040 --> 00:19:05,020 And this guy would be degree q minus 1. 266 00:19:05,020 --> 00:19:07,390 Degree q minus 1. 267 00:19:07,390 --> 00:19:11,200 And when I multiply them, what's the degree? 268 00:19:11,200 --> 00:19:13,230 Just add. 269 00:19:13,230 --> 00:19:15,640 And how many coefficients? 270 00:19:15,640 --> 00:19:17,380 Well, one more I have to remember 271 00:19:17,380 --> 00:19:20,710 for that stupid 0 order term. 272 00:19:20,710 --> 00:19:26,294 So this would have p plus q minus 1 components. 273 00:19:30,493 --> 00:19:31,910 So that would have been the number 274 00:19:31,910 --> 00:19:36,440 that you've somehow had to work out in that first lab. 275 00:19:36,440 --> 00:19:39,620 So that if this had n components and this had n, 276 00:19:39,620 --> 00:19:41,840 this would have 2n minus 1. 277 00:19:41,840 --> 00:19:44,360 It's just what you would have-- like you 278 00:19:44,360 --> 00:19:51,390 say 3 plus x times 1 plus 2x. 279 00:19:51,390 --> 00:19:57,470 In this case, p is 2, q is two, two components, two components. 280 00:19:57,470 --> 00:20:07,160 And if I multiply those, I get 3x and 6x is 7x and 2x squared. 281 00:20:07,160 --> 00:20:18,230 And so I have 2 plus 2 minus 1 equals 3 components. 282 00:20:18,230 --> 00:20:20,210 The constant x and x squared. 283 00:20:20,210 --> 00:20:22,690 Yeah, clear, right. 284 00:20:22,690 --> 00:20:24,590 Yeah, so that's not the-- 285 00:20:24,590 --> 00:20:28,850 that's what I would get if I multiplied these matrices, if I 286 00:20:28,850 --> 00:20:32,030 had a two diagonal matrix, Toeplitz matrix, 287 00:20:32,030 --> 00:20:35,420 times a two diagonal Toeplitz matrix, that would give me 288 00:20:35,420 --> 00:20:39,650 a three diagonal answer. 289 00:20:39,650 --> 00:20:43,370 But if I am doing it periodically, 290 00:20:43,370 --> 00:20:46,040 I would only have two. 291 00:20:46,040 --> 00:20:51,020 That 2x squared would come back if I-- 292 00:20:51,020 --> 00:20:53,420 come back as a 2. 293 00:20:53,420 --> 00:20:56,600 so I just have 5 plus 7x. 294 00:20:56,600 --> 00:20:59,130 Right, good, good, good. 295 00:20:59,130 --> 00:21:01,350 OK. 296 00:21:01,350 --> 00:21:04,740 So that's a reminder of what convolution is. 297 00:21:04,740 --> 00:21:08,830 Cyclic and non-cyclic, vectors and functions. 298 00:21:08,830 --> 00:21:15,150 OK, then eigenvalues and eigenvectors are the next step, 299 00:21:15,150 --> 00:21:17,790 and then the convolution rule is the last step. 300 00:21:17,790 --> 00:21:21,980 So eigenvectors. 301 00:21:21,980 --> 00:21:24,351 Eigenvectors of the circulant. 302 00:21:26,940 --> 00:21:28,858 Of course, I can only do square matrices. 303 00:21:32,880 --> 00:21:35,430 So I'm doing the periodic case. 304 00:21:35,430 --> 00:21:40,380 So the eigenvectors are the columns 305 00:21:40,380 --> 00:21:45,950 of the eigenvector matrix. 306 00:21:45,950 --> 00:21:49,260 And I'm going to call it F for Fourier. 307 00:21:49,260 --> 00:21:54,410 So F is-- the first eigenvector is all 1s. 308 00:21:54,410 --> 00:21:58,620 An x eigenvector is the fourth root of 1, 309 00:21:58,620 --> 00:22:07,380 then the square root of 1, i6, i8, i fourth, i6, 310 00:22:07,380 --> 00:22:12,640 and finally, 1 i cubed i sixth i ninth. 311 00:22:12,640 --> 00:22:21,160 OK, that's F. Those are the four eigenvectors of the permutation 312 00:22:21,160 --> 00:22:25,690 p and of any polynomial in p. 313 00:22:25,690 --> 00:22:33,760 So my circulant is some c0 i plus c 1p plus c 314 00:22:33,760 --> 00:22:37,200 2p squared and c3 pq. 315 00:22:37,200 --> 00:22:38,980 OK. 316 00:22:38,980 --> 00:22:44,110 And finally, this is the step we've been almost ready to do 317 00:22:44,110 --> 00:22:45,520 but didn't quite do. 318 00:22:45,520 --> 00:22:48,900 What are the eigenvectors-- 319 00:22:48,900 --> 00:22:54,995 what eigenvectors are its eigenvectors? 320 00:22:57,850 --> 00:23:02,730 So those are the eigenvectors of p. 321 00:23:02,730 --> 00:23:07,320 And now we have just a combination of p's. 322 00:23:07,320 --> 00:23:12,500 So I think the eigenvectors I just multiply. 323 00:23:12,500 --> 00:23:16,664 I take that same combination of the eigenvectors. 324 00:23:24,050 --> 00:23:27,410 Does that look right? 325 00:23:27,410 --> 00:23:29,180 So sorry. 326 00:23:31,820 --> 00:23:34,450 I'm sorry. 327 00:23:34,450 --> 00:23:39,250 Its eigenvectors, they're the columns of f. 328 00:23:39,250 --> 00:23:47,120 The question I meant to ask is what are its eigenvalues? 329 00:23:51,070 --> 00:23:52,810 That's the key question. 330 00:23:52,810 --> 00:23:54,970 What are the eigenvalues? 331 00:23:54,970 --> 00:23:58,240 And I think that if I just multiply F times c, 332 00:23:58,240 --> 00:24:03,880 I get the eigenvalues of the matrix C. 333 00:24:03,880 --> 00:24:05,530 That's the beauty. 334 00:24:05,530 --> 00:24:06,610 That's the nice formula. 335 00:24:09,280 --> 00:24:14,920 If my matrix is just P alone, then this is 0, 1, 0, 0, 336 00:24:14,920 --> 00:24:18,400 and I get 1, i, i squared, i cubed. 337 00:24:18,400 --> 00:24:22,330 But if c is some other combination of the p's, then I 338 00:24:22,330 --> 00:24:30,130 take the same combination of the eigenvectors to see-- 339 00:24:30,130 --> 00:24:32,920 yeah. 340 00:24:32,920 --> 00:24:34,330 Do you see it? 341 00:24:34,330 --> 00:24:41,080 So I'm claiming that I'll get four eigenvalues of C 342 00:24:41,080 --> 00:24:44,500 from this multiplication. 343 00:24:44,500 --> 00:24:49,140 So of course, if there's only c0, 344 00:24:49,140 --> 00:24:54,430 then I only get c0, c0, c0, c0. 345 00:24:54,430 --> 00:24:57,280 It's four times repeated. 346 00:24:57,280 --> 00:25:02,520 But if it's this combination, then that matrix multiplication 347 00:25:02,520 --> 00:25:04,590 takes the same combination of-- 348 00:25:04,590 --> 00:25:11,950 this is a combination of the eigenvectors. 349 00:25:11,950 --> 00:25:17,120 And that gives us the right thing. 350 00:25:17,120 --> 00:25:17,620 OK. 351 00:25:22,730 --> 00:25:26,660 Now I just have one more step for this convolution rule, 352 00:25:26,660 --> 00:25:27,530 and then I'm happy. 353 00:25:38,370 --> 00:25:44,610 Really, the convolution rule is stating what we-- 354 00:25:44,610 --> 00:25:48,610 it's stating a relation between multiplication, 355 00:25:48,610 --> 00:25:53,520 which we saw here, and the convolution, which 356 00:25:53,520 --> 00:25:56,070 we saw for the coefficients. 357 00:25:56,070 --> 00:26:00,270 So the convolution rule is a connection between multiplying 358 00:26:00,270 --> 00:26:04,150 and convolution. 359 00:26:04,150 --> 00:26:08,180 And so let me say what that convolution rule is 360 00:26:08,180 --> 00:26:10,120 and let me write it correctly. 361 00:26:15,000 --> 00:26:21,060 So here I take a cyclic convolution. 362 00:26:21,060 --> 00:26:22,860 I'm dealing with square matrices. 363 00:26:22,860 --> 00:26:25,200 Everything is cyclic here. 364 00:26:25,200 --> 00:26:28,440 And then I get-- 365 00:26:28,440 --> 00:26:32,190 if I multiply by F, what do I have now? 366 00:26:32,190 --> 00:26:34,500 What does that represent? 367 00:26:34,500 --> 00:26:38,520 This was c and d, and I convolve them. 368 00:26:38,520 --> 00:26:43,290 So I got another circulant matrix. 369 00:26:43,290 --> 00:26:50,784 So up here, the multiplication of matrices is C times D. 370 00:26:50,784 --> 00:26:55,970 I want to connect multiplying those matrices with convolving 371 00:26:55,970 --> 00:26:58,310 the c's. 372 00:26:58,310 --> 00:27:02,570 I want to make that connection. 373 00:27:02,570 --> 00:27:05,580 And that connection is the convolution rule. 374 00:27:08,120 --> 00:27:15,360 So this would be the eigenvalues of CD. 375 00:27:19,160 --> 00:27:21,080 Let's just pause there. 376 00:27:21,080 --> 00:27:23,990 Why am I looking at the eigenvalues of CD? 377 00:27:23,990 --> 00:27:27,150 Because if I do that multiplication, 378 00:27:27,150 --> 00:27:32,390 I get another Toeplitz matrix, C times D. 379 00:27:32,390 --> 00:27:36,740 And the polynomial-- the coefficients associated 380 00:27:36,740 --> 00:27:42,170 on the diagonals of C times D are the coefficients 381 00:27:42,170 --> 00:27:43,650 of the convolution. 382 00:27:43,650 --> 00:28:00,390 So its diagonals come from convolving c with d cyclically. 383 00:28:00,390 --> 00:28:03,050 OK. 384 00:28:03,050 --> 00:28:07,160 Now I want to find the same eigenvalues in a second way 385 00:28:07,160 --> 00:28:07,910 and match-- 386 00:28:07,910 --> 00:28:11,300 and the equation will be the convolution rule. 387 00:28:11,300 --> 00:28:14,720 So how can I find the eigenvalues of CD? 388 00:28:14,720 --> 00:28:18,710 Well, amazingly, they are the eigenvalues 389 00:28:18,710 --> 00:28:34,540 of C times the eigenvalues of D. I'm going 390 00:28:34,540 --> 00:28:36,770 to test this rule on 2 by 2. 391 00:28:36,770 --> 00:28:39,820 So you'll see everything happening. 392 00:28:39,820 --> 00:28:42,290 So this is the main-- this is the fact that I want to use. 393 00:28:46,880 --> 00:28:49,460 Because C and D commute. 394 00:28:49,460 --> 00:28:50,540 C and D commute. 395 00:28:50,540 --> 00:28:52,970 They have the same eigenvectors. 396 00:28:52,970 --> 00:28:56,900 And then the eigenvalues just multiply. 397 00:28:56,900 --> 00:28:58,760 So I can multiply. 398 00:28:58,760 --> 00:29:04,850 I can get that in a second way by taking the eigenvalues of c 399 00:29:04,850 --> 00:29:10,040 and multiplying those by the eigenvalues of d. 400 00:29:10,040 --> 00:29:13,790 And I multiply component by component. 401 00:29:13,790 --> 00:29:18,560 I multiply the eigenvalue for the all 1s vector 402 00:29:18,560 --> 00:29:22,790 by the eigenvalue for the all 1s vector. 403 00:29:22,790 --> 00:29:25,760 Do you know this MATLAB command? 404 00:29:25,760 --> 00:29:28,380 Component by component multiplication? 405 00:29:28,380 --> 00:29:30,690 This is an important one. 406 00:29:30,690 --> 00:29:35,110 There's a guy's name is also associated with that. 407 00:29:35,110 --> 00:29:36,470 So that's a vector. 408 00:29:36,470 --> 00:29:37,400 That's a vector. 409 00:29:37,400 --> 00:29:40,850 And what comes out of that operation? 410 00:29:40,850 --> 00:29:43,710 If I have a vector with three components. 411 00:29:43,710 --> 00:29:45,590 So n is 3 here. 412 00:29:45,590 --> 00:29:49,580 And I do point star or dot star. 413 00:29:49,580 --> 00:29:52,580 I'm not sure what people usually say. 414 00:29:52,580 --> 00:29:55,820 Component by component, a three component vector times a three 415 00:29:55,820 --> 00:30:01,860 component vector, I get a three component vector, 416 00:30:01,860 --> 00:30:03,870 just like that. 417 00:30:03,870 --> 00:30:07,833 So this is the convolution rule. 418 00:30:07,833 --> 00:30:09,000 That's the convolution rule. 419 00:30:12,050 --> 00:30:18,390 And the proof is the fact that when matrices commute, 420 00:30:18,390 --> 00:30:21,750 the eigenvalues of the product are just 421 00:30:21,750 --> 00:30:24,360 these eigenvalues times these eigenvalues, 422 00:30:24,360 --> 00:30:26,520 because they have the same-- 423 00:30:26,520 --> 00:30:29,190 the eigenvectors are always the same 424 00:30:29,190 --> 00:30:31,650 here for all these circulants. 425 00:30:31,650 --> 00:30:36,930 So there's the convolution rule that I can convolve and then 426 00:30:36,930 --> 00:30:38,970 transform. 427 00:30:38,970 --> 00:30:45,490 Or I can transform separately and then multiply. 428 00:30:45,490 --> 00:30:50,700 So I just maybe better right that convolution rule. 429 00:30:50,700 --> 00:30:52,320 Let's call it the C rule. 430 00:30:55,140 --> 00:31:24,650 Convolve then transform by F. Or transform separately by F. 431 00:31:24,650 --> 00:31:34,390 And then multiply point one. 432 00:31:38,120 --> 00:31:39,950 Element by element. 433 00:31:39,950 --> 00:31:41,658 Component by component. 434 00:31:45,250 --> 00:31:47,690 OK. 435 00:31:47,690 --> 00:31:51,870 So that's the convolution rule. 436 00:31:51,870 --> 00:31:56,750 And why is it sort of-- why is it so important? 437 00:31:56,750 --> 00:32:00,500 Because transforming by F, multiplying 438 00:32:00,500 --> 00:32:05,630 by the Fourier matrix, is extremely fast by the FFT. 439 00:32:05,630 --> 00:32:16,760 So it's useful because of the FFT, the Fast Fourier 440 00:32:16,760 --> 00:32:19,640 Transform, to multiply. 441 00:32:26,570 --> 00:32:28,370 Or to transform. 442 00:32:28,370 --> 00:32:29,970 Whichever. 443 00:32:29,970 --> 00:32:32,660 Equal to transform. 444 00:32:32,660 --> 00:32:36,500 Multiply by F transform. 445 00:32:43,370 --> 00:32:48,980 So it's the presence of the FFT that makes this-- 446 00:32:48,980 --> 00:32:51,930 it gives us really two different ways to do it. 447 00:32:51,930 --> 00:32:56,990 In fact, which is the faster way? 448 00:32:56,990 --> 00:33:06,520 So we can produce the same result this way or this way. 449 00:33:09,280 --> 00:33:14,700 And if I don't count the cost of-- if the cost of multiplying 450 00:33:14,700 --> 00:33:20,400 by F is low, because I have the FFT, which would you do? 451 00:33:20,400 --> 00:33:22,260 Which would you do? 452 00:33:22,260 --> 00:33:26,010 So let me just think aloud before we answer that question, 453 00:33:26,010 --> 00:33:27,810 and then we're good. 454 00:33:30,600 --> 00:33:34,430 So my vectors have n components. 455 00:33:34,430 --> 00:33:38,370 So one way I can do is to do convolution. 456 00:33:38,370 --> 00:33:40,920 How many steps is that? 457 00:33:40,920 --> 00:33:43,240 If I take a vector with n components 458 00:33:43,240 --> 00:33:45,870 and I convolve with a vector with n components, 459 00:33:45,870 --> 00:33:50,100 how many little multiplications do I have to do? 460 00:33:50,100 --> 00:33:52,250 N squared, right? 461 00:33:52,250 --> 00:33:55,830 Because each of the c's has to multiply each of the d's. 462 00:33:55,830 --> 00:33:57,210 So that takes N squared. 463 00:33:59,940 --> 00:34:02,800 And Fourier is cheap. 464 00:34:02,800 --> 00:34:10,881 It's N log N. Log to base 2. 465 00:34:10,881 --> 00:34:16,090 So the left hand side is effectively N cubed. 466 00:34:16,090 --> 00:34:19,179 What about this one? 467 00:34:19,179 --> 00:34:23,050 How many to do these two guys? 468 00:34:25,920 --> 00:34:32,690 To find the Fourier transform to multiply by the matrix F. OK, 469 00:34:32,690 --> 00:34:33,850 those are fast again. 470 00:34:33,850 --> 00:34:36,889 That's just I've got two multiplications by F. 471 00:34:36,889 --> 00:34:43,654 So that's 2 N log N. And what's the cost of this? 472 00:34:47,340 --> 00:34:50,850 I have a vector with n components. 473 00:34:50,850 --> 00:34:52,199 Dot star vector. 474 00:34:52,199 --> 00:34:53,889 Another vector with n components. 475 00:34:53,889 --> 00:34:55,650 How many little multiplications do I 476 00:34:55,650 --> 00:35:00,420 have to do for a Hadamard product or a component 477 00:35:00,420 --> 00:35:02,490 by component product? 478 00:35:02,490 --> 00:35:03,750 N, only n. 479 00:35:06,770 --> 00:35:13,100 Plus N. Yeah, maybe I should have made that plus. 480 00:35:13,100 --> 00:35:14,280 I had two. 481 00:35:14,280 --> 00:35:18,080 No, I had one N log N. Plus it took 482 00:35:18,080 --> 00:35:20,280 N squared to find that vector. 483 00:35:20,280 --> 00:35:25,700 And then N log N. So it's effectively N squared. 484 00:35:25,700 --> 00:35:32,300 But this one where I do the N log N twice and then it 485 00:35:32,300 --> 00:35:34,160 only takes me N more. 486 00:35:34,160 --> 00:35:35,600 So this is the fast way. 487 00:35:41,260 --> 00:35:44,020 So if you wanted to multiply two really big, 488 00:35:44,020 --> 00:35:50,860 long integers, as you would want to do in cryptography, if you 489 00:35:50,860 --> 00:35:58,360 had two long integers, say, of length 125, 126, 128 490 00:35:58,360 --> 00:36:02,170 components, to multiply those, you 491 00:36:02,170 --> 00:36:05,760 would be better off to separately take 492 00:36:05,760 --> 00:36:11,770 the cyclic transform of each of those 128 guys 493 00:36:11,770 --> 00:36:15,040 and do it this way. 494 00:36:17,590 --> 00:36:22,180 Take the transforms, do the component by component product, 495 00:36:22,180 --> 00:36:24,650 and then transform back to get that. 496 00:36:28,202 --> 00:36:31,140 The convolution rule is what makes that go. 497 00:36:35,184 --> 00:36:38,190 Oh, one more thought, I guess, about all 498 00:36:38,190 --> 00:36:40,470 this convolution stuff. 499 00:36:40,470 --> 00:36:43,140 Suppose we're in 2D. 500 00:36:43,140 --> 00:36:48,060 We have to think what is a two dimensional convolution? 501 00:36:48,060 --> 00:36:52,320 What does this become in two dimensions? 502 00:36:52,320 --> 00:36:54,400 Suppose we have functions. 503 00:36:54,400 --> 00:37:01,020 So now I'm gonna do 2D functions of x and y. 504 00:37:01,020 --> 00:37:03,750 Periodic or not periodic. 505 00:37:03,750 --> 00:37:07,050 But what's a convolution? 506 00:37:07,050 --> 00:37:11,040 What's the operation we have to do in two dimensions? 507 00:37:11,040 --> 00:37:14,490 Well, it's a double integral, of course. 508 00:37:14,490 --> 00:37:15,990 t and u. 509 00:37:15,990 --> 00:37:29,025 We would do f of t and u times g of x minus ty minus u dtdu. 510 00:37:33,710 --> 00:37:36,510 And that would produce a function. 511 00:37:36,510 --> 00:37:40,920 So I'm convolving a function of x and y 512 00:37:40,920 --> 00:37:43,410 with another function of x and y. 513 00:37:43,410 --> 00:37:47,016 And again, I'm looking for this. 514 00:37:47,016 --> 00:37:49,230 This is the key to watch for. 515 00:37:49,230 --> 00:37:50,850 x minus t. 516 00:37:50,850 --> 00:37:52,650 y minus u. 517 00:37:52,650 --> 00:37:57,420 That's the signal of a convolution integral. 518 00:37:57,420 --> 00:38:01,750 So that's what we would have in 2D. 519 00:38:01,750 --> 00:38:06,700 In general, So maybe now my final thought 520 00:38:06,700 --> 00:38:13,450 is to move to think about two dimensional matrices 521 00:38:13,450 --> 00:38:15,530 and their products and so on. 522 00:38:15,530 --> 00:38:17,080 And this is why you need them. 523 00:38:17,080 --> 00:38:21,190 Because if you have two dimensional signals, 524 00:38:21,190 --> 00:38:25,780 then the components fit into a matrix. 525 00:38:25,780 --> 00:38:30,960 And we just have to operate in both dimensions. 526 00:38:33,710 --> 00:38:46,545 So the key operation in 2D is in MATLAB. 527 00:38:49,680 --> 00:38:55,980 The MATLAB command that you need to know to get-- 528 00:38:55,980 --> 00:38:59,430 if you know what you're doing in 1D and you want to do it in 2D, 529 00:38:59,430 --> 00:39:03,580 the MATLAB command is Kron. 530 00:39:03,580 --> 00:39:12,250 So imagine we have one dimensional matrices A and B. 531 00:39:12,250 --> 00:39:19,570 And so those are in 1D, and we want to produce a natural two 532 00:39:19,570 --> 00:39:20,660 dimensional matrix. 533 00:39:20,660 --> 00:39:27,770 So these will be N by N. And we want 534 00:39:27,770 --> 00:39:31,060 the sort of natural product, let me call it 535 00:39:31,060 --> 00:39:35,080 K for Kron, which will be N squared by N squared. 536 00:39:37,710 --> 00:39:47,860 I want to create a 2D matrix connected to an image that's 537 00:39:47,860 --> 00:39:50,030 N in each direction. 538 00:39:50,030 --> 00:39:53,720 So it has N squared pixels. 539 00:39:53,720 --> 00:39:59,180 These are 1D signals, and K is a 2D one. 540 00:39:59,180 --> 00:40:03,950 And this K would be the-- 541 00:40:03,950 --> 00:40:06,938 this is the operation to know. 542 00:40:06,938 --> 00:40:13,520 Given two one dimensional n by n matrices, 543 00:40:13,520 --> 00:40:19,040 Kron produces an N squared by N squared matrix. 544 00:40:19,040 --> 00:40:21,560 It's the operation to know. 545 00:40:21,560 --> 00:40:25,220 So I'll just write it, and if you know what Kron is, 546 00:40:25,220 --> 00:40:29,630 then you know it before I write it. 547 00:40:29,630 --> 00:40:33,860 So I want to produce a big matrix, N squared by N squared. 548 00:40:37,450 --> 00:40:41,860 Somehow appropriately multiplying these two guys. 549 00:40:41,860 --> 00:40:48,805 And the appropriate way to do it is to take a11 and multiply it 550 00:40:48,805 --> 00:40:52,240 by B. So there, what do I have? 551 00:40:52,240 --> 00:40:55,030 What size have I got there already 552 00:40:55,030 --> 00:40:57,850 just in that one corner? 553 00:40:57,850 --> 00:40:59,530 N by N, right? 554 00:40:59,530 --> 00:41:03,100 It's a number of times an N by N matrix. 555 00:41:03,100 --> 00:41:07,900 Then a12 times B. That's another N by N matrix. 556 00:41:07,900 --> 00:41:14,530 Up to a1N times B. So I have now-- 557 00:41:14,530 --> 00:41:18,010 sorry, cap N. 558 00:41:18,010 --> 00:41:22,720 So I have cap N matrices in a row. 559 00:41:22,720 --> 00:41:25,150 Each of those matrices is N by N. 560 00:41:25,150 --> 00:41:27,790 So that row has length n squared. 561 00:41:27,790 --> 00:41:31,105 And of course, the next row is-- 562 00:41:37,330 --> 00:41:41,420 I've allowed myself to number from 1 to N, 563 00:41:41,420 --> 00:41:46,210 but very often that numbering should be 0 to n minus 1. 564 00:41:46,210 --> 00:41:59,710 And finally on down here down to an1 B to aNN B. So so 565 00:41:59,710 --> 00:42:01,633 that's the N squared by N squared matrix 566 00:42:01,633 --> 00:42:02,800 that you would need to know. 567 00:42:02,800 --> 00:42:07,990 For example, if you wanted to do a two dimensional Fourier 568 00:42:07,990 --> 00:42:11,245 transform, that would be-- 569 00:42:15,130 --> 00:42:19,420 yeah, so what would a two dimensional Fourier transform 570 00:42:19,420 --> 00:42:20,120 produce? 571 00:42:20,120 --> 00:42:21,340 What matrix? 572 00:42:21,340 --> 00:42:24,190 Is this the matrix you would use for a 2D? 573 00:42:30,280 --> 00:42:33,220 I haven't sort of got started properly on 2D 574 00:42:33,220 --> 00:42:34,720 Fourier transforms. 575 00:42:34,720 --> 00:42:36,760 So would it be F times F? 576 00:42:39,520 --> 00:42:43,330 So let me write down the full name of this guy. 577 00:42:43,330 --> 00:42:44,350 Kronecker. 578 00:42:44,350 --> 00:42:45,925 So it's called the Kronecker product. 579 00:42:51,500 --> 00:42:53,450 It's just the right thing to know 580 00:42:53,450 --> 00:42:57,920 in moving from one dimension to two dimensions. 581 00:42:57,920 --> 00:42:59,760 For example. 582 00:42:59,760 --> 00:43:01,390 Let me do an example. 583 00:43:07,708 --> 00:43:09,990 Oops, that's full. 584 00:43:09,990 --> 00:43:11,900 Have I got one board left? 585 00:43:11,900 --> 00:43:12,400 Yeah. 586 00:43:16,770 --> 00:43:20,070 So here's a standard matrix. 587 00:43:20,070 --> 00:43:25,410 Call it A. 2s and minus 1s. 588 00:43:25,410 --> 00:43:32,010 So that corresponds to a second derivative 589 00:43:32,010 --> 00:43:35,350 or actually minus a second derivative. 590 00:43:35,350 --> 00:43:39,210 Now, suppose I have another, the same matrix, 591 00:43:39,210 --> 00:43:47,086 corresponding to second derivatives in the y direction. 592 00:43:47,086 --> 00:43:47,586 Same. 593 00:43:50,420 --> 00:43:53,150 And what I really want to do is both. 594 00:43:53,150 --> 00:43:58,040 I want to have a matrix K that corresponds 595 00:43:58,040 --> 00:44:03,380 to minus the second in the x direction 596 00:44:03,380 --> 00:44:07,430 minus the second in the y. 597 00:44:07,430 --> 00:44:10,330 So this is the Laplace. 598 00:44:10,330 --> 00:44:10,950 Laplacian. 599 00:44:14,840 --> 00:44:18,930 Which is all over the place in differential equations. 600 00:44:18,930 --> 00:44:23,300 At a typical point, I want to do minus one 601 00:44:23,300 --> 00:44:27,110 of these, two of these minus one of those in the x direction, 602 00:44:27,110 --> 00:44:29,710 and I want to add to that minus 1. 603 00:44:29,710 --> 00:44:34,550 Now that 2 becomes a 4 and minus 1 in the y direction. 604 00:44:34,550 --> 00:44:37,970 So I'm looking for the 2 by 2 matrix-- 605 00:44:37,970 --> 00:44:43,080 sorry, the two dimensional matrix that takes-- 606 00:44:43,080 --> 00:44:45,800 that does that five point scheme. 607 00:44:45,800 --> 00:44:49,520 Five weights at each point. 608 00:44:49,520 --> 00:44:51,350 It takes four of the-- 609 00:44:51,350 --> 00:44:56,690 on the diagonal and minus 1 on the four neighbors. 610 00:44:56,690 --> 00:45:01,280 And the operation that would do that would be you 611 00:45:01,280 --> 00:45:03,230 would use Kron. 612 00:45:03,230 --> 00:45:07,700 It wouldn't be Kron of A B. That would just-- 613 00:45:07,700 --> 00:45:10,090 K of A B is not what I-- 614 00:45:10,090 --> 00:45:12,470 a Kron of A B is not what I want. 615 00:45:17,390 --> 00:45:19,700 Yeah, that would do one and then the other one. 616 00:45:19,700 --> 00:45:24,650 And then that would probably produce nine non-zeroes. 617 00:45:24,650 --> 00:45:26,690 I want something that adds here. 618 00:45:26,690 --> 00:45:32,060 So I want Kron of A times the identity. 619 00:45:32,060 --> 00:45:36,650 That gives me the two dimensional thing 620 00:45:36,650 --> 00:45:38,390 for this part. 621 00:45:38,390 --> 00:45:50,280 And then I'll add on Kron of I B for the vertical derivative, 622 00:45:50,280 --> 00:45:54,350 the derivatives in the y direction. 623 00:45:54,350 --> 00:45:56,180 So that's called a Kronecker sum. 624 00:45:59,600 --> 00:46:02,420 The other was a Kronecker product. 625 00:46:02,420 --> 00:46:04,400 So that would be a Kronecker product. 626 00:46:04,400 --> 00:46:06,290 This would be another Kronecker product, 627 00:46:06,290 --> 00:46:09,290 and the total is called the Kronecker sum. 628 00:46:11,820 --> 00:46:14,110 OK. 629 00:46:14,110 --> 00:46:17,380 I wanted just to get those notations out. 630 00:46:17,380 --> 00:46:21,120 Because really, Fourier transforming 631 00:46:21,120 --> 00:46:27,590 is such a central operation in all of applied math, 632 00:46:27,590 --> 00:46:30,500 and especially in signal processing. 633 00:46:30,500 --> 00:46:33,180 OK, so I'm good for it today. 634 00:46:33,180 --> 00:46:33,680 Let's see. 635 00:46:33,680 --> 00:46:39,080 I've got one volunteer so far to talk about a project. 636 00:46:39,080 --> 00:46:43,730 Can I encourage an email from anybody that doesn't-- you 637 00:46:43,730 --> 00:46:46,950 don't have to be a superstar. 638 00:46:46,950 --> 00:46:52,130 You're just willing to do it. 639 00:46:52,130 --> 00:46:54,020 Tell us something about what you've learned. 640 00:46:54,020 --> 00:46:56,210 Get comments from the audience. 641 00:46:56,210 --> 00:47:02,960 And 10 or 15 minutes is all I'm thinking about. 642 00:47:02,960 --> 00:47:07,520 OK, I'll let you send me an email if you'd like to tell us 643 00:47:07,520 --> 00:47:09,530 that and get some feedback. 644 00:47:09,530 --> 00:47:11,240 OK, good. 645 00:47:11,240 --> 00:47:12,590 So I'll see you Wednesday. 646 00:47:15,590 --> 00:47:17,140 Thanks.