1 00:00:00,000 --> 00:00:03,890 2 00:00:03,890 --> 00:00:09,130 Yes, OK, four, three, two, one, OK, I 3 00:00:09,130 --> 00:00:10,880 see you guys are in a happy mood. 4 00:00:10,880 --> 00:00:13,980 I don't know if that means 18.06 is ending, 5 00:00:13,980 --> 00:00:16,680 or, the quiz was good. 6 00:00:16,680 --> 00:00:21,030 Uh, my birthday conference was going 7 00:00:21,030 --> 00:00:24,880 on at the time of the quiz, and in the conference, of course, 8 00:00:24,880 --> 00:00:27,190 everybody had to say nice things, 9 00:00:27,190 --> 00:00:30,390 but I was wondering, what would my 18.06 10 00:00:30,390 --> 00:00:36,200 class be saying, because it was at the exactly the same time. 11 00:00:36,200 --> 00:00:39,300 But, what I know from the grades so far, 12 00:00:39,300 --> 00:00:45,440 they're basically close to, and maybe slightly above the grades 13 00:00:45,440 --> 00:00:48,590 that you got on quiz two. 14 00:00:48,590 --> 00:00:52,790 So, very satisfactory. 15 00:00:52,790 --> 00:00:56,550 And, then we have a final exam coming up, 16 00:00:56,550 --> 00:01:00,780 and today's lecture, as I told you by email, 17 00:01:00,780 --> 00:01:05,019 will be a first step in the review, 18 00:01:05,019 --> 00:01:07,890 and then on Wednesday I'll do all I can 19 00:01:07,890 --> 00:01:13,560 in reviewing the whole course. 20 00:01:13,560 --> 00:01:16,300 So my topic today is -- actually, 21 00:01:16,300 --> 00:01:24,740 this is a lecture I have never given before in this way, 22 00:01:24,740 --> 00:01:28,310 and it will -- well, four subspaces, 23 00:01:28,310 --> 00:01:32,190 that's certainly fundamental, and you know that, 24 00:01:32,190 --> 00:01:34,810 so I want to speak about left-inverses 25 00:01:34,810 --> 00:01:37,150 and right-inverses and then something called 26 00:01:37,150 --> 00:01:39,190 pseudo-inverses. 27 00:01:39,190 --> 00:01:43,110 And pseudo-inverses, let me say right away, 28 00:01:43,110 --> 00:01:47,240 that comes in near the end of chapter seven, 29 00:01:47,240 --> 00:01:52,170 and that would not be expected on the final. 30 00:01:52,170 --> 00:01:54,840 But you'll see that what I'm talking about 31 00:01:54,840 --> 00:02:01,600 is really the basic stuff that, for an m-by-n matrix of rank r, 32 00:02:01,600 --> 00:02:06,080 we're going back to the most fundamental picture in linear 33 00:02:06,080 --> 00:02:07,130 algebra. 34 00:02:07,130 --> 00:02:12,020 Nobody could forget that picture, right? 35 00:02:12,020 --> 00:02:15,890 When you're my age, even, you'll remember the row space, 36 00:02:15,890 --> 00:02:17,560 and the null space. 37 00:02:17,560 --> 00:02:21,130 Orthogonal complements over there, the column space 38 00:02:21,130 --> 00:02:23,530 and the null space of A transpose column, 39 00:02:23,530 --> 00:02:26,190 orthogonal complements over here. 40 00:02:26,190 --> 00:02:29,220 And I want to speak about inverses. 41 00:02:29,220 --> 00:02:30,320 OK. 42 00:02:30,320 --> 00:02:33,960 And I want to identify the different possibilities. 43 00:02:33,960 --> 00:02:37,980 So first of all, when does a matrix 44 00:02:37,980 --> 00:02:42,440 have a just a perfect inverse, two-sided, you know, 45 00:02:42,440 --> 00:02:51,600 so the two-sided inverse is what we just call inverse, right? 46 00:02:51,600 --> 00:02:57,720 And, so that means that there's a matrix that 47 00:02:57,720 --> 00:03:02,010 produces the identity, whether we write it on the left 48 00:03:02,010 --> 00:03:03,670 or on the right. 49 00:03:03,670 --> 00:03:10,170 And just tell me, how are the numbers r, 50 00:03:10,170 --> 00:03:16,580 the rank, n the number of columns, m the number of rows, 51 00:03:16,580 --> 00:03:19,070 how are those numbers related when 52 00:03:19,070 --> 00:03:21,320 we have an invertible matrix? 53 00:03:21,320 --> 00:03:23,690 So this is the matrix which was -- 54 00:03:23,690 --> 00:03:26,540 chapter two was all about matrices like this, 55 00:03:26,540 --> 00:03:30,420 the beginning of the course, what was the relation of th- 56 00:03:30,420 --> 00:03:36,110 of r, m, and n, for the nice case? 57 00:03:36,110 --> 00:03:39,360 They're all the same, all equal. 58 00:03:39,360 --> 00:03:43,260 So this is the case when r=m=n. 59 00:03:43,260 --> 00:03:46,910 Square matrix, full rank, period, just -- 60 00:03:46,910 --> 00:03:50,480 so I'll use the words full rank. 61 00:03:50,480 --> 00:03:51,840 OK, good. 62 00:03:51,840 --> 00:03:53,800 Everybody knows that. 63 00:03:53,800 --> 00:03:55,860 OK. 64 00:03:55,860 --> 00:03:56,875 Then chapter three. 65 00:03:56,875 --> 00:03:59,660 66 00:03:59,660 --> 00:04:03,240 We began to deal with matrices that were not of full rank, 67 00:04:03,240 --> 00:04:07,050 and they could have any rank, and we learned what the rank 68 00:04:07,050 --> 00:04:08,480 was. 69 00:04:08,480 --> 00:04:12,140 And then we focused, if you remember 70 00:04:12,140 --> 00:04:17,110 on some cases like full column rank. 71 00:04:17,110 --> 00:04:20,810 Now, can you remember what was the deal with full column rank? 72 00:04:20,810 --> 00:04:25,020 So, now, I think this is the case in which we 73 00:04:25,020 --> 00:04:28,650 have a left-inverse, and I'll try to find it. 74 00:04:28,650 --> 00:04:32,930 75 00:04:32,930 --> 00:04:35,130 So we have a -- 76 00:04:35,130 --> 00:04:37,800 what was the situation there? 77 00:04:37,800 --> 00:04:43,780 It's the case of full column rank, and that means -- 78 00:04:43,780 --> 00:04:47,160 what does that mean about r? 79 00:04:47,160 --> 00:04:51,770 It equals, what's the deal with r, now, 80 00:04:51,770 --> 00:04:54,460 if we have full column rank, I mean 81 00:04:54,460 --> 00:04:59,730 the columns are independent, but maybe not the rows. 82 00:04:59,730 --> 00:05:03,780 So what is r equal to in this case? 83 00:05:03,780 --> 00:05:04,450 n. 84 00:05:04,450 --> 00:05:05,780 Thanks. 85 00:05:05,780 --> 00:05:06,480 n. 86 00:05:06,480 --> 00:05:06,979 r=n. 87 00:05:06,979 --> 00:05:10,270 The n columns are independent, but probably, we 88 00:05:10,270 --> 00:05:12,920 have more rows. 89 00:05:12,920 --> 00:05:17,060 What's the picture, and then what's the null space for this? 90 00:05:17,060 --> 00:05:19,990 So the n columns are independent, 91 00:05:19,990 --> 00:05:22,155 what's the null space in this case? 92 00:05:22,155 --> 00:05:25,420 93 00:05:25,420 --> 00:05:27,450 So of course, you know what I'm asking. 94 00:05:27,450 --> 00:05:30,690 You're saying, why is this guy asking something, I know that-- 95 00:05:30,690 --> 00:05:32,680 I think about it in my sleep, 96 00:05:32,680 --> 00:05:33,260 right? 97 00:05:33,260 --> 00:05:36,900 So the null space of this matrix if the rank is 98 00:05:36,900 --> 00:05:47,280 n, the null space is what vectors are in the null space? 99 00:05:47,280 --> 00:05:48,410 Just the zero vector. 100 00:05:48,410 --> 00:05:51,320 101 00:05:51,320 --> 00:05:51,820 Right? 102 00:05:51,820 --> 00:05:53,360 The columns are independent. 103 00:05:53,360 --> 00:05:54,300 Independent columns. 104 00:05:54,300 --> 00:05:59,020 105 00:05:59,020 --> 00:06:03,320 No combination of the columns gives zero except that one. 106 00:06:03,320 --> 00:06:06,270 And what's my picture over, -- 107 00:06:06,270 --> 00:06:10,050 let me redraw my picture -- 108 00:06:10,050 --> 00:06:14,330 the row space is everything. 109 00:06:14,330 --> 00:06:18,450 110 00:06:18,450 --> 00:06:20,860 No. 111 00:06:20,860 --> 00:06:23,030 Is that right? 112 00:06:23,030 --> 00:06:26,580 Let's see, I often get these turned around, right? 113 00:06:26,580 --> 00:06:31,640 So what's the deal? 114 00:06:31,640 --> 00:06:34,810 The columns are independent, right? 115 00:06:34,810 --> 00:06:39,330 So the rank should be the full number of columns, so what 116 00:06:39,330 --> 00:06:41,250 does that tell us? 117 00:06:41,250 --> 00:06:43,051 There's no null space, right. 118 00:06:43,051 --> 00:06:43,550 OK. 119 00:06:43,550 --> 00:06:45,400 The row space is the whole thing. 120 00:06:45,400 --> 00:06:48,510 Yes, I won't even draw the picture. 121 00:06:48,510 --> 00:06:51,640 And what was the deal with -- 122 00:06:51,640 --> 00:06:56,960 and these were very important in least squares problems 123 00:06:56,960 --> 00:06:58,750 because -- 124 00:06:58,750 --> 00:07:06,230 So, what more is true here? 125 00:07:06,230 --> 00:07:09,460 If we have full column rank, the null space is zero, 126 00:07:09,460 --> 00:07:14,640 we have independent columns, the unique -- 127 00:07:14,640 --> 00:07:22,990 so we have zero or one solutions to Ax=b. 128 00:07:22,990 --> 00:07:25,790 129 00:07:25,790 --> 00:07:28,660 There may not be any solutions, but if there's a solution, 130 00:07:28,660 --> 00:07:32,910 there's only one solution because other solutions are 131 00:07:32,910 --> 00:07:35,470 found by adding on stuff from the null space, 132 00:07:35,470 --> 00:07:39,490 and there's nobody there to add on. 133 00:07:39,490 --> 00:07:43,250 So the particular solution is the solution, 134 00:07:43,250 --> 00:07:45,860 if there is a particular solution. 135 00:07:45,860 --> 00:07:49,030 But of course, the rows might not be - 136 00:07:49,030 --> 00:07:51,990 are probably not independent -- and therefore, 137 00:07:51,990 --> 00:07:56,880 so right-hand sides won't end up with a zero equal zero after 138 00:07:56,880 --> 00:08:00,990 elimination, so sometimes we may have no solution, 139 00:08:00,990 --> 00:08:02,710 or one solution. 140 00:08:02,710 --> 00:08:03,380 OK. 141 00:08:03,380 --> 00:08:10,350 And what I want to say is that for this matrix A -- 142 00:08:10,350 --> 00:08:13,910 oh, yes, tell me something about A transpose A in this case. 143 00:08:13,910 --> 00:08:20,240 So this whole part of the board, now, is devoted to this case. 144 00:08:20,240 --> 00:08:23,690 What's the deal with A transpose A? 145 00:08:23,690 --> 00:08:27,280 I've emphasized over and over how important that combination 146 00:08:27,280 --> 00:08:32,120 is, for a rectangular matrix, A transpose A 147 00:08:32,120 --> 00:08:37,120 is the good thing to look at, and if the rank is n, 148 00:08:37,120 --> 00:08:39,750 if the null space has only zero in it, 149 00:08:39,750 --> 00:08:43,340 then the same is true of A transpose A. 150 00:08:43,340 --> 00:08:49,250 That's the beautiful fact, that if the rank of A is n, well, 151 00:08:49,250 --> 00:08:52,300 we know this will be an n by n symmetric matrix, 152 00:08:52,300 --> 00:08:53,820 and it will be full rank. 153 00:08:53,820 --> 00:08:55,230 So this is invertible. 154 00:08:55,230 --> 00:08:58,450 This matrix is invertible. 155 00:08:58,450 --> 00:08:59,960 That matrix is invertible. 156 00:08:59,960 --> 00:09:04,070 And now I want to show you that A itself has 157 00:09:04,070 --> 00:09:06,880 a one-sided inverse. 158 00:09:06,880 --> 00:09:08,450 Here it is. 159 00:09:08,450 --> 00:09:17,190 The inverse of that, which exists, times A transpose, 160 00:09:17,190 --> 00:09:20,390 there is a one-sided -- shall I call it A inverse? 161 00:09:20,390 --> 00:09:24,160 -- left of the matrix A. 162 00:09:24,160 --> 00:09:28,680 Why do I say that? 163 00:09:28,680 --> 00:09:35,740 Because if I multiply this guy by A, what do I get? 164 00:09:35,740 --> 00:09:37,630 What does that multiplication give? 165 00:09:37,630 --> 00:09:40,360 Of course, you know it instantly, 166 00:09:40,360 --> 00:09:44,110 because I just put the parentheses there, 167 00:09:44,110 --> 00:09:47,000 I have A transpose A inverse times A transpose A 168 00:09:47,000 --> 00:09:49,090 so, of course, it's the identity. 169 00:09:49,090 --> 00:09:52,120 So it's a left inverse. 170 00:09:52,120 --> 00:09:58,830 And this was the totally crucial case for least squares, 171 00:09:58,830 --> 00:10:02,920 because you remember that least squares, the central equation 172 00:10:02,920 --> 00:10:06,230 of least squares had this matrix, A transpose A, 173 00:10:06,230 --> 00:10:08,920 as its coefficient matrix. 174 00:10:08,920 --> 00:10:11,770 And in the case of full column rank, 175 00:10:11,770 --> 00:10:15,910 that matrix is invertible, and we're go. 176 00:10:15,910 --> 00:10:19,830 So that's the case where there is a left-inverse. 177 00:10:19,830 --> 00:10:26,140 So A does whatever it does, we can find a matrix that 178 00:10:26,140 --> 00:10:30,530 brings it back to the identity. 179 00:10:30,530 --> 00:10:33,710 Now, is it true that, in the other order -- 180 00:10:33,710 --> 00:10:36,830 so A inverse left times A is the identity. 181 00:10:36,830 --> 00:10:42,131 182 00:10:42,131 --> 00:10:42,630 Right? 183 00:10:42,630 --> 00:10:46,100 This matrix is m by n. 184 00:10:46,100 --> 00:10:49,580 This matrix is n by m. 185 00:10:49,580 --> 00:10:52,060 The identity matrix is n by n. 186 00:10:52,060 --> 00:10:53,560 All good. 187 00:10:53,560 --> 00:10:56,300 All good if you're n. 188 00:10:56,300 --> 00:11:02,850 But if you try to put that matrix on the other side, 189 00:11:02,850 --> 00:11:05,280 it would fail. 190 00:11:05,280 --> 00:11:12,240 If the full column rank -- if this is smaller than m, 191 00:11:12,240 --> 00:11:14,480 the case where they're equals is the beautiful case, 192 00:11:14,480 --> 00:11:16,270 but that's all set. 193 00:11:16,270 --> 00:11:18,100 Now, we're looking at the case where 194 00:11:18,100 --> 00:11:21,440 the columns are independent but the rows are not. 195 00:11:21,440 --> 00:11:25,370 So this is invertible, but what matrix is not 196 00:11:25,370 --> 00:11:26,670 invertible? 197 00:11:26,670 --> 00:11:30,890 A A transpose is bad for this case. 198 00:11:30,890 --> 00:11:32,570 A transpose A is good. 199 00:11:32,570 --> 00:11:35,710 So we can multiply on the left, everything good, 200 00:11:35,710 --> 00:11:39,130 we get the left inverse. 201 00:11:39,130 --> 00:11:42,140 But it would not be a two-sided inverse. 202 00:11:42,140 --> 00:11:46,680 A rectangular matrix can't have a two-sided inverse, 203 00:11:46,680 --> 00:11:50,900 because there's got to be some null space, right? 204 00:11:50,900 --> 00:11:53,840 If I have a matrix that's rectangular, 205 00:11:53,840 --> 00:11:58,920 then either that matrix or its transpose 206 00:11:58,920 --> 00:12:02,340 has some null space, because if n and m are different, 207 00:12:02,340 --> 00:12:06,440 then there's going to be some free variables around, 208 00:12:06,440 --> 00:12:09,380 and we'll have some null space in that direction. 209 00:12:09,380 --> 00:12:17,070 OK, tell me the corresponding picture for the opposite case. 210 00:12:17,070 --> 00:12:20,840 So now I'm going to ask you about right-inverses. 211 00:12:20,840 --> 00:12:22,325 A right-inverse. 212 00:12:22,325 --> 00:12:26,010 213 00:12:26,010 --> 00:12:28,970 And you can fill this all out, this 214 00:12:28,970 --> 00:12:31,410 is going to be the case of full row rank. 215 00:12:31,410 --> 00:12:34,420 216 00:12:34,420 --> 00:12:41,870 And then r is equal to m, now, the m rows are independent, 217 00:12:41,870 --> 00:12:45,270 but the columns are not. 218 00:12:45,270 --> 00:12:47,000 So what's the deal on that? 219 00:12:47,000 --> 00:12:50,640 Well, just exactly the flip of this one. 220 00:12:50,640 --> 00:12:57,400 The null space of A transpose contains only zero, 221 00:12:57,400 --> 00:13:00,480 because there are no combinations of the rows that 222 00:13:00,480 --> 00:13:02,390 give the zero row. 223 00:13:02,390 --> 00:13:03,750 We have independent rows. 224 00:13:03,750 --> 00:13:08,860 225 00:13:08,860 --> 00:13:13,660 And in a minute, I'll give an example of all these. 226 00:13:13,660 --> 00:13:17,545 So, how many solutions to Ax=b in this case? 227 00:13:17,545 --> 00:13:23,159 228 00:13:23,159 --> 00:13:24,200 The rows are independent. 229 00:13:24,200 --> 00:13:27,050 230 00:13:27,050 --> 00:13:30,980 So we can always solve Ax=b. 231 00:13:30,980 --> 00:13:34,540 Whenever elimination never produces a zero row, 232 00:13:34,540 --> 00:13:38,290 so we never get into that zero equal one problem, 233 00:13:38,290 --> 00:13:43,470 so Ax=b always has a solution, but too many. 234 00:13:43,470 --> 00:13:50,140 So there will be some null space, the null space of A -- 235 00:13:50,140 --> 00:13:54,460 what will be the dimension of A's null space? 236 00:13:54,460 --> 00:13:58,910 How many free variables have we got? 237 00:13:58,910 --> 00:14:03,410 How many special solutions in that null space have we got? 238 00:14:03,410 --> 00:14:06,090 So how many free variables in this setup? 239 00:14:06,090 --> 00:14:12,680 We've got n columns, so n variables, 240 00:14:12,680 --> 00:14:16,530 and this tells us how many are pivot 241 00:14:16,530 --> 00:14:19,170 variables, that tells us how many pivots there are, 242 00:14:19,170 --> 00:14:21,930 so there are n-m free variables. 243 00:14:21,930 --> 00:14:27,940 So there are infinitely many solutions to Ax=b. 244 00:14:27,940 --> 00:14:37,220 We have n-m free variables in this case. 245 00:14:37,220 --> 00:14:38,240 OK. 246 00:14:38,240 --> 00:14:45,600 Now I wanted to ask about this idea of a right-inverse. 247 00:14:45,600 --> 00:14:46,720 OK. 248 00:14:46,720 --> 00:14:52,710 So I'm going to have a matrix A, my matrix A, and now 249 00:14:52,710 --> 00:14:54,960 there's going to be some inverse on the right that 250 00:14:54,960 --> 00:14:57,710 will give the identity matrix. 251 00:14:57,710 --> 00:15:05,500 So it will be A times A inverse on the right, will be I. 252 00:15:05,500 --> 00:15:12,290 And can you tell me what, just by comparing 253 00:15:12,290 --> 00:15:18,560 with what we had up there, what will be the right-inverse, 254 00:15:18,560 --> 00:15:21,150 we even have a formula for it. 255 00:15:21,150 --> 00:15:22,760 There will be other -- 256 00:15:22,760 --> 00:15:24,820 actually, there are other left-inverses, 257 00:15:24,820 --> 00:15:26,610 that's our favorite. 258 00:15:26,610 --> 00:15:28,350 There will be other right-inverses, 259 00:15:28,350 --> 00:15:31,620 but tell me our favorite here, what's the nice right-inverse? 260 00:15:31,620 --> 00:15:35,240 261 00:15:35,240 --> 00:15:39,470 The nice right-inverse will be, well, there we 262 00:15:39,470 --> 00:15:43,820 had A transpose A was good, now it 263 00:15:43,820 --> 00:15:46,860 will be A A transpose that's good. 264 00:15:46,860 --> 00:15:49,680 The good matrix, the good right -- 265 00:15:49,680 --> 00:15:53,200 the thing we can invert is A A transpose, 266 00:15:53,200 --> 00:16:00,540 so now if I just do it that way, there sits the right-inverse. 267 00:16:00,540 --> 00:16:03,930 You see how completely parallel it is to the one above? 268 00:16:03,930 --> 00:16:11,220 269 00:16:11,220 --> 00:16:11,964 Right. 270 00:16:11,964 --> 00:16:13,130 So that's the right-inverse. 271 00:16:13,130 --> 00:16:20,960 So that's the case when there is -- 272 00:16:20,960 --> 00:16:25,390 In terms of this picture, tell me 273 00:16:25,390 --> 00:16:29,770 what the null spaces are like so far for these three cases. 274 00:16:29,770 --> 00:16:32,530 What about case one, where we had 275 00:16:32,530 --> 00:16:36,930 a two-sided inverse, full rank, everything great. 276 00:16:36,930 --> 00:16:41,290 The null spaces were, like, gone, right? 277 00:16:41,290 --> 00:16:44,480 The null spaces were just the zero vectors. 278 00:16:44,480 --> 00:16:49,590 Then I took case two, this null space was gone. 279 00:16:49,590 --> 00:16:52,990 280 00:16:52,990 --> 00:16:58,890 Case three, this null space was gone, and then case four is, 281 00:16:58,890 --> 00:17:04,250 like, the most general case when this picture is all there -- 282 00:17:04,250 --> 00:17:10,680 when all the null spaces -- this has dimension r, of course, 283 00:17:10,680 --> 00:17:14,700 this has dimension n-r, this has dimension r, 284 00:17:14,700 --> 00:17:26,800 this has dimension m-r, and the final case will be when r is 285 00:17:26,800 --> 00:17:29,210 smaller than m and n. 286 00:17:29,210 --> 00:17:38,940 But can I just, before I leave here 287 00:17:38,940 --> 00:17:43,750 look a little more at this one? 288 00:17:43,750 --> 00:17:46,510 At this case of full column rank? 289 00:17:46,510 --> 00:17:52,070 So A inverse on the left, it has this left-inverse 290 00:17:52,070 --> 00:17:53,930 to give the identity. 291 00:17:53,930 --> 00:17:56,540 I said if we multiply it in the other order, 292 00:17:56,540 --> 00:17:57,970 we wouldn't get the identity. 293 00:17:57,970 --> 00:18:02,650 But then I just realized that I should ask you, what do we get? 294 00:18:02,650 --> 00:18:05,260 So if I put them in the other order -- 295 00:18:05,260 --> 00:18:19,010 if I continue this down below, but I write A times A inverse 296 00:18:19,010 --> 00:18:21,040 left -- so there's A times the left-inverse, 297 00:18:21,040 --> 00:18:23,370 but it's not on the left any more. 298 00:18:23,370 --> 00:18:26,970 So it's not going to come out perfectly. 299 00:18:26,970 --> 00:18:35,990 But everybody in this room ought to recognize that matrix, 300 00:18:35,990 --> 00:18:38,150 right? 301 00:18:38,150 --> 00:18:42,080 Let's see, is that the guy we know? 302 00:18:42,080 --> 00:18:43,195 Am I OK, here? 303 00:18:43,195 --> 00:18:51,280 304 00:18:51,280 --> 00:18:53,000 What is that matrix? 305 00:18:53,000 --> 00:18:56,130 P. Thanks. 306 00:18:56,130 --> 00:18:59,230 P. That matrix -- 307 00:18:59,230 --> 00:19:02,070 it's a projection. 308 00:19:02,070 --> 00:19:07,750 It's the projection onto the column space. 309 00:19:07,750 --> 00:19:12,340 It's trying to be the identity matrix, right? 310 00:19:12,340 --> 00:19:17,620 A projection matrix tries to be the identity matrix, 311 00:19:17,620 --> 00:19:22,140 but you've given it, an impossible job. 312 00:19:22,140 --> 00:19:25,240 So it's the identity matrix where it can be, 313 00:19:25,240 --> 00:19:27,750 and elsewhere, it's the zero matrix. 314 00:19:27,750 --> 00:19:29,820 So this is P, right. 315 00:19:29,820 --> 00:19:34,190 A projection onto the column space. 316 00:19:34,190 --> 00:19:38,190 And if I asked you this one, and put these in the opposite 317 00:19:38,190 --> 00:19:41,340 OK. order -- so this came from up here. 318 00:19:41,340 --> 00:19:45,610 And similarly, if I try to put the right inverse on the left 319 00:19:45,610 --> 00:19:48,380 -- 320 00:19:48,380 --> 00:19:51,840 so that, like, came from above. 321 00:19:51,840 --> 00:19:53,810 This, coming from this side, what 322 00:19:53,810 --> 00:19:56,620 happens if I try to put the right inverse on the left? 323 00:19:56,620 --> 00:20:05,090 Then I would have A transpose A, A transpose inverse A, 324 00:20:05,090 --> 00:20:08,350 if this matrix is now on the left, what 325 00:20:08,350 --> 00:20:09,880 do you figure that matrix is? 326 00:20:09,880 --> 00:20:17,390 It's going to be a projection, too, right? 327 00:20:17,390 --> 00:20:19,090 It looks very much like this guy, 328 00:20:19,090 --> 00:20:22,400 except the only difference is, A and A transpose 329 00:20:22,400 --> 00:20:24,100 have been reversed. 330 00:20:24,100 --> 00:20:27,960 So this is a projection, this is another projection, 331 00:20:27,960 --> 00:20:29,710 onto the row space. 332 00:20:29,710 --> 00:20:33,520 333 00:20:33,520 --> 00:20:35,850 Again, it's trying to be the identity, 334 00:20:35,850 --> 00:20:39,980 but there's only so much the matrix can do. 335 00:20:39,980 --> 00:20:44,580 And this is the projection onto the column space. 336 00:20:44,580 --> 00:20:50,600 So let me now go back to the main picture 337 00:20:50,600 --> 00:20:55,600 and tell you about the general case, the pseudo-inverse. 338 00:20:55,600 --> 00:20:58,060 These are cases we know. 339 00:20:58,060 --> 00:21:01,500 So this was important review. 340 00:21:01,500 --> 00:21:08,480 You've got to know the business about these ranks, 341 00:21:08,480 --> 00:21:11,350 and the free variables -- 342 00:21:11,350 --> 00:21:14,960 really, this is linear algebra coming together. 343 00:21:14,960 --> 00:21:19,130 And, you know, one nice thing about teaching 18.06, 344 00:21:19,130 --> 00:21:23,260 It's not trivial. 345 00:21:23,260 --> 00:21:25,610 But it's -- 346 00:21:25,610 --> 00:21:28,590 I don't know, somehow, it's nice when it comes out right. 347 00:21:28,590 --> 00:21:31,360 I mean -- well, I shouldn't say anything bad about calculus, 348 00:21:31,360 --> 00:21:33,050 but I will. 349 00:21:33,050 --> 00:21:35,150 I mean, like, you know, you have formulas 350 00:21:35,150 --> 00:21:40,190 for surface area, and other awful things and, you know, 351 00:21:40,190 --> 00:21:46,580 they do their best in calculus, but it's not elegant. 352 00:21:46,580 --> 00:21:52,300 And, linear algebra just is -- well, you know, 353 00:21:52,300 --> 00:21:54,770 linear algebra is about the nice part of calculus, 354 00:21:54,770 --> 00:22:00,880 where everything's, like, flat, and, the formulas come out 355 00:22:00,880 --> 00:22:01,830 right. 356 00:22:01,830 --> 00:22:04,040 And you can go into high dimensions 357 00:22:04,040 --> 00:22:06,820 where, in calculus, you're trying 358 00:22:06,820 --> 00:22:09,780 to visualize these things, well, two or three dimensions 359 00:22:09,780 --> 00:22:10,810 is kind of the limit. 360 00:22:10,810 --> 00:22:12,430 But here, we don't -- 361 00:22:12,430 --> 00:22:16,280 you know, I've stopped doing two-by-twos, 362 00:22:16,280 --> 00:22:18,160 I'm just talking about the general case. 363 00:22:18,160 --> 00:22:22,430 OK, now I really will speak about the general case here. 364 00:22:22,430 --> 00:22:27,810 What could be the inverse -- 365 00:22:27,810 --> 00:22:29,910 what's a kind of reasonable inverse 366 00:22:29,910 --> 00:22:34,070 for a matrix for the completely general matrix where 367 00:22:34,070 --> 00:22:38,410 there's a rank r, but it's smaller than n, 368 00:22:38,410 --> 00:22:41,680 so there's some null space left, and it's smaller 369 00:22:41,680 --> 00:22:44,930 than m, so a transpose has some null space, 370 00:22:44,930 --> 00:22:48,430 and it's those null spaces that are screwing up inverses, 371 00:22:48,430 --> 00:22:49,650 right? 372 00:22:49,650 --> 00:22:53,090 Because if a matrix takes a vector to zero, 373 00:22:53,090 --> 00:23:01,390 well, there's no way an inverse can, like, bring it back 374 00:23:01,390 --> 00:23:03,240 to life. 375 00:23:03,240 --> 00:23:05,810 My topic is now the pseudo-inverse, 376 00:23:05,810 --> 00:23:09,190 and let's just by a picture, see what's 377 00:23:09,190 --> 00:23:11,170 the best inverse we could have? 378 00:23:11,170 --> 00:23:15,820 So, here's a vector x in the row space. 379 00:23:15,820 --> 00:23:18,170 I multiply by A. 380 00:23:18,170 --> 00:23:22,050 Now, the one thing everybody knows is you take a vector, 381 00:23:22,050 --> 00:23:25,360 you multiply by A, and you get an output, 382 00:23:25,360 --> 00:23:28,220 and where is that output? 383 00:23:28,220 --> 00:23:31,090 Where is Ax? 384 00:23:31,090 --> 00:23:35,050 Always in the column space, right? 385 00:23:35,050 --> 00:23:37,750 Ax is a combination of the columns. 386 00:23:37,750 --> 00:23:39,310 So Ax is somewhere here. 387 00:23:39,310 --> 00:23:42,800 388 00:23:42,800 --> 00:23:46,750 So I could take all the vectors in the row space. 389 00:23:46,750 --> 00:23:49,400 I could multiply them all by A. 390 00:23:49,400 --> 00:23:53,870 I would get a bunch of vectors in the column space 391 00:23:53,870 --> 00:23:59,280 and what I think is, I'd get all the vectors in the column space 392 00:23:59,280 --> 00:24:00,670 just right. 393 00:24:00,670 --> 00:24:03,440 I think that this connection between an x 394 00:24:03,440 --> 00:24:07,257 in the row space and an Ax in the column space, this 395 00:24:07,257 --> 00:24:07,840 is one-to-one. 396 00:24:07,840 --> 00:24:12,160 397 00:24:12,160 --> 00:24:14,170 We got a chance, because they have the same 398 00:24:14,170 --> 00:24:15,150 dimension. 399 00:24:15,150 --> 00:24:17,480 That's an r-dimensional space, and that's 400 00:24:17,480 --> 00:24:20,060 an r-dimensional space. 401 00:24:20,060 --> 00:24:22,970 And somehow, the matrix A -- 402 00:24:22,970 --> 00:24:27,260 it's got these null spaces hanging around, 403 00:24:27,260 --> 00:24:30,820 where it's knocking vectors to 404 00:24:30,820 --> 00:24:33,510 And then it's got all the vectors in between, zero. 405 00:24:33,510 --> 00:24:35,190 which is almost all vectors. 406 00:24:35,190 --> 00:24:38,460 Almost all vectors have a row space component 407 00:24:38,460 --> 00:24:39,250 and a null space 408 00:24:39,250 --> 00:24:39,950 component. 409 00:24:39,950 --> 00:24:42,880 And it's killing the null space component. 410 00:24:42,880 --> 00:24:45,690 But if I look at the vectors that are in the row space, 411 00:24:45,690 --> 00:24:48,740 with no null space component, just in the row space, 412 00:24:48,740 --> 00:24:51,110 then they all go into the column space, 413 00:24:51,110 --> 00:24:55,340 so if I put another vector, let's say, y, in the row space, 414 00:24:55,340 --> 00:25:02,730 I positive that wherever Ay is, it won't hit Ax. 415 00:25:02,730 --> 00:25:04,680 Do you see what I'm saying? 416 00:25:04,680 --> 00:25:05,830 Let's see why. 417 00:25:05,830 --> 00:25:09,340 418 00:25:09,340 --> 00:25:09,840 All right. 419 00:25:09,840 --> 00:25:12,600 So here's what I said. 420 00:25:12,600 --> 00:25:23,430 If x and y are in the row space, then A x 421 00:25:23,430 --> 00:25:27,650 is not the same as A y. 422 00:25:27,650 --> 00:25:30,110 They're both in the column space, of course, 423 00:25:30,110 --> 00:25:31,210 but they're different. 424 00:25:31,210 --> 00:25:36,690 425 00:25:36,690 --> 00:25:39,780 That would be a perfect question on a final exam, 426 00:25:39,780 --> 00:25:45,360 because that's what I'm teaching you 427 00:25:45,360 --> 00:25:48,690 in that material of chapter three 428 00:25:48,690 --> 00:25:53,920 and chapter four, especially chapter three. 429 00:25:53,920 --> 00:25:58,430 If x and y are in the row space, then Ax is different from Ay. 430 00:25:58,430 --> 00:26:01,280 So what this means -- 431 00:26:01,280 --> 00:26:03,630 and we'll see why -- 432 00:26:03,630 --> 00:26:09,000 is that, in words, from the row space to the column space, 433 00:26:09,000 --> 00:26:12,980 A is perfect, it's an invertible matrix. 434 00:26:12,980 --> 00:26:16,700 If we, like, limited it to those spaces. 435 00:26:16,700 --> 00:26:19,930 And then, its inverse will be what 436 00:26:19,930 --> 00:26:21,290 I'll call the pseudo-inverse. 437 00:26:21,290 --> 00:26:23,770 So that's that the pseudo-inverse is. 438 00:26:23,770 --> 00:26:28,940 It's the inverse -- so A goes this way, from x to y -- sorry, 439 00:26:28,940 --> 00:26:35,340 x to A x, from y to A y, that's A, going that way. 440 00:26:35,340 --> 00:26:38,910 Then in the other direction, anything in the column space 441 00:26:38,910 --> 00:26:41,400 comes from somebody in the row space, 442 00:26:41,400 --> 00:26:45,010 and the reverse there is what I'll call the pseudo-inverse, 443 00:26:45,010 --> 00:26:51,010 and the accepted notation is A plus. 444 00:26:51,010 --> 00:26:55,390 So y will be A plus x. 445 00:26:55,390 --> 00:26:55,890 I'm sorry. 446 00:26:55,890 --> 00:27:05,900 No, y will be A plus times whatever it started with, A y. 447 00:27:05,900 --> 00:27:09,030 Do you see my picture there? 448 00:27:09,030 --> 00:27:11,340 Same, of course, for x and A x. 449 00:27:11,340 --> 00:27:15,280 This way, A does it, the other way is the pseudo-inverse, 450 00:27:15,280 --> 00:27:18,430 and the pseudo-inverse just kills this stuff, 451 00:27:18,430 --> 00:27:20,360 and the matrix just kills this 452 00:27:20,360 --> 00:27:20,860 stuff. 453 00:27:20,860 --> 00:27:25,260 So everything that's really serious here is going 454 00:27:25,260 --> 00:27:27,840 on in the row space and the column space, and now, 455 00:27:27,840 --> 00:27:31,550 tell me -- 456 00:27:31,550 --> 00:27:34,910 this is the fundamental fact, that between those two 457 00:27:34,910 --> 00:27:37,560 r-dimensional spaces, our matrix is perfect. 458 00:27:37,560 --> 00:27:40,900 459 00:27:40,900 --> 00:27:41,400 Why? 460 00:27:41,400 --> 00:27:44,960 461 00:27:44,960 --> 00:27:47,180 Suppose they weren't. 462 00:27:47,180 --> 00:27:49,230 Why do I get into trouble? 463 00:27:49,230 --> 00:27:51,780 Suppose -- so, proof. 464 00:27:51,780 --> 00:27:54,090 I haven't written down proof very much, 465 00:27:54,090 --> 00:27:57,640 but I'm going to use that word once. 466 00:27:57,640 --> 00:28:00,710 Suppose they were the same. 467 00:28:00,710 --> 00:28:07,990 Suppose these are supposed to be two different vectors. 468 00:28:07,990 --> 00:28:10,540 Maybe I'd better make the statement correctly. 469 00:28:10,540 --> 00:28:13,930 If x and y are different vectors in the row space -- 470 00:28:13,930 --> 00:28:20,790 maybe I'll better put if x is different from y, 471 00:28:20,790 --> 00:28:23,370 both in the row space -- 472 00:28:23,370 --> 00:28:25,620 so I'm starting with two different vectors in the row 473 00:28:25,620 --> 00:28:29,900 space, I'm multiplying by A -- so these guys are in the column 474 00:28:29,900 --> 00:28:33,520 space, everybody knows that, and the point is, 475 00:28:33,520 --> 00:28:36,980 they're different over there. 476 00:28:36,980 --> 00:28:38,950 So, suppose they weren't. 477 00:28:38,950 --> 00:28:40,570 Suppose A x=A y. 478 00:28:40,570 --> 00:28:44,940 479 00:28:44,940 --> 00:28:48,800 Suppose, well, that's the same as saying A(x-y) is zero. 480 00:28:48,800 --> 00:28:54,960 481 00:28:54,960 --> 00:28:57,140 So what? 482 00:28:57,140 --> 00:29:00,090 So, what do I know now about (x-y), 483 00:29:00,090 --> 00:29:03,500 what do I know about this vector? 484 00:29:03,500 --> 00:29:07,810 Well, I can see right away, what space is it in? 485 00:29:07,810 --> 00:29:10,390 It's sitting in the null space, right? 486 00:29:10,390 --> 00:29:11,580 So it's in the null space. 487 00:29:11,580 --> 00:29:14,520 488 00:29:14,520 --> 00:29:17,030 But what else do I know about it? 489 00:29:17,030 --> 00:29:20,950 Here it was x in the row space, y in the row space, 490 00:29:20,950 --> 00:29:23,170 what about x-y? 491 00:29:23,170 --> 00:29:29,870 It's also in the row space, right? 492 00:29:29,870 --> 00:29:32,030 Heck, that thing is a vector space, 493 00:29:32,030 --> 00:29:35,550 and if the vector space is anything at all, 494 00:29:35,550 --> 00:29:38,610 if x is in the row space, and y is in the row space, 495 00:29:38,610 --> 00:29:43,040 then the difference is also, so it's also in the row space. 496 00:29:43,040 --> 00:29:47,990 497 00:29:47,990 --> 00:29:48,850 So what? 498 00:29:48,850 --> 00:29:52,420 Now I've got a vector x-y that's in the null space, 499 00:29:52,420 --> 00:29:56,270 and that's also in the row space, so what vector is it? 500 00:29:56,270 --> 00:29:58,380 It's the zero vector. 501 00:29:58,380 --> 00:30:00,840 So I would conclude from that that x-y 502 00:30:00,840 --> 00:30:07,200 had to be the zero vector, x-y, so, in other words, 503 00:30:07,200 --> 00:30:09,120 if I start from two different vectors, 504 00:30:09,120 --> 00:30:11,480 I get two different vectors. 505 00:30:11,480 --> 00:30:14,020 If these vectors are the same, then those vectors 506 00:30:14,020 --> 00:30:16,050 had to be the same. 507 00:30:16,050 --> 00:30:21,600 That's like the algebra proof, which we understand completely 508 00:30:21,600 --> 00:30:28,230 because we really understand these subspaces of what 509 00:30:28,230 --> 00:30:32,480 I said in words, that a matrix A is really 510 00:30:32,480 --> 00:30:37,560 a nice, invertible mapping from row space to columns pace. 511 00:30:37,560 --> 00:30:40,160 If the null spaces keep out of the way, 512 00:30:40,160 --> 00:30:43,200 then we have an inverse. 513 00:30:43,200 --> 00:30:46,840 And that inverse is called the pseudo inverse, 514 00:30:46,840 --> 00:30:51,410 and it's a very, very, useful in application. 515 00:30:51,410 --> 00:30:54,360 Statisticians discovered, oh boy, this 516 00:30:54,360 --> 00:30:56,580 is the thing that we needed all our lives, 517 00:30:56,580 --> 00:30:59,100 and here it finally showed up, the pseudo-inverse 518 00:30:59,100 --> 00:31:02,070 is the right thing. 519 00:31:02,070 --> 00:31:04,300 Why do statisticians need it? 520 00:31:04,300 --> 00:31:11,360 And because statisticians are like least-squares-happy. 521 00:31:11,360 --> 00:31:14,440 I mean they're always doing least squares. 522 00:31:14,440 --> 00:31:20,700 And so this is their central linear regression. 523 00:31:20,700 --> 00:31:22,920 Statisticians who may watch this on video, 524 00:31:22,920 --> 00:31:28,940 please forgive that description of your interests. 525 00:31:28,940 --> 00:31:35,170 One of your interests is linear regression and this problem. 526 00:31:35,170 --> 00:31:41,150 But this problem is only OK provided we have full column 527 00:31:41,150 --> 00:31:42,080 rank. 528 00:31:42,080 --> 00:31:46,810 And statisticians have to worry all the time about, oh, God, 529 00:31:46,810 --> 00:31:49,900 maybe we just repeated an experiment. 530 00:31:49,900 --> 00:31:52,220 You know, you're taking all these measurements, 531 00:31:52,220 --> 00:31:54,750 maybe you just repeat them a few times. 532 00:31:54,750 --> 00:31:56,730 You know, maybe they're not independent. 533 00:31:56,730 --> 00:32:00,910 Well, in that case, that A transpose A matrix 534 00:32:00,910 --> 00:32:04,230 that they depend on becomes singular. 535 00:32:04,230 --> 00:32:06,890 So then that's when they needed the pseudo-inverse, 536 00:32:06,890 --> 00:32:09,180 it just arrived at the right moment, 537 00:32:09,180 --> 00:32:13,840 and it's the right quantity. 538 00:32:13,840 --> 00:32:14,440 OK. 539 00:32:14,440 --> 00:32:21,590 So now that you know what the pseudo-inverse should do, let 540 00:32:21,590 --> 00:32:25,080 me see what it is. 541 00:32:25,080 --> 00:32:27,010 Can we find it? 542 00:32:27,010 --> 00:32:30,170 So this is my -- to complete the lecture is -- 543 00:32:30,170 --> 00:32:42,740 how do I find this pseudo-inverse A plus? 544 00:32:42,740 --> 00:32:45,310 OK. 545 00:32:45,310 --> 00:32:46,340 OK. 546 00:32:46,340 --> 00:32:48,170 Well, here's one way. 547 00:32:48,170 --> 00:32:50,710 548 00:32:50,710 --> 00:32:53,860 Everything I do today is to try to review stuff. 549 00:32:53,860 --> 00:33:00,770 One way would be to start from the SVD. 550 00:33:00,770 --> 00:33:02,630 The Singular Value Decomposition. 551 00:33:02,630 --> 00:33:05,380 And you remember that that factored A 552 00:33:05,380 --> 00:33:10,480 into an orthogonal matrix times this diagonal matrix 553 00:33:10,480 --> 00:33:12,500 times this orthogonal matrix. 554 00:33:12,500 --> 00:33:16,330 But what did that diagonal guy look like? 555 00:33:16,330 --> 00:33:25,510 This diagonal guy, sigma, has some non-zeroes, 556 00:33:25,510 --> 00:33:28,280 and you remember, they came from A transpose A, 557 00:33:28,280 --> 00:33:31,400 and A A transpose, these are the good guys, and then 558 00:33:31,400 --> 00:33:34,680 some more zeroes, and all zeroes there, and all zeroes there. 559 00:33:34,680 --> 00:33:38,110 560 00:33:38,110 --> 00:33:41,700 So you can guess what the pseudo-inverse is, 561 00:33:41,700 --> 00:33:45,030 I just invert stuff that's nice to invert -- well, 562 00:33:45,030 --> 00:33:47,280 what's the pseudo-inverse of this? 563 00:33:47,280 --> 00:33:50,610 That's what the problem comes down to. 564 00:33:50,610 --> 00:33:55,310 What's the pseudo-inverse of this beautiful diagonal matrix? 565 00:33:55,310 --> 00:33:58,380 But it's got a null space, right? 566 00:33:58,380 --> 00:34:01,480 What's the rank of this matrix? 567 00:34:01,480 --> 00:34:06,420 What's the rank of this diagonal matrix? 568 00:34:06,420 --> 00:34:07,430 r, of course. 569 00:34:07,430 --> 00:34:09,790 It's got r non-zeroes, and then it's otherwise, 570 00:34:09,790 --> 00:34:10,780 zip. 571 00:34:10,780 --> 00:34:21,000 So it's got n columns, it's got m rows, and it's got rank r. 572 00:34:21,000 --> 00:34:23,989 It's the best example, the simplest example we could ever 573 00:34:23,989 --> 00:34:28,089 have of our general setup. 574 00:34:28,089 --> 00:34:31,179 575 00:34:31,179 --> 00:34:31,699 OK? 576 00:34:31,699 --> 00:34:36,800 So what's the pseudo-inverse? 577 00:34:36,800 --> 00:34:39,020 What's the matrix -- 578 00:34:39,020 --> 00:34:41,296 so I'll erase our columns, because right below it, 579 00:34:41,296 --> 00:34:42,754 I want to write the pseudo-inverse. 580 00:34:42,754 --> 00:34:46,199 581 00:34:46,199 --> 00:34:49,179 OK, you can make a pretty darn good guess. 582 00:34:49,179 --> 00:34:53,170 If it was a proper diagonal matrix, invertible, 583 00:34:53,170 --> 00:34:57,000 if there weren't any zeroes down here, if it was sigma one 584 00:34:57,000 --> 00:35:01,010 to sigma n, then everybody knows what the inverse would be, 585 00:35:01,010 --> 00:35:08,760 the inverse would be one over sigma one, down to one over s- 586 00:35:08,760 --> 00:35:13,340 but of course, I'll have to stop at sigma r. 587 00:35:13,340 --> 00:35:17,350 And, it will be the rest, zeroes again, of course. 588 00:35:17,350 --> 00:35:22,620 And now this one was m by n, and this one 589 00:35:22,620 --> 00:35:25,340 is meant to have a slightly different, you know, 590 00:35:25,340 --> 00:35:29,230 transpose shape, n by m. 591 00:35:29,230 --> 00:35:31,740 They both have that rank r. 592 00:35:31,740 --> 00:35:39,820 593 00:35:39,820 --> 00:35:45,350 My idea is that the pseudo-inverse is the best -- 594 00:35:45,350 --> 00:35:48,200 is the closest I can come to an inverse. 595 00:35:48,200 --> 00:35:53,220 So what is sigma times its pseudo-inverse? 596 00:35:53,220 --> 00:35:56,020 Can you multiply sigma by its pseudo-inverse? 597 00:35:56,020 --> 00:35:57,920 Multiply that by that? 598 00:35:57,920 --> 00:35:59,130 What matrix do you get? 599 00:35:59,130 --> 00:36:05,040 600 00:36:05,040 --> 00:36:07,280 They're diagonal. 601 00:36:07,280 --> 00:36:10,200 Rectangular, of course. 602 00:36:10,200 --> 00:36:18,780 But of course, we're going to get ones, R ones, 603 00:36:18,780 --> 00:36:20,590 and all the rest, zeroes. 604 00:36:20,590 --> 00:36:24,420 And the shape of that, this whole matrix will be m by 605 00:36:24,420 --> 00:36:25,780 m. 606 00:36:25,780 --> 00:36:30,200 And suppose I did it in the other order. 607 00:36:30,200 --> 00:36:32,067 Suppose I did sigma plus sigma. 608 00:36:32,067 --> 00:36:33,525 Why don't I do it right underneath? 609 00:36:33,525 --> 00:36:38,190 610 00:36:38,190 --> 00:36:40,500 in the opposite order? 611 00:36:40,500 --> 00:36:42,640 See, this matrix hasn't got a left-inverse, 612 00:36:42,640 --> 00:36:45,170 it hasn't got a right-inverse, but every matrix 613 00:36:45,170 --> 00:36:46,590 has got a pseudo-inverse. 614 00:36:46,590 --> 00:36:51,300 If I do it in the order sigma plus sigma, what do I get? 615 00:36:51,300 --> 00:36:55,920 Square matrix, this is m by n, this is m by m, 616 00:36:55,920 --> 00:37:01,590 my result is going to m by m -- is going to be n by n, 617 00:37:01,590 --> 00:37:03,030 and what is it? 618 00:37:03,030 --> 00:37:05,930 619 00:37:05,930 --> 00:37:08,260 Those are diagonal matrices, it's 620 00:37:08,260 --> 00:37:10,600 going to be ones, and then zeroes. 621 00:37:10,600 --> 00:37:14,670 622 00:37:14,670 --> 00:37:19,020 It's not the same as that, it's a different size -- 623 00:37:19,020 --> 00:37:22,090 it's a projection. 624 00:37:22,090 --> 00:37:25,560 One is a projection matrix onto the column space, 625 00:37:25,560 --> 00:37:30,590 and this one is the projection matrix onto the row space. 626 00:37:30,590 --> 00:37:35,470 That's the best that pseudo-inverse can do. 627 00:37:35,470 --> 00:37:38,030 So what the pseudo-inverse does is, 628 00:37:38,030 --> 00:37:41,050 if you multiply on the left, you don't get the identity, 629 00:37:41,050 --> 00:37:42,510 if you multiply on the right, you 630 00:37:42,510 --> 00:37:46,080 don't get the identity, what you get is the projection. 631 00:37:46,080 --> 00:37:51,300 It brings you into the two good spaces, the row space 632 00:37:51,300 --> 00:37:52,760 and column space. 633 00:37:52,760 --> 00:37:55,240 And it just wipes out the null space. 634 00:37:55,240 --> 00:37:57,800 So that's what the pseudo-inverse of this diagonal 635 00:37:57,800 --> 00:38:04,710 one is, and then the pseudo-inverse of A itself -- 636 00:38:04,710 --> 00:38:06,290 this is perfectly invertible. 637 00:38:06,290 --> 00:38:09,180 What's the inverse of V transpose? 638 00:38:09,180 --> 00:38:11,800 Just another tiny bit of review. 639 00:38:11,800 --> 00:38:18,590 That's an orthogonal matrix, and its inverse is V, good. 640 00:38:18,590 --> 00:38:21,360 This guy has got all the trouble in it, 641 00:38:21,360 --> 00:38:25,940 all the null space is responsible for, 642 00:38:25,940 --> 00:38:28,610 so it doesn't have a true inverse, 643 00:38:28,610 --> 00:38:32,010 it has a pseudo-inverse, and then the inverse of U 644 00:38:32,010 --> 00:38:37,320 is U transpose, thanks. 645 00:38:37,320 --> 00:38:40,200 Or, of course, I could write U inverse. 646 00:38:40,200 --> 00:38:43,640 So, that's the question of, how do you find the pseudo-inverse 647 00:38:43,640 --> 00:38:45,110 -- 648 00:38:45,110 --> 00:38:49,350 so what statisticians do when they're in this -- 649 00:38:49,350 --> 00:38:54,980 so this is like the case of where least squares breaks down 650 00:38:54,980 --> 00:38:58,420 because the rank is -- you don't have full rank, 651 00:38:58,420 --> 00:39:05,020 and the beauty of the singular value decomposition is, 652 00:39:05,020 --> 00:39:09,620 it puts all the problems into this diagonal matrix where 653 00:39:09,620 --> 00:39:10,980 it's clear what to do. 654 00:39:10,980 --> 00:39:13,960 It's the best inverse you could think of is clear. 655 00:39:13,960 --> 00:39:16,430 You see there could be other -- 656 00:39:16,430 --> 00:39:18,720 I mean, we could put some stuff down here, 657 00:39:18,720 --> 00:39:20,980 it would multiply these zeroes. 658 00:39:20,980 --> 00:39:27,270 It wouldn't have any effect, but then the good pseudo-inverse 659 00:39:27,270 --> 00:39:33,290 is the one with no extra stuff, it's sort of, like, 660 00:39:33,290 --> 00:39:36,430 as small as possible. 661 00:39:36,430 --> 00:39:41,040 It has to have those to produce the ones. 662 00:39:41,040 --> 00:39:46,260 If it had other stuff, it would just be a larger matrix, 663 00:39:46,260 --> 00:39:51,050 so this pseudo-inverse is kind of the minimal matrix that 664 00:39:51,050 --> 00:39:53,960 gives the best result. 665 00:39:53,960 --> 00:39:57,250 Sigma sigma plus being r ones. 666 00:39:57,250 --> 00:39:59,710 SK. 667 00:39:59,710 --> 00:40:03,122 so I guess I'm hoping -- 668 00:40:03,122 --> 00:40:04,830 pseudo-inverse, again, let me repeat what 669 00:40:04,830 --> 00:40:06,038 I said at the very beginning. 670 00:40:06,038 --> 00:40:09,080 671 00:40:09,080 --> 00:40:11,990 This pseudo-inverse, which appears 672 00:40:11,990 --> 00:40:17,700 at the end, which is in section seven point four, 673 00:40:17,700 --> 00:40:24,090 and probably I did more with it here than I did in the book. 674 00:40:24,090 --> 00:40:28,460 The word pseudo-inverse will not appear on an exam in this 675 00:40:28,460 --> 00:40:34,900 course, but I think if you see this all will appear, 676 00:40:34,900 --> 00:40:40,010 because this is all what the course was about, chapters one, 677 00:40:40,010 --> 00:40:41,527 two, three, four -- 678 00:40:41,527 --> 00:40:44,150 679 00:40:44,150 --> 00:40:47,900 but if you see all that, then you probably see, 680 00:40:47,900 --> 00:40:51,810 well, OK, the general case had both null spaces around, 681 00:40:51,810 --> 00:40:56,470 and this is the natural thing to do. 682 00:40:56,470 --> 00:41:01,190 So, this is one way to find the pseudo-inverse. 683 00:41:01,190 --> 00:41:03,500 Yes. 684 00:41:03,500 --> 00:41:07,260 The point of a pseudo-inverse, of computing a pseudo-inverse 685 00:41:07,260 --> 00:41:10,220 is to get some factors where you can find 686 00:41:10,220 --> 00:41:12,000 the pseudo-inverse quickly. 687 00:41:12,000 --> 00:41:15,720 And this is, like, the champion, because this 688 00:41:15,720 --> 00:41:20,850 is where we can invert those, and those two, easily, 689 00:41:20,850 --> 00:41:26,560 just by transposing, and we know what to do with a diagonal. 690 00:41:26,560 --> 00:41:33,210 OK, that's as much review, maybe -- 691 00:41:33,210 --> 00:41:37,920 let's have a five-minute holiday in 18.06 and, I'll see you 692 00:41:37,920 --> 00:41:40,060 Wednesday, then, for the rest of this 693 00:41:40,060 --> 00:41:40,560 course. 694 00:41:40,560 --> 00:41:42,110 Thanks. 695 00:41:42,110 --> 00:41:52,467