This is a quite interesting problem that "beautifully" connects linear algebra to a linear regression model and the recommender systems.
Here, the matrix $M$ is a projection matrix and you can write it as:
$$M= A(A^TA)^{-1}A^T$$
Where the $A$ is a dataset or you can call it a design matrix.
Now, when we say that $U$ is 2-dimensional then it means that matrix $A$ is having two linearly independent vectors which span the whole column space of $A$
Now,
$M^2 =MM=(A(A^TA)^{-1}A^T)(A(A^TA)^{-1}A^T) =A[(A^TA)^{-1}(A^TA)](A^TA)^{-1}A^T$
$M^2=AI(A^TA)^{-1}A^T=A(A^TA)^{-1}A^T=M$
Therefore, $M^2=M$ and $M^3=M^2M=MM=M^2=M$
The rank of the matrix $M$ and the matrix $A$ should be the same. (can be proved easily)
Now, Assuming $U$ is two-dimensional subspace of $\mathbb{R}^3$
Since, column space of matrix $A$ is the subspace $U$ which is 2-dimensional and dimension of the column space and row space of a matrix is called the rank of that matrix. It means:
$rank(M)=rank(A)=2$
Now, according to the rank-nullity theorem:
$rk(M)+nullity(M)=\#columns(M)$
So,
$2+nullity(M)=3 \Rightarrow nullity(M)=1$
nullity(M) is nothing but the dimension of the null space of matrix $M.$
Therefore,
The null space of matrix $M$ is one-dimensional.
So, the answers are $(B),(C),(D)$
--------------------------------------------------------------------------------------------------------------------------------------------
$\textbf{Proof}$
Here, we will prove that $M=A(A^TA)^{-1}A^T$
Suppose we have a $\textbf{linear regression problem}$ in which we have to find a best fit line say $y=mx+c$ for a given set of points:
https://gateoverflow.in/?qa=blob&qa_blobid=12799168104057043330
Suppose, we are given the $3$ points on this line as:
$mx_1+c=y_1$
$mx_2+c=y_2$
$mx_3+c=y_3$
It is a overdetermined system and here $m$ and $c$ are unknown which we need to find. These are called the "Estimated Coefficients" in the language of statistics.
We can write system as:
So, we have $Az=b.$
Say vectors $\vec{a_1}$ and $\vec{a_2}$ are the columns of the matrix $A$ and they are linearly independent and span the whole column space of $A.$
Geometrically, we can represent it as:
https://gateoverflow.in/?qa=blob&qa_blobid=2155493817043769815
The system $Az=b$ is fully solvable if $\vec{b}$ is in the column space of $A$ but it is not the case here and we can't solve this problem perfectly but we can solve the relax/approximate version of this problem as:
Make the perpendicular projection of $\vec{b}$ in the column space of $A$ and say it is $\hat{b}$ and we can write $b=\hat{b}+e$ or $e=b-\hat{b}$ where $e$ is perpendicular to both $\vec{a_1}$ and $\vec{a_2}.$ and so $a_1^T e=0$ and $a_2^T e=0$
So, $A^Te=0$
$A^T(b-\hat{b})=0$
$A^Tb=A^T\hat{b} = 0$ $;\;\; (1)$
Here, system for our relaxed version of the problem is $Az^*=\hat{b}$
So, replace the value of $\hat{b}$ in $(1)$, we get:
$A^Tb = A^TAz^*$ which implies $z^*=(A^TA)^{-1}A^Tb$
This is the answer of our best fit line.
Now, here,
$\hat{b}=Az^* = [A(A^TA)^{-1}A^T]b=Pb$
This matrix $P$ is called the Projection Matrix and it is equal to $A(A^TA)^{-1}A^T$.
-------------------------------------------------------------------------------------------------------------------------------------------------
These figures and explanation are taken from my course work at ISI. Since this problem is very close to my heart, So, I wanted to add more things here but due to space constraint I could not add more here. If anyone have any doubt then you can ask it in comments.
