If $A = \begin{bmatrix}1 & -1 \\ 2 & -1\end{bmatrix}$, $B = \begin{bmatrix}a & 1 \\ b & -1\end{bmatrix}$ and $(A + B)^2 = A^2 + B^2$ then

$(A + B)^2 = A^2 + AB + BA + B^2$

Since matrix multiplication is not commutative, $AB \neq BA$ in general, so the cross terms cannot be combined.

Given $(A + B)^2 = A^2 + B^2$, this means:

$AB + BA = O$

---

$AB = \begin{bmatrix}1 & -1 \\ 2 & -1\end{bmatrix}\begin{bmatrix}a & 1 \\ b & -1\end{bmatrix}$

$= \begin{bmatrix}a - b & 2 \\ 2a - b & 3\end{bmatrix}$

---

$BA = \begin{bmatrix}a & 1 \\ b & -1\end{bmatrix}\begin{bmatrix}1 & -1 \\ 2 & -1\end{bmatrix}$

$= \begin{bmatrix}a+2 & -a-1 \\ b-2 & -b+1\end{bmatrix}$

---

$AB + BA = \begin{bmatrix}2a - b + 2 & 1 - a \\ 2a - 2 & 4 - b\end{bmatrix} = \begin{bmatrix}0 & 0 \\ 0 & 0\end{bmatrix}$

From entry $(1,2)$:

$1 - a = 0$

$a = 1$

From entry $(2,2)$:

$4 - b = 0$

$b = 4$

From entry $(2,1)$:

$2(1) - 2 = 0$ ✓

From entry $(1,1)$:

$2(1) - 4 + 2 = 0$ ✓

---

$a = 1, \quad b = 4$