If $x = a\cos\alpha + b\sin\alpha$ and $y = a\sin\alpha - b\cos\alpha$, then $\left(x\frac{dy}{dx} - y^2\frac{d^2y}{dx^2}\right)$ is equal to:

Given $x = a\cos\alpha + b\sin\alpha$ and $y = a\sin\alpha - b\cos\alpha$.

$x^2 + y^2 = (a\cos\alpha + b\sin\alpha)^2 + (a\sin\alpha - b\cos\alpha)^2$

$= a^2\cos^2\alpha + 2ab\cos\alpha\sin\alpha + b^2\sin^2\alpha + a^2\sin^2\alpha - 2ab\sin\alpha\cos\alpha + b^2\cos^2\alpha$

The $2ab\cos\alpha\sin\alpha$ terms cancel out:

$= a^2(\cos^2\alpha + \sin^2\alpha) + b^2(\sin^2\alpha + \cos^2\alpha)$

$x^2 + y^2 = a^2 + b^2 \quad \text{...(i)}$

---

$\frac{dx}{d\alpha} = -a\sin\alpha + b\cos\alpha = -y$

$\frac{dy}{d\alpha} = a\cos\alpha + b\sin\alpha = x$

---

$\frac{dy}{dx} = \frac{dy/d\alpha}{dx/d\alpha}$

$= \frac{x}{-y}$

$= -\frac{x}{y}$

---

$\frac{d^2y}{dx^2} = \dfrac{\frac{d}{d\alpha}\!\left(\frac{dy}{dx}\right)}{\frac{dx}{d\alpha}}$

$\frac{d}{d\alpha}\!\left(-\frac{x}{y}\right) = -\left(\frac{y \cdot \frac{dx}{d\alpha} - x \cdot \frac{dy}{d\alpha}}{y^2}\right)$

$= -\left(\frac{y(-y) - x(x)}{y^2}\right)$

$= -\left(\frac{-y^2 - x^2}{y^2}\right)$

$= \frac{x^2 + y^2}{y^2}$

$\frac{d^2y}{dx^2} = \frac{(x^2 + y^2)/y^2}{-y}$

$= -\frac{x^2 + y^2}{y^3}$

From (i):

$\frac{d^2y}{dx^2} = -\frac{a^2 + b^2}{y^3}$

---

$x\frac{dy}{dx} - y^2\frac{d^2y}{dx^2} = x\!\left(-\frac{x}{y}\right) - y^2\!\left(-\frac{a^2+b^2}{y^3}\right)$

$= -\frac{x^2}{y} + \frac{a^2 + b^2}{y}$

$= \frac{-x^2 + (a^2 + b^2)}{y}$

From (i), $a^2 + b^2 - x^2 = y^2$:

$= \frac{y^2}{y}$

$= y$