As mentioned by Dan, the homogeneous equation has solution $y(t) = e^{-t}(A\cos(t) + B\sin(t))$ for arbitrary constants $A$ and $B$. We shall denote the two basis functions as $y_1(t) = e^{-t}\cos(t)$ and $y_2(t) = e^{-t}\sin(t)$. Using a standard method to solve second-order linear ODE (see Particular solution of second order differential equation), I shall show how one can use the "Variation of Parameters" here.
As a recap, the general solution using Variation of Parameters (see https://tutorial.math.lamar.edu/classes/de/VariationofParameters.aspx) is given by $$y_p(t) = -y_1(t)\int^t \frac{y_2(s)f(s)}{W(y_1,y_2)(s)}\mathrm{d}s
+ y_2(t)\int^t \frac{y_1(s)f(s)}{W(y_1,y_2)(s)}\mathrm{d}s$$
where $f(t)$ is the forcing function of the second-order linear ODE $\mathcal{L}y(t) = f(t)$, and $W(y_1,y_2)(t)$ is the Wronskian, defined by $$W(y_1,y_2)(t) = y_1(t)y_2'(t) - y_2(t)y_1'(t). $$
All that's left is to compute the relevant terms. Substituting $y_1(t)$ and $y_2(t)$ as deduced above, we have
$$\begin{aligned}W(y_1,y_2)(t) &= y_1(t)y_2'(t) - y_2(t)y_1'(t) \\ &= e^{-t}\cos(t)(e^{-t}\sin(t))' - e^{-t}\sin(t)(e^{-t}\cos(t))' \\ &= e^{-2t}.\end{aligned} $$
Let's now compute the two integrals in the formula given above. Before we begin, we note that for any smooth function $g(t)$, we have
$$\int^t g(s)\delta(s) \mathrm{d}s = \int^t g(0)\delta(s)\mathrm{d}s = g(0) \int^t \delta(s)\mathrm{d}s = g(0)\theta(t)$$
and using integration by parts, we have
$$\int^t g(s)\delta'(s) \mathrm{d}s = g(t)\delta(t) - \int^t g'(s)\delta(s) \mathrm{d}s = g(0)\delta(t) - g'(0)\theta(t).$$
Here, we have used the fact that $\delta(t)g(t) = \delta(t)g(0)$ since the values of $g$ outside of $t = 0$ does not matter as $\delta = 0$ there. (These are the physicists type of argument, in which from the way the question is phrased, would be more suitable as a solution to this question!)
The first is given by
$$\begin{aligned}\int^t \frac{y_2(s)f(s)}{W(y_1,y_2)(s)}\mathrm{d}s &= \int^t e^{2s}e^{-s}\sin(s)(2\delta'(s) + 2\delta(s)) \mathrm{d}s \\
&= 2 \int^t e^{s}\sin(s)\delta'(s)\mathrm{d}s + 2\int^t e^{s}\sin(s)\delta(s) \mathrm{d}s \\ &= -2(e^t \sin(t))'|_{t=0}\theta(t) = -2 \theta(t)\end{aligned}$$
where I have ignored the other terms that vanishes at $t = 0$ in the last line. Similarly, the second is given by
$$\begin{aligned}\int^t \frac{y_1(s)f(s)}{W(y_1,y_2)(s)}\mathrm{d}s
&= 2 \int^t e^{s}\cos(s)\delta'(s)\mathrm{d}s + 2\int^t e^{s}\cos(s)\delta(s) \mathrm{d}s \\ &= 2(e^t \cos{t}|_{t=0} - (e^t \cos{t})'|_{t=0} ) \theta(t) + 2(e^t \cos{t}|_{t=0})\delta(t) \\ &= 2 \delta(t).\end{aligned}$$
Combining all the above computations, we get $y_p(t) = 2\theta(t)y_1(t) + 2 \delta(t)y_2(t)$. Thus,
$$\boxed{y_p(t) = 2e^{-t}(\cos(t)\theta(t) + \delta(t)\sin(t)) = 2e^{-t}(\sin(t)\theta(t))'}. $$
In retrospect, $2\theta(t)y_1(t)$ looks like what you have written in your solution so I'm guessing there might be a shorter/slicker solution to this. Nonetheless, this illustrates that standard calculus techniques still work for this (though from a rigorous mathematical point of view, one has to rely on distribution theory).