As I said in the comments, this is indeed called torus or toroidal space when it comes to the topology. Even if the images suggest something 3 dimensional, this is just a visualization of the embedding of such a space in $\mathbb R^3$.
Regarding the distance between two points, I think you mean following:
Just consider the coordinates $p=(x_1,y_1)$ and $q=(x_2,y_2)$ in the $[0,X)\times [0,Y) \simeq (\mathbb R / X\mathbb Z)\times (\mathbb R / Y\mathbb Z)$ square. Then define all the translations $q_i$ of $q$ :
$$
\begin{align*}
q_0 &= (x_2\hphantom{+X\,\,\,},y_2\hphantom{+X\,\,\,})\\
q_1 &= (x_2+X,y_2+Y)\\
q_2 &= (x_2\hphantom{+X\,\,\,},y_2+Y)\\
q_3 &= (x_2-X,y_2+Y)\\
q_4 &= (x_2-X,y_2\hphantom{+X\,\,\,})\\
q_5 &= (x_2-X,y_2-Y)\\
q_6 &= (x_2\hphantom{+X\,\,\,},y_2-Y)\\
q_7 &= (x_2+X,y_2-Y)\\
q_8 &= (x_2+X,y_2\hphantom{+X\,\,\,})
\end{align*}$$
Then I think the distance you're looking for is
$$d(p,q) = \min_{i=0}^8 \Vert p-q_i \Vert$$