Been a while since I last dabbled on Reflection X but here goes.
By default, the scologin and X server program are running on your SCO OpenServer box. So there is really nothing needed to setup on the server side unless this is not true.
On the client side, configure Reflection X to connect to your host. Login and once authenticated, from the prompt, issue the following command:
$ DISPLAY=<hostname>:0.0 --> replace <hostname> with your IP
$ export DISPLAY
$ startx
This should start the X Windows environment for your desktop. Note that this is regardless of username. For more information, do a man on startx.
All these steps can be automated, with auto or manual logins (refer to Reflection X docs for more info). You could also have single or multiple display windows.
Hope this helps.
aLRip