A few ideas here:
If you’re using NT, create a security rule that only permits certain IP address and blocks the rest WHILE you’re gaming. Once the rule is created, you can turn it on and off with a few clicks.
Can you change the port? If so, use a different port in the upper range. Using a high range, such as 45555 makes it very time consuming for someone to scan your ports looking for an open IP.
What server are you using and what is the game? Perhaps I can provide more information.