Server timeout after invite receive

[Leonos]

Due to NLB our system has two IP's bound to one nic.
192.168.10.50 and 192.168.10.52

We receive the invite on 192.168.10.50 and apparantly this is forwarded to 192.168.10.52 and it's recognized by our voip.

Then our voip on 192.168.10.52 responses to the invite, but gets a 504 timeout.

the wireshark looks like this:

(5060) provider --> 192.168.10.50 --> 192.168.10.52 INVITE (5060)
(5060) provider <-------------------- 192.168.10.52 100 TRYING
(5060) provider <-------------------- 192.168.10.51 SERVER TIMEOUT

We cannot make our voip listen to one IP only.