Had to read this a couple of times to see what it was that you really wanted to do...
What it looks like you are trying to do is to place two NICs inside of the unit, one facing in to handle your e-mail users and one facing out to handle outbound SMTP traffic.
That being said, obviously if you are doing this, you have a seperate network IP segment that faces in and a seperate network segment that faces outward. You probably are NATing the outward facing address, so the route is not apparent to the network and the OS has no idea where to send the packet.
I am assuming that you already have your Foreign SMTP document created and a connection document that points to the Foreign SMTP document. If not, you need to look this up in the help file and create these, so that SMTP will work in the first place.
Assuming that these are in place, all you need to do to Domino is change the connection document to point at the correct IP for your external SMTP relay, then start and stop the router task.
Now then, this fixes the issue in Domino, but you need to correct the issue in the OS.
What you need to do is add a persistent route that indicates that this one external IP address is accessed through the outward facing NIC. I'm not certain as to the OS400 statement for this, but in the Windows environment it would be:
route add -p 12.64.80.51 MASK 255.255.255.255 192.168.1.1
where: -p makes the connection persistent, so that it will remain after a reboot,
12.64.80.51 is the single IP for your external relay,
255.255.255.255 is the mask that specifies this single explicit address rather than a range,
and 192.168.1.1 is the outward facing NIC.
Every OS has something to this effect, so that you can specify an interface to try when the route has not built into the internal route tables. By specifying an explicit route, the OS will always try to send any packet it receives for the external relay to the outbound interface.
Hope this helps.
Leo L'Homme, PCLP