MICROS RES 3700: IFS error status -20 / IFS_RECEIVE_TIMEOUT / IFS_PROXY_ERROR / 0xc70700d7 1

[JimFromCanada]

These errors will appear in OPS when a SIM event is fired.
The BOH was replaced quite recently and the configurator seems to keep reverting some settings for the IP addresses. At one point, using hostnames for the interfaces seemed to work for a few days but it just stopped working again.
The steps to reproduce are to send an event to a SIM server (that's it).

There are a few other mentions of this (or a similar) issue:

https://www.tek-tips.com/viewthread.cfm?qid=1748111

/ "Micros 3700 "IFS error status -20""

https://www.tek-tips.com/viewthread.cfm?qid=1816466

/ "IFS status error -20"

https://www.tek-tips.com/viewthread.cfm?qid=1813132

/ "RES 3700 Unable to print reports at Workstations"
These are somewhat similar but there was either no reply or it was seemingly-unrelated.

The first error was slightly different and then the subsequent errors are all the same.

This is the first instance in the 3700d.log:

<Date in question> | UWS06           | MDSHTTPTransprt |        0 | CMDSHTTPTransport::DoRequest_S | 169736 | MDS_Transact : Trace [0x3418ABFE] | *** ERROR *** Select returned nothing to read. | 
<Date in question> | UWS06           | MDSHTTPTransprt |        0 | CMDSHTTPTransport::DoRequest_S | 169736 | - : 		 Trace [0x3418ABFE] | Error Code = [0xc70700d7] | 
<Date in question> | UWS06           | OPS             |        0 | IFS error [IFS_PROXY_ERROR] | 
<Date in question> | MICROS1         | KDSController   |        0 | KDSC SuborderDone - Suborder was bumped.  Order[11:20240622153656982] suborder[10716] nodename[192.168.XXX.XXX] check[14918] table[0] | 
<Date in question> | MICROS1         | IFS             |        0 | Interface [interface name here] id [9] receive timeout | 
<Date in question> | MICROS1         | IFS             |        0 | Interface [interface name here] receive failed for id [000000009] | 
<Date in question> | MICROS1         | MDSIFSAdapter   |        0 | IfsReceive COM Error [0x80004005] | 
<Date in question> | MICROS1         | MDSIFSAdapter   |        0 | Timeout waiting for interface [interface name here] on server [MICROS1] | 
<Date in question> | MICROS1         | MDSIFSAdapter   |        0 | IFS error [IFS_RECEIVE_TIMEOUT] |

Note how it has a proxy error followed by a timeout.
From here on out, the errors are just the 'receive timeout' one.

The components involved are primarily IFS, MDSHTTPTransprt, and MDSIFSAdapter.

Restarting all workstations, BOH, and even reloading the DB does not yield any resolution unfortunately.

Again, perhaps it's just a red herring but when viewing entities that have IP addresses in POS Configurator it seems as though POS Configurator was reverting changes on its own where the hosts files were concerned.

Despite attempting the same fix (force a hostname instead of the IP address for the PMS interface / SIM def) when this happened again, nothing seems to resolve the issue.

Is there a quick fix for this, or is this an Oracle Support-only type of situation?

 

[JimFromCanada]

Update: it appears as though this can also happen mid-flight. We originally thought it was only a "while the system is idle even if only for a few seconds" issue. This was an incorrect assumption.

Our SIM message processor (in our system) received a message, processed it, and then as it was issuing a response the server had already closed the connection.

Our .NET-based server observed a SocketException indicating that "An existing connection was forcibly closed by the remote host". For posterity, that was the inner exception contained within an IOException indicating that it was "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

This seems extra odd because it means the communications were working just fine and during an actual communication, the RES 3700 side of the comms decided to exit outright. Given the errors in the 3700d.log it seems as though IFS is not running at all, yet the issue is not resolved after restarting services or the BOH (and terminals) outright.

 

[JimFromCanada]

I found a potential source of the problem.
The server was quite sluggish and while attempting to inspect the 3700d.log file it became obvious that the logs for our interface were rather large (4GB). I did not get a chance to look at the log retention settings but we usually do not configure "Log Transactions" at the interface level yet it was enabled at this site (and others).

Other sites of ours also have the 'transactions' log at 4GB but seeing as this site in particular was struggling even within POS Configurator (it indicated it was unresponsive and even attempting to close a dialog took nearly 10 seconds), it seems like the disk was the bottleneck.

After turning OFF the "Log Transactions" in the interface settings it appears as though things began working again.

TL;DR: If you hit odd interface errors be sure to check for "Log Transactions" at the interface definition level to make sure you are not logging things needlessly / unexpectedly and slowing down the system by having it attempt to roll the 4GB log file with every message that comes in or goes out via your SIM.