RE: MICROS RES 3700: IFS error 1

[JimFromCanada]

Regarding these two previous posts from June 2024 and July 2024 (I can't post a follow-up within those topics, likely due to age):

The culprit was seemingly a 'ghost' node which did not exist but was somehow in the database acting as a backup server.

The reason why the IFS errors were so sporadic and inconsistent was due to this 'ghost' server and how often the hosts file would be updated on the terminals.

The processing of any IFS activities would halt seemingly at random and interrupt workflows. It appeared as though messages were actually being redirected to this non-existent IFS server mid-workflow.

For searchability, this primarily manifested itself as 'IFS_ADAPTER_ERROR' but there were other breadcrumbs that were also frequent including 'IFS error status -20', 'IFS_RECEIVE_TIMEOUT', 'IFS_PROXY_ERROR', '0xc70700d7' and more.

There was an issue with having "Log Transactions" enabled at the interface level due to the volume of logging (it would max out the 4GB log file relatively quickly) but the issue was fundamentally due to a non-existent backup server interrupting the IFS requests by having all requests sent to a specific, not-in-use IP address.

 

[pradeepckmeta]

Hi Jim,

I am facing the same issue in a micros 5.7 installation, not sure about the exact minor version.

[0xF678B9F6] | *** ERROR *** Select returned nothing to read. |
Mon Sep 16 20:46:47 2024 | BOTTLESHOP | MDSHTTPTransprt | 0 | CMDSHTTPTransport:oRequest_S | 371134 | - : Trace [0xF678B9F6] | Error Code = [0xc70700d7] |
Mon Sep 16 20:46:47 2024 | BOTTLESHOP | OPS | 0 | IFS error [IFS_PROXY_ERROR] |
Mon Sep 16 20:47:10 2024 | BOTTLESHOP | PrintControllerCE | 0 | printerModelType [EpsonThermal_], printerPrimaryBuModelType [

Observed that this happens exactly 30 seconds after message is sent to external interface server.

It shows a message "select returned nothing to read".

Am trying to solve this by sending heartbeat messages back to ISL every 5 seconds, not sure if it is possible.

Will keep you posted. meanwhile, do let me know if you find something.

 

[JimFromCanada]

When defining the interface there is an option for a timeout, but that timeout determines how long the POS should wait for a SIM server to respond to a message. If your timeout is set to 30 seconds in the POS Configurator then it points towards your server not communicating (outright).

The heartbeat in my previous scenarios was every 1 second, give or take (ISL isn't overly accurate) and it would still encounter issues--the heartbeat was more of a state check to see if the workflow should proceed, but it fired roughly once per second. In other words: a heartbeat will help diagnose the issue but the IFS communications don't require a heartbeat for the communication channels to stay open. Once your SIM server comes online, the IFS server will send a PING message (empty message, basically) that you need to respond to, and it will send a similar PING at certain intervals (every X minutes). That's basically its own internal heartbeat to ensure your SIM server is still alive.

What component was logging the "Select returned nothing to read" message?

If I had to speculate: the SIM server that is attempting to be communicated with is not available, blocked by firewall if not on the same network, or could generally be refusing communication. Is the SIM server accessible to you? It would be worth looking at the server logs to see if it's being reached (at all) and what it thinks the issue is.

TL;DR: Given that there's a database query error involved, and given the precise timing, it "feels" as though it either can't find the details for contacting the SIM server, the SIM server doesn't exist or is misconfigured (address) or the SIM server is refusing communications (outright). If the query that is being attempted is happening against a remote server (not the OPS / CAPS / local database) then that may also be a component to diagnose and debug.

 

[pradeepckmeta]

Thanks a lot, Jim. It was good to see a post regarding this issue, and of course to talk to someone about it.

Regarding the interface timeout, we have set it to 120 seconds.

Messages that are replied to within 30 seconds work fine, so the SIM server is very much accessible. Besides, I am getting requests to my SIM server.

What I am not getting are the heartbeats. I was not aware that I should listen for them and that I should reply to them. Could you point me to the part of the documentation that refers to them? Perhaps this is what I am missing.

Earlier versions of Micros work fine without response to the heartbeat.

 

[JimFromCanada]

I stand corrected: I thought it was mandatory to respond to the PING requests (ACK) but apparently the sample code in the documentation just ignores them outright--the sample code (written in C) in the documentation explicitly ignores the ACK/NACK requests. Huh.

I log these and respond to them to better understand what comms happen and when they're happening, but it doesn't appear mandatory. (TIL)

Regarding the heartbeats: I see them when the SIM server starts up and IFS was already running. I also see them when restarting IFS on the BOH. They're likely not a huge deal--I was more talking about the notion of a heartbeat from within our application where we check on the status of a workflow roughly every 1 second. The PING messages with IFS seemingly come every 5 or 15 minutes. I can't recall the specific interval but it was very (very) precise in its timing.

Regarding the 30 second timeout: again, what component is issuing that database / SQL query error? That error code is seemingly unique to RES/OPS/Micros3700 but I only performed a cursory search. If the timeout you're experiencing is very consistent in nature (30 seconds) then it could also be a built-in timeout for things like database queries in different components (perhaps not even yours).

I would say the more you can log within your own code, the better. It also helps to (even if only temporarily) increase the logging in RES/OPS/Micros to determine if other components are part of the issue.

TL;DR: you don't necessarily need to reply to the PING messages but it does help to at least log them to know what comms are happening and when. You may also want to see if something is failing before even sending your message due to a configuration or communication issue. The fact that it's 30 seconds "smells" like a timeout, and the main hint that something behind the scenes is causing the issue is likely the fact that it is the "PROXY ERROR"--I only saw that when it was timing out and failing to send the message to our server due to the logs being very (very) full (4GB). The "Log Transactions" checkbox was enabled so it kept trying to write every message to that log file and seemingly struggled once it hit the maximum of 4GB (go figure).

 

[pradeepckmeta]

I am also discarding ACK packets apparently

What should I reply to ack? Should I respond with an ACK too?

 

[JimFromCanada]

It's typically just a "passthrough". It will only contain the workstation ID and interface name iirc, and you can just respond with the same properties on your response, or, depending on your implementation, just send the request right back as the response.

Edit: It might actually just be the workstation ID that's sent (no interface name).

 

[JimFromCanada]

@pradeepckmeta: did you manage to resolve the issue? I'm curious what the source of your issues may have been and if it is different from the ones I encountered.

 

[pradeepckmeta]

@pradeepckmeta: did you manage to resolve the issue? I'm curious what the source of your issues may have been and if it is different from the ones I encountered.

Hi Jim,

we are not very sure of what the origin of the error is. We just know that if the external server does not respond within 30 seconds, this error occurs.

Just putting a response to the heartbeat did not work since we receive a heartbeat from micros only once every 5 minutes.

So we put in a mechanism of sending a heartbeat while our server is busy.

That worked and the IFS errors are gone.

Thanks for all the help.