Aloha credit card transactions failing daily

[bbrunning]

We're having a lot of credit card transactions failing and the server is needing to be rebooted daily, sometimes a few times a day. I had them write down the times they experienced this and here's some dumps from the debout.edc

Jun 11, 11:14:55 [2616] Can not access Terminal \\TERM2\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:15:12 [2616] Can not access Terminal \\TERM1\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:16:08 [2616] Found Terminal \\TERM3\BOOTDRV\Aloha while scanning.
Jun 11, 11:16:26 [2616] Can not access Terminal \\TERM3\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:17:02 [2616] Found Terminal \\TERM2\BOOTDRV\Aloha while scanning.
Jun 11, 11:17:19 [2616] Found Terminal \\TERM1\BOOTDRV\Aloha while scanning.
Jun 11, 11:17:20 [2616] Can not access Terminal \\TERM2\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:17:37 [2616] Can not access Terminal \\TERM1\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:18:33 [2616] Found Terminal \\TERM3\BOOTDRV\Aloha while scanning.
Jun 11, 11:18:51 [2616] Can not access Terminal \\TERM3\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:19:26 [2616] Found Terminal \\TERM2\BOOTDRV\Aloha while scanning.
Jun 11, 11:19:43 [2616] Found Terminal \\TERM1\BOOTDRV\Aloha while scanning.
Jun 11, 11:19:44 [2616] Can not access Terminal \\TERM2\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:20:01 [2616] Can not access Terminal \\TERM1\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:20:57 [2616] Found Terminal \\TERM3\BOOTDRV\Aloha while scanning.
Jun 11, 11:21:15 [2616] Can not access Terminal \\TERM3\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:21:50 [2616] Found Terminal \\TERM2\BOOTDRV\Aloha while scanning.
Jun 11, 11:22:07 [2616] Found Terminal \\TERM1\BOOTDRV\Aloha while scanning.
Jun 11, 11:22:08 [2616] Can not access Terminal \\TERM2\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:22:25 [2616] Can not access Terminal \\TERM1\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:23:21 [2616] Found Terminal \\TERM3\BOOTDRV\Aloha while scanning.
Jun 11, 11:23:39 [2616] Can not access Terminal \\TERM3\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 11, 11:27:23 [1684] EdcSvr v5.3.20.233 starting...
Jun 11, 11:27:23 [1684] LoadIberStrings() - Loaded version 5.3.20.0, IBERSTR.DLL
Jun 11, 11:27:23 [1684] EDCSVR Debug=0
Jun 11, 11:27:23 [1684] pms type = 0
Jun 11, 11:27:23 [1684] pms debug = 0
Jun 11, 11:27:23 [1684] pms ExtendedRecovery = 0
Jun 11, 11:27:23 [1684] HITISUser = 'Aloha'
Jun 11, 11:27:23 [1684] HITISPassword = 'Aloha'
Jun 11, 11:27:23 [1684] HITISServer = ''
Jun 11, 11:27:23 [1684] HITISInterface = 0
Jun 11, 11:27:23 [1684] HITISPostSession = ''
Jun 11, 11:27:23 [1684] HITISPostDetail = ''
Jun 11, 11:27:23 [1684] Duplicate Credit/Debit REQ file checking disabled.
Jun 11, 11:27:23 [1684] Duplicate Gift card REQ file checking disabled.
Jun 11, 11:27:24 [1684] InitEDC completed
Jun 11, 11:27:24 [1684]
Jun 11, 11:27:24 [2068]
Jun 11, 11:27:24 [1684] POS is ON
Jun 11, 11:27:24 [2068] Begin Request Manager thread...
Jun 11, 11:27:24 [2068] EDCSVR Debug=0
Jun 11, 11:27:24 [2068] pms type = 0
Jun 11, 11:27:24 [2068] pms debug = 0
Jun 11, 11:27:24 [2068] pms ExtendedRecovery = 0
Jun 11, 11:27:24 [2068] HITISUser = 'Aloha'
Jun 11, 11:27:24 [2068] HITISPassword = 'Aloha'
Jun 11, 11:27:24 [2068] HITISServer = ''
Jun 11, 11:27:24 [2068] HITISInterface = 0
Jun 11, 11:27:24 [2068] HITISPostSession = ''
Jun 11, 11:27:24 [2068] HITISPostDetail = ''
Jun 11, 11:27:24 [2068] First Data IPN : Trying to connect...
Jun 11, 11:27:24 [2068] First Data IPN : Trying to connect...
Jun 11, 11:27:24 [2068] CEngine: Read in 3 terminals
Jun 11, 11:27:24 [2068] Visanet: GetProcessorParms() - m_bUsingSockets is OFF because SSL is selected.
Jun 11, 11:27:24 [2068] Create new internet session!!
Jun 11, 11:27:24 [2068] Visanet: GetProcessorParms() - m_bUsingSockets is OFF because SSL is selected.
Jun 11, 11:27:24 [2068] Create new internet session!!
Jun 11, 11:27:25 [2068] ARequestManager:reProcessFiles-- pre-processing files...
Jun 11, 11:27:25 [2068] Starting IPN Manager. [0]
Jun 11, 11:27:25 [2156] Begin Modem Manager thread for modem 1...
Jun 11, 11:27:25 [2068] Starting IPN Manager. [1]
Jun 11, 11:27:25 [2068] Starting SSL Manager. [0]
Jun 11, 11:27:25 [2068] Starting SSL Manager. [1]
Jun 11, 11:27:25 [2068] RequestManager: Read in 3 terminals
Jun 11, 11:27:25 [2068] RequestManager: Read in 1 stores
Jun 11, 11:27:25 [2068] RequestManager: Monitoring 3 terminals with SpoolChecker...
Jun 11, 11:27:25 [2176] Begin Spool Checker thread...
Jun 11, 11:27:25 [2068]
Jun 11, 11:27:25 [2068] Begin main loop...
Jun 11, 11:27:25 [2156] CEngine::OrderStoredTxnsByTime() - Sorting...0 txns
Jun 11, 11:27:25 [2156] CEngine::OrderStoredTxnsByTime() - Sorting...0 txns
Jun 11, 11:27:25 [2156] CEngine::OrderStoredTxnsByTime() - Sorting...0 txns
Jun 11, 11:27:25 [2160] CEngine::OrderStoredTxnsByTime() - Sorting...0 txns
Jun 11, 11:27:27 [2064] The number of clients connected now is: 0
Jun 11, 11:35:18 [2068] ARequestManager:rocessRequestFile: FOUND: C:\Aloha\EDC\10000006.REQ, size = 607
Jun 11, 11:35:18 [2068] ARequestManager:rocessRequestFile: --1 about to process .req file: C:\Aloha\EDC\10000006.REQ
Jun 11, 11:35:18 [2068] ARequestManager:rocessTxnFile --2 txn stored. (renamed to HLD)
Jun 11, 11:35:18 [2068] ARequestManager:rocessTxnFile --3 .req file processed.
Jun 11, 11:35:18 [2164] CEngine:reProcessTxn() begin
Jun 11, 11:35:18 [2164] CEngine:rocessStoredTxn() - after 'PreProcessTxn(parms)
Jun 11, 11:35:19 [2164] ResponseBuf = 1AP0035170611A014E1621635611622212DTC 000000000000000000000000
Jun 11, 11:35:19 [2164] CEngine:rocessStoredTxn() - after 'ProcessTxn(parms)
Jun 11, 11:35:19 [2164] LastTxnNumber before increment is: 0
Jun 11, 11:35:19 [2164] LastTxnNumber after increment is: 1
Jun 11, 11:35:19 [2164] Writing C:\Aloha\EDC\CES2\TXN00001...


I also noticed these errors around the same times:


Jun 13, 01:06:35 [2036] pms mode = 0
Jun 13, 01:06:35 [2036] Failed to open port, status -1
Jun 13, 01:06:35 [2036] Failed to initialize port (1)
Jun 13, 01:06:35 [2036] Failed to initialize port (1)
Jun 13, 01:06:35 [2036] CEngine:rocessStoredTxn() - after 'ProcessTxn(parms)
Jun 13, 01:06:35 [2036] LastTxnNumber before increment is: 15
Jun 13, 01:06:35 [2036] LastTxnNumber after increment is: 16
Jun 13, 01:06:35 [2036] Writing C:\Aloha\EDC\CES2\TXN00016...
Jun 13, 01:06:35 [2036] ProcessStoredTxn(): WRITE: C:\Aloha\EDC\10000083.WRK
Jun 13, 01:06:35 [2036] ProcessStoredTxn(): WROTE: C:\Aloha\EDC\10000083.WRK, size = 632
Jun 13, 01:06:35 [2036] ProcessStoredTxn(): RENAME C:\Aloha\EDC\10000083.ANS
Jun 13, 01:06:35 [2036] |: CEngine:rocessNextStoredTxn() - END - Modem Manager
Jun 13, 01:06:35 [2036] |: ----------------------------------------
Jun 13, 01:06:35 [2036] |:
Jun 13, 01:06:50 [616] Can not access Terminal \\TERM1\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 13, 01:06:52 [616] Found Terminal \\TERM3\BOOTDRV\Aloha while scanning.
Jun 13, 01:07:09 [616] Can not access Terminal \\TERM2\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 13, 01:07:10 [616] Can not access Terminal \\TERM3\BOOTDRV\Aloha for scanning. Check your permissions and your network connection.
Jun 13, 01:07:19 [4068] ARequestManager:rocessRequestFile: FOUND: C:\Aloha\EDC\10000084.REQ, size = 619
Jun 13, 01:07:19 [4068] ARequestManager:rocessRequestFile: --1 about to process .req file: C:\Aloha\EDC\10000084.REQ
Jun 13, 01:07:19 [4068] ARequestManager:rocessTxnFile --2 txn stored. (renamed to HLD)
Jun 13, 01:07:19 [4068] ARequestManager:rocessTxnFile --3 .req file processed.
Jun 13, 01:07:19 [3376] |: StoredTxn corresponding to FileNumber (10000084) deleted
Jun 13, 01:07:19 [3376] |:
Jun 13, 01:07:19 [3376] |: ----------------------------------------
Jun 13, 01:07:19 [3376] |: CEngine:rocessNextStoredTxn() - START - IPN Manager
Jun 13, 01:07:19 [3376] |:
Jun 13, 01:07:19 [3376] |: CEngine:rocessStoredTxn() FileNumber = 10000084 - begin.

Jun 13, 01:07:19 [3376] |: WrapMessage(2) - len after = 132
Jun 13, 01:07:19 [3376] |: ADatawire::SendTransaction() begin.
Jun 13, 01:07:19 [3376] |: SimpleTransaction() returned 50016
Jun 13, 01:07:19 [3376] |: ADatawire:etermineTypeOfError() Received status / error code 50016
Jun 13, 01:07:19 [3376] |: Error Code (50016) Server not available at specified IP Address/Port.
Jun 13, 01:07:19 [3376] |: **CEngine::SendDatawireMessage - Failed to SendMessage**
Jun 13, 01:07:19 [3376] |: Will attempt to Send this Transaction Via Modem
Jun 13, 01:07:19 [3376] |:
Jun 13, 01:07:19 [3376] |: SendDefaultMessage() - RESULT = 50016
Jun 13, 01:07:19 [3376] |:
Jun 13, 01:07:19 [3376] Adding transaction to modem backup list
Jun 13, 01:07:19 [3376] |: Will try modem if it is configured...

Any ideas on what's causing our CC processes to stop working?
Note: We are using AT&T U-Verse now instead of Nuvox. Not sure if anyone has had errors with U-Verse or not.

 

[coorsman]

To check for internet connection problems I usually run a ping test to google.

Open a command prompt on your BOH.
type in

ping google.com -t

and let this run continuoiusly, look for any timeouts.

If you have some timeouts then call your Internet provider, your internet is not stable and is causing your problems.





Cheers,
Coorsman

 

[bbrunning]

Thanks for the quick response coorsman. I've set the server to ping and output to a text file. I also set up nagios to monitor their internet from outside. Looks like the waiting game again...

 

[coorsman]

I had issues all week about EDC not working - found some interesting information today about Visanet and noticed all my calls were related to Visanet as a processor and windows 2000 and XP without auto update of the windows operating system.

This is what I found:

Tech at Visanet told us on Monday they have been slowly migrating their URLs which causes issues with their Verisign Certificates. They also said sites will have issues daily because of this and eventually stop working all together.

Try installing root certificate update available from Microsoft. KB931125 for Windows XP, at least. Could not authorize any cards over SSL until I applied a root certificate update to the system, that I believe was designed for XP but worked on W2000.

But also getting quite a few calls from customers running Windows XP having the problem. Don't think any calls about this from anyone running Win7 or Server2008, because these updates are done automatically.

Cheers,
Coorsman

 

[bbrunning]

Thanks Coorsman,

That could be part of the problem. I found a virus on their system which wasn't even in any databases and a mail server setup called Sockz.conf. I've since removed but they are still having strange issues with citrix and logmein not working. The owner said he was going to have his aloha updated by his vender to the newest version. So, at the moment, I'm staying out of it.

 

[USofAScott]

I dont know if this is related to the post that Coorsman made or not. I have a boh server running Windows XP. automatic updates are active. EDC Processor is Visanet using "VITAL "network"

I am having batches rejected because of a rejected transaction. The Aloha message says that it is Field 19. the processor says that the transaction ID is invalid. All of the successful transactions have an ID of 000000000000000. The rejected transaction has an id that is a 15 digit number. It is the only transaction in the batch that does not have an id of 000000000000000.

Any hints or ids.

 

[tikky212]

Hi coorsman

It's Tikky, you helped me with the Aloha systems before.

I'm in big trouble again, I just got a call from Heartland saying that Visanet is deleting the old URL so we won't be able to process the credit card longer. They gave me the new URL but I have no idea how to change it. Did you know by any chances?

Thank you for you time

Tikky

 

[BIGGBYNinja]

Thank you coorsman for your post about the TSYS tech. We have a store that has been having issues, but is running Aloha EDC 6.4.28 on Windows 7 Professional x86, so the SSL root certificates should be updated.

One of our stores has been having intermittent credit card processing issues since the start of the month. While I am not sure of the actual merchant that the store uses, I do know that it is processed on the Visanet / TSYS processing network according to how it was set up in Aloha EDC. The majority of cards are processing correctly without any issues. When the issue does occur, the message displayed on the FOH terminal is "Failed to initialize port." When this happens, we stop processing in Aloha EDC, restart the edcsvr service, and then start processing in Aloha EDC. This seems to resolve the issue temporarily. Checking EDC's debug output logs when this issue occurs yields the following:

Jul 17, 14:59:27, [5848], [INFO],,"Could not make SSL connection to Host"

and

Jul 12, 15:21:03, [5892], [INFO],,"|: CEngineVISANET::SendSSLMessage() begin....."
Jul 12, 15:21:03, [5892], [INFO],,"** Trying to connect to ssl2.vitalps.net:5003 timeout 20 secs"
Jul 12, 15:21:04, [5892], [INFO],,"** AVisanetSSL2Client2::TimedConnect OK"
Jul 12, 15:21:24, [5892], [INFO],,"** VisanetSSL2Client Timeout at 20"

This does not really tell us too much other than there is a connection issue. This could be caused by a number of variables. It could be a spotty internet connection at the store, an internet-level networking / routing issue between the store and the processing servers, or it could be caused by issues at the processing end (in Virginia).

To test whether this issue was related to the internet connection at the store, I ran what is called a continuous ping test. This test sends a packet of data every second to a google server and gets a response back. It's a basic "Is my internet working?" test.

I compared the times in the ping test log to when the credit cards failed to make the SSL connection and found there to be no issues with the ping test to google. Every packet that was sent to google when a card failed to process was sent back by google. This tells us that the issue does not appear to be caused by the internet connection at the store level.

14:59:22.232 : Reply[86837] from google.com: bytes=32 time=14.8 ms TTL=53
14:59:23.246 : Reply[86838] from google.com: bytes=32 time=14.1 ms TTL=53
14:59:24.260 : Reply[86839] from google.com: bytes=32 time=13.9 ms TTL=53
14:59:25.275 : Reply[86840] from google.com: bytes=32 time=14.5 ms TTL=53
14:59:26.289 : Reply[86841] from google.com: bytes=32 time=13.7 ms TTL=53
14:59:27.303 : Reply[86842] from google.com: bytes=32 time=14.2 ms TTL=53
14:59:28.316 : Reply[86843] from google.com: bytes=32 time=13.9 ms TTL=53
14:59:29.330 : Reply[86844] from google.com: bytes=32 time=13.4 ms TTL=53
14:59:30.345 : Reply[86845] from google.com: bytes=32 time=14.1 ms TTL=53
14:59:31.359 : Reply[86846] from google.com: bytes=32 time=14.5 ms TTL=53
14:59:32.376 : Reply[86847] from google.com: bytes=32 time=17.4 ms TTL=53

Some research online yielded the following blog entry, for a different point of sale software:

http://www.advanced-business-system...ng-ChargeLogic-Customers-ACTION-REQUIRED.aspx

"TSYS (also known as Vital and Visa Net), a ChargeLogic authorized payment processor, is retiring the URL that ChargeLogic Payments comes preconfigured to use for live transactions. In order to avoid a service interruption, ChargeLogic customers using TSYS, Vital, or VisaNet must configure their system to use the new URL. Please find instructions for making this change in the attached document. Making the change is easy and can be completed in less than five minutes.

The old URL will be retired on August 6th, 2012. The settings change described in Vital URL Retirement Communication 2012 must be completed by that date. We suggest making the change as soon as possible."

http://www.chargelogic.com/blog/wp-...nt-Configuration-Change-Instructions-2012.pdf

This information was corroborated by coorsman in a post above.

The majority of the BIGGBY stores that we support are currently using FirstData for credit card processing and are not experiencing these issues. I know of at least one other store of ours using VisaNet and they reported at least one issue yesterday as well.

By all accounts, it appears that VisaNet (now called TSYS), has made changes on their end that have caused these connection issues. I tried calling TSYS support yesterday and all the agent could confirm were failed batch settlement attempts so she verified she could see the connection issues. I brought up the URL retirement article I had found and she said I would have to contact Aloha support.

At this time, I believe all of the recent credit card processing issues appear to have been caused by changes made on TSYS’s end.

I have tried adding the following line to the EDC.ini but it would not connect.

IPHost=

https://ssl1.tsysacquiring.net

I have also tried changing DNS servers to Google's Public DNS but this did not resolve the issue.

Has anybody else had issues like this and resolved them?

 

[ltso]

I continue to have connection problems with TSYS Acquiring at 2 locations. If I open up the firewall I get connected and authorizations. If I lock down the firewall but whitelist all the urls and ip's I received from my processor, opening ports 443 and 5003, I can see that the firewall passes the connection out, but it seems that the response cannot get back in. My processor assures me that the response is coming back on the same ip and port it goes out on. I am wondering if there is an EDC setting I am missing.
The server is 2008, the firewall is PFSense. Anyone have any ideas? I am currently on dial up.

 

[BIGGBYNinja]

I contacted our Aloha dealer and asked if they had any issues with customers processing cards over Visanet / TSYS and he was able to email this PDF:

http://www.mediafire.com/view/?yg5bhmddu8xgs8u

Unfortunately, it looks like it is saying we shouldn't have to do anything.

Note: The VirtualNet SSL domains are not changing. Most Merchants should
not have to make any changes to processing parameters, when using a
TSYS Acquiring Solutions domain name (such as ssl2.vitalps.net) for
SSL Gateway access

So then I contacted TSYS support (again) and this time I was able to get them to send me some SSL certificates that they said I should install and may help. Those are here:

http://www.mediafire.com/?ysrx2jsvzhc7d87

I will be installing these at our 2 sites still using VisaNet that have been having issues, despite both using Windows 7.

 

[djoeg]

We are also still having on-going issues with the EDC, be it on Win XP or Win 2007.
"BiggyNinja" - Were you able to get the CC issue sorted out with the SSL certificate?
I believe radiant need to update there SSL library that EDC uses to accomodate the IP address changes from Visanet, but I have but havent had any luck with further info from our Aloha dealer. Does anyone have an update?

 

[dauphin2]

Just so everyone knows, the whole domain change thing was pushed back until next year, too many issues with the new TSYS domain.

Visanet Domain Change

 

[BIGGBYNinja]

I installed the certificates at one of our sites but they called back the following day with the same issue. I think it's all down to changes being made at TSYS' end and there is nothing we can do about it. We only have two franchises currently processing with VISANET, so I recommended switching to FirstData if the problems persist.

Thanks for the update, dauphin2.

 

[djoeg]

Just an update on this issue. we finally made progress by upgrading EDC to version 7.0
I tested the upgrade at one of our stores that were having major CC issues, (having to restart EDC every 30 mins.) Tested for a week and we never had any further problems. Right now upgrading all the other stores.

I personally dont think Radiant are in any rush to make these updates available too quickly, (they let you stew a while) one of the reasons is they have there own Processor "Radiant Payment Services" (RPS) which they make money on. Tsys is a competitor

 

[djoeg]

Just an Update ....its been about 4 weeks since running the new version of EDC ver 7.0. I mentioned in my previous post that we had no further problems, well that was true then, but seems that although there was an big improvement, we are still having some issues ("Could not make SSL connection to host") especially busy times on a Friday / Saturday.

Coorsman - did installing the root certificate help ?

Has anyone had further issues or resolutions to this - any feed back is apprciated.

thanks

 

[BIGGBYNinja]

We still have occasional issues at our locations using VisaNet (TSYS).

 

[eburks]

I had similar intermittent problems when using heartland via SSL

When I switched back to First Data using the North Platform and Datawire ALL my intermittent problems went away. We sometimes run 350 cards a day at 1 store.

The only drawback was that settling the batch took < 15 sec with Heartland and settling with First Data over DataWire takes 3 - 5 minutes.