Trojans are back after 10 days, and overloading entire network again

[1LUV1T]

Hello all, I've posted here before regarding trojans found on my Win2003 server...

http://www.tek-tips.com/viewthread.cfm?qid=1567911

Thought the problem was over since I removed the suspicious and identified files that were causing the issue.

Last night (10 days later) the problem reappeared in same manner; At 6:05pm, bandwidth spiked 100% and continued through the night leaving all my computers without internet and virtually inaccessible. Trojans were detected by Hitman Anti-malware software, otherwise TrendMicro didn't even blink.

All trojans were located in c:\windows\system32 and were named as 2.exe, 3.exe, and 4.exe.

Consequently, I ran antivirus, malware-bytes, hijackthis and hitman pro.


Logfile of Trend Micro HijackThis v2.0.2
Scan saved at 7:58:47 AM, on 9/16/2009
Platform: Windows 2003 SP2 (WinNT 5.02.3790)
MSIE: Internet Explorer v8.00 (8.00.6001.18702)
Boot mode: Normal

Running processes:
C:\WINDOWS\System32\smss.exe
C:\WINDOWS\system32\winlogon.exe
C:\WINDOWS\system32\services.exe
C:\WINDOWS\system32\lsass.exe
C:\WINDOWS\system32\svchost.exe
C:\WINDOWS\System32\svchost.exe
C:\WINDOWS\system32\spoolsv.exe
C:\Program Files\Symantec\Backup Exec\NT\dlomaintsvcu.exe
C:\WINDOWS\System32\svchost.exe
C:\Program Files\Java\jre6\bin\jqs.exe
C:\Program Files\Trend Micro\OfficeScan Client\ntrtscan.exe
f:\oracle9i\bin\omtsreco.exe
f:\oracle9i\bin\agntsrvc.exe
f:\oracle9i\BIN\TNSLSNR.exe
C:\WINDOWS\system32\cmd.exe
f:\oracle9i\bin\dbsnmp.exe
f:\oracle9i\bin\ORACLE.EXE
C:\Program Files\Trend Micro\RUBotted\TMRUBotted.exe
C:\WINDOWS\System32\snmp.exe
C:\WINDOWS\system32\svchost.exe
C:\WINDOWS\System32\svchost.exe
C:\Program Files\Trend Micro\OfficeScan Client\tmlisten.exe
C:\Program Files\Trend Micro\OfficeScan Client\TmPfw.exe
C:\Program Files\UltraVNC\WinVNC.exe
C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe
C:\WINDOWS\TEMP\IZ599A.EXE
C:\WINDOWS\System32\svchost.exe
C:\Program Files\Trend Micro\OfficeScan Client\CNTAoSMgr.exe
C:\WINDOWS\Explorer.EXE
C:\Program Files\Trend Micro\OfficeScan Client\pccntmon.exe
C:\Program Files\Trend Micro\RUBotted\TMRUBottedTray.exe
C:\Program Files\YourWare Solutions\FreeRAM XP Pro\FreeRAM XP Pro.exe
C:\Program Files\Symantec\Backup Exec\RAWS\vxmon.exe
C:\WINDOWS\system32\ctfmon.exe
C:\WINDOWS\system32\winlogon.exe
C:\WINDOWS\system32\userinit.exe
C:\WINDOWS\system32\rdpclip.exe
C:\WINDOWS\Explorer.EXE
C:\Program Files\Trend Micro\OfficeScan Client\pccntmon.exe
C:\Program Files\UltraVNC\WinVNC.exe
C:\Program Files\Hitman Pro 3.5\HitmanPro35.exe
C:\Program Files\NetStat Agent\NetAgent.exe
C:\WINDOWS\system32\ctfmon.exe
C:\Program Files\Secunia\PSI\psi.exe
C:\Program Files\Trend Micro\HijackThis\HijackThis.exe

R1 - HKCU\Software\Microsoft\Internet Explorer\Main,Default_Page_URL = res://shdoclc.dll/softAdmin.htm
R0 - HKCU\Software\Microsoft\Internet Explorer\Main,Start Page = about:blank
R1 - HKLM\Software\Microsoft\Internet Explorer\Main,Default_Page_URL =

http://go.microsoft.com/fwlink/?LinkId=69157

R1 - HKLM\Software\Microsoft\Internet Explorer\Main,Default_Search_URL =

http://go.microsoft.com/fwlink/?LinkId=54896

R1 - HKLM\Software\Microsoft\Internet Explorer\Main,Search Page =

http://go.microsoft.com/fwlink/?LinkId=54896

R0 - HKLM\Software\Microsoft\Internet Explorer\Main,Start Page =

http://go.microsoft.com/fwlink/?LinkId=69157

R1 - HKCU\Software\Microsoft\Internet Connection Wizard,ShellNext =

http://google.com/

O2 - BHO: Java(tm) Plug-In 2 SSV Helper - {DBC80044-A445-435b-BC74-9C25C1C588A9} - C:\Program Files\Java\jre6\bin\jp2ssv.dll
O2 - BHO: JQSIEStartDetectorImpl - {E7E6F031-17CE-4C07-BC86-EABFE594F69C} - C:\Program Files\Java\jre6\lib\deploy\jqs\ie\jqs_plugin.dll
O4 - HKLM\..\Run: [OfficeScanNT Monitor] "C:\Program Files\Trend Micro\OfficeScan Client\pccntmon.exe" -HideWindow
O4 - HKLM\..\Run: [WinVNC] "C:\Program Files\UltraVNC\WinVNC.exe" -servicehelper
O4 - HKLM\..\Run: [TMRUBottedTray] "C:\Program Files\Trend Micro\RUBotted\TMRUBottedTray.exe"
O4 - HKLM\..\Run: [HitmanPro35] "C:\Program Files\Hitman Pro 3.5\HitmanPro35.exe" /scan:boot
O4 - HKCU\..\Run: [VxBeMon] "C:\Program Files\VERITAS\Backup Exec\RANT\vxmon.exe"
O4 - HKCU\..\Run: [NetAgent] C:\Program Files\NetStat Agent\NetAgent.exe
O4 - HKCU\..\Run: [ctfmon.exe] C:\WINDOWS\system32\ctfmon.exe
O4 - HKUS\S-1-5-19\..\RunOnce: [tscuninstall] %systemroot%\system32\tscupgrd.exe (User 'LOCAL SERVICE')
O4 - HKUS\S-1-5-20\..\RunOnce: [tscuninstall] %systemroot%\system32\tscupgrd.exe (User 'NETWORK SERVICE')
O4 - HKUS\S-1-5-21-4092164673-791646736-449835827-1004\..\Run: [FreeRAM XP] "C:\Program Files\YourWare Solutions\FreeRAM XP Pro\FreeRAM XP Pro.exe" -win (User 'gcs')
O4 - HKUS\S-1-5-18\..\RunOnce: [tscuninstall] %systemroot%\system32\tscupgrd.exe (User 'SYSTEM')
O4 - HKUS\.DEFAULT\..\RunOnce: [tscuninstall] %systemroot%\system32\tscupgrd.exe (User 'Default user')
O4 - Startup: Secunia PSI.lnk = C:\Program Files\Secunia\PSI\psi.exe
O15 - ESC Trusted Zone:

http://runonce.msn.com

O16 - DPF: {076169AA-8C3D-4CFC-AC23-3ACA88FC21B5} (F-Secure Online Scanner Launcher) -

http://download.sp.f-secure.com/ols/f-secure-rtm/resources/fslauncher.cab

O16 - DPF: {215B8138-A3CF-44C5-803F-8226143CFC0A} (Trend Micro ActiveX Scan Agent 6.6) -

http://housecall65.trendmicro.com/housecall/applet/html/native/x86/win32/activex/hcImpl.cab

O16 - DPF: {6414512B-B978-451D-A0D8-FCFDF33E833C} (WUWebControl Class) -

http://update.microsoft.com/windowsupdate/v6/V5Controls/en/x86/client/wuweb_site.cab?1245883567517

O16 - DPF: {73ECB3AA-4717-450C-A2AB-D00DAD9EE203} (GMNRev Class) -

http://h20270.www2.hp.com/ediags/gmn2/install/HPProductDetection2.cab

O16 - DPF: {A90A5822-F108-45AD-8482-9BC8B12DD539} (Crucial cpcScan) -

http://www.crucial.com/controls/cpcScanner.cab

O17 - HKLM\System\CCS\Services\Tcpip\Parameters: Domain = MYDOMAIN.COM
O17 - HKLM\Software\..\Telephony: DomainName = MYDOMAIN.COM
O17 - HKLM\System\CCS\Services\Tcpip\..\{155499C4-4503-4D95-B5F0-E8F9689E64C7}: NameServer = 192.168.1.2,192.168.1.3
O17 - HKLM\System\CS1\Services\Tcpip\Parameters: Domain = MYDOMAIN.COM
O17 - HKLM\System\CS1\Services\Tcpip\..\{155499C4-4503-4D95-B5F0-E8F9689E64C7}: NameServer = 192.168.1.2,192.168.1.3
O23 - Service: pcAnywhere Host Service (awhost32) - Symantec Corporation - C:\Program Files\Symantec\pcAnywhere\awhost32.exe
O23 - Service: Backup Exec Remote Agent for Windows Systems (BackupExecAgentAccelerator) - Symantec Corporation - C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe
O23 - Service: Backup Exec DLO Maintenance Service (DLOMaintenanceSvc) - Symantec Corporation - C:\Program Files\Symantec\Backup Exec\NT\dlomaintsvcu.exe
O23 - Service: Java Quick Starter (JavaQuickStarterService) - Sun Microsystems, Inc. - C:\Program Files\Java\jre6\bin\jqs.exe
O23 - Service: OfficeScanNT RealTime Scan (ntrtscan) - Trend Micro Inc. - C:\Program Files\Trend Micro\OfficeScan Client\ntrtscan.exe
O23 - Service: OracleClientCache80 - Unknown owner - f:\orant\BIN\ONRSD80.EXE
O23 - Service: OracleMTSRecoveryService - Oracle Corporation - f:\oracle9i\bin\omtsreco.exe
O23 - Service: Oracleoracle9iAgent - Oracle Corporation - f:\oracle9i\bin\agntsrvc.exe
O23 - Service: Oracleoracle9iClientCache - Unknown owner - f:\oracle9i\BIN\ONRSD.EXE
O23 - Service: Oracleoracle9iHTTPServer - Unknown owner - f:\oracle9i\Apache\Apache\apache.exe
O23 - Service: Oracleoracle9iPagingServer - Unknown owner - f:\oracle9i/bin/pagntsrv.exe
O23 - Service: Oracleoracle9iSNMPPeerEncapsulator - Unknown owner - f:\oracle9i\BIN\ENCSVC.EXE
O23 - Service: Oracleoracle9iSNMPPeerMasterAgent - Unknown owner - f:\oracle9i\BIN\AGNTSVC.EXE
O23 - Service: Oracleoracle9iTNSListener - Unknown owner - f:\oracle9i\BIN\TNSLSNR.exe
O23 - Service: OracleServiceA2000 - Oracle Corporation - f:\oracle9i\bin\ORACLE.EXE
O23 - Service: OracleServiceA2TEST - Oracle Corporation - f:\oracle9i\bin\ORACLE.EXE
O23 - Service: Trend Micro RUBotted Service (RUBotted) - Trend Micro Inc. - C:\Program Files\Trend Micro\RUBotted\TMRUBotted.exe
O23 - Service: OfficeScan NT Listener (tmlisten) - Trend Micro Inc. - C:\Program Files\Trend Micro\OfficeScan Client\tmlisten.exe
O23 - Service: OfficeScanNT Personal Firewall (TmPfw) - Trend Micro Inc. - C:\Program Files\Trend Micro\OfficeScan Client\TmPfw.exe
O23 - Service: OfficeScan NT Proxy Service (TmProxy) - Trend Micro Inc. - C:\Program Files\Trend Micro\OfficeScan Client\TmProxy.exe
O23 - Service: VNC Server (winvnc) - UltraVNC - C:\Program Files\UltraVNC\WinVNC.exe

--
End of file - 7909 bytes

P.S. Please disregard the WinVNC as it is a necessary service and the Oracle services.

 

[Roadki11]

I would start by looking at who was on vacation last week and had their workstation shutdown or who has a laptop. The 10 day time frame would make that logical. Might be the source of your reinfection. What services do you expose to the internet? I know older versions of VNC had an authentication bypass bug that would leave your network exposed. Or you could have a naughty web user that needs to satisfy his porn craving every ten days.

RoadKi11

"This apparent fear reaction is typical, rather than try to solve technical problems technically, policy solutions are often chosen." - Fred Cohen

 

[1LUV1T]

I'm positive that something triggered the infection again. However, I am stumped at where to begin looking. Event Viewer doesn't show much. Also, this particular server that is infected again, NO ONE, has access to it. It is an application server that hosts an Oracle service (that workstations access through a shared drive). But no one, has access to the root (C drive which is where the infection is.

 

[Lemon13]

at the time of the scan, why was
C:\WINDOWS\system32\cmd.exe
C:\WINDOWS\TEMP\IZ599A.EXE
running?

you didnt do a restore from a good known backup, right?!

check event-viewer, is there something suspicious logged just before the time it started again?

like Roadki11 posted, the problem might be a client that wasnt connected to the network for 10 days, but since you only disinfected the server you cant say for sure.

again, i never would repair a server, use your last good-known backup or do a fresh install

M. Knorr

MCSE, MCTS, MCSA, CCNA

 

[1LUV1T]

@Lemon13: I can explain the CMD.EXE running but what definitely did not look right was the presence of C:\WINDOWS\TEMP\IZ599A.EXE. I checked that file, however, and it seemed to pass for Clean and seemed like it is owned by TrendMicro (my antivirus). Still, doesn't seem right that each bootup, TM is pulling a file from a TEMP directory?!

Event Viewer doesn't show anything in particular. Just standard error or service messages.

I guess my question is, how do I figure out where the source of the infection is? I keep deleting trojans as they are detected, problem goes away after reboot, but as today showed the infection reappears with no warning?

Is there a way to trace something like this?

 

[Lemon13]

get a bootable cd/dvd like avira rescue disk

http://www.free-av.com/en/products/12/avira_antivir_rescue_system.html

burn it on cd, boot the server ........, when done do it with your clients too

meanwhile read about auditing

http://support.microsoft.com/?scid=kb;en-us;814595&x=15&y=10

M. Knorr

MCSE, MCTS, MCSA, CCNA

 

[1LUV1T]

Ah ha! The first crack in my case. So in my effort to track down the source of the infection, I had instructed all my users to shutdown their workstations for the night. My thinking was that if the infections exists on a workstation it would inoperable until the workstation turns on again. Sure enough, right around the start of business day 12 hours later, the infection came back. Thus, my hypothesis is that it exists on a workstation and targets other devices on the network when awake.

@Lemon13: whatever I have, the antiviruses cannot "see it". Only Hitman (cloud computing based spyware detector) seems to be detecting it.

 

[GrimR]

The temp file is a Trend Micro file, why are there two instances of explorer running is my concern. Are there no DEP errors on startup or strange dos box flashes?

MCITP:EA/SA, MCSE, MCSA, MCDBA, MCTS, MCP+I, MCP

 

[1LUV1T]

@GrimR: Good catch on the two explorers running, however, the explanation for that is - at the time of running HijackThis, I was logged in twice on the effected server (through Console & on RDP). That would explain the two instances of explorer.

I did, however, run a scan on the found trojans and they definately check out as different variants ranging from DLOADER to Eldorado to Crypt. Unfortunately Trend Micro (my network antivirus) doesn't do a thing.

I think the next step, and please let me know if this doesn't make sense, is to install a temporary antivirus on ALL workstations and run a scan that way. Perhaps another antivirus will find the offending workstation? Right now, it's all a shot in the dark.

 

[Lemon13]

I think the next step,...."

it makes no sence to install a antivirus prog on an infected machine, therefor you have the bootable cd to test them.

M. Knorr

MCSE, MCTS, MCSA, CCNA

 

[technome]

Two cents...
Ran into a network taken over by a hacker via an insecure wifi.
The hacker installed rookits or equivalent, which made 3 anti virus scanners ignore his attack (Symantec, Trend, Kaspersky), all 3 caught some of his resident "programs" but missed others. Had the same "downloaders", 2,3 and 4.exe, plus multiple keyloggers, basically the keyloggers gave him all passwords, credit card and bank pin numbers. He went so far as to place programs in the Symantec folders, which were hidden from the OS and scanners via the rootkit. Caught him by logging in at 5 am on a weekend, while he was busy purchasing tons of stuff on Internet sites with all the personal info he obtained. Up until this point, he had created enough issues on the network, I was not looking for a hacker.
Anyway, if you have a rootkit in play, I would not count on the virus scanners. Combofix, Mbam (and any other anti malware programs you can run,more the better) are your best bet, if your not willing to do a total network rebuild . Combofix should run on 2003, on a couple of highly infected machine, Combofix only worked after a number of trojans were removed by other utils and manual deletions. The obvious, if your not running all AV scanners and utils which reveal rootkits from safe mode you have very little, if any, chance of a cleanup. As far as Svchost, if it dies in the process, an over the top repair install should repair it, but letting any util replace it is a last resort. I have cleared a few very infected small networks, including the servers but it is stressful/tedious and can take days ( 18 hour days) to repair the damage. It is gambling, if you miss one piece of malware (their could be dozens), you can lose... so the advice to nuke the network is advised.


........................................
Chernobyl disaster..a must see pictorial

http://www.kiddofspeed.com/default.htm

 

[1LUV1T]

@Lemon13: Makes sense, I will download the Avira AntiVir bootable CD and scan at boot-time. Are there any other bootable alware/antivirus apps that you can recommend? Thanks in advance.

technome, thanks for sharing your experience. Sounds very similar to what I am in the process of going through. It is definately hell week for me as the network at random times of the night will be bombarded and I have to come in middle of the night or early morning to restart the infected/effected server. The programs are definately replicating and I cannot keep up. I will follow Lemon's advice and take it offline and do some repair work in Safe Mode (Away from a live system). You mention two things; Combofix you said can run under Windows 2003 but I was not able to. Second, you mention "nuking the network". My network is fairly large, did you mean nuke the server? It doesn't seem like the infection is spreading at this point.

 

[58sniper]

FWIW - any time you have a compromised server, you should disable its Internet access. It could be sending your data out. Kill access to it while you're working on it. Problem for the users? Probably. But a much bigger problem for the org when the financial and customer data gets out.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[1LUV1T]

I understand that just not very feasible during a work week. My plan of attack for this weekend, during downtime, is to use Avira's rescue disk first and see what comes up. Then boot-up in Safe Mode, and ran Malware-bytes and the other plethoria of anti-malware tools for a scan. Third, will try to find other bootcd programs and run those as well. If none of this works, I will have no choice but to format the server and start anew, however, business will be down which is -$$$.

If anyone has any thing to add before I start the plan, please feel free to share.

Thanks for everyone who has replied here and helped me out. Much appreciated.

 

[Lemon13]

I will have no choice but to format the server and start anew, however, business will be down which is -$$$."

so for the next time plan for desaster recovery, first thing to start would be backups of your server......

http://www.techmixer.com/free-bootable-antivirus-rescue-cds-download-list/

M. Knorr

MCSE, MCTS, MCSA, CCNA

 

[58sniper]

I understand that just not very feasible during a work week.

So, business stability be damned? If that machine is compromised, then they ARE getting data. Not stopping that immediately means the company isn't concerned about that data getting out.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[1LUV1T]

@Lemon13: thanks for the excellent link. I've downloaded quite a few live cd's friday, but as of today (saturday), most do not work from bootcd. I am guessing that's based on that fact that the infected server is on a raid5 array? I always use live cds for windows xp/2000 repair work but first time that I am attempting to use on a production server and it's a no-go. I'm reading up on mounting harddisks within command-lines as an alternative method but that's quite time consuming.

@58sniper: I won't argue your point about keeping an infected machine AS IS, but from a business perspective, it is literally hundreds of thousands of dollars per day, if not more, that is being conducted using these servers. This partcular one, is not a file server, or anything that holds sensitive data in plainview. There are no spreadsheets or reports here. It is an application server, and good luck to the hacker, who can download gigs worth of database files and then make sense of it all. Still, the machine is compromised and it will be fixed but not at the expense of business down-time.

 

[Lemon13]

create a windows livecd with bartpe

http://www.nu2.nu/pebuilder/

and include a antivirus solution and the raid driver

M. Knorr

MCSE, MCTS, MCSA, CCNA

 

[Zelandakh]

good luck to the hacker"...database files taking hundreds of thousands per day?

What company do you work for so that we can see if the PCI compliancy people think your recovery model of continuing to trade on a compromised box is a good plan.

Are your databases encrypted enough to pass PCI? If not, you may as well publish all financial details in plain text on your web site.

 

[58sniper]

And if one server is compromised, are the service accounts? No doubt the same service accounts that have rights on other servers?

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/