How can I analyze this potential security breach on my PIX?

[skhoury]

Hello all,

I have a general question on how to prove that I am being DDOS'ed.

Here is what happened: About 30min ago, we noticed our T1 became considerably slow. So I logged into our PIX to check our bandwidth usage and on the outside it was over 3Mbits/sec! (more than twice a T1).

Then, I started to observe the syslog enteries and we had literaly thousands of attempts at hitting our PIX outside address on port 55727 UDP.

Very unusual behavior. In order to break the flow without killing everyones internet access (completely), I reset all the xlate enteries and the flood seemed to stop...then our bandwidth returned to normal (about 500-700kbits/second on outside).

Any thoughts on how to go about actually researching what happened? How can I tell if this really was a DDOS or if someone was just downloading a huge file?

We don't have an IDS or IPS system, just our PIX515E running 7.0.

Also we have a VPN tunnel between this site and our production site.

Any guidance is much appreciated - many thanks!

Sam

 

[NetworkGhost]

Is the address that was being hit a nat'd address?

 

[skhoury]

No - interestingly enough the address that was being hit was the external address of our PIX, not one of the nat'd bunch.

 

[lgarner]

Are your workstations nat'd to the external address, or to a specific address?

What was the source port of the traffic? That might tell you something. Normal downloads don't use UDP, though.

 

[skhoury]

All our workstations are Pat'd to the external IP address of the PIX. We have such a small block of IPs that we save the others for service like mail etc.

The source port was just a random number - I can't remmeber, but it certinaly wasn't a registered port or anything.

 

[lgarner]

It does sound like it's something replying to an internal workstation at the dynamic NAT address. I don't know how you'd figure out what it is after the fact, but putting something like an IDS on the Pix's inside segment would help track it down in the future.

 

[skhoury]

Yea - I guess I am going to build out an Snort box for the time being.

What is the easiest way to determine who is using the most bandwidth on the network.

I would imagine that would be a good place to start if you suspect an internal user is doing something goofy.

 

[lgarner]

I like Ntop for finding top talkers, protocols, and other stuff.

 

[lashboy]

hi lgarner,

what is Ntop ? can u send any link or some thing for Ntop

 

[lgarner]

http://www.google.com/search?q=ntop,

first link shown.

 

[karmic]

You should setup syslogging to an internal server from your pix unit. At least that'll keep a running log of what's going on. There are many free versions on the net but I usually use the kiwi syslog.

~ K.I.S.S - Don't make it any more complex than it has to be ~

 

[skhoury]

Karmic,

That is what i've done - we generate on average about 300MBs of log files per day.

We also use Kiwi.

Sifting through the logs manually is a pain however and really doesn't tell me to much other than what I stated above.

Is there a better way to analyze the syslog enteries?

 

[lgarner]

http://www.splunk.com/

looks promising. I've played around with it a little.

 

[karmic]

Ok, my stupid, didn't read your first post well enuf.

Little story for you, maybe something, maybe nothing...

I got a call from one of my clients a while back, switches were locked up (actually maxxed out). They have a managed, transparent T1 and a VPN linkup to their head office in central canada, they receive internet access thru this vpn linkup. Head office controlled the security. The difference is, they did NOT have a decent firewall in place like they do now.

Anyhow, the server (and just the server) kept getting hit with a virus and they were hacked in a really big way. When the smoke cleared, head office knew nothing, there was nothing in their logs to show anything. Yet someone basically uploaded the server. (I had no control over security).

I installed another router at this point to provide a gateway out of our network, just to see what was going on. Half the world was surfing thru the T1, both incoming and outbound. Go figure.

Come to find out, the ISP (Bell) did "something wrong with their routing" on their end, and left the managed T1 wide open. They admitted to this suprisingly. The virus kept hitting the server and finally infected it, allowing the hack. If I remember correctly, the breach happened between the company and the bell gateway if that makes sense. Bell didn't see the hack or log any IP's (like they should have), just that the line was up and down due to huge traffic volume.

Good luck in your search. It's frustrating but fun.

~ K.I.S.S - Don't make it any more complex than it has to be ~

 

[fs483]

Doesn't flushing the xlates kill any open connections ?

 

[skhoury]

yes, that is why I ran clear xlate.

I wanted to kill the connections but not by rebooting the firewall.

Most people don't even notice the broken connections when you clear the translations tables, whereas my help desks phone doesn't stop ringing when the firewall is rebooted...not sure why.