Traffic surge halting all access to servers

[MS1441]

Hello,
I work for a voice and video chat service, and recently we have been experiencing total service outages lasting minutes at a time usually once a day. When it happens access to any server on the internal Pix interface is not possible. Before and after these problems happen, the pix CPU usage doesn't seem to go above 11%, but then again I can't access it during the outages. Memory seems fine, I have logging enabled to send messages to a Linux syslogd. So far I'm guessing there is some kind of DOS attack going on.. but servers inside the firewall don't experience any dangerous load.

Here are 2 snapshots of what the graphs look like when there is a spike:

http://w3dev.net/temp/

I 'did' have logging trap critical messages, but have since moved it up to alert messages since critical messages (below) were coming through every second or two. These messages occur throughout the day.. just not during a traffic surge


Sep 16 18:50:10 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 66.218.59.73/137 to 209.92.37.166/137 on interface outside
Sep 16 18:50:10 209.92.37.160 %PIX-2-106001: Inbound TCP connection denied from 216.177.60.54/7973 to 209.92.37.180/25 flags SYN on interface outside
Sep 16 18:50:11 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 12.94.116.127/137 to 209.92.37.161/137 on interface outside
Sep 16 18:50:11 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 212.37.44.65/137 to 209.92.37.166/137 on interface outside
Sep 16 18:50:11 209.92.37.160 %PIX-2-106001: Inbound TCP connection denied from 216.177.60.47/12629 to 209.92.37.180/25 flags SYN on interface outside
Sep 16 18:50:11 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 66.218.59.73/137 to 209.92.37.166/137 on interface outside
Sep 16 18:50:12 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 208.191.70.204/137 to 209.92.37.166/137 on interface outside
Sep 16 18:50:13 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 66.218.59.73/137 to 209.92.37.166/137 on interface outside
Sep 16 18:50:13 209.92.37.160 %PIX-2-106001: Inbound TCP connection denied from 216.39.115.119/3368 to 209.92.37.180/25 flags SYN on interface outside
Sep 16 18:50:13 209.92.37.160 %PIX-2-106006: Deny inbound UDP from 208.191.70.204/137 to 209.92.37.166/137 on interface outside

Here is information about the firewall:

Cisco PIX Firewall Version 6.2(1)

Compiled on Wed 17-Apr-02 21:18 by morlee

fw-w1 up 30 mins 22 secs

Hardware: SE440BX2, 128 MB RAM, CPU Pentium II 350 MHz
Flash i28F640J5 @ 0x300, 16MB
BIOS Flash AT29C257 @ 0xfffd8000, 32KB

0: ethernet0: address is 00d0.b7af.b2ba, irq 11
1: ethernet1: address is 00d0.b7c8.1a08, irq 15
2: ethernet2: address is 00d0.b7af.c4e6, irq 10
Licensed Features:
Failover: Enabled
VPN-DES: Enabled
VPN-3DES: Disabled
Maximum Interfaces: 6
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: Unlimited
Throughput: Unlimited
IKE peers: Unlimited

If anyone has suggestions on what I can try I would really appreactiate it. Or if it might be an intermittent hardware failure issue, how could I find out?

Thanks,
Marc

 

[yizhar]

HI.

* Is this an old upgraded pix, or a brand new one?
Anyway check here:

http://www.cisco.com/warp/public/tech_tips/index/hardware/security/500/fn.html

* Check your servers to see if any application or service from the inside is causing the problem.

* What multimedia protocols are you using?

* You can try the built in limitted IDS of the pix, to see if you get any info about or can block an attack.

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_62/cmdref/gl.htm#36470

ip audit name attack1 attack action alarm drop reset
ip audit name info1 info action alarm drop
ip audit interface outside attack1
ip audit interface outside info1

* You will probably need to use a high syslog trap level like level 6 (for a short troubleshooting time) or better a network sniffer connected with a HUB to the pix inside interface that will capture more data during the spike so you can get more info.

* Did you use the PDM graphs? Try it before and after the spike to look for CPU and other resources, and for bit rates of each interface.

Bye
Yizhar Hurwitz

http://come.to/yizhar

http://teachers.sivan.co.il/yizhar