Cisco Pix 506e running out of memory over time - have to reset to fix

[TrentGreenawalt]

Good Morning,

We have a Cisco Pix 506e firewall, and it has been in service for us for many years with minimal changes, until recently it has been a great firewall.

Here is what is happening: On 8/13, I get a call that the internet is crawling, so I search around a bit and end up rebooting our internet router, and the firewall and the problem goes away. Until 8/20, when the same thing happens again and this time I take a peek at the running config and memory used and see that we are using 31 of 32mb of ram. So I reboot again, and problem solved (again).

So the good news is I can solve the "problem", but the bad news is I don't know why this is happening. Now it is 8/22 and it is already in need of another reboot. I am attaching a snapshot of the memory used from 8/20 until today 8/22 and you will see what I mean.

So I have scoured our email logs (Kerio mail server), content management logs (cymphonix box), and firewall logs and I am not finding any problems. Does anyone have any thoughts on what to do here?

Thanks in advance for your help,
Trent Greenawalt
Anderson Pump & Process

 

[BuckWeet]

Could you check the connection count when this happens.. I assume someone internally or externally is opening up thousands of connections causing this..


BuckWeet

 

[TrentGreenawalt]

Hi Buckweet,

Yes actually, I believe you are correct. See attached screenshot of a graph that I found for connections. Now how to I find out how they are doing this and how to stop "them"?

Thanks a ton for your help,
Trent

 

[BuckWeet]

Not sure how to do it via the PDM that you are using.. But via the CLI you can do a 'show conn' I believe.. Or maybe its 'show session'... Its changed between PIX 6.3x and 7/8x..


BuckWeet

 

[TrentGreenawalt]

There is a command line interface built into the PDM, however since there is only about a 1mb of RAM left it can't display all the connections. I think I might try to telnet into it to save on the RAM portion of things.

I will report back what I find.

Okay here is a "Show Local"
Interface inside: 30 active, 37 maximum active, 0 denied
local host: <192.168.30.46>,
TCP connection count/limit = 0/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 0/unlimited
AAA:
Xlate(s):
Conn(s):

local host: <192.168.30.42>,
TCP connection count/limit = 0/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 0/unlimited
AAA:
Xlate(s):
Conn(s):

local host: <CymphonixBox>,
TCP connection count/limit = 0/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 0/unlimited
AAA:
Xlate(s):
Conn(s):

local host: <192.168.30.2>,
TCP connection count/limit = 14/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 49663/unlimited
AAA:
Xlate(s):
PAT Global xx.xxx.xxx.x(1024) Local 192.168.30.2(1052)
PAT Global xx.xxx.xxx.x (25) Local 192.168.30.2(25)
PAT Global xx.xxx.xxx.x (443) Local 192.168.30.2(443)
PAT Global xx.xxx.xxx.x (143) Local 192.168.30.2(143)
PAT Global xx.xxx.xxx.x (43777) Local 192.168.30.2(1175)
Conn(s):
UDP out 65.43.19.26:53 in 192.168.30.2:1052 idle 0:00:00 flags -

UDP out 65.43.19.26:53 in 192.168.30.2:1052 idle 0:00:01 flags -

UDP out 65.43.19.26:53 in 192.168.30.2:1052 idle 0:00:00 flags -

So there are 49,000 connections to 192.168.30.2, when I click more here it just goes on forever..well 50k times. I quit out of it.

So how do I limit these connections or block the SOB that is doing this.

Thanks,
Trent

 

[BuckWeet]

I would find out why 192.168.30.2 is doing it.. Not fix it from the PIX..

 

[TrentGreenawalt]

I don't even know where to begin with this. Are you saying that 192.168.30.2 has 50k connections to the internet through my firewall. When I run a "netstat" from "jim" let's call it, there are no where near 50k connections. I bet there is less then 50-75 that show up.

Are you leaning towards spyware or adware? Virus? Nothing is showing up in scans. I thought that there should be a limit on the firewall for UDP connections or a timeout at least. Can you inform me what command lines to input into the CLI to do that and do you think it will work?

Thanks,
Trent

 

[BuckWeet]

Thats exactly what I'm saying..

local host: <192.168.30.2>,
TCP connection count/limit = 14/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 49663/unlimited


There are 49K UDP connections.. So they won't show up in a netstat typically.

You need to look at them and see what they are.. From what you posted above it looks like outbound DNS queries.


BuckWeet

 

[TrentGreenawalt]

Good morning,

Well I wasn't able to figure anything out, but I have finally found users with similar problems with my build of Cisco 6.3(3), the problem is these links are from 2003!!

http://puck.nether.net/pipermail/cisco-nsp/2003-September/005890.html

So I will read through these and see what is happening. Buckweet do you have any insight on this thread and what they are suggesting? I haven't a clue how to manage this router. Your help is greatly appreciated.

 

[BuckWeet]

Interesting.. One thought would be to upgrade to the latest version of PIX 6.3x.. I believe the default xlate timeout for UDP is 5 minutes..

It could be this bug.. CSCec45748 or CSCsc61300..

These are fixed in 6.3(5)...

however the fact that it worked for years without issue makes me think something changed on the server..


BuckWeet