cisco 831 memory leak in Chunk Manager?

[dpetzinger]

I'm sorry, this is unfortunately going to be a bit long-winded in an effort to provide as much information as possible.

My school used to have a T1, but in an effort to lower costs and increase bandwidth, we switched to cable modem. a brief description of our network is below

-=-=-=-=-=-

172.16.40.0/255.255.254.0 (2 subnets)
total of 220 machines or so and about 20 network printers/switches (explaining why we need the second subnet).

-=-=-=-=-=-

The router that cablevision uses is an 831. Below is the running config

-=-=-=-

Building configuration...

Current configuration : 2712 bytes
!
! Last configuration change at 23:13:55 America Thu Apr 1 2004
! NVRAM config last updated at 18:03:13 America Thu Apr 1 2004
!
version 12.3
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname router
!
boot-start-marker
boot-end-marker
!
logging buffered 80000 debugging
enable secret 5 1234123412341234123412342134
enable password 7 1234123412341234
!
username isplogin privilege 15 password 7 1234123412341234
clock timezone America/New_York -5
clock summer-time America/New_York date Apr 6 2003 2:00 Oct 26 2003 2:00
no aaa new-model
ip subnet-zero
!
!
!
!
ip domain name myisp.com
ip name-server 34.140.118.3
ip name-server 34.140.118.4
ip cef
ip urlfilter server vendor n2h2 162.206.102.9 outside timeout 300
ip audit notify log
ip audit po max-events 100
no ftp-server write-enable
no scripting tcl init
no scripting tcl encdir
!
!
!
no crypto isakmp enable
!
!
!
!
interface Ethernet0
description $FW_INSIDE$$ETH-LAN$$ETH-SW-LAUNCH$
ip address 172.16.40.1 255.255.254.0
ip nat inside
!
interface Ethernet1
description $FW_OUTSIDE$$ETH-WAN$
ip address 34.140.118.68 255.255.255.240
ip nat outside
duplex auto
!
interface FastEthernet1
no ip address
duplex auto
speed auto
!
interface FastEthernet2
no ip address
duplex auto
speed auto
!
interface FastEthernet3
no ip address
duplex auto
speed auto
!
interface FastEthernet4
no ip address
duplex full
speed auto
!
ip nat inside source list 1 interface Ethernet1 overload
ip classless
ip route 0.0.0.0 0.0.0.0 34.140.118.65
!
ip http server
ip http authentication local
ip http secure-server
!
!
logging 172.16.40.30
access-list 1 remark INSIDE_IF=Ethernet0
access-list 1 remark SDM_ACL Category=2
access-list 1 permit 172.16.40.0 0.0.1.255
snmp-server community dbug RO
snmp-server enable traps tty
snmp-server host 172.16.40.30 justin
!
control-plane
!
banner login ^C
-----------------------------------------------------------------------

WARNING!

Accessing this device without express permission of MY ISP
is a violation of the terms of agreement and any attempt to alter the
configuration of this device will void this agreement and result in the
termination of services.

^C
!
line con 0
password 7 12341234123412341234
login
no modem enable
transport preferred all
transport output all
line aux 0
transport preferred all
transport output all
line vty 0 4
privilege level 15
password 7 12341234123412341234
login local
transport preferred all
transport input telnet ssh
transport output telnet ssh
!
scheduler max-task-time 5000
!
end

-=-=-=-=-=-

names and ip addresses have been changed to protect the inno... er... make sure my isp doesn't get pissed off.

As installed right now, it crashes once every 4 hours of normal use. snmp crash log:

-=-=-

Apr 01 11:15:15 172.16.40.1 76: Apr 1 16:17:23.939: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x802E6D60, alignment 8
Apr 01 11:15:15 172.16.40.1 77: Pool: Processor Free: 112612 Cause: Memory fragmentation
Apr 01 11:15:15 172.16.40.1 78: Alternate Pool: None Free: 0 Cause: No Alternate pool
Apr 01 11:15:15 172.16.40.1 79:
Apr 01 11:15:15 172.16.40.1 80: -Process= "Chunk Manager", ipl= 4, pid= 1
Apr 01 11:15:15 172.16.40.1 81: -Traceback= 802D4F14 802D6A50 802E6D64 802E5F98 802E5D1C 802C9BEC 802CE3B8
Apr 01 11:15:15 172.16.40.1 82: Apr 1 16:17:23.939: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available
Apr 01 11:15:15 172.16.40.1 83: -Process= "Chunk Manager", ipl= 4, pid= 1
Apr 01 11:15:15 172.16.40.1 84: -Traceback= 802E5D64 802C9BEC 802CE3B8
Apr 01 11:15:27 172.16.40.1 85: Apr 1 16:17:36.195: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available

-=-=-=-

The end part repeats for about 30 minutes, at which point, the router is incapable of any communication. I decided to monitor the console a bit, and see if i could figure out what and when. I ran a show proc mem and show mem sum every 5 minutes or so after the router was up for 2 hours. As you can see, Chunk Manager was already high when i started monitoring, and grew rapidly until it brought the router to its knees again.

-=-=-=-

1:30
Total: 28186024, Used: 16308940, Free: 11877084
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 52720 400 0 0 *Sched*
0 0 715804 641372 77184 547704 0 *Dead*
1 0 6892496 0 6899404 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 4152 3188 13932 0 0 Virtual Exec
5 0 0 0 6908 0 0 Check heaps
6 0 100128 0 107036 32232 0 Pool Manager

Router#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 14605196 8547868 8542912 8545528
I/O 2B33400 5033016 1705768 3327248 3327248 3326712
1:40
Router#show process memory
Total: 28186024, Used: 16311044, Free: 11874980
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 52720 400 0 0 *Sched*
0 0 715804 641372 77184 547704 0 *Dead*
1 0 6892496 0 6899404 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 4292 3188 14072 0 0 Virtual Exec
5 0 0 0 6908 0 0 Check heaps
6 0 100128 0 107036 32232 0 Pool Manager


Router#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 14605616 8547448 8542912 8545184
I/O 2B33400 5033016 1705768 3327248 3327248 3326712

1:45
Router#show process memory
Total: 28186024, Used: 16312104, Free: 11873920
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 52720 400 0 0 *Sched*
0 0 715804 641372 77184 547704 0 *Dead*
1 0 6892496 0 6899404 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 106112 104776 14304 0 0 Virtual Exec


Router#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 14606748 8546316 8496816 8540956
I/O 2B33400 5033016 1705768 3327248 3327248 3326712
1:50
Router#show process memory
Total: 28186024, Used: 16730848, Free: 11455176
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 52720 400 0 0 *Sched*
0 0 715804 641372 77184 547704 0 *Dead*
1 0 7310464 0 7317372 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 106304 104776 14496 0 0 Virtual Exec

20 0 0 0 9908 0 0 Per-Second Jobs

Router#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 15025632 8127432 8119044 8118528
I/O 2B33400 5033016 1705768 3327248 3327248 3326712
2:00
Router#show process memory
Total: 28186024, Used: 17145028, Free: 11040996
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 52720 400 0 0 *Sched*
0 0 715804 641372 77184 547704 0 *Dead*
1 0 7721452 0 7728360 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 106584 104776 14776 0 0 Virtual Exec
5 0 0 0 6908 0 0 Check heaps
6 0 100128 0 107036 32232 0 Pool Manager


Router#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 15439772 7713292 7708016 7709948
I/O 2B33400 5033016 1705768 3327248 3327248 3326712
2:12
Router#show proc mem
Total: 28186024, Used: 17553852, Free: 10632172
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 65688 400 0 0 *Sched*
0 0 827224 754240 77184 547704 0 *Dead*
1 0 8133412 0 8140320 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 4264 3652 13544 0 0 Virtual Exec
5 0 0 0 6908 0 0 Check heaps
6 0 100128 0 107036 32232 0 Pool Manager


Router#show mem sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 15848552 7304512 7289964 7285808
I/O 2B33400 5033016 1705768 3327248 3327248 3326712
2:20
Router#show proc mem
Total: 28186024, Used: 18190756, Free: 9995268
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 65688 400 0 0 *Sched*
0 0 827224 754240 77184 547704 0 *Dead*
1 0 8752892 0 8759800 0 0 Chunk Manager
2 0 188 188 3908 0 0 Load Meter
3 2 4416 3652 13696 0 0 Virtual Exec
4 3 4332 3184 14116 0 0 Virtual Exec
5 0 0 0 6908 0 0 Check heaps
6 0 100128 0 107036 32232 0 Pool Manager


Router#show mem sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153432 16485620 6667812 6667812 6661328
I/O 2B33400 5033016 1705768 3327248 3327248 3326712

2:30
Router#show proc mem
Total: 28186024, Used: 18523208, Free: 9662816
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 12603372 3933208 7787112 0 0 *Init*
0 0 400 78656 400 0 0 *Sched*
0 0 836976 765352 77272 547704 0 *Dead*
1 0 9096960 0 9103868 0 0 Chunk Manager


Router#show mem sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8151EA58 23153064 16817988 6335076 6309408 6315440
I/O 2B33400 5033016 1705768 3327248 3327248 3326712

-=-=-=-

Does anyone know what could possibly cause the chunk manager to grow out of control like that? i didn't continue to log it, but eventually the chunk manager bankrupted the system memory. even when disconnected from the network, the switch never recovers any of the memory. when connected to only a couple of computers, it continues to lose memory.

thanks for any help!

dave

 

[pmesjar]

I think these kind of errors can only be solved by TAC or by upgrading you IOS version. Anyway have a look at:

http://www.cisco.com/warp/public/63/mallocfail.shtml

Hope this helps

Peter Mesjar
CCNP, A+ certified
pmesjar@centrum.sk

"The only true wisdom is in knowing you know nothing.

 

[gconnect]

probably a bug in the version of IOS. I have seen those types of problems relating to Encryption Processes (IPSec Keying Process or SSH Key Process). Especially on 1600's and 800's.

As with any memory leak problem, there is a process running that is not properly releasing itself as it should. I suspect it has something to do with encryption.

i'd try these things one at a time and see what happens first...
make sure no debugging is on, or being sent to syslog at error level
disable CEF
disable SSH and Erase the Key
disable SNMP
disable the URL filter stuff
make sure that the IDS is not getting flooded


Did 12.3 come with your router or was it an upgrade. If it came stock (or even if it didn't) you need to get with TAC and/or change your IOS version.

-gC-

 

[computerhighguy]

I would change the IOS. It easy and will probably solve your issue. Remember, that IOS can be a bit buggy especially at the beginning of a n e release. It the bugs very from image to image (that is why there are so many on the TAC).

 

[dpetzinger]

Those sound like some good debugging steps, I'll give them a shot. Unfortunately, i can't just change the ios's on the machine because it doesn't belong to me... I'm not even supposed to have access to it..

thanks again,

dave