cisco memory issues

[paublo]

Hi recently I had an issue with two flexwan cards on a cisco 6509 reporting memory issues at the exact same time. On the 6509's log I notice some memory allocation messages related to the flexwan on ports 8/1 and 9/1, it complaints that it can't allocate memory and then disables Dcef.


The log on the flexwan says

"Memory allocation of 65536 bytes failed from 0x60197FA0, alignment 16
Pool: Processor Free: 800304 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool"

What exactly does this mean, it seems like at the time I had 800304 bytes free which was enough for the requested 65536 bytes. I'm assuming is requesting Processor memory and not I/O memory so those this mean even though I have 800304 bytes free its fragmented memory and therefore can't be allocated? If so what can I do to fix this issue.



The current memory for the flexwan is:

FlexWAN-8/1>sh memory sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 6124B260 98258336 19751320 78507016 788768 1567384
I/O 20000000 16777216 16544104 233112 230448 230264


The I/O pool concerns me it seems kind of low if there anything I can do without adding more ram to allocate more memory to the I/O pool?


The flexwan log shows:

SLOT 8/1: Jun 15 19:29:39: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
from 0x60197FA0, alignment 16
Pool: Processor Free: 800304 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 44
-Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404 60498AC8 6
0491054 60491484 60491760 604923E0
SLOT 8/1: Jun 15 19:29:39: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard

Cisco Internetwork Operating System Software
IOS (tm) cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)
Technical Support:

http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Thu 08-May-08 00:57 by kellythw
Image text-base: 0x60011038, data-base: 0x6099C000

ROM: System Bootstrap, Version 12.1(1r)EX, RELEASE SOFTWARE (fc1)
ROM: cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

FlexWAN-8/1 uptime is 36 weeks, 21 hours, 53 minutes
System returned to ROM by power-on
System restarted at 12:44:50 edt Wed Oct 6 2010
Running default software

cisco FLEXWAN (R7000) processor (revision 0xFF) with 114688K/16384K bytes of memory.
R7000 CPU at 262Mhz, Implementation 0x27, Rev 2.1, 256KB L2 Cache
Last reset from power-on
1 FastEthernet/IEEE 802.3 interface
2 Serial network interfaces

Configuration register is 0x101

thanks paul

 

[paublo]

FYI - i had another issue and this time i looked at it closely and i think the issue is one of running low on memory and not so much a memory fragmentation issue although its possible its both.

below is the error log from the flexwan and the sh mem, you will notice that a task called VIP Encap IPC" is asking for 65536 bytes but only 76352 bytes from the processor pool. Unlike i previously thought the problem seems to be with this pool that has 98 megs total and not the I/O as i previously thought. So for whatever reason, maybe because i have more customers frame-relay/ppp/MLPPP with interleave on the card is running out of memory. I think the solution is going to be a new card or more ram.

thanks paul

-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:52:48: %SYS-2-MALLOCFAIL: Memory allocation of 65592 bytes failed from 0x6031FEAC, a lignment 32

Pool: Processor Free: 72192 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool



-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:53:37: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x60197FA0, a lignment 16

Pool: Processor Free: 76352 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool



-Process= "CEF LC IPC Background", ipl= 0, pid= 74 -Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404

60498AC8 60491054 60491484 604

91760 604923E0

SLOT 13/1: Jun 17 07:53:37: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard FlexWAN-13/1> sh mem FlexWAN-13/1> sh memory

Head Total(b) Used(b) Free(b) Lowest(b)

Largest(b)

Processor 6124B260 98258336 21595032 76663304 64576

68952160

I/O 20000000 16777216 16515952 261264 258992

243224

 

[MrOyvind]

Experience with a 1812 router connected to Internet using fiber ethernet and DHCP.
The router loaded a huge CEF table because of a misconfiguration.
When I changed IP route 0.0.0.0 0.0.0.0 fastethernet 0 to IP route 0.0.0.0 0.0.0.0 DHCP the CEF table was getting much smaller and the memory problem was gone.