Cisco 2950s with high memory utilization

[jkaftan]

I have implimented Solar Winds NPM. It is telling me that most of my cisco 2950s are experiencing high memory utilization. I am trying to determine if this is normal or if I have a problem.

I seem to remember that Cisco does not report memory utilization accurately. My CPU on these switches is low.

Should I be worried?

I ran a sh processes memory it nothing jumped out at me. I was looking at the 'Holding' column for each process.

See attachment.

 

[burtsbees]

You can't post attachments---only links.

Burt

 

[VinceWhirlwind]

If the high memory utilisation is an issue, you should see other problems.

Find your busiest interfAces on those switches (easy with SOlarwinds!) and do a show int. See if any buffer drops are happening.

 

[jkaftan]

Sorry the attachment did not work. I will just paste the text here.

User Access Verification
Password:
Password:
WH-2950-3>en
Password:
WH-2950-3#sh mem
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 80BA72C0 4547904 3868972 678932 339752 559596
I/O A0750D80 4547872 2638704 1909168 1632188 1859548



WH-2950-3#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(9)EA1, RELEASE SOFTWARE (
fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Wed 24-Apr-02 06:57 by antonino
Image text-base: 0x80010000, data-base: 0x804E8000
ROM: Bootstrap program is CALHOUN boot loader
WH-2950-3 uptime is 1 year, 44 weeks, 5 days, 5 hours, 41 minutes
System returned to ROM by power-on
System restarted at 08:22:21 Eastern Wed Jan 10 2007
System image file is "flash:c2950-i6q4l2-mz.121-9.EA1.bin"
cisco WS-C2950G-48-EI (RC32300) processor (revision C0) with 20815K bytes of mem
ory.
Processor board ID FHK0626X03D
Last reset from system-reset
Running Enhanced Image
48 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0A:41:23:60:40
Motherboard assembly number: 73-7409-08
Power supply part number: 34-0965-01
Motherboard serial number: FOC06250M5A
Power supply serial number: PHI06210B6F
Model revision number: C0
Motherboard revision number: B0
Model number: WS-C2950G-48-EI
System serial number: FHK0626X03D
Configuration register is 0xF


Total : 909 5776, Used : 6505892, Free: 25898
PID TTY Allocated Freed Holding Process
0 0 66904 5048 4743564 *Init*
0 0 820 165937756 820 *Sched*
0 0 411328556 435825128 318640 *Dead*
1 0 280 280 3808 Load Meter
2 0 76323752 339024308 10132 Span Tree
3 0 0 1084 6808 Check heaps
4 0 96 0 6904 Chunk Manager
5 0 2 380297132 3332524 93656 Pool Manager
6 0 280 280 6808 Timers
7 0 3504552 2878460 447076 Entity MIB API
8 0 0 0 6808 HC Counter Timer
9 0 232 48668 7040 ARP Input
10 0 192 0 7000 Net Input
11 0 96 0 6904 Critical Bkgnd
12 0 29949196 29282092 16124 Net Background
13 0 63408 280 12996 Logger
14 0 280 5264 6808 TTY Background
15 0 0 0 9808 Per-Second Jobs
16 0 512 280 4040 Calhoun Storm Co
17 0 0 0 6808 Calhoun LED Proc
18 0 1 856598044 1856597908 3808 Calhoun Statisti
19 0 280 280 6808 Compute load avg
PID TTY Allocated Freed Holding Process
20 0 401664 1323855468 6808 Per-minute Jobs
21 0 699302176 699275184 33800 HMATM Learn proc
22 0 50194612 12495656 24052 CalhounPM
23 0 280 280 6808 Link Status Proc
24 0 248 0 7056 CalhounInsIpopti
25 0 265400460 265323476 14388 CDP Protocol
26 0 14773992 28010252 14352 IP Input
27 0 43202796 1044032 68004 DTP Protocol
28 0 280 280 6808 EtherChnl
29 0 12021384 11808636 94012 VLAN Manager
30 0 280 280 6808 PI MATM Aging Pr
31 0 604 280 7132 UDLD
32 0 19840 280 21304 dot1x
33 0 4 91716724 4091694648 21804 DHCPD Receive
34 0 0 0 6808 HTTP Timer
35 0 96 0 6904 RARP Input
36 0 0 575016 12808 TCP Timer
37 0 183973064 0 14412 TCP Protocols
38 0 0 0 6808 Socket Timers
39 0 33048 379280008 35400 L2MM
40 0 317947268 2734783780 71828 MRD
41 0 791496 24387300 7976 IGMPSN
42 0 248 818540524 4056 IGMP Snooping Pr
PID TTY Allocated Freed Holding Process
43 0 2 804007436 53932888 7032 IGMP Snooping Re
44 0 232 0 7040 pSecureProcess
45 0 376 280 6904 Cluster L2
46 0 96 0 6904 Cluster RARP
47 0 253402496 253401576 7728 Cluster Base
48 1 1200 0 14056 Virtual Exec
49 0 248 68 7056 Router Autoconf
50 0 96 0 6904 Syslog Traps
51 0 0 0 6808 AggMgr Process
52 0 286080 70297480 91112 PM Callback
53 0 5108 280 11636 VTP Trap Process
54 0 0 0 6808 DHCPD Timer
55 0 376 280 3904 STP STACK TOPOLO
56 0 376 280 3904 STP FAST TRANSIT
57 0 158352 109756 6944 SpanTree Helper
58 0 280 280 6808 IP NAT Ager
59 0 0 0 6808 SNMP Timers
60 0 4 213058304 4212959828 14148 IP SNMP
61 0 96 0 12904 SNMP ConfCopyPro
62 0 631760 701280 12904 SNMP Traps
63 0 7720 280 8276 NTP
6489892

 

[jkaftan]

I am seeing small and medium buffer misses. I reboot the switch and as soon as it comes back up it is right back to the same utilization.

Also in Solar Winds when I look at Mem Utilization over time it is unwavering. It is a straight line at 85% or whatever.

 

[jkaftan]

This problem is happening network wide with 2950's. 3750s, 4506s, 6509s seem fine. I have added SNMP to all switches so that I could do monitoring and NAC. Is there any way I could be taxing the switches with my monitoring? It does not seem likely and when I do the "sh proc mem" command it does not look like snmp is doing much. But then again I am not quite sure how to read the output from this command.

I am just looking at the 'Holding' column for the amount of memory a process is using. Is that correct?

Also when it says Total and Used at the top I assume they mean total Memory on the box. However I get numbers like 7947968 for total which is not close to any of the usual boundaries. I am verifing Solar Winds numbers by taking Used\Total from this command. I get a number that is close but slightly less then what Solar Winds reports.

I have to admit that determining what is going on with memory on Cisco devices has always eluded me. For me just trying to determine how much memory a switch has is guesswork at best.

Anyway if anyone has any insight I would be greatful.

 

[burtsbees]

Post a sh run and let's see what your switch is doing...

Burt

 

[vipergg]

Just keep an eye on it . Memory is allocated and freed up as needed . If it doesn't get any worse or stays in the same area of mem utilization don't worry about it . To get a good view of what the switch fabric and individual port utilization is use the "show controllers utilization" command. I really kind of doubt there is an issue. Of course make sure there isn't unneeded stuff in the config thats not being used like unneeded span sessions etc...

 

[burtsbees]

How many link lights light up when you press the MODE button to UTIL?

Burt

 

[jkaftan]

Burt:

I will have to go over and look at the lights. In the meantime here is the config.

We have been messing around with SNMP both for NAC as well as Solar Winds. I never really noticed what my memory utilizaion was before implimenting Solar Winds. I tried removing all SNMP commands from one of the switches and also removing it from Solar Winds with no change.

Also it doesn't look like SNMP is doing anything from the "sh proc memory" command.

Keep in mind that this is happening on all 2950s on campus. I read about a memory leak on CDP but I tried rebooting one swtich and disabling CDP with no change.



Current configuration : 6265 bytes
!
! Last configuration change at 14:50:57 Eastern Sat Aug 16 2008
! NVRAM config last updated at 14:50:57 Eastern Sat Aug 16 2008
!
version 12.1
no service pad
service timestamps debug datetime
service timestamps log datetime
service password-encryption
!
hostname WH-2950-2
!
enable secret
!
clock timezone Eastern -5
clock summer-time Eastern recurring
ip subnet-zero
!
!
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
no spanning-tree vlan 623
!
!
!
!
interface FastEthernet0/1
description B-1
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/2
description B-25 Admiss-HP4200
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/3
description B-2
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/4
description B-26
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/5
description Rm-239
switchport access vlan 10
spanning-tree portfast
!
interface FastEthernet0/6
description B-27
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/7
description C-41 Registrar Office
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/8
description B-28
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/9
description C-47 P&A Color Ptr
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/10
description B-29
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/11
description B-6
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/12
description B-30
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/13
description B-7 PayRoll-XN2125
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/14
description B-31
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/15
description B-8 HR-XN2125
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/16
description B-32
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/17
description B-9
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/18
description B-33
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/19
description B-10
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/20
description B-34
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/21
description C43
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/22
description B-35
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/23
description B-12
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/24
description B-36
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/25
description B-13
switchport access vlan 59
!
interface FastEthernet0/26
description B-37
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/27
description C-48 Lobby Desk
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/28
description B-38 Rm123
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/29
description B-15
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/30
description B-39 Bursar-HP8150
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/31
description B-16
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/32
description B-40
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/33
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/34
description B-41
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/35
description B-18
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/36
description B-42
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/37
description B-19
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/38
description B-43 Rm 123
switchport access vlan 2
spanning-tree portfast
!
interface FastEthernet0/39
description B-20
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/40
description B-44
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/41
description B-21
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/42
description B-45
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/43
description B-22
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/44
description B-46
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/45
description B-23
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/46
description B-47
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/47
description B-24
switchport access vlan 59
spanning-tree portfast
!
interface FastEthernet0/48
switchport access vlan 59
switchport mode trunk
speed 100
duplex full
!
interface GigabitEthernet0/1
description To SS-6509
switchport mode trunk
!
interface GigabitEthernet0/2
description To NH-6509
switchport mode trunk
!
interface Vlan1
ip address 10.0.1.42 255.255.255.0
no ip route-cache
!
ip default-gateway 10.0.1.1
no ip http server
logging facility local1
logging 10.0.1.25
access-list 10 permit 10.3.1.10
access-list 50 permit 10.0.1.6
access-list 50 permit 10.0.1.5
access-list 50 permit 10.3.1.7
access-list 50 permit 10.0.1.25
access-list 50 permit 10.3.1.16
access-list 50 permit 10.3.1.46
access-list 50 permit 10.3.1.47
snmp-server community ****** RO
snmp-server enable traps snmp linkdown linkup
snmp-server enable traps MAC-Notification
snmp-server host 10.0.1.25 ******
snmp-server host 10.0.1.5 ******
snmp-server host 10.0.1.6 ******
line con 0
line vty 0 4
access-class 50 in
exec-timeout 20 0
password
login
line vty 5 15
login
!
ntp clock-period 17180095
ntp server 10.3.1.5
!
end

WH-2950-2#

 

[burtsbees]

Are there a lot of people trying to telnet in?
I see 5 vlans...i will look at a similar switch of mine...

Burt

 

[vipergg]

Configs look good and a lot cleaner than many I have seen . I really think you don't have any issue . That is probably just the memory level they operate at , just like some of the older 2900 series will have the cpu at 35% with nothing hooked to it which was normal.

 

[burtsbees]

I agree---all your 2950's doing the same thing with the memory.

Burt

 

[burtsbees]

I will say that you have a good sized bottleneck routing 5 vlans on 47 ports with router-on-a-stick. Have you considered breaking up the network into a traditional 3 layer heirarchy, and using a true distribution layer with L3 switches?

Burt

 

[jkaftan]

Hmm

Burt:

Very few folks telnet into this switch.

I am not following you in regards to 5 VLANS. This switch is just a layer 2 device it is not doing any routing. I do have a colapsed core in that my distribution and core layers are both on a single layer 3 6509. I have 50 VLANS on that are trunked to all of the Layer 2 devices. I could prune but did not think it really mattered. The 6509 at the core is not breaking a sweat.

As to this possibly being normal behavior I am seeing small buffer misses.

This doc tells about how I might tune the buffer settings.

http://www.cisco.com/application/pdf/paws/14620/41.pdf

My current plan is to take a new switch and fire it up to see what is going on in terms of memory.

The I will add my standard config slowly while I watch the memory usage.

SolarWinds tells me what my memory usage but I am trying to tell if solar winds is part of the problems so I need to get an accurate memory usage before I add the new switch to SolarWinds. What is the best way to do that? I know it sounds crazy to ask something so basic but honestly I cannot tell from sh ver, sh mem, or sh proc mem.

Thanks so much for all of your help!

 

[burtsbees]

I know that the 2950's are not routing---it is whatever is going the routing on a single interface (or etherchannel) that might be the bottleneck. Five vlans all having traffic from 47 access ports on multiple switches---that's a lot of traffic for a single routing interface. I know that they are all going to gig links though. Perhaps I should have clarified---not on any one device taking a hit on the proc or memory, but just a throughput issue is all. If you are not having people complaining about slow throughput, disregard. I myself just would not trunk all that traffic via router-on-a-stick like that.
Like I say---no muss, no fuss...
As far as the memory issues go---I would say that it is indeed normal. Just for grins, maybe clear the log and lower the buffer, just to see if the memory maybe goes down...
switch#clear log
switch#conf t
switch(config)#logg buff ?
and choose the lowest, just for the time being. If you really wanted to log everything down to simple system messages, then you could log to a syslog daemon (like kiwi, free, work great) and have the buffer in the switch to 4096, which I believe is the lowest. This may all be meaningless, but it couldn't hurt and could potentially help?

Burt

 

[jkaftan]

I have a single Gig link to the closet in all of my buildings. It typically terminates in a 2950 and then I have 2-3 other switches daisy chained off of the main switch for the building. It is not elegant or redundant but has been working fine.

As for load none of fiber links to any of the closets seem to be going over 3-4 Mb. I do not think I am hurting for uplink capacity.

Can anyone speak to how to determine how much RAM I have in a switch and how much is being used?

 

[burtsbees]

cisco WS-C2950G-48-EI (RC32300) processor (revision C0) with 20815K bytes of mem
ory.

That is from sh version. I am not sure of a way to actually see how much is being used, other than sh stacks, sh mem, show proc, etc.

Burt

 

[jkaftan]

Ok when I do a sh ver I get the same thing. Looks like I have about 20 MB of memory. Seems like a strange number but I can go along with it. I would expect it to fall along the lines of binary, e.g. 8, 16, 24, 32 etc.


cisco WS-C2950G-48-EI (RC32300) processor (revision P0) with 20873K bytes of memory.
Processor board ID FOC0839Z11Y

Now I do a sh mem and get the following.

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Proc 4869568 3556668 1312900 722544 1108232
I/O 3078400 1426316 1652084 1504344 1635516

So from this it looks like I might have 7.8 MB if I add Proc and I/O. Does the small b mean bytes? Usually it does not but we never measure memory in bits.

See my confusion? I have been reading but I have not found a clear answer. It has me wondering why something so basic is not more straight forward.

 

[Ashok2612]

Hi,

I am also facing similar problem with one of Cisco 2950 Switch.

Can I know the solution if you have any on your problem?

I appreciate the early reply.

Regds,
Ashok