Memory problems with 1812w

[MrOyvind]

Hi

When downloading torrents I get this error message in the log after a while.
Is it possible to do something with my config, upgrade RAM og is it a hardware failure ?

000092: Dec 12 13:07:02.413 GMT+01.: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x80430E38, alignment 16
Pool: Processor Free: 40908 Cause: Not enough free memory
Alternate Pool: I/O Free: 11564 Cause: Not enough free memory
-Process= "ARP Input", ipl= 4, pid= 11 -Traceback= 0x80957AD4 0x8040A77C 0x80410540 0x80430E3C 0x804A9B28 0x804AA4A4 0x804AA730 0x80BD9540 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC
000093: Dec 12 13:07:02.413 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
000094: Dec 12 13:07:14.937 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80C0F144 0x80C0F500 0x80C0FBFC 0x800C6038 0x800C9690
000095: Dec 12 13:07:32.429 GMT+01.: %SYS-2-MALLOCFAIL: Memory allocation of 2560 bytes failed from 0x80BFD9C0, alignment 8
Pool: Processor Free: 40832 Cause: Memory fragmentation
Alternate Pool: I/O Free: 11564 Cause: Memory fragmentation
-Process= "CEF process", ipl= 4, pid= 99 -Traceback= 0x80957AD4 0x8040A77C 0x80416760 0x80BFD9C4 0x80BFDAA0 0x80BF82F4 0x80BE9E14 0x800C6038 0x800C9690


Thanks,

Øyvind

 

[Alterac]

The amount of connections you are establishing is causing the router to run out of memory.

Can you post a Sh Run (wihtout passwords) so we can see what features are enabled?

You might have to turn down the timeouts on connections to fix this.

Also, you could upgrade the memory on the router.


Bill

 

[MrOyvind]

Hi, thanks.

Here is my output modified, wireless and other things is removed. It should not be relevant to read.

Øyvind

Box# sh run
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
no service password-encryption
service sequence-numbers
!
hostname Box
!
boot-start-marker
warm-reboot
boot-end-marker
!
logging userinfo
logging buffered 51200 warnings
logging console critical
enable secret 5 xxxxxxxxxxxxxx
!
aaa new-model
!
!
aaa authentication login default line group SSH
aaa authentication login ssh enable
!
aaa session-id common
!
resource policy
!
clock timezone GMT+01. 1
clock summer-time GMT+01.00 recurring
clock calendar-valid
no ip source-route
no ip gratuitous-arps
!
!
ip cef
ip cef accounting non-recursive
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.10.1
!
ip dhcp pool POOL_10
import all
network 192.168.10.0 255.255.255.0
default-router 192.168.10.1
!
ip tcp synwait-time 10
no ip bootp server
no ip domain lookup
ip domain name xxxx.no-ip.org
ip ssh logging events
ip ssh version 2
!
login block-for 180 attempts 2 within 20
!
interface FastEthernet0
ip address dhcp
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
!
interface FastEthernet2
switchport access vlan 10
no logging event link-status
spanning-tree portfast
!
interface FastEthernet4
switchport trunk native vlan 10
switchport mode trunk
!
interface FastEthernet5
switchport access vlan 10
spanning-tree portfast
!
interface Vlan10
ip address 192.168.10.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip flow egress
ip nat inside
ip virtual-reassembly
ip route-cache flow
!
ip default-gateway 192.168.10.1
ip route 0.0.0.0 0.0.0.0 FastEthernet0
!
ip dns primary 81.167.x.x soa ns1isp.mro.lyse.net dns@lyse.net 21600 900 7776000 86400
ip dns primary ns1isp.mro.lyse.net soa 81.167.x.x sdsd@sd.dsd 21600 900 7776000 86400
!
no ip http server
no ip http secure-server
ip nat pool ISP 81.167.x.x 81.167.x.x netmask 255.255.255.0
ip nat source list 10 interface FastEthernet0 overload
!
ip nat source static tcp 192.168.10.10 3389 interface FastEthernet0 3389
ip nat inside source static tcp 192.168.10.10 61001 interface FastEthernet0 61001
!
!
logging facility auth
logging source-interface Vlan10
access-list 10 permit 192.168.0.0 0.0.255.255
!
!
!
!
!
!
control-plane
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
password xxxxxxxxxxxx
login authentication ssh
transport input ssh
!
ntp logging
ntp clock-period 17180214
ntp source Vlan10
ntp master
ntp server 129.240.x.x prefer
!
webvpn context Default_context
ssl authenticate verify all
!
no inservice
!
end

Box#

 

[MrOyvind]

This is added to my config. Hope it can help.

Box(config)#ip nat tran max-entries 300

Øyvind

 

[Alterac]

you could also play with these values to see what works out for you:

I have used these for a home connection:

ip nat translation tcp-timeout 512
ip nat translation udp-timeout 512


Look up the details on what they do, on cisco's website.

Setting them lower causes the translation to expire, too low, and it breaks stuff, too high and the router will run out of memory.

Bill

 

[MrOyvind]

My CEF table is very high. My Fast etehernet 0 interface is directly connected to a fiber network. They run Cisco on the ISP`s distribution nodes.

Output from CEF summary:

Box#show ip cef summary
IP CEF with switching (Table Version 4346), flags=0x0
4264 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 0
4264 leaves, 6936 nodes, 7861568 bytes, 4350 inserts, 86 invalidations
4248 load sharing elements, 1597248 bytes, 4248 references
universal per-destination load sharing algorithm, id 5F79EAB5
3(0) CEF resets, 1 revisions of existing leaves
Resolution Timer: Exponential (currently 1s, peak 1s)
1 in-place/0 aborted modifications
refcounts: 1790231 leaf, 1775872 node

Table epoch: 0 (4264 entries at this epoch)

Adjacency Table has 4243 adjacencies
Box#

 

[Alterac]

Also do this:
no ip cef accounting non-recursive


Let us know how its been working.




Bill

 

[Alterac]

Another note (Really wish you could edit on these forums):

Change
ip route 0.0.0.0 0.0.0.0 FastEthernet0
to
ip route 0.0.0.0 0.0.0.0 dhcp


Bill

 

[MrOyvind]

Hi

Thanks for the answers. After adding the NAT parameters to the config I thought the problems was resolved but it failed again today. I will try to configure the two suggestions in the latest posts and tell how it works. The torrenst have to run about 24 hours before crashing.
When executing show ip arp before the last memory failure it listed around 8500 ARP enries.

Øyvind

009166: Dec 14 17:39:05.947 GMT+01.: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x80430E38, alignment 16
Pool: Processor Free: 62168 Cause: Not enough free memory
Alternate Pool: I/O Free: 42364 Cause: Not enough free memory
-Process= "ARP Input", ipl= 4, pid= 11 -Traceback= 0x80957AD4 0x8040A77C 0x80410540 0x80430E3C 0x804A9B28 0x804AA4A4 0x804AA730 0x80BD9540 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC
009167: Dec 14 17:39:05.947 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009168: Dec 14 17:39:11.979 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009169: Dec 14 17:39:20.511 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009170: Dec 14 17:39:25.935 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009171: Dec 14 17:39:39.607 GMT+01.: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x80430E38, alignment 16
Pool: Processor Free: 61300 Cause: Not enough free memory
Alternate Pool: I/O Free: 42364 Cause: Not enough free memory
-Process= "ARP Input", ipl= 4, pid= 11 -Traceback= 0x80957AD4 0x8040A77C 0x80410540 0x80430E3C 0x804A9B28 0x804AA4A4 0x804AA730 0x80BD9540 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC
009172: Dec 14 17:39:39.607 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009173: Dec 14 17:39:45.315 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80C0F144 0x80C0F500 0x80C0FBFC 0x800C6038 0x800C9690
009174: Dec 14 17:39:50.939 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009175: Dec 14 17:39:59.923 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009176: Dec 14 17:40:06.936 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009177: Dec 14 17:40:17.928 GMT+01.: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x80430E38, alignment 16
Pool: Processor Free: 62036 Cause: Not enough free memory
Alternate Pool: I/O Free: 39400 Cause: Not enough free memory
-Process= "ARP Input", ipl= 4, pid= 11 -Traceback= 0x80957AD4 0x8040A77C 0x80410540 0x80430E3C 0x804A9B28 0x804AA4A4 0x804AA730 0x80BD9540 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC
009178: Dec 14 17:40:17.928 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690
009179: Dec 14 17:40:22.936 GMT+01.: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x80957AD4 0x80BD959C 0x80BDF4E8 0x80BE33FC 0x80BE3AE4 0x80738E5C 0x80C0E684 0x80DF07D0 0x80DF0DA4 0x80DF18BC 0x805BB324 0x800C6038 0x800C9690

 

[MrOyvind]

Hi. Thank`s a lot.

IP route 0.0.0.0 0.0.0.0 dhcp is the solution.
The huge ARP table and CEF adjaceny table is now small and CPU usage is low too.
But what is the reason ? It seems that the IP route 0.0.0.0 0.0.0.0 Fast ethernet 0 updates my CEF table via the routing table on the ISP`s distribution switch and then knock`s out my router with the large tables.
In both cases I got my IP adress via DHCP and the IP adress never changes ( 3 years ). Can`t manipulate the IP address by changing the MAC address on Fa0 either.
I think the IP address is bounded to the homecentral`s MAC address or the interfcae MAC address of the distrtribution switch due to administrative issues. The ISP takes a higher price for a static IP but my DHCP address never changes.

Øyvind

 

[Alterac]

Cisco actually explains it quite nicely and they recommend against setting a static route to a dhcp via the interface name.

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml

"If you point a static route to a broadcast interface, the route is inserted into the routing table only when the broadcast interface is up. This configuration is not recommended because when the next hop of a static route points to an interface, the router considers each of the hosts within the range of the route to be directly connected through that interface. For example, ip route 0.0.0.0 0.0.0.0 Ethernet0.

With this type of configuration, a router performs Address Resolution Protocol (ARP) on the Ethernet for every destination the router finds through the default route because the router considers all of these destinations as directly connected to Ethernet 0.

This kind of default route, especially if it is used by a lot of packets to many different destination subnets, can cause high processor utilization and a very large ARP cache (along with attendant memory allocation failures). "

Bill

 

[MrOyvind]

Great answers.
Good to know my hardware is OK.

Øyvind