Internet Performance

[Denda]

Good Morning -
Wondering if anyone can help me with figuring this out. For the last week our Internet (T1) has been intermittantly dropping web site attempts for about 20 minutes each hour and then will come back with no problems. ISP (Qwest) states there is no problem on their end at all. What else could we check to make sure it's not an issue on our end?

 

[plshlpme]

does the T1 interface into your router?
or do they have a terminating router?

look at the serial interface on your router and look for errors...

i would clear the counters
and then montitor it..

here is an example

router#sh int s0/0
Serial0/0 is up, line protocol is up
Hardware is PowerQUICC Serial
Description: wan link = 08091, SITE ID = 60114
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 41/255, rxload 49/255
Encapsulation FRAME-RELAY, loopback not set
Keepalive set (10 sec)
LMI enq sent 415595, LMI stat recvd 415586, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is CCITT frame relay DTE
Broadcast queue 0/64, broadcasts sent/dropped 0/0, interface broadcasts 0
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 6w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 93786
Queueing strategy: dual fifo
Output queue: high size/max/dropped 0/256/0
Output queue: 0/128 (size/max)
30 second input rate 300000 bits/sec, 91 packets/sec
30 second output rate 251000 bits/sec, 89 packets/sec
141550736 packets input, 2953327179 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 2 giants, 0 throttles
27 input errors, 1 CRC, 2 frame, 0 overrun, 0 ignored, 20 abort <<<<<<<<<<<<<<<<<<<
101745055 packets output, 2505463257 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets <<<<<<<<<<<<<<<<
0 output buffer failures, 0 output buffers swapped out 46 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up
router#

i would clear the counters and then run that command (replaceing the s0/0 for whatever your physical wan interface is) and look for errors.

during periods where you cant surf you should log into the router and see if it is still seeing the t1 up and if so if it can pass traffic.

 

[Denda]

Thanks plshlpme!

I'm brain dead lately and couldn't remember what the next step would be...

Here's the output from the current status... I'll post when everything starts slowing down again.
Router1>sh int s0/0
Serial0/0 is up, line protocol is up
Hardware is QUICC with integrated T1 CSU/DSU
Internet address is 10.53.53.1/24
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 8/255, rxload 6/255
Encapsulation PPP, loopback not set
Keepalive set (10 sec)
LCP Open
Open: IPCP, CDPCP
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters 5w5d
Queueing strategy: fifo
Output queue 0/40, 15670 drops; input queue 0/75, 0 drops
5 minute input rate 38000 bits/sec, 19 packets/sec
5 minute output rate 53000 bits/sec, 19 packets/sec
76643988 packets input, 4152732603 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 4 giants, 0 throttles
3891 input errors, 732 CRC, 2672 frame, 0 overrun, 0 ignored, 484 abort
74893668 packets output, 744039292 bytes, 0 underruns
0 output errors, 0 collisions, 2980 interface resets
0 output buffer failures, 0 output buffers swapped out
15 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=u

 

[Denda]

Sorry, I'm an idiot.. Grabbed the wrong one.. Here is the correct one.

FastEthernet1/0 is up, line protocol is up
Hardware is AmdFE, address is 0030.9499.6e90 (bia 0030.9499.6e90)
Internet address is 10.35.1.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 47160 drops
5 minute input rate 202000 bits/sec, 140 packets/sec
5 minute output rate 194000 bits/sec, 142 packets/sec
444818635 packets input, 342199418 bytes
Received 6071629 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
446050454 packets output, 2650239663 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

[plshlpme]

you connect to your isp through the fa0/1 interface? or the s0/0 ?
typically s0/0 would be the t1
and it is showing alot of errors

 

[burtsbees]

Last clearing of "show interface" counters 5w5d
That's a long time...

Burt

 

[Denda]

I need to go home. It's been a long day already.

grabbed my test router accidently.. here's the Internet Router..

Serial0/0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: Frame Circuit
Internet address is XX.XXX.X.34/30
MTU 1500 bytes, BW 256 Kbit, DLY 20000 usec,
reliability 255/255, txload 86/255, rxload 252/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 6032 drops; input queue 0/75, 0 drops
5 minute input rate 507000 bits/sec, 73 packets/sec
5 minute output rate 87000 bits/sec, 59 packets/sec
43125515 packets input, 102091025 bytes, 0 no buffer
Received 210081 broadcasts, 0 runts, 0 giants, 0 throttles
2966 input errors, 595 CRC, 2217 frame, 0 overrun, 0 ignored, 154 abort
35695976 packets output, 1933515779 bytes, 0 underruns
0 output errors, 0 collisions, 817 interface resets
0 output buffer failures, 0 output buffers swapped out
3 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up


It actually looks really good for counters never being cleared in the last 6 years. I did clear them and will wait. While I wait, anything else I could look at?

 

[plshlpme]

well the fact that your rx load is pinned 252/255
might have something to do with your problems too...

do you have a cir of 256k? or is it full t1?

the telco could be dumping alot of your packets..

 

[Denda]

We have a full T1.

We pulled the stats this morning which it started around 8:30 and no errors are coming up on the router at all, been pulling stats for about 20 minutes, now Internet is back working. Trace routes are hanging for about 25ms before the next router after ours, then start increasingly getting worse from there on. This has to be an issue with our T1, what do you think?

 

[Denda]

A colleague brought up the EDNS issue with server 2003. Don't know why all of a sudden this would affect us since it's been working fine for the last 3 years with our 2003 DNS servers, but it's worth a look.

Has anyone else had this type of issue, in conjunction with a CheckPoint NGX R60 HFA02 firewall? If so, how did you get it to work since EDNS is not listed in the protocol list. I know I can create the protocol, but how do I set it to allow over 512K. If I can't do it then we will just have to disable it (EDNS) on the 2003 DNS servers.

While we are researching this, we have our T1 (dedicated internet) provider monitoring the line to see if they can find anything.

 

[texnut]

It's very possible you could be under a DoS attack considering your rx load is so high ...

Do you have any sniffer's running on the network so you can take a peak at what's going on?

I would recommend running ntop on a port monitor or enabling netflow on your router and feeding the data to a netflow program. Netflow Analyzer or ntop can both handle it.

 

[Denda]

Thanks for the advice texnut, but the sniffer is coming up clean.

The T1 provider finally called back and let us know that they were having problems with a couple DNS servers in the Chicago area, which most of our traffic hits. Once they bounced one of the servers, problems were very few, the test will be within the first couple of hours of business and then we will see if it was actually fixed. I will report back.

 

[texnut]

Interesting ... well, that being the case, I highly suggest you switch over to OpenDNS (opendns.com). They are free, redundant and way faster than anything your ISP could provide!

 

[Denda]

Yup. Everything is working fine now. I also just wanted to say that the rx load does not stay that high, it averages between 60 - 70%.

thanks again for all your help.