Leased T1's suddely got s-l-o-w 1

[ITsmyfault]

Hi -
We have 2 clearchannel T1's, both full, running interoffice into Cisco 3640s (both ends) Each router is maxed on RAM, no ACL.. nothing fancy at all. sh cpu hist is clean. Set up per packet load balancing with CEF and it's been running great for a long while. Now suddenly it is slow as molasses. I did not change anything on the routers or the lines. sh int does not show errors accumulating. Traffic is low. Ping times are erratic. Backing up over the lines is horrble (sub 100KB/sec) FTP over the lines gets to maybe 10% of throughput.
I tried adminning one T down to see if things sped up but they got worse (although throughput did rise to the previous sum of both T's) Both T's are B8ZS/ESF. Routers keep time via NTP (we provide time for the circuits AFAIK)
My first instinct is to call them out for testing, but without errors showing up I am not sure what to tell the telco. Don't want to be another wanker who yells at the phone before I get my facts straight.
I was able to get great backup rates over these previously, up to 80-90% of bw (so sayeth mrtg & netflow)

Any ideas of what to look at first? Thanks for any comments you may care to share.

 

[ph0neguy]

Is you're sync set up ok on these circuits? If you've got any T1 test gear you test it youreself its easy. Fluke microbert 635 makes a good test set its just over 2000g's but it saves you alot of $ in the long run. Could be a line coding issue, since the T1 is a 2 way transmission, sounds like you have a point to point config so the only thing the provider cares about is the line coding. So what happens sometimes is during a power outage of any sort the providers equipment has hard switches which are set to ami. During the power outage these switches are all thats left and the b8zs config is gone. This wouldn't be a big deal if you knew about it. The way to catch this is to loop one end back and run an all zero's test back to yourself. IF you can run error free and keep you're frame pattern syncronization up you can rule out the facility.

 

[ITsmyfault]

Hi - and thanks! Great idea.
In this case it turned out to be something on the carrier's network.. they wouldn't say what. Probably something like what you suggest.. they rebooted something and now it's fine again.

 

[ISDNman]

Ah yes, a case of CCBM......







From Newton's Telecom Dictionary 18th edition:

CCBM: Came Clear By Magic. This term created (but not contributed by) NYNEX (now part of Verizon). This is pseudo-technical lingo used by Nynex when they repair something that they broke (i.e. the problem was their fault), but they are unwilling to admit it was their fault. They say this "CCBM" to interconnect companies who are trying to get thier customers' phone lines fixed.

By the way, if you don't own Newton's you should. I have no affiliation with Newton's Telecom Dictionary or with the publisher CMP books. I just happen to think it is a very handy reference!