switching from T1s to wireless

[tekkie123]

we have 2 cisco 3620 routers connected over two T1s. we use multilink to load balance traffic. we have fastEthernet in slots 0 and 1 on both routers. we shut down the T1s and the multilink and brought up the second fastEthernet with motorola wireless radio towers plugged in to them - motos run at 19Mb/s. the network on one 3620 is 10.7.0.0/16, and the other has 10.10.0.0/16. the moto radio link is 172.22.0.0/16 and the multilink/T1s was 172.20.0.0/16.

after we do this, the network goes in and out. i think it's bad hardware on one of the 3620 slots or the fastEthernet cards in them. my associate wants to be sure it's not a config problem first.

any ideas where to look for issues?

 

[lerdalt]

Post your configs. I have a similar setup here in a couple of locations, and don't have the problem you have described.

 

[tekkie123]

configs: first sidea, then sideb
----------------------------------------------------------
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname sidea-sideb
!
ip subnet-zero
!
ip name-server xx.xx.xx.xx
ip name-server xx.xx.xx.xx
!
no call rsvp-sync
!
partition flash 2 16 16
!
interface Multilink10
ip address 172.20.1.2 255.255.255.252
ppp multilink
shutdown
multilink-group 10
!
interface FastEthernet0/0
description Ethernet interface to radio tower at 172.22.1.1 255.255.0.0
ip address 172.22.1.3 255.255.0.0
duplex auto
speed auto
!
interface FastEthernet1/0
description Ethernet interface for 10.7.0.0 network
ip address 10.7.1.1 255.255.0.0
duplex auto
speed auto
!
interface Serial1/0
description T1 1 to sideb
no ip address
encapsulation ppp
ppp multilink
shutdown
multilink-group 10
!
interface Serial1/1
description T1 2 to sideb
no ip address
encapsulation ppp
ppp multilink
shutdown
multilink-group 10
!
router eigrp 1
network 10.0.0.0
network 172.20.0.0
network 172.22.0.0
auto-summary
!
no ip classless
ip route 0.0.0.0 0.0.0.0 10.7.1.2
no ip http server
!
!
snmp-server community public RO
snmp-server enable traps tty
!
dial-peer cor custom
!
banner exec
Authorized Personnel Only.
!
end
-------------------------------------------------------
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname sideb-sidea
!
logging buffered 4096 debugging
logging console informational
!
ip subnet-zero
!
ip name-server xx.xx.xx.xx
ip name-server xx.xx.xx.xx
!
no call rsvp-sync
!
interface Multilink10
ip address 172.20.1.1 255.255.255.252
ppp multilink
shutdown
multilink-group 10
!
interface FastEthernet0/0
description Ethernet interface for 10.10.0.0 network
ip address 10.10.1.1 255.255.0.0
load-interval 30
duplex auto
speed auto
!
interface Serial0/0
description T1 1 to sidea
no ip address
encapsulation ppp
load-interval 30
ppp multilink
shutdown
multilink-group 10
!
interface Serial0/1
description T1 2 to sidea
no ip address
encapsulation ppp
load-interval 30
ppp multilink
shutdown
multilink-group 10
!
interface FastEthernet1/0
description Ethernet interface to radio tower at 172.22.1.2 255.255.0.0
ip address 172.22.1.4 255.255.0.0
duplex auto
speed auto
!
router eigrp 1
network 10.0.0.0
network 172.20.0.0
network 172.22.0.0
no auto-summary
!
no ip classless
ip route 0.0.0.0 0.0.0.0 10.10.1.4
no ip http server
!
snmp-server community public RO
snmp-server enable traps tty
!
dial-peer cor custom
!
banner exec
Authorized Personnel Only.
!
end

 

[lerdalt]

The big thing I question, not that it's an impact is why you aren't using a smaller mask on your wireless link.

I think I may have mis-read your post before. The connection over the wireless is dropping? Or is there a drop in connection between the time you drop the Serial interfaces and routing starts over the Wireless again?

If it's dropping while running on the wireless, you may want to look at the bridges and make certain they aren't dropping the association between them.

 

[tekkie123]

>The big thing I question, not that it's an impact is why you aren't using a smaller mask on your wireless link.

good point. out of the box, the motos come config'd 169.254.1.1 and 169.254.1.2 255.255.0.0. we just changed over to 172.22.x.x and didn't bother with the subnet mask. but you're right that there's only going to be a small number of items on that network.

>The connection over the wireless is dropping?

we don't know (or know enough to find out) where it drops. we have a program (intermapper) to monitor devices and services on both sides of the wireless/3620s. after we switch to wireless from the T1s devices are green (good and running) then red (dead), green, red,... etc. to us that means the network between us is going up and down. we don't think the wireless itself drops as it's steady 19Mb/s when not in use, and we have other identical wireless here that works rock steady.

 

[dtrtel]

>...monitor devices and services on both sides of the wireless/3620s. after we switch to wireless from the T1s devices are green (good and running) then red (dead), green, red,... etc. to us that means the network between us is going up and down.

This makes sense.

>...we don't think the wireless itself drops as it's steady 19Mb/s when not in use

So what else is there between the wireless bridges and the router on each end? A switch or larger fabric?

It's probably a good idea to set your speed and duplex on the FastEthernet Interfaces that connect to the bridges.


--jeff

http://linkedin.com/in/JeffLynch

 

[tekkie123]

>So what else is there between the wireless bridges and the router on each end? A switch or larger fabric?

the radio towers (PoE) have cat5 that runs through lightning arrestors and then directly into the fastEthernet on the routers. we thought maybe we're beyond the distance limit for cat5 from the radio to the router, but that's not the case. we have satellite buildings connected via the identical wireless systems on each side, run the same distance, and they run fine. the only difference is that they plug into cisco 3500 series switches.

>It's probably a good idea to set your speed and duplex on the FastEthernet Interfaces that connect to the bridges.

the wireless, as we said, runs around 19Mb/s full-duplex. the fastEthernet is, as you saw in the configs, set to auto.

should we set the fastEthernet to 100Mb/s full-duplex? or is there some config command to set a different speed, other than 10 or 100?

and if we need to do just that, why do the switches handle this fine and the fastEthernet on the routers don't?

a few more questions: 1) is there some debugging or logging on the routers that we can turn on which will help us pinpoint the problem? 2) after we switch to wireless, do we need to take the 'network 172.20.0.0 255.255.0.0' out of the 'router eigrp 1' - ie, that's not causing our problem, is it? 3) after switching we run a 'clear arp-cache' and 'clear counters' - is there anything else we should do? 4) so the configs look okay other than possibly the auto-neg speed/duplex?

 

[dtrtel]

For permanent or semi-permanent connection you *should* always set speed and duplex. Since your bridges are >10mbps and full duplex, set your FastEthernet to 100mbps/full-duplex, unless the installation guide says otherwise.

>and if we need to do just that, why do the switches handle this fine and the fastEthernet on the routers don't?

This is exactly why you set the speed and duplex explicitly. Autonegotiation isn't always consistent.

The 172.20.0.0 network is down when the T1s are shutdown, so EIGRP should remove the route from the routing table. After switching, I would also do advanced ping from the router to watch the link. Increase therRepeat count and datagram size up close to the max MTU (usually 1500 on Ethernet, so maybe 1400 since we're not looking at a fragmentation problem at this time), and see if you can get the link to fail. You'll be pinging the remote interface of router that is connected to the remote bridge 172.22.1.?.

mon-rtr-1#ping
Protocol [ip]:
Target IP address: 192.168.252.1
Repeat count [5]: 1000
Datagram size [100]: 1000
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1000-byte ICMP Echos to 192.168.252.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<snip>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/3/20 ms
mon-rtr-1#


--jeff

http://linkedin.com/in/JeffLynch

 

[jneiberger]

No, do NOT hard set speed and duplex. Leave both sides to auto and you'll be just fine. Statistically speaking, depending on the combination of hardware types, hard setting speed and duplex to 100/full is far more likely to cause problems than it is to help. Under many circumstances, hard setting both sides will still lead to a duplex mismatch. Just set both sides to auto and leave it that way unless you see evidence of a problem.

If either sides negotiates to half duplex then you have a problem and should hard set them, but that shouldn't happen if both sides are in compliance with the Fast Ethernet specs.

 

[tekkie123]

thanks jeff. we've set the speed/duplex to 100/full on both the routers and both the radios. we will not test until tomorrow AM early. i'll post the result.

 

[lerdalt]

I have to agree on the speed/duplex. Don't set it unless you can hard set both sides. No matter what, watch the errors on the interface.

I really think for what you have described, you need to try and look at the bridges themselves. I would hope there is some sort of logging on them so you can see if the radios are dropping.

I have a similar setup in a few areas, only leaving the T1's up as a backup. Only difference is I'm using OSPF instead of EIGRP.

 

[jneiberger]

I doubt this is related, but we had a Proxim bridge that was acting weird recently. It turns out that it simply couldn't handle being on a VLAN with a lot of broadcast and multicast traffic. It seemed to overwhelm the radio's administrative interface and it would reboot several times a day. We moved the admin interface into a VLAN with very little traffic and it hasn't rebooted since.

 

[dtrtel]

> Under many circumstances, hard setting both sides will still lead to a duplex mismatch.

My experience is exactly the opposite. Oh well.

--jeff

http://linkedin.com/in/JeffLynch

 

[dtrtel]

Ah, I correct myself. I have experienced problems with hard setting duplex. It's when I set it wrong. Full-duplex disables collision detection, which is required for old-school shared-media hubs to function. So the proper setting in that case would be half-duplex (on shared media hubs non-switched) or an interface that does not support full-duplex. Packet loss and slow throughput will tell you every time if you've got it wrong when you load it up with traffic.


--jeff

http://linkedin.com/in/JeffLynch

 

[tekkie123]

jeff et al, i have to agree with lerdalt and jneiberger - we set the speed and auto on both routers and radios yesterday for the switchover this morning and the radio on one side was saying "ethernet speed and duplex mismatch" this morning. we changed all back to auto.

our tower guys were out this past monday and must have done something - they claim they were checking alignment and putting in a key for higher speed - but whatever they did fixed our dropout problem.

thanks for all help

 

[jneiberger]

I'm glad it's working again. Try to troubleshoot radio dropouts can be an irritating process.

Regarding the duplex thing, let me hijack this conversation to explain what is actually happening.

The only method for setting speed and duplex mentioned in the Fast Ethernet standard is autonegotiation, also called Nway. There is no specified behavior for when a device has its speed and duplex manually configured. The problem is that there are two possible behaviors:

#1 Disable nway autonegotiation completely and only use the configured settings

#2 Participate in nway autonegotiation but only "offer" the configured settings

You'll have no problem if you connect devices with similar behaviors. However, you will have a problem if you connect a device using behavior #1 to a device using behavior #2. If a device with behavior #2 does not detect an autonegotiating link partner, it will assume that it is connected to a hub and drop back to half duplex despite being manually configured for full duplex. This *is* proper behavior! But it causes a big problem when it is connected to a device using behavior #1. That will results in a duplex mismatch.

All Cisco switches made in the last five or six years use behavior #1. They completely disable nway if you manually configure those settings. On the other hand, many common NICs tend to use behavior #2. You're gambling if you choose to use manually configured settings. It's best to go by the specs and use autonegotiation. That will assure that both sides are happy.

The only time you'll have a problem with autonegotiation is if you have a physical cabling problem, in which case you just need to fix your cabling, or if the NIC or switch hardware is out of spec. I haven't seen that in several years.

I hope that clears it up. It can be a confusing mess.

 

[tekkie123]

thanks for the background.

we spoke too soon. the link ran solid for 20-30 minutes, then began dropping out, just like before. we suspect that there was not much traffic when it was solid. we ran pings, connected to a few services on the other side of the link, etc, and it all seemed good.

then came the drops. so we're back to square one.

 

[jneiberger]

Can you check the uptime on the radios? Just for grins, let's see if they're rebooting. Also, can you check the signal on them? The Proxim radios that we use have a tool to monitor the signal. If you have a strong signal, it shouldn't be dropping out like that.

Are you seeing a change of status on any interfaces?

 

[lerdalt]

Another thought for testing (probably already mentioned) use extended ping from interface to interface and adjust the packet size. I've had this on a t1 circuit where a small packet size ran fine, but as we increased it we got more and more drops.

I agree on troubleshooting the wireless dropouts. Unless you have good logging or good visibility into the device, it can be almost impossible. We inherited a set of bridges once that had absolutely no management to them. "Set it and forget it" sort of thing. To make it worse, we couldn't connect them to a switch, for whatever reason, it would only stay up if the one end was connected to a hub.

 

[dtrtel]

I suppose I show my age with this fixed speed/duplex vs auto disagreement. Not a problem, I agree that auto is sometimes the appropriate setting, but usually because of anomalous behavior or convenience, not whether it's right or wrong to set it for infrastructure links (unless GigE then auto only). But recall that I did mention 100/full as a first try unless the install manual says otherwise. Just because the link fails at 100/full, doesn't mean auto is better. So what if 100/full-duplex isn't stable, the next thing to try, if so inclined, would be 100/half-duplex.

The larger point is when you auto negotiate, what does it settle on? Does it change from time to time? Does it have to re-negotiate from time to time?

But that's not really the point now. Now you may desire to ease your troubleshooting environment. You really don't have to do "switchovers" to troubleshoot this. You can have the T1s and the wireless link up at the same time. Just remove the eigrp announcement for network 172.22.0.0 until you get the wireless link stable. The traffic between sites will only know the one route via 172.20.x.x provided your T1s are up.

Then you can use extended ping at your convenience per my example above from directly attached router to directly attached router to find the problem and make it stable. You can also do extended ping test between the router and it's directly attached radio on each end of the link provided the radio management IP addresses are in the same subnet as the router interfaces directly attached.

--jeff

http://linkedin.com/in/JeffLynch