Layer2 vs Layer3

[jkaftan]

Sorry, this is a long one.

We have a flat layer 2 network with a single 6509 at the core. It's a collasped core so we are doing our routing on the same 6509. We have been running this way for 8 years now but have gotten a bit VLAN happy. We are a school and and it has become convienent to have a seperate VLAN for each building, LAB, and various other special use networks. I find myself saying, "We'll just create another VLAN.....". We are up to about 70 VLANs now that are all being handled by our 6509 VTP server.

Some of the problems we are having is excessive BPDU traffic due to Cisco's PVST that may be leading to poor performance on our wireless access points that are trunked to our network. Also our 2950s at the access layer cannot handle more than 64 VLANs.

Sould we move to a model of routing at each building to the core rather than going layer 2 to the core and routing there? That way the core would not need to know about VLANs in each building. It would just need to know how to route to each building. Also, down the road, as we build up a DR data center we would have a way to get to it if the core swtich went down as we would be making a routing decision at each building.

What do you think?

 

[unclerico]

Some of the problems we are having is excessive BPDU traffic due to Cisco's PVST that may be leading to poor performance on our wireless access points that are trunked to our network. Also our 2950s at the access layer cannot handle more than 64 VLANs.

you talk about VTP so are you pruning VLANs?? as for poor wireless performance, i don't know what your coverage is in terms of AP placement, but the APs could be getting overloaded with clients.

Sould we move to a model of routing at each building to the core rather than going layer 2 to the core and routing there?

typically this is desirable. one side effect of this design comes from the wireless perspective. unless you have an enterprise scale solution you're going to end up with many different wireless network ranges. the good wireless vendors will allow you to do identity based tunnels so that you can have a single network range for each SSID regardless of whether or not you cross a L3 boundary. another side effect is cost. you'll need multilayer switches in each building and they cost a lot more than your typical access layer/edge switches

It would just need to know how to route to each building

static routing or dynamic routing depending on how many paths you have from each building back into the core

Also, down the road, as we build up a DR data center we would have a way to get to it if the core swtich went down as we would be making a routing decision at each building.

you will want to have redundant core then. if the connection from DR is going into the core and it fails, it doesn't matter how many paths you have downstream.

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[VinceWhirlwind]

Just one point here - ask yourself why on earth your edge 2950s would want to see any more than 2 or 3 VLANs each.

Nothing wrong with lots of VLANs on a core.
Nothing right about lots of VLANs on an edge.

I'd ditch VTP and stick with a design instead.

 

[vipergg]

If you aren't manually pruning all trunk links this could be your problem. All trunks should be "manually" pruned , do not rely on vtp pruning . All trunks links on both sides should have the statement "switchport trunk allowed vlan xx where xx is only the vlans that are actually needed on that switch . This willl also cure your issue with your 2950's and spanning tree instances . This stops all broadcasts and arps from going to all cpu's where it has to handled or dropped if its not on that switch . A lot of people just use the statement switchport mode trunk and never prune anything and this allows all vlans across the trunk along with all the traffic that is not needed down at the switch level. This will also cut down on bpdu's going down to switch if its not even allowed across the trunk.

 

[jkaftan]

Yes I am pruning but not manually. I had played with this in the past and found that even if I did pruning and "manually pruned" that I still could see all of my VLANs when I did a show vlan on an uplinked switch. I also saw that each uplinked switch still had an instance of spanning tree going for every vlan. So I determined that even though I turned on pruning and mannually pruned I was still dealing with all of those spanning tree instances at the access layer. I never did a packet capture to see of BDPUs were coming over or not.

I even called Cisco on this and they told me the only way to really knock down the instances of Spanning tree on each access layer switch was to put them into Transparrent mode and then delete all of the VLANs I do not care about.

I have played with this and it does seem to work. I just have not gotten around to doing this on every switch. Plus I am waiting to see if any issues come up.

We looked at this issue some more today and noticed some funky things with Spanning Tree. Three of the VLANs involved with wireless were not the root bridge on our core switch where we expected them to be. When we looked on the core their priority was set to 5048. We have all of the other VLANs on the 6509 set to 8192 (Cisco's Root). We're not sure how to find the other device acting as root bridge. The MAC address that was given as the root bridge did not lead us anywhere. So we just set the priority for those VLANs to 0 and it seems to have calmed down a bit although we are not seeing a big change in throughput.

 

[Cluebird]

Look into RSTP and MIST to help with your spanning-tree. Add bpduguard to your edge ports and rootguard to protect the root bridge. You may also want to look into implementing Dynamic ARP Inspection and DHCP snooping.

I'd definitely trim the VLANS on the trunks with allowed lists as vipergg states.

 

[vipergg]

I would disagree with cisco a little on the spanning tree issue . We have a setup where a 2950 is running in a client server environment with 80 vlans . Just because you see say 100 vlans does mean there is a spanning tree instance created for each one what you are seeing is just a vtp advertisement and really has nothing to do with spanning tree instances. If you do not manually prune then yes a spanning tree instance will be created by the switch . Manually prune off whatever is not needed and check with a show spanning tree active and you will see only the ones allowed across that trunk as a active spanning tree instance. Below is that switch ...


xxxxxxx#sh vtp status
VTP Version : 2
Configuration Revision : 361
Maximum VLANs supported locally : 250
Number of existing VLANs : 83
VTP Operating Mode : Client
VTP Domain Name : XXXXXXXXXX
VTP Pruning Mode : Enabled
VTP V2 Mode : Enabled
VTP Traps Generation : Disabled
MD5 digest : 0x17 0x50 0x0C 0x58 0x81 0x86 0x7E 0x9E
Configuration last modified by 150.102.250.134 at 10-20-10 12:55:17
kpb207f01a_s# sh int trunk

Port Mode Encapsulation Status Native vlan
Gi0/1 desirable 802.1q trunking 757

Port Vlans allowed on trunk
Gi0/1 757

Port Vlans allowed and active in management domain
Gi0/1 757

Port Vlans in spanning tree forwarding state and not pruned
Gi0/1 757
kpb207f01a_s#sh span
kpb207f01a_s#sh spanning-tree act
kpb207f01a_s#sh spanning-tree active

VLAN0757
Spanning tree enabled protocol ieee
Root ID Priority 32768
Address 000e.3972.82f4
Cost 23
Port 25 (GigabitEthernet0/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33525 (priority 32768 sys-id-ext 757)
Address 0012.8040.8a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Fa0/6 Desg FWD 19 128.6 Edge P2p
Fa0/7 Desg FWD 19 128.7 Edge P2p
Gi0/1 Root FWD 19 128.25 P2p

 

[Quadratic]

jkaftan, when it comes to tracking down a root bridge you didn't expect, "show spanning vlan ..." will give you the forwarding root port for the non-root switch, so you can trace it that way. Are you hard-coding your access ports and filtering BPDUs where you don't expect them? I agree with Cluebird on those points.

For the VTP question, my personal preference (and Cisco's, officially) is to use VTP transparent across the board for a number of reasons. But if you do insist, at a minimum you need to find a way to prune vlans off trunks that don't need them.

For the question of how far L3 should be extended, in my opinion that depends on a lot more than the info that's been presented. How large is "a building" to you? The size, use and nature of your network will help gauge how modular you need to be, where your L2/L3 barrier should be, and how much cost is justified.

As for BPDU's, 80 STP instances by default comes to 40 35-byte frames per second, or 11.2 Kbps of overhead, which isn't that unreasonable. I've seem people get a LOT more vlan-happy than that.

CCNP, CCDP

 

[MrOyvind]

On our MAN we are running transparent VLAN`s on the 6509 switces.
The 6509`s are placed in fiber rings with redundance and runs OSPF\IBGP\MPLS VPN with trunks to each location. Each BVI on the 6509`s is redisributed into MPBGP.
Some of the 6509`s have up to 250 VLAN`s.
The first access switch on each loacation runs as server, and the switches back as clients. As we have to manually create each VLAN we want on the VTP server at each site, everything is under full controll, this will stop accidently creating new VLAN`s on the 6509`s, and only the desirable VLANS are visible at the sites. Typicially running 2960 and 2950`s.
Switchport trunked allowed vlan are used to advertise only the VLAN`s we want. We have good experience with BPDU guard that stops IT personel, employers, students etc. on the sites making loops on access ports ( the spanning tree port fast ports ).
The CPU on two of the 6509`s was pushed to the limit and rebooted because of loops created for at time back.
IP DHCP snooping works well to, the log shows many hits. We never know what someone does on our 250-300 sites.
ARP insepection wil be implemented on the schools, since we have experince with small hackers.
Works very well.
Sorry about my english.

Best regards

 

[jkaftan]

Thank you all for your feedback. Sorry it has take me so long to respond. Each building is typically <100 clients so we have lots of small VLANs. What I gather from all of your responses is that I should go to transparent mode and lock down my VLANs crossing links. I have played around with that and feel I can make that work.

The other part I am not sure about is how to get my folks to DR if I do not route at each building. I guess I could have 2 layer two connections to each closet, 1 to the core and 1 to a router that could lead me to DR. Then I suppose I could configure VRRP between the core router and the DR router so my clients could still get to DR if the core went down. I don't think we would ever want an automatic failove to DR should our core go down so we have some flexibility there. I mean I could manually just change the gateway IPs via a script on the DR router so folks would have access etc.