Network Issues - Communication between vlans

[mamoser]

Hi everyone,

I really hope someone can help me out here. I've need to troubleshoot a network that has extreme issues. However, the client has mentioned that it's not broken so don't break it due to critical data flowing.

Here's the messed up setup:

Server1 (Vlan1)
|
Switch1 (Vlan1)
|
Switch2 (internal lan switch) - FW - Switch3 (another internal sw) - PC
|
Switch4(Vlan30)
|
Server2 (Vlan30)

I'm concerned about their setup but have no clue where to start troubleshooting. They do not want to implement any layer 3 device but need to keep the traffic from different vlans segregated.

What I find alarming is the fact that a wirshark capture from server 2 shows communication between two different servers on vlan 1 from Switch1.

How is this possible? Could a hacker be on the network and turning the switch into a hub by flooding the CAM table?

Here's the output for arp/CAM table:

PSA#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.12.175.226 232 0015.173e.1055 ARPA Vlan30
Internet 10.12.175.231 4 0015.1771.55e0 ARPA Vlan30
Internet 10.12.175.201 - 001d.e524.6042 ARPA Vlan30
Internet 10.12.175.221 18 001c.23c2.0742 ARPA Vlan30
Internet 10.12.186.201 184 0024.9778.bef0 ARPA Vlan30
Internet 10.12.175.222 3 001c.23e2.8cc4 ARPA Vlan30
Internet 10.12.184.52 191 0024.9778.bef0 ARPA Vlan30
Internet 10.12.175.36 7 0013.7270.7000 ARPA Vlan30
Internet 10.12.175.13 9 0001.0284.b7cf ARPA Vlan30
Internet 10.12.175.17 15 0009.444a.2bff ARPA Vlan30
Internet 10.0.1.178 87 0024.9778.bef0 ARPA Vlan30
Internet 10.12.175.23 198 0090.2798.44ed ARPA Vlan30
Internet 10.0.2.204 170 0024.9778.bef0 ARPA Vlan30
Internet 10.12.175.112 1 0090.a408.117c ARPA Vlan30
Internet 10.12.175.117 9 0013.2052.d6b6 ARPA Vlan30

PSA#sh mac-address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All ffff.ffff.ffff STATIC CPU
10 0014.0b20.0a47 DYNAMIC Gi0/24
10 0014.0b20.0a53 DYNAMIC Gi0/24
10 0014.0b20.0a59 DYNAMIC Gi0/24
10 0014.0b20.0a5b DYNAMIC Gi0/24
10 0014.0b20.0d32 DYNAMIC Gi0/24
10 0014.0b20.0fba DYNAMIC Gi0/24
10 0014.0b20.0fd4 DYNAMIC Gi0/24
10 0014.0b20.0fda DYNAMIC Gi0/24
10 0014.0b20.0fdc DYNAMIC Gi0/24
10 0014.0b20.0fe2 DYNAMIC Gi0/24
10 0014.0b20.0fec DYNAMIC Gi0/24
10 0014.0b20.100a DYNAMIC Gi0/24
10 0014.0b20.100c DYNAMIC Gi0/24
10 0014.0b20.100e DYNAMIC Gi0/24
10 0014.0b20.1010 DYNAMIC Gi0/24
10 0014.0b20.1016 DYNAMIC Gi0/24
10 0014.0b20.101c DYNAMIC Gi0/24
10 0014.0b20.1040 DYNAMIC Gi0/24
10 0014.0b20.1042 DYNAMIC Gi0/24
10 0014.0b20.10ac DYNAMIC Gi0/24
10 0014.0b20.10b8 DYNAMIC Gi0/24
10 0014.0b20.10bc DYNAMIC Gi0/24
10 0014.0b20.10c6 DYNAMIC Gi0/24
10 0014.0b20.10c8 DYNAMIC Gi0/24
10 0014.0b20.10cc DYNAMIC Gi0/24
10 0014.0b20.10d2 DYNAMIC Gi0/24
10 0014.0b20.10e4 DYNAMIC Gi0/24
10 0014.0b20.110a DYNAMIC Gi0/24
10 0014.0b20.11c0 DYNAMIC Gi0/24
10 0014.0b20.11c6 DYNAMIC Gi0/24
10 0014.0b20.1232 DYNAMIC Gi0/24
10 0014.0b20.1244 DYNAMIC Gi0/24
10 0014.0b20.1260 DYNAMIC Gi0/24
10 0014.0b20.1266 DYNAMIC Gi0/24
10 0014.0b20.128c DYNAMIC Gi0/24
10 0014.0b20.12d6 DYNAMIC Gi0/24
10 0014.0b20.1356 DYNAMIC Gi0/24
10 0014.0b20.1364 DYNAMIC Gi0/24
10 0014.0b20.136e DYNAMIC Gi0/24
10 0014.0b20.137c DYNAMIC Gi0/24
10 0014.0b20.13a0 DYNAMIC Gi0/24
10 0014.0b20.13ca DYNAMIC Gi0/24
10 0014.0b20.13cc DYNAMIC Gi0/24
10 0014.0b20.13f4 DYNAMIC Gi0/24
10 0014.0b20.2530 DYNAMIC Gi0/24
10 0014.0b20.2546 DYNAMIC Gi0/24
10 0014.0b20.256e DYNAMIC Gi0/24
10 0014.0b20.259c DYNAMIC Gi0/24
10 0014.0b20.259e DYNAMIC Gi0/24
10 0014.0b20.25b0 DYNAMIC Gi0/24
10 0014.0b20.25b6 DYNAMIC Gi0/24
10 0014.0b20.25c0 DYNAMIC Gi0/24
10 0014.0b20.25ce DYNAMIC Gi0/24
10 0014.0b20.25d2 DYNAMIC Gi0/24
10 0014.0b20.25da DYNAMIC Gi0/24
10 0014.0b20.260e DYNAMIC Gi0/24
10 0014.0b20.2610 DYNAMIC Gi0/24
10 0014.0b20.2612 DYNAMIC Gi0/24
10 0014.0b20.262e DYNAMIC Gi0/24
10 0014.0b20.265c DYNAMIC Gi0/24
10 0014.0b20.265e DYNAMIC Gi0/24
10 0014.0b20.2674 DYNAMIC Gi0/24
10 0014.0b20.267c DYNAMIC Gi0/24
10 0014.0b20.267e DYNAMIC Gi0/24
10 0014.0b20.2690 DYNAMIC Gi0/24
10 0014.0b20.26a4 DYNAMIC Gi0/24
10 0014.0b20.26a6 DYNAMIC Gi0/24
10 0014.0b20.26ba DYNAMIC Gi0/24
10 0015.173e.098a DYNAMIC Gi0/3
10 0019.b9f1.a44d DYNAMIC Gi0/7
10 001c.23c2.0744 DYNAMIC Gi0/6
10 001c.23e1.4c5c DYNAMIC Gi0/2
10 001c.23e2.8b22 DYNAMIC Gi0/4
10 001c.23e2.8cc3 DYNAMIC Gi0/1
10 001d.a1d6.1e30 DYNAMIC Gi0/24
10 0040.ca81.534b DYNAMIC Gi0/24
30 0003.2d0b.9475 DYNAMIC Gi0/21
30 0003.2d0c.d20d DYNAMIC Gi0/21
30 0003.2d0e.0b19 DYNAMIC Gi0/21
30 0004.23bb.7f9e DYNAMIC Gi0/21
30 0008.02df.9b44 DYNAMIC Gi0/21
30 0009.444a.2bce DYNAMIC Gi0/21
30 0009.444a.2bfb DYNAMIC Gi0/21
30 000b.cdc5.632c DYNAMIC Gi0/21
30 000b.db92.f85a DYNAMIC Gi0/21
30 0010.4b2f.b9ee DYNAMIC Gi0/21
30 0011.0a9d.1e9f DYNAMIC Gi0/21
30 0011.43dc.7fdc DYNAMIC Gi0/21
30 0013.724d.31d5 DYNAMIC Gi0/21
30 0013.724d.ba67 DYNAMIC Gi0/21
30 0013.7270.3577 DYNAMIC Gi0/21
30 0013.7270.7000 DYNAMIC Gi0/21
30 0014.2211.4100 DYNAMIC Gi0/21
30 0014.221c.2b42 DYNAMIC Gi0/21
30 0014.225c.2665 DYNAMIC Gi0/21
30 0015.173e.1055 DYNAMIC Gi0/16
30 0015.1771.55e0 DYNAMIC Gi0/21
30 0015.1771.55e9 DYNAMIC Gi0/21
30 0019.e24f.8316 DYNAMIC Gi0/21
30 001a.6b67.d061 DYNAMIC Gi0/21
30 001c.23c2.0742 DYNAMIC Gi0/14
30 001c.23c4.6ee1 DYNAMIC Gi0/21
30 001c.23d9.6ddb DYNAMIC Gi0/21
30 001c.23e1.4c5d DYNAMIC Gi0/15
30 001c.23e2.8bec DYNAMIC Gi0/22
30 001c.23e2.8cc4 DYNAMIC Gi0/13
30 001d.7eae.d0f4 DYNAMIC Gi0/21
30 001d.a1ca.9c16 DYNAMIC Gi0/22
30 001d.e548.8842 DYNAMIC Gi0/24
30 001d.e68a.c307 DYNAMIC Gi0/21
30 001d.e68a.c340 DYNAMIC Gi0/21
30 001f.9d13.b116 DYNAMIC Gi0/21
30 001f.c906.8402 DYNAMIC Gi0/21
30 0020.4a86.1893 DYNAMIC Gi0/21
30 0021.9b32.7cb3 DYNAMIC Gi0/21
30 0021.9bb3.7f43 DYNAMIC Gi0/21
30 0024.9778.bef0 DYNAMIC Gi0/21
30 0050.8b69.1915 DYNAMIC Gi0/21
30 0090.275b.e9bd DYNAMIC Gi0/21
30 0090.2798.44ed DYNAMIC Gi0/21
30 0090.2798.a162 DYNAMIC Gi0/21
30 0090.a408.117c DYNAMIC Gi0/21
30 00b0.d076.8e5f DYNAMIC Gi0/21
30 0201.0a0c.afd6 DYNAMIC Gi0/21
30 0202.0a0c.afd6 DYNAMIC Gi0/21
1 001d.a1d6.1e30 DYNAMIC Gi0/24
Total Mac Addresses for this criterion: 146

I'm in a tough position, they don't want to add any new equipment and they can't afford the system to go down.

I also noticed that they are using the same subnet 10.12.175.0/24 for both vlan1, vlan30 and for the internal switches. The tech onsite states that this is not a problem since they do not use intervlan routing.

Could someone point me in the right direction to fix this god awful mess?

 

[North323]

what is the problem?

 

[jeter]

What's the subnet mask on your network?

Did you segment the network ip's for each vlan?

Suggestion would be to install a router, implement 802.1q
encapsulation to segment the networks.

Program the switches with trunk ports to pass the vlans. Verify switchports are programmed correctly for vlans.









Go Army!
Tek-TIP Member 19,650

 

[unclerico]

I would like to see switch configs. Where does the firewall come into play?? Are the PC's on the right side accessing server1 or server2 on the left and vice versa??

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[mamoser]

Guys,

Sorry for not getting back to you sooner. The request to post the configs was denied. With that being said, I had to troubleshoot this entire setup even though access to certain switches was restricted. Suffice to say, I resolved the problem with basic troubleshooting.

The config I was told was completely off, so much so, I've lost respect for their network team.

Sw1 had no vlan assignment. It was directly connected to another switch that had no vlan assignment as well. This led to traffic flowing from one switch to the other. I then noticed that someone had created a trunk port from the second switch to the third one by passing the firewall. This trunk port was propagating the traffic from the switch to the third one.

I will now take the appropriate steps and assign a vlanid and all ports on Sw1 to Vlan50. I will also remove the trunk port since it's not required.

Thanks to all those that posted their comments. It's very much appreciated when you are stuck and under a lot of pressure to fix something.

I guess this is the life of an network admin, dealing with complex problem created by idiots that have certs but no nothing.

Evan

 

[unclerico]

Evan, thanks for posting back. I completely understand where you're coming from. I've been at my current position for almost 2 years and I'm STILL cleaning up crappy implementations at my remote locations.

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[mamoser]

I have another question for you guys regarding a new problem.

There is a server running linux that requires multicast traffic (40 address' to be exact). At the moment, they have it setup so that it flood the switch which in turn makes it to the linux server through one of the ports.

I found out yesterday that their increasing the traffic to 80mbps for each stream but noticed that they are not doing any kind of multicast control. This means that the traffic is flooding through all the daisy chained switches.

*Another example of poor network admins creating horrible designs.

Looking into the custom linux server, I noticed that IGMP does not work. Wireshark captures do not show IGMP join requests from the linux box.

After speaking with the support team for the custom linux box, I found out that the feature is there but does not work and they don't plan on fixing it. (Great!)

I'm really scared of the effects of 40 streams of 80mbps flooding all the switches. Being someone that hates getting woken up at 3am because the entire network is down, I've decided to be proactive and find a solution.

My thoughts are the following:
I heard that you can create static port mapping and force specific mcast address' to a port.

Do you think the amount of packets with this setup will affect the switches performance (Cisco 2960s) or even worst take it down?

Also, could someone point me to documentation on setting this up?

This is what I've found so far:

Get MAC for the mcast address.
230.0.0.10 = 0100.5e00.000A

Issue Static Port Mapping on Sw:
2960(config)#mac-address-table static 0100.5e00.000A vlan 10 interface g0/24

I'm guessing I'll need to enable IGMP snooping right? Can this be done without having a L3 device?

Thanks again,

Evan

 

[burtsbees]

http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/l2pt.html

/

 

[burtsbees]

Crap...though that doc may prove to be useful (encapsulating with MPLS WITHIN your LAN? Huh?), 'tis the wrong link...

I believe you may want to read this one...and I agree...poor design...40 lashes with a wet noodle!

ftp://ftp-eng.cisco.com/ipmulticast/training/Module2.pdf

/

 

[mamoser]

Thanks for the doc, I found it really usefull. However, this brings up a question.

Is there any way of controlling multicast without a layer 3 device. I noticed when I use the following command on my Cisco 3750 (no ip services module) I still see multicast flooding:

conf-t)#ip igmp snooping vlan 20

This leads me to believe that I need to setup a querier/RP/mrouter port which I don't have. I was hoping to be able to stop the flooding without a l3 device and use static port entries using the multicast address' MAC.

Is this even possible?

Thanks,

Evan

 

[burtsbees]

I am not sure, but I don't think so. What kind of switch is this? If it is L3, then you could implement acl's, MAC or IP.

Also, this statement...

"What I find alarming is the fact that a wirshark capture from server 2 shows communication between two different servers on vlan 1 from Switch1.

How is this possible? Could a hacker be on the network and turning the switch into a hub by flooding the CAM table?"

That depends on how the capture was set up. Looks to me that someone may have a SPAN config in there...

/

 

[mamoser]

Burtsbee,

Don't worry about the above problem that has since been resolved.

From my understanding of what I've read so far, I don't think it's possible to control multicast with just a layer 2 device.

The Cisco 3760G that I have does not have ip services module. It's just a L2 sw and from what I understand, my superiours are not interested in purchasing the L3 services.

The question that I'm looking for the answer to is the following:

Is there anyway of controlling multicast at Layer2 without having a L3 device involved?

I thought this was possible with igmp snooping and converting multicast IP to mac then create static entries. However issuing the ip igmp snooping command does nothing and multicast is still flooding to all ports.

Hope this clears it up a little.

 

[unclerico]

you said that the application does not use igmp so do you know what multicast protocol(s) the application uses?? igmp snooping will only work for igmp traffic. you could do what you said and create static mac mappings for each client to group association. it will not scale well, but it may be your only option.

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[mamoser]

Here's the current setup in my lab:

Multicast Server (smart bits)
|
Cisco 2960 or Cisco 3760G
|
Linux box

The Linux server is really a special transparent bridge which does not have IGMP capabilities. The smart bits floods 40 streams to the Sw which in turn floods all ports and the Linux box gets the mcast traffic.

I don't want flooding happening on the sw so I need a command that stops the flooding. I thought that i could use igmp snooping but it seems as if I need a L3 device for it to work.

Does anyone know a way of stopping the flooding without a L3 device? Once I manage to stop the flooding, I will then use static entries for my test.

Obviously, I'll do some baseline and performance testing to ensure it meets the requirements of the service offered before putting it onto the production system.