Multiple return routes

[OutOfHisElement]

Here's a fun problem: I have two VLANs (A & B), each with a subnet of its own. I have a server that is connected to both VLANs, so it has an address on each subnet. If I ping the server's address on A from a machine on VLAN B, no go. Swap A and B in the last sentence and you get another true statement.

And it's not just ping. Nothing works.

I suspect that what is happening is that the return traffic is traveling by a different route. Does that seem like it could be the problem? Any suggestions for a solution?

The server is a Linux box.

 

[jimbopalmer]

I am unclear why you would put a server on multiple subnets unless you wanted to address it 'locally' on each subnet.

Do other devices on the two subnets see each other when you ping?

I always used the local subnet on my unix servers and made the route daemon quiet.

routed -q

http://www.oreillynet.com/linux/cmd/cmd.csp?path=r/routed

My Sun Engineer said "Let your routers route, and your servers serve."

I tried to remain child-like, all I acheived was childish.

 

[OutOfHisElement]

That's exactly what I'm trying to do. Here is my motivation: The server is connected to the network with a 4X GigE aggregate link (using a Linux bonding device, like EtherChannel). If I go through our router, then one of the subnets can only access the server at 1XGigE. We could solve this with an expensive router, but this is something I'd rather not do.

As we add more VLANs, this will be still more of a problem.

All the other hosts can ping one another as they should. It's just this one server that has a problem when it is accessed via the address on the "other" subnet/VLAN. This is more of an inconvenience than a genuine problem.

The symmetry surprises me. I would expect that it would work from one network and not the other.

 

[jimbopalmer]

I am still not feeling the inconvenience, It works locally in both subnets, and gets 4x gig throughput locally. It does not work through the router, but would only get gig throughput if it DID work through the router.

I tried to remain child-like, all I acheived was childish.

 

[OutOfHisElement]

It's not all that inconvenient, and you are right that the access via the local subnets gets full throughput. But you'd be surprised how often one tries to access a host via the wrong subnet and then goes off on a wild goose chase. (I spent about an hour on such a goose chase yesterday.) In my opinion, all addresses should work.

But mostly, I just think it is an interesting issue.

In writing this reply, I've come to the conclusion that what is happening is as follows: The hosts can't use the other VLAN, so they use the router. The server receives the traffic, sees that it is from a subnet that it is directly connected to (as it is connected to both VLANs) so it by-passes the router. Traffic from hosts to the server uses the router, but return traffic doesn't.

To be honest, I can't see why this wouldn't work.

 

[jimbopalmer]

Tell me more about what options you have tried for routed. If you do not start it at all, what effects?
If you run routed -q, what effects?
If you run routed -s, what effects?

routed -q worked for me, except for tftp across subnets.

I tried to remain child-like, all I acheived was childish.

 

[OutOfHisElement]

I'm not using routed on the server. Just static routes for the two VLANs. These routes are created when the interfaces are brought up (ie. I had no chance to screw them up).

Interestingly, I've ran tcpdump on the server. It receives pings to the "wrong" address from the other VLAN, but doesn't send anything in return. This makes me think that the problem is with the server and not the router or the workstation.

 

[jimbopalmer]

I'm not using routed on the server."

I bet you are. Did you disable it in the startup script in your linux? Perhaps /etc/rc.d/init.d/

http://www.policyrouting.org/PolicyRoutingBook/ONLINE/CH01.web.html

Section 1.5.1 talks about routed some more

I really suspect forcing -q on routed will help.

I tried to remain child-like, all I acheived was childish.

 

[OutOfHisElement]

I bet you are."

In the most good natured way: I'll take that bet. ;-)

"Did you disable it in the startup script in your linux? Perhaps /etc/rc.d/init.d/"

Nope. There was no need as routed isn't installed. Just to be sure, I ran a find. Nothing.

In the documentation you reference, it makes it clear that routed is used for dynamic routing with RIP. On this server, there was exactly one interface (good ol' eth0) until I added a VLAN last week. Eventually, after I have a few VLANs, I'll start using dynamic routing, but as of right now, there is really no need. It's just plain old static routing.

Unfortunately, I've been swamped with higher priority tasks. Next week should yield some to focus on this problem. Looking forward to it.

 

[OutOfHisElement]

I looked into this a little further yesterday and was able to find a work around.

Just to review, I was looking for a way to have a Linux server on two VLANs (the default and VLAN 5) and corresponding subnets. The server would have direct access to each VLAN/subnet, allowing it to use two aggregated GigabitEthernet connections. For workstations, which are only on one of the VLANs/subnets, access from one VLAN/subnet to the other would be controlled by a Cisco router. This worked, except that I couldn't access the server by its address on one VLAN/subnet by a workstation on the other VLAN/subnet. In this situation, the server would receive pings from the workstation, but would not send replies (confirmed by running tcpdump on the server).

I had configured the router to access the one VLAN via one interface and the other by another interface. On a whim, I decided to change this so that both VLANs were trunked over a single interface, using a sub-interface configured to access VLAN 5. For whatever reason, this works, and very nicely.

The only gotcha is that both VLANs share a single GigE connection. For our application this is a non-issue, at least for the foreseeable future.

My guess is that this was a Linux configuration issue, but can't be sure yet.