Weird Comcast routing

[pcrequest]

I noticed that our latency from our Minneapolis office (which has 50/10 Comcast Business class) to a server in our Chicago office (a different ISP), is either routed a fast way through Chicago, or a slower route through Denver->Dallas->Chicago depending on which public, static IP address we connect from in Minneapolis. Comcast assigned us 11.11.0.8/29 subnet, so there are 5 addresses I can assign after Comcast assigns our gateway 11.11.0.14. We run a latency sensitive service over this connection. We noticed it was either roughly 41 ms round trip, or 18 ms round trip. Doing traceroutes from Minneapolis, we see it either routes through Denver, or through Chicago. If we assign our Minneapolis server 11.11.0.9 or 11.11.0.11 it always routes through Chicago and is fast:


>ipconfig

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 11.11.0.11
Subnet Mask . . . . . . . . . . . : 255.255.255.248
Default Gateway . . . . . . . . . : 11.11.0.14

>tracert 66.28.82.19

Tracing route to 66.28.82.19 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]
2 * * * Request timed out.
3 4 ms 2 ms 3 ms ge-9-1-ur01.minnetonka.mn.minn.comcast.net [68.85.164.113]
4 16 ms 7 ms 5 ms te-8-1-ur02.crosstown.mn.minn.comcast.net [68.86.232.33]
5 7 ms 7 ms 8 ms 68.87.174.193
6 18 ms 17 ms 17 ms te-0-4-0-5-cr01.chicago.il.ibone.comcast.net [68.86.91.185]
7 17 ms 18 ms 17 ms pos-0-0-0-0-pe01.350ecermak.il.ibone.comcast.net [68.86.86.34]
8 18 ms 17 ms 21 ms te0-2-0-1.ccr22.ord03.atlas.cogentco.com [154.54.11.253]
9 17 ms 17 ms 17 ms te0-0-0-1.ccr21.ord01.atlas.cogentco.com [154.54.1.1]
10 21 ms 18 ms 17 ms te0-1-0-0.mpd21.ord01.atlas.cogentco.com [154.54.30.250]
11 19 ms 173 ms 294 ms te4-2.mpd01.ord05.atlas.cogentco.com [154.54.3.206]
12 19 ms 17 ms 18 ms vl3805.na01.b000268-0.ord05.atlas.cogentco.com [66.28.6.86]
13 18 ms 19 ms 17 ms 66.28.82.19

Trace complete.



If we assign the same server 11.11.0.10, 11.11.0.12, 11.11.0.13 It routes through the slower Denver route.

>ipconfig

Windows IP Configuration


Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 11.11.0.12
Subnet Mask . . . . . . . . . . . : 255.255.255.248
Default Gateway . . . . . . . . . : 11.11.0.14

>tracert 66.28.82.19

Tracing route to 66.28.82.19 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]
2 * * * Request timed out.
3 6 ms 7 ms 7 ms ge-9-1-ur01.minnetonka.mn.minn.comcast.net [68.85.164.113]
4 7 ms 9 ms 8 ms te-8-1-ur02.crosstown.mn.minn.comcast.net [68.86.232.33]
5 7 ms 9 ms 7 ms te-8-3-ar02.roseville.mn.minn.comcast.net [68.87.174.6]
6 26 ms 25 ms 25 ms te-1-0-0-5-cr01.denver.co.ibone.comcast.net [68.86.93.133]
7 43 ms 41 ms 41 ms pos-0-8-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.85.178]
8 43 ms 43 ms 46 ms pos-0-1-0-0-pe01.1950stemmons.tx.ibone.comcast.net [68.86.86.94]
9 41 ms 41 ms 41 ms as174-pe01.1950stemmons.tx.ibone.comcast.net [75.149.230.150]
10 42 ms 43 ms 42 ms te7-4.mpd01.dfw01.atlas.cogentco.com [154.54.7.45]
11 56 ms 53 ms 55 ms te0-3-0-0.mpd22.mci01.atlas.cogentco.com [154.54.25.206]
12 42 ms 46 ms 42 ms te0-4-0-3.mpd22.ord01.atlas.cogentco.com [154.54.30.177]
13 41 ms 41 ms 41 ms te4-1.mpd01.ord05.atlas.cogentco.com [154.54.5.6]
14 41 ms 41 ms 41 ms vl3805.na01.b000268-0.ord05.atlas.cogentco.com [66.28.6.86]
15 41 ms 41 ms 41 ms 66.28.82.19

Trace complete.


Is there a good reason for things to work this way? Comcast tech support cannot answer. I want the faster route for any of our public IP Address.

Note: First hop IP address have been obfuscated here to protect the innocent.

 

[Quadratic]

It's odd but it's comcast's problem to correct. If I were them I'd start by zeroing in on the routing table of 68.86.232.33, and asking why what was assigned to a customer as a /29 subnet is getting broken up and routed differently.

I can only speculate on cause with this info, but routing between 68.86.232.33 and its two upstream links to your second office is a problem that they alone would have to solve. Perhaps you can point them in the right direction by asking them to look there specifically?

Also, which group in Comcast are you speaking with directly? Their NOC? Has this been escallated to their routing group? These are questions you might want to ask to get them thinking in the right direction.

CCNP, CCDP