NTP won't sync 2

[plshlpme]

I have 3 2650XM routers that i cannot get to sync to a NTP server.

cisco of course wanted me to upgrade the IOS. and then said that it was due to the inconsistency in delay on the circuit.
i had my customer turn their router into a ntp master and i was still not able to sync over the lan.

other then a hardware problem does anybody have any ideas of what else the problem may be?

ios is c2600-is-mz.123-10d
and all three routers are configured exactly teh same as my other 65 routers that sync no problem.

the 3 routers have this for the ntp config

ntp source Loopback0
ntp server y.y.y.y
ntp server x.x.x.x

ntp server y.y.y.y:
ntp source Loopback0
ntp master 3
ntp peer x.x.x.x

ntp server x.x.x.x:
ntp source Loopback0
ntp master 3
ntp peer y.y.y.y

this has been bugging me for a while and im at the point where i think im going to change out the routers for new ones unless anybody has come across this before?

thanks

 

[tecnikall]

Hello there, to be able to have ntp running fine we will need to enable the local clock on those routers before enabling ntp command. Just check that and let me how it goes. At this stage I don't belive that it is a hardware failure with those routers -- at most it will be software defect... Let me know how it goes and we can run few debugs at a later stage. Cheers and good luck

 

[plshlpme]

what steps are required to set enable the local clock?
set the timezone?

 

[plshlpme]

the one thing that is missing from the config on the routers that won't sync is the
ntp clock-period

does the router have to sync before it can derive its clock period?

 

[tecnikall]

Hello there, the command is in global config mode (config t): clock

For the client router to be able to update it's clock from ntp master/server has to have it's own clock up and running before being able to syn it to the ntp server.

More info about clock and ntp clock-period at:

http://www.cisco.com/univercd/cc/td...2/122cgcr/ffun_c/fcfprt3/fcf012.htm#wp1001926

Cheers and good luck!!!

 

[plshlpme]

thanks tecnikall,
ive tried all of those commands all ready. one thing though that i tried many times was to set the local time zone on the router and it wasn't taking.
also the clock calendar-valid
isn't available in the ios/platform ?

in all the other routers i never had to enter any clock commands to have them sync to the same source though.
we are using the ip plus ios and there are some 2650XM's and some 2651XM's.

when i set them as a ntp master they cync to themselves but don't update properly from the peers.

 

[vipergg]

Have you tried debugging ntp ? Might give you some insight into what is happening and what the router is outputting and coming back from the server .

 

[tecnikall]

Hello there, It looks like we should have a fresh start. There are few questions:

1. I am assuming that we have the following setup ;


Public atomic clock----------------| NTP Server router 1 |---------| NTP-Client 2600xm |

|
|
| NTP-Client 2600xm |

Where NTP server router has access to the Internet and the rest of the router are inside your customer's network and we can ping all the routers in the network both ways.

2. On the NTP Server router 1 we will need the following commands:

clock timezone CST -6 (depending where the router is located)
ntp server x.x.x.x (pointing to the public atomic clock
ntp master
end

Before moving any further use the command: show ntp status . Look for the clock to be synchronised, it should have a reference clock of x.x.x.x
next command:
show ntp associations------When ntp is synchronised an * reveals that the router recived UDP packets from x.x.x.x

In is important to allow UDP packets on the ntp Server router.

3. On the rest of the routers:
clock timezone CST -6
ntp peer a.b.c.d (ntp server's interface)

The ntp clock-period is automatically added into the router configs when NTP is enabled. It helps to start the NTP frequency compensation on router reloads.


Now, if still not working run on all routers :
debug ntp select
show ntp status
show ntp assoc
Save all of them to a text file and we'll take it from there. I am sure that we should be able to fix this issue... Cheers and good luck!!!

 

[plshlpme]

ok here is some debug and i do have it set up as you show.
and just to make sure everybody is aware. we have 65 global routers syncing off of the two main ones at the hub site. this is via a mpls ipvpn network.
of the 65 sites 63 sync fine 2 do not.

client#sho debug
NTP:
NTP clock adjustments debugging is on
NTP events debugging is on
NTP clock synchronization debugging is on
NTP clock selection debugging is on
NTP peer validity debugging is on

client#
.May 29 19:17:28 UTC: NTP: packet from x.x.x.x failed validity tests 80
.May 29 19:17:28 UTC: Root delay/dispersion failed bounda
ry check
.May 29 19:17:32 UTC: NTP: packet from x.x.x.x failed validity tests 80
.May 29 19:17:32 UTC: Root delay/dispersion failed bounda
ry check

client#sho ntp stat
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 249.5901 Hz, actual freq is 249.5901 Hz, precision is 2**16
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.00 msec, peer dispersion is 0.00 msec
client#sho ntp ass

address ref clock st when poll reach delay offset disp
~server1 0.0.0.0 16 58 64 0 0.0 0.00 16000.
~server2 0.0.0.0 16 54 64 0 0.0 0.00 16000.
* master (synced), # master (unsynced), + selected, - candidate, ~ configured
client#

ntp source Loopback0
ntp peer server1
ntp peer server2


here is the server config:
server1#sho ntp ass

address ref clock st when poll reach delay offset disp
+~127.127.7.1 127.127.7.1 3 32 64 377 0.0 0.00 0.0
*~y.y.y.y 132.246.168.164 3 109 1024 373 1.3 270852 6.5
* master (synced), # master (unsynced), + selected, - candidate, ~ configured
server1#sho ntp sta
Clock is synchronized, stratum 4, reference is y.y.y.y
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**18
reference time is C6449D20.11E22ACB (19:57:20.069 UTC Sun May 29 2005)
clock offset is 0.0000 msec, root delay is 7.34 msec
root dispersion is 270886.22 msec, peer dispersion is 6.50 msec
server1#

ntp master 4
ntp update-calendar
ntp peer y.y.y.y

 

[plshlpme]

also to add to the above... i have added the clock time-zone command but since we use UTC as the time... it does not take.

 

[tecnikall]

Hello there,

Some information in regards to this problem:

Validity (sanity) tests are specified in the RFC1305 to test the reply packet received.
Eight of them are defined and when you do a “debug ntp validity” each of those failed test
is being represent by a bit. Test1 to 3 check for the validity of the data portion of the
packet and Test 5 to 8 check for the header information.

Test # Mask Meaning
1 0x01 Duplicate packet received
2 0x02 Bogus packet received
3 0x04 Protocol unsynchronized
4 0x08 Peer delay/dispersion failed boundary check
5 0x10 Peer authentication failed
6 0x20 Peer clock unsynchronized (common for unsynched server)
7 0x40 Peer stratum out of bound
8 0x80 Root delay/dispersion failed boundary check

Packet data is valid if test 1 to 4 are passed. Then the data will be used to calculate
offset, delay, and the dispersion.

Test 8 is failing and this means the following
Root Dispersion:
"Maximum error relative to the primary reference source at the root of the NTP "subnet"
Root Delay:
"Total roundtrip delay to the primary reference source at the root of the NTP subnet".
It looks like the packets are taking too long to get from the server to the client.


At this stage I will check for any delays on a round trip from the client to the server. Also, you may try a very simple config and

on Server
ntp server 198.82.1.203

also for redundancy add 1 more public time

ntp server 192.5.41.209 (Naval Observatory)

do a show ntp stat, when it becomes synchronized add the following on the client

client
ntp server x.x.x.x

If all fails I would try to move the server closer to the client for further testing. Maybe those 2 routers are on the edge of your network. Cheers and good luck

 

[plshlpme]

ok here is another update:
i manage this router, but my customer has a router which is back to back against mine. they are now using the same sync source and they sync ok. i had them set their router up as a master and i can not sync to them across the fast ethernet.
some mroe debug

DFAIT-CE-BUC-1#
.May 30 11:58:38 gmt: NTP: 10.112.0.42 reachable
.May 30 11:58:38 gmt: NTP: nlist 1, allow 0, found 0, low -28.566483, high 3.676559
.May 30 11:58:38 gmt: NTP: candidate 10.112.0.42 cdist 80.121521 error 16.023911
.May 30 11:58:38 gmt: NTP: survivor 10.112.0.42 offset -12.444956, cdist 80.12152
.May 30 11:58:38 gmt: NTP: synced to new peer 10.112.0.42
.May 30 11:58:38 gmt: NTP: step(-12.444956369): local_offset = 0.000000000, curtime = -968524177.056450950
.May 30 11:58:26 gmt: NTP: nlist 0, allow 0, found 0, low 0.000000, high 0.000000
.May 30 11:58:26 gmt: NTP: no select intersection
.May 30 11:58:26 gmt: NTP: peer stratum change
.May 30 11:58:26 gmt: NTP: no select intersection
.May 30 11:58:26 gmt: NTP: clock reset
CLIENT#
CLIENT#sho ntp ass

address ref clock st when poll reach delay offset disp
~10.112.0.42 160.106.86.3 4 54 64 0 3.3 -12444 16000.
* master (synced), # master (unsynced), + selected, - candidate, ~ configured
CLIENT#
.May 30 11:59:30 gmt: NTP: 10.112.0.42 reachable
.May 30 11:59:30 gmt: NTP: nlist 1, allow 0, found 0, low -28.566498, high 3.676514
.May 30 11:59:30 gmt: NTP: candidate 10.112.0.42 cdist 80.121506 error 16.023911
.May 30 11:59:30 gmt: NTP: survivor 10.112.0.42 offset -12.444983, cdist 80.12151
.May 30 11:59:30 gmt: NTP: synced to new peer 10.112.0.42
.May 30 11:59:30 gmt: NTP: step(-12.444983390): local_offset = 0.000000000, curtime = -968524125.500492055
.May 30 11:59:18 gmt: NTP: nlist 0, allow 0, found 0, low 0.000000, high 0.000000
.May 30 11:59:18 gmt: NTP: no select intersection
.May 30 11:59:18 gmt: NTP: peer stratum change
.May 30 11:59:18 gmt: NTP: no select intersection
.May 30 11:59:18 gmt: NTP: clock reset

 

[tecnikall]

Hello there, the day just strated here... Ok, there are few issues we can look into it:



1. Run a debug ntp packet on client and look for the following:

leap 3 on the client, the value should be leap 0

leap 3: alarm condition (clock unsynchronized).

2. For the client to be able to sync the master must sync himself first. Check this one first before checking the client

3. In the configs if you have more than one ntp server commands put in ntp server x.x.x.x prefer this way ntp will try to sync with one server and avoid the errors you posted.

4." no select intersection " means that all the filters are reset and start all over. The incoming packet will be tested for it validity. Then this new sample will be inserted into the filter (actually it is some sort of array storing the history about this particular server). This is the Clock Filtering Procedure. The purpose of the Clock Filtering
Procedure is to select the best offset samples from a given clock (a particular NTP server). In the past I have seen this issue solved by just adding "prefer" to the server command.

5. If all fails I would check the IOS on the 26xx to see if idendical with the rest. Maybe it is a software defect, if not identical I would upgrade/downgrade the code to see if same error. If same issue I would use a different router or replace the router with the router platform.

Cheers and good luck

 

[plshlpme]

thanks for all the help tecnikall,
i did the debug and was getting leap 3 errors...
also we had upgraded the ios on these routers just recently.
we had a 12.2 T train on them and just went to 12.3-10d. this problem existed before the ios upgrade.

i think weve done enough testing at this point to change out the router... its hard to do anything more since one router is in eaurope, one is in canada, one is in asia etc...

ill post again next week once the router has been swapped.

 

[tecnikall]

Hello there, not a problem. Looking forward to your next post. I hope that the replacement router will fix the issue. Cheers and all the best!!!

 

[plshlpme]

well the rotuer has been swapped... same problem still exists. we had a conference call with cisco our international provider and our tech support in country and nobody has any idea why it will not sync.

ill draw a picture so you can be as confused as me.


NTP Server --- (international mpls cloud) --- Edge router --- Customer router

so our edge router cannot sync its clock.
the customer router can sync to the same ntp source that we are trying to sync to.

so the cutomer can sync through our router, through the network but we cannot.

also when the customer router is set to master, we cannot sync to it through the fast ethernet..

i have stripped the configs down to bare bones just keeping conectivity up and it has no effect.

the only thing left for me to verify is the power source.. is it possible that bad power can make the clock act up?

the one way i had it sync was to make it think it was a master and then peer to another router... this appeared good. when i did sho ntp ass and sho ntp stat it all looked good. but the clock would drift very badly. within a minute i could be out by 20 seconds.

we have another router replacement lined up for this week because the provider didn't have the right model and temp swapped it for an older model.
the new one has arrived from cisco so we will swap it.

 

[tecnikall]

Hello there, what a shame to see the replacement router fail.

Did you try to use a different router platform? Maybe there is a software defect which applies to your existing platform. Did you try to use a different IOS? Also, can you post the errors you are getting now? Where abouts are you located? Cheers and good luck.

 

[plshlpme]

well we had a 2650XM at this site... and they brought a 2621XM.
they had some 12.2 ios that didn't have the features in it to get our frame relay link up. the frame relay fragment command wasn't in the ios.
so i upgraded it to the 12.3-10D IP Plus image that we are using on the other 50+ routers in the network.

the issue is still the same. the clock won't sync to any source. even over the fast ethernet to the customer via a crossover cable. the customer router has no problem syncing to the master clock at their headquarters.
the funny thing too is that our router reports a longer delay then theirs when youuse sho ntp stat.



Router#sho clo
.12:00:43.863 UTC Sat Jun 25 2005
Router#sho ntp ass

address ref clock st when poll reach delay offset disp
~A.A.A.A X.X.X.X 4 27 64 0 238.5 -12444 16000.
* master (synced), # master (unsynced), + selected, - candidate, ~ configured
Router#sho ntp stat
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 249.5901 Hz, actual freq is 249.5901 Hz, precision is 2**18
reference time is C667C5E0.A083CA6D (12:00:32.627 UTC Sat Jun 25 2005)
clock offset is -12444.1054 msec, root delay is 246.00 msec
root dispersion is 28350.56 msec, peer dispersion is 16000.00 msec
Router#sho ntp ass de
A.A.A.A configured, insane, invalid, stratum 4
ref ID X.X.X.X, time C667C4E4.546A0B3F (11:56:20.329 UTC Sat Jun 25 2005)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 7.52 msec, root disp 31.43, reach 0, sync dist 16029.449
delay 238.48 msec, offset -12444.1054 msec, dispersion 16000.00
precision 2**18, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

one thign that is common and i assume that these are maximum numbers are the delay and dispersion are the same if i sync over the lan or the wan.

cisco has had all the debug for over a week now apparently they have given it to their engineers to look at...
hopefully we hear something soon on it.

 

[Cotton9]

Is it likely the MPLS provider is corrupting the NTP UDP packets or random small packets?

Our provider was doing ATM-LAN emulation on our Lan-Switches-Service 100M link and was clobbering about half of our 64byte packtes. Our 10M link was actually faster at the time. This caused us a lot of problems and time proving it to them. They finally fixed it after a month --- replaced their bridges at both ends.

Just an old idea.

 

[plshlpme]

well the fact that the customer router can sync the clock has to put down that theory... as they are syncing over the same link.
also our router can't link over the fast ethernet which is purely a crossover cable between two back to back routers.

we use this provider for about 50 international sites.. soon to be 100 and we only have 2 sites where this has occured.