Weird hourly system time change? 1

[LGJ]

Hi all,

Please can someone help, can anyone explain why the following might suddenly be happening on our Solaris box?:

Thu Jan 6 11:59:52 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 11:59:54 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 10:59:56 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 11:00:01 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 11:00:09 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 11:03:41 GMT 2005
pin:/export/home/pin> date
Thu Jan 6 12:03:46 GMT 2005

As you can see almost on the hour the time goes back an hour and then resolves itself within a couple of minutes. This began on about Dec 27 and has never been a problem before.

This is happening every hour.

Thanks for any help.

LGJ

 

[bfitzmai]

Is ntp running on your workstation? You should be able to see if the daemon is running with command:

ps -ef | grep ntp

 

[LGJ]

thanks for the reply.

The output is shown below:

pin:/var/portal/6.5/cm> ps -ef | grep ntp
root 579 1 0 Jul 23 ? 0:01 /usr/lib/inet/xntpd

could it be this xntpd?

LGJ

 

[bfitzmai]

Yes... You are running the Network Time Protocol (NTP). Now the question was why your time was so far off or did something change on the NTP server? Is the time on your workstation wrong?

 

[LGJ]

thanks, looking into this now.

(The workstation time is OK).

We noticed the problem as we have log file rotations running from logadm. This runs from root crontab at 03:10 every morning. Unfortunately the rotations have been failing and we believe this is related, even though the system time appears to correct itself before 03:10.

This has got me scratching my head?

Thanks
LGJ

 

[daFranze]

I have seen something like that many times: it is a typical effect if NVRAM battery is low -> replace this soon and your time will be stable again... pressing thumbs

Best Regards, Franz
--
Solaris System Manager from Munich, Germany
I used to work for Sun Microsystems Support (EMEA) for 5 years

 

[spamly]

Is there a chance that you have a cron running ntpdate or rdate manually to a different server? While I can see the time getting messed up if a battery is low, I find it a little strange that it seems to be almost exactly one hour off.

Just my 2 cents. daFranze is usually right on the money.

 

[daFranze]

>> daFranze is usually right on the money.

funny phrase, what does it mean? I mean, I can translate it word by word, but.... ;-)

Best Regards, Franz
--
Solaris System Manager from Munich, Germany
I used to work for Sun Microsystems Support (EMEA) for 5 years

 

[LGJ]

thanks again for all the replies - daFranze and spamly.

We have the other teams involved in supporting the ntp servers looking into this. No luck so far unfortunately.

Thanks
LGJ

 

[spamly]

Right on the money" is slang that pretty much means that you're correct or accurate.

 

[LGJ]

We have the following in roots crontab: (could rtc be causing a problem?? - its not present in /usr/sbin/):

#ident "@(#)root 1.20 01/11/06 SMI"
#
# The root crontab should be used to perform accounting data collection.
#
# The rtc command is run to adjust the real time clock if and when
# daylight savings time changes.
#
10 3 * * * /usr/sbin/logadm
15 3 * * 0 /usr/lib/fs/nfs/nfsfind
1 2 * * * [ -x /usr/sbin/rtc ] && /usr/sbin/rtc -c > /dev/null 2>&1
30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] && /usr/lib/gss/gsscred_clean
#
# Weekly vxprint config dumps
#
0 5 * * 1 /usr/local/scripts/vxdump.sh > /tmp/vxdump.lis 2>&1

Thanks again
LGJ

 

[spamly]

I've never seen rtc cause an issue. It's cron'ed up by default.

Do you have an application that my be performing a date sync internally? I work with a medical program that has an rdate that it calls. This particular one is called by an application daemon that I can't modify.

Do you see any unusual activity in /var/adm/messages?

Do you have a trusted host relationship with any other servers that could be issues commands remotely?

 

[daFranze]

cron does not execute rtc if there is no executable in /usr/sbin

look at the messages, as spamly says, if there is nothing to see get a new battery and try again

@spamly: thanks!

Best Regards, Franz
--
Solaris System Manager from Munich, Germany
I used to work for Sun Microsystems Support (EMEA) for 5 years

 

[LGJ]

Please find below some messages from /var/adm/messages. Are these normal? I have done a quick bit of research but do not have the experience of you guys to tell if this could cause system time changes?

Once again thanks for your time on this, I only support the appication on this server and with some reorganisation it seems there is a lack of knowledge in the support teams for the servers!

Here are some of the last messages:

Jan 8 01:49:13 beezer in.mpathd[324]: [ID 822913 daemon.error] Improved failure detection time 10101 ms on (inet qfe6)
Jan 8 01:49:13 beezer in.mpathd[324]: [ID 822913 daemon.error] Improved failure detection time 10000 ms on (inet qfe5)
Jan 9 04:28:15 beezer in.mpathd[324]: [ID 398532 daemon.error] Cannot meet requested failure detection time of 10000 ms on (inet qf
e6) new failure detection time is 20832 ms
Jan 9 04:29:15 beezer in.mpathd[324]: [ID 822913 daemon.error] Improved failure detection time 10416 ms on (inet qfe1)
Jan 9 04:29:16 beezer in.mpathd[324]: [ID 822913 daemon.error] Improved failure detection time 10000 ms on (inet qfe6)


Best Regards
LGJ

 

[LGJ]

Hi,

Just an update, /etc/inet/ntp.conf has been updated to point to different NTP servers also the drift reference in ntp.conf has been uncommented and a ntp.drift file has been created so that if connection is lost to the NTP server then time can be kept through this file.

We are presently monitoring this to see if the issue is resolved?

LGJ

 

[LGJ]

Hi,

Just an update for anyone interested.

Having contacted SUN, this was known to be a known fault with this hardware. A script was provided and applied successfully.

The problem seems to take effect if the system controller has not been rebooted for 528 days

Thanks to all involved for the input on this thread

LGJ

 

[KenCunningham]

LGJ - thanks for posting back.

 

[spamly]

You mean you don't reboot your system controller every 527 days? And you call yourself an administrator?!

Congrats at finding the problem! Hardware issues are always the toughest to troubleshoot.

 

[LGJ]

Thanks guys, (lol @ spamlys comment )FYI :

There is a known issue on Sun Fire 3800/4800/4810/6800, V1280, and Netra 1280 Domains where the time jumps usually by one hour.

The problem is desbribed in Sun Alert 56680, which is available at:

http://uk.sunsolve.sun.com/search/printfriendly.do?assetkey=1-26-56680-1

Thanks
LGJ