[Xen-users] Problem with time in 64bit domUs

["Paul Reiber" <reiber@xxxxxxxxx>]

Greetings, Xen developers and power-users!

I have a client who's using Xen in their staging environment.  This client
is technically advanced, well beyond your typical sysadmin shop.  For example,
they're comfortable building Xen locally and packaging their own RPMs.

We've been noticing some issues with TIME within 64bit HVM-based domU's -
their time-of-day clock appears to get out-of-whack in short order.

For example, in our early testing, running "sleep 1; date" from the
commandline, repeatedly, exhibited the following behavior:
- the "real time" that passes each time is indeed approx. one second
- the "system clock" time that's reported can jump forward randomly
from between 1 through 6-7 seconds per execution.

Various changes (newest version of Xen, etc) have obfuscated the problem
but not eliminated it.  The above test no longer fails, but the following
test still shows there are problems.

The client used the shmux shell multiplexer to execute 'date' in parallel
across various domU's:

[root@hostname]# SHMUX_SSH_OPTS='-xa -o "StrictHostKeyChecking no"'
shmux -S all `grep -vh '#' /etc/hosts.stage-orion` -c 'date'
orion-xen01.stage2: Thu Dec 20 16:49:02 PST 2007
orion-xen03.stage2: Thu Dec 20 17:14:38 PST 2007
orion-xen02.stage2: Thu Dec 20 16:31:36 PST 2007
orion-xen08.stage2: Thu Dec 20 16:31:37 PST 2007
orion-xen05.stage2: Thu Dec 20 16:37:47 PST 2007
orion-xen04.stage2: Thu Dec 20 16:31:37 PST 2007
orion-xen06.stage2: Thu Dec 20 16:31:39 PST 2007
orion-xen10.stage2: Thu Dec 20 16:37:55 PST 2007
orion-xen07.stage2: Thu Dec 20 16:31:39 PST 2007
orion-xen09.stage2: Thu Dec 20 16:31:38 PST 2007
orion-xen12.stage2: Thu Dec 20 16:59:18 PST 2007
orion-xen11.stage2: Thu Dec 20 16:39:24 PST 2007
orion-xen14.stage2: Thu Dec 20 16:47:35 PST 2007
orion-xen13.stage2: Thu Dec 20 16:40:47 PST 2007

14 targets processed in 2 seconds.
Summary: 14 successes

You can see from the variations in timestamps above that the various domUs are
indeed quite far out of sync.  This isn't being off by a second or two; it's
quite beyond that.

While there are some work-arounds (like running ntpdate periodically,
for example)
it's disconcerting to both me and the client that this is occurring,
and we'd like
to help actually _fix_ the problem rather than just work around it.

I've learned there are three different processor schedulers within Xen.  Should
my client focus on testing whether changes at this level affect this problem?

Are there any other "obvious" things to check/test regarding this?

I believe this client could provide valuable help with problem investigation and
debugging. Please feel free to contact them with any initial questions you
might have regarding the specifics.  Brendan Wood - bwood@xxxxxxxxxxxxx - will
be happy to provide in-depth answers to any questions you might have.

Thanks so much for your time and attention to this issue!

Happy Holidays!
-Paul Reiber,
Linux specialist, Taos Mountain Consulting

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users