[Xen-devel] IVTV and xen: silent intermittent failures reading from /dev/video0

[Dan Schmick <dschmick@xxxxxxxxxxxx>]

[Disclaimer:  I realize that I probably won't get this problem solved
as it's probably an unsual setup. I thought I'd ask, though.]

Problem:  'cat /dev/video0 > test.mpg'
(a simple IVTV test of my hardware mpeg2 encoder) doesn't work
properly under Xen. 
When this command is run, test.mpg should grow approximately 1MB/sec in
perpetuity, and does so when the machine is booted under a 'normal'
(i.e. non-Xen) kernel.  When I boot under Xen, test.mpg grows
properly for a while (say, 2 to 10 minutes) and then stops growing.  Typical final file sizes
are 50MB
to 500MB.  Note that there are *no* errors in dmesg, syslog, etc.  The
cat command doesn't exit, either.  It seems to be blocked on read. 
This same behavior is observed whether IVTV is in dom0 or domu.

More information:

- I can usually hasten the failure by doing something with the Xen
machine that causes lots of disk and/or network activity.  Failure-hastening
activity can occur in any Xen domain, not just the one with /dev/video0
or Domain 0.
- When 'cat /dev/video0' is running and IVTV is working, 'xm top' shows
3-4% cpu usage in the IVTV domu.  Inside the IVTV domu, however, I see
1% cpu usage or so.  As soon as the failure occurs, 'xm top' shows that
the IVTV domu cpu usage has dropped to 1%.

This failure occurs identically whether IVTV is in Domain 0 or
in an unprivileged domain.  No difference.

My setup:

- Hardware:  Nforce2 motherboard, 2GB ram,
1.8GHz Athlon XP, Hauppauge PVR-150
- Xen: 3.0.2 
- Kernel: 2.6.16-xen 
- ivtv: 0.6.6 

What I've tried so far:

1.  Verified that this is not a hardware problem (I've done stability testing,
prime95, memtest86, etc.  The system works perfectly and the IVTV device is rock-solid on a non-Xen Linux 2.6.16 kernel.)
2.  Moved the encoder (IVTV) to its own IRQ
3.  Set PCI latency timers to reasonable values for all devices (e.g.
128)
4.  Verified that there are *no* errors in dmesg, xm dmesg, syslogs on domu, syslogs on dom0, etc.
5.  Verified that there are no throughput problems with disks.  The
output of 'cat' is redirected to a dedicated disk on a dedicated
controller where I can get 30MB/sec writing speeds sustained.
6.  Verified that I'm using the proper firmware for my encoder card
7.  Verified that swiotlb=force is not the problem.  Using swiotlb=force is
required to get IVTV working under Xen kernels but not under 'normal' kernels.  However,
passing it under a 'normal' 2.6.16 kernel doesn't cause any problems --
the IVTV device works fine.
8.  Tuned the Xen scheduler to reduce latency for Domain 0 and the
domain with /dev/video0.  This helped a little, but not much.

Consequences of problem:

My recordings 'hiccup' every 2-10 minutes or so.  The hiccup is a jump
in the recording of probably just a few seconds.  I believe that mythtv
detects that reading from the device has failed and restarts the read. 
These interruptions are annoying but I don't have a machine to dedicate
to this purpose, so I'm stuck with them.  If there's anything that I
can do to help diagnose this problem please let me know.

Xen rocks, by the way.  Thanks very much for your work on this excellent platform, and for your time reading this post.


Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel