[Xen-devel] Infiniband from userland in dom0, process killed, bad pagetable

[Vivien Bernet-Rollande <vbernetr@xxxxxxxxx>]

Hello all.

I'm new to Xen or kernel development.

I'm trying to fix the following issue :
 - I'm in dom0, I don't care about having IB in domU.
 - Hardware is mellanox, driver is ib_mthca + libmthca
 - IB from the kernel seems to work fine.
 - However, when a process tries to access the hardware from userland (through libmthca) it is killed by the kernel. The following message is sent to the console :
Message from syslogd@x-dev-4 at Nov 10 14:00:42 ...
 kernel:Bad pagetable: 000f [#1] SMP 

Message from syslogd@x-dev-4 at Nov 10 14:00:42 ...
 kernel:last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mthca0/node_guid


Dmesg shows :
client: Corrupted page table at address 7ff332e74020
PGD 21abc067 PUD 4a9b4067 PMD 4c018067 PTE fffffffffffff237
Bad pagetable: 000f [#3] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mthca0/node_guid
CPU 0 
Modules linked in: bridge stp llc sunrpc rdma_ucm ib_sdp rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_umad iw_nes libcrc32c iw_cxgb3 cxgb3 mlx4_ib mlx4_en mlx4_core xen_netback xen_blkback blkback_pagemap xen_gntdev xen_evtchn xenfs ib_mthca snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device ib_mad snd_pcm tg3 hp_wmi ppdev snd_timer rfkill k8temp ib_core snd edac_core parport_pc soundcore i2c_piix4 snd_page_alloc wmi edac_mce_amd shpchp parport serio_raw xfs exportfs pata_acpi ata_generic dm_multipath pata_atiixp radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
Pid: 3869, comm: client Tainted: G      D W  2.6.32.21-167.xendom0.fc12.x86_64 #1 HP Compaq dc5750 Microtower
RIP: e033:[<00007ff332653d3f>]  [<00007ff332653d3f>] 0x7ff332653d3f
RSP: e02b:00007fff029299a8  EFLAGS: 00010206
RAX: 0000000086000212 RBX: 0000000000401568 RCX: 0000000000000001
RDX: 00007ff332e74000 RSI: 0000000000000000 RDI: 0000000001d10b70
RBP: 00007fff029299c0 R08: 0000000000000000 R09: 0000000000000008
R10: 0000000000000001 R11: 00000030b1eecea0 R12: 0000000000400a90
R13: 00007fff02929cc0 R14: 0000000000000000 R15: 0000000000000000
FS:  00007ff332e6a700(0000) GS:ffff880003ea2000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff332e74020 CR3: 0000000011a08000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Process client (pid: 3869, threadinfo ffff880014d64000, task ffff88005a531760)

RIP  [<00007ff332653d3f>] 0x7ff332653d3f
 RSP <00007fff029299a8>
---[ end trace a7919e7f17c0a729 ]---



xm dmesg :

(XEN) d0:v0: reserved bit in page table (ec=000F)
(XEN) Pagetable walk from 00007f9478d3c020:
(XEN)  L4[0x0ff] = 00000000690c8067 000000000000d550
(XEN)  L3[0x051] = 00000000496a8067 000000000002cf30
(XEN)  L2[0x1c6] = 000000004989c067 000000000002cd24 
(XEN)  L1[0x13c] = fffff7fffffff237 ffffffffffffffff
(XEN) ----[ Xen-4.0.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<00007f947851bd3f>]
(XEN) RFLAGS: 0000000000010206   EM: 0   CONTEXT: pv guest
(XEN) rax: 0000000086000312   rbx: 0000000000401568   rcx: 0000000000000001
(XEN) rdx: 00007f9478d3c000   rsi: 0000000000000000   rdi: 0000000002429b70
(XEN) rbp: 00007fffbb95c3e0   rsp: 00007fffbb95c3c8   r8:  0000000000000000
(XEN) r9:  0000000000000008   r10: 0000000000000001   r11: 00000030b1eecea0
(XEN) r12: 0000000000400a90   r13: 00007fffbb95c6e0   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 0000000043848000   cr2: 00007f9478d3c020
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=00007fffbb95c3c8:
(XEN)    0000000000400bea 00000000bb95c6e0 0000000002429b70 00007fffbb95c600
(XEN)    0000000000400f74 00007fffbb95c6e8 0000000203d8f538 0000000200000000
(XEN)    0000000000000001 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000002429ad0 00000030b1a00784 00000030b1a00380
(XEN)    00000030b1c1fa00 00007fffbb95c5b0 0000000000000000 00000030b1c1f0e8
(XEN)    0000000000000000 00007f9478d33078 00007f9478d3db30 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 00000001000007f0
(XEN)    00000030b1c1f678 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000001
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00007fffbb95c5d0 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000004005ba 000000000242d1b0
(XEN)    0000000002430120 00007fffbb95c700 00007fffbb95c700 0000000000000004
(XEN)    00000000024303c0 0000000002429980 00000000024299d0 0000000002429b70
(XEN)    0000000000401455 0000000002429ad0 0000000000401410 0000000000000000
(XEN)    0000000000400a90 00007fffbb95c6e0 0000000000000000 0000000000000000
(XEN)    00000030b1e1eb1d 0000000000000000 00007fffbb95c6e8 0000000200000000
(XEN)    0000000000400c68 0000000000000000 36674056a3d08701 0000000000400a90


As the logs say, there's a 1 in a reserved bit in a page table entry, to which a write access is performed in user mode ("ec=000F").

Now, this error looks a lot like a few of those : http://wiki.xensource.com/xenwiki/XenPVOPSDRM so I expect the problem is similar : physical addresses used are not the true physical addresses, and bad things happen.

BTW, I am aware of the xen-smartio repository, and will use it as an example. However, I will have to identify the changes it made and port from 2.6.18 xenlinux to something more recent.


My main question is the following : how is it possible to end up with an invalid page table ? I would expect that such a bug would trigger a segmentation violation. But since I suppose all page descriptors are initialized by the kernel or Xen, how does a 1 end up in a reserved field ? Or did I miss something ?



-- Vivien Bernet-Rollande

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel