Re: [Xen-devel] Compile pvops kernel xen/stable-2.6.32.x error

[Teck Choon Giam <giamteckchoon@xxxxxxxxx>]

Hi Kevin,

Thanks for sharing... ...

for me, the error messages are effectively toggled if one either
defines (see errors) or does not define (no errors) the kernel
config value:

CONFIG_DEBUG_FORCE_WEAK_PER_CPU 

What do you mean either define or not define?  Do you mean when the above option is enabled or set to y the compilation will fail and errors are what I posted previously in that thread?  If that is so, I will have doubt... ... see below... ...

# cd /boot
# grep CONFIG_DEBUG_FORCE_WEAK_PER_CPU ./config-2.6.32*
./config-2.6.32.25:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
./config-2.6.32.25-xenpvops.choon.centos5:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
./config-2.6.32.25-xenpvops.choon.centos6:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
./config-2.6.32.25-xenU.pvops.centos6.0:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
./config-2.6.32.25-xenU.pvops.choon.centos6.0:CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y
./config-2.6.32-44.2.el6.x86_64:CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y

config-2.6.32-44.2.el6.x86_64 is stock RHEL6 kernel which I used to build xen pvops using the xen.git tree and got those errors as described in that thread.  However, my test compile using config-2.6.32.25-xenU.pvops.choon.centos6.0 is successful and as you can see is with the CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y.  However it can't be booted as I will get kernel panic in xenU/guest.  So all my future pvops kernel configuration will not have that option set since I disable majority of the debug options.

Xen Minimal OS!
  start_info: 0xb9c000(VA)
    nr_pages: 0x40000
  shared_inf: 0xde3a2000(MA)
     pt_base: 0xb9f000(VA)
nr_pt_frames: 0xb
    mfn_list: 0x99c000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: (hd0)/boot/grub/grub.conf
  stack:      0x95b840-0x97b840
MM: Init
      _text: 0x0(VA)
     _etext: 0x69c3d(VA)
   _erodata: 0x84000(VA)
     _edata: 0x8cae0(VA)
stack start: 0x95b840(VA)
       _end: 0x99be48(VA)
  start_pfn: bad
    max_pfn: 40000
Mapping memory range 0x1000000 - 0x40000000
setting 0x0-0x84000 readonly
skipped 0x1000
MM: Initialise page allocator for da5000(da5000)-40000000(40000000)
MM: done
Demand map pfns at 40001000-2040001000.
Heap resides at 2040002000-4040002000.
Initialising timer interface
  Booting 'Red Hat Enterprise Linux Server (2.6.32.25-xenU.pvops.choon.centos6.
0)'

root (hd0)
Error ENOENT when reading the backend path device/vkbd/0/backend
Thread "kbdfront" exited.
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32.25-xenU.pvops.choon.centos6.0 ro root=/dev/xvda1 pa
nic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8
initrd /boot/initramfs-2.6.32.25-xenU.pvops.choon.centos6.0.img

close blk: backend=/local/domain/0/backend/vbd/12/51713 node=device/vbd/51713
close blk: backend=/local/domain/0/backend/vbd/12/51714 node=device/vbd/51714
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32.25-xenU.pvops.choon.centos6.0 (root@xxxxxxxxxxxxxxxxxx) (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC) ) #1 SMP Tue Nov 9 05:01:24 SGT 2010
Command line: ro root=/dev/xvda1 panic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000040000000 (usable)
DMI not present or invalid.
last_pfn = 0x40000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000040000000
RAMDISK: 01c7e000 - 14405000
No NUMA configuration found
Faking a node at 0000000000000000-0000000040000000
Bootmem setup node 0 0000000000000000-0000000040000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [000000000003c000 -  0000000000043fff] pages 8
(7 early reservations) ==> bootmem [0000000000 - 0040000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0014608000 - 00146af000]   XEN PAGETABLES ==> [0014608000 - 00146af000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001c5d418]    TEXT DATA BSS ==> [0001000000 - 0001c5d418]
  #4 [0001c7e000 - 0014405000]          RAMDISK ==> [0001c7e000 - 0014405000]
  #5 [0014405000 - 0014608000]   XEN START INFO ==> [0014405000 - 0014608000]
  #6 [0000100000 - 0000258000]          PGTABLE ==> [0000100000 - 0000258000]
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x000000a0
    0: 0x00000100 -> 0x00040000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 4 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
Booting paravirtualized kernel on Xen
Xen version: 4.0.2-rc1-pre (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff8800146e5000 s92824 r8192 d21864 u122880
pcpu-alloc: s92824 r8192 d21864 u122880 alloc=30*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 258117
Policy zone: DMA32
Kernel command line: ro root=/dev/xvda1 panic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8
PID hash table entries: 4096 (order: 3, 32768 bytes)
Initializing CPU#0
Checking aperture...
No AGP bridge found
Memory: 713456k/1048576k available (4813k kernel code, 384k absent, 334736k reserved, 3867k data, 1136k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:304
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 10485760 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2133.446 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 4266.89 BogoMIPS (lpj=2133446)
Security Framework initialized
SELinux:  Disabled at boot.
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
CPU 0/0x1 -> Node 0
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only.
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20529 entries in 81 pages
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324
PGD 0
Oops: 0002 [#1] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32.25-xenU.pvops.choon.centos6.0 #1
RIP: e030:[<ffffffff814a29d4>]  [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324
RSP: e02b:ffff88003ea73e80  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8800146e5000 RSI: ffffffff818693c0 RDI: 0000000000000000
RBP: ffff88003ea73ee0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800146faf80
R13: 0000000000000000 R14: 0000000000015f80 R15: 00000020404046e0
FS:  0000000000000000(0000) GS:ffff8800146e5000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001001000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88003ea72000, task ffff88003ea71460)
Stack:
 ffffffff8100f8cf ffffffff814ac096 ffff88003ea73f10 ffffffff81063fc6
<0> ffff88003ea71460 ffff88003ea70a50 0000000000000000 ffffffff81987ee0
<0> 0000000000001000 0000000000000000 ffffffffffffffff 00000020404046e0
Call Trace:
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ac096>] ? _spin_unlock_irqrestore+0x16/0x20
 [<ffffffff81063fc6>] ? set_cpus_allowed_ptr+0xb6/0x1d0
 [<ffffffff8189800a>] xen_smp_prepare_cpus+0x36/0x135
 [<ffffffff81893692>] kernel_init+0xc0/0x204
 [<ffffffff8101414a>] child_rip+0xa/0x20
 [<ffffffff81013311>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013a9d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81014140>] ? child_rip+0x0/0x20
Code: c0 01 48 63 d0 e8 0d cb d8 ff 3b 05 07 68 3c 00 0f 8c 54 ff ff ff eb 17 48 8b 14 d5 80 96 86 81 48 c7 c0 b8 e3 00 00 48 8b 04 10 <3e> 0f ab 38 49 8b 84 24 a0 00 00 00 3e 0f ab 18 48 c7 c0 80 5f
RIP  [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324
 RSP <ffff88003ea73e80>
CR2: 0000000000000000
---[ end trace a7919e7f17c0a725 ]---
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: swapper Tainted: G      D    2.6.32.25-xenU.pvops.choon.centos6.0 #1
Call Trace:
 [<ffffffff814a91e0>] panic+0x78/0x137
 [<ffffffff8106a0d4>] ? put_files_struct+0xc4/0xf0
 [<ffffffff81074c02>] ? exit_ptrace+0xb2/0x140
 [<ffffffff8106c66b>] do_exit+0x6bb/0x7e0
 [<ffffffff814ad1ab>] oops_end+0xab/0xf0
 [<ffffffff81041bf2>] no_context+0xf2/0x260
 [<ffffffff81041e85>] __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff81041f53>] bad_area_nosemaphore+0x13/0x20
 [<ffffffff814aec41>] do_page_fault+0x2d1/0x380
 [<ffffffff814ac535>] page_fault+0x25/0x30
 [<ffffffff814a29d4>] ? set_cpu_sibling_map+0x12e/0x324
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ac096>] ? _spin_unlock_irqrestore+0x16/0x20
 [<ffffffff81063fc6>] ? set_cpus_allowed_ptr+0xb6/0x1d0
 [<ffffffff8189800a>] xen_smp_prepare_cpus+0x36/0x135
 [<ffffffff81893692>] kernel_init+0xc0/0x204
 [<ffffffff8101414a>] child_rip+0xa/0x20
 [<ffffffff81013311>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013a9d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81014140>] ? child_rip+0x0/0x20
Rebooting in 5 seconds..

Since after that compile and test boot failure, I optimized even more for the configuration... ...

I have seen Jeremy posted related to this (especially what I submitted to him) to LKML but can't remember the details.

Once again, many thanks for sharing.  Now at least I know what is mostly causing that kind of errors and make note to avoid having that option set.

Kindest regards,
Giam Teck Choon
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel