Re: [Xen-devel] Linux 4.15-rc6 + xen-unstable: BUG: unable to handle kernel NULL pointer dereference at (null), [ 0.000000] IP: zero_resv_unavail+0x8e/0xe1

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Since it's already rc7:
"Give me a subtle ping, Vasili. One subtle ping only, please."

On 04/01/18 21:02, Sander Eikelenboom wrote:
> On 04/01/18 12:44, Juergen Gross wrote:
>> On 04/01/18 11:17, Sander Eikelenboom wrote:
>>> Hi Boris / Juergen,
>>>
>>> First of all best wishes for a quite turbulent starting new year.
>>>
>>> Now the holidays are over I finally gotten to test a linux 4.15-rc6 kernel
>>> and experienced a crash in early dom0 boot on my system (AMD phenom x6).
>>>
>>> I tested some earlier linux 4.15 rc's but experienced crashes then as well, 
>>> but didn't have time to setup serial console to send them in 
>>> (and waited to see if the issue Boris fixed with AMD PCI 64bit bar's could 
>>> be it). 
>>>
>>> But since that patch went in before 4.15 rc6, that doesn't seem to be the 
>>> issue. 
>>> So it could be that the culprit went in pretty earlier in the 4.15 cycle.
>>>
>>> The 4.15-rc6 kernel boots fine on bare metal, as does a 4.14.6 kernel on 
>>> xen-unstable.
>>>
>>> Hopefully you have a pointer to what is wrong, if not i can try to do a 
>>> bisect.
>>
>> A bisect would be very welcome.
> 
> Hi Juergen / Boris / Pavel,
> 
> Bisection result is:
> 
> a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b is the first bad commit
> commit a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b
> Author: Pavel Tatashin <pasha.tatashin@xxxxxxxxxx>
> Date:   Wed Nov 15 17:36:31 2017 -0800
> 
>     mm: zero reserved and unavailable struct pages
>     
>     Some memory is reserved but unavailable: not present in memblock.memory
>     (because not backed by physical pages), but present in memblock.reserved.
>     Such memory has backing struct pages, but they are not initialized by
>     going through __init_single_page().
>     
>     In some cases these struct pages are accessed even if they do not
>     contain any data.  One example is page_to_pfn() might access page->flags
>     if this is where section information is stored (CONFIG_SPARSEMEM,
>     SECTION_IN_PAGE_FLAGS).
>     
>     One example of such memory: trim_low_memory_range() unconditionally
>     reserves from pfn 0, but e820__memblock_setup() might provide the
>     exiting memory from pfn 1 (i.e.  KVM).
>     
>     Since struct pages are zeroed in __init_single_page(), and not during
>     allocation time, we must zero such struct pages explicitly.
>     
>     The patch involves adding a new memblock iterator:
>             for_each_resv_unavail_range(i, p_start, p_end)
>     
>     Which iterates through reserved && !memory lists, and we zero struct pages
>     explicitly by calling mm_zero_struct_page().
>     
>     ===
>     
>     Here is more detailed example of problem that this patch is addressing:
>     
>     Run tested on qemu with the following arguments:
>     
>             -enable-kvm -cpu kvm64 -m 512 -smp 2
>     
>     This patch reports that there are 98 unavailable pages.
>     
>     They are: pfn 0 and pfns in range [159, 255].
>     
>     Note, trim_low_memory_range() reserves only pfns in range [0, 15], it does
>     not reserve [159, 255] ones.
>     
>     e820__memblock_setup() reports linux that the following physical ranges 
> are
>     available:
>         [1 , 158]
>     [256, 130783]
>     
>     Notice, that exactly unavailable pfns are missing!
>     
>     Now, lets check what we have in zone 0: [1, 131039]
>     
>     pfn 0, is not part of the zone, but pfns [1, 158], are.
>     
>     However, the bigger problem we have if we do not initialize these struct
>     pages is with memory hotplug.  Because, that path operates at 2M
>     boundaries (section_nr).  And checks if 2M range of pages is hot
>     removable.  It starts with first pfn from zone, rounds it down to 2M
>     boundary (sturct pages are allocated at 2M boundaries when vmemmap is
>     created), and checks if that section is hot removable.  In this case
>     start with pfn 1 and convert it down to pfn 0.  Later pfn is converted
>     to struct page, and some fields are checked.  Now, if we do not zero
>     struct pages, we get unpredictable results.
>     
>     In fact when CONFIG_VM_DEBUG is enabled, and we explicitly set all
>     vmemmap memory to ones, the following panic is observed with kernel test
>     without this patch applied:
>     
>       BUG: unable to handle kernel NULL pointer dereference at          (null)
>       IP: is_pageblock_removable_nolock+0x35/0x90
>       PGD 0 P4D 0
>       Oops: 0000 [#1] PREEMPT
>       ...
>       task: ffff88001f4e2900 task.stack: ffffc90000314000
>       RIP: 0010:is_pageblock_removable_nolock+0x35/0x90
>       Call Trace:
>        ? is_mem_section_removable+0x5a/0xd0
>        show_mem_removable+0x6b/0xa0
>        dev_attr_show+0x1b/0x50
>        sysfs_kf_seq_show+0xa1/0x100
>        kernfs_seq_show+0x22/0x30
>        seq_read+0x1ac/0x3a0
>        kernfs_fop_read+0x36/0x190
>        ? security_file_permission+0x90/0xb0
>        __vfs_read+0x16/0x30
>        vfs_read+0x81/0x130
>        SyS_read+0x44/0xa0
>        entry_SYSCALL_64_fastpath+0x1f/0xbd
>     
>     Link: 
> http://lkml.kernel.org/r/20171013173214.27300-7-pasha.tatashin@xxxxxxxxxx
>     Signed-off-by: Pavel Tatashin <pasha.tatashin@xxxxxxxxxx>
>     Reviewed-by: Steven Sistare <steven.sistare@xxxxxxxxxx>
>     Reviewed-by: Daniel Jordan <daniel.m.jordan@xxxxxxxxxx>
>     Reviewed-by: Bob Picco <bob.picco@xxxxxxxxxx>
>     Tested-by: Bob Picco <bob.picco@xxxxxxxxxx>
>     Acked-by: Michal Hocko <mhocko@xxxxxxxx>
>     Cc: Alexander Potapenko <glider@xxxxxxxxxx>
>     Cc: Andrey Ryabinin <aryabinin@xxxxxxxxxxxxx>
>     Cc: Ard Biesheuvel <ard.biesheuvel@xxxxxxxxxx>
>     Cc: Catalin Marinas <catalin.marinas@xxxxxxx>
>     Cc: Christian Borntraeger <borntraeger@xxxxxxxxxx>
>     Cc: David S. Miller <davem@xxxxxxxxxxxxx>
>     Cc: Dmitry Vyukov <dvyukov@xxxxxxxxxx>
>     Cc: Heiko Carstens <heiko.carstens@xxxxxxxxxx>
>     Cc: "H. Peter Anvin" <hpa@xxxxxxxxx>
>     Cc: Ingo Molnar <mingo@xxxxxxxxxx>
>     Cc: Mark Rutland <mark.rutland@xxxxxxx>
>     Cc: Matthew Wilcox <willy@xxxxxxxxxxxxx>
>     Cc: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx>
>     Cc: Michal Hocko <mhocko@xxxxxxxxxx>
>     Cc: Sam Ravnborg <sam@xxxxxxxxxxxx>
>     Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
>     Cc: Will Deacon <will.deacon@xxxxxxx>
>     Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>     Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> 
> :040000 040000 b0422cb4f5ef60f5bc7f0686d135c869680c603d 
> 51ef20afe641afceaf5530b83b4f1b9a51563939 M    include
> :040000 040000 55be7a5dd879578dc3f88bec059bcc392e3f1a1c 
> b4c9f81df05629bb034b6d0bdc0454579f2986fe M    mm
> 
> 
> --
> Sander
> 
>>
>> Juergen
>>
>>>
>>> --
>>> Sander
>>>
>>> Attached: .config and full serial log
>>>
>>>  0.000000] ACPI: Early table checksum verification disabled
>>> [    0.000000] ACPI: RSDP 0x00000000000FB100 000014 (v00 ACPIAM)
>>> [    0.000000] ACPI: RSDT 0x00000000C7F90000 000048 (v01 MSI    OEMSLIC  
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: FACP 0x00000000C7F90200 000084 (v01 7640MS A7640100 
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: DSDT 0x00000000C7F905E0 009427 (v01 A7640  A7640100 
>>> 00000100 INTL 20051117)
>>> [    0.000000] ACPI: FACS 0x00000000C7F9E000 000040
>>> [    0.000000] ACPI: APIC 0x00000000C7F90390 000088 (v01 7640MS A7640100 
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: MCFG 0x00000000C7F90420 00003C (v01 7640MS OEMMCFG  
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: SLIC 0x00000000C7F90460 000176 (v01 MSI    OEMSLIC  
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: OEMB 0x00000000C7F9E040 000072 (v01 7640MS A7640100 
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: SRAT 0x00000000C7F9A5E0 000108 (v03 AMD    FAM_F_10 
>>> 00000002 AMD  00000001)
>>> [    0.000000] ACPI: HPET 0x00000000C7F9A6F0 000038 (v01 7640MS OEMHPET  
>>> 20100913 MSFT 00000097)
>>> [    0.000000] ACPI: IVRS 0x00000000C7F9A730 000110 (v01 AMD    RD890S   
>>> 00202031 AMD  00000000)
>>> [    0.000000] ACPI: SSDT 0x00000000C7F9A840 000DA4 (v01 A M I  POWERNOW 
>>> 00000001 AMD  00000001)
>>> [    0.000000] ACPI: Local APIC address 0xfee00000
>>> [    0.000000] Setting APIC routing to Xen PV.
>>> [    0.000000] NUMA turned off
>>> [    0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffffff]
>>> [    0.000000] NODE_DATA(0) allocated [mem 0x7fc15000-0x7fc1efff]
>>> [    0.000000] tsc: Fast TSC calibration using PIT
>>> [    0.000000] Zone ranges:
>>> [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
>>> [    0.000000]   DMA32    [mem 0x0000000001000000-0x000000007fffffff]
>>> [    0.000000]   Normal   empty
>>> [    0.000000] Movable zone start for each node
>>> [    0.000000] Early memory node ranges
>>> [    0.000000]   node   0: [mem 0x0000000000001000-0x0000000000095fff]
>>> [    0.000000]   node   0: [mem 0x0000000000100000-0x000000007fffffff]
>>> [    0.000000] Initmem setup node 0 [mem 
>>> 0x0000000000001000-0x000000007fffffff]
>>> [    0.000000] On node 0 totalpages: 524181
>>> [    0.000000]   DMA zone: 64 pages used for memmap
>>> [    0.000000]   DMA zone: 21 pages reserved
>>> [    0.000000]   DMA zone: 3989 pages, LIFO batch:0
>>> [    0.000000]   DMA32 zone: 8128 pages used for memmap
>>> [    0.000000]   DMA32 zone: 520192 pages, LIFO batch:31
>>> [    0.000000] BUG: unable to handle kernel NULL pointer dereference at     
>>>       (null)
>>> [    0.000000] IP: zero_resv_unavail+0x8e/0xe1
>>> [    0.000000] PGD 0 P4D 0 
>>> [    0.000000] Oops: 0002 [#1] SMP
>>> [    0.000000] Modules linked in:
>>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 
>>> 4.15.0-rc6-20180104-linus-doflr+ #1
>>> [    0.000000] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS 
>>> V1.8B1 09/13/2010
>>> [    0.000000] RIP: e030:zero_resv_unavail+0x8e/0xe1
>>> [    0.000000] RSP: e02b:ffffffff82803d68 EFLAGS: 00010006
>>> [    0.000000] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 
>>> 0000000000000010
>>> [    0.000000] RDX: 000000000007ffff RSI: 0000000000000100 RDI: 
>>> ffffea0002000000
>>> [    0.000000] RBP: ffffffff82803d70 R08: ffffea0002000000 R09: 
>>> 0000000000000002
>>> [    0.000000] R10: 0000000000000002 R11: 0000000000000003 R12: 
>>> ffffea0000000000
>>> [    0.000000] R13: 0000000000000000 R14: ffffffff82803f20 R15: 
>>> 0000000000000000
>>> [    0.000000] FS:  0000000000000000(0000) GS:ffffffff82e16000(0000) 
>>> knlGS:0000000000000000
>>> [    0.000000] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [    0.000000] CR2: 0000000000000000 CR3: 0000000002823000 CR4: 
>>> 0000000000000660
>>> [    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
>>> 0000000000000000
>>> [    0.000000] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 
>>> 0000000000000000
>>> [    0.000000] Call Trace:
>>> [    0.000000]  ? free_area_init_nodes+0x690/0x69f
>>> [    0.000000]  ? zone_sizes_init+0x4b/0x50
>>> [    0.000000]  ? xen_pagetable_init+0x13/0x43f
>>> [    0.000000]  ? memblock_find_dma_reserve+0x141/0x15b
>>> [    0.000000]  ? memblock_find_dma_reserve+0x150/0x15b
>>> [    0.000000]  ? numa_init+0x43c/0x453
>>> [    0.000000]  ? setup_arch+0x7a0/0x87f
>>> [    0.000000]  ? start_kernel+0x58/0x3a8
>>> [    0.000000]  ? iommu_shutdown_noop+0x10/0x10
>>> [    0.000000]  ? xen_start_kernel+0x528/0x534
>>> [    0.000000] Code: da 49 c1 e0 06 4d 01 e0 48 8b 44 24 08 48 8d 0c 1a 48 
>>> 05 ff 0f 00 00 48 c1 e8 0c 48 39 c8 76 16 4c 89 c7 b9 10 00 00 00 44 89 e8 
>>> <f3> ab 48 ff c3 49 83 c0 40 eb d2 6a 00 55 31 d2 49 c7 c0 90 78 
>>> [    0.000000] RIP: zero_resv_unavail+0x8e/0xe1 RSP: ffffffff82803d68
>>> [    0.000000] CR2: 0000000000000000
>>> [    0.000000] ---[ end trace b788f32e38f6de39 ]---
>>> [    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>>> (XEN) [2018-01-04 09:52:49.218] Hardware Dom0 crashed: rebooting machine in 
>>> 5 seconds.
>>>
>>
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel