Re: [Xen-devel] Re: [Xen-users] Max. PV and HVM Guests

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Mon, Nov 09, 2009 at 08:06:54AM -0700, Nick Couchman wrote:
> 
> 
> >>> On 2009/11/09 at 05:05, Pasi Kärkkäinen<pasik@xxxxxx> wrote: 
> > On Mon, Nov 09, 2009 at 08:01:00PM +0800, Mr. Teo En Ming (Zhang
> Enming) 
> > wrote:
> >>    No, I didn't limit dom0 memory in grub.conf.
> >> 
> > 
> > You should. 
> 
> Really?  I thought current conventional wisdom was to allow Xen to
> self-manage memory in both dom0 and domUs, and not to manually adjust
> this?  I run several Xen systems with anywhere from 8 to 24 GB of RAM
> and 20 to 30 domUs on some of these systems and have *never* specified
> the dom0 memory at boot time - the Xen ballooning has always functioned
> perfectly fine, and never crashed my dom0.  
>

Yes, Xen is totally OK with this, but dom0 Linux has more problems..

> Furthermore, while I'm not
> Linux developer and so not familiar with how Linux calculates buffering
> and caching, I do know that my Linux systems dynamically manage buffers
> and caches, and when memory is reduced or some application requires a
> larger amount of physical memory, Linux reduces the amount of data in
> buffers and caches.
> 

Yeah, it has to do with sizing the network buffers, caches etc.. 

It shouldn't _crash_, so Teo is seeing some bug I believe. But it has
always been "best practice" to limit dom0 memory - and prevent weird
things happening later (like "memory squeeze in netback driver").

> Of course, a lot of this depends on what you're doing in dom0 - on my
> Xen servers, my dom0 is strictly for Xen management - I'm not running
> anything else in dom0 that would require large amounts of memory, memory
> buffers and caches, etc.
> 

Teo is running graphical stuff, X etc so it's a bit different..

-- Pasi

> 
> > 
> > If dom0 has all the memory at boot time, you need to balloon down
> dom0 
> > memory every time you create a new guest - this can (and will) cause
> 
> > problems with the dom0 linux kernel.
> > 
> > Linux calculates some internal parameters/buffers/values based on
> the
> > _boot time_ amount of memory. And when the amount of memory goes down
> to
> > only a small fraction of that while creating new guests bad things
> can
> > happen..
> > 
> > It still shouldn't crash though.. I bet your problem will get fixed
> when
> > you limit the dom0 memory to say dom0_mem=512M and reboot.
> > 
> > -- Pasi
> > 
> >>    Here's my xm info output after I have shutdown all the virtual
> machines.
> >> 
> >>    [root@fedora11-x86-64-host ~]# xm list
> >>    Name                                        ID   Mem VCPUs     
> State
> >>    Time(s)
> >>    Domain-0                                     0  2812     2    
> r-----
> >>    3242.5
> >>    [root@fedora11-x86-64-host ~]# xm info
> >>    host                   : fedora11-x86-64-host
> >>    release                : 2.6.30-rc3-enming.teo-tip
> >>    version                : #1 SMP Wed Aug 19 23:14:15 SGT 2009
> >>    machine                : x86_64
> >>    nr_cpus                : 2
> >>    nr_nodes               : 1
> >>    cores_per_socket       : 2
> >>    threads_per_core       : 1
> >>    cpu_mhz                : 2800
> >>    hw_caps                :
> >>   
> bfebfbff:20100800:00000000:00000140:0400e3bd:00000000:00000001:00000000
> >>    virt_caps              : hvm hvm_directio
> >>    total_memory           : 6039
> >>    free_memory            : 3124
> >>    node_to_cpu            : node0:0-1
> >>    node_to_memory         : node0:3124
> >>    xen_major              : 3
> >>    xen_minor              : 5
> >>    xen_extra              : -unstable
> >>    xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
> hvm-3.0-x86_32
> >>    hvm-3.0-x86_32p hvm-3.0-x86_64
> >>    xen_scheduler          : credit
> >>    xen_pagesize           : 4096
> >>    platform_params        : virt_start=0xffff800000000000
> >>    xen_changeset          : Tue Sep 01 11:34:31 2009 +0100 
> > 20143:a7de5bd776ca
> >>    xen_commandline        : iommu=1
> >>    cc_compiler            : gcc version 4.4.1 20090725 (Red Hat
> 4.4.1-2)
> >>    (GCC)
> >>    cc_compile_by          : root
> >>    cc_compile_domain      : (none)
> >>    cc_compile_date        : Thu Sep 10 07:01:13 SGT 2009
> >>    xend_config_format     : 4
> >> 
> >>    --
> >>    Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics)
> BEng(Hons)(Mechanical
> >>    Engineering)
> >>    Alma Maters:
> >>    (1) Singapore Polytechnic
> >>    (2) National University of Singapore
> >>    My Primary Blog:
> [1]http://teo-en-ming-aka-zhang-enming.blogspot.com
> >>    My Secondary Blog: [2]http://enmingteo.wordpress.com
> >>    My Youtube videos: [3]http://www.youtube.com/user/enmingteo
> >>    Email: [4]space.time.universe@xxxxxxxxx
> >>    Mobile Phone (Starhub Prepaid): +65-8369-2618
> >>    Street: Bedok Reservoir Road
> >>    Country: Singapore
> >> 
> >>    On Mon, Nov 9, 2009 at 7:54 PM, Pasi Kärkkäinen <[5]pasik@xxxxxx>
> wrote:
> >> 
> >>      On Mon, Nov 09, 2009 at 06:52:37PM +0800, Mr. Teo En Ming
> (Zhang 
> > Enming)
> >>      wrote:
> >>      >    Hi,
> >>      >
> >>      >    Please watch this 4-minute video at
> >>      >    [1][6]http://www.youtube.com/watch?v=LbLaPpwNAx4
> >>      >
> >>      >    I have only started 3 HVM Linux guests with 1 GB ram each.
> I can't
> >>      start
> >>      >    the 4th HVM guest. If I attempt to start the 4th instance,
> it will
> >>      crash
> >>      >    dom0.
> >>      >
> >>      >    Are there anything in the xm dmesg output that could
> explain the
> >>      low limit
> >>      >    to the number of VMs that I could start before dom0
> becomes
> >>      unresponsive?
> >>      >
> >> 
> >>      Have you limited dom0 memory (by specifying dom0_mem=XMB option
> in
> >>      grub.conf for xen.gz) ?
> >> 
> >>      What does "xm info" say about free memory before starting any
> guests?
> >>      -- Pasi
> >> 
> >> References
> >> 
> >>    Visible links
> >>    1. http://teo-en-ming-aka-zhang-enming.blogspot.com/
> >>    2. http://enmingteo.wordpress.com/
> >>    3. http://www.youtube.com/user/enmingteo
> >>    4. mailto:space.time.universe@xxxxxxxxx
> >>    5. mailto:pasik@xxxxxx
> >>    6. http://www.youtube.com/watch?v=LbLaPpwNAx4
> 
> 
> 
> --------
> This e-mail may contain confidential and privileged material for the sole use 
> of the intended recipient.  If this email is not intended for you, or you are 
> not responsible for the delivery of this message to the intended recipient, 
> please note that this message may contain SEAKR Engineering (SEAKR) 
> Privileged/Proprietary Information.  In such a case, you are strictly 
> prohibited from downloading, photocopying, distributing or otherwise using 
> this message, its contents or attachments in any way.  If you have received 
> this message in error, please notify us immediately by replying to this 
> e-mail and delete the message from your mailbox.  Information contained in 
> this message that does not relate to the business of SEAKR is neither 
> endorsed by nor attributable to SEAKR.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users