[Xen-devel] [PATCH 0/6] add NUMA support to Xen

[Ryan Harper <ryanh@xxxxxxxxxx>]

The following patchset adds NUMA support to the hypervisor.  This
includes:

 - A full SRAT table parser for 32-bit and 64-bit, based on the ACPI
   NUMA parser from linux 2.6.16.29 with data structures to represent 
   NUMA cpu and memory topology, and NUMA emulation (fake=).
 - Changes to the Xen page allocator adding a per-node bucket for each
   zone.  Xen will continue to prioritize using the requested zone at
   the cost of using non-local memory.
 - Modified xen memory_ops to use NUMA-aware page allocator functions to
   help allocate node-local memory for guests
 - export NUMA topology through physinfo hypercall, integreated into
   Xend
 - fix-up nr_cpus calculation now that nr_nodes is not static
 - Add a tool to explore NUMA allocations and Xen heap status.

Nothing has functionally changed since the last posting of the patches;
I've just been maintaining the patchset against unstable.

Additionally, I've spent the past few weeks collecting some performance
benchmarks comparing with and without the NUMA patches on a system
configured as a 4-socket server(no NUMA characteristics) and as a
multi-node 16-socket NUMA machine.

Machine: x460 4-node, 3Ghz Xeon PIV, 1MB L2, 8MB L3
         16 Socket (4 sockets per node) HT, logical 32-way,
         64GB RAM, 16G per node
Xen Changeset used: 11536:041be3f6b38e

We ran sdet and dbench, recording aggregate throughput for 4, 8, 12, and
16domUs.  This loads the system with 4domUs on each node, one node, two
nodes, etc up to 4 nodes pinning the guest VCPUs to the proper physical
cpus in their respective nodes.  

To look for possible regressions on non-NUMA systems, we used a single
node (4 sockets and 16G) for the first 4domU test as a single node does
not have any NUMA characteristics.  

In summary, there are no regressions for the single node, non-numa case.
As we scale the number of domains out the NUMA patches show significant
gains.

dbench(tmpfs)  4domU  8domU  12domU  16domU
-------------------------------------------
with           590    1175   1750    2175
without        590    1090   1405    1450
-------------------------------------------
%gain         +0.0%   +7.8%  +24.5%  +50.0%

sdet       4domU  8domU  12domU  16domU
---------------------------------------
with       14000  25900  39500   49000
without    13900  24950  34900   38500
---------------------------------------
%gain      +0.8%  +3.8%  +13.2%  +27.0%

I've also been working with Raj at Unisys to test out the NUMA patches
on their ES7000.  Just today Raj confirmed that the previous "dom0
slowdown" that had been reported has been reproduced without the NUMA
patches.  I believe there are no outstanding machines that have tested
the NUMA patches that have any issues.

I would like to see the NUMA patches included in xen-unstable once it
is open for wider testing prior to the next release after 3.0.3.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@xxxxxxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel