|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy
On Thu, Aug 27, 2015 at 11:24 AM, George Dunlap <george.dunlap@xxxxxxxxxx> wrote: > On 08/18/2015 04:55 PM, Dario Faggioli wrote: >> Hey everyone, >> >> So, as a followup of what we were discussing in this thread: >> >> [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest >> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03241.html >> >> I started looking in more details at scheduling domains in the Linux >> kernel. Now, that thread was about CPUID and vNUMA, and their weird way >> of interacting, while this thing I'm proposing here is completely >> independent from them both. >> >> In fact, no matter whether vNUMA is supported and enabled, and no matter >> whether CPUID is reporting accurate, random, meaningful or completely >> misleading information, I think that we should do something about how >> scheduling domains are build. >> >> Fact is, unless we use 1:1, and immutable (across all the guest >> lifetime) pinning, scheduling domains should not be constructed, in >> Linux, by looking at *any* topology information, because that just does >> not make any sense, when vcpus move around. >> >> Let me state this again (hoping to make myself as clear as possible): no >> matter in how much good shape we put CPUID support, no matter how >> beautifully and consistently that will interact with both vNUMA, >> licensing requirements and whatever else. It will be always possible for >> vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time t1, and >> on two different NUMA nodes at time t2. Hence, the Linux scheduler >> should really not skew his load balancing logic toward any of those two >> situations, as neither of them could be considered correct (since >> nothing is!). >> >> For now, this only covers the PV case. HVM case shouldn't be any >> different, but I haven't looked at how to make the same thing happen in >> there as well. >> >> OVERALL DESCRIPTION >> =================== >> What this RFC patch does is, in the Xen PV case, configure scheduling >> domains in such a way that there is only one of them, spanning all the >> pCPUs of the guest. >> >> Note that the patch deals directly with scheduling domains, and there is >> no need to alter the masks that will then be used for building and >> reporting the topology (via CPUID, /proc/cpuinfo, /sysfs, etc.). That is >> the main difference between it and the patch proposed by Juergen here: >> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg05088.html >> >> This means that when, in future, we will fix CPUID handling and make it >> comply with whatever logic or requirements we want, that won't have any >> unexpected side effects on scheduling domains. >> >> Information about how the scheduling domains are being constructed >> during boot are available in `dmesg', if the kernel is booted with the >> 'sched_debug' parameter. It is also possible to look >> at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat. >> >> With the patch applied, only one scheduling domain is created, called >> the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs. You can >> tell that from the fact that every cpu* folder >> in /proc/sys/kernel/sched_domain/ only have one subdirectory >> ('domain0'), with all the tweaks and the tunables for our scheduling >> domain. >> >> EVALUATION >> ========== >> I've tested this with UnixBench, and by looking at Xen build time, on a >> 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0 only, for >> now, but I plan to re-run them in DomUs soon (Juergen may be doing >> something similar to this in DomU already, AFAUI). >> >> I've run the benchmarks with and without the patch applied ('patched' >> and 'vanilla', respectively, in the tables below), and with different >> number of build jobs (in case of the Xen build) or of parallel copy of >> the benchmarks (in the case of UnixBench). >> >> What I get from the numbers is that the patch almost always brings >> benefits, in some cases even huge ones. There are a couple of cases >> where we regress, but always only slightly so, especially if comparing >> that to the magnitude of some of the improvement that we get. >> >> Bear also in mind that these results are gathered from Dom0, and without >> any overcommitment at the vCPU level (i.e., nr. vCPUs == nr pCPUs). If >> we move things in DomU and do overcommit at the Xen scheduler level, I >> am expecting even better results. >> >> RESULTS >> ======= >> To have a quick idea of how a benchmark went, look at the '% >> improvement' row of each table. >> >> I'll put these results online, in a googledoc spreadsheet or something >> like that, to make them easier to read, as soon as possible. >> >> *** Intel(R) Xeon(R) E5620 @ 2.40GHz >> *** pCPUs 16 DOM0 vCPUS 16 >> *** RAM 12285 MB DOM0 Memory 9955 MB >> *** NUMA nodes 2 >> ======================================================================================================================================= >> MAKE XEN (lower == better) >> ======================================================================================================================================= >> # of build jobs -j1 -j6 >> -j8 -j16** -j24 >> vanilla/patched vanilla patched vanilla patched >> vanilla patched vanilla patched vanilla patched >> --------------------------------------------------------------------------------------------------------------------------------------- >> 153.72 152.41 35.33 34.93 >> 30.7 30.33 26.79 25.97 26.88 26.21 >> 153.81 152.76 35.37 34.99 >> 30.81 30.36 26.83 26.08 27 26.24 >> 153.93 152.79 35.37 35.25 >> 30.92 30.39 26.83 26.13 27.01 26.28 >> 153.94 152.94 35.39 35.28 >> 31.05 30.43 26.9 26.14 27.01 26.44 >> 153.98 153.06 35.45 35.31 >> 31.17 30.5 26.95 26.18 27.02 26.55 >> 154.01 153.23 35.5 35.35 >> 31.2 30.59 26.98 26.2 27.05 26.61 >> 154.04 153.34 35.56 35.42 >> 31.45 30.76 27.12 26.21 27.06 26.78 >> 154.16 153.5 37.79 35.58 >> 31.68 30.83 27.16 26.23 27.16 26.78 >> 154.18 153.71 37.98 35.61 >> 33.73 30.9 27.49 26.32 27.16 26.8 >> 154.9 154.67 38.03 37.64 >> 34.69 31.69 29.82 26.38 27.2 28.63 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Avg. 154.067 153.241 36.177 35.536 >> 31.74 30.678 27.287 26.184 27.055 26.732 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Std. Dev. 0.325 0.631 1.215 0.771 >> 1.352 0.410 0.914 0.116 0.095 0.704 >> --------------------------------------------------------------------------------------------------------------------------------------- >> % improvement 0.536 1.772 >> 3.346 4.042 1.194 >> ======================================================================================================================================== >> ==================================================================================================================================================== >> UNIXBENCH >> ==================================================================================================================================================== >> # parallel copies 1 parallel 6 >> parrallel 8 parallel 16 parallel** 24 parallel >> vanilla/patched vanilla patched vanilla >> pached vanilla patched vanilla patched vanilla patched >> ---------------------------------------------------------------------------------------------------------------------------------------------------- >> Dhrystone 2 using register variables 2302.2 2302.1 13157.8 >> 12262.4 15691.5 15860.1 18927.7 19078.5 18654.3 18855.6 >> Double-Precision Whetstone 620.2 620.2 3481.2 >> 3566.9 4669.2 4551.5 7610.1 7614.3 11558.9 11561.3 >> Execl Throughput 184.3 186.7 884.6 >> 905.3 1168.4 1213.6 2134.6 2210.2 2250.9 2265 >> File Copy 1024 bufsize 2000 maxblocks 780.8 783.3 1243.7 >> 1255.5 1250.6 1215.7 1080.9 1094.2 1069.8 1062.5 >> File Copy 256 bufsize 500 maxblocks 479.8 482.8 781.8 >> 803.6 806.4 781 682.9 707.7 698.2 694.6 >> File Copy 4096 bufsize 8000 maxblocks 1617.6 1593.5 2739.7 >> 2943.4 2818.3 2957.8 2389.6 2412.6 2371.6 2423.8 >> Pipe Throughput 363.9 361.6 2068.6 >> 2065.6 2622 2633.5 4053.3 4085.9 4064.7 4076.7 >> Pipe-based Context Switching 70.6 207.2 369.1 >> 1126.8 623.9 1431.3 1970.4 2082.9 1963.8 2077 >> Process Creation 103.1 135 503 >> 677.6 618.7 855.4 1138 1113.7 1195.6 1199 >> Shell Scripts (1 concurrent) 723.2 765.3 4406.4 >> 4334.4 5045.4 5002.5 5861.9 5844.2 5958.8 5916.1 >> Shell Scripts (8 concurrent) 2243.7 2715.3 5694.7 >> 5663.6 5694.7 5657.8 5637.1 5600.5 5582.9 5543.6 >> System Call Overhead 330 330.1 1669.2 >> 1672.4 2028.6 1996.6 2920.5 2947.1 2923.9 2952.5 >> System Benchmarks Index Score 496.8 567.5 1861.9 >> 2106 2220.3 2441.3 2972.5 3007.9 3103.4 3125.3 >> ---------------------------------------------------------------------------------------------------------------------------------------------------- >> % increase (of the Index Score) 14.231 >> 13.110 9.954 1.191 0.706 >> ==================================================================================================================================================== >> >> *** Intel(R) Xeon(R) X5650 @ 2.67GHz >> *** pCPUs 24 DOM0 vCPUS 16 >> *** RAM 36851 MB DOM0 Memory 9955 MB >> *** NUMA nodes 2 >> ======================================================================================================================================= >> MAKE XEN (lower == better) >> ======================================================================================================================================= >> # of build jobs -j1 -j8 >> -j12 -j24** -j32 >> vanilla/patched vanilla patched vanilla patched >> vanilla patched vanilla patched vanilla patched >> --------------------------------------------------------------------------------------------------------------------------------------- >> 119.49 119.47 23.37 23.29 >> 20.12 19.85 17.99 17.9 17.82 17.8 >> 119.59 119.64 23.52 23.31 >> 20.16 19.99 18.19 18.05 18.23 17.89 >> 119.59 119.65 23.53 23.35 >> 20.19 20.08 18.26 18.09 18.35 17.91 >> 119.72 119.75 23.63 23.41 >> 20.2 20.14 18.54 18.1 18.4 17.95 >> 119.95 119.86 23.68 23.42 >> 20.24 20.19 18.57 18.15 18.44 18.03 >> 119.97 119.9 23.72 23.51 >> 20.38 20.31 18.61 18.21 18.49 18.03 >> 119.97 119.91 25.03 23.53 >> 20.38 20.42 18.75 18.28 18.51 18.08 >> 120.01 119.98 25.05 23.93 >> 20.39 21.69 19.99 18.49 18.52 18.6 >> 120.24 119.99 25.12 24.19 >> 21.67 21.76 20.08 19.74 19.73 19.62 >> 120.66 121.22 25.16 25.36 >> 21.94 21.85 20.26 20.3 19.92 19.81 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Avg. 119.919 119.937 24.181 23.73 >> 20.567 20.628 18.924 18.531 18.641 18.372 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Std. Dev. 0.351 0.481 0.789 0.642 >> 0.663 0.802 0.851 0.811 0.658 0.741 >> --------------------------------------------------------------------------------------------------------------------------------------- >> % improvement -0.015 1.865 >> -0.297 2.077 1.443 >> ======================================================================================================================================== >> ==================================================================================================================================================== >> UNIXBENCH >> ==================================================================================================================================================== >> # parallel copies 1 parallel 8 >> parrallel 12 parallel 24 parallel** 32 parallel >> vanilla/patched vanilla patched vanilla >> pached vanilla patched vanilla patched vanilla patched >> ---------------------------------------------------------------------------------------------------------------------------------------------------- >> Dhrystone 2 using register variables 2650.1 2664.6 18967.8 >> 19060.4 27534.1 27046.8 30077.9 30110.6 30542.1 30358.7 >> Double-Precision Whetstone 713.7 713.5 5463.6 >> 5455.1 7863.9 7923.8 12725.1 12727.8 17474.3 17463.3 >> Execl Throughput 280.9 283.8 1724.4 >> 1866.5 2029.5 2367.6 2370 2521.3 2453 2506.8 >> File Copy 1024 bufsize 2000 maxblocks 891.1 894.2 1423 >> 1457.7 1385.6 1482.2 1226.1 1224.2 1235.9 1265.5 >> File Copy 256 bufsize 500 maxblocks 546.9 555.4 949 >> 972.1 882.8 878.6 821.9 817.7 784.7 810.8 >> File Copy 4096 bufsize 8000 maxblocks 1743.4 1722.8 3406.5 >> 3438.9 3314.3 3265.9 2801.9 2788.3 2695.2 2781.5 >> Pipe Throughput 426.8 423.4 3207.9 >> 3234 4635.1 4708.9 7326 7335.3 7327.2 7319.7 >> Pipe-based Context Switching 110.2 223.5 680.8 >> 1602.2 998.6 2324.6 3122.1 3252.7 3128.6 3337.2 >> Process Creation 130.7 224.4 1001.3 >> 1043.6 1209 1248.2 1337.9 1380.4 1338.6 1280.1 >> Shell Scripts (1 concurrent) 1140.5 1257.5 5462.8 >> 6146.4 6435.3 7206.1 7425.2 7636.2 7566.1 7636.6 >> Shell Scripts (8 concurrent) 3492 3586.7 7144.9 >> 7307 7258 7320.2 7295.1 7296.7 7248.6 7252.2 >> System Call Overhead 387.7 387.5 2398.4 >> 2367 2793.8 2752.7 3735.7 3694.2 3752.1 3709.4 >> System Benchmarks Index Score 634.8 712.6 2725.8 >> 3005.7 3232.4 3569.7 3981.3 4028.8 4085.2 4126.3 >> ---------------------------------------------------------------------------------------------------------------------------------------------------- >> % increase (of the Index Score) 12.256 >> 10.269 10.435 1.193 1.006 >> ==================================================================================================================================================== >> >> *** Intel(R) Xeon(R) X5650 @ 2.67GHz >> *** pCPUs 48 DOM0 vCPUS 16 >> *** RAM 393138 MB DOM0 Memory 9955 MB >> *** NUMA nodes 2 >> ======================================================================================================================================= >> MAKE XEN (lower == better) >> ======================================================================================================================================= >> # of build jobs -j1 -j20 >> -j24 -j48** -j62 >> vanilla/patched vanilla patched vanilla patched >> vanilla patched vanilla patched vanilla patched >> --------------------------------------------------------------------------------------------------------------------------------------- >> 267.78 233.25 36.53 35.53 >> 35.98 34.99 33.46 32.13 33.57 32.54 >> 268.42 233.92 36.82 35.56 >> 36.12 35.2 34.24 32.24 33.64 32.56 >> 268.85 234.39 36.92 35.75 >> 36.15 35.35 34.48 32.86 33.67 32.74 >> 268.98 235.11 36.96 36.01 >> 36.25 35.46 34.73 32.89 33.97 32.83 >> 269.03 236.48 37.04 36.16 >> 36.45 35.63 34.77 32.97 34.12 33.01 >> 269.54 237.05 40.33 36.59 >> 36.57 36.15 34.97 33.09 34.18 33.52 >> 269.99 238.24 40.45 36.78 >> 36.58 36.22 34.99 33.69 34.28 33.63 >> 270.11 238.48 41.13 39.98 >> 40.22 36.24 38 33.92 34.35 33.87 >> 270.96 239.07 41.66 40.81 >> 40.59 36.35 38.99 34.19 34.49 37.24 >> 271.84 240.89 42.07 41.24 >> 40.63 40.06 39.07 36.04 34.69 37.59 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Avg. 269.55 236.688 38.991 37.441 >> 37.554 36.165 35.77 33.402 34.096 33.953 >> --------------------------------------------------------------------------------------------------------------------------------------- >> Std. Dev. 1.213 2.503 2.312 2.288 >> 2.031 1.452 2.079 1.142 0.379 1.882 >> --------------------------------------------------------------------------------------------------------------------------------------- >> % improvement 12.191 3.975 >> 3.699 6.620 0.419 >> ======================================================================================================================================== > > I'm a bit confused here as to why, if dom0 has 16 vcpus in all of your > tests, you change the -j number (apparently) based on the number of > pcpus available to Xen. Wouldn't it make more sense to stick with > 1/6/8/16/24? That would allow us to have actually comparable numbers. > > But in any case, it seems to me that the numbers do show a uniform > improvement and no regressions -- I think this approach looks really > good, particularly as it is so small and well-contained. That said, it's probably a good idea to make this optional somehow, so that if people do decide to do a pinning / partitioning approach, the guest scheduler actually can take advantage of topological information. -George _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |