Xen project Mailing List

2013/12/17 Alice Wan <wanjia19870902@xxxxxxxxx>

hi all,

recently, our group did some test with vcpu-pin, and found vcpu-list result wasn't the same as we thought. our scheduler is default scheduler(credit), our host has 16cores with ht.

netback-test1 1 0 11 -b- 7.1 0-1,11-12
netback-test1 1 1 0 -b- 0.3 0-1,11-12
netback-test1 1 2 11 -b- 0.2 0-1,11-12

netback-test1 1 3 0 -b- 0.4 0-1,11-12
netback-test2 2 0 7 -b- 7.1 0-1,11-12
netback-test2 2 1 11 -b- 0.2 0-1,11-12

netback-test2 2 2 0 -b- 0.2 0-1,11-12
netback-test2 2 3 11 -b- 0.3 0-1,11-12
netback-test3 3 0 11 -b- 6.9 0-1,11-12

netback-test3 3 1 13 -b- 0.2 0-1,11-12
netback-test3 3 2 14 -b- 0.2 0-1,11-12
netback-test3 3 3 0 -b- 0.3 0-1,11-12

from the result, we found domain3's vcpu1, vcpu2 were running on the cpus which weren't in the cpu_affinity

then, i read some code about schedule_credit's picking up cpu and find its implement like this

nxt = cpumask_cycle(cpu, &cpus);

if ( cpumask_test_cpu(cpu, per_cpu(cpu_core_mask, nxt)) )
{
/* We're on the same socket, so check the busy-ness of threads.

* Migrate if # of idlers is less at all */
ASSERT( cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
migrate_factor = 1;
cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_sibling_mask, cpu));

cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_sibling_mask, nxt));
}
else
{
/* We're on different sockets, so check the busy-ness of cores.

* Migrate only if the other core is twice as idle */
ASSERT( !cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
migrate_factor = 2;
cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_core_mask, cpu));

cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_core_mask, nxt));
}

weight_cpu = cpumask_weight(&cpu_idlers);
weight_nxt = cpumask_weight(&nxt_idlers);

/* smt_power_savings: consolidate work rather than spreading it */
if ( sched_smt_power_savings ?
weight_cpu > weight_nxt :
weight_cpu * migrate_factor < weight_nxt )

{
cpumask_and(&nxt_idlers, &cpus, &nxt_idlers);
spc = CSCHED_PCPU(nxt);
cpu = cpumask_cycle(spc->idle_bias, &nxt_idlers);

cpumask_andnot(&cpus, &cpus, per_cpu(cpu_sibling_mask, cpu));

if the cpu_affinity which isn't in the same socket , the picked cpu might be one of the intersection of nxt cpu_core_mask and idlers, which isn't in the cpu_affinity and even is in the same socket, picked cpu might be one of intersection of nxt cpu_sibling_mask and idlers.

it really depends on the setting of cpu_affinity. if you set all the siblings cpu of one socket to vcpu, it works well.

  in our test environment, we use xen4.0, code doesn't have migrate_factor, so it's easier to trigger.

i wanna know whether this is a bug or a design consideration, in our opinion, this scheduling result must be the same as we set even though we need better setting.

can anyone give some ideas about my question, thanks in advance

regards,
wanjia

Re: [Xen-devel] schedule_credit not strict in cpu_affinity