So, as I understand it, you expect the 6.5M cache misses for cases 1, 2,
4, but don't understand the much lower results for 3 and 5? And you
suspect that the numbers you're getting aren't valid, but are the result
of some mismanagement?
For one, you don't check in your {start,stop}PMC() functions whether the
domain ID is less than MAX_DOMAIN_NUMBER-1.
You're aware also that context_switch() isn't called when switching from a
It might be worth adding a traces for the values you're reading from and
writing to the registers, and using xenalyze to see if you notice
anything strange.
The xenalyze source can be found here:
http://xenbits.xensource.com/ext/xenalyze.hg
You'd have to make xenalyze understand your performance trace record,
but it will understand most everything else.
-George
On 21/11/10 03:28, alex wrote:
Hi all,
I am running 64bit xen-3.4.2 on AMD Phenom II Quad-core processor, on
which each core has sepearate performance counter. My research is to use
performance counter to track interesting events(say, L3 cache miss) for
each VM. Thus, I developed software multiplexing to support PMC tracking
for individual PVOPS-VMs. Each domain has its own logical performance
counter. Every time it will be scheduled, performance counter for that
domain is reloaded and started. When a domain (VCPU) is de-scheduled
after 30ms, the performance counter is stopped and stored to logical
counter. I modified context_switch (in arch/x86/domain.c) by adding
statements below:
#define MAX_DOMAIN_NUMBER 8
volatile uint64_t perfcounter[MAX_DOMAIN_NUMBER] = { 0, 0, 0, 0, 0, 0,
0, 0 }; // element 0 is reserved for dom0, element 7 is reserved for
IDLE_DOMAIN_ID.
// multiplexing the performance counter for more than 4 VMs
void startPMC(unsigned int pcpu_id, unsigned int domain_id)
{
uint32_t eax, edx;
/* reload performance counter for next dom */
if(domain_id == IDLE_DOMAIN_ID) {
wrmsrl(MSR_K7_PERFCTR0, perfcounter[7]);
}
else {
wrmsrl(MSR_K7_PERFCTR0, perfcounter[domain_id]);
}
edx = 0x4;
eax = 0xE1 | (0x3 << 16) | (0x1 << (12 + pcpu_id)) | (0x7 << 8) |
(0x1 << 22); // L3 cache misses for accesses from a core(cpu)
wrmsr(MSR_K7_EVNTSEL0, eax, edx);
}
// multiplexing the performance counter for more than 4 VMs
void stopPMC(unsigned int pcpu_id, unsigned int domain_id)
{
uint32_t eax, edx;
edx = 0x4;
eax = (0xE1 | (0x3 << 16) | (0x1 << (12 + pcpu_id)) | (0x7 << 8))
& ~(0x1 << 22); // L3 cache misses for accesses from core(cpu)
wrmsr(MSR_K7_EVNTSEL0, eax, edx);
/* save current performance counter */
if(domain_id == IDLE_DOMAIN_ID) {
rdmsrl(MSR_K7_PERFCTR0, perfcounter[7]);
}
else {
rdmsrl(MSR_K7_PERFCTR0, perfcounter[domain_id]);
}
}
void context_switch(struct vcpu *prev, struct vcpu *next)
{
unsigned int cpu = smp_processor_id();
......
stopPMC(cpu, prev->domain->domain_id);
startPMC(cpu, next->domain->domain_id);
.......
}
In my experiment, I run 6 pvops DomUs and each pv domain has one VCPU
and 1GB allocated memory. Dom0 has four VCPUs, and 1GB memory. The
testing code is very simple and listed below.
int main(void)
{
int i;
char *buf;
buf = (char *)malloc(400 * 1024 * 1024);
for(i=0; i<400*1024*1024;i++)
buf[i] = 1;
return 0;
}
I run the testing code only in DomUs and found the mismatching results
in different scenarios.
Results:
1) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1..... and run the
testing code in a DomU whose vcpu is also pinned, the performance
counter I checked is around 6553600 (L3 cache misses). That means DomU
accessed about 400MB data. (The cache line size is 64B).
2) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1..... but don't
pin DomU's vcpu, the performance counter is a little less than 6553600
(L3 cache misses).
3) If I don't pin Dom0 vcpus, and still run testing code in a DomU, the
performance counter I got is much less than 6553600 (L3 cache misses),
and the average value is 2949120.
Now I move to run testing code in 2 DomUs, and also found mismatching
results.
4) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1..... and pin each
DomU vcpu to different pcpu, the result is around 6553600 (L3 cache misses).
5) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1..... and pin each
DomU vcpu to the same pcpu, the result is much less than 6553600 (L3
cache misses).
To validate each VM accesses the specified data, I used "page-fault
approach" which is another way to track memory accesses and find out
each VM has about 102400 page faults, which is equivalent to 400MB.
Since there is no paging sharing between VMs, each VM should access the
same amount of 400MB data. I am not sure what is wrong with the
performance counter multiplexing. So can anyone give me some suggestions?
Thank you,
Lei
University of Arizona
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home