[Xen-devel] Performance Monitoring Counter(PMC) Problem

[alex <leiye.xen@xxxxxxxxx>]

Hi all,

 

I am running 64bit xen-3.4.2 on AMD Phenom II Quad-core processor, on which each core has sepearate performance counter. My research is to use performance counter to track interesting events(say, L3 cache miss) for each VM. Thus, I developed software multiplexing to support PMC tracking for individual PVOPS-VMs. Each domain has its own logical performance counter. Every time it will be scheduled, performance counter for that domain is reloaded and started. When a domain (VCPU) is de-scheduled after 30ms, the performance counter is stopped and stored to logical counter.  I modified context_switch (in arch/x86/domain.c) by adding statements below:

 

#define MAX_DOMAIN_NUMBER 8

volatile uint64_t perfcounter[MAX_DOMAIN_NUMBER] = { 0, 0, 0, 0, 0, 0, 0, 0 };   // element 0 is reserved for dom0, element 7 is reserved for IDLE_DOMAIN_ID.

 

// multiplexing the performance counter for more than 4 VMs

void startPMC(unsigned int pcpu_id, unsigned int domain_id)
{

    uint32_t eax, edx;

    /* reload performance counter for next dom */

    if(domain_id == IDLE_DOMAIN_ID) {
          wrmsrl(MSR_K7_PERFCTR0, perfcounter[7]);
    }
    else {
         wrmsrl(MSR_K7_PERFCTR0, perfcounter[domain_id]);
    }
    edx = 0x4;
    eax = 0xE1 | (0x3 << 16) | (0x1 << (12 + pcpu_id)) | (0x7 << 8) | (0x1 << 22); // L3 cache misses for accesses from a core(cpu)
    wrmsr(MSR_K7_EVNTSEL0, eax, edx); 

}

 

// multiplexing the performance counter for more than 4 VMs

void stopPMC(unsigned int pcpu_id, unsigned int domain_id)
{
      uint32_t eax, edx;

      edx = 0x4;
      eax = (0xE1 | (0x3 << 16) | (0x1 << (12 + pcpu_id)) | (0x7 << 8)) & ~(0x1 << 22); // L3 cache misses for accesses from core(cpu)
      wrmsr(MSR_K7_EVNTSEL0, eax, edx);

 

     /* save current performance counter */
    if(domain_id == IDLE_DOMAIN_ID) {
          rdmsrl(MSR_K7_PERFCTR0, perfcounter[7]);
    }
    else {
         rdmsrl(MSR_K7_PERFCTR0, perfcounter[domain_id]);
    }

}

 

void context_switch(struct vcpu *prev, struct vcpu *next)
{

   unsigned int cpu = smp_processor_id();

   ......

 

    stopPMC(cpu, prev->domain->domain_id);
    startPMC(cpu, next->domain->domain_id);

 

   .......

}

 

In my experiment, I run 6 pvops DomUs and each pv domain has one VCPU and 1GB allocated memory. Dom0 has four VCPUs, and 1GB memory. The testing code is very simple and listed below.

int main(void)

{

  int i;

  char *buf;

  buf = (char *)malloc(400 * 1024 * 1024);

  for(i=0; i<400*1024*1024;i++)

    buf[i] = 1;

  return 0;

}

 

I run the testing code only in DomUs and found the mismatching results in different scenarios.

 

Results:

1) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1.....  and run the testing code in a DomU whose vcpu is also pinned, the performance counter I checked is around 6553600 (L3 cache misses). That means DomU accessed about 400MB data. (The cache line size is 64B).

 

2) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1.....  but don't pin DomU's vcpu, the performance counter is a little less than 6553600 (L3 cache misses).

 

3) If I don't pin Dom0 vcpus, and still run testing code in a DomU, the performance counter I got is much less than 6553600 (L3 cache misses), and the average value is 2949120.

 

Now I move to run testing code in 2 DomUs, and also found mismatching results.

4) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1.....  and pin each DomU vcpu to different pcpu, the result is around 6553600 (L3 cache misses).

5) If I pin Dom0 vcpu 0 to pcpu 0, vcpu 1 to pcpu 1.....  and pin each DomU vcpu to the same pcpu, the result is much less than 6553600 (L3 cache misses).

 

To validate each VM accesses the specified data, I used "page-fault approach" which is another way to track memory accesses and find out each VM has about 102400 page faults, which is equivalent to 400MB. Since there is no paging sharing between VMs, each VM should access the same amount of 400MB data. I am not sure what is wrong with the performance counter multiplexing.  So can anyone give me some suggestions?

 

Thank you,

 

Lei

University of Arizona

 

 

 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel