Re: [Xen-devel] [PATCH v2] tools: libxc: flush data cache after loading images into guest memory

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Fri, 2013-12-13 at 00:49 +0000, Julien Grall wrote:
> 
> On 12/12/2013 02:23 PM, Ian Campbell wrote:
> > On ARM guest OSes are started with MMU and Caches disables (as they are on
> > native) however caching is enabled in the domain running the builder and
> > therefore we must flush the cache as we load the blobs, otherwise when the
> > guest starts running it may not see them. The dom0 build in the hypervisor 
> > has
> > the same requirements and already does the right thing.
> >
> > The mechanism for performing a cache flush from userspace is OS specific, so
> > implement this as a new osdep hook:
> >
> >   - On 32-bit ARM Linux provides a system call to flush the cache.
> >   - On 64-bit ARM Linux the processor is configured to allow cache flushes
> >     directly from userspace.
> >   - Non-Linux platforms will need to provide their own implementation. If
> >     similar mechanisms are not available then a new privcmd ioctl should be 
> > a
> >     suitable alternative.
> >
> > No cache maintenance is required on x86, so provide a stub for all non-Linux
> > platforms which returns success on x86 only and log an error otherwise.
> >
> > This fixes guest building on Xgene which has a very large L3 cache and so is
> > particularly susceptible to this problem. It has also been observed
> > sporadically on midway.
> 
> This patch doesn't solve issue on Midway.

That's a shame. I think we should go ahead with this patch regardless,
since it does fix arm64 and introduces the infrastructure for arm32. I
think there is no harm in adding the syscall on arm32 for now.

> cacheflush syscall on ARM32 is calling DCCMVAU (Data Clean Cache by MVA
> to PoU), that is not enough.
> As I understand the ARM ARM B2.2.6 (page B2-1275):
>      - PoC means the data will be written to the RAM
>      - PoU means, in a same inner shareable domain, instruction/data
> cache and translation page table will see the same value for a specific
> MVA. It doesn't means that the data will reach the RAM.

This is essentially my understanding as well.

> I did some test and indeed DCCMVAC (Data Clean Cache By MVA to PoC)
> resolves the problem on Midway (and generally on ARMv7).

Good.

> Unfortunately Linux doesn't provide any syscall to call this function
> for ARMv7 and it's not possible to call cache instruction from
> userspace. What we could do is:
>      - Use the "flags" parameters of cacheflush syscall and call a
> function which DCCMVAC (for instance __cpuc_flush_dcache_area)
>      - Extend privcmd to have a flush cache ioctl

Personally I think the first is nicer, but ultimately we need input from
l-a-k on this one and would be happy with either.

> Both solution would mean waiting Linux 3.14 (I don't think we can get an
> accepted patch for 3.13).

That's a shame, but it is what it is. We could perhaps tag it for a
stable backport.

> I have also tried to trap PoU cache instruction (via HCR.TPU). But when 
> Xen call DCCIMVAC/DCCIMVAU, the processor will raise a data abort fault.

Xen MVAs are different to guest MVAs, and the guest MVA is very likely
to not correspond to a mapping in the Xen space, so this approach
can't/won't work I think.

> Any thoughts?

I think we'll just have to wait for the Linux part of the fix to land.
Are you going to look into this?

Ian.




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel