RE: Proposal for Porting Xen to Armv8-R64 - DraftA

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 2 Mar 2022, Wei Chen wrote:
> > > > > > If not, and considering that we have to generate
> > > > > > ARM_MPU_*_MEMORY_START/END anyway at build time, would it make
> > sense
> > > > to
> > > > > > also generate mpu,guest-memory-section, xen,static-mem, etc. at
> > build
> > > > > > time rather than passing it via device tree to Xen at runtime?
> > > > > >
> > > > >
> > > > > Did you mean we still add these information in device tree, but for
> > > > build
> > > > > time only. In runtime we don't parse them?
> > > >
> > > > Yes, something like that, but see below.
> > > >
> > > >
> > > > > > What's the value of doing ARM_MPU_*_MEMORY_START/END at build time
> > and
> > > > > > everything else at runtime?
> > > > >
> > > > > ARM_MPU_*_MEMORY_START/END is defined by platform. But other things
> > are
> > > > > users customized. They can change their usage without rebuild the
> > image.
> > > >
> > > > Good point.
> > > >
> > > > We don't want to have to rebuild Xen if the user updated a guest
> > kernel,
> > > > resulting in a larger boot-module-section.
> > > >
> > > > So I think it makes sense that "mpu,boot-module-section" is generated
> > by
> > > > the scripts (e.g. ImageBuilder) at build time, and Xen reads the
> > > > property at boot from the runtime device tree.
> > > >
> > > > I think we need to divide the information into two groups:
> > > >
> > > >
> > > > # Group1: board info
> > > >
> > > > This information is platform specific and it is not meant to change
> > > > depending on the VM configuration. Ideally, we build Xen for a
> > platform
> > > > once, then we can use the same Xen binary together with any
> > combination
> > > > of dom0/domU kernels and ramdisks.
> > > >
> > > > This kind of information doesn't need to be exposed to the runtime
> > > > device tree. But we can still use a build-time device tree to generate
> > > > the addresses if it is convenient.
> > > >
> > > > XEN_START_ADDRESS, ARM_MPU_DEVICE_MEMORY_*, and
> > ARM_MPU_NORMAL_MEMORY_*
> > > > seem to be part of this group.
> > > >
> > >
> > > Yes.
> > >
> > > >
> > > > # Group2: boot configuration
> > > >
> > > > This information is about the specific set of binaries and VMs that we
> > > > need to boot. It is conceptually similar to the dom0less device tree
> > > > nodes that we already have. If we change one of the VM binaries, we
> > > > likely have to refresh the information here.
> > > >
> > > > "mpu,boot-module-section" probably belongs to this group (unless we
> > find
> > > > a way to define "mpu,boot-module-section" generically so that we don't
> > > > need to change it any time the set of boot modules change.)
> > > >
> > > >
> > >
> > > I agree.
> > >
> > > > > > It looks like we are forced to have the sections definitions at
> > build
> > > > > > time because we need them before we can parse device tree. In that
> > > > case,
> > > > > > we might as well define all the sections at build time.
> > > > > >
> > > > > > But I think it would be even better if Xen could automatically
> > choose
> > > > > > xen,static-mem, mpu,guest-memory-section, etc. on its own based on
> > the
> > > > > > regular device tree information (/memory, /amba, etc.), without
> > any
> > > > need
> > > > > > for explicitly describing each range with these new properties.
> > > > > >
> > > > >
> > > > > for mpu,guest-memory-section, with the limitations: no other usage
> > > > between
> > > > > different guest' memory nodes, this is OK. But for xen,static-mem
> > (heap),
> > > > > we just want everything on a MPU system is dertermistic. But, of
> > course
> > > > Xen
> > > > > can select left memory for heap without static-mem.
> > > >
> > > > It is good that you think they can be chosen by Xen.
> > > >
> > > > Differently from "boot-module-section", which has to do with the boot
> > > > modules selected by the user for a specific execution,
> > > > guest-memory-section and static-mem are Xen specific memory
> > > > policies/allocations.
> > > >
> > > > A user wouldn't know how to fill them in. And I worry that even a
> > script
> > >
> > > But users should know it, because static-mem for guest must be allocated
> > > in this range. And users take the responsibility to set the DomU's
> > > static allocate memory ranges.
> > 
> > Let me premise that my goal is to avoid having many users reporting
> > errors to xen-devel and xen-users when actually it is just a wrong
> > choice of addresses.
> > 
> > I think we need to make a distinction between addresses for the boot
> > modules, e.g. addresses where to load xen, the dom0/U kernel, dom0/U
> > ramdisk in memory at boot time, and VM static memory addresses.
> > 
> > The boot modules addresses are particularly difficult to fill in because
> > they are many and a small update in one of the modules could invalidate
> > all the other addresses. This is why I ended up writing ImageBuilder.
> > Since them, I received several emails from users thanking me for
> > ImageBuilder :-)
> > 
> 
> Thanks +999 😊
> 
> 
> > The static VM memory addresses (xen,static-mem) should be a bit easier
> > to fill in correctly. They are meant to be chosen once, and it shouldn't
> > happen that an update on a kernel forces the user to change all the VM
> > static memory addresses. Also, I know that some users actually want to
> > be able to choose the domU addresses by hand because they have specific
> > needs. So it is good that we can let the user choose the addresses if
> > they want to.
> > 
> 
> Yes.
> 
> > With all of that said, I do think that many users won't have an opinion
> > on the VM static memory addresses and won't know how to choose them.
> > It would be error prone to let them try to fill them in by hand. So I
> > was already planning on adding support to ImageBuilder to automatically
> > generate xen,static-mem for dom0less domains.
> > 
> 
> Let me make sure that's what you said: Users give an VM memory size to
> ImageBuilder, and ImageBuilder will generate xen,static-mem = <start, size>.
> For specific VM, ImageBuilder also can accept start and size as inputs?
> 
> Do I understand this correctly?

Yes, exactly

 
> > Going back to this specific discussion about boot-module-section: I can
> > see now that, given xen,static-mem is chosen by ImageBuilder (or
> 
> By hand : )
> 
> > similar) and not Xen, then it makes sense to have ImageBuilder (or
> > similar) also generate boot-module-section.
> > 
> 
> If my above understanding is right, then yes.

Yes, I think we are on the same page
 
 
> > > > like ImageBuilder wouldn't be the best place to pick these values --
> > > > they seem too "important" to leave to a script.
> > > >
> > > > But it seems possible to choose the values in Xen:
> > > > - Xen knows ARM_MPU_NORMAL_MEMORY_* because it was defined at build
> > time
> > > > - Xen reads boot-module-section from device tree
> > > >
> > > > It should be possible at this point for Xen to pick the best values
> > for
> > > > guest-memory-section and static-mem based on the memory available.
> > > >
> > >
> > > How Xen to pick? Does it mean in static allocation DomU DT node, we just
> > > need a size, but don't require a start address for static-mem?
> > 
> > Yes the idea was that the user would only provide the size (e.g.
> > DOMU_STATIC_MEM[1]=1024) and the addresses would be automatically
> > calculated. But I didn't mean to change the existing xen,static-mem
> > device tree bindings. So it is best if the xen,static-mem addresses
> > generation is done by ImageBuilder (or similar tool) instead of Xen.
> > 
> 
> If we still keep the option for user to specify the start and size
> parameters for VM memory, because it maybe very important for a
> deterministic system (fully static system), I agree with you.
> 
> And in current static-allocation, I think Xen doesn't generate
> xen,static-mem addresses, all by hands...

Yeah 


> > Sorry for the confusion!
> > 
> 
> NP ; )