[Xen-devel] Xen Project 4.7 and 4.6.3 Release

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

Dear Community Members,

I’m pleased to announce the release of Xen Project Hypervisor 4.7 and Xen 
Project Hypervisor 4.6.3.

Best Regards
Lars, Wei, Jan

= Xen Project Hypervisor 4.7 =

This new release focuses on improving code quality, security hardening, 
security features, live migration support, usability improvements and support 
for new hardware features — this is also the first release of our fixed term 
June – December release cycle.

We continue to strive to make Xen Project Hypervisor the most secure open 
source hypervisor to match the security challenges in cloud computing, and for 
embedded and IoT use-cases. We are continuing to improve upon the performance 
and scalability for our users, and aim to continuously bring many new features 
to our users in a timely manner.

To make it easier to understand the major changes during this release cycle, 
I’ve grouped them below into several categories:

* Security Features
* Migration Support
* Performance and Workloads
* Support for new Hardware Features
* Drivers and Devices (Linux, FreeBSD and other)

== Security Features ==

Reboot-free Live Patching: Xen Project Hypervisor 4.7 comes equipped with Live 
Patching, a technology that enables re-boot free deployment of security patches 
to minimize disruption and downtime during security upgrades for system 
administrators and DevOps practitioners. Xen Project 4.7 implements version 1 
of the Xen Project’s Live Patching specification, which is designed to encode 
the vast majority of security patches (approximately 90%) as Live Patching 
payloads. This version ships with a Live Patching enabled hypervisor and 
payload deployment tools and is available as a technology preview.

KCONFIG support: For security, embedded automotive and IoT use cases, Xen 
Project introduced the ability to remove core Xen Hypervisor features at 
compile time via KCONFIG. This ability creates a more lightweight hypervisor 
and eliminates extra attack surfaces that are beneficial in security-first 
environments, microservice architectures and environments that have heavy 
compliance and certification needs, like automotive.

Improvements to the Virtual Machine Introspection (VMI) subsystem: A number of 
performance, scalability, robustness and interface improvements have been added 
to theVirtual Machine Introspection subsystem, that was introduced in Xen 4.5. 
In addition,Bitdefender Hypervisor Introspection leveraging Xen Project Virtual 
Machine Introspection, has recently been released as a new enterprise security 
solution to discover and remedy deep threats that remain hidden via traditional 
endpoint security tools.

Foundation work to tolerate a restartable Dom0: Several key components in a Xen 
Project system run in Dom0, which make Dom0 the single point of failure. Xen 
Project has been able to run xenstored, the daemon for managing the 
hypervisor’s central settings repository on a Xen Project host, in a sandboxed 
Virtual Machine called xenstored stub domain since Xen Project version 4.2. In 
Xen 4.7, we have made it easier to build xenstored stub domains and for them to 
tolerate a Dom0 restart. This will make Dom0 less critical to a Xen Project 
system and help us move towards a more robust and secure architecture in the 
future. More work in this area is expected in subsequent releases.

== Migration Support ==

Improved Migration support: CPU ID Levelling enables migration of VM’s between 
a larger range of non-identical hosts than previously supported.

Fault Tolerance / Coarse-grained Lock-stepping (COLO): Xen 4.5 laid the 
foundation forCOLO while improving the Xen Project’s Hypervisors Live Migration 
and Remus High Availability support. The COLO Manager, which introduces a 
relaxed approach to checkpointing that avoids unnecessary checkpoints enabling 
near native performance for many workloads, has been fully integrated as an 
experimental feature into Xen 4.7. Note that the COLO Block Replication and 
COLO Proxy components, both of which are QEMU components, are currently still 
reviewed by the QEMU community. Both components are available as out-of-tree 
add-ons to the Xen Project Hypervisor, until fully integrated into QEMU.

== Performance and Workloads ==

Support for a wider range of workloads and applications: The PV guest limit 
restriction of 512GB has been removed to allow the creation of huge PV domains 
in the TB range. TB sized VMs, coupled with Xen Project’s existing support for 
512 vCPUs per VM, enable execution of memory and compute intensive workloads 
such as big data analytics workloads and in-memory databases.

Improved Credit 2 scheduler: The Credit2 scheduler is one (big) step closer to 
being ready for production use. It is now possible to instruct the scheduler to 
organize its runqueues and perform load balancing at core, socket or NUMA node 
granularity. More fine grained (core) configurations, deliver more aggressive 
load balancing, and are best suited for medium size systems. This feature has 
been proven to enable very good performance, especially if Hyper Threading is 
present.

Less fine grained configurations entail less overhead, and is suitable for 
larger servers or when no Hyper Threading is available. In addition, Credit2 
has been extended to allow pinning of vCPUs to pCPUs (also known as “hard 
affinity”), allowing system administrators to configure the system in the exact 
way they want, and achieve the best setup for a given workload (for instance, a 
guarantee that a certain subset of vCPUs are always able to run when they need 
to run).

Improved RTDS scheduler: The RTDS scheduler is a real-time CPU scheduler built 
to provide guaranteed CPU capacity to guest VMs on SMP hosts, which primarily 
targets embedded, real-time and low-latency workloads. In Xen Project 4.7, the 
scheduling model has been changed from a quantum-driven to an event-driven 
model, which reduces scheduling overhead and thus scalability and performance 
for embedded and realtime workloads. In addition, per-VCPU parameter 
configuration has been added to allow better scheduler control for specialised 
workloads.

Per-cpu reader-writer lock: This new infrastructure allows for the fast path 
read case to have low overhead by only setting/clearing a per-cpu variable for 
using the read lock. After transforming various hypervisor locks to this 
infrastructure, VM-VM network transfer with 16 queues jumped from 15 gbit/s to 
48 gbit/s on a 2 socket Haswell-EP host.

== Usability Improvements ==

PVUSB Support: In Xen Project 4.7, a new XL command line interface to manage 
PVUSB devices has been introduced to manage PVUSB devices for PV guests. Both 
in kernel PVUSB backend and QEMU backend are supported.

Hot plugging of QEMU disk backends: Xen Project now enables hot-plugging of USB 
devices as well as QEMU disk backends, such as drbd, iscsi, and more in HVM 
guests. This new feature allows users to add and remove disk backends to 
virtual machines without the need to reboot the guest.

Soft-reset: The soft reset feature for HVM guests allows for a more graceful 
shutdown and restart of the HVM guest.

== New Hardware Support ==

=== Features specific to the ARM Architecture ===

SBBR Compliance: Xen Project now supports booting on hosts that expose ACPI 6.0 
(and later) information. The ARM Server Base Boot Requirements (SBBR) stipulate 
that compliant systems need to express hardware resources with ACPI; thus this 
support will come in useful for ARM Servers. This effort was carried out by 
Shannon Zhao of Linaro with minor patches from Julien Grall of ARM.

PCSI 1.0 Compatibility: PSCI 1.0 compatibility allows Xen Project software to 
operate on systems that expose PSCI 1.0 methods. Now, all 1.x versions of PSCI 
will be compatible with Xen Project software. More information on Power State 
Coordination Interface can be found here. This effort was also carried out by 
Julien Grall with a patch from Dirk Behme of Bosch.

vGIC-v3: Virtual Generic Interrupt Controller version 3. Reworked to be 
spec-compliant and optimised in some code paths.

Wallclock support: ARM guest can now get wallclock time directly from Xen 
Project via shared info page.

=== Features specific to Intel® Xeon® processor product family ===

Improved Interrupt Efficiency: Xen Project 4.7 supports VT-d Posted Interrupts, 
which provides hardware-level acceleration to increase interrupt virtualization 
efficiency. It reduces latency and improves user experience through performance 
improvements, especially for interrupt-intensive front-end workloads such as 
web servers. Note that Posted Interrupts in Xen Project 4.7 are still 
experimental and disabled by default.

Code and Data Prioritization: Xen Project 4.7 is the first to include Code and 
Data Prioritization (CDP), part of the Intel® Resource Director Technology 
(RDT) Framework and an extension of Cache Allocation Technology (CAT), first 
introduced in Xen Project 4.6. The introduction of CDP allows isolation of 
code/data within the shared L3 cache of multi-tenant environments, reducing 
contention and improving performance.

Other Intel Features: Additional features specific to the Intel Xeon processor 
family in Xen Project 4.7 include: VMX TSC Scaling, which allows for easier 
migration between machines with different CPU frequencies and support for 
Memory Protection Keys, a new security feature for hardening the software stack.

== Drivers and Devices (Linux, FreeBSD and other) ==

During the Xen Project 4.7 release cycle, we made significant improvements to 
major operating systems and components we rely on to improve interoperability. 
During this development cycle 1494 Xen Project only related changesets – mostly 
bug fixes and small improvements – were applied to Linux, FreeBSD, NetBSD, QEMU 
and the Windows PV drivers: more than twice as many as in the 4.6 release cycle.

== Summary ==

With dozens of major improvements, many more bug fixes and small improvements, 
and significant improvements to Drivers and Devices, Xen Project 4.7 reflects a 
thriving community around the Xen Project Hypervisor.

We are extremely proud of achieving the highest quality of the release while 
increasingdevelopment velocity across the hypervisor and its upstream 
dependencies by about 16%. In particular, our latest security related features 
enable Xen Project software to compete in the security appliance market and 
help answer some of the difficult questions regarding security in the cloud era.

We set out at the beginning of this release cycle to foster greater 
collaboration among vendors, individual developers, upstream maintainers, other 
projects and distributions. During this release cycle we continued to see an 
increasing influx of patches and newcomers such as Star Lab, Bosch and Netflix. 
We had a significant amount of contributions from cloud providers, software 
vendors, hardware vendors, academic researchers and individuals to help with 
this release. Major contributors for this particular release come from Citrix, 
SUSE, Intel, Star Lab, Oracle, Linaro, Fujitsu, Bitdefender, Red Hat, Huawei, 
ARM, Novetta, Broadcom, Xilinx, Bosch, AMD, GlobalLogic, NSA, Netflix and a 
number of universities and individuals. Thank you to all who participated.

As the release manager, I would like to thank everyone for their contributions 
(either in the form of patches, bug reports or packaging efforts) to the Xen 
Project. This release wouldn’t have happened without contributions from so many 
people around the world. Please check out our4.7 contributor acknowledgement 
page.

The source can be located in the 

  http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.7 
tree (tag RELEASE-4.7.0) 

or can be downloaded as tarball from our website at 

  http://xenproject.org/downloads/xen-archives/xen-47-series/xen-470.html

More information can be found at

* Building Xen 4.7 without Raisin: 
http://wiki.xenproject.org/wiki/Xen_Project_4.7_Release_Notes#Build_without_Raisin
* Building Xen 4.7 with Raisin: 
http://wiki.xenproject.org/wiki/Xen_Project_4.7_Release_Notes#Build_with_Raisin
* Xen 4.7 Release Notes: 
http://wiki.xenproject.org/wiki/Xen_Project_4.7_Release_Notes
* Xen 4.7 Man Pages: http://wiki.xenproject.org/wiki/Xen_Project_4.7_Man_Pages
* All Xen 4.7 related documentation: 
http://wiki.xenproject.org/wiki/Category:Xen_4.7

= Xen Project Hypervisor 4.6.3 =

The Xen Project 4.6.3 release is a maintenance release which comprises bug 
fixes and security updates. This is release is available immediately from its 
git repository

  http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6 
(tag RELEASE-4.6.3) 

or from the Xen Project download page

  http://www.xenproject.org/downloads/xen-archives/xen-46-series/xen-463.html

(where a list of changes can also be found).

We recommend all users of the 4.6 stable series which do not wish to upgrade to 
Xen 4.7, to update to this latest point release.

Note regarding version numbering: an issue was found late in the release 
process, after one of the affected qemu trees was already tagged with a signed 
release git tag. Signed git tags provide a secure way of accounting for the 
source code, but once created they cannot be removed. Thus, the project could 
have released this maintenance release with a known issue, or fix the issue and 
skip a version number. We opted for the latter and decided to skip version 
4.6.2.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel