[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Network performance issues on Qubes OS Server prototype


  • To: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Frédéric Pierret <frederic.pierret@xxxxxxxxxxxx>
  • Date: Mon, 23 Mar 2020 18:36:43 +0100
  • Arc-authentication-results: i=1; mx.zohomail.com; dkim=pass header.i=qubes-os.org; spf=pass smtp.mailfrom=frederic.pierret@xxxxxxxxxxxx; dmarc=pass header.from=<frederic.pierret@xxxxxxxxxxxx> header.from=<frederic.pierret@xxxxxxxxxxxx>
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584985009; h=Content-Type:Cc:Date:From:MIME-Version:Message-ID:Subject:To; bh=D07aIrUpUhoHFQzcJgeP6HisL4IoBMJROhjdZ0dD0wE=; b=Zkr5/DTkPPEBvf6Y9+QoWJjlyT9mJK9ydcEBDom/yCNQt//vuGlNaMrj5z5T+SpBEtrC2DWMdR4EFsYZCN3yBAlAAuni91dWvyQLs8wAVeDsy32b0F7n0uHWDD74SQVUEu6Rosn7eUUvjeeR0u38QCVFwp7DY7Pn6UP7uDWgqmU=
  • Arc-seal: i=1; a=rsa-sha256; t=1584985009; cv=none; d=zohomail.com; s=zohoarc; b=NqP9hGyfyQFvXxeutbhVZfrNorZN9wVW9PuzowSRubcLk+ZA9AQkw6xywT4/Y/gVBrpk7MECg5Vo7MUbsa1VWh55g5G4q8geJO7XTZnLm2YQxw2mXE3Dmy8PveTx0P+LrkQx2xXVCNSaop3HxgyvDfgakQ9boam+mYJtVEYqo64=
  • Cc: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Mon, 23 Mar 2020 17:36:53 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Hi all,

I'm currently working on a Qubes OS server version (example architecture can 
been seen at 
https://raw.githubusercontent.com/fepitre/qubes-mgmt-salt-qubes-server/devel-140320-extra/qubes-server.png).
 I'm using this configuration since several months on Qubes R4.0 (xen-4.8) and 
recently on Qubes R4.1 (xen-4.13). I'm writing to you because since the 
beginning I'm having network performance issues that I never succeeded to solve.

This setup is done on a HP Gen8 DL360p with 2*CPUs, 160GB memory, 1TB RAID6 SAS.

On the picture I linked you, all the colored rectangles {zone}-* for zone in 
(wan, dmz, lan, admin) are PVH VMs (Debian 10). There exist a VM not drawn in 
the picture, called 'sys-net-interfaces' which holds four 1Gbits Ethernet 
controllers of the server using PCI passthrough. It is a HVM with Linux-based 
stubdomain.

All the inner links between VMs are NAT interfaces. All the outer links on 
*-sys-net VMs are BRIDGE interfaces with backend 'sys-net-interfaces'. In VM 
'sys-net-interfaces' a LACP bond0 is done with two Ethernet controllers, which 
is a trunk, then several vlan interfaces are generated with parent device this 
bond, and finally, bridges are created and associated to those vlans.

Here are my issues. Consider one computer named 'PC-LAN' in LAN network and 
another 'PC-DMZ' in DMZ network. The considered network path is the following:

        PC-LAN (1) <-- B --> lan-sys-net (2) <-- N --> lan-sys-firewall (3) <-- 
N --> dmz-sys-firewall (4) <-- N --> dmz-sys-net (5) <-- B --> PC-DMZ (6)

where B denotes bridge interface, N denotes NAT interface and numbers for 
numbering machines. Up to 'wget', 'scp' (limited normally by ciphers), etc., I 
ran multiple iperf3 tests over 20 seconds for having a clearer view of network 
issues.

Example 1: Full path

        From (1) to (6): 165 Mbits/s
        From (2) to (6): 196 Mbits/s
        From (3) to (6): 205 Mbits/s
        From (4) to (6): 203 Mbits/s
        From (5) to (6): 714 Mbits/s


Example 2: 'dmz-sys-net' as end node

        From (1) to (5): 194 Mbits/s
        From (2) to (5): 189 Mbits/s
        From (3) to (5): 258 Mbits/s
        From (4) to (5): 500 Mbits/s

Example 3: 'lan-sys-net' as end node

        From (1) to (2): 830 Mbits/s


I've another HP Gen8 with almost the same physical configuration and network 
configuration (LACP+vlan+bridges) running under Debian 10 as bare metal KVM, 
and I obtain 1Gbits/s network workflows over bridges. The almost physical 
configuration is due to the related mail I sent you in july 2019 '[Xen-devel] 
Ethernet PCI passthrough problem'. The provided Ethernet card with 4 ports (HP 
Ethernet 1Gb 4-port 331FLR Adapter) makes the driver tg3 crashing when 
attaching those into a VM. So the Debian KVM has those HP Ethernet controllers 
whereas on the Qubes server, it has a cheap PCI express 4 Ethernet Realtek 8169 
card.

Of course physical connections on the switches have been changed, 'switched' 
between the two servers for eliminating any hardware problem.

I had a look to 
https://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide. 
Unfortunately, trying some change of options with 'ethtool' in 
'sys-net-interfaces', changing amount of RAM/VCPUs of it and other *-sys-net, 
does not do that much.

I'm writing to you for having some clues into where I can dig and what I can 
look in order to put in evidence the bottleneck. If it's purely dom0 side or 
backend network VM side (sys-net-interfaces) or elsewhere.

I would like to thank you a lot in advance for any help on this problem.

Best regards,
Frédéric

Attachment: signature.asc
Description: OpenPGP digital signature


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.