RE: RE: [Xen-users] VLAN and BRIDGE HELP

I have refined my howto a bit. I fixed a few typos and had a few things in the wrong order.

Although they work, I still have a few questions on these steps.

1) Do we need to create a VIF on DOM0 in order to set up the management interface? I am starting to think not.

2) When setting up the VLAN should I connect to the BOND (as I indicated below) or to the network that initially gets connected to the bond?

Any help on these would be great.

Thanks;

James

While I am still in the process of working through this here are the steps I am taking to set up each machine.

1) Since each of my servers have multiple NIC ports I bond them together (see page 34 of the Administrators Guide) This is best done from the physical server and not via the console. If you do it remotely you will then have to reconfigure the management port before the next step. I think it is cleaner if you do not have to do that.

a. Shutdown all VM’s (this is easy since I am pretty much a new installation)

i. xe vm-list

ii. xe vm-shutdown uuid=

b. Create the network (this is like a virtual switch), write down the first part of the UUID that is returned after this command since the next step may cause it to scroll off the page

i. xe network-create name-label=bond0

c. Create the actual bond (keep track of the uuid of this bond since it will be used in step 2)

i. xe pif-list

ii. xe bond-create network-uuid=<UUID from above> pif-uuids=<UUID of the first interface from the last step>,<UUID of the second interface from the last step>

2) Next for the VLAN work. First, of course, each NIC port must be on a trunked switch port. IN our case, we forced the port encapsulation to dot1q and the mode to trunk with a native VLAN of 1. We did this using ranges so that we know everything is configured the same. I am not sure if allowing the ports to negotiate the encapsulation or making the trunk mode dynamic would have worked but, we were not taking any chances. This way we had two fewer things to troubleshoot if we had problems.

a. Create a new network. This will be like a VLAN specific switch for all of your DOM’s

i. xe network-create name-label=vlan103

b. Next tie the network to the network interface on the physical server (in our case, the bonded interfaces) and specify the VLAN. Keep track of the UUID returned as you will need it when we create VIF’s for the DOM’s

i. xe vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the bond in step 1.c.ii> vlan=103

c. Reconfigure the IP address info. I do not use DHCP for my servers so I make this static. Also, I do not know for sure how to enter multiple DNS servers. You may just separate them with a comma but I have not tested that yet. Finally, the case seems to be important here. I noticed it on the DNS keyword. If it is lowercase it does not seem to work.

i. xe pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK> DNS=<DNS SERVER ADDRESS>

3) The next step is to get the management port back up and running. I am not sure if you have to actually set up a VIF (steps a through d. Step e may be enough.) These are the same steps for setting up VIF’s on other VM’s.

a. get the UUID of DOM0

i. xe vm-list

b. Create a VIF to tie DOM0 to the management VLAN (this is the VLAN I created above). In this case I had one interface (BOND0) so I set the device to 1. Keep track of this UUID as you will need it to configure the management interface.

i. xe vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network created just above in 2.b.i> device=1

c. As long as a VM is not running, you can just start it and the new interface will be ready. For DOM0 we will have to plug it in or reboot the server.

i. xe vif-plug uuid=<UUID of the VIF created above>

d. Configure IP information of the VIF. First run ifconfig to get the device and then run it again to configure it

i. ifconfig

ii. ifconfig xapi2 <IP ADDRESS> netmask <NETMASK>

e. Configure the VLAN PIF as the management interface

i. xe host-management-reconfigure pif-uuid=<UUID of the PIF created above 2.b.i>

f. Clean up after yourself. This will help to keep you from getting confused later when you look at the settings.

i. xe pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None

Notes

1) The XenNetworking WiKi page helped me wrap my head around what was going on. http://wiki.xensource.com/xenwiki/XenNetworking

2) I found this thread on the forums that basically became my template. http://forums.xensource.com/thread.jspa?messageID=15451&#15451

3) The above thread led me to the knowledge that there is more about VLANS in the admin guide than the index would have you believe. Page 33 – 34 tells you how to set them up. I hope that this omission can be fixed in the next document release.

4) I found that the UUID’s work with tab completion. This may be obvious to everyone else but I never saw it mentioned in the manual (it may be there but I never saw it).

5) One problem I had with configuring bridges manually in Linux instead of doing it the Xensource way above, was that Xensource automatically removes interfaces it does not know about. Since I was not sure how to make it aware of things I had manually created, my interfaces and bridges would stay up for 30 – 60 seconds or so before being torn down.

6) Once you get the management interface setup, you can create links to other VLANS using the XenCenter.

7) I have not tested yet but, my hope is that this information will travel to each of the other machines I add to the resource pool. That is why I took the above steps on the master server.

8) If you dig through this entire thread you will find a nugget of information that sounds like it could cause problems if set incorrectly. “The secret seems to be that the native VLAN on the trunkport must be 1 (which is the default).” http://forums.xensource.com/thread.jspa?messageID=15780

9) Before you can take all of the above steps you must install your license. This presents a Catch-22 since you would normally install it using the XenCenter, however, this requires the network already be working. To install via the command line you need to:

1. copy the license to a thumbdrive, take it to the server and mount it.

2. mv /etc/xensource/license /etc/xensource/license.old

3. copy the license from the thumb drive to /etc/xensource/license

4. If necessary, chmod a+x /etc/xensource/license to make the permissions match the original file.

5. reboot the box

James Alspach

Systems Analyst II
Shasta County Office of Education

From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of James Alspach
Sent: Monday, June 09, 2008 3:37 PM
To: augusto lopes
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: RE: [Xen-users] VLAN and BRIDGE HELP

You are very welcome.

One thing that I am working on now involves removing the step toward the beginning:

xe pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK> DNS=<DNS SERVER ADDRESS>

I do not believe that this step is necessary since we will not use the pif directly. We will use vif’s connected to this pif (actually to the network connected to the pif) and the vif’s will each get their own IP addresses.
I am working on this right now and will let you know how it goes.

Thanks;

James

James Alspach

Systems Analyst II
Shasta County Office of Education

From: augusto lopes [mailto:nhanonme@xxxxxxxxxxxx]
Sent: Monday, June 09, 2008 3:29 PM
To: James Alspach
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: RE: [Xen-users] VLAN and BRIDGE HELP

Thank you VERY, VERY MUCH JAMES!
This is a great detail oriented document that answers all my noobie's doubts: now I should be able to set without issues. I will let you know how it goes for me, but I truly have a good feeling about it now that you provided me with this feed...

Thanks again for you help,

Augusto Lopes
Intern Systems Admin
Apollo System

James Alspach <jalspach@xxxxxxxxxxxxx> escreveu:

While I am still in the process of working through this here are the steps I am taking to set up each machine.

Since each of my servers have multiple NIC ports I bond them together (see page 34 of the Administrators Guide) first. This is best down from the physical server and not via the remote console. If you do it remotely you will then have to reconfigure the management port before the next step. I think it is cleaner if you do not have to do that.

Shutdown all VM’s (this is easy since I am pretty much a new installation)

xe vm-list

xe vm-shutdown uuid=

Create the network (this is like a virtual switch), write down the first part of the UUID that is returned after this command since the next step may cause it to scroll off the page

xe network-create name-label=bond0

Create the actual bond (keep track of the uuid of this bond since it will be used in step 2)

xe pif-list

xe bond-create network-uuid=<UUID from above> pif-uuids=<UUID if the first interface from the last step>,<UUID if the second interface from the last step>

Reconfigure the IP address info. I do not use DHCP for my servers so I make this static. Also, I do not know for sure how to enter multiple DNS servers. You may just separate them with a comma but I have not tested that yet. Finally, the case seems to be important here. I noticed it on the DNS keyword. If it is lowercase it does not seem to work.

xe pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK> DNS=<DNS SERVER ADDRESS>

Next for the VLAN work (see page 33 of the Administrators Guide). First, of course, each NIC port must be on a trunked switch port. IN our case, we forced the port encapsulation to dot1q and the mode to trunk with a native VLAN of 1. We did this using ranges so that we know everything is configured the same. I am not sure if allowing the ports to negotiate the encapsulation or making the trunk mode dynamic would have worked but, we were not taking any chances. This way we had two fewer things to troubleshoot if we had problems.

Create a new network. This will be like a VLAN specific switch for all of your DOM’s

xe network-create name-label=vlan103

Next tie the network to the network interface on the physical server (in our case, the bonded interfaces) and specify the VLAN. Keep track of the UUID returned as you will need it when we create VIF’s for the DOM’s

xe vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the bond in step 1.c.ii> vlan=103

The next step is to get the management port back up and running. These are the same steps for setting up VIF’s on other VM’s.

Get the UUID of DOM0

xe vm-list

Create a VIF to tie DOM0 to the management VLAN (this is the VLAN I created above). In this case I had one interface (BOND0) so I set the device to 1. Keep track of this UUID as you will need it to configure the management interface.

xe vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network created just above in 2.b.i> device=1

Configure IP information of the VIF. First run ifconfig to get the device name and then run it again to configure it

ifconfig

ifconfig eth1 <IP ADDRESS> netmask <NETMASK>

Configure it as the management interface

xe host-management-reconfigure pif-uuid=<UUID of the VIF created above>

Clean up after yourself. This will help to keep you from getting confused later when you look at the settings.

xe pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None

As long as a VM is not running, you can just start it and the new interface will be ready. For DOM0 we will have to plug it in.

xe vif-plug uuid=<UUID of the VIF created above>

Notes

The XenNetworking WiKi page helped me wrap my head around what was going on. http://wiki.xensource.com/xenwiki/XenNetworking

I found this thread on the forums that basically became my template. http://forums.xensource.com/thread.jspa?messageID=15451&#15451

The above thread led me to the knowledge that there is more about VLANS in the admin guide than the index would have you believe. Page 33 – 34 tells you how to set them up. I hope that this omission can be fixed in the next document release.

I found that the UUID’s work with tab completion from the command line. This may be obvious to everyone else but I never saw it mentioned in the manual (it may be there but I never saw it).

One problem I had with configuring bridges manually in Linux instead of doing it the Xensource way above was that Xensource automatically removes interfaces it does not know about. Since I was not sure how to make it aware of things I had manually created, my interfaces and bridges would stay up and running for 30 – 60 seconds or so before being torn down.

Once you get the management interface setup, you can create links to other VLANS using the XenCenter and skip much of the above.

I have not tested yet but, my hope is that this information will travel to each of the other machines I add to the resource pool. That is why I took the above steps on the master server.

I also attached the above info as a PDF. If anyone sees any problems with what I have described or better ways to go about it, please let me know so that I can update this in the hope that it helps somebody somewhere.

James Alspach

P.S. Thank you for everyone who provided suggestions and help while I was (and still am) trying to figure this all out.

James Alspach

Systems Analyst II
Shasta County Office of Education

From: augusto lopes [mailto:nhanonme@xxxxxxxxxxxx]
Sent: Monday, June 09, 2008 9:17 AM
To: James Alspach
Subject: En: RE: [Xen-users] VLAN and BRIDGE HELP

Hello James;

I was just checking all the recent help emails and found this particular one which is basically referring to a similar environment I would like to set up. Basically, in my scenario I am asked to set up three guest domains (domU1 - domU3). The first tow will provide web and mail services sequentially. And the last one will basically be the DBM server.
Since in a normal network environment domU3 should be on a protected subnet and the other tow on a dmz subnet, I have not been able to set up VLANs appropriately for them. I have practically set up netfilter (iptables firewall ) on the Dom0, but do not understand VLAN concept in xen's virtual environment well enough to accomplish the main goal of separating each service to its own domain.

After reading your help, I can picture the all thing better, but I am still not sure how to bring it all about. Would you please give me some ropes regarding this topic?

I am working with RHEL5 as the Dom0 and guest domains will be various win server 2k3 as well as rhel5.

Thank you in advance for you help.

Augusto Lopes
Intern System Admin
Apollo System

James Alspach <jalspach@xxxxxxxxxxxxx> escreveu:

Data: Thu, 5 Jun 2008 09:24:04 -0700
De: "James Alspach" <jalspach@xxxxxxxxxxxxx>
Para: "Emil Palm" <empa@xxxxxxxxxx>,
<xen-users@xxxxxxxxxxxxxxxxxxx>
Assunto: RE: [Xen-users] VLAN help

Emil –

Thank you! This gives me another avenue to explore.

So what you are saying is that one does not have to explicitly create the new interface on the VLAN by making changes in /etc/sysconfig/network-scripts/ ? By configuring the vlan in vconfig it builds the new interface automagically? Will this setup survive a reboot or should I build a script to set this up each time?

In my case I have bonded two NICs (well actually two ports on the same NIC but no need to split hairs) so I would imagine I would replace eth0 in your example with bond0.

The host OS I am using is, I believe, CentOS. I am using the Xen Enterprise version installed right off of the CD which uses some RedHat derived distro.

Thank you for your help

James

James Alspach

Systems Analyst II
Shasta County Office of Education

From: Emil Palm [mailto:empa@xxxxxxxxxx]
Sent: Thursday, June 05, 2008 12:35 AM
To: James Alspach; xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-users] VLAN help

Hi!

I'v just recently set that kind of eviroment up. What Host OS are you running becouse i wrote 2 diffrent howtos for Red Hat and one for SuSe.

But in theory you just have 1 interface for ex: eth0

then you do vconfig add eth0 VLANID

when that is done you create a bridge something like this:
brctl addbr brVLANID

brctl addif brVLANID eth0.VLANDID

when that is setup you should change "network-script=network-bridge" to "network-script=network-dummy" within your xend-config.sxp so Xend doesnt screw up your real physical interface.

When that is done just put: vif=["mac=XX:XX:XX:XX:XX:XX,bridge=brVLANID",] in your domU configuration file.

If you want more info just give me a mail and i will help you as the best as i can.

Emil Palm

Cardium AB

Sweden

-----Original message-----
From: James Alspach <jalspach@xxxxxxxxxxxxx>
Sent: Wed 06/04/08 19:56:40
To: xen-users@xxxxxxxxxxxxxxxxxxx;
Subject: [Xen-users] VLAN help

We are in the process of setting up a few Xensource servers whose initial function will be to run Exchange 2007. As part of this (and for future VM’s) I need to be able to provide access to various VLANS to the various DOM’s.

In theory this sounds fairly straight forward: DOM0 gets a PIF for each VLAN. This PIF connects to a VLAN specific bridge and then, for each DOM that needs one, a VIF is created and connected to the bridge.

Does this sound correct?

If so, my question is how to specify the VLAN for a PIF. I can list it but I am not able to set it since it is read only.

How do virtual networks fit into the above and how is a virtual network different from a virtual bridge?

Any help or pointers to information are greatly appreciated.

Thank you for your help;

James

James Alspach
Systems Analyst II

Shasta County Office of Education

1644 Magnolia avenue

Redding, California

96003
jalspach@xxxxxxxxxxxxx
(530) 225-0293

IT Hotline: 225-0279

hotline@xxxxxxxxxxxxx

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

Abra sua conta no Yahoo! Mail, o único sem limite de espaço para armazenamento!

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

Abra sua conta no Yahoo! Mail, o único sem limite de espaço para armazenamento!

xensource networking.pdf
Description: xensource networking.pdf

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

WARNING - OLD ARCHIVES

xen-users

RE: RE: [Xen-users] VLAN and BRIDGE HELP