This is an update and bit of a rewrite of an earlier post I made here. I wanted to add more detail and information now that I’ve worked with the Nexus 1000v more.
If you’ve read a lot of the Varrow blogs you’ll see information and talk about Cisco’s Nexus technology and products. To be blunt, it can be confusing and a bit convoluted. The hardware products, the Nexus 5000 and 7000, have been out for a little while now and we’re seeing more and more interest in those as companies see the need for high speed connectivity and the benefits of FCoE, especially now that FCoE is a standard. But I think the Nexus 1000v is still a mystery to a lot of people. There is a lot of “It’s really cool!” information out there but not a lot on how it really works. One thing I’ve found is that the concept of the 1000v is very nebulous to many customers. I can white board out how it works, the pieces, how they talk but it’s hard to “get it” without seeing it. For that reason I created a nice demonstration video that is here. If you’re new to the 1000v I highly suggest you check it out and see if it makes things click a little better for you.
Now let’s get in to some specifics.
The New Distributed Switch
Before getting in to the 1000v you first need to understand the new distributed switch in vSphere. The Nexus 1000v uses this framework to provide much of its functionality. In fact, from within vCenter the 1000v looks very much like the standard shipping distributed switch (dvSwitch). But, while you can make changes to the dvSwitch in the vCenter GUI you can’t do that with the 1000v. All changes to that must be done via the Nexus-OS command-line environment. The idea behind the distributed switch is you now have one place to do the configuration and management of the network connectivity for your entire ESX cluster. In the past you had to manually create vSwitches and Port Groups on every ESX server you brought up in the cluster. With the distributed switch you configure your Port Groups just like you want in vCenter. When a new ESX server moves in to the cluster and is joined to the dvSwitch (distributed virtual switch) it automatically sees the configuration in place. It’s really great. I’d say network configuration is the hardest part of the install for most VMware administrators and often a point of contention between the server admins and the network admins..maybe almost as bad as the contention between server guys and storage guys!
The benefit for organizations with separate teams, such as network and server admins, is that this moves “the line” back. Before we virtualized everything the demarcation point between the server and the network was pretty simple, it was at the switch port. Now that we’ve virtualized that has moved and sits somewhere in the ESX server so now a lot of that responsibility falls, correctly or not, on to the server admin. They are the ones configuring networking and networking policies. While the Nexus 1000v doesn’t move the line back to the physical switch port it gives the network team a virtual switch in the cluster that looks, feels, and acts like a hardware Cisco switch just like they are used to managing.
Components of the Nexus 1000v
When deploying the 1000v there are a few moving parts, not a lot but a couple. The two primary pieces are the Virtual Supervisor Module (VSM) and the Virtual Ethernet Module (VEM). One thing that helps a lot of people is to imagine the 1000v like a multi-slot chassis switch such as a 6509 or a 4507R. The VSMs are the supervisor management modules and the VEMs are the I/O port blades that provide connectivity. The 1000v can have up to two VSMs and 64 VEMs which equates to a 66-slot chassis switch. That’s a big switch!
So let’s look at the components in a bit more detail. First is the VSM as it is the central management authority.
Virtual Supervisor Module
The VSM is a virtual version of a hardware supervisor module. Some switches have redundant modules and the Nexus 1000v is no different. The VSM runs as a virtual machine on an ESX server in the cluster. To provide fault tolerance you can run a second VSM in a standby role. The secondary VSM will take over if the primary should fail. Like a physical Supervisor Module there really isn’t any extra maintenance or management needed to run the second one. Any configuration change on the primary is automatically replicated to the secondary. So…I don’t see why you wouldn’t run two. Also like many physical switches the supervisor modules do not have stateful failover, meaning they don’t share current information. When the primary supervisor fails the secondary reboots, reads in the configuration, and starts working.
It’s important to note that the VSM is not in the data path. That means it does not stop data flow through the cluster if the VSM should go down. You won’t be able to make management changes but your VMs will continue to talk. As of the latest version, 4.0(4)SV1(1), you can now vMotion the VSMs around as long as it’s on an ESX server with a VEM installed that the VSM itself manages. Just be sure to exclude it from DRS so you know when it moves!
Virtual Ethernet Module
This is where it gets really cool. On each vSphere server in the cluster you install the Nexus 1000v VEM. This is a piece of software that ties that server in to the distributed switch of the Nexus 1000v. When you install and attach the VEM to the VSM for the cluster that ESX server’s VEM appears as a module on the switch. So just like you log in to a Cisco chassis switch and do a “show module” you’ll do the same here. Each ESX server will be its own module. And that’s why it’s called a Virtual Ethernet Module. There is an example of the “show module” output below.
How the Modules Communicate
So you have one or more VSMs installed as the brains of the switch and you also have some VEMs installed on ESX servers to act as the access port modules. How do they talk to each other? This is an important thing to understand as you need to get this before you start rolling out your VSMs. Basically the Nexus 1000v uses a couple of VLANs as layer 2 communication channels. These are the control and packet VLANs. Their purpose is:
- The Packet VLAN is used by protocols such as CDP, LACP, and IGMP.
The Control VLAN is used for the following:
- VSM configuration commands to each VEM, and their responses.
- VEM NetFlow exports are sent to the VSM, where they are then forwarded to a NetFlow Collector.
- VEM notifications to the VSM, for example a VEM notifies the VSM of the attachment or detachment of ports to the distributed virtual switch (DVS).
Cisco recommends that the Control VLAN and Packet VLAN be separate VLANs; and that they also be on separate VLANs from those that carry data. In the original version of the 1000v the VSMs and VEMs had to be on the same layer 2 network. As of version 1.2 you can have layer 3 connectivity from the VSMs to the VEMs. Just be sure all of the VEMs that are managed by these VSMs can talk via layer 3. To put it simply, the VSM can be on another network but all the VEMs it manages must be together. With Cisco releasing a hardware VSM appliance you’ll see more use cases for this functionality. The vCenter server can be on a different layer 3 network than the VSM, that doesn’t matter. Before you start deploying VSMs and VEMs you need to decide which VLANs you plan to use for packet and control and then create them on your physical switches that connect ESX servers. To reiterate, the Control and Packet VLANs are for layer 2 communication. This means that no IP addresses or default gateway needs to be assigned to these VLANs so just pick any unused VLAN numbers and get them going.
Deploying the Components
This particular post is not meant to be an in-depth guide to installing the 1000v. Cisco offers their documentation here. The point of this post is to give you a deeper understanding of how everything fits together so you better understand the function of the 1000v and how it integrates in to your environment.
Deploying the VSM is very simple. It’s distributed as a ..ova/ovf file set and is easily added to the cluster via the vCenter client. Version 1.2 and above make installation even easier than the original version. On the original you just imported the .ovf and booted the VSM. Once booted you were presented the familiar “basic setup” Cisco walkthrough. As of 1.2 you import the VSM appliance using a .ova file which walks you through a GUI wizard to configure those same items. It just makes it simpler and faster than before. In the wizard you’ll be asked for the packet and control VLANs as well as a “domain ID”. This just defines the VSMs and VEMs that are working together. Pick an unused number, I start with 1, and continue. If you have multiple 1000v installations you’ll need to assign unique Domain IDs to each one.
Installing the VEMs isn’t much more difficult. You have the option of doing it manually on each ESX server or using Update Manager. The choice is yours. Manually is easy as it’s a single command once you get the package file on the server. There is no manual configuration needed on each VEM. Everything will come from the VSM or vCenter. If you scp the .vib file to the ESX server all you need to type to install it is:
esxupdate -b *.vib update
Once that is done you can check the installation by using the “vem status” command, such as:
[root@labclt-esx01 ~]# vem status
VEM modules are loaded
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 32 10 32 1500 vmnic0
DVS Name Num Ports Used Ports Configured Ports Uplinks
nexus1kv 256 51 256 vmnic1
VEM Agent (vemdpa) is running
That’s it. Nothing else to do on each system. Going forward I recommend updating the VEMs using Update Manager. You can just add a repository in to Update Manager to get patches for the VEMs. Makes it very easy. The VSMs take a bit more work to update…but we’ll cover that in another post soon.
The final piece is some configuration in vCenter. You have to install the Nexus 1000v plugin as well as perform some configuration on the VSM to connect it to vCenter. This is easily done by opening a web browser and pointing it to the IP of the VSM you installed. From there you can install the plugin, which is just a simple XML file to allow communication between the VSMs and vCenter. When that happens your new distributed switch appears and the real fun begins….
Beyond Installation
Assuming everything went well and all the pieces are talking you are ready for configuration. The first step is to bring each ESX host in the cluster and assign some unused NICs to uplink ports. You can kind of think of uplinks like vSwitches in the normal vSwitch that we are all used to. Basically, an uplink defines which traffic goes over which physical NICs. These are how you separate traffic. So if you want vMotion over one set of NICs and VM Network traffic over another set you’d create two uplinks and assign the appropriate NICs to each. Here is an example config for an uplink:
port-profile type ethernet system-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 20-30
no shutdown
system vlan 10-12
state enabled
This is a very simple uplink called system-uplink. It is allowed to carry all VLANs so all traffic in all port-groups will flow over any NICs assigned to this uplink. Notice the “system vlan” option. This specifies that the special VLANs such as Control and Packet can flow over this uplink. You need to do this if you’re going to move those to the dvSwitch, which you’ll probably eventually do. One thing worth mentioning is the “switchport trunk allowed vlan” command. This tells the switch which VLANs can ride of this uplink. This is how you relate port-groups to uplinks. So a port-group used for VM traffic on VLAN 21 will ride over this uplink and these NICs. If another port-group is used for VM traffic or a service console on VLAN 40 it will not go over this uplink and you’d need to define another uplink that could service that VLAN. If you assign more than one they must be in a channel group. The Nexus 1000v does not run spanning tree and therefore doesn’t like loops and will disable the connections if it sees duplicates.
If you’re using 10Gb CNA adapters you may only have two ports on the server which means, usually, everything will ride over a single uplink with redundant connections. If you’re concerned about bandwidth problems due to heavy traffic features such as vMotion or FT you can use QoS to alleviate that. If you’re in a legacy type configuration with 6 or 8 Gb connections you’ll probably have multiple uplinks just like you had multiple vSwitches before the move to the dvSwitch. Some of the best practices we’ve had for a while still apply here, they may just be implemented a little differently.
Now that all hosts are attached to the distributed switch you can start creating port groups. Creating port groups is done from the VSM via the Cisco NX-OS command-line. NX-OS is the Nexus OS and is very similar to IOS. Anyone used to IOS should feel at home and will find many nice enhancements. Within NX-OS you create Port Profiles, which turn in to port groups within vCenter and the distributed switch. Below is an example configuration for a Port Profile used to let VMs talk to the network.
port-profile VM_VLAN300
vmware port-group
switchport mode access
switchport access vlan 300
no shutdown
state enabled
In many ways it is similar to a port configuration on a physical switch. This creates a port group in vCenter named VM_VLAN300 that lets VMs talk to other devices on VLAN 300. Below is a screenshot showing this port group available in the settings for a VM.
Behind the scenes the VMs and port groups act like Cisco devices. As mentioned before, each ESX server is like a switch module. For example:
vc4labsw# show mod
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 0 Virtual Supervisor Module Nexus1000V active *
3 248 Virtual Ethernet Module NA ok
4 248 Virtual Ethernet Module NA ok
Mod Server-IP Server-UUID Server-Name
--- --------------- ------------------------------------ --------------------
1 10.13.7.49 NA NA
3 10.13.7.53 33393935-3234-5553-4539-32344e36464b vc4lab1.varrow.com
4 10.13.7.52 33393935-3234-5553-4539-32344e36464c vc4lab2.varrow.com
From that you can see there are two VEMs installed. One on vc4lab1 and the other on vc4lab2, my two vSphere servers in this lab. When a new connection is made to the dvswitch, whether it’s a VM or something like a Service Console, that connection is given a Virtual Ethernet interface or port assignment. The connection will always keep the same port number no matter where it lives in the cluster. Example again:
vc4labsw# show int bri
--------------------------------------------------------------------------------
Interface VLAN Type Mode Status Reason MTU
--------------------------------------------------------------------------------
Veth1 300 virt access up none 1500
Veth2 300 virt access up none 1500
Veth3 1 virt access up none 1500
Veth4 1 virt access up none 1500
Veth5 400 virt access up none 1500
Veth6 400 virt access up none 1500
vc4labsw# show int ve3
Vethernet3 is up
Port description is JN_Win2k8, Network Adapter 1
Hardware is Virtual, address is 0050.568a.31cf
Owner is VM "JN_Win2k8", adapter is Network Adapter 1
Active on module 3
VMware DVS port 352
Port-Profile is VM_VLAN1
Port mode is access
Rx
13616 Input Packets 13220 Unicast Packets
23 Multicast Packets 373 Broadcast Packets
1039898 Bytes
Tx
198323 Output Packets 87263 Unicast Packets
13016 Multicast Packets 98044 Broadcast Packets 19 Flood Packets
80092259 Bytes
8 Input Packet Drops 0 Output Packet Drops
In the example I performed a “show int brief” to get a concise list. Notice it shows the interface and VLAN but not the type of connection. If I narrow it down and look at ve3, Virtual Ethernet 3, I see that this is my Windows 2008 server named JN_Win2K8. Notice the information displayed. Very similar to a port on a physical switch. No matter where this machine goes those statistics and information follow it.
Conclusion
There isa lot of good information out there on the Nexus 1000v now. Cisco has a very good community site here. If you’re starting out with the new virtual switch hopefully this post was useful and filled in some of the gaps. Take a look at the video referenced above as well as the documentation from Cisco. Cisco has very good documentation and install guides that will walk you through a normal deployment. The key is to extrapolate what you need from those documents and fit it in to your environment.
