vSwitch Load Balancing – Stop Getting it Wrong!
OK, so this is one of my pet-hates.
The subject of Network Load Balancing comes up all the time when I’m visiting clients and it’s always a cause of some pain.
James’ article on Spotting The Red Flags made me think about what triggers the alarm bells ringing in the network world.
For me, if often starts with the Switch configuration, when I see something like this:
interface GigabitEthernet0/1
description ESX Server NIC0
switchport mode access
channel-group 1 mode on
Some of you might be thinking, well, there’s nothing wrong with that. In fact you’d be right. “Technically” there’s nothing wrong with that IF the vSwitch on the ESX Server is using the right load-balancing method.
BUT, it’s a red flag, because 9 times out of 10, the vSwitch configuration is using the default ‘Route based on originating port ID’ method.
And listen closely folks, as I’m not going to say it again, this method DOES NOT REQUIRE SWTICH CONFIGURATION. (Sorry for shouting, but it really, really bugs me).
There is only one ESX vSwitch load-balancing method which requires switch-side configuration and that’s ‘Route based on IP hash’.
From page 7 of the ESX Configuration Guide:
“NOTE: IP-based teaming requires that the physical switch be configured with etherchannel. For all other options, etherchannel should be disabled.”
I get even more scared when I see this:
interface GigabitEthernet0/1
description ESX Server NIC0
switchport mode access
channel-group 1 mode [auto|desirable|active|passive]
This is because any of those channel-group modes imply the use of PAgP or LACP. Let’s put an end to this debate once and for all – “VMware ESX Server does not support dynamic PAgP or LCAP.”
These dynamic link aggregation protocol are absolutely not supported on ESX, so don’t use them on the switch-side!
If you’re an ESX Admin or a Network Admin, please get together and make sure that you’re both on the same page. Time and time again, we find that there’s a disconnect between what each team thinks the other is doing.
I would strongly encourage everyone to read Ken Clien’s series of Blogs called ‘The Great vSwitch Debate‘. Especially if you’re new to the world of VMware and ESX. Part 3 focuses on the different Load Balancing methods, so check it out.
For the Network folks, this is a great reference:
VMware Infrastructure 3 in a Cisco Network Environment
For the ESX folks, these are also handy:
ESX Server host requirements for link aggregation
Sample configuration of EtherChannel / Link aggregation with ESX 3.x and Cisco/HP switches
Have a read, build some understanding and stop getting it wrong! (Please…)
It’s the Bloody Defaults
If you’re a DBA having problems with Oracle Data Guard dropping connections for no apparent reason, the first point of call is usually the Network Team.
Especially if you’re seeing errors like this:
“RFS[7]: Possible network disconnect with primary database
Dataguard log shipping is failing”
The Network Team will hopefully do all the investigations they can. Checking:
- ICMP ping repsonses
- ICMP trace routes
- Routing
- Interface speed/duplex mismatches on the hosts, switches, routers, firewall
They’ll then hopefully start looking at tcpdumps or packet captures from switches.
Then they’ll scratch their heads and go, “I can’t see anything wrong…”
Both teams will then start hypothesising on more convoluted possibilities. Is it an MTU issues? Is the traffic going over the Internet / VPN and it’s ‘just the way it is’?
…No
…We’re all guilty.
…We should all be fired.
The first thing to ask yourself is this, “Does the traffic pass through a Firewall?”
Then ask, “Is it a Cisco Firewall?”
If the answer is yes to both, then the place to start looking is Application Inspection. By default, both the Cisco PIX and Cisco ASA have SQL*Net Application Inspection turned on. This is great for SQL*Net traffic, but the problem is that Oracle Data Guard uses the same TCP Port – 1521
So that means that Data Guard traffic is also subject to the same Inspection and time and time again, this causes issues.
Look for this in your PIX 6.x configs:
fixup protocol sqlnet 1521
Or this in your PIX/ASA 7.x or 8.x configs:
policy-map global_policy
class inspection_default
inspect sqlnet
If you see either of these, the raise the red flag.
Now, you could disable the Inspection, but this is usually a global change. So the simple answer:
Always run Oracle Data Guard on a different TCP port
Life should then be good again.
Sometimes I really hate the Cisco defaults…
ASA Quick’n'Dirty Web Filtering
A subject that comes up again and again with some of our smaller clients is that of Web Filtering.
Whilst there are a whole host of solutions out there, the client’s requirements are often very straightforward:
“I don’t want my staff wasting time on the Internet!”
I’m sure we could all spend hours (or days) debating the pros and cons of allowing unrestricted Internet access. But, let’s be honest, we’ve all spent some of our work hours browsing when we should have been working (sorry James!).
Whilst researching the options, I came across this excellent article by René Jorissen:
Cisco ASA: DNS Reply Filtering
I love this solution.
It tackles the problem at the right point in the stack; the beginning, with the initial DNS request.
If all you’re looking to do is stop users (and it would be all users) from accessing specific sites or services, then this could be the solution for you. By dropping DNS replies for the specific sites, you knee-cap the connection at the start.
Yes, it’s not perfect.
There’s no flexibility in relation to specific users, groups, machines or times of day. But that’s not the point. This is a simple solution to meet a simple requirement.
So when a client comes to you and says, “I want to block that bloody [insert current social networking fad of the month] site,” you can now make it happen.
No cost, no fuss.
I like these kind of solutions…
VMware Fusion 2.0.6 Released
Just a quick one.
VMware have released the latest update for Fusion:
This is free to existing licensed users.
You can find the Release Notes here: http://www.vmware.com/support/fusion2/doc/releasenotes_fusion_206.html
- Fixes multiple issues when running VMware Fusion 2.0.x on Mac OS X Snow Leopard (32-bit kernel mode)
- Provides improved 3D performance on Macs with NVIDIA graphics cards running Mac OS X 10.6
- Contains fixes for more than 20 bugs
So, if you’re a OS X Snow Leopard user (like myself) check it out!
State of the Nation
For me, one of the important aspects of managing a VMWare environment, is to quickly be able to ’see’ what’s happening and react accordingly.
Specifically, I want to know what my Guest VMs are up to. Luckily ESX provides with this information in the form of ‘State Transition’ events.
In ESX 4 these are logged to the hostd log (/var/log/vmware/hostd.log). They look a bit like this:
[2009-09-28 11:22:02.048 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_ON -> VM_STATE_OFF)
[2009-09-28 11:22:02.340 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_RECONFIGURING)
[2009-09-28 11:22:02.554 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_OFF)
[2009-09-28 11:22:07.670 F5A6DB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_RECONFIGURING)
[2009-09-28 11:22:07.954 F5A6DB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_OFF)
This is a ‘power off’ sequence for shutting down a VM Guest.
Or more fun things like:
[2009-09-28 11:18:26.399 F6450B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_ON -> VM_STATE_EMIGRATING)
[2009-09-28 11:18:52.732 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_EMIGRATING -> VM_STATE_OFF)
[2009-09-28 11:18:52.744 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_UNREGISTERING)
[2009-09-28 11:18:53.034 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_UNREGISTERING -> VM_STATE_GONE)
Which was a VMotion event from the perspective of the ESX server that was the original host. (I love the terminology by the way – ‘EMIGRATING’ / ‘GONE’).
Now, connecting to your ESX Servers and searching for these specific events would be a bit tedious. So one approach is to send them all to a remote Syslog server.
There are already a number of articles on the web on how to set-up remote sysloging for ESX, so I wont re-invent the wheel. Here’s what worked for me:
- SSH to your ESX Server
- Edit the end of the /etc/syslog.conf file to include the line
- *.* @[Your remote syslog server name / IP]
- Restart the Syslog service using the command:
- /etc/init.d syslog restart
- Allow outbound Syslog through the ESX Firewall
- esxcfg-firewall -o 514,udp,out,syslog
- Restart the ESX Firewall
- esxcfg-firewall -l
This should be enough to get the basic syslog information going where you want it to. But, it wont give you information from the /var/log/vmware/hostd.log
To get this, I did the following:
- SSH to your ESX Server
- Edit the file /etc/vmware/hostd/config.xml to look like this:
<log>
<directory>/var/log/vmware/</directory>
<name>hostd</name>
<outputToConsole>true</outputToConsole>
<level>info</level>
</log>
- Restart the ESX Management Agents with this command:
- /etc/init.d/mgmt-vmware restart
Pay close attention to the ‘level’ tag. By default it reads ‘verbose’. I tried this first and managed to generate about 4000 messages a minute. This was a bit more than I needed…
So what next?
Well, that’s really up to you. Now that you can capture these events, use your favourite Syslog server to either alert or report on whatever you’re interested in.
This sort of information is vital in environments where there are multiple teams administering the VMWare infrastructure. It’s not always easy to know what your colleagues are working on, so capturing this information can provide a quick oversight so that everyone is kept in the picture.
For example, once you’ve got the right Transition event, you can map these to a timeline, to very quickly see what’s happing to your VMs:
Or summarise them into a table:
Or make the names on the table a little more readable:
Once you’ve captured the information, it’s really up to you!
And if your Syslog server doesn’t let you produce reports like the ones above, then you need to start Splunking…
G.
(More information on how to use the exceptional Splunk Server in the next few weeks)
Also, check out these links for a bit more depth on how to configure the Syslog component in VMWare:
Move to the Ghetto
So,
After ‘playing’ around in the VMWare backup world for a little while, I stumbled across this page:
http://communities.vmware.com/docs/DOC-8760
This is the page of the rather excellent ghettoVCB.sh script, written by William Lam.
Here you’ll find complete instructions on how to execute VCB-style backups from either ESX or ESXi (including the free version). The logic is essentially the same as the normal VCB ‘fullvm’ backup:
- Snapshot the Guest VM
- Copy all of the VM data to another location
- Delete the Guest VM snapshot
The process is also insanely quick. One of my clients wanted to see how it would perform in their environment, so we did a little testing.
Currently, they are using the VCB Framework, installed on a Windows XP machine, to backup VM Guests to a CIFS share. This is essentially using the vcbMounter.exe like this:
VCBMOUNTER.EXE -h [ESX Server IP] –u [ESX Username] -p [ESX User Password] -a name:”[Name of VM Guest]” -r “[Destination Path]” -t fullvm -m nbd
They are using the ‘nbd’ method, as the VM Guests are stored on local disk in the ESXi Server, so there’s no option to use the ’san’ mode. Their normal destination is a small single disk NAS, which whilst cheap, is a bit limited in performance terms. But it’s OK for their current needs.
For testing purposes, we changed the destination to an 8-disk SATA NAS (a QNAP TS-809U-RP). This is because previous testing has shown that it’s got some good raw performance:
This is recorded from the perspective of the switchport, so ‘Outbound’ (leaving the switch) is write performance. It’s around:
- 700 Mbits/s Write
- 950 Mbits/s Read
Which should mean that it’s good enough for our testing purposes…
So running their normal backup (of all their Guest VMs) to this location gives us a network profile that looks like this:
The section highlighted in red is a 200GB vmdk of one of the VM Guests.
- 200GB in 4 hours = 50GB / Hour
So what did we get when we ran the ghettoVCB.sh script?
Well, this graph shows the backup of just one Guest VM:
The red highlight section is the same 200GB vmdk file.
- 200GB in 40 minutes = 300GB / Hour
The performance was (*cough*) slightly better. In fact, 600% better!
They were certainly pleased.
I should say that there is a little gotcha in all of this. The CPU on the ESX server can still be a limiting factor. These tests were performed using an HP DL380 G6 (1 x Quad-Core Intel 5530 2.4GHz).
I repeated some similar tests in our office lab using a Sun Fire X4100 (2 x Dual-Core AMD Opteron 285 2.53GHz) and the maximum speeds I could get were in the 300 Mbits/s (135GB / Hour) range. It’s not the number of cores or the CPU speed, it’s simply the fact that the latest generation of Intel Processors appear to be rather good…
What’s the conclusion?
All of this testing was done on ESXi (and ESX) 4.0, where is appears that VMWare may have introduced ’some issues’ when it comes to ‘nbd’ / Service Console backups:
If you’re using VCB in the ‘nbd’ mode, then maybe it’s time to think about some other strategies. Especially if you’re about to upgrade from ESX 3.5 to ESX 4.0
Your backup strategy may be influenced by other factors, but for my money, maybe it’s time we all moved to the Ghetto?
G.
Are Cisco the new Microsoft?
Cisco have released 11 new Security Advisories today (23rd September 2009):
http://www.cisco.com/en/US/products/products_security_advisories_listing.html
I remember when IOS used to be considered stable…
VMWare, Backup and VCB
(The Good, The Bad and The Ugly)
I hate Backup.
Actually, that’s not entirely true. I hate having to ‘do’ backups. After too many years in IT, I know the value of having good backups. But it doesn’t change the fact that I hate having to perform and manage them.
I love VMWare.
Without a shadow of doubt, I love working with VMWare for the simple fact that it makes my life easier.
So now I find myself in the dubious position of having the reconcile my love / hate relationship between VMWare and Backup. In my role as a Consultant as Scale Abilities, I’m often asked by clients what they should do with regards to VMWare backups. How long is that piece of sting..?
The problem is that there are almost too many ways to address the question. Agents in the VM Guest? Tape or disk? D2D2T? SAN Snapshots? Fibre attached or iSCSI? Vendor A vs Vendor B? What’s the recommended ‘Best Practice’?
As a company, we’ve banned the use of the phrase ‘Best Practice’, because at the end of the day, every client’s situation is different. What you should be asking is, what’s the ‘Right Practice’.
The ‘Right Practice’ is a lot of cases is the balance between cost vs functionality. I like to keep the costs low, as do the majority of our clients. So this is why VMWare’s Consolidated Backup (VCB) is often the backup choice in many situations.
I like it, because one of the tools, vcbMounter, is included in ESX and it lets me take backups. This seems like a good start for a backup tool…
For myself, I like the simplicity of just taking a full image-based backup of my VMs. Then I’ve got everything I need should a want to restore them. You can keep your incremental / differential backups; I’m sure they have their place, but I want the whole thing, every time.
I hate tape.
The stuff always keeps going wrong, which isn’t a good recommendation for a backup technology. It’s also really, really expensive. You don’t believe me?
LTO4 tapes, which hold about 800GB native, are about £50 each. Doesn’t sound like much right? But what about the cost of the LTO drive? Or more normally, the cost of an autoloader as no-one want to be swapping tapes at midnight…
So say you had to backup 2TB of data, you’d probably need something like:
- 3 x LTO4 Tapes = £150.00
- 1 x 1/8 HP LTO4 Autoloader = £3200.00
- 1 x SAS HBA = £100.00
- 1 x NBD Hardware Support = £300.00
- Total = £3750.00
That’s your initial investment to backup 2TB data. If you want to keep historical backups, it’s £150 each time for another set of 3 LTO4 Tapes.
So what’s the alternative? Well, I’m a big fan of disk. You can now pickup a small 2TB NFS NAS for less than £400.
With disk, what do we need to backup 2TB of data?:
- 1 x 2TB NFS NAS = £400
- Total = £400
Hummmm… I’d struggle to justify the tape solution to a client!
Need to keep historical backups? Cool, buy another NAS. Buy another 8 and you’ll still be better off!
Want to keep monthly backups? Well, that’s 36 LTO4 Tapes (3 per month), so now your backup costs are:
- 36 x LTO4 Tapes = £1800.00
- 1 x 1/8 HP LTO4 Autoloader = £3200.00
- 1 x SAS HBA = £100.00
- 1 x NBD Hardware Support = £300.00
- Total = £5400.00
The disk / NAS costs are still less than that of tape:
- 12 x 2TB NFS NAS = £4800.00
- Total = £4800.00
Oh yes, and that’s just hardware costs… Backup to tape almost certainly means that you need some backup software at some stage, so just throw another £500-£1000 in there. And of course, we all know how reliable those tapes are… They never break do they?
Why is all of this important? (Besides the cost saving aspect) Well, it means that with tape out of the equation, we can just use vcbMounter to make our backups straight to disk. Yay!
Again, there are still plenty of ways to skin the cat, but my preference is:
- Mount the NAS as an NFS datastore
- Run your vcbMounter backups
- Unmount the NAS
All of this is easily scriptable and can be done on the ESX Server.
The performance is dependant on a combination of your NAS and ESX Server CPU.
As an example, I know that our NetApp SAN will happily do 900 Mbit/s without breaking a sweat. So using this for testing, means that we can eliminate NAS performance. So backing up a 20GB VM Guest, the network performance looks like this:
VCB - 20GB VM Guest Backup
Although it’s not a very scientific test, checking out the CPU performance on ESX server, shows that the ‘host’ process is hammering the CPU. But interesting, it seems to be limited to one core, so actually may not impact VM Guest performance as much as you might think…
The key to all of this is to make sure that you get the ‘right’ NAS, that supports NFS at the performance levels you need.
Some cheaper NAS devices, which claim to support NFS, are sadly limited in their performance. Make sure that you do some testing first to ensure that you get what you need.
And remember, there’s no ‘Best Practice’, only ‘Right Practice’.
Simples.
G.
