State of the Nation
For me, one of the important aspects of managing a VMWare environment, is to quickly be able to ‘see’ what’s happening and react accordingly.
Specifically, I want to know what my Guest VMs are up to. Luckily ESX provides with this information in the form of ‘State Transition’ events.
In ESX 4 these are logged to the hostd log (/var/log/vmware/hostd.log). They look a bit like this:
[2009-09-28 11:22:02.048 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_ON -> VM_STATE_OFF)
[2009-09-28 11:22:02.340 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_RECONFIGURING)
[2009-09-28 11:22:02.554 F64D2B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_OFF)
[2009-09-28 11:22:07.670 F5A6DB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_RECONFIGURING)
[2009-09-28 11:22:07.954 F5A6DB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/LNX-TEST03/LNX-TEST03.vmx'] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_OFF)
This is a ‘power off’ sequence for shutting down a VM Guest.
Or more fun things like:
[2009-09-28 11:18:26.399 F6450B90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_ON -> VM_STATE_EMIGRATING)
[2009-09-28 11:18:52.732 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_EMIGRATING -> VM_STATE_OFF)
[2009-09-28 11:18:52.744 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_UNREGISTERING)
[2009-09-28 11:18:53.034 F634CB90 info 'vm:/vmfs/volumes/4ac2ea2f-95f7957e/VM-CACTI01/VM-CACTI01.vmx'] State Transition (VM_STATE_UNREGISTERING -> VM_STATE_GONE)
Which was a VMotion event from the perspective of the ESX server that was the original host. (I love the terminology by the way – ‘EMIGRATING’ / ‘GONE’).
Now, connecting to your ESX Servers and searching for these specific events would be a bit tedious. So one approach is to send them all to a remote Syslog server.
There are already a number of articles on the web on how to set-up remote sysloging for ESX, so I wont re-invent the wheel. Here’s what worked for me:
- SSH to your ESX Server
- Edit the end of the /etc/syslog.conf file to include the line
- *.* @[Your remote syslog server name / IP]
- Restart the Syslog service using the command:
- /etc/init.d syslog restart
- Allow outbound Syslog through the ESX Firewall
- esxcfg-firewall -o 514,udp,out,syslog
- Restart the ESX Firewall
- esxcfg-firewall -l
This should be enough to get the basic syslog information going where you want it to. But, it wont give you information from the /var/log/vmware/hostd.log
To get this, I did the following:
- SSH to your ESX Server
- Edit the file /etc/vmware/hostd/config.xml to look like this:
<log>
<directory>/var/log/vmware/</directory>
<name>hostd</name>
<outputToConsole>true</outputToConsole>
<level>info</level>
</log>
- Restart the ESX Management Agents with this command:
- /etc/init.d/mgmt-vmware restart
Pay close attention to the ‘level’ tag. By default it reads ‘verbose’. I tried this first and managed to generate about 4000 messages a minute. This was a bit more than I needed…
So what next?
Well, that’s really up to you. Now that you can capture these events, use your favourite Syslog server to either alert or report on whatever you’re interested in.
This sort of information is vital in environments where there are multiple teams administering the VMWare infrastructure. It’s not always easy to know what your colleagues are working on, so capturing this information can provide a quick oversight so that everyone is kept in the picture.
For example, once you’ve got the right Transition event, you can map these to a timeline, to very quickly see what’s happing to your VMs:
Or summarise them into a table:
Or make the names on the table a little more readable:
Once you’ve captured the information, it’s really up to you!
And if your Syslog server doesn’t let you produce reports like the ones above, then you need to start Splunking…
G.
(More information on how to use the exceptional Splunk Server in the next few weeks)
Also, check out these links for a bit more depth on how to configure the Syslog component in VMWare:
