Toni Westbrook dot Com

The Joy of Computer Science and Engineering

We recently upgraded from ESX 3.5 to vSphere at work, and man – is it awesome. The infrastructure manager (vSphere Client) supports a whole boatload of new options for better monitoring and managing your VMs and clusters – including a patch management system for keeping your VMs and hosts up to date. If you’d like the full details of all the new features, check out the VMware page here.

The Console OS

One new aspect of vSphere is it separates out the hypervisor from the service console to a greater degree, actually creating a VM for the console. Great from an architectural point of view – however, one gotcha is where it actually STORES this VM. During the upgrade (and I’m assuming new install process), it asks where you’d like to store this Console OS VM. It suggests you store it on local storage, but you can just as easily store it in a Datastore on your SAN. Which is what I (and others judging from the web) chose as an option.

An issue arises in that there is no easy way (that I know of, after scouring the web for ages) to move the console VM in the future. We recently needed to redo all the LUNs on our SANs, which meant we needed them empty – which is when we ran into this issue. The service console was on the SAN with no good way to move it.

The Solution

First off – proceed at your own risk. [UPDATE] When asking VMware support about this solution, they said it works, but is not officially supported. [END UPDATE] So far we’ve had no issues, but that’s not to say in 2 months our whole ESX cluster won’t detonate, showering death and destruction down on us. That being said, I think we’re pretty safe.

This question has been asked before, and everyone on the VMware forums said you needed to reinstall your ESX server to move the console VM – this is the official stance by VMware as well. I originally decided to do a little digging though, as it just seemed like there must be some way of doing it. It was just a file that needed to be moved, and the underlying OS is Linux, so I guessed it was all being done through scripts. I was right.

If you take a look at /etc/vmware/esx.conf on your ESX host in question, you’ll see all your configuration options. One of which is

/boot/cosvmdk

This points to the path and filename of the service console. This value is later used by the initialization script “/etc/vmware/init/init.d/66.vsd-mount” to mount the service console. We can change this value to anything we want, inclung a new location.

1. Identify the correct service console VM for the ESX server in question. This isn’t an issue if you’ve got 1 ESX server, but if you’ve got a cluster, it’s not always clear the one in question. The console VM is stored with the name “esxconsole-<uuid>”. You can find the unique identifier/cos vmdk filename for your server within the /etc/vmware/esx.conf file.
2. Identify the Datastore where you want to keep the service console. Take VMware’s advice and keep it on local storage – that way, if your SAN dies or you need to do maintenance, you aren’t in a pickle. Look in /vmfs/volumes and write down the ID of the storage you want to use.
3. Put the ESX host in question into service mode – you’ll need to reboot it to perform the move, and you’ll need local access as it won’t reboot to a point where you have ssh.
4. Make a backup copy of /etc/vmware/esx.conf in case you make a boo-boo.
5. Edit /etc/vmware/esx.conf and change the path for the “/boot/cosvmdk” option to point to the new Datastore you recorded in #2. Save the conf file.
6. Reboot the server. It will go smoothly until it hits the point where it attempts to mount the COS – at this point, it will choke as it can’t find esxconsole in the new place you told it to look. At this point, you’ll get a shell prompt.
7. Do a recursive copy of the esxconsole-<uuid> directory from its old location to the new location. You should have all your /vmfs/volumes mounted since this takes place in the initialization sequence before the COS is mounted. [UPDATE] Jim points out that while local and FC storage will be available, it looks like iSCSI mounts take place later in the boot process, so they will not. Just a heads up if you’re planning on moving your COS to/from iSCSI [END UPDATE]
8. Reboot. It should boot back up completely at this point. Ensure life is good, and then delete the old copy of the esxconsole-<uuid> directory.
9. Ensure ESX automatically updated any other values to the new path (like /adv/Misc/CosCorefile)

Please don’t hesitate to comment with your experiences or if you know a better way to handle this (or if this solution could cause issues). Good luck!

Though by night I practice in the dark arts of computer science and engineering, during the day I play the part of a mild mannered network administrator. Recently I was taking stock of our backups, and as I was looking through some items that needed to be included in the nightly routine, I checked out our IAS server. We run IAS as we have a number of RADIUS clients, such as switches and other devices, that we like to have authenticate against our Active Directory. RADIUS connected to AD via IAS is super sweet, as there are quite a number of devices out there that support RADIUS, and you can get pretty detailed with the authentication rules of what is and isn’t granted access.

In IAS, to establish trust between the server and the RADIUS client, an administrator sets up a shared secret – basically a password that both ends agree to use to prove they are who they say they are during communication. Normally, you would expect such a password to at least be encrypted, or at least obfuscated in some manner to add a level of protection to snooping eyes. Microsoft has however, to my surprise, decided not take this route.

Viewing Shared Secrets

IAS stores its settings in two files under C:\windows\system32\ias – ias.mdb and dnary.mdb. If you’re a database user, you’ll notice mdb being the file extension used by Jet/Access databases. For the heck of it, being a tinkerer, I decided to link to these files with MS Access and see what I could see. They are indeed standard Jet databases – which is pretty neat from an integration perspective – with a simple ODBC connection you can read/write your IAS settings. There is a table called “Objects” that contains an entry for each one of your RADIUS clients. What was a little surprising, however, is there is a field labeled “Shared Secret” that contains, in very clear text, the shared secret password for each RADIUS client.

Now while users shouldn’t have access to this file normally, having a big, easy to use database full of passwords always makes me a bit nervous. Understandably hashing might not have been an option due to the need to deduce the original cleartext – but where authentication is involved, a little encryption would be nice, to at least dissuade the average script kiddie.

Not the security hole of the century, but certainly something to be aware of.