How to Move the COS (esxconsole.vmdk) in VMware ESX 4 (vSphere)

We recently upgraded from ESX 3.5 to vSphere at work, and man – is it awesome. The infrastructure manager (vSphere Client) supports a whole boatload of new options for better monitoring and managing your VMs and clusters – including a patch management system for keeping your VMs and hosts up to date. If you’d like the full details of all the new features, check out the VMware page here.

The Console OS

One new aspect of vSphere is it separates out the hypervisor from the service console to a greater degree, actually creating a VM for the console. Great from an architectural point of view – however, one gotcha is where it actually STORES this VM. During the upgrade (and I’m assuming new install process), it asks where you’d like to store this Console OS VM. It suggests you store it on local storage, but you can just as easily store it in a Datastore on your SAN. Which is what I (and others judging from the web) chose as an option.

An issue arises in that there is no easy way (that I know of, after scouring the web for ages) to move the console VM in the future. We recently needed to redo all the LUNs on our SANs, which meant we needed them empty – which is when we ran into this issue. The service console was on the SAN with no good way to move it.

The Solution

First off – proceed at your own risk. [UPDATE] When asking VMware support about this solution, they said it works, but is not officially supported. [END UPDATE] So far we’ve had no issues, but that’s not to say in 2 months our whole ESX cluster won’t detonate, showering death and destruction down on us. That being said, I think we’re pretty safe.

This question has been asked before, and everyone on the VMware forums said you needed to reinstall your ESX server to move the console VM – this is the official stance by VMware as well. I originally decided to do a little digging though, as it just seemed like there must be some way of doing it. It was just a file that needed to be moved, and the underlying OS is Linux, so I guessed it was all being done through scripts. I was right.

If you take a look at /etc/vmware/esx.conf on your ESX host in question, you’ll see all your configuration options. One of which is

/boot/cosvmdk

This points to the path and filename of the service console. This value is later used by the initialization script “/etc/vmware/init/init.d/66.vsd-mount” to mount the service console. We can change this value to anything we want, inclung a new location.

  1. Identify the correct service console VM for the ESX server in question. This isn’t an issue if you’ve got 1 ESX server, but if you’ve got a cluster, it’s not always clear the one in question. The console VM is stored with the name “esxconsole-<uuid>”. You can find the unique identifier/cos vmdk filename for your server within the /etc/vmware/esx.conf file.
  2. Identify the Datastore where you want to keep the service console. Take VMware’s advice and keep it on local storage – that way, if your SAN dies or you need to do maintenance, you aren’t in a pickle. Look in /vmfs/volumes and write down the ID of the storage you want to use.
  3. Put the ESX host in question into service mode – you’ll need to reboot it to perform the move, and you’ll need local access as it won’t reboot to a point where you have ssh.
  4. Make a backup copy of /etc/vmware/esx.conf in case you make a boo-boo.
  5. Edit /etc/vmware/esx.conf and change the path for the “/boot/cosvmdk” option to point to the new Datastore you recorded in #2. Save the conf file.
  6. Reboot the server. It will go smoothly until it hits the point where it attempts to mount the COS – at this point, it will choke as it can’t find esxconsole in the new place you told it to look. At this point, you’ll get a shell prompt.
  7. Do a recursive copy of the esxconsole-<uuid> directory from its old location to the new location. You should have all your /vmfs/volumes mounted since this takes place in the initialization sequence before the COS is mounted. [UPDATE] Jim points out that while local and FC storage will be available, it looks like iSCSI mounts take place later in the boot process, so they will not. Just a heads up if you’re planning on moving your COS to/from iSCSI [END UPDATE]
  8. Reboot. It should boot back up completely at this point. Ensure life is good, and then delete the old copy of the esxconsole-<uuid> directory.
  9. Ensure ESX automatically updated any other values to the new path (like /adv/Misc/CosCorefile)

Please don’t hesitate to comment with your experiences or if you know a better way to handle this (or if this solution could cause issues). Good luck!

15 Responses to How to Move the COS (esxconsole.vmdk) in VMware ESX 4 (vSphere)

  1. Francesco Rana says:

    Well it went straight and smooth like you wrote. I actually run a couple of ESX/ESXi on Dell PowerConnect 2900 servers (in cluster mode with HA) and all went fine. Point number 9 was not to be performed as the ESX host modified all the COS entries making them point to the right partition.
    Also I forced a manual reboot instead of changing ESX host status to “maintenance mode”, as I wanted to test the HA configuration as well.
    Just to say that you made my work so much easier and want to thank you for this post.

    Keep Up the good work!

  2. Toni says:

    Glad I could help – and thanks for the info about #9 – when I performed my move, I actually changed all the values at the same time, which is why I never noticed that ESX automatically updates the rest of the values after changing /boot/cosvdmk. Very cool, one less step!

  3. Alfred says:

    Hello Toni

    I have the same Problem in my Company but i have 3 conf Files (esx.conf, esx.conf.old and esx.conf.esx4) what should i do now ????

    Last Modified is the esx.conf but there are a lot of differences between the esx.conf and the esx.conf.esx4

    THx Alfred

  4. Toni says:

    Hi Alfred -

    It’s normal to have those three files, you’ll want to just edit the esx.conf one. Did you originally upgrade that server from 3.x to 4? I haven’t taken a look at the content of those other files, but they make be related to the upgrade (e.g. the original esx.conf became esx.conf.old). Let me know if just changing the esx.conf doesn’t work.

  5. Alfred says:

    Hi Toni

    first i done it like you and it works fine

    I do the Upgrade from 3.5 to 4 by using the Update Manager and i found it very easy.

    Thanks for your answer and sorry for my English its a little bit rusty. :-)

    Alfred

  6. Toni says:

    Yeah, Update Manager is truly awesome – and it sounds like those files are indeed related to upgrading. Glad everything went okay!

  7. [...] Westbrook has a good article on how to move the COS VMDK in VMware ESX 4.0. Key note: this solution is currently unsupported by VMware, so use at your own [...]

  8. Jim says:

    Hey Toni, thanks for the great article, it came in very handy for me. One thing of note is that upon the first reboot while my local storage and FC LUNs showed up, the iSCSI ones did not. Luckily in my case I was moving it from FC to local so it was no big deal, but it may be an issue for others.

  9. Toni says:

    Glad it worked – and thanks for the info about the iSCSI drives being missing, good to know. I will update the article, thanks Jim.

  10. Groone says:

    Thanks for this. I have 4 servers I need to reclaim auto made primary datastores on so this is going to make my life simple. I dreaded the idea of having to reinstall esx and reset up resource pools and stuff. Thanks!

    • Toni says:

      Glad it helped – the dread you felt was the same I felt at the prospects of redoing everything as well. Hopefully VMware releases this (or another GUI based) solution for people who are stuck in this boat. It seems silly to make people go through so much work needlessly.

  11. AJ says:

    I am glad this post existed because it saved me alot of admin time on some ESX hosts wherein I THOUGHT it would have been a better idea for me to save the COS to a shared disk!

    Thank you much!

  12. Avram Woroch says:

    Thank you sir. Those steps worked like a dream, and I’ve shared with my internal team your tricks!

  13. [...] post did however get me thinking. I got to the source of his how-to (written up by Toni Westbrook here).The fix (also known as ‘Dirty Workaround’)There I found a real gem in the form of a [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>