March 25th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Some time around the release of vSphere 5.5 (Update 2 maybe?) VMware officially(?) didn’t not support vCenter on a Windows Failover Cluster. I say didn’t not support because there still seems to be very limited documentation and KB’s on how to do this. The VMware vCenter Server Availability Guide documents available options such as using HA for vCenter availability, but also how to install vCenter on a Windows Failover Cluster, and configure the services appropriately since the application itself is not other cluster aware, for example like installing SQL on a failover cluster.

If you have done a failover cluster on Windows before, the process is a bit different so don’t just dive in as I did. So what does my environment look like:

  • SSO has been already deployed and working
  • A management vCenter is running; you will need this or some other means to clone the first virtual machine after installation

So wait why are you clustering vCenter if there is already a vCenter you ask? Many reasons, but primarily our availability of our management vCenter is less of a concern. The clustered vCenter is being deployed to support vRealize Automation so end users will rely on this vCenter to be able to request catalog items. Availability was more of a concern for this purpose than strictly management.

  • Start with only a single Window 2012 R2 64-bit virtual machine (not 2) as you will later clone this virtual machine to act as the 2nd node
  • I placed the original, and clone on two separate physical hosts
  • Each virtual machine has a single 60GB (C) drive for the OS
  • 2 additional volumes will be added which, in my case, are XtremIO volumes presented as a physical RDM. This should also work using in-guest iSCSI for example
  • 1 of the 2 additional volumes is a 60GB (D) drive which vCenter will be installed on and the other a quorum disk for the failover cluster
  • Each virtual machine has two NICs – one for production/client access the other for cluster communication
  • The Windows Failover Cluster will have an IP address, as well as the vCenter Service role which you will create; in total this is 6 IP address
  • An AD account was created for the vCenter services, added to the local administrators group and given permission on the SQL server as required

A few notes before I review the process;

  • If you are using RDMs, make sure you read this KB to mark the RDMs as perennially claimed otherwise storage rescans and boot times will be drastically affected (hosts were taking roughly an hour to boot)
  • The directions have you install the vCenter Web Client, Inventory Service, and vCenter services to the D drive. There is a known bug that causes the web client to not function properly when installed to a non-default location (though it seems more that it doesn’t work when not installed to the C drive). You’ll need this KB article which walks you through creating a symbolic link, after implementing this the web client operated as expected. Also, once installation is complete and working on the primary node, you’ll need to failover to the secondary node to create the sym link (well at least I did, would it let you create a sym link to a drive that didn’t exist? hmmm)

So, with that out of the way there is a few things to define before you bring up your first virtual machine – specifically the names and IP addresses of both virtual machines, the Window cluster, and the vCenter cluster. For example:

jQuery(document).ready(function() {jQuery(‘#table_95684235’).dataTable( {“bPaginate”: false,”bLengthChange”: false,”bFilter”: false,”bSort”: false,”bInfo”: false,”bStateSave”: true,”bAutoWidth”: true,”sPaginationType”: “full_numbers”,”oLanguage”: {
“sLengthMenu”: “Display _MENU_ records per page”,
“sZeroRecords”: “Nothing found – sorry”,
“sInfo”: “Showing _START_ to _END_ of _TOTAL_ records”,
“sInfoEmpty”: “Showing 0 to 0 of 0 records”,
“sSearch”: “Search: “,
“sInfoFiltered”: “(filtered from _MAX_ total records)”
}} );});

Purpose Name IP
vCenter Cluster VC2
Windows Cluster VC2Win
Primary vCenter Node VC2-1
Secondary vCenter Node VC2-2


This is important, and I misinterpreted this step the first time I did this: When you create the first virtual machine – give it the name and IP address of what will ultimately be the vCenter cluster – using the example above you will name the computer VC2, with an IP address of and join it to your domain. After the initial install this will be changed.

Create the virtual machine, with 2 NICs and the RDMs. Mount one of the RDMs as D and one as whatever letter makes you happy, for my OCD that would be Q for quorum. Create your system DSN as you normally would, log in as your vCenter service account and perform a custom installation (not simple), installing each of the components to the D drive. During the installation process note that the name being added to SSO is the name that will ultimately be the vCenter cluster.

Before removing the RDMs, make sure to note their original file name, volume ID, and SCSI controller; they need to be added back in the same order.

These steps are pretty straight forward in the guide, change all of the vCenter services to manual, shutdown the virtual machine, remove the RDMs, and make a clone of the virtual machine. One item not clear was when to re-add the RDMs, I chose to play it safe and kept them out of the virtual machine for now. Once the clone is complete, power on the cloned virtual machine and rename it to the secondary vCenter node hostname and IP address. Power on the original virtual machine, unjoin it from the domain, rename and IP it with the hostname for the primary vCenter node, and rejoin the domain. Now you can power off the virtual machines, re-add the RDMs to the primary node, then the secondary as you typically would, making sure the SCSI controller is set to physical sharing.

Power on the virtual machines and install the Failover Cluster feature on each. Once complete, create a new cluster on the primary node – during the creation you will be asked for a cluster name and IP address – use the Windows Cluster name (VC2Win) from the example above – this is NOT the vCenter cluster name and IP address which you used on the initial virtual machine during installation. Unlike with the SQL post I wrote, you can add all available cluster storage as both additional drives are used for the cluster (D – App, Q – Quorum). Now that the cluster has been created, you should have an AD object called VC2Win. Using option #2 from this MSDN blog post, create your vCenter cluster AD object. Failing to do this will cause the cluster to fail when you attempt to start it.

The rest of the steps for creating the vCenter cluster role are well documented with one caveat, so rather than copy paste them here finish reading the VMware vCenter Server Availability Guide. That caveat, because your vCenter services were set to manual, and thus not started after the reboots, when you create the initial vCenter role service it will come us as failed – which made me go  ZOMG not again! This message is actually just the status of the clustered service, which is stopped, thus failed from a Windows Failover cluster perspective – it is okay to proceed with creating the remaining services and setting the dependencies.

At this point, you should be able to start the cluster and have all services come up.

vCenter services on Windows Failover Cluster

vCenter services on Windows Failover Cluster

Once it is up, access the web client and set permissions as required. For example, as you can see in this screenshot, here is both vCenters in the web client after since my account was given the appropriate permissions to both.

vCenter on a Windows Failover Cluster

vCenter on a Windows Failover Cluster

The last item I have to tackle is automating the backup, copy, and restore of the ADAM database. There are a lot of words in the doc which basically says – xcopy the backup to the correct location. The document talks about stopping/starting services before placing the file. But if the services aren’t running on VC2-2, I should just be able to drop it in. Now when the services start there is an up to date file which will get loaded.

So, quick a dirty like…

del d:backup*.* /Q
%windir%system32dsdbutil.exe “ac i VMwareVCMSDS” ifm “create full D:backup” q q
xcopy /osy d:backupadamntds.dit “VC2-2C$ProgramDataVMwareVmware VirtualCenterVMwareVCMSDS”

VMware vCenter on Windows 2012 Failover Cluster

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , ,

February 9th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Generally, installing virtual appliances has been pretty straight forward – import an OVA and enter the necessary details in the deployment wizard, or access the virtual appliances management interface (such as those typically on port 5480 from VMware). However, as of the Release Candidate for VMware vSphere 6.0, the vCenter Server Appliance (VCSA) installation takes a much different approach than what you’ve been used to.

A few vCenter Server Appliance prerequisites

First, it should be noted that you can only install the vCenter Server Appliance (VCSA) from Windows. I was first turned onto the VCSA because I was at an all OSX/Linux shop so it made sense to use something we were accustomed to using already. For now, you’ll need a Windows box to at least get the appliance deployed;  then you can punt (please note also this is based on Release Candidate (RC) code and could change in the final release).

You CAN deploy the VCSA 6.0 to both ESXi 5.5 or 6.0 host. If you currently have a 5.5 environment you can deploy the VCSA without upgrading your hosts, but if you did not take  the plunge into 5.5 you’ll have to bring at least one host online running 5.5. or 6.0.

Finally, before getting started, you MUST create DNS records before running the installer. I was struggling with the new installer because I’ve just been used to doing my DNS records after I deployed the VCSA, but before running the setup through the management interface. However with a little help from Emad Younis (@Emad_Younis) I was able to point me in the right direction. With 6.0 all of the configuration is done from the initial setup wizard. When it’s finished installing, vCenter is ready to run.

The installation wizard will NOT give you an error if this does not exist, instead it will fail during the installation!

As you can see here I have my forward and reverse DNS records ready to go on .9


Installing the vCenter Server Appliance

As with the older versions of the VCSA, it all starts with a download; however in this case you will be downloading an ISO image. Once the ISO image is downloaded either mount the ISO on your Windows box or extract the ISOs into a folder (as seen here).


Now that you have access to the files, drill down into the vcsa folder, there you will find the VMware-ClientIntegrationPlugin-6.0.0. Install this application on your Windows box (double click, Next, Accept/Next, Next, Install, Finish). Once the plugin finishes installing, back up one folder level and open the index file. As you can see here I am on Windows Server 2012, thus at least IE10 however opening the index in IE10 gives me a warning that I need to upgrade to at least IE10 or 11, so yea I’m going with Chrome. As with any plugin, you must enable it in Chrome. Click on the puzzle piece with the red x, then click Always allow and refresh the page and click the Allow button.


You should now see the vCenter icon along with a large Install button, click on it. You will get a UI very similar to what you would get deploying a virtual appliance.

vmware-vcsa-6 -installer

1.  After carefully reading the license agreement, printing it for your records, and having it signed by an attorney, click the I accept… check box and click Next.

2.  Now you can chose to deploy to your target server. Specify your ESXi host (5.5 or above!), username and password – now click Next.

If you are using self signed/untrusted certificates click Yes when prompted.

3.  The next step is to name your appliance. In my case, like I have created in DNS, my appliance name will be vxprt-vc02.vxprt.local. Click Next

4.  On the deployment type you can chose to install an embedded Platform Services Controller (which includes Single Sign-On in vSphere 6.0), just the the PSC, or just vCenter. You can have multiple Platform Services Controllers, and they can be different types. For example you could do a stand-alone PSC and have an embedded one with the VCSA. When the installer says “embedded” it really just means the components will be installed on the same virtual appliance as vCenter. I’ll be doing embedded here. Click Next

5.  Chose whether you have an existing SSO domain or you will be creating a new one. I will do this install as a greenfield type deployment, so select Configure Single Sign-On. Now enter the administrator password, and domain. To stay consistent with what I know about SSO, I’ll enter vsphere.local here. Click Next.

VMware vCenter Server Appliance (VCSA) step 5 - configure SSO

VMware vCenter Server Appliance (VCSA) step 5 – configure SSO

6.  Select the appliance size that supports your environment, including the new “tiny” deployment for up to 20 hosts. Click Next

7.  Select the datastore you will to install to, and whether to THIN PROVISION the vmdk (no VMware, I’m not calling it “Thin Disk Mode” – THIN PROVISION!). Click Next

8.  If you’re an Oracle shop, you have a choice on step 8, otherwise just click Next.

9.  Chose a network (this will be based on the host you deployed to), and how to assign IP information including the host name – This MUST match DNS. I’ll select static as that is what I would want to do for this type of server. Finally enter the NTP server and click next (I’ve also enabled SSH so I can connect directly to the virtual machine.

VMware vCenter Server Appliance (VCSA) installation - Network Settings

VMware vCenter Server Appliance (VCSA) installation – Network Settings

10.  Review the settings you’ve enter, make sure your IP information and host name are all correct and click Finish. The installation of vCenter and the VCSA will start. You’ll even see it installing packages, that’s right this is a ground up build, not just a bunch of packages pre-installed on a virtual machine!

VMware vCenter Server Appliance (VCSA) installation process

VMware vCenter Server Appliance (VCSA) installation process

Once the installation is complete, you can connect to https://fqdn/vsphere-client (no more 9443! One less question on the VCP6 I guess :) ).

vmware-vcsa-6-installation-completeLog in as the [email protected] you configured during the installation.


So far on the release candidate I’ve had trouble deploying to a port group on a VDS (it gives errors almost immediately) even though it appears as a valid port group on the network settings page. It would be nice if VMware added more validation on the various steps to ensure there will be no errors during the installation. If you do run into an error, you need to re-run the installation wizard.

Installing the VMware vCenter Server Appliance 6.0 VCSA

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , , , , , ,

December 22nd, 2014 by JFrappier

Jonathan Frappier Virtxpert

SamSnowmanI don’t know what I would have done without Commitmas to pull us through. Anyway – uh, Commitmas? Huh, could it be that some of you are not acquainted with the story of Commitmas?

Yesterday was the first day of Commitmas, a community event thought up by Matt Brender to help us all get used to sharing code and working with GitHub. The challenges vary based on your level of comfort – I am starting in the beginner track and hoping to work my comfort level up to being a “beginner intermediate” by the end. Now I am no developer but I see the train coming, and for all the vCommunity out there I hope you see it coming at well. The future is code and scripts; Commitmas is a great way to prepare.

Last month I published a few posts on Ansible, as part of that I created a repository on GitHub to put my playbooks in. To get back into the swing of GitHub I decided to install Git on Windows, clone my Ansible repository and create a simple README file.

What little knowledge and hands on with GitHub I have has all been from a Linux based system. On Windows you lack some of the common tools you have with Linux such as the ability to create SSH keys or an SSH client. The Git install for Windows provides these for you. Installing Git for Windows is easy, thanks for course to; if you have not used Chocolatey before installing it is also quite simple. Open a cmd prompt as admimistrator and run

@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString(''))" && SET PATH=%PATH%;%ALLUSERSPROFILE%chocolateybin

Now packages are as easy to install as running

choco install git

Once installed, we need to verify a few system settings:

  • Go to Advanced system settings (Start >> Control Panel >> System >> Advanced system settings)
  • Click on Environment Variables
  • In the System variables section locate ‘Path’ and verify C:Program Files (x86)Gitcmd; is there
  • Add C:Program Files (x86)Gitbin; directly after C:Program Files (x86)Gitcmd; with no spaces after the ; (make sure you also end with a ;)
  • For SSH commands to work later, you need to add a variable.  In the User variables for user section, click the New… button
  • Create a variable named HOME with a value of %USERPROFILE%
  • Click OK three times and reboot the computer. After the reboot you

After the reboot you should be able to open a cmd prompt and run ssh and not get a “ssh is not a recognized command” message


We are now ready to setup GitHub on Windows. If you haven’t done so already, create a user account on GitHub. There are a few commands we need to run to get everything ready.

  • git config –global username [where username is your actual username, for example jfrappier]
  • git config –global [email protected] [where [email protected] is your actual email that you signed up for GitHub with]
  • ssh-keygen -t rsa -C [email protected]
  • Select the location to save the key, I accepted the default
  • Enter a passprhase

CD to %userprofile%.ssh; you should see two files – id_rsa and

  • Open in note pad and copy the contents
  • Log into GitHub and click on Settings (the gear icon in the upper right corner) >> ssh keys
  • Click Add SSH key
  • Provide a name, paste the contents from into the key textbox, and click Add key

Now switch back to your cmd prompt window

  • Type ssh -T [email protected]
  • Type yes to accept the certificate
  • Type the passphrase you set for your key previously

You are now authenticated with GitHub, you can now enjoy the 12 days of Commitmas!



On the first day of Commitmas – Windows Git, SSH, and Keys

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , ,

December 22nd, 2014 by JFrappier

Jonathan Frappier Virtxpert

Day two of commitmas, I have my Windows computer setup and SSH keys added to my GitHub account. Time to clone my existing repository to make a few edits. First, I created a directory on my computer called ‘git’ where I’ll save all my work; cd to that directory and run git init

Now, log into GitHub and find the repository you wish to clone, in the lower right corner of the screen you will see the Git clone URL for the repository. In my case I want the SSH URL so click on SSH.

git-cloneNow in your console type git clone [email protected]:jfrappier/ansible-test-playbooks.git or your repository; when prompted enter the passphrase for your SSH key. You should now have your files on your local machine.


Edit a file, for example the README file. Once the file has been edited and saved, we want to get the file back into the repository. To see what files have been changed, run git status. As you can see here my README file was modified.


As you can also see in the screen show we need to run git add, for example git add README or git add . – with the file added you now commit your changes by running git commit (this launches vi so press i to go to insert mode, enter a note then press esc :wq enter). Finally, git push to put the file back into GitHub. As you can see here I just updated my README file.

git-pushTomorrow we will get into forking!

On the second day of Commitmas – git clone, git add, git commit, git push

Posted in Tech Tagged with: , , , , , , , , , , , , ,

December 4th, 2014 by JFrappier

Jonathan Frappier Virtxpert

In part 4 we published an application blueprint through Application Serivces, that is pretty awesome but we still really haven’t done anything just yet.  I mean its all just about working but the real hard part is creating the application blueprints.  Just for fun, lets create a generic blueprint and run a deployment.  While logged into Application Services go to Applications and click on the green + (plus) button to create a new application.

  • Name the application and select a business group, if you’ve followed along my various home lab series you would select StarWars here since it is the only business group we gave permission to in vRealize Automation.
  • Click save, click Create Application Version then click Save
  • Now you are able to create a blueprint; click Create Blueprint
  • Drag the logical template to the design pane, again if you’re following along with me this would be the CentOS 64 logical template
VMware Application Services / Application Director application designer

VMware Application Services / Application Director application designer

  • Now all this would do is create a virtual machine like you could do through vRealize Automation or vSphere; here however we also have several preconfigured services we can drag into our logical template to install applications.
  • Let’s do a typical single node web and database server
  • Drag Apache, vFabric RabbitMQ and and vFabric Postgres into the logical template, it should look something like this:

apps-app-services-addedNow one of the hardest parts about automating something is now all the dependencies.  In this scenario I happen to know a few things are missing, not because I am a genius but because I went through several iterations of this blueprint before getting it to work.  This, however also allows me to demo some other features of Application Services.  In my CentOS template, SELinux is enabled – now I could convert my template to a virtual machine, disable it, clean up the virtual machine machine again and convert it back to a template.  It’s what I would have done not 6-8 months ago.  Now, however, I’ll simply use the tools available to me, tools like Application Services or Ansible to put the virtual machine into the state I want it:

  • From the Application Components page, drag two “script” items into the logical template
  • Edit the first script by clicking on it; name it (no spaces), click on Actions, click “Click here to Edit,” copy the following into the window and click the reboot checkbox

# set SELinux disabled
cp /etc/selinux/config /etc/selinux/config.bak
sed -i s/SELINUX=permissive/SELINUX=disabled/g /etc/selinux/config

  • SELinux will now be disabled upon reboot.
  • We also have to tweak the EPEL install to allow it to pull data properly (seems to be a known issues right now).  Rather than letting the EPEL package install as part of the services we used earlier, we can also do that in a script and configure the options we need for it to work.
  • Edit the 2nd script as you did before but copy the following into the window

# install EPEL
yum -y install
sed -i “s/mirrorlist=https/mirrorlist=http/” /etc/yum.repos.d/epel.repo

  • Click the OK button, you should now see something like this:


  • Now click the deploy button, name the deployment, and select the business group
  • Click Map Details, ensure all details match what you have setup, and click Next
  • Provide a name to your virtual machine and edit CPU and memory as needed (and to match your vRA blueprint limits) – click Next
  • Review the deployment blueprint and click Next
  • Click the deploy  button (you could also publish to vRA here as we did in part 4, but I’m just demonstrating the deployment)
  • The deployment will start

Now at one point I wasn’t sure it was working, I could see Application Services say it was working (system was under 80-90% load consistently) however I wanted to see what vSphere was doing.  As you an see in the two screenshots below, the virtual machines are being deployed as you might expect (they are from two different deployments so yes the dates are different)

Application Services - virtual machined deployed via the web client

Application Services – virtual machined deployed via the web client

VMware Application Services deployment viewed in vSphere Client

VMware Application Services deployment viewed in vSphere Client

In addition, you can zoom in on the Execution Plan pane to see what step the deployment is currently on

Application Services provisioning a virtual machine

Application Services provisioning a virtual machine

This process took quite a while in my lab, but it I am pretty resource bound now.  Now, as I mentioned this is an iterative processes, good chance it may have failed in your environment, review errors and run the deployment again.  After working through any specific environment issues you should be able to successfully deploy the application components.


Deploy an Application Blueprint – Application Services Series Part 5

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,