May 7th, 2015 by JFrappier

Jonathan Frappier Virtxpert

The PowerCLI Cookbook by Phillip Sellers is an excellent resource for any skill level, whether you are a beginner or looking for a great reference to have with you.

PowerCLI Cookbook by Phillip Sellers

First and foremost, this book far exceeds what I expect out of a technology cookbook. If you step back and think about a (food) cookbook you get the recipe for what you are going to make (i.e. what you are going to do in PowerCLI) and the ingredients to make it (i.e. the cmdlets necessary to perform the task). Phillip took that a step further and began the cookbook with how to actually start the oven, or in this case a simple recipe to connect to vCenter and get started using PowerCLI.

The chapters in the book are laid out very well, starting with basic hosts related tasks, before moving on to vCenter, virtual machines, and other more complex scenarios – the build up in this format makes it excellent for those who are new to PowerCLI, or even VMware for that matter. Each recipie also has a “how it works” section where the components use are explained (no one has ever told me how food flavors work together!).

You could quite literally use the book to just about stand up a complete vSphere environment as all the major topics such as networking, datastores, clusters, and virtual machine management (including using PowerCLI to invoke in guest scrips) is covered.

**Disclaimer – I have a book published with Packt Publishing and spoke to Phillip before he decided to write the book. This book was provided to me by the author but the review was not read, or approved by Phillip, it is simply my opinion on the book and its contents.**

Yummy! – PowerCLI Cookbook Review by Phillip Sellers (@pbsellers)

Posted in Tech Tagged with: , , , , , , , , ,

March 25th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Some time around the release of vSphere 5.5 (Update 2 maybe?) VMware officially(?) didn’t not support vCenter on a Windows Failover Cluster. I say didn’t not support because there still seems to be very limited documentation and KB’s on how to do this. The VMware vCenter Server Availability Guide documents available options such as using HA for vCenter availability, but also how to install vCenter on a Windows Failover Cluster, and configure the services appropriately since the application itself is not other cluster aware, for example like installing SQL on a failover cluster.

If you have done a failover cluster on Windows before, the process is a bit different so don’t just dive in as I did. So what does my environment look like:

  • SSO has been already deployed and working
  • A management vCenter is running; you will need this or some other means to clone the first virtual machine after installation

So wait why are you clustering vCenter if there is already a vCenter you ask? Many reasons, but primarily our availability of our management vCenter is less of a concern. The clustered vCenter is being deployed to support vRealize Automation so end users will rely on this vCenter to be able to request catalog items. Availability was more of a concern for this purpose than strictly management.

  • Start with only a single Window 2012 R2 64-bit virtual machine (not 2) as you will later clone this virtual machine to act as the 2nd node
  • I placed the original, and clone on two separate physical hosts
  • Each virtual machine has a single 60GB (C) drive for the OS
  • 2 additional volumes will be added which, in my case, are XtremIO volumes presented as a physical RDM. This should also work using in-guest iSCSI for example
  • 1 of the 2 additional volumes is a 60GB (D) drive which vCenter will be installed on and the other a quorum disk for the failover cluster
  • Each virtual machine has two NICs – one for production/client access the other for cluster communication
  • The Windows Failover Cluster will have an IP address, as well as the vCenter Service role which you will create; in total this is 6 IP address
  • An AD account was created for the vCenter services, added to the local administrators group and given permission on the SQL server as required

A few notes before I review the process;

  • If you are using RDMs, make sure you read this KB to mark the RDMs as perennially claimed otherwise storage rescans and boot times will be drastically affected (hosts were taking roughly an hour to boot)
  • The directions have you install the vCenter Web Client, Inventory Service, and vCenter services to the D drive. There is a known bug that causes the web client to not function properly when installed to a non-default location (though it seems more that it doesn’t work when not installed to the C drive). You’ll need this KB article which walks you through creating a symbolic link, after implementing this the web client operated as expected. Also, once installation is complete and working on the primary node, you’ll need to failover to the secondary node to create the sym link (well at least I did, would it let you create a sym link to a drive that didn’t exist? hmmm)

So, with that out of the way there is a few things to define before you bring up your first virtual machine – specifically the names and IP addresses of both virtual machines, the Window cluster, and the vCenter cluster. For example:

jQuery(document).ready(function() {jQuery(‘#table_95684235’).dataTable( {“bPaginate”: false,”bLengthChange”: false,”bFilter”: false,”bSort”: false,”bInfo”: false,”bStateSave”: true,”bAutoWidth”: true,”sPaginationType”: “full_numbers”,”oLanguage”: {
“sLengthMenu”: “Display _MENU_ records per page”,
“sZeroRecords”: “Nothing found – sorry”,
“sInfo”: “Showing _START_ to _END_ of _TOTAL_ records”,
“sInfoEmpty”: “Showing 0 to 0 of 0 records”,
“sSearch”: “Search: “,
“sInfoFiltered”: “(filtered from _MAX_ total records)”
}} );});

Purpose Name IP
vCenter Cluster VC2 192.168.1.100
Windows Cluster VC2Win 192.168.1.99
Primary vCenter Node VC2-1 192.168.1.101
Secondary vCenter Node VC2-2 192.168.1.102

 

This is important, and I misinterpreted this step the first time I did this: When you create the first virtual machine – give it the name and IP address of what will ultimately be the vCenter cluster – using the example above you will name the computer VC2, with an IP address of 192.168.1.100 and join it to your domain. After the initial install this will be changed.

Create the virtual machine, with 2 NICs and the RDMs. Mount one of the RDMs as D and one as whatever letter makes you happy, for my OCD that would be Q for quorum. Create your system DSN as you normally would, log in as your vCenter service account and perform a custom installation (not simple), installing each of the components to the D drive. During the installation process note that the name being added to SSO is the name that will ultimately be the vCenter cluster.

Before removing the RDMs, make sure to note their original file name, volume ID, and SCSI controller; they need to be added back in the same order.

These steps are pretty straight forward in the guide, change all of the vCenter services to manual, shutdown the virtual machine, remove the RDMs, and make a clone of the virtual machine. One item not clear was when to re-add the RDMs, I chose to play it safe and kept them out of the virtual machine for now. Once the clone is complete, power on the cloned virtual machine and rename it to the secondary vCenter node hostname and IP address. Power on the original virtual machine, unjoin it from the domain, rename and IP it with the hostname for the primary vCenter node, and rejoin the domain. Now you can power off the virtual machines, re-add the RDMs to the primary node, then the secondary as you typically would, making sure the SCSI controller is set to physical sharing.

Power on the virtual machines and install the Failover Cluster feature on each. Once complete, create a new cluster on the primary node – during the creation you will be asked for a cluster name and IP address – use the Windows Cluster name (VC2Win) from the example above – this is NOT the vCenter cluster name and IP address which you used on the initial virtual machine during installation. Unlike with the SQL post I wrote, you can add all available cluster storage as both additional drives are used for the cluster (D – App, Q – Quorum). Now that the cluster has been created, you should have an AD object called VC2Win. Using option #2 from this MSDN blog post, create your vCenter cluster AD object. Failing to do this will cause the cluster to fail when you attempt to start it.

The rest of the steps for creating the vCenter cluster role are well documented with one caveat, so rather than copy paste them here finish reading the VMware vCenter Server Availability Guide. That caveat, because your vCenter services were set to manual, and thus not started after the reboots, when you create the initial vCenter role service it will come us as failed – which made me go  ZOMG not again! This message is actually just the status of the clustered service, which is stopped, thus failed from a Windows Failover cluster perspective – it is okay to proceed with creating the remaining services and setting the dependencies.

At this point, you should be able to start the cluster and have all services come up.

vCenter services on Windows Failover Cluster

vCenter services on Windows Failover Cluster

Once it is up, access the web client and set permissions as required. For example, as you can see in this screenshot, here is both vCenters in the web client after since my account was given the appropriate permissions to both.

vCenter on a Windows Failover Cluster

vCenter on a Windows Failover Cluster

The last item I have to tackle is automating the backup, copy, and restore of the ADAM database. There are a lot of words in the doc which basically says – xcopy the backup to the correct location. The document talks about stopping/starting services before placing the file. But if the services aren’t running on VC2-2, I should just be able to drop it in. Now when the services start there is an up to date file which will get loaded.

So, quick a dirty like…

del d:backup*.* /Q
%windir%system32dsdbutil.exe “ac i VMwareVCMSDS” ifm “create full D:backup” q q
xcopy /osy d:backupadamntds.dit “VC2-2C$ProgramDataVMwareVmware VirtualCenterVMwareVCMSDS”

VMware vCenter on Windows 2012 Failover Cluster

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , ,

March 16th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Working on building out a lab that is going to be used to demonstrate setting up a vCenter environment. We were fortuneate enough to be given some time to set it up “right” – meaning setup a SQL cluster for vCenter, SSO in HA behind a load balancer with valid certificates. I drew the SQL straw, and it s the first time I have setup SQL clustering. I had to pull from a few different resources, none were completely what I was trying to do but thank you to Derek Seaman’s blog and the MSDN blogs for being able to answer questions when they came up. You can find more information on Windows Failover Clustering on vSphere 5.5 here (nope not on 6 yet). An over view of our setup:

  • Two Windows 2012 R2 virtual machines on separate hosts; SQL1 and SQL2
  • Each virtual machines with two NICs; one for production/client access the other for cluster communication.
  • Each virtual machine has 2 drives; 60GB “C” for OS and 20GB “D” for SQL installation
  • 3 XtremIO drives presented via VPLEX
  • AD accounts for SQL and SQL Agent were created in AD
  • IP addresses for each of the SQL virtual machines, the Windows cluster, and the SQL cluster; for this setup that is 4 total.

Windows was installed, patched and joined to the domain. On each virtual machine I ensured that Windows Ethernet0 was first in the biding order and used for “production.” NIC1 would be used for cluster communication. Ensure RSS is not enabled on the NICs.

  • In Windows 2012 R2 open the Network Connections window, click on Organize >> Layout >> Menu Bar.
  • Once the menu bar appears click on Advanced >> Advanced settings. Ensure Ethernet0 is listed first.

ethernet-order

With networking setup, add the .NET 3.5 (needed for SQL), and Failover Clustering features. Next, shutdown SQL1, and SQL2. On SQL1 I added 3 new drives as physical RDMs; 1 each for the vCenter, and vRealize Automation database files and 1 for a quorum volume. The mapping file will need to be stored on a shared datastore that is accessible by both hosts. When adding these to SQL1, add 1 at a time, select the next available SCSI adapter – for example the first volume was set to 1:0, the 2nd 2:0, and the quorum volume 3:0. The OS (C), and SQL (D), drives were 0:0 and 0:1 respectively. A few considerations:

  • In this format you will use all 4 of the possible SCSI controllers
  • The SCSI controllers for the RDMs must have their SCSI Bus Sharing set to Physical (the virtual machine has to be off to do this, thus why I mentioned shutting down SQL1)
  • Drives need to be added in the same order on both SQL1 and SQL2

Repeat adding drives to SQL2, except this time select “Use an existing virtual disk” and navigate to the SQL1 folder on the datastore to select the mapping file. You can check SQL1 to ensure that the correct mapping file is on the correct SCSI controller. For example, here you can see that SQL1_2.vmdk is on SCSI 1:0, so I want to make sure that SQL1_2.vmdk is using 1:0 on SQL2.

sql-rdm

Power on SQL1, and SQL2; configure each of the drives in Server Manager >> File and Storage Services on SQL1. DO NOT configure the drives on SQL2, when you build the cluster, and ultimately fail the cluster over from SQL1 to SQL2 it will mount those volumes. SQL2 should only have C (OS) and D (SQL) configured.

sql1-drives

Now that the drives on SQL1 are configured, open Server Manager >> Tools >> Failover Cluster Manager and click Create Cluster… Follow the wizard to add both SQL1 and SQL2. You will likely receive warnings about drives 0 and 1 since they can’t be used for the cluster. When prompted, DO NOT select the option to add available storage, otherwise it will add D. We only want to add the RDMs to the cluster. During this process is you will also create an AD object for the Windows cluster and assign 1 of the 4 IP addresses mentioned earlier (2 others were already given to SQL1 and SQL2)

Once the cluster is created, expand Storage and click on Disks. Uncheck the 20GB D drive, in my case I added 2x 250GB volumes, and the 50GB volume. The 50GB volume will be the quorum drive. Once added they should all be online. Click on your SQL cluster, and in the Actions pane click More Actions >> Configure Cluster Quorum Settings…

configure-quorum

When the Wizard opens, click Next and select Use default quorum configuration; in my case it correctly selected the 50GB volume for the quorum witness. Now if I go back to Storage >> Disks you can see Disk 3 is set as the Disk Witness in Quorum.

sql-drives

Next create an AD object for the SQL cluster, I opted to go with option #2 for from this MSDN post. This allows the AD Windows cluster object to manage the SQL AD object. You can finally start the SQL installer. When the SQL Installation menu opens, click Installation >> New SQL Server Failover Cluster Installation.

Select the options you wish for SQL, ensure that you install to the appropriate drive – in the case of this example D. Verify the installer correctly identified the RDMs as the data destination, and complete the installation. Once the installation finishes, enable FILESTREAM on the SQL Server Service. Start SQL Server Configuration Manager, click on SQL Server Services, right click on SQL Server and select properties. Click on the FILESTREAM tab and enable for both SQL and I/O access.

sql-server-filestream

Start the SQL installation on SQL2, except this time select Add Node to a SQL Failover Cluster. Select the cluster and most of the configuration options will be pulled from SQL1 (including the SQL installation directory, in my case D). Once complete you should be able to connect to the SQL instance, I did a named instance and called it VC so I connect to sqlc1vc

sql-cluster-management-studio

Apologies for lack for explicit steps, it would have been quite a lengthy post but happy as always to answer any questions I can. Here is a quick demo where I connected to the instance, created a DB, ran a query to list all the DBs, powered off SQL1 (hard shutdown, no role draining) and re-ran the query. It took a few seconds for the failover to happen but was fairly quick.

SQL Failover Cluster

SQL Failover Cluster Hosting vCenter Database

Configure Windows 2012 Failover Cluster for SQL 2014

Posted in Tech Tagged with: , , , , , , , , , , , , , ,

February 23rd, 2015 by JFrappier

Jonathan Frappier Virtxpert

I wanted to share some of the example Ansible playbooks used during last Wednesday’s US #vBrownBag. During the show I went over examples of how you can use Ansible to create, clone, and update virtual machines in vCenter without the need for other provisioning tools. Based on my testing (and I’m still learning as well), the items noted in the comments are the bare minimum needed to run the playbook, even though the official documentation may currently state otherwise. If you are already using Ansible for configuration management, this is a handy option to have as you can perform the provisioning tasks without leaving Ansible.

All playbooks have been uploaded to my GitHub Ansible-Test-Playbooks repository (https://github.com/jfrappier/ansible-test-playbooks/).

#vBrownBag Using Ansible with vCenter Examples

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , ,

February 10th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Scenario: You try to install the VMware vCenter Server Appliance (VCSA) or Platform Services Controller but receive an error during the installation. After correcting the problem during installation you attempt to re-install the appliance but receive the following error message:

Virtual Machine Already Exists

Virtual Machine Already Exists

As of the release candidate of vSphere 6.0, the vCenter Server Appliance installation wizard does not clean up deployed virtual machines after failed deployments. Virtual Machines deployed are still present on the selected ESXi hosts inventory. Log into the ESXi host, power off, and delete the virtual machine from the failed deployment.

vcsa-failed-deployment-vms-not-delete

If you attempt to redploy the virtual machine with a different name (appliance and host name) using the same IP address you receive the following error message:

Encountered an internal error. see /var/log/firstboot/vmafd-firstboot.py_6399_stderr.log

Because the virtual machine was deployed and powered on, there is a duplicate IP address on the network.

Error during deployment of vCenter Server Appliance or Platform Services Controller following error

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , , , ,