May 7th, 2015 by JFrappier

Jonathan Frappier Virtxpert

**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**

Up until now I went through a basic ViPR SRM installation, getting a basic single VM environment setup. What I want to show in this post is my favorite ViPR SRM feature – topology maps. To understand why these are useful, lets step back and give some scenarios:

You are the personal responsible for supporting the storage within your environment, you may support other things but ultimately when there is a storage related problem your name is called. An application own comes to you and says their application is slow, and that the network team said everything on their end is fine so its probably the storage. Great – now what?

  1. You come into a new organization – whether as an internal IT person or a var and you’ve inherited an environment cabled by 3 monkeys and a cat with no documentation – now what?

This is where topology maps can be very useful. The topology maps is that end-to-end visualization and monitoring component I mentioned in previous posts. I see from my virtual machine or even some applications such as SQL Server all the way through to the underlying storage, and drill down on each component. Let me shows you some examples.

To access the topology maps, click on Explore >> Hosts – small aside here – host could be any physical or virtual server in the environment discovered by ViPR SRM, not just ESXi hosts. So this could be an ESXi host, a virtual machine, or a physical host running its own OS.


From this report, you can see a list of all the hosts in the environment, which for some could be a very extensive list. I should mention that the filter field is not a search field, so you cannot type the end of a machine name; for example maybe all your VM names end in OS type or some other identifier, you couldn’t just type W2K8 to find a server name myserver-w2k8, you would have to start with myserver, but would then see a list of all servers starting with that string. You can filter on any column that has the funnel icon, so for example I could filter on just physical hosts, or virtual machines by clicking the funnel icon in the host type column;


Using the example above, let’s say an application owner has complained about performance and you need to investigate to see if storage could be the problem. Filter on the host name, in this case I will pick on mhmbd078-W2K8, as you can see below I start typing that name and can select it from a the list or type it in full and hit enter to filter on that one host



Now I just see that specific host, in this case a virtual machine as you can see here with 16GB of memory and 4 vCPU:


This much information is available in just a few clicks, now there are many places you could get this information but as I continue to drill deeper, you will start to see just how much information we have at hand. With just what is available so far, you might be able to say to the application owner who issued the complain that there is not enough memory, for example maybe you know that this particular application needs 32GB of memory, so disk I/O could be a problem if the application and OS are constantly swapping to disk. But, maybe so far everything checks out, if I click on any of the text here, it will take me into the detail of that virtual machine.

Now, this is where it gets interesting; what you see below is the topology map for mbmbd078-w2k8, we can see the host, the datastore it is on, the host it is on, the VSANs it is connected to and the arrays connected to those VSANs. Also, notice to the right we have different reports related to the host, we can see attributes about the host which is show by default, you can also see:

  • Capacity information about the hosts local disks, in this case VMDKs and since it is a virtual machine, the datastore
  • Path details for the disks attached to the host
  • Related storage performance
  • Events related to the host


You can click on any element in the map to see details specific to that item, for example if you click on the datastore – DS_Bootcamp_D you can see reports about the datastore, or on the host – you guessed it, reports about the host. You may have also noticed the + icon next to some of the elements, this is because there are additional components, using VSAN0040 as an example, we can click on the + sign to see switches in that VSAN


Now I see two switches, each with their own + icon, I can keep drilling down and see ports on that switch as well. I can expand different elements and hover over different components to see how they are connected. For example I have expanded my host to see my HBAs, I can see that the particular HBA I am interested in is connected to VSAN mptb023 so I have expanded that as well and drilled down to see the switch ports. While I have some limited lab resolution available, you can see here that when I hover over the HBA from the host it highlights the path to the port on the switch – in this case fc1/6 (as shown by the blue highlighted line)


This is just one specific report, and I have only skimmed the surface of the data available in this report. Imagine being able to show this to an application owner as you troubleshoot each component, and explain how/why any particular piece of the infrastructure supporting the application is, or isn’t doing what it is supposed to. For those folks who worked in a silo’d type group, I’d urge you not use this information to punt back over your wall to someone else, but rather be the person to start poking some pinholes in the silo, call up a virtualization, OS, or network person depending on what you might think the problem is and work with them, sharing knowledge and help the application owner be a happy customer. After all, even if you are “internal” IT – you are still providing a service to the business – they are you customers, treat them like it. Silos will only fall if someone starts poking holes, no reason it can’t be you.

If you haven’t done so, chat with your EMC rep (they can likey get you in touch with an SE who can help if you have any setup questions) and head over to to sign up for an account and download ViPR SRM which comes with a 30 day license.

ViPR SRM Explore Reports and Topology Maps

Posted in Tech Tagged with: , , , , , , , , , , , , , , , ,

January 24th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Now that Nagios Log Server is installed, it’s time to get some log files in there. I got myself all fired up ready to comb through page after page of documentation to figure out how to set it up… then those nice folks over at Nagios did this…

Nagios Log Server linux source setup

Nagios Log Server linux source setup

That’s right, if you click on Linux Source from the home screen, it gives you scripts to download and run to set it all up. They even pulled the IP address from the Nagios Log Server…it was like they wanted you to succeed in making this all work! It can’t be that easy right? Let’s try!

Linux Source - Nagios Log Server

Linux Source – Nagios Log Server

That was easy, no way there are actually logs showing up in Nagios Log Server though, right? Almost, SELinux was preventing log files from being shipped as you can see in the middle of the above screenshot. So…

cp /etc/selinux/config /etc/selinux/config.bak && sed -i s/SELINUX=enabled/SELINUX=disabled/g /etc/selinux/config && shutdown now -r

And BOOM goes the log file goodness after a reboot!

Remote linux source sending to Nagios Log Server - Dashboard search

Remote linux source sending to Nagios Log Server – Dashboard search

In probably less than 5 minutes, you can have a fully functional Nagios Log Server, based on ELK, deployed and receiving log files from a remote source – that is damn impressive. Of course in this example we haven’t looked at which logs we are sending – maybe you only want specific log files being sent from Apache or Ansible for instance, but that is a finer art form that we can save for another blog post. Happy logging!

Configuring CentOS to to send logs to Nagios Log Server

Posted in Tech Tagged with: , , , , , , , , , , ,

January 8th, 2015 by JFrappier

Jonathan Frappier Virtxpert

Chalk this up in the “useful error messages” column. When you attempt to enter a license key in the vRealize Automation appliance you receive “Error code: 500.”


Now when I saw this I immediately thought “internal server error,” however in the case of vRA it may simply be an expired or invalid license key. Before extensive troubleshooting validate that your license key is correct, and it has not expired.


vRealize Automation Error Code: 500 when submitting license key

Posted in Tech Tagged with: , , , , , , , , , , , , , , , ,

December 30th, 2014 by JFrappier

Jonathan Frappier Virtxpert

In my last video, we walked through how to install VMware Tools for CentOS 6.x. Now we are going to prepare the virtual machine for cloning. This requires we remove a specific file; /etc/udev/rules.d/70-persistent-net.rules. This file contains the MAC address for the virtual machine. Once removed, the file will be created during the initial boot with the matching MAC address for the cloned virtual machine.

Prepare CentOS 6.x virtual machine for cloning

Video: Prepare CentOS 6.x virtual machine for cloning

Posted in Tech Tagged with: , , , , , , , , , , , , , , ,

December 1st, 2014 by JFrappier

Jonathan Frappier Virtxpert

When deploying an Application Services blueprint, you notice that the workflow does not move past the 2nd step in the provisioning process – agent_bootstrap node setup, however the previous step which renames the virtual machines appears to work fine.  In this scenario you have also successfully installed the AppD agent in the vSphere template.

VMware Application Services / Application Director hangs at agent_bootstrap node setup

VMware Application Services / Application Director hangs at agent_bootstrap node setup

If you log into the virtual machine, you can see that the vmware_appdirector_agent should be set to on, however when checking the running services (for example by running ps -ef | grep vmware_appdirector_agent) you do not see any processes running.

When working with vSphere templates that are used in Application Services / Application Director blueprints there are a few things to be aware of.  After the agent installation initially, you shutdown the virtual machine, however making changes to the template after the initial shutdown requires additional steps to be performed.   Not only do you need to remove the /etc/udev/rules/70-persistent-net.rules file and make sure the vmnic MAC is not in /etc/sysconfig/network-scripts/ifcfg-eth0 but you also need to run


This reset the agent configuration and allows it to start properly after being cloned.  Once the agent_reset script has been run, shutdown the host and convert it back to a template.  You should now be able to run your Application Services / Application Director blueprint (or at least get past that step :) )

VMware Application Services / Application Director agent_bootstrap completes node setup

VMware Application Services / Application Director agent_bootstrap completes node setup

VMware Application Services Linux deployment hangs at agent_bootstrap node setup

Posted in Tech Tagged with: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,