April 1st, 2014 by JFrappier

Jonathan Frappier Virtxpert

During a recent vCenter deployment using the VCSA I ran into an error I hadn’t run into before with the VCSA (or vCenter/SSO on Windows for that matter).  After an error free install and setup wizard, I logged in to vCenter as [email protected] to set my roles and assign my AD groups permission.  However I noticed that there was no identity source for my Active Directory domain, no problem add it in, boom now hop on over to my vCenter permissions tab and get people vCentering.  This is where I ran into errors.  When trying to search for a user I received a pop up that said

Cannot load users for the selected domain

Before I ran the setup wizard, I had SSH’d to the VCSA did some pings and digs to make sure the network bits were flowing properly and everything seemed fine.  I could ping and dig both local and remote  AD resources so I was confident that was all working fine.  Easy fix I assumed, so I headed over to the global KB search tool known as Google and was lead to this KB, http://kb.vmware.com/kb/2033742 which suggested checking DNS, time synchronization and joining and re-joining to the domain.  I manually re-checked DNS records were present, that the AD join process had worked correctly and the account was still enabled.

Looking through the /storage/log/vmware/sso/ssoAdminServer.log I saw several exceptions with the following error (stripped excess text)

Failed to establish server connection

I searched the KB again for this error, but all I found was problems related to accent characters which wasn’t my thing.  At this point it was worth while to open a support case.  I wanted to make sure I had all my tier 1 support boxes ticked so I rebooted, verified I could ping/dig records, went back to the Identity Source page removed and re-added the domain and went to look up an AD group to get the error message and… all my users were listed.  I’ve not quite tracked down what was wrong, but if you are getting this error, and you know your DNS was square try just re-adding the sdentity source.

vCenter Active Directory SSO Error – Cannot load users for the selected domain

Posted in Tech Tagged with: , , , , , , , , , , , , , ,

January 22nd, 2014 by JFrappier

Jonathan Frappier Virtxpert

Splunk allows you to create either real time or scheduled alerts, and while real time alerts would seem logical they can be quite CPU intensive.  In fact, some suggest limiting to 1 real time search per CPU core even though the default limits will be about 3 per core.  You can see more on the limits.conf and how to change it here.  Rather than creating real time alerts, you can set or change existing alerts to run on a schedule.  In reality, most monitoring software works on a polling interval anyways so this is not far from what you are likely doing today with something like Nagios.

The process for creating or changing Splunk alerts from real time to scheduled is fairly straight forward.  By default, however, the shortest time period Splunk provides is 1 hour.  If you want to schedule Splunk alerts to run more frequently you will need to use the “Run on CRON scheduler” option.  Cron schedule examples give me a bit of a headache, but since we only have to worry about a per minute time interval since Splunk provides other options here is a quick how to on how to set these up, or change them.

Changing a Splunk Alert from Real Time to Scheduled

  • Log into your Splunk server
  • Under Search and Reporting click on Alerts
  • Find the Alert you wish to change and click Edit >> Edit Alert type and trigger
  • Under Alert type click on Scheduled
  • Change the time range to Run on CRON schedule, or one of the other options if those better suit your need
  • The “earliest” text box should be a negative number that matches how often you will run the alert.  For example if you want it to run every 5 minutes, set this to -5m
  • In the “latest” box enter Now so that it will search logs between 5 minutes ago and run time
  • In the CRON Expression box enter
    */5 * * * *
  • You can find more information about alert schedules here
  • Your alert window should look something like:


  • Click Next and configure the Actions you want to enable, such as email subject, recipients and how to include information (inline, CSV or PDF) or even options such as running a script
  • Click Save.  Your alert will now run at the defined schedule.


While alerts can be swell, they will have an impact on your server and, if you have to many alerts, you might not actually receive any of them, which would be bad.  Scheduling alerts is an easy way to make sure your Splunk server is not resource constrained.

How to change Splunk alerts from Real time to scheduled

Posted in Tech Tagged with: , , , , , , , ,

January 20th, 2014 by JFrappier

Jonathan Frappier Virtxpert

Here is a quick guide to adding additional Windows Active Directory groups for Splunk authentication to allow users to log in.  You can see how to configure Splunk for Active Directory here.  If you have not already done so, create the Active Directory group that you want to grant access to and add the users you want to to access Splunk into the group.

  • Log into Splunk as an administrative user
  • Click on Settings >> Access controls
  • Click on Authentication Method, then Configure Splunk to use LDAP and map groups
  • Click on the Map groups links for your AD
  • Click on the group you want to provide access to
  • Select the roles you want to provide to the group and click Save.
  • Log out of Splunk and log back in as a user who is part of the AD group you added.

If you are not able to log into Splunk, do the following:

  • Verify the user account object you are trying to log in as is within your user base DN setting for AD
  • Click on Settings >> Access controls >> Authentication method and click Reload authentication configuration
  • Click on Settings >> Server Controls >> Restart Splunk

Adding additional Active Directory Groups for Splunk Authentication

Posted in Tech Tagged with: , , , , , , , , , ,

December 20th, 2013 by JFrappier

Jonathan Frappier Virtxpert

Having been working on Splunk for the last few days, one of my last hurdles was to easily monitor, present and alert on Active Directory events such as account lock outs, group changes etc.  Low and behold Splunk has an app called Splunk App for Microsoft Windows Active Directory – problem solved!  Well as Lee Corso would say – “Not so fast my friend!”



The documentation for the Splunk App for AD starts with a warning, that its complicated to install…meh they just want PS revenue right?  The documentation starts off straight forward, mostly configuring your audit policies to log relevant information but ends up in a twisty turvey rabbit hole of horror!  So much so, that the sales rep even suggested I use another app!  So, here is how to actually monitor AD events, not using the Splunk App for AD.

The app you should look at, of course if it meets all of your requirements, is the Windows Security Operations Center app.  This app is not available for direct install from the app market directly from Splunk, you will need to download it.  Also it should be noted that as of this writing its only certified to work  Splunk 5, however I am testing it with Splunk 6 and having only a few minor problems that I suspect are related to my audit policy versus the application not working.  In my limited test environment, its generating just under 100MB per day (about 75 user accounts and tests scenarios like adding/removing/deleting accounts from the domain admins group).  For purposes of this walk through, I’ll assume you have a central Splunk server already installed and the Universal Forwarder installed on your Domain Controller(s).

  • In my test environment, I have configured a Group Policy for my domain controllers for Audit Policy to log success and failure of all defined events (logon, account management, policy change, privilege use, and system events).
  • Download the Windows Security Operations Center app from http://apps.splunk.com/app/647/
  • Log into Splunk and click on the Manage Apps button
  • Click Install app from file and browse the location you saved the file in the first step and click Upload.
  • Click Set up now
  • If you have used the default installation and configuration as we’ve done in the Using Splunk to Monitor Windows Event Logs post, then leave index and sourcetype at their defaults and click Save.
  • If you return to the dashboard you’ll now see the Windows Security Operations dashboard, click on the Login events menu in the dashboard and click Active Directory unsuccessful logins.  If you have had any, you should now see those events.



  • As I navigate through the other menus, I am also seeing data related to new accounts (if you recall we created an account for Splunk to read AD accounts for authentication), accounts I’ve unlocked and software I’ve installed (the Splunk UniversalForwarder specifically).


The Windows Security Operations Center App for Splunk is an easy to install and configure app, unlike the official Splunk App.  If you have a need to monitor AD events, this app should come in handy.  Of course at the end of the day its still Splunk, so if you can find it in your event log, you can find it in Splunk.

Using Splunk to Monitor Windows Active Directory Events

Posted in Tech Tagged with: , , , , , , , , , ,

December 17th, 2013 by JFrappier

Jonathan Frappier Virtxpert

A quick how-to for enabling Active Directory authentication for Splunk:

  • Create user for BIND, basically a service account
  • Check the box next to LDAP
  • Click COnfigure Splunk to use LDAP and map groups
  • Click the New button
  • Enter a name for your LDAP
  • Enter your AD or LDAP host
  • Enter the port (389 or 689)
  • Enter the BIND DN, for example CN=Jeff Smith,OU=Sales,DC=Fabrikam,DC=COM (*If you are using default OUs such as Users change OU to CN)
  • Enter the password for the user
  • Fill in the User and Group settings, the help for each section should be adequate to guide you, but here is an example screenshot


  • Click Save
  • You should now be taken to a page with the LDAP strategies listed.
  • Click on Map groups
  • Click on the group you wish to map, for example you may wish for all Domain Admins to have the admin role, or you may want to create a specific AD group to give access to splunk
  • Select the role(s) you wish to add for that group and click save.
  • Return to the Splunk login page and log in with your AD credentials


Enabling Active Directory authentication for Splunk is quite simple and allows you to leverage Active Directory for all user access.

Enabling Active Directory Authentication for Splunk

Posted in Tech Tagged with: , , , , , ,