During a recent vCenter deployment using the VCSA I ran into an error I hadn’t run into before with the VCSA (or vCenter/SSO on Windows for that matter). After an error free install and setup wizard, I logged in to vCenter as [email protected] to set my roles and assign my AD groups permission. However I noticed that there was no identity source for my Active Directory domain, no problem add it in, boom now hop on over to my vCenter permissions tab and get people vCentering. This is where I ran into errors. When trying to search for a user I received a pop up that said
Cannot load users for the selected domain
Before I ran the setup wizard, I had SSH’d to the VCSA did some pings and digs to make sure the network bits were flowing properly and everything seemed fine. I could ping and dig both local and remote AD resources so I was confident that was all working fine. Easy fix I assumed, so I headed over to the global KB search tool known as Google and was lead to this KB, http://kb.vmware.com/kb/2033742 which suggested checking DNS, time synchronization and joining and re-joining to the domain. I manually re-checked DNS records were present, that the AD join process had worked correctly and the account was still enabled.
Looking through the /storage/log/vmware/sso/ssoAdminServer.log I saw several exceptions with the following error (stripped excess text)
Failed to establish server connection
I searched the KB again for this error, but all I found was problems related to accent characters which wasn’t my thing. At this point it was worth while to open a support case. I wanted to make sure I had all my tier 1 support boxes ticked so I rebooted, verified I could ping/dig records, went back to the Identity Source page removed and re-added the domain and went to look up an AD group to get the error message and… all my users were listed. I’ve not quite tracked down what was wrong, but if you are getting this error, and you know your DNS was square try just re-adding the sdentity source.
Posted in Tech Tagged with: active directory, Cannot load users for the selected domain, dns, error, identity source, Security, Shared, SSO, Technology, vcenter, vcsa, Vendors, Virtualization, VMware, Windows
Splunk allows you to create either real time or scheduled alerts, and while real time alerts would seem logical they can be quite CPU intensive. In fact, some suggest limiting to 1 real time search per CPU core even though the default limits will be about 3 per core. You can see more on the limits.conf and how to change it here. Rather than creating real time alerts, you can set or change existing alerts to run on a schedule. In reality, most monitoring software works on a polling interval anyways so this is not far from what you are likely doing today with something like Nagios.
The process for creating or changing Splunk alerts from real time to scheduled is fairly straight forward. By default, however, the shortest time period Splunk provides is 1 hour. If you want to schedule Splunk alerts to run more frequently you will need to use the “Run on CRON scheduler” option. Cron schedule examples give me a bit of a headache, but since we only have to worry about a per minute time interval since Splunk provides other options here is a quick how to on how to set these up, or change them.
Changing a Splunk Alert from Real Time to Scheduled
*/5 * * * *
While alerts can be swell, they will have an impact on your server and, if you have to many alerts, you might not actually receive any of them, which would be bad. Scheduling alerts is an easy way to make sure your Splunk server is not resource constrained.
Here is a quick guide to adding additional Windows Active Directory groups for Splunk authentication to allow users to log in. You can see how to configure Splunk for Active Directory here. If you have not already done so, create the Active Directory group that you want to grant access to and add the users you want to to access Splunk into the group.
If you are not able to log into Splunk, do the following:
Having been working on Splunk for the last few days, one of my last hurdles was to easily monitor, present and alert on Active Directory events such as account lock outs, group changes etc. Low and behold Splunk has an app called Splunk App for Microsoft Windows Active Directory – problem solved! Well as Lee Corso would say – “Not so fast my friend!”
The documentation for the Splunk App for AD starts with a warning, that its complicated to install…meh they just want PS revenue right? The documentation starts off straight forward, mostly configuring your audit policies to log relevant information but ends up in a twisty turvey rabbit hole of horror! So much so, that the sales rep even suggested I use another app! So, here is how to actually monitor AD events, not using the Splunk App for AD.
The app you should look at, of course if it meets all of your requirements, is the Windows Security Operations Center app. This app is not available for direct install from the app market directly from Splunk, you will need to download it. Also it should be noted that as of this writing its only certified to work Splunk 5, however I am testing it with Splunk 6 and having only a few minor problems that I suspect are related to my audit policy versus the application not working. In my limited test environment, its generating just under 100MB per day (about 75 user accounts and tests scenarios like adding/removing/deleting accounts from the domain admins group). For purposes of this walk through, I’ll assume you have a central Splunk server already installed and the Universal Forwarder installed on your Domain Controller(s).
The Windows Security Operations Center App for Splunk is an easy to install and configure app, unlike the official Splunk App. If you have a need to monitor AD events, this app should come in handy. Of course at the end of the day its still Splunk, so if you can find it in your event log, you can find it in Splunk.
A quick how-to for enabling Active Directory authentication for Splunk:
Enabling Active Directory authentication for Splunk is quite simple and allows you to leverage Active Directory for all user access.