Today we had a policy failure on a newly built SQL server. The policy verifies that every SQL server has a backup file stored on a remote directory. Simple enough, but in the past, this policy has been prone to permissions errors.
In case you would like to implement something like this on your system, the dumbed-down condition of the policy is below:
ExecuteSql('NUMERIC', ' --Get DBs on server DECLARE @DBCount INT,@File VARCHAR(250) SELECT @DBCount = COUNT(*) FROM sys.databases WHERE database_id <> 2 SELECT @File = ''\\BACKUPSERVER\BACKUP\''+CONVERT(VARCHAR(100),SERVERPROPERTY(''MachineName'')) --Insert DirTree results into a table variable to return a numeric result DECLARE @Check TABLE (SubDirectory VARCHAR(250), Depth BIT, [File] BIT) INSERT @Check EXEC master.dbo.xp_dirtree @File,1,1 --Numeric result for easy policy evaluation 1=Pass if( # files = # dbs ) IF((SELECT COUNT(*) FROM @Check) = @DBCount) SELECT 1 ELSE SELECT 0 ')
The dumbed-down policy checks a backup server, which has a folder named for each SQL server, to see if it has the correct count of backups. Each folder should have a backup file for each database, except for TempDB. This isn’t exactly how we do it, but it gets my point across and it provides a good example of how a moderately complex policy works using
ExecuteSQL. There aren’t a lot of examples on the web, so I try to provide as many as I can.
The new server wasn’t processing the policy correctly. My first idea was, “Well, there’s no files!” Easy! Problem solved! Or not…the files were in the folder, and the number was even correct. So it wasn’t the backups or copy failing. That meant that there was something wrong with the permissions, the problem was, what?
I verified the logins were all present in SQL by reviewing Security -> Logins through Management Studio. This was a new server, so I assumed it would be something simple like that. The account in question was the SQL Server service account. Policy Based Management and
xp_dirtree use this account. The account was in SQL and it had all the proper permissions. OK, next thing.
The folder where all the files are stored is a shared folder, so I went into the Properties of the folder and verified the account that was shared. Everything looked right there. Every time I’ve had an issue with the permissions of
xp_dirtree reading servers, it’s been because the service account in the shared folder was wrong, so my next stop was checking out Active Directory.
Drilling down through Active Directory, I found the account used for sharing the folder. This account had every SQL service account as a member. It looked fine in AD, it looked fine on the Shared folder. Everything worked perfectly except on this new server. What was wrong?!?
At this point, I just decided to throw a hammer at it and throw in extra service accounts until hopefully one worked. Eventually I added the SQL Agent account. Suddenly, the policy passed. The new server could read the backup directory! So I took the normal account out of the permissions. Now everything failed except this new server. Wait, what…?
I logged into the new server and loaded up SQL Configuration Manager…
When the server was setup, someone switched the accounts. SQL Server was running using the SQL Agent service account. SQL Agent was running using the SQL Server service account. I restarted each with the correctly corresponding service and account. I then fixed the permissions for the shared backup directory back to the original setup.
Voila! Everything worked perfectly!
The Moral of the Story
Make sure new servers and settings are configured properly before trying to fix things that weren’t broken before the new stuff existed. I’m sad to say it took me so long to understand the problem. Even with other DBAs chiming in. None of us thought about checking the Configuration Manager…that or they assumed I had already checked that.
Sometimes, it’s the little things that get you…OK, normally its the little things that get you.