Who Made the Backup

As DBAs, we often obsess over backups. When was the backup made, how big is it, how quickly can I restore, can I restore it, etc. But sometimes, we need to know who made a backup.

Shouldn’t that be me? Well, of course, it should be you. But sometimes, it ain’t.

A few months back, I was asked to troubleshoot a SQL Server where they were getting disk full alerts. After a few quick checks, I was able to see it was the backup LUN that was filling up.

When I looked in the folder, I noticed there was a FULL backup file with an unusual timestamp. Most of the backup files were all created around the same time of day, but there was one with a completely different time. Next, I looked in the job history and this matched what I saw at the file system level.

OK, so now I knew what caused the space alerts, I still didn’t know who had caused them. Time to dig into the msdb database. msdb is a system database that contained all the job history for your SQL Server. A quick query revealed to me who it was that ran the ad-hoc backup.


-- look for who ran a manual backup
select top 30
s.database_name, s.user_name, s.backup_start_date,
m.physical_device_name
from msdb.dbo.backupset s
join msdb.dbo.backupmediafamily m
on s.media_set_id = m.media_set_id
where s.type = 'D'
and s.database_name = 'YOUR_DB_NAME_HERE'
order by s.backup_start_date desc
go

Who Made the Backup

Who Made the Backup

Next, I wanted to know how the user had access to create the backup. Let’s look at security. The first place I typically look is, who has admin access.


-- check server role membership
select l.name,
-- 'People with elevated access' as 'name',
l.sysadmin, l.securityadmin, l.serveradmin, l.setupadmin,
l.processadmin, l.diskadmin, l.dbcreator, l.bulkadmin,
l.createdate, l.updatedate
--, l.*
from sys.syslogins l
where l.sysadmin = 1
or l.securityadmin = 1
or l.serveradmin = 1
or l.setupadmin = 1
or l.processadmin = 1
or l.diskadmin = 1
or l.dbcreator = 1
or l.bulkadmin = 1
order by l.name;
go

Who has Admin Access

Who has Admin Access

And there’s my new best friend. Sometimes referred to as the Extra DBA, or People Who Can Get You Fired. In this case, there were several people who had been granted admin access. Time to start pruning some permissions.

The biggest hole on any system is extra people who have admin access. I see this all the time; perhaps a manager needs to run reporting queries, or developers need to maintain and troubleshoot an application.

When deadlines are looming and people are standing over your shoulder, the simplest fix often seems to just grant the person admin access. You’ll remember to remove it later, or when the app is deployed, right?

 

T-SQL Tuesday #41

T-SQL Tuesday

T-SQL Tuesday

T-SQL Tuesday is a rotating blog series brought to life by Adam Machanic. Basically, a different person is chosen to host the topic each month. We all participate by writing a blog post on that topic. And slowly world peace ensues.

How I Came to Love Presenting

This month the host is Bob Pusateri and the topic is how did you come to love presenting. In his invitation, Bob laments the fact that is has been almost two years since the last time he hosted T-SQL Tuesday.

Well, here I am lamenting the fact that it has been almost two years since I posted on T-SQL Tuesday. In fact, the last time I did it, he was the host. I guess there must be something about Bob.

Theatre

Drama Masks

Drama Masks

For me, everything began with Theatre. All through middle school and high school I did Theatre; I even did a little in college. When I took my sabbatical to become a teacher for several years, one of the subjects I did was, yeah you guessed it, Theatre. I helped the Drama teacher run the department, put on plays, and did a few plays with the kids.

Tons of fun.

There is something really cool about taking on a role and letting it consume you, motivate you, and drive you to new places that you didn’t realize were inside you. That, and the cast parties are lot’s of fun.

I also did lots of crew; set building, lights, sound, etc. For my money, the best troupes are the ones where the cast does crew as well.

SWT Computer Club

College

College

I went to college at Southwest Texas and started off as an Anthropology Major / Music Minor. Somewhere along the way, I was seduced by the dark side of Computer Science and endless fun of flame wars on the university’s mainframe message boards.

I became involved in the Computer Club and one year they made me the President. As part of this, I was the public face of the club for several years. I would organize and run the meetings, trick people into paying club dues, throw a few parties, and cajole speakers to come. One time we got Richard Garriott to come!

Really cool guy…

Teaching

Spicoli and Mr. Hand

Spicoli and Mr. Hand

While in college, I also worked as a Teaching Assistant. I taught the lab sections of CS1308 – Intro to Computer Science and CS3409 Intro to Computer Architecture.

At another point in my computer career, I took a sabbatical from IT and became a high school teacher. I taught Computer Science, Math, Robotics, and Theatre. I also helped found Akins New Tech High as a school within a school for AISD.

I tell ya, hanging around kids, you will never feel so old and so young all at the same time.

SQL Server

Now that I’m all grown up, I work with SQL Server; focusing on areas such as Clustering, Indexing, and Troubleshooting. One of the things that has really inspired me in my career is the SQL Community. It is so vibrant, welcoming, and supportive. Once you start getting involved and meeting people you realize you are part of a growing family who will help you anytime you need it, provide mentoring to help you grow, and an outlet for you to give back to the community that gives so much to you.

I’m talking about your local  SQL Server User Group and SQL Saturday! Oh, and also that little thing known as SQL PASS.

Before I discovered these, I wasn’t very happy with how my career was progressing, and not even sure I wanted to continue working with SQL Server. It was totally random that I learned about my user group, CACTUSS, and even more random when I learned about SQL PASS and SQL Saturday.

I attended a meeting, then another, then another, and pretty soon I started to realize that I was part of a SQL Family who wanted to help me fix my career and watch me grow.

After a while I took the plunge and signed up to present at one of the monthly meetings. I was a little nervous presenting to a room full of experts ready to pounce on every little mistake I made. But I soon realized that everyone there had the same questions and experiences that I had, and couldn’t have been more supportive.

I was hooked.

So I reworked my presentation, adding stuff, removing stuff; and then presented it to another user group. During that time I attended my first SQL Saturday in Houston. Simply amazing. I couldn’t believe all these folks were giving up their weekends to put together free training for me. Très cool.

SQL Saturday #97 - Austin

SQL Saturday #97 – Austin

A few months later, Austin had its inaugural SQL Saturday. I immediately submitted my session, and I guess the judges were sleeping or drunk because they picked me. Now the pressure was really on. I reworked my session a few more times and presented it at a user group again.

When the big day came, I couldn’t believe I had a full room and people asking questions; and no one threw tomatoes at me. After I finished, I had several people hanging around asking questions about my session and about problems they were experiencing in their environments. I traded emails and scripts with a few folks and even ended up hiring one of them the following year.

Get Out There

Get Out There

Get Out There

If you’re looking to get more involved in the SQL Server Community there is no better way than to start attending your local SQL Server User Group and sign up to give a presentation.

Don’t worry about being a beginner. I think you will find the User Groups to be appreciative and supportive environments where you can hone your presentation skills and your technical chops.

You might even give your career a boost to boot.

Looking in the Backup History

The other day I had a developer request a refresh from production from a date in the past. It seems they were testing a bug that cropped up and needed to replicate the way the system looked on a particular date.

Since this was beyond our local retention policy, we needed to restore the backup file from the virtual tape library.

Time to make friends with the Backup Administrator…

You want to tell your Backup Administrator the name of the server and the file, along with the date of the backup. This last part is slightly tricky. It is fairly typical for the file you are looking for to be on the following days’ backup.

For example, if you wrote a SQL Backup file on January 1st, it probably isn’t backed up until January 2nd. You may want to spend some time with your Backup Administrators to understand how your backups work and what the procedure is to get a file restored.

So how do you find out what the backup file was called two weeks ago? Query the msdb database, of course.


-- look for a backup filename from the past...

select top 30
s.database_name, s.backup_start_date,
m.physical_device_name
--, s.*
--, m.*
from msdb.dbo.backupset s
join msdb.dbo.backupmediafamily m
on s.media_set_id = m.media_set_id
where s.database_name = 'YOUR_DATABASE_NAME_HERE'
and s.type = 'D'
order by s.backup_start_date desc
go

This query will return the names of the FULL backup files for the past thirty day. You can modify this query as needed.

backup history

Backup History

SQL Server on Ubuntu

I’m excited by the announcement, Microsoft SQL Server is finally supported on Linux, specifically the Ubuntu distribution. Apparently, this is in response to the Chinese government’s choosing of Ubuntu as the state-sponsored operating system of choice.

While the Minister of Technology initially looked at using MySQL in conjunction with Ubuntu, ultimately he wanted a database platform with the best support model, sources close to the minister are quoted as saying.

Then he tried looking at Oracle, but apparently the proper price point could not be derived. 大象爆炸式的拉肚子 (Da-shiang bao-tza shr duh lah doo-tze) which roughly translates to ‘Costs too much, does too little’ the minister was overheard saying during lunch recently.

Chaz Brockwell, lead user experience developer for the Silicon Valley start-up SighberCool exclaimed, ‘This is the biggest development in the past eighteen months. Finally, Microsoft is opening up their flagship product to the rest of us!’ His company is working on a new wearable-computing device that combines Ubuntu, SQL Server, and the iPhone.

The NASDAQ is up 17 points today in light trading, while the Hang Seng trading down.

SQL Server on Ubuntu

SQL Server on Ubuntu

A Reboot Ate My Log Shipping

I was recently contacted to troubleshoot some Log Shipping issues that arose after some SAN and server maintenance. The Server and SAN teams were working on the server; performing some maintenance on the MPIO drivers. After the work was finished and the server was rebooted, log shipping started to get out of sync on this server.

First, Get a High-Level View, Then Dive Into the Details

The best place to get a high-level view of Log Shipping is by using the built-in reports from SQL Server. In SSMS, right-click on the server node, select Reports, and then choose the Transaction Log Shipping Status report. Do this on both the primary and the secondary server in your log shipping configuration.

Log Shipping Report - Primary Server

Log Shipping Report – Primary Server

The Secondary Server’s report is a bit more informative in my opinion…

Log Shipping Report - Secondary Server

Log Shipping Report – Secondary Server

By viewing the report, I could see at a glance that the log backups were running fine, but the copy job was having trouble. Next I looked at the copy job history and saw the following error message:

  • The network name cannot be found.

The name of the share can be found in two locations, the error log, or the Log Shipping Setup Properties. In the error log, look for this text in one of the substeps:

  • Backup Source Directory
  • Backup Destination Directory
Log Shipping Errors

Log Shipping Errors

Or, in the Log Shipping Setup Properties look for this:

Log Shipping Backup Share

Log Shipping Backup Share

From there, I logged onto the primary server and verified that the share was indeed missing. I also verified that the LOG backups were being written properly on the primary server, but missing from the secondary server.

To fix the situation, all that was necessary was to re-create the share on the primary server and grant read-only permissions to the secondary server’s service account.

After that, I manually ran the copy job to start the log backups flowing to the secondary server once again. Then, I ran the restore job and watched the secondary server get back in sync.

But Don’t Rush In

Often, when confronted with a Log Shipping sync problem, a person’s first reaction is to remove Log Shipping, and then reconfigure it. In this case, nothing would have been solved since the LOG share would have still been missing.

Instead, take the time to troubleshoot the problem and try to find the root cause. Log Shipping is a very simple and stable technology and not much will break it. I have never had to remove and reconfigure Log Shipping to get it to work again.

Give Yourself Some Cushion

Another key to success with Log Shipping is to keep the LOGs around long enough so that you don’t break the LSN chain in case of a problem. You want to make sure you keep enough LOGs on hand to cover a long weekend or holiday break in case your DBAs or other staff are not available to respond immediately.

Log Shipping Retention

Log Shipping Retention

My preference is to keep seven days worth of LOGs available. If you can’t get approval for that much space, walk it back a bit, but hold firm at three days worth. Think about the weekend/holiday factor. If management pushes back, remind them how long it took to initialize the secondary server with the FULL backup. Ask them which is worse, a little extra space, or a long delay while you get a 1TB FULL backup copied from one coast to another.

But What About the Root Cause

Oh yeah, so why did the share disappear to begin with? This server had been migrated from one SAN to another and ended up with two different vendor’s MPIO drivers present. So, the old one was slated to be removed. After the work was completed and the server rebooted, all of the shares went missing.

It turns out this is a known issue with the iSCSI Initiator if it is not configured correctly. Basically, the Server Service needs to have a dependency set on the iSCSI Initiator Service. Take a look at kb870964 for some more details on how to prevent this from happening to you.

Where are your indexes located?

When taking over a database or looking at it for the first time, one area I like to focus on is the indexes. There are the standard things, such as index fragmentation, usage, etc. However, one area that is often overlooked is where the indexes are.

But aren’t they in the database?

Yes, but where in the database. If your database only has one filegroup, PRIMARY, then that is where your indexes are. However, some databases have multiple filegroups. They could be for archiving purposes, security segregation, performance, or any number of reasons.

A fairly common setup to encounter is to have a separate filegroup for non-clustered indexes. This is not as common a technique as it used to be, and there is some debate as to whether this helps performance. The point here is not to debate the wisdom of index filegroups, but rather to get a sense of how your database is laid out.

The following script will give you a high-level breakdown of your database and show you which filegroup your clustered indexes, non-clustered indexes, and heaps are living in.


-- index location aggregations
select ds.name as 'FG Name',
i.type_desc as 'Index Type',
count(*) as 'Index Count'
from sys.objects o
join sys.indexes i
on o.object_id = i.object_id
join sys.data_spaces ds
on i.data_space_id = ds.data_space_id
where o.type_desc = 'USER_TABLE'
-- and i.type_desc = 'CLUSTERED'
-- and i.type_desc = 'NONCLUSTERED'
-- and i.type_desc = 'HEAP'
-- and ds.name = 'PRIMARY'
-- and ds.name = 'MyDataFG'
-- and ds.name = 'MyIndexFG'
group by ds.name, i.type_desc
order by 'FG Name', 'Index Type'
go

Index Filegroup Aggregations

What catches my eye in this example, aside from all the heaps, are any clustered indexes that are not in the data filegroup, and any non-clustered indexes that are not in the index filegroup. So I might flag those indexes to be moved to the appropriate filegroup during a future maintenance window.

The next part of the script will give you a detailed list of each index or heap and where it lives.


-- index location details
select
ds.name as 'FG Name',
object_name(i.object_id) as 'Table Name',
i.name as 'Index Name',
i.type_desc as 'Index Type'
-- , o.*
-- , i.*
from sys.objects o
join sys.indexes i
on o.object_id = i.object_id
join sys.data_spaces ds
on i.data_space_id = ds.data_space_id
where o.type_desc = 'USER_TABLE'
-- and i.type_desc = 'CLUSTERED'
-- and i.type_desc = 'NONCLUSTERED'
-- and i.type_desc = 'HEAP'
-- and ds.name = 'PRIMARY'
-- and ds.name = 'MyDataFG'
-- and ds.name = 'MyIndexFG'
order by 'FG Name', 'Table Name', 'Index Name', 'Index Type'
go

Index Filegroup Details

You can modify the where clause to filter the results based on the type of index or the specific filegroup you are interested in.

Go forth and analyze…

SQL in Seattle

This past week, I went to Seattle to attend the SQL PASS Summit. I had a great week learning about SQL Server, exploring Seattle, and networking with my fellow SQL Server professionals.

Pikes Place Market

You can’t talk about Seattle without mentioning the weather. Personally, I found the weather to be fairly nice. Recently, I had purchased a new rain jacket for the Austin City Limits Festival, and was looking forward to putting it to use in Seattle. Funny, I left Texas only to have it rain while I was away. In Seattle, it only rained once, any only for a few hours. Not really that cold either, a fleece was all you needed.

Announcements

SQL Server 2012 Service Pack 1 has been released. It’s time to update your servers. If you’ve been sitting on the sidelines, no more excuses.

Heckaton, as in you have a heck-o-ton of data and need things to be quick. This is an in-memory database technology that promises to greatly improve performance.

Column Store Indexes will now be updateable and able to be the clustered index. This will help performance in Data Warehouses. Nice.

And perhaps the biggest announcement…

Business Analytics PASS, aka BI PASS

There will be a BI-focused Data Analytics Conference in Chicago next year. I think this is a good thing. I like BI, but it is has grown to the point where it needs its own conference. However, I would like to see PASS retain 10-20% of BI sessions at the Summit. At the same time, BI PASS should also retain a similar percentage of DBA content.

SQL Family

One of the best parts of PASS is reconnecting with old friends and making new ones. I especially enjoyed seeing my old SQLskills classmates and SQL Cruise shipmates. Because of the intimate nature of these two courses, you build a solid relationship with your peers. This was very evident seeing each other at PASS over a year later and picking up right where we left off. It seemed everywhere I turned, I was running into someone from one or both of these two classes.

I met so many new people from countries all around the world. One interesting person I got to spend some time with was Benjamin Nevarez. If you haven’t checked out his book, Inside the Query Optimizer, go grab a copy and prepare for your brain to melt.

Session Highlights

There were so many cool sessions, but these were some of my favorites…

Klaus Aschenbrenner had a pre-con where he took you through his Trouble Shooting Methodology. I enjoyed how he showed us some of his performance tuning techniques and a few different ways to deal with SQL problems.

Got Security?

Andreas Wolter is a security guru and he did not disappoint. In this breakneck session we took us through security basics, a few encryption gotchas, and got into some advanced topics. This was a lot to squeeze into a seventy-five minute session, so I hope to see him in a half-day session in the future.

Both Andreas and Klaus were in my SQLskills rotation last Summer. So when I saw their names on the schedule, they were some of the first sessions I picked. Both are top-notch SQL guys. If you get the opportunity to see one of them, go for it!

Adam Machanic took us deep into parallel execution plans and showed us how to tell what was going on and why. Even if you don’t have parallelism issues, everything we learned can be applied to ‘normal’ query plans and tuning, as well. This was my favorite session of the week.

Waddayamean the server failed over?

Brent Ozar had a couple of nice sessions this year. In one, he discussed some of the projects where he has used SQL Server 2012 Availability Groups. He spoke about the need to test your DR plans befre disaster strikes so you will know if they work. We were able to discuss some real world examples involving the Stack Exchange sites and Hurricane Sandy.

Renaissance Festival Anyone?

Later in the week, Brent demoed a new version of sp_blitz which adds Plan Cache Analysis to the tool. I enjoyed that he actually went behind the scenes and explained what the code was doing instead of just showing us the finished product.

Jason Strate had an excellent session where he showed us how to dig into the Plan Cache using XML Queries. I love using the Plan Cache, and feel it is an often overlooked area of performance tuning. He took it to the next level showing us how to do this more efficiently using XML Queries. I’ve been chatting with Jason for a while over Yammer, so it was great to finally meet him face to face.

Suffering from Success

I’m starting to think PASS Summit is suffering from its own success. This year is the largest PASS Summit yet, but I think it has gotten a bit unmanageable. The attendance is large enough that the session rooms are spread out on opposite ends of the convention center.

Many of the sessions I tried to attend were filled to capacity and I was not able to get in. In one case, I had to go to my 4th choice session. For some of the popular sessions, people were lining up 30-45 minutes early. So if you wanted to guarantee you had a chance to see some of these sessions, then you either had to leave the previous session early, or skip it altogether.

To make matter worse, several times there were popular, filled sessions in smaller rooms; while the large room 6E was empty. I would understand the occasional miss, but this seemed to happen quite a few times.

Furthermore, with so many people it was difficult to navigate the crowds or find time to meet new people. Everyone seemed to be rushing to their next session so they could get a seat.

When I consider all of this, I think having a dedicated BI-focused conference will help alleviate the crowding problems that I experienced this year.

See You Next Year in Charlotte

Next year, Pass will be held in Charlotte, North Carolina October 15-18. I think it will be cool to check out a new city for PASS, but I am a little bummed about the dates. It is coming right on the heels of the Austin City Limits Festival so it might be difficult for me to attend. We shall see…

Gimme the IOPS and no one gets hurt!

Disabling Time Sync in VirtualBox

Have you ever need to control the date or time on one of your VirtualBox Guests? Typically, the Guest OS gets the date and time from the Host.

Usually, this is fine. However, there are times when you may wish to control how the Guest operates. This could be for a variety of testing reasons, particularly with Edge-Case Testing.

For example, Active Directory Authentication, Mirroring and Witnesses, Replication, Clustering, etc. In each of these applications, if the servers involved get beyond five minutes from one another, they may cease to operate normally.

By disabling time synchronization between the Host and the Guest, you will be able to move each servers’ time backwards and forwards while you observe how they behave. Will your application continue to function? Will you be able to trap error messages and inform the user?

VBoxManage Utility

VirtualBox includes a command-line utility, VBoxManage, that will allow you to control every aspect of your Guest VMs.

On a Windows host you will find this in the following directory: C:\Program Files\Oracle\VirtualBox

This example lists all the VMs that you have defined on your VirtualBox installation.

.vbox Configuration Files

Before we dive into this utility, another piece we need to look at are the VirtualBox Machine Definition Files. These are XML file which control how your Guest VMs operate. Basically, when you use the GUI to setup and control your VM, all the various settings are stored here.

On my Host, these files are stored in C:\Users\<USERNAME>VirtualBox VMs

You will see one directory for each VM that you have defined. Within that folder you will see a file with the .vbox extension. Before we go too much further, go ahead and make a copy of that file. This way, if something happens while you are working with it, you can restore your VM’s configuration.

Now, use Notepad to open the .vbox file and poke around a bit. Just be careful not to modify anything yet, that comes later. You should see something like this…

 

This is an XML file that contains all the configuration information for your VM.

Let’s go take a look at a couple of sections and see what they do.

ExtraData

As you might guess, this section contains ‘extra’ data about your Guest VM.

BIOS

And this section contains information on how the virtual BIOS operates.

OK, now that we’ve seen that, it’s time to get to work.

Disable Time Synchronization

First, let’s turn off Time Synchronization.

http://www.virtualbox.org/manual/ch08.html

VBoxManage SetExtraData “YOURVMNAME” “VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled” “1″

Note, if your VM has spaces in its name, then surround it with quotes.

If you want to turn Time Synchronization back on, use a ’0′.

VBoxManage SetExtraData “YOURVMNAME” “VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled” “0″

If you don’t remember the names of all your VMs, try this first.

OK, let get down to business.  To turn off Time Synchronization, use this command:

Now, the Extra Data section of your .vbox file should look like this.

So now when you start your VM, you will be able to change the Date and Time. Presumably, one could simply edit the .vbox file to make this change.

Time Offset

OK, but what if you would like your VM to start with a different time. That’s easy to do, as well. For this step, there isn’t a command-line option, so we need to edit the .vbox file manually.

Find the BIOS section, then edit the TimeOffset line with something similar…

<TimeOffset value=”-123456789″/>

It should look like this when done.

The number represents milliseconds so adjust as necessary for your needs. Also, the number can be positive or negative.

Et voila! Your Guest VM now has a completely different time from your Host.