T-SQL Tuesday #13 – Business Is as Business Does

This blog entry is participating in T-SQL Tuesday #13, hosted this month by Steve Jones (Blog|@way0utwest). You are invited to visit his blog to join the blog party. You are welcome to write your own participating blog post for the party or just to read more blogs participating in this month’s theme: What the Business Says Is Not What the Business Wants.

For my part in this month’s T-SQL Tuesday, I am going to reflect on the wise words of that great American philosopher Forest Gump. In the movie with the same name, Forest Gump stated, “Stupid is as stupid does.” The same applies to the part of our organizations that we like to refer to as The Business. We DBAs love to joke and tell stories about the crazy things the Business has made us do. I’ve definitely had to do my share of unfavorable things at the behest of the Business. I commiserate with my DBA colleagues on this matter; however, I am often saddened by some of the things I hear as well. I believe that it is our responsibility to educate the Business and guide them on making informed decisions.

Business Does as Business Knows

Unfortunately, I see a lot of DBAs that just do what the Business tells them to do without any sort of pushback even though they know the results will be bad. To some degree, our job is to do what we’re told, but it goes further than that. I’ve blogged a couple of times before comparing us to doctors. If our doctors merely did what we told them, we’d all be in worse physical condition. If I told my doctor that I wanted a prescription for antibiotics because I have a cold, he’s going to examine me, confirm what my illness is, and then prescribe the treatment or medication that he feels is appropriate. Ultimately, it’s up to me (as his Business) to decide whether to follow his advice or not, but he’s not just going to blindly do what I say.

So Business is as Business does. If your Business is making you do bad things, you need to ask yourself if you are contributing to it. Business will generally have a reason for why they want things done a certain way, and if it is contrary to what you know to be best, then you must make your case for doing it the right way. Failure to do so makes you complicit in the offense and guilty by association. Here are some examples from my past of when I failed to help Business make the wise decision and when I successfully persuaded business to make the right decision.

Bad Business #1

I have written about this before though not on my own blog. My second DBA job was at a company that had never had a DBA before and felt they were past due for one. I took on the challenge of completely redoing the way their development team operated. I was a hybrid DBA performing both administration and development DBA duties. In short, I took on quite a load of work, especially for a DBA career as young as mine. The company was outsourcing their systems administration work to a company in Spokane, WA, where their servers were located. I would be taking quite of bit of the load of the work off of their plate, and the company was spooked that it was a move to replace them permanently. To keep the company happy, the Business informed me on my first day that I would NOT be taking over the database backups from them. I was feeling a little overwhelmed at everything I was taking on, so I just agreed.

Just agreeing to it was a bad move. What makes it even worse is that I knew that I should be taking it over. I knew it was a bad move. What they wanted me to accomplish in my first 3 months was going to be a monumental undertaking, so I let it go. I decided that there would be time to fight that particular fight later.

Well, you can probably guess that disaster struck, and we were not able to recover from it. It affected a very important client’s database in the middle of their busy season, and the whole disk array was lost. As it turns out, there were no backups for this database at all and they had just moved the database to a new server moments before, so the DR processes that I had created to automatically log ship every new database had not run for this database yet. A lot of data was lost and a lot of money was lost for both the comapny and the client. Business looked bad because they had made a bad decision to let the external company keep managing backups. I looked bad because I allowed Business to make the bad decision.

Bad Business #2

When I went to work at the company mentioned previously, one of the things I did fight for is that no SQL code gets put into production without going through our QA Test process and no SQL code goes to QA Test without being reviewed and approved by me. The developers had mixed feelings about this. Some thought it was great and one person even commented that he always looked forward to my code reviews because he never failed to learn something. Others resented that their code now had to be reviewed by this new guy even though they had been coding at the company for years. One developer in particular seemed to despise my mere existence.

Every month the development team set goals for how many new features they would deliver and how many bugs they would fix. One month they were just a wee bit short on meeting the goal, and the CTO and this developer who despised me were up until 2 AM trying to get some code finished and put into production. The bad part is that the code had never been reviewed or tested. It implemented a trigger on a merge replicated table. When users started using this new feature, it blew up. Not only did it blow up for people using the new feature, it killed merge replication. When I walked in, I was bum-rushed by people telling that replication was down and had been for many hours. I quickly tracked down the problem, disabled the trigger, and got replication running again. I then rewrote the process to utilize Service Broker rather than a synchronous trigger that relied on being able to update external tables.

This is definitely a case where I had guided the Business on making the right decision, but the Business decided to go against their own decision and my advice and allow it “this one time.” The Business assured everyone that they would not make that mistake again. The developer who wrote the offending code seemed to like me even less than she did before.

Summary

In a way, you could say that Business is as we allow Business to do. We are not just a tool of the Business. We are the educators and guides for the Business. If you find yourself being asked to do something that you know you shouldn’t do, you need to offer the Business alternatives and intelligent reasoning.