Fortunately for me, the database was in an Availability Group so after recreating the LUN and attaching it, I restored the database from the read-only secondary. The steps I followed were:
- Didn’t panic
- Removed the database from the Availability Group (I’m not convinced this part was necessary, but I am loathe to try it again)
- Recovered the secondary database
- Removed the secondary database from log backups (we back up on the secondary)
- Created a copy_only full backup of the secondary database
- Created a copy_only log backup of the secondary database WITH NORECOVERY (this puts it in a restoring state)
- Restored both backups on the primary
- Recovered the original primary WITH RECOVERY
- Joined the database back to the Availability Group
- Added the secondary back to the log backup job
It was a fairly small database, so it was really fast and nobody even noticed anything had happened. When my team mates returned from lunch, we all had a good laugh over it and shared a few other war stories.
Yesterday may have been a Tuesday, but for me it was Monday all day long! I made a mistake that could have been a huge mistakes if our team was not so diligent about having redundancy in the database system (Availability Groups, backups, etc). The key here is that I didn’t panic, I worked out a solution logically in my head, and implemented it. We all make mistakes, but if you have prepared and can keep your cool and correct them, you’ll almost always come out of it looking good.