The rollback plan… what to do when things go wrong to get back to where you were before the deployment or implementation. I’ve seen too many cases where a rollback plan is required, but it’s never tested prior to a deployment. Because it’s never tested, it rarely gets properly developed. And when deployments go well or well enough that issues can be fixed during the deployment, the deployment’s rollback plan doesn’t get used. Eventually, folks wonder why we do one. And they wonder until the one deployment when things do go wrong and they don’t have a properly tested rollback plan. And they can’t rollback.
That doesn’t happen in modern enterprises? I beg to differ. Recently I was talking with a representative from an organization that I receive services on. The representative apologized profusely because they couldn’t pull up my records. They and many other representatives were locked out of their system. Talk about a nightmare. What’s worse was the estimated time when their ability to access their system would be restored. The representative indicated it would be one, possibly two, days.
If the estimated time to fix the issues in the deployment are measured in days for something critical like authenticating to the system, it’s time for the rollback plan. The fact that the estimated time was in days tells me either there wasn’t a rollback plan or it wasn’t fully tested and they weren’t able to execute it when the implementation failed. One to two days for an identity platform sounds like they chose to roll forward, likely meaning they had no other choice.
Now I could be wrong. They could have had a rollback plan that they did fully test but management decided to roll forward anyway. I’ve seen that happen. If you’ve been in information technology long enough, you likely have, too. But regardless of what they did do, hopefully this real life scenario has convinced some of the imperative to have a proper rollback plan and to test it before the production deployment.