Several months ago CrossRealms was asked to help one of our hospital customers create a Business Continuity Program. Great. BC Program initiation is usually at least one quarter of the battle of making BC a reality. Even if you have a dedicated BC champion it can be an uphill climb to get management buy-in for the investment. Luckily for our customer-champion, her management team was aware of the value and, furthermore, compelled by regulation and audits within their industry. Today, we have a living, breathing and amendable BC Plan in place that is executable. We’re currently conducting testing and then plan to start the training and awareness program but the majority of this program is complete.
So ….. How did we do it?
Here is a brief overview of Business Continuity Programs:
First of all, we didn’t re-invent the wheel. At CrossRealms, we subscribe to the professional practices that DRI International promotes for BC. These are an accepted, common body of knowledge within the BC profession that describe the tasks, procedures, and activities required to successfully create and run a BC Program.
In keeping with these practices we started with the Program Initiation and Management. As I mentioned, we were able to skip a few steps because this organization already established the need for a BC Program. Senior management was willing to support the program. The BC champion (coordinator and planner) was our eyes and ears on the inside. She recommended a group of key players to be present at our kickoff meeting. They became oursteering committee and were responsible for guidance, oversight and determining the scope of the project.
We performed a Risk Analysis. I contacted their state’s Emergency Management Agency and tabulated data on the last few decades of state and federal emergencies in their county. I ordered the crises in decreasing order of probability and presented the findings to the steering committee.
We then performed a Business Impact Analysis (which I believe is the most important part of BC – I’ll save that for another blog). I created a questionnaire that gave us answers about each department or business unit’s most critical processes. We learned about their recovery point and recovery time objectives as well as how these processes fit in the overall flow of the organization. We essentially painted a picture of the organization by its most important processes. The BIA became iterative to the extent that we saw “grade inflation” of some of the departments’ processes, so we had to re-work some of the results to more appropriately match the results from other departments. If you want to save yourself some re-work, you can have your department heads fill out their data in a collaborative document management tool like SmartSheet or SharePoint, as some of our other customers did. You may find that this will help reduce grade inflation. We then determined the most critical staff and infrastructure dependencies for these processes. Ultimately we determined RPOs and RTOs for each major process. The last step was to conduct multiple tabletop exercises to determine how all of these processes were interdependent and how they were to be recovered. What was initially met with some eye-rolls and skepticism ended up being a valuable study in how so many teams were responsible for recoveries in various types of scenarios.
Based on the findings of these two analyses and the table-top exercises we were able to come up with a Business Continuity Strategy. We did a gap analysis on where the organization was currently, compared to where they collectively decided they needed to be. Their tape backup system did not meet their RPOs or RTOs. It was determined that local backups and recoveries were necessary for RTOs and a remote replication site was required for DR. We ran a Proof of Concept regarding their backup footprint and replication to our cloud infrastructure for costing purposes and then proceeded to implement the new solution.
During the entirety of these steps, we re-created a BC Plan for them that was more suited to quicker recoveries and had specific information about their processes and people. Their original BC Plan was fairly impressive compared to many organizations we have helped, but still wasn’t adequate for their purposes. The new plan is never going to be complete because their organization is always changing and processes are always being created, expired, or replaced. This was also the most time-consuming step and updates are still being made to the plan as I expect they will indefinitely. I also have to note that our customer-champion was extremely involved in the plan updates and became more aware of every piece of the organization as a result. I’m sure she feels more connected and capable than ever before.
I don’t want to imply that there weren’t challenges and issues we faced along the way. Getting staff buy-in was a challenge until the CEO communicated the program value to them. The Emergency Management Agency of their state was extremely unresponsive for our low-priority requests. The BIA questionnaire responses not only had grade inflation, but sometimes didn’t even address the spirit of the questions. Our customer-champion already had a full-time job as the hospital’s IT director and was often forced to prioritize other matters ahead of our program. And last, but not least, we had three customer technical liaisons during the course of the project because the first two resigned. We were clearly able to overcome these challenges with some rework, persistence and reliance on their management. As is usually the case, fighting the occasional fire always takes precedence over non-emergency planning projects, so you can expect scheduling issues with your program as well.
That’s where we are today. We are currently testing our new infrastructure and recovery processes. Our next step is to create a Training and Awareness Program to make sure all relevant personnel can assist during an outage or recovery event. I also expect that some of the trainees will train the trainers and help our recovery runbooks become even more accurate.
That’s it. That is essentially how we got one hospital’s BC Program up and running. I am currently working on 5 simultaneous BC Programs from industries that range from healthcare to legal, to finance to retail and I learn something new every time I do this. No two customers are ever the same. Some want to skip as many steps as possible and jump right into the strategy and infrastructure implementation. Some want the most complete and defensible BC Plans in their industries.
Of course, the devil is in the details, so I don’t feel like we’ve given away our secret sauce, but I hope this helps you understand how we take our customers from kickoff to resilience.
We’ve Got Your Back(up)
CTO Storage and Business Continuity