Round-the-clock and without interruption, Sharp HealthCare is dedicated to providing top-quality patient care. To further this end in 2014, Sharp embarked on a mission to improve the integrity of their IT environment. To facilitate the critical task of building a more robust infrastructure, the San Diego-based healthcare organization partnered with leading IT advisory and consulting firm T2 Tech Group™.
After Sharp had taken the initiative to begin work on the buildout of a redundant architecture, the provider realized the potential ROI of augmenting their proficient application team’s abilities with T2 Tech’s premier project management services. Working together, Sharp and T2 Tech successfully completed the design and implementation of a redundant architecture and disaster recovery site at Sharp’s Rees-Steely Medical Group facility (RSM) in Sorrento Mesa, California. Leveraging the skills of Sharp’s adept IT team along with T2 Tech’s effective management staff and methodology, the two organizations efficiently tested failover for Sharp’s key applications without causing any interruption to business.
Completing this project marked the end of phase one in a large, two-part data center transformation. Along with completing the redundant buildout at the RSM, the second phase of the project includes a migration out of the Sharp Operations Center (SOC) data center in San Diego. By utilizing an agile-based methodology and rigorously analyzing Sharp’s application environment in phase one, the Sharp and T2 Tech team will be better prepared to optimize efforts during phase two. When both phases are complete, Sharp will have redundancy to support failover along with a state-of-the-art, highly-secure primary data center.
Facilitating a Failover Solution
To select a partner in finding a failover solution, Sharp underwent a rigorous selection process. Upon reviewing a host of potential candidates, Sharp chose to work with T2 Tech. The consulting company was chosen because of its highly-experienced team and effective project management methodology.
“With entire communities depending on us for their medical needs, Sharp HealthCare must maximize its use of resources and avoid downtime in its critical software and services,” said Teri Moraga, VP of IT Infrastructure, Sharp HealthCare. “When it was time to find a project management partner to help reshape our approach to IT disaster recovery, T2 Tech Group was selected because of their experienced team and unique project management methodology. Helping drive the successful completion of our redundant buildout, their project managers allowed our very capable application team to work more efficiently, and they helped provide crucial knowledge and processes that will streamline our future move to a new primary data center.”
For the project, T2 Tech lent crucial project management expertise that streamlined efforts. Working side by side, the Sharp and T2 Tech team completed the system build, the testing and the documentation of the redundant architecture implementation.
The team worked to secure immediate failover capability for Sharp’s critical applications and services. During the project, they put a top priority on maintaining hospital operations and delivering uninterrupted services to patients and staff. For more than 60 of Sharp’s applications — including the organization’s Cerner Millennium EHR software — the team created an architectural diagram, mapped out a failover process, built out and configured solutions at the failover facility, and developed runbooks and playbooks. The organizations went through the process of testing failover procedures and functionalities for each of the applications without taking down the production site or causing any disruption or degradation of normal hospital activity.
When collaborating with Sharp to develop the disaster recovery solution, T2 Tech worked to ensure the healthcare organization’s IT staff could continue to complete their existing operational responsibilities during the redundant buildout. At the same time, T2 Tech’s project-management services helped teams incorporate lessons learned before close out. These benefits gave Sharp’s team flexibility, and it saved valuable time for its busy engineers.
During all project phases, T2 Tech’s project managers helped coordinate the team using its hybrid methodology that balances upfront planning with an agile-based execution approach. Within the framework of the management method, T2 Tech parsed long-term project plans into two-week iterations, held daily meetings with core team members and provided accurate stakeholder reporting. These processes yielded significant benefits, including simplified goals, effective collaboration and transparent communications.
The short iterations allowed for flexibility while Sharp’s engineers were juggling a multitude of important initiatives. After every two-week iteration, team members could decide whether or not to commit time to the project depending on their operational requirements. They could also place part of the project on hold if they had more pressing duties to attend to. The iterations also facilitated improvements to team efforts. Each iteration within the management framework represented a vertical slice of the entire project. Using the framework, the team could work all the way through one application thread and apply lessons learned when starting on the next application. By applying lessons learned before the entire project’s completion, the team was more efficient.
Assessing Sharp’s Architecture For Failover
During the assessment stage, the Sharp and T2 Tech team evaluated the architecture for replication and failover by reviewing documentation and conducting interviews with staff. The evaluation focused on working with Sharp to understand the rationale of key architectural decisions and identifying opportunities for improvements. As part of this effort, Sharp worked with
T2 Tech to evaluate key areas, including the following:
- Single points of failure
- Hardware and software lifecycles
- Capacity and concurrency capabilities
- Other services dependencies
- Anticipated recovery time and point objectives (RTO and RPO)
- Key performance metrics and service-level agreements for each identified application
- The wherewithal to test failover of applications without impacting production
- The capability to recover back to the primary data center after failing over to the secondary data center
During the assessment process, Sharp and T2 Tech also evaluated Sharp’s progress on building redundant infrastructure. In addition to involving more documentation review and staff interviews, this segment of the assessment included hands-on validation and an evaluation of testing procedures.
Enhancing Designs For A Redundant Architecture
After assessing Sharp’s current IT environment and the healthcare organization’s progress on developing a redundant architecture, the team developed a plan to make Sharp’s existing IT environment more resilient. This required a solution that accomplished important goals, including the following:
- Ensuring the services and applications within the scope of the project could be fully supported in either the primary or secondary data center if one of Sharp’s data centers experiences an outage
- Avoiding any impact on production during the implementation and testing process
- Providing an RTO of no longer than four hours where feasible
- Consolidating and standardizing infrastructure as appropriate
- Reducing application dependencies on dedicated hardware where possible
Developing An Execution Plan And Implementing The Solution
Sharp and T2 Tech worked together to understand the current state of the redundant plan. Then they further developed and executed a plan to complete it. This process included the following actions:
- Identifying additional hardware and software needs
- Installing additional hardware and software as required
- Configuring replication, load balancing and failover tools
- Ensuring that appropriate backup and recovery processes were in place
- Testing application failover and redundancy where production will not be impacted
Plans For An Improved IT Ecosystem
Sharp’s investment in a redundant architecture marks the first phase of Sharp’s data center transformation project. The next phase of the project includes a migration out of the SOC, Sharp’s old data center. To efficiently implement the next phase, the team will leverage the application environment knowledge and process improvements gained in phase one.
Working together on the second phase, Sharp and T2 Tech’s efforts will help Sharp implement an ideal, fully-redundant architecture that supports its long-term requirements. Aligning with the overall goals of the two organizations, the data center transformation project will ultimately contribute to improved IT services and healthcare for patients throughout San Diego.