CHGI Transition
To improve the sustainability of the CHGI cluster compute and storage infrastructure, the decision was made to integrate this infrastructure with the University of Calgary's High Performance Computing site managed by Research Computing Services (RCS). This will allow CHGI users to leverage the extensive RCS staff and infrastructure. For more information on RCS services, please see the RCS Home Page .
What does this really mean?
PIs, researchers, or students using any of the following storage or compute infrastructure will be contacted by analysts from the RCS team on moving their workflows to ARC:
- Synergy compute cluster
- Theia compute cluster
- Galaxy compute
- Storage on /gpfs and /tiered
If necessary, analysts from RCS will provide you with one-on-one time to support you in transitioning your workflows to ARC. RCS will continue to provide support to you and your workflows on ARC after the transition is complete.
Project Objectives
The primary objectives of this transition are to:
- Develop a transition plan for the equipment and technical services CHGI offers
- Define ongoing maintenance governance, processes, data privacy, retention and other related processes
- Transition data, users and workflows to RCS managed infrastructure
Status
User account status in CHGI and transition
| What | Total CHGI Accounts Count | CHGI Accounts Active | CHGI Accounts Locked-out | With ARC Accounts | Without ARC Accounts | Logged In - Last 12 months | Logged In - More than a year ago | Not logged In | IT Account enabled | IT Account that is disabled | No IT Account |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Baseline Dec 04, 2019 | 379 | 335 | 43 | 76 | 259 | 139 | 180 | 16 | 243 | 43 | 49 |
| 88% | 11% | 23% | 77% | 41% | 54% | 5% | 73% | 13% | 15% | ||
| Current month statistics | 399 | 218 | 181 | 88 | 130 | 189 | 51 | 22 | 196 | 12 | 10 |
| 55% | 45% | 40% | 60% | 87% | 23% | 10% | 90% | 6% | 5% | ||
| Changes from baseline Dec/2019 | 20 | 138 | 12 |
Cluster status:
Hyperion - moved to ARC, users transitioned.
Theia - moved to ARC, users transitioned.
Crick - Restricted access.
Synergy - moved to ARC, users transitioned.
Background
The Center for Health Genomics and Informatics (CHGI) is an initiative that provides a wide range of next-generation genome sequencing services and access to high-performance bioinformatics for sequence analysis to all University of Calgary researchers. New researchers are provided a basic level of networking and storage to make use of CHGI services but also have the option of purchasing additional servers and storage to integrate within the CHGI network to upgrade their usage capabilities. Many of the pieces of the CHGI network equipment have been purchased by researchers and institutions.
The CHGI in collaboration with RCS is working on a planned transition of CHGI data, services and workflows to infrastructure managed by RCS. This will allow CHGI researchers to:
- Focus on core duties instead of managing IT services
- Open the potential for scalability in the infrastructure
- Standardize several common services to RCS offerings.
Approach
The project team interviewed 15 Principal Investigators and researchers in the Analysis phase to understand the current use of the equipment and services provided by CHGI. This engagement took four weeks during which we interacted with the CHGI manager, System Administrators and PIs that leverage CHGI's equipment for their research. An important aspect in the interviews was the investment made in equipment.
The project team took the information available and discussed the technical options based on current RCS practices.
Implementation
Tasks identified as low impact include:
- CHGI workflows and services that can be transitioned into existing RCS infrastructure
- CHGI Workflows and services with very little impact and costs.
- CHGI equipment which are off or almost off warranty and have low dependence on storage.
All other tasks, CHGI web related services, or workflows heavily dependent on the the IBM storage system will require deeper analysis and more preparation time. The longer transition period will give researchers extra time to prepare and integrate their processes into RCS infrastructure. Equipment and workflows requiring this extra work will remain in their current location.
Two years into the transition, a decision whether to physically move any remaining equipment to RCS infrastructure will be determined.
End of Warranty Issues
To address the inevitable problem of equipment falling off warranty and funding ceasing, clear communication to the equipment owners and researchers will offer two options if they wish to continue operations. Researchers can choose either to:
- Transition their workflows onto existing RCS offerings in ARC.
- Purchase new equipment under RCS guidelines.
Engagement and collaboration with RCS in this stage will assist in purchasing new equipment that can be integrated with the rest of the RCS infrastructure. These services and workflows will then be transitioned into standard RCS infrastructure, and the new RCS administrator will manage this by following the service levels described in a signed Operating Level Agreement (OLA). After the workflows have been transitioned, the off-warranty equipment would be decommissioned.
On Going
Continuous monitoring of the transition will guide the implementation and may improve the transition process.
Critical Success Factors
- Minimize impact to researcher’s workflows/processes
- Consolidate services and infrastructure
Scope
In Scope
- Develop a consistent process to transition users, data, workflows from CHGI to RCS infrastructure
- Define ongoing maintenance governance, processes, data privacy, retention and other related processes
- Transition Individual Researchers from CHGI to RCS infrastructure.
Out of Scope
- RCS and CHGI will agree when equipment is ready for decommissioning. CHGI is responsible to put in the requests to dispose of equipment.
Schedule/Milestones
| Milestone | Dates |
|---|---|
| Alternatives Evaluation | Q2/2019 |
| Implementation Plan | Q3/2019 |
| Transition to ARC of appropriate users and workflows | Q3/2020 |
| Analysis of remaining equipment and users | Q3/2020 |
| Additional transition work - dependent on equipment and users remaining | Q4/2020 |