CHGI Transition: Difference between revisions

From RCSWiki
Jump to navigation Jump to search
No edit summary
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
To meet building code requirements, the CHGI Data Centre (DC) in HSC B151 must be decommissioned. To facilitate this change, the storage and compute capabilities housed in the DC are being transferred to the University of Calgary's High Performance Computing site managed by Research Computing Services.
To improve the sustainability of the CHGI cluster compute and storage infrastructure, the decision was made to integrate this infrastructure with the University of Calgary's High Performance Computing site managed by Research Computing Services (RCS).
This will allow CHGI users to leverage the extensive RCS staff and infrastructure.
For more information on RCS services, please see the [[ RCS Home Page ]].


= Background =
== What does this really mean? ==
PIs, researchers, or students using any of the following storage or compute infrastructure will be contacted by analysts from the RCS team on moving their workflows to ARC:
* Synergy compute cluster
* Theia compute cluster
* Galaxy compute
* Storage on /gpfs and /tiered


If necessary, analysts from RCS will provide you with one-on-one time to support you in transitioning your workflows to ARC. RCS will continue to provide support to you and your workflows on ARC after the transition is complete.
== Project Objectives ==
The primary objectives of this transition are to:
* Develop a transition plan for the equipment and technical services CHGI offers
* Define ongoing maintenance governance, processes, data privacy, retention and other related processes
* Transition data, users and workflows to RCS managed infrastructure
== Status ==
User account status in CHGI and transition
{| class="wikitable"
|-
! What !! Total CHGI Accounts Count !! CHGI Accounts Active !! CHGI Accounts Locked-out !! With ARC Accounts !! Without ARC Accounts !! Logged In - Last 12 months !! Logged In - More than a year ago !! Not logged In !! IT Account enabled !! IT Account that is disabled !! No IT Account
|-
| Baseline Dec 04, 2019 || 379 || 335 || 43 || 76 || 259 || 139 || 180 || 16 || 243 || 43  || 49
|-
| || || 88% || 11% || 23% || 77% || 41% || 54% || 5% || 73% || 13% || 15%
|-
| Current month statistics || 399 || 218 || 181 || 88 || 130 || 189 || 51 || 22 || 196 || 12 || 10
|-
| || || 55% || 45% || 40% || 60% || 87%  || 23%  || 10%  || 90% || 6% || 5%
|-
| Changes from baseline Dec/2019 || || || 20 || || 138 || 12
|}
Cluster status:
Hyperion - moved to ARC, users transitioned.
Theia - moved to ARC, users transitioned.
Crick - Restricted access.
Synergy - moved to ARC, users transitioned.
== Background ==
The Center for Health Genomics and Informatics (CHGI) is an initiative that provides a wide range of next-generation genome sequencing services and access to high-performance bioinformatics for sequence analysis to all University of Calgary researchers. New researchers are provided a basic level of networking and storage to make use of CHGI services but also have the option of purchasing additional servers and storage to integrate within the CHGI network to upgrade their usage capabilities. Many of the pieces of the CHGI network equipment have been purchased by researchers and institutions.
The Center for Health Genomics and Informatics (CHGI) is an initiative that provides a wide range of next-generation genome sequencing services and access to high-performance bioinformatics for sequence analysis to all University of Calgary researchers. New researchers are provided a basic level of networking and storage to make use of CHGI services but also have the option of purchasing additional servers and storage to integrate within the CHGI network to upgrade their usage capabilities. Many of the pieces of the CHGI network equipment have been purchased by researchers and institutions.


The CHGI in collaboration with RCS is working on a planned transition of CHGI data, services and workflows to infrastructure managed by RCS. This will allow CHGI researchers to focus on core duties instead of IT services, and open up the potential for scalability in the infrastructure, and standardize a number of common services to IT’s offerings.
The CHGI in collaboration with RCS is working on a planned transition of CHGI data, services and workflows to infrastructure managed by RCS. This will allow CHGI researchers to:
* Focus on core duties instead of managing IT services  
* Open the potential for scalability in the infrastructure
* Standardize several common services to RCS offerings.


= Project Objectives =
== Approach ==
The project team interviewed 15 Principal Investigators and researchers in the Analysis phase to understand the current use of the equipment and services provided by CHGI. This engagement took four weeks during which we interacted with the CHGI manager, System Administrators and PIs that leverage CHGI's equipment for their research. An important aspect in the interviews was the investment made in equipment.


* Develop a transition plan for the equipment and technical services CHGI offers
The project team took the information available and discussed the technical options based on current RCS practices.
* Define ongoing maintenance governance, processes, data privacy, retention and other related processes
* Transition data, users and workflows to IT/RCS managed infrastructure


= Approach =
== Implementation ==
Tasks identified as low impact include:
* CHGI workflows and services that can be transitioned into existing RCS infrastructure
* CHGI Workflows and services with very little impact and costs.
* CHGI equipment which are off or almost off warranty and have low dependence on storage.


The project team interviewed 15 Principal Investigators and researchers in the Analysis phase to understand the current use of the equipment and services provided by CHGI. This engagement took four weeks during which we interacted with the CHGI manager, System Administrator and PIs that leverage CHGI’s equipment for their research. An important aspect in the interviews was the investment made in equipment.
All other tasks, CHGI web related services, or workflows heavily dependent on the the IBM storage system will require deeper analysis and more preparation time. The longer transition period will give researchers extra time to prepare and integrate their processes into RCS infrastructure. Equipment and workflows requiring this extra work will remain in their current location.  


The project team took the information available and discussed the technical options based on current IT’s practices. The equipment, services and workflows have been categorized into 2 groups which was determined based on their ease of transition: Quick Wins, and medium to high complexity.
Two years into the transition, a decision whether to physically move any remaining equipment to RCS infrastructure will be determined.


= Implementation =
== End of Warranty Issues ==
To address the inevitable problem of equipment falling off warranty and funding ceasing, clear communication to the equipment owners and researchers will offer two options if they wish to continue operations. Researchers can choose either to:
# Transition their workflows onto existing RCS offerings in ARC.
# Purchase new equipment under RCS guidelines.


We identified quick wins that can be used as test cases to build trust with the CHGI user base. The Quick Wins are CHGI equipment, workflows and services that should be easy to transition into existing IT/RCS infrastructure, workflows and services with very little impact and costs. This group of equipment are off or almost off warranty, and also have low dependence on storage. These Quick Wins will be the first step in the implementation.
Engagement and collaboration with RCS in this stage will assist in purchasing new equipment that can be integrated with the rest of the RCS infrastructure. These services and workflows will then be transitioned into standard RCS infrastructure, and the new RCS administrator will manage this by following the service levels described in a signed Operating Level Agreement (OLA). After the workflows have been transitioned, the off-warranty equipment would be decommissioned.  


All other clusters and equipment providing web related services or heavily dependent on the IBM Storage (Medium to High complexity transition) will be transitioned using the option described as Option B – PLANNED 12-MONTH TERM TRANSITION to IT.
== On Going ==
Continuous monitoring of the transition will guide the implementation and may improve the transition process.


Option B keeps the equipment and workflows in their current location of HSC B151 providing CHGI researchers a schedule to integrate their processes within IT infrastructure prior to their current warranty running out.
== Critical Success Factors ==


Quick Wins workflows and data are transitioned into ARC following a defined schedule.  At the end of 12-months UCIT will start looking at transitioning the medium and high complexity workflows and data.
* Minimize impact to researcher’s workflows/processes
* Consolidate services and infrastructure


Two years into the transition, a decision whether to transition the remaining equipment physically to IT needs to be discussed.
== Scope ==


= End of Warranty Issues =
=== In Scope ===
 
* Develop a consistent process to transition users, data, workflows from CHGI to RCS infrastructure
To address the inevitable problem of equipment falling off warranty and funding ceasing, clear communication to the equipment owners and researchers will offer two options if they wish to continue operations. Researchers can choose either to transition their workflows onto existing RCS offerings in ARC, or to purchase new equipment under RCS guidelines. Engagement and collaboration with RCS in this stage will assist in purchasing new equipment that can be integrated with the rest of the RCS infrastructure.  These services and workflows will then be transitioned into standard RCS infrastructure, and the new RCS administrator will manage this by following the service levels described in a signed Operating Level Agreement (OLA). After the workflows have been transitioned, the off-warranty equipment would be decommissioned.  
* Define ongoing maintenance governance, processes, data privacy, retention and other related processes
* Transition Individual Researchers from CHGI to RCS infrastructure.  


= On Going =
=== Out of Scope ===
* RCS and CHGI will agree when equipment is ready for decommissioning. CHGI is responsible to put in the requests to dispose of equipment.


Once the first wave of transition happens, a revision to the plan will be made to decide whether to forklift part or all of the remaining equipment to an IT data center.
== Schedule/Milestones ==
{| class="wikitable"
|-
! Milestone !! Dates
|-
| Alternatives Evaluation || Q2/2019
|-
| Implementation Plan || Q3/2019
|-
| Transition to ARC of appropriate users and workflows || Q3/2020
|-
| Analysis of remaining equipment and users || Q3/2020
|-
| Additional transition work - dependent on equipment and users remaining || Q4/2020
|}

Latest revision as of 19:35, 21 April 2021

To improve the sustainability of the CHGI cluster compute and storage infrastructure, the decision was made to integrate this infrastructure with the University of Calgary's High Performance Computing site managed by Research Computing Services (RCS). This will allow CHGI users to leverage the extensive RCS staff and infrastructure. For more information on RCS services, please see the RCS Home Page .

What does this really mean?

PIs, researchers, or students using any of the following storage or compute infrastructure will be contacted by analysts from the RCS team on moving their workflows to ARC:

  • Synergy compute cluster
  • Theia compute cluster
  • Galaxy compute
  • Storage on /gpfs and /tiered

If necessary, analysts from RCS will provide you with one-on-one time to support you in transitioning your workflows to ARC. RCS will continue to provide support to you and your workflows on ARC after the transition is complete.

Project Objectives

The primary objectives of this transition are to:

  • Develop a transition plan for the equipment and technical services CHGI offers
  • Define ongoing maintenance governance, processes, data privacy, retention and other related processes
  • Transition data, users and workflows to RCS managed infrastructure

Status

User account status in CHGI and transition

What Total CHGI Accounts Count CHGI Accounts Active CHGI Accounts Locked-out With ARC Accounts Without ARC Accounts Logged In - Last 12 months Logged In - More than a year ago Not logged In IT Account enabled IT Account that is disabled No IT Account
Baseline Dec 04, 2019 379 335 43 76 259 139 180 16 243 43 49
88% 11% 23% 77% 41% 54% 5% 73% 13% 15%
Current month statistics 399 218 181 88 130 189 51 22 196 12 10
55% 45% 40% 60% 87% 23% 10% 90% 6% 5%
Changes from baseline Dec/2019 20 138 12

Cluster status:

Hyperion - moved to ARC, users transitioned.

Theia - moved to ARC, users transitioned.

Crick - Restricted access.

Synergy - moved to ARC, users transitioned.

Background

The Center for Health Genomics and Informatics (CHGI) is an initiative that provides a wide range of next-generation genome sequencing services and access to high-performance bioinformatics for sequence analysis to all University of Calgary researchers. New researchers are provided a basic level of networking and storage to make use of CHGI services but also have the option of purchasing additional servers and storage to integrate within the CHGI network to upgrade their usage capabilities. Many of the pieces of the CHGI network equipment have been purchased by researchers and institutions.

The CHGI in collaboration with RCS is working on a planned transition of CHGI data, services and workflows to infrastructure managed by RCS. This will allow CHGI researchers to:

  • Focus on core duties instead of managing IT services
  • Open the potential for scalability in the infrastructure
  • Standardize several common services to RCS offerings.

Approach

The project team interviewed 15 Principal Investigators and researchers in the Analysis phase to understand the current use of the equipment and services provided by CHGI. This engagement took four weeks during which we interacted with the CHGI manager, System Administrators and PIs that leverage CHGI's equipment for their research. An important aspect in the interviews was the investment made in equipment.

The project team took the information available and discussed the technical options based on current RCS practices.

Implementation

Tasks identified as low impact include:

  • CHGI workflows and services that can be transitioned into existing RCS infrastructure
  • CHGI Workflows and services with very little impact and costs.
  • CHGI equipment which are off or almost off warranty and have low dependence on storage.

All other tasks, CHGI web related services, or workflows heavily dependent on the the IBM storage system will require deeper analysis and more preparation time. The longer transition period will give researchers extra time to prepare and integrate their processes into RCS infrastructure. Equipment and workflows requiring this extra work will remain in their current location.

Two years into the transition, a decision whether to physically move any remaining equipment to RCS infrastructure will be determined.

End of Warranty Issues

To address the inevitable problem of equipment falling off warranty and funding ceasing, clear communication to the equipment owners and researchers will offer two options if they wish to continue operations. Researchers can choose either to:

  1. Transition their workflows onto existing RCS offerings in ARC.
  2. Purchase new equipment under RCS guidelines.

Engagement and collaboration with RCS in this stage will assist in purchasing new equipment that can be integrated with the rest of the RCS infrastructure. These services and workflows will then be transitioned into standard RCS infrastructure, and the new RCS administrator will manage this by following the service levels described in a signed Operating Level Agreement (OLA). After the workflows have been transitioned, the off-warranty equipment would be decommissioned.

On Going

Continuous monitoring of the transition will guide the implementation and may improve the transition process.

Critical Success Factors

  • Minimize impact to researcher’s workflows/processes
  • Consolidate services and infrastructure

Scope

In Scope

  • Develop a consistent process to transition users, data, workflows from CHGI to RCS infrastructure
  • Define ongoing maintenance governance, processes, data privacy, retention and other related processes
  • Transition Individual Researchers from CHGI to RCS infrastructure.

Out of Scope

  • RCS and CHGI will agree when equipment is ready for decommissioning. CHGI is responsible to put in the requests to dispose of equipment.

Schedule/Milestones

Milestone Dates
Alternatives Evaluation Q2/2019
Implementation Plan Q3/2019
Transition to ARC of appropriate users and workflows Q3/2020
Analysis of remaining equipment and users Q3/2020
Additional transition work - dependent on equipment and users remaining Q4/2020
Retrieved from "https://rcs.ucalgary.ca/index.php?title=CHGI_Transition&oldid=1310"