ARC Cluster Status

2026-01-16T19:49:32Z

Dschulz:

ARC Cluster Status

2026-01-16T19:47:37Z

Dschulz:

{{ARC Cluster Status}}

== System Messages ==
{{Message of the day item
| title = ⚠️ January System Updates
| date = 2023/01/01
| message =
Beginning January 16, 2023, the ARC cluster will undergo operating system updates. We shall do our utmost to minimize disruption and allow ongoing jobs to be completed. New jobs may be temporarily held from scheduling.

The ARC login node will reboot on the morning of January 16. Please save your work and log out if possible.

The upgrade is planned to be fully complete by January 20.

If you encounter any system issues, do not hesitate to let us know.

Thank you for your cooperation.
}}

{{Message of the day item
| title = System Updates Completed
| date = 2023/01/24
| message =
The upgrade has been completed. The following has been changed:
* OS Updated to Rocky Linux 8.7
* Slurm updated to 22.05.7
* Apptainer replaces Singularity
* Each job will have its own /tmp, /dev/shm, /run/user/$uid mounted

If you encounter any system issues, do not hesitate to let us know.

Thank you for your cooperation.
}}

{{Message of the day item
| title = Filesystem Issues
| date = 2023/02/28
| message =
We are currently investigating a filesystem issue that is causing filesystem slowdowns across ARC.

We will update you with more information as it becomes available.

Thank you for your patience.
}}

{{Message of the day item
| title = Filesystem Issues
| date = 2023/03/1
| message =
We are still currently investigating a filesystem issue that is causing filesystem slowdowns across ARC. Some jobs on ARC have been paused to help us find the root cause of the slowdowns.

We will update you with more information as it becomes available.

Thank you for your patience.
}}

{{Message of the day item
| title = ARC Login node reboot
| date = 2023/03/2
| message =
The ARC login node will be rebooted this afternoon for an emergency maintenance. This downtime is needed to help mitigate the filesystem slowdowns experienced on the login node. Jobs will continue running and scheduling during this time.

All logins to the ARC login node will be terminated at 3:00PM and will remain unavailable until 4:00PM.

We apologize for the inconvenience and thank you for your patience.
}}

{{Message of the day item
| title = ⚠️ Filesystem Issues
| date = 2023/03/2
| message =
We are still currently investigating a filesystem issue that is causing filesystem slowdowns on specific nodes in our MSRDC location.

We will update you with more information as it becomes available.

We apologize for the inconvenience and thank you for your patience.
}}

{{Message of the day item
| title = Filesystem Issues Resolved
| date = 2023/03/10
| message =
We have upgraded the filesystem routers in our MSRDC location to address the performance issues.

Please let us know if you experience any issues with the filesystem performance.

Thank-you for your patience.
}}

{{Message of the day item
| title = Open OnDemand reboot
| date = 2023/05/01
| message =
On May 1, 2023, the ARC Open OnDemand node will be rebooted between 5PM and 6PM. Expected downtime will be approximately 15 minutes.

If you encounter any system issues, do not hesitate to let us know.

Thank you for your cooperation.
}}

{{Message of the day item
| title = Apptainer (Singularity) on ARC Login Node
| date = 2023/06/22
| message =
Apptainer (Singularity) containers may experience an error when
running on the Arc login node. If apptainer complains that a system
administrator needs to enable user namespaces, simply run your
containers inside a job.

This is a temporary measure due to security vulnerability that will be
patched soon.

}}

{{Message of the day item
| title = Lattice, Single, cpu2013 partition changes
| date = 2023/07/13
| message =
The Lattice and Single, and cpu2013 have all been decomissioned. The Single
partition will be replaced by the nodes formerly in the cpu2013 partition but
will be called single.

}}

{{Message of the day item
| title = Open OnDemand reboot
| date = 2023/10/17
| message =
Open OnDemand will be rebooted on October 17, 2023 for an update. It will be down for up to 30 minutes.

}}

{{Message of the day item
| title = Storage Upgrade MARC/ARC cluster
| date = 2023/10/23
| message =
We will be performing storage upgrades on the MARC/ARC cluster on
November 16 and 17, 2023. To facilitate this, we will be throttling
down the number of jobs on both clusters while the upgrades are
performed
}}

{{Message of the day item
| title = Systems Operating Normally
| date = 2024/05/3
| message =
}}

{{Message of the day item
| title = Power Interruption
| date = 2024/05/07
| message = Arc Experienced an brief power outage around 11AM May 7, 2024.
Most compute nodes have or are rebooting. Most jobs running at this time
were lost. Arc administrators are actively working on restarting compute
nodes. Sorry for the inconvenience.
}}

{{Message of the day item
| title = GPU a100 Node Reservation
| date = 2024/06/03
| message = Job submissions targeted to the GPU a100 partition will be
affected by a temporary reservation on the nodes to accommodate the RCS
summer school class taking place on 2024/Jun/10. Reservation will end
shortly after. Please submit your jobs normally and the scheduler will
start them as soon as the nodes are available. Sorry for the inconvenience.
}}

{{Message of the day item
| title = GPU a100 Node Reservation Removed
| date = 2024/06/11
| message = GPU a100 Nodes in ARC have been returned to normal scheduling.
}}

{{Message of the day item
| title = Notice of Upcoming Partial Outage
| date = 2024/08/23
| message = Several compute nodes from the ARC cluster will be unavailable
between Sept 23 to Sept 27 inclusive (subject to change). All compute nodes
in cpu2019, cpu2021/2, gpu-v100 most nodes from bigmem and gpu-a100 will be
affected. These nodes will return to service as soon as the work is complete.
}}

{{Message of the day item
| title = Partial Outage Update I
| date = 2024/09/25
| message = Due to hardware issues that is blocking our original maintenance window, most compute nodes that were taken offline on Monday has been brought back online today. An additional partial outage will occur again starting next Tuesday for the same nodes.

On Tuesday, October 1, 2024, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until Friday October 4, 2024.

We apologise for the inconvenience.
}}

{{Message of the day item
| title = Partial Outage Update II
| date = 2024/10/04
| message = The maintenance window will be extended until at least Monday, October 7, 2024 due to a power distribution issue in our renovated data centre.

Currently, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until at least Monday, October 7, 2024. Affected WDF-Altis GPU nodes include: wdfgpu[1-2,6,8-12].

We apologize for the extended downtime and will update you as soon as we have additional information from our operations team.
}}

{{Message of the day item
| title = Partial Outage Update III
| date = 2024/10/07
| message = Due to technical issues beyond our control the maintenance window will be extended until at least Tuesday, October 15, 2024.

Currently, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until at least Tuesday, October 15, 2024. Affected WDF-Altis GPU nodes include: wdfgpu[1-2,6,8-12].

We apologize for the extended downtime and will update you as soon as we have additional information from our operations team.
}}

{{Message of the day item
| title = Normal Scheduling has resumed.
| date = 2024/10/08
| message = The ARC cluster has been successfully brought online and nodes are running jobs normally. We apologize for the extended downtime.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = Scheduled Maintenance
| date = 2024/12/11
| message = The ARC login node will be rebooted on Tuesday December 17 for scheduled maintenance. It will be down for a few minutes and return shortly. Job scheduling and jobs running on the cluster will not be affected. Thank you for understanding.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = Scheduled Maintenance and OS Update
| date = 2025/01/07
| message = The ARC cluster will be rebooted for OS updates on Monday January 20, 2025. Please make sure to save your work and log out before the reboot happens. Scheduling will be paused until the cluster is back, but queued jobs will remain in the queue and nodes will start scheduling when the cluster is ready. Thank you for understanding.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = ⚠️ Scheduled Maintenance and OS Update
| date = 2025/01/15
| message = The ARC cluster will be down for maintenance and upgrades starting 9AM Monday, January 20, 2025 through Wednesday, January 22, 2025.

For the duration of the upgrade window:
* Scheduling will be paused and new jobs will be queued. Any queued jobs will start scheduling only after the upgrade is complete.
* Access to files via the login node and arc-dtn will generally be available but intermittent. File transfers on the DTN node, including Globus file transfers, may be interrupted during this window.

Please make sure to save your work prior to this outage window to avoid any loss of work.

During this time the following changes will happen:

1. Ethernet will replace the 11 year old, unsupported Infiniband on the following partitions:
* cpu2023 (temporary)
* Parallel
* Theia/Synergy/cpu2017-bf05
* Single
Any multi-node jobs (MPI) running on these partitions will have increased latency going forward. If you run multi-node jobs, make sure to run on a partition such as cpu2019, cpu2021, cpu2022.

2. A component of the NetApp filer will be replaced. Access to /bulk will be unavailable on Wednesday, January 22, 2025.

3. The compute node operating system will be updated to Rocky Linux 8.10.

4. The Slurm scheduling system will be upgraded.

5. The Open OnDemand web portal will be upgraded.

Please reach out to support@hpc.calgary.ca with any issues or concerns.

⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️
Update Jan 18, 2025

Around 10AM Arc experienced an electrical power brownout. Some percentage (how many is unknown at this time) of the nodes lost electrical power during this time causing a loss of a number of running jobs.

Sorry for the inconvenience.

Since Arc is shutting down for maintenance on Monday Jan 20, replacement jobs will likely not start unless they request a timelimit less than the time until 8AM Monday.
⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️
}}

{{Message of the day item
| title = Maintenance Complete
| date = 2025/01/22
| message = The ARC cluster upgrade is complete

During this time the following changes happened:

1. Ethernet will replace the 11 year old, unsupported Infiniband on the following partitions:
* cpu2023 (temporary)
* Parallel
* Theia/Synergy/cpu2017-bf05
* Single
Any multi-node jobs (MPI) running on these partitions will have increased latency going forward. If you run multi-node jobs, make sure to run on a partition such as cpu2019, cpu2021, cpu2022.

2. A component of the NetApp filer was replaced successfully.

3. The compute node operating was updated to Rocky Linux 8.10.

4. The Slurm scheduling system was upgraded.

5. The Open OnDemand web portal was upgraded.

6. The Parallel partition was renamed to Legacy to show the lack of an interconnect for parallel MPI work and was restricted to maximum 4 node jobs.

Please reach out to support@hpc.calgary.ca with any issues or concerns.

Jan 23, 9:08AM

Remount complete. arc is back in full service.
}}

{{Message of the day item
| title = Module Command Upgrade
| date = 2025/02/03
| message = Upgrade of the module command

On Tuesday, February 11, 2025 the module command will be upgraded to a new verson on Arc. This should result in new capabilities and a slightly different visual experience when using the module command. Loading modules is not expected to change.
}}

{{Message of the day item
| title = Module Command Upgraded
| date = 2025/02/12
| message = The module command was upgraded

On Tuesday, February 12, 2025 the module command was upgraded to a new verson on Arc. This should result in new capabilities and a slightly different visual experience when using the module command. Loading modules should not have changed.

}}

{{Message of the day item
| title = Support email address down
| date = 2025/03/07
| message = support@hpc.ucalgary.ca Unavailable

Please be informed that our support email address (support@hpc.ucalgary.ca) for RCS is currently not working. We are working to bring it back as soon as possible. Please keep an eye on this space for updates. The clusters are working normally, but support will not receive your messages at this time. We will begin responding as soon as we can get it back.

Apologies for the inconvenience.

}}

{{Message of the day item
| title = Support email address functional
| date = 2025/03/07
| message = support@hpc.ucalgary.ca is back

support@hpc.ucalgary.ca has been repaired and RCS can be contacted there. If you had reached out for assistance in recent days without response please follow up as we may not have received your initial email.

Apologies for the inconvenience.
}}

{{Message of the day item
| title = Interactive Job Timelimit will be Enforced
| date = 2025/04/11
| message = In order to improve the scheduling and job throughput efficiency of ARC, interactive jobs will be limited to a maximum of 5 hours of runtime. Interactive jobs that are submitted with a timelimit over 5 hours will be rejected at submission time. This change will be made on Monday, April 28, 2025.
}}

{{Message of the day item
| title = Interactive Job Timelimit Is Now Enforced
| date = 2025/04/28
| message = In order to improve the scheduling and job throughput efficiency of ARC, interactive jobs are now limited to a maximum of 5 hours of runtime. Interactive jobs that are submitted with a timelimit over 5 hours will be rejected at submission time.

Apr 29, 2025
To increase the security posture of the Arc cluster administrators will be installing Trend Micro on cluster login nodes over the week starting Apr 30. Please report any inconsistencies to support@hpc.ucalgary.ca
}}

{{Message of the day item
| title = ⚠️ Bulk Filesystem Emergency Maintenance
| date = 2025/07/29
| message = The filer that provides the /bulk filesystem will be down for emergency repairs at 12 Noon on Thursday July 31. No access to files on /bulk will be possible for the duration of the multi-hour outage. Any jobs running that access /bulk will start and then pause when access to /bulk is attempted. Jobs should continue once service is restored. Sorry for the inconvenience.
}}

{{Message of the day item
| title = Bulk Filesystem Maintenance Complete
| date = 2025/08/01
| message = Maintenance on /bulk was completed successfully and all filesystems are back in service on Arc.
}}

{{Message of the day item
| title = GPU partition changes on ARC
| date = 2025/08/15
| message = New gpu-h100 and gpu-l40 partitions are available for general scheduling with new gpu hardware. gpu-v100 also has 6 fewer nodes. You can view more details about the node specs by running the arc.hardware script on the login node.
}}

{{Message of the day item
| title = Update and Bulk outage on ARC
| date = 2025/09/15
| message = ⚠️ Arc Update September 15-19

An update of the Arc cluster will begin on Sep 15. This will result in fewer resources while the compute nodes are restarted. The login node and Scheduler will be restarted on Sep 17.

⚠️ Bulk Filesystem Maintenance September 17
The filer that provides the /bulk filesystem will be down for emergency repairs at 9 AM on Wednesday Sep 17th. No access to files on /bulk will be possible for the duration of the multi-hour outage. Any jobs running that access /bulk will start and then pause when access to /bulk is attempted. Jobs should continue once service is restored. We apologize for any inconvenience.

}}

{{Message of the day item
| title = Update and Bulk outage on ARC
| date = 2025/09/18
| message = Arc update complete

The update to arc is complete.

}}

{{Message of the day item
| title = GPU partition changes on ARC
| date = 2025/09/23
| message = gpu-v100 has 6 more nodes. You can view more details about the node specs by running the arc.hardware script on the login node.
}}

{{Message of the day item
| title = Systems Operating Normally
| date = 2025/11/4
| message =
}}

{{Message of the day item
| title = Legacy compute nodes are being retired
| date = 2025/11/20
| message = ARC nodes cn[0513-1096] are being removed from the arc cluster. They will be removed from scheduling and removed from the cluster over the next while.
}}

{{Message of the day item
| title = Planned outage in Jan 2026
| date = 2025/12/26
| message = ⚠️ Arc Update Week of January 12

ARC will be going down periodically during the week of 2026/Jan/12 to allow for system maintenance. All nodes and filesystems will be unavailable for use during the outage, and filesystem performance may be variable following the upgrade for a time until the changes finish.

If you have questions or concerns, please contact support@hpc.ucalgary.ca
}}

{{Message of the day item
| title = Arc Filesystem Returned to service, scheduling outages continue
| date = 2025/01/13
| message = ⚠️ Data access now possible ⚠️

The first stage of the maintenance is complete and the filesystems are once again available.

There is still significant work to do on the cluster so jobs will not start until this maintenance is complete but clients will be able to access data via the login node and DTN.

If you have questions or concerns, please contact support@hpc.ucalgary.ca
}}

{{Message of the day item
| title = Arc Returned to service, scheduling resumed
| date = 2025/01/16
| message = Maintenance Complete

Arc has been returned to service.

If you have questions or concerns, please contact support@hpc.ucalgary.ca
}}

{{Navbox ARC}}
[[Category:ARC]]

TALC Cluster Status

2026-01-13T20:49:40Z

Dschulz:

TALC Cluster Status

2026-01-13T20:04:59Z

Dschulz:

ARC Cluster Status

2026-01-13T19:54:55Z

Dschulz:

Running alphafold3

2025-11-12T17:55:55Z

Dschulz: Updated Alphafold3 usage instructions

= Running Alphafold3 on Arc =

Alphafold3 has been compiled into an Apptainer image for use on Arc.

Alphafold3 has two separate steps, one that requires cpus only, and reads much of the large public_databases directory. The second step runs the inference and requires the use of a GPU. The pipeline step takes much longer than the inference step. It is wasteful to occupy a GPU node for the duration of the pipeline step. To run Alphafold3 efficiently we have to run it two separate times in different modes on compute nodes that have appropriate resources.

== Prerequisites ==

* Due to licensing reasons, every client who would like to use alphafold3 must register for and download the model parameters. See https://forms.gle/svvpY4u2jsHEwWYS6 to register. The modelparameters are relatively small and can be stored anywhere on Arc that you have access to and can be used by multiple jobs at the same time (you do not have to have a separate copy of it for every job).
* The alphafold3 container is stored in /global/software/alphafold3. There are example job scripts in that directory that are referenced below.
* Due to the nature of the pipeline stage, we need to split each alphafold run into separate pipeline and inference stages. If we don't do this, a valuable GPU node is tied up for a long time preventing others from using it. In our testing using an input of 120 proteins the pipeline stage took a day and the inference stage took only an hour
* The public databases that Alphafold uses have been pre-downloaded and are stored in a location for anyone on Arc to use. This location, /bulk/public/alphafold3/public_databases, is reflected in the example job scripts.

== The Pipeline stage ==

* This stage is very time consuming. Much of the time is spent waiting to load a very large amount of data from the filesystem. The example jobscript in /global/software/alphafold3 copies the /bulk/public/alphafold3/public_databases/mmcif_files to /dev/shm which is a filesystem that exists entirely in the compute node's memory (this makes it very fast). This staging of data takes 20-30 min so you'll want to run a fairly large number of proteins in one job to amortize the cost of copying the mmcif_files to the compute node.
* Download the model and put it in ./models (This is a legal/license requirement)
* Generate input data. Make sure you've split your input sequences into 5 roughly equally sized files (by number of input sequences). For the examples we'll use filenames starting with xa and ending with .json so xa*.json. The example job script runs 5 copies of alphafold at the same time on the compute node as alphafold3's pipeline stage can really only effectively make use of 4 cpus for a single job. This way the system is running multiple 4 cpu jobs at the same time.
* Create ./pipelineoutputs for the resulting files
* Run the pipeline stage on cpus by submitting runpipeline.slurm. Required Job resources:
** --mem=500GB # We need to request enough memory for the public_databases/mmcif_files plus whatever the job needs.
** -c 20 # 20 cpus
** -N 1 # 1 node

== The Inference stage ==

* This is the part of alphafold that requires a GPU.
* Compared to the pipeline stage, the inference is fairly quick.
* copy the runinference.slurm file to a directory you have write access to.
* Take the outputs from the pipeline stage (which were put in ./pipelineoutputs in the example, move/copy the files into a directory called something like inference_inputs. Note that the pipeline stage makes a directory for each of the files but the inference stage expects them to all be directly in the inference_inputs. You'll have to copy them all. You can use find to help:
cd pipelineoutputs
cp $(find -type f -name \*.json) ../inferenceinputs
* submit the runpipeline.slurm to the scheduler. Required job resources:
** -p gpu-h100
** --gres=gpu:1 # Alphafold can only use 1 gpu
** --mem=120G # This might need to be increased depending on the inputs.
** --ntasks-per-node=4

== References ==
* https://github.com/google-deepmind/alphafold3/blob/main/docs/performance.md
* https://docs.alliancecan.ca/wiki/AlphaFold3

Template:ARC Cluster Status

2025-09-18T19:55:03Z

Dschulz:

{{Cluster Status
| cluster = ARC
| status = green
| title = Cluster maintenance complete.
| message = System is functioning normally. No known issues.

See the [[ARC Cluster Status]] page for system notices.
}}

Altis Login Node Status

2025-09-18T19:52:57Z

Dschulz: /* Software pages */

Research Computing Services (RCS) is a group within the wider University of Calgary Information Technologies team that plans, manages, and supports high performance computing (HPC) systems in use by researchers throughout the University of Calgary. Our primary focus is to meet the increasing demand for engineering and scientific computation by offering a wide range of specialized services to help researchers solve highly complex real-world problems or run large scale computationally intensive workloads on our high-end HPC resources.

This RCS Wiki contains technical documentation for use by users of HPC systems operated by RCS


=== Contact us for support ===

* For general RCS/HPC inquiries, please email: [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca]
* For IT related issues (networking, VPN, email), please email: [mailto:it@ucalgary.ca it@ucalgary.ca]
* For Compute Canada specific questions: [mailto:support@tech.alliancecan.ca support@tech.alliancecan.ca]

{{Clear}}
<div class="row">
<div class="col-md-6">

== General information ==
* [[General Cluster Guidelines and Policies]]
* [[How to get an account]]
* [[Data ownership]]
* [[Connecting to RCS HPC Systems]]
* [[External collaborators]]

* [[CloudStack|Cloud/Virtual Machine Infrastructure (CloudStack)]]

* [[On-line resources for new Linux and ARC users]]
* [[Acknowledging Research Computing Services Group]]

== Cluster Guides ==
* [[ ARC Cluster Guide]] - ARC is a general purpose cluster for University of Calgary researchers.
* [[GLaDOS Cluster Guide]] - GLaDOS is a researcher-owned cluster maintained by Research Computing Services.
* [[TALC Cluster Guide]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.
* [[MARC Cluster Guide]] -- Medical Advanced Research Computing cluster at the University of Calgary created by Research Computing Services in 2020.

== Other services ==

* [[Jupyter Notebooks]]
* [[Open OnDemand | Open OnDemand portal]]

== Software pages ==
* [[Managing software on ARC]]
* [[Gaussian on ARC]] -- How to use Gaussian 16 on ARC.
* [[Apache Spark on ARC]]
* [[ARC Software pages]]
* [[Bioinformatics applications]]
* [[Running_alphafold3]]
</div>
<div class="col-md-6">

== Running courses on HPC resources ==
* [[TALC Cluster|TALC]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.
* [[TALC Terms of Use]] - Terms of use to which TALC account holders must agree to use the cluster.
* [[List of courses on TALC]] - A list of current and historical courses taught using TALC.

== Training ==
* Our [[HPC Systems]]
* [[HPC Linux topics]] - A list of topics on which RCS technical support staff can provide one-on-one or group training
* [[Courses]]
* [[Linux Introduction]]
* [[What is a scheduler?]]
* [[Running jobs]]
* [[Data storage options for UofC researchers]]
* [[Security and privacy]]
* [[How to transfer data]]

* [[UofC Services]]

* [[Book online training sessions]]

* [[How-Tos | More How-Tos]]
</div>
</div>

{{Clear}}

__NOTOC__

Running alphafold3

2025-07-29T18:35:37Z

Dschulz:

Running alphafold3

2025-07-29T17:22:14Z

Dschulz: Created page with "= Running Alphafold3 on Arc = Alphafold3 has been compiled into an Apptainer image for use on Arc. Alphafold3 has two separate steps, one that requires cpus only, and reads much of the large public_databases directory. The second step runs the inference and requires the use of a GPU. The pipeline step takes much longer than the inference step. It is wasteful to occupy a GPU node for the duration of the pipeline step. To run Alphafold3 efficiently we have to run it t..."

MARC Cluster Status

2025-06-17T19:38:54Z

Dschulz:

Think Login Node Status

2025-04-29T20:36:40Z

Dschulz:

{{ARC Cluster Status}}

== System Messages ==
{{Message of the day item
| title = Systems Operating Normally
| date = 2024/09/03
| message =
The ARC Cluster and the Think login node is operational. No upcoming upgrades are planned.
}}

{{Message of the day item
| title = Notice of Upcoming Partial Outage
| date = 2024/08/23
| message = Several compute nodes from the ARC cluster will be unavailable
between Sept 23 to Sept 27 inclusive (subject to change). Some Think GPU nodes will be affected during this maintenance window. These nodes will return to service as soon as the work is complete.
}}

{{Message of the day item
| title = Partial Outage Update I
| date = 2024/09/25
| message = Due to hardware issues that is blocking our original maintenance window, most compute nodes that were taken offline on Monday has been brought back online today. An additional partial outage will occur again starting next Tuesday for the same nodes.

On Tuesday, October 1, 2024, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until Friday October 4, 2024.

We apologise for the inconvenience.
}}

{{Message of the day item
| title = Partial Outage Update II
| date = 2024/10/04
| message = The maintenance window will be extended until at least Monday, October 7, 2024 due to a power distribution issue in our renovated data centre.

Currently, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until at least Monday, October 7, 2024. Affected WDF-Altis GPU nodes include: wdfgpu[1-2,6,8-12].

We apologize for the extended downtime and will update you as soon as we have additional information from our operations team.
}}

{{Message of the day item
| title = Partial Outage Update III
| date = 2024/10/07
| message = Due to technical issues beyond our control the maintenance window will be extended until at least Tuesday, October 15, 2024.

Currently, the compute nodes in cpu2019, cpu2021, cpu2022, gpu-v100, gpu-a100, and most nodes from bigmem will be unavailable until at least Tuesday, October 15, 2024. Affected WDF GPU nodes include: wdfgpu[1-2,6,8-12].

We apologize for the extended downtime and will update you as soon as we have additional information from our operations team.
}}

{{Message of the day item
| title = Normal Scheduling has resumed.
| date = 2024/10/08
| message = The ARC cluster has been successfully brought online and nodes are running jobs normally. We apologize for the extended downtime.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = wdfgpu[1-12] System Update Reboots
| date = 2024/12/02
| message = wdfgpu[1-12] will be updated today for a short reboot to install important system updates and will return shortly.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = Scheduled Maintenance and OS Update
| date = 2025/01/07
| message = The ARC cluster will be rebooted for OS updates on Monday January 20, 2025. Please make sure to save your work and log out before the reboot happens. Scheduling will be paused until the cluster is back, but queued jobs will remain in the queue and nodes will start scheduling when the cluster is ready. Thank you for understanding.

Please reach out to support@hpc.ucalgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = ⚠️ Scheduled Maintenance and OS Update
| date = 2025/01/15
| message = The ARC cluster will be down for maintenance and upgrades starting 9AM Monday, January 20, 2025 through Wednesday, January 22, 2025.

For the duration of the upgrade window:
* Scheduling will be paused and new jobs will be queued. Any queued jobs will start scheduling only after the upgrade is complete.
* Access to files via the login node and arc-dtn will generally be available but intermittent. File transfers on the DTN node, including Globus file transfers, may be interrupted during this window.

Please make sure to save your work prior to this outage window to avoid any loss of work.

During this time the following changes will happen:

1. Ethernet will replace the 11 year old, unsupported Infiniband on the following partitions:
* cpu2023 (temporary)
* Parallel
* Theia/Synergy/cpu2017-bf05
* Single
Any multi-node jobs (MPI) running on these partitions will have increased latency going forward. If you run multi-node jobs, make sure to run on a partition such as cpu2019, cpu2021, cpu2022.

2. A component of the NetApp filer will be replaced. Access to /bulk will be unavailable on Wednesday, January 22, 2025.

3. The compute node operating system will be updated to Rocky Linux 8.10.

4. The Slurm scheduling system will be upgraded.

5. The Open OnDemand web portal will be upgraded.

Please reach out to support@hpc.calgary.ca with any issues or concerns.

⚠️⚠️⚠️⚠️⚠️
Update Jan 18, 2025

Around 10AM Arc experienced an electrical power brownout. Some percentage (how many is unknown at this time) of the nodes lost electrical power during this time causing a loss
of a number of running jobs.

Sorry for the inconvenience.

Since Arc is shutting down for maintenance on Monday Jan 20, replacement jobs will likely not start unless they request a timelimit less than the time until 8AM Monday.
⚠️⚠️⚠️⚠️⚠️
}}

{{Message of the day item
| title = Maintenance Complete
| date = 2025/01/22
| message = The ARC/Think cluster upgrade is complete

During this time the following changes happened:

1. Ethernet will replace the 11 year old, unsupported Infiniband on the following partitions:
* cpu2023 (temporary)
* Parallel
* Theia/Synergy/cpu2017-bf05
* Single
Any multi-node jobs (MPI) running on these partitions will have increased latency going forward. If you run multi-node jobs, make sure to run on a partition such as cpu2019, cpu2021, cpu2022.

2. A component of the NetApp filer was replaced successfully.

3. The compute node operating was updated to Rocky Linux 8.10.

4. The Slurm scheduling system was upgraded.

Please reach out to support@hpc.calgary.ca with any issues or concerns.
}}

{{Message of the day item
| title = Support email address down
| date = 2025/03/07
| message = support@hpc.ucalgary.ca Unavailable

Please be informed that our support email address (support@hpc.ucalgary.ca) for RCS is currently not working. We are working to bring it back as soon as possible. Please keep an eye on this space for updates. The clusters are working normally, but support will not receive your messages at this time. We will begin responding as soon as we can get it back.

Apologies for the inconvenience.

}}

{{Message of the day item
| title = Support email address functional
| date = 2025/03/07
| message = support@hpc.ucalgary.ca is back

support@hpc.ucalgary.ca has been repaired and RCS can be contacted there. If you had reached out for assistance in recent days without response please follow up as we may not have received your initial email.

Apologies for the inconvenience.
}}

{{Message of the day item
| title = Trend Micro Installation
| date = 2025/04/29
| message = To increase the security posture of the Arc cluster administrators will be installing Trend Micro on cluster login nodes over the week starting Apr 30. Please report any inconsistencies to support@hpc.ucalgary.ca

}}

[[Category:ARC]]
{{Navbox ARC}}

Altis Login Node Status

2025-04-29T20:36:04Z

Dschulz:

TALC Cluster Status

2025-04-29T20:35:15Z

Dschulz:

2025-01-22T19:10:55Z

Dschulz: