https://rcs.ucalgary.ca/api.php?action=feedcontributions&feedformat=atom&user=Fridman 2026-04-29T02:52:48Z User contributions MediaWiki 1.43.3 https://rcs.ucalgary.ca/index.php?title=High-Performance_Computing_Courses_at_RCS&diff=2976 2023-11-17T23:02:13Z

<p>Fridman: /* Past courses offered by the Research Computing Services Team */</p> <hr /> <div>For University of Calgary researchers intending to leverage the HPC infrastructure for their research work and seeking customized sessions, please contact us at support@hpc.ucalgary.ca for further discussion.<br /> <br /> To customize our courses according to your requirements, please furnish the following details at least four weeks in advance:<br /> <br /> # Specify the number of researchers to undergo training (minimum of 5 or more).<br /> # Share the domain of your research.<br /> # Let us know if you have a particular workflow or application in mind.<br /> # Inform us about the Linux background of the researchers; this information aids us in designing the course effectively.<br /> <br /> == Past courses offered by the Research Computing Services Team ==<br /> <br /> === Year 2023 ===<br /> <br /> * May 01, 2023- Grad Success Week (GSW) 2023 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing'' <br /> * Jan 04, 2023 - Session for undergraduate bioinformatics students (invited by: David Anderson)<br /> : ''Introduction to High-Performance Computing Infrastructure at the University of Calgary''<br /> *Oct 10, 2023 - ENSF 619<br /> <br /> === Year 2022 ===<br /> * May 05, 2022- Grad Success Week (GSW) 2022 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing''<br /> * Jan 04, 2022- Block week 2022 by David Anderson (invited by: David Anderson)<br /> : ''Introduction to High-Performance Computing Infrastructure at the University of Calgary''<br /> *Oct 11, 2022 - ENSF 619<br /> <br /> === Year 2021 ===<br /> * May 05, 2021- Grad Success Week (GSW) 2021 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing'' <br /> * Apr 22, 2021 - Centre for Health Informatics <br /> : ''Secure Computing on the MARC cluster''<br /> *Oct 21, 2021 - ENSF 619<br /> :<br /> :<br /> <br /> === Year 2020 ===<br /> * May 05, 2020- Grad Success Week (GSW) 2020 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing''<br /> <br /> <br /> **********************************************************************************************************************************************<br /> <br /> == Overview of the courses we offer ==<br /> <br /> === Introduction to Bash and Linux - Basic ===<br /> <br /> : ''I. Understanding the Command Line Interface (CLI)''<br /> :* Overview of CLI vs. Graphical User Interface (GUI)<br /> :* Importance of CLI in Bioinformatics and Data Science<br /> <br /> : ''II. Getting Started with Bash''<br /> :* Opening the Terminal<br /> :* Basic Shell Commands<br /> <br /> : ''III. File and Directory Manipulation''<br /> :* Creating and Deleting Files/Directories<br /> :* Copying and Moving Files/Directories<br /> :* Understanding Permissions (chmod)<br /> <br /> : ''IV. Text Processing with Bash''<br /> :* cat, head, and tail commands<br /> :* grep for pattern matching<br /> :* Redirection and Pipelines<br /> <br /> : ''V. Basic Scripting''<br /> :* Creating and Executing Bash Scripts<br /> :* Variables and Basic Control Structures<br /> :* Introduction to Functions<br /> <br /> === Introduction to Bash and Linux - Advanced ===<br /> : ''I. Advanced Text Processing''<br /> :* Regular Expressions<br /> :* Text Manipulation<br /> <br /> : ''II. Shell Scripting Best Practices<br /> :* Error Handling and Logging<br /> :* Command-Line Arguments<br /> :* Debugging Techniques<br /> .<br /> .<br /> .<br /> .<br /> .<br /> <br /> === Introduction to Python on ARC ===<br /> <br /> <br /> === Introduction to R on ARC ===<br /> <br /> <br /> === How to Speed Up R Codes? ===<br /> <br /> <br /> === Introduction to Singularity/Apptainer - Basic ===<br /> <br /> <br /> === Introduction to Singularity/Apptainer - Advanced ===</div>

Fridman https://rcs.ucalgary.ca/index.php?title=High-Performance_Computing_Courses_at_RCS&diff=2975 2023-11-17T23:01:06Z

<p>Fridman: </p> <hr /> <div>For University of Calgary researchers intending to leverage the HPC infrastructure for their research work and seeking customized sessions, please contact us at support@hpc.ucalgary.ca for further discussion.<br /> <br /> To customize our courses according to your requirements, please furnish the following details at least four weeks in advance:<br /> <br /> # Specify the number of researchers to undergo training (minimum of 5 or more).<br /> # Share the domain of your research.<br /> # Let us know if you have a particular workflow or application in mind.<br /> # Inform us about the Linux background of the researchers; this information aids us in designing the course effectively.<br /> <br /> == Past courses offered by the Research Computing Services Team ==<br /> <br /> === Year 2023 ===<br /> <br /> * May 01, 2023- Grad Success Week (GSW) 2023 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing'' <br /> * Jan 04, 2023 - Session for undergraduate bioinformatics students (invited by: David Anderson)<br /> : ''Introduction to High-Performance Computing Infrastructure at the University of Calgary''<br /> <br /> === Year 2022 ===<br /> * May 05, 2022- Grad Success Week (GSW) 2022 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing''<br /> * Jan 04, 2022- Block week 2022 by David Anderson (invited by: David Anderson)<br /> : ''Introduction to High-Performance Computing Infrastructure at the University of Calgary''<br /> <br /> === Year 2021 ===<br /> * May 05, 2021- Grad Success Week (GSW) 2021 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing'' <br /> * Apr 22, 2021 - Centre for Health Informatics <br /> : ''Secure Computing on the MARC cluster''<br /> *Oct 21, 2021 - ENSF 619<br /> :<br /> :<br /> <br /> === Year 2020 ===<br /> * May 05, 2020- Grad Success Week (GSW) 2020 (invited by: Paul Pappin)<br /> : ''Data Analysis in R and Accessing Advanced Research Computing''<br /> <br /> <br /> **********************************************************************************************************************************************<br /> == Overview of the courses we offer ==<br /> <br /> === Introduction to Bash and Linux - Basic ===<br /> <br /> : ''I. Understanding the Command Line Interface (CLI)''<br /> :* Overview of CLI vs. Graphical User Interface (GUI)<br /> :* Importance of CLI in Bioinformatics and Data Science<br /> <br /> : ''II. Getting Started with Bash''<br /> :* Opening the Terminal<br /> :* Basic Shell Commands<br /> <br /> : ''III. File and Directory Manipulation''<br /> :* Creating and Deleting Files/Directories<br /> :* Copying and Moving Files/Directories<br /> :* Understanding Permissions (chmod)<br /> <br /> : ''IV. Text Processing with Bash''<br /> :* cat, head, and tail commands<br /> :* grep for pattern matching<br /> :* Redirection and Pipelines<br /> <br /> : ''V. Basic Scripting''<br /> :* Creating and Executing Bash Scripts<br /> :* Variables and Basic Control Structures<br /> :* Introduction to Functions<br /> <br /> === Introduction to Bash and Linux - Advanced ===<br /> : ''I. Advanced Text Processing''<br /> :* Regular Expressions<br /> :* Text Manipulation<br /> <br /> : ''II. Shell Scripting Best Practices<br /> :* Error Handling and Logging<br /> :* Command-Line Arguments<br /> :* Debugging Techniques<br /> .<br /> .<br /> .<br /> .<br /> .<br /> <br /> === Introduction to Python on ARC ===<br /> <br /> <br /> === Introduction to R on ARC ===<br /> <br /> <br /> === How to Speed Up R Codes? ===<br /> <br /> <br /> === Introduction to Singularity/Apptainer - Basic ===<br /> <br /> <br /> === Introduction to Singularity/Apptainer - Advanced ===</div>

Fridman https://rcs.ucalgary.ca/index.php?title=Template:ARC_Cluster_Status&diff=2547 2023-07-05T18:35:56Z

<p>Fridman: </p> <hr /> <div>{{Cluster Status<br /> | cluster = ARC<br /> | status = green<br /> | title = Cluster operational<br /> | message = No issues to report.<br /> See the [[ARC Cluster Status]] page for system notices. <br /> }}</div>

Fridman https://rcs.ucalgary.ca/index.php?title=External_collaborators&diff=2541 2023-06-28T20:00:38Z

<p>Fridman: /* Background */</p> <hr /> <div>= Background =<br /> <br /> Accessing resources provided by Research Computing Services at the University of Calgary <br /> requires that potential new users have an IT account with the University as well as an email address associated with the account, <br /> such as &lt;code&gt;name@ucalagary.ca&lt;/code&gt;. <br /> This makes it impossible for a person outside the University of Calgary to access the resources,<br /> including external collaborators.<br /> The solution is to formally associate an external collaborator who needs access to HPC resources at UofC with the University.<br /> <br /> = General Associate =<br /> <br /> An '''External Research Collaborator''' designation is for those who need to remotely access <br /> the S'''ecure Compute''' and '''High Performance Compute (HPC)''' resources at the University of Calgary. <br /> This designation allows you to request a '''General Associate (GA)''' access for external research collaborators, <br /> that are not University of Calgary employees or associated with AHS, to our HPC and Secure Compute services in an expedited manner. <br /> Researchers in this category require a Principal Investigator (PI) or a PI delegate to submit a Template Based Hire (TBH) <br /> with the GA template. <br /> <br /> Please note that '''AHS external researchers''' have their own GA template available for them and they don’t need to use this new GA template.<br /> <br /> Principal Investigators or their delegates can request the creation of a new General Associate External Collaborator following <br /> the Template Base Hire form process in PeopleSoft and selecting the template “UC_CWR_EXT_RES_CL – Gen Associate – External Research Collaborator”. <br /> These requests will need to be approved by Research Computing Services, who is managing HPC. Once the transaction is approved, <br /> it will go to HR to complete the hiring process and the new account will be ready for your associate.<br /> <br /> = Official documents =<br /> <br /> * This reference guide provides instructions on how to initiate a template-based hire for a '''General Associate Relationship'''<br /> :https://www.ucalgary.ca/hr/sites/default/files/teams/241/creating-new-relationship-general-associate-qrg.pdf<br /> <br /> * This link gives details on the '''various types''' of general associate hires:<br /> :https://www.ucalgary.ca/hr/hiring-managing/administration/general-associate-set-up</div>

Fridman https://rcs.ucalgary.ca/index.php?title=RCS_Home_Page&diff=2385 2023-03-20T20:51:27Z

<p>Fridman: /* What&#039;s New */</p> <hr /> <div>Research Computing Services (RCS) is a group within the wider University of Calgary Information Technologies team that plans, manages, and supports high performance computing (HPC) systems in use by researchers throughout the University of Calgary. Our primary focus is to meet the increasing demand for engineering and scientific computation by offering a wide range of specialized services to help researchers solve highly complex real-world problems or run large scale computationally intensive workloads on our high-end HPC resources.<br /> <br /> This RCS Wiki contains technical documentation for use by users of HPC systems operated by RCS<br /> <br /> &lt;!-- <br /> In case cluster status changes:<br /> * set the status to yellow or red <br /> * provide a custom 'title' and 'message'<br /> <br /> {{Cluster Status<br /> |status=green<br /> }}<br /> --&gt;<br /> === Contact us for support ===<br /> <br /> * For general RCS/HPC inquiries, please email: [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca]<br /> * For IT related issues (networking, VPN, email), please email: [mailto:it@ucalgary.ca it@ucalgary.ca]<br /> * For Compute Canada specific questions: [mailto:support@computecanada.ca support@computecanada.ca]<br /> <br /> {{Clear}}<br /> &lt;div class=&quot;row&quot;&gt;<br /> &lt;div class=&quot;col-md-6&quot;&gt;<br /> <br /> == General information ==<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[How to get an account]]<br /> * [[Data ownership]]<br /> * [[Connecting to RCS HPC Systems]]<br /> * [[External collaborators]]<br /> <br /> * [[CloudStack|Cloud/Virtual Machine Infrastructure (CloudStack)]]<br /> <br /> * [[On-line resources for new Linux and ARC users]]<br /> * [[Acknowledging Research Computing Services Group]]<br /> <br /> == Cluster Guides ==<br /> * [[ ARC Cluster Guide]] - ARC is a general purpose cluster for University of Calgary researchers.<br /> * [[GLaDOS Cluster Guide]] - GLaDOS is a researcher-owned cluster maintained by Research Computing Services.<br /> * [[TALC Cluster Guide]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.<br /> * [[MARC Cluster Guide]] -- Medical Advanced Research Computing cluster at the University of Calgary created by Research Computing Services in 2020.<br /> <br /> == Other services ==<br /> <br /> * [[Jupyter Notebooks]]<br /> * [[Open OnDemand | Open OnDemand portal]]<br /> <br /> == Software pages ==<br /> * [[Managing software on ARC]]<br /> * [https://hpc.ucalgary.ca/arc/software/conda Using Conda (external link)]<br /> * [[Gaussian on ARC]] -- How to use Gaussian 16 on ARC.<br /> * [[Apache Spark on ARC]]<br /> * [[ARC Software pages]]<br /> * [[Bioinformatics applications]]<br /> &lt;/div&gt;<br /> &lt;div class=&quot;col-md-6&quot;&gt;<br /> <br /> == Running courses on HPC resources ==<br /> * [[TALC Cluster|TALC]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.<br /> * [[TALC Terms of Use]] - Terms of use to which TALC account holders must agree to use the cluster.<br /> * [[List of courses on TALC]] - A list of current and historical courses taught using TALC.<br /> <br /> == Training ==<br /> * Our [[HPC Systems]]<br /> * [[HPC Linux topics]] - A list of topics on which RCS technical support staff can provide one-on-one or group training<br /> * [[Courses]]<br /> * [[Linux Introduction]]<br /> * [[What is a scheduler?]]<br /> * [[Running jobs]]<br /> * [[Data storage options for UofC researchers]]<br /> * [[Security and privacy]]<br /> * [[How to transfer data]]<br /> <br /> * [[UofC Services]]<br /> <br /> * [[Book online training sessions]]<br /> <br /> <br /> * [[How-Tos | More How-Tos]]<br /> &lt;/div&gt;<br /> &lt;/div&gt;<br /> <br /> {{Clear}}<br /> <br /> ==What's New==<br /> <br /> __NOTOC__</div>

Fridman https://rcs.ucalgary.ca/index.php?title=RCS_Home_Page&diff=2384 2023-03-20T20:49:28Z

<p>Fridman: /* Cluster Guides */</p> <hr /> <div>Research Computing Services (RCS) is a group within the wider University of Calgary Information Technologies team that plans, manages, and supports high performance computing (HPC) systems in use by researchers throughout the University of Calgary. Our primary focus is to meet the increasing demand for engineering and scientific computation by offering a wide range of specialized services to help researchers solve highly complex real-world problems or run large scale computationally intensive workloads on our high-end HPC resources.<br /> <br /> This RCS Wiki contains technical documentation for use by users of HPC systems operated by RCS<br /> <br /> &lt;!-- <br /> In case cluster status changes:<br /> * set the status to yellow or red <br /> * provide a custom 'title' and 'message'<br /> <br /> {{Cluster Status<br /> |status=green<br /> }}<br /> --&gt;<br /> === Contact us for support ===<br /> <br /> * For general RCS/HPC inquiries, please email: [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca]<br /> * For IT related issues (networking, VPN, email), please email: [mailto:it@ucalgary.ca it@ucalgary.ca]<br /> * For Compute Canada specific questions: [mailto:support@computecanada.ca support@computecanada.ca]<br /> <br /> {{Clear}}<br /> &lt;div class=&quot;row&quot;&gt;<br /> &lt;div class=&quot;col-md-6&quot;&gt;<br /> <br /> == General information ==<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[How to get an account]]<br /> * [[Data ownership]]<br /> * [[Connecting to RCS HPC Systems]]<br /> * [[External collaborators]]<br /> <br /> * [[CloudStack|Cloud/Virtual Machine Infrastructure (CloudStack)]]<br /> <br /> * [[On-line resources for new Linux and ARC users]]<br /> * [[Acknowledging Research Computing Services Group]]<br /> <br /> == Cluster Guides ==<br /> * [[ ARC Cluster Guide]] - ARC is a general purpose cluster for University of Calgary researchers.<br /> * [[GLaDOS Cluster Guide]] - GLaDOS is a researcher-owned cluster maintained by Research Computing Services.<br /> * [[TALC Cluster Guide]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.<br /> * [[MARC Cluster Guide]] -- Medical Advanced Research Computing cluster at the University of Calgary created by Research Computing Services in 2020.<br /> <br /> == Other services ==<br /> <br /> * [[Jupyter Notebooks]]<br /> * [[Open OnDemand | Open OnDemand portal]]<br /> <br /> == Software pages ==<br /> * [[Managing software on ARC]]<br /> * [https://hpc.ucalgary.ca/arc/software/conda Using Conda (external link)]<br /> * [[Gaussian on ARC]] -- How to use Gaussian 16 on ARC.<br /> * [[Apache Spark on ARC]]<br /> * [[ARC Software pages]]<br /> * [[Bioinformatics applications]]<br /> &lt;/div&gt;<br /> &lt;div class=&quot;col-md-6&quot;&gt;<br /> <br /> == Running courses on HPC resources ==<br /> * [[TALC Cluster|TALC]] - Teaching and Learning Cluster (TALC) is a cluster created by Research Computing Services to support academic courses and workshops.<br /> * [[TALC Terms of Use]] - Terms of use to which TALC account holders must agree to use the cluster.<br /> * [[List of courses on TALC]] - A list of current and historical courses taught using TALC.<br /> <br /> == Training ==<br /> * Our [[HPC Systems]]<br /> * [[HPC Linux topics]] - A list of topics on which RCS technical support staff can provide one-on-one or group training<br /> * [[Courses]]<br /> * [[Linux Introduction]]<br /> * [[What is a scheduler?]]<br /> * [[Running jobs]]<br /> * [[Data storage options for UofC researchers]]<br /> * [[Security and privacy]]<br /> * [[How to transfer data]]<br /> <br /> * [[UofC Services]]<br /> <br /> * [[Book online training sessions]]<br /> <br /> <br /> * [[How-Tos | More How-Tos]]<br /> &lt;/div&gt;<br /> &lt;/div&gt;<br /> <br /> {{Clear}}<br /> <br /> ==What's New==<br /> * [[CHGI Transition]] - Information on the current CHGI Transition<br /> <br /> <br /> __NOTOC__</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1721 2022-02-10T17:52:24Z

<p>Fridman: /* Current Courses */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> <br /> = Information =<br /> <br /> * TALC Terms of Use: https://rcs.ucalgary.ca/TALC_Terms_of_Use<br /> <br /> * TALC Guide: https://rcs.ucalgary.ca/TALC_Cluster<br /> <br /> <br /> == A New Course Checklist ==<br /> <br /> At least '''three (3) weeks''' before the course.<br /> * Please contact us about the course you are going to teach using TALC.<br /> <br /> * Please provide us with the '''course ID''', '''name of the course''', and the '''name of the instructor''' for the course.<br /> <br /> <br /> '''Soon after''':<br /> * Request '''accounts''' on TALC for yourself, and TA who is going to be helping with the course.<br /> <br /> <br /> '''Please note that'''<br /> : it is '''instructor's responsibility''' to make sure that the '''software''' required for the course is '''available''' and '''works''' on TALC. <br /> : if '''special software''' is required for the course it is recommended that the instructor contacts us '''well before the course starts''' to allow enough time for installation and testing of the software.<br /> <br /> <br /> * If you are planning to share data with the students, request a '''shared directory for the course'''. <br /> : There will also be a '''unix group''' on TALC to control access to the shared directory.<br /> : The directory name and the unix group will probably have the same name, like &quot;course601-21&quot;. <br /> : Your and TA's accounts have to be added to the access group.<br /> <br /> <br /> * The data for courses run on TALC is deleted once the course is over. So that the '''shared directory''' and '''software''' for a course have to be setup every time the course is run.<br /> : If you need to build / have some '''specific software''' on TALC for the course, please start early. <br /> : It needs to be '''installed and tested''' well before the course start, so that the solution can be found in case something does not work.<br /> <br /> <br /> * If you '''need help with setting up the software''' for the course, let the RCS support know at support@hpc.ucalgary.ca .<br /> <br /> <br /> * During the course, if students have '''difficulties with using TALC''', the TAs are expected to help the students.<br /> : If the TA cannot solve the issue, we prefer that the TA contact us rather than the student who has the problem.<br /> : If TAs need training, this has to be arranged before the course with us (RCS);<br /> <br /> <br /> * As soon as you have a '''list of students''' who are going to take the course, please send it to support@hpc.ucalgary.ca .<br /> : The list has to have students' '''names''' as well as associated '''UofC email addresses'''.<br /> : The accounts will be created and added to the access group.<br /> <br /> <br /> * If you have any concerns or questions about running a course on TALC please let us know.<br /> <br /> = Courses =<br /> <br /> == Current Courses ==<br /> <br /> === Winter 2022 ===<br /> * ENSF 511 - Industrial Internet of Things Systems and Data Analytics, Hatem Abou-Zeid <br /> * MDSC 301 - Introduction to Bioinformatics, Tatiana Maroilley<br /> * GLGY 605 - Groundwater Flow and Reactive Transport Modelling, Benjamin Tutolo (requested Nov 2021)<br /> <br /> * MDSC 201 - Applied Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * MDSC 519 - Advanced Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * DATA 608 - Developing Big Data Applications, Leanne Wu (requested Nov 2021).<br /> <br /> * BMEN 415 - Sensor Systems and Data Analytics, Ethan MacDonald (requested Dec 2021).<br /> <br /> * CPSC 601.04 - High Performance Scientific Computing and Visualization, Usman Alim (late request, Jan 2022);<br /> <br /> * ENEL 645 - Data Mining &amp; Machine Learning, Roberto Souza (late request, Jan 2022);<br /> <br /> == Past Courses ==<br /> <br /> === Fall 2021 ===<br /> <br /> * ENSF 619.01 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> * ENSF 619.02 - Roberto Medeiros de Souza<br /> * ENSF 612 - Gias Uddin and Ajoy Das<br /> <br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> === 2020 Winter ===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> === 2020 Winter Block Week ===<br /> * MDSC 395 D Anderson<br /> <br /> = Resources =<br /> <br /> == Academic Schedule ==<br /> * See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=MARC_Cluster_Guide&diff=1659 2022-01-19T22:41:18Z

<p>Fridman: /* Hardware */ changed 8 nodes to 4 nodes</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> {{Message Box<br /> |title=Need Help or have other MARC Related Questions?<br /> |message=For all general RCS related issues, questions, or comments, please contact us at support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> <br /> This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary.<br /> <br /> It is intended to be read by new account holders getting started on MARC, covering such topics as the hardware and performance characteristics, available software, usage policies and how to log in.<br /> <br /> If you are looking for how to login to MARC or how to get an account, please see [[MARC_accounts]]<br /> <br /> == Introduction ==<br /> MARC is a cluster comprised of Linux-based computers purchased in 2019<br /> <br /> The MARC cluster has been designed with controls appropriate for Level 3 and Level 4 classified data. The University of Calgary Information Security Classification Standard is published here: https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf<br /> <br /> Due to security requirements for Level 3/4 data, some necessary restrictions have been placed on MARC to prevent accidental (or otherwise) data exfiltration.<br /> * Compute nodes and login nodes have no access to the internet<br /> * All data must be ingested to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.<br /> * Resulting data (outputs of analyses) must be copyied to SCDS and then fetching it from SCDS to wherever it needs to go using established means.<br /> * All file accesses are recorded for auditing purposes.<br /> * ssh connections to MARC must be through the IT Citrix system (Admin VPN is not sufficient nor necessary)<br /> * All accounts must be IT accounts<br /> * A project ID is required to use MARC. This project ID is the same number that is used on SCDS<br /> <br /> == Hardware ==<br /> MARC has compute nodes of two different varieties: <br /> * 4 GPU (Graphics Processing Unit)-enabled nodes containing:<br /> ** 40-cores: each node having 2 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz. <br /> ** The 40 cores on the individual compute nodes share about 750 GB of RAM (memory) but, jobs should request no more than 753000 MB.<br /> ** Two Tesla V100-PCIE-16GB GPUs.<br /> * 1 Bigmem Node<br /> ** 80-cores: node with 4 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz. <br /> ** The 80 cores on the node share about 3 TB of RAM (memory), but, jobs should request no more than 3000000 MB.<br /> <br /> === cpu2019 ===<br /> Allows non-GPU jobs to use:<br /> * Up to 38 cpus per node <br /> * No gpus.<br /> * Up to 500GB memory<br /> * Are the same <br /> <br /> === gpu2019 ===<br /> Allows jobs requiring nVidia v100 gpu jobs to use:<br /> * 1 or 2 gpus per node <br /> * Up to 40 cpus per node.<br /> * Up to 750GB memory<br /> <br /> === bigmem ===<br /> For very large memory jobs:<br /> * Up to 80 cpus<br /> * Up to 3TB memory<br /> * No gpus<br /> <br /> == Storage ==<br /> About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits. As described below, there are two storage areas: home and project.<br /> <br /> === Home file system: /home ===<br /> There is a per-user quota of 25 GB under /home. This limit is fixed and cannot be increased. Each user has a directory under /home, which is the default working directory when logging in to MARC. It is expected that most researchers will work from /project and only use home for software and such things. /home is expected to be used only for L1/L2 data and not for your patient identifiable files. The identifiable files go in the appropriate directory under /project.<br /> <br /> === Project file system for larger projects: /project ===<br /> Directories will be created in /project named after your project ID. This name will be the same as your SCDS share name. The expectation is that all files to do with that project will be stored in /project/projectid. Quotas in /project are somewhat flexible. Please write to support@hpc.ucalgary.ca with an estimate of how much space you will require.<br /> <br /> == Software installations ==<br /> <br /> === Python ===<br /> <br /> There are some complications in using Python on MARC relative to using ARC. <br /> Normally, we would recommend installing conda in user's home directory. <br /> On MARC, security requirements for working with L4 data require that we block outgoing and incoming internet connections. <br /> As a result, new packages cannot be downloaded with conda. <br /> <br /> Depending on what you need, the two recommendations we can make are<br /> <br /> * Download the standard anaconda distribution from the anaconda website to a personal computer: https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh<br /> ** Transfer the script to MARC via SCDS<br /> ** Copy it to your /home directory<br /> ** Install it in your home directory with $bash Anaconda3-2020.07-Linux-x86_64.sh<br /> ** you will be asked to agree to a license agreement and to confirm that you wish to create a folder anaconda3 once the installation completes, you will have a new directory under your home directory ~/anaconda3. In order to make it possible to use the local conda instance you will need to change the system path to include your local python directories $ export PATH=~/anaconda3/bin:$PATH<br /> * Download a docker container with the software that you need including python (e.g. tensorflow-gpu)<br /> ** Transfer the docker container to MARC via SCDS<br /> ** Copy it to your /home directory<br /> ** Run it with singularity<br /> <br /> <br /> * Non-open source software which requires a connection to a license server may require admin assistance to set up. contact [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] for support.&lt;br /&gt;<br /> <br /> == Further Reading ==<br /> See [[Running_jobs]] for information on starting a job<br /> <br /> [[Category:MARC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=CloudStack&diff=1658 2022-01-18T18:07:49Z

<p>Fridman: </p> <hr /> <div>Apache CloudStack is the RCS Virtual Machine (VM) Infrastructure as a Service (IaaS) offering for University of Calgary Researchers in order to allow them to quickly deploy VMs to support short term projects. It is intended as a proof of concept or for prototyping environments, where resources are quickly spun up as required. Once the project matures, the expectation would be to transition to a more permanent environment such as I.T. VMware.<br /> <br /> Researchers have a great degree of freedom in how they use CloudStack. As such, they are solely responsible for maintaining any VMs they deploy. The expectation is that the owners of the VMs will be patching and doing other maintenance work on a regular basis.<br /> <br /> Note, as the CloudStack deployment is a research environment, it is not intended for long term resilient service provision.<br /> <br /> ==Release Date==<br /> We expect that CloudStack will be available for UofC researchers at the end of March.<br /> ==Requesting Access==<br /> Requests to use CloudStack are done though I.T's Service Now request software. The request form is being finalized and further instruction will be forthcoming.<br /> ==End User Agreement==<br /> Please see https://rcs.ucalgary.ca/CloudStack_End_User_Agreement</div>

Fridman https://rcs.ucalgary.ca/index.php?title=CloudStack&diff=1657 2022-01-18T18:07:23Z

<p>Fridman: </p> <hr /> <div>Apache CloudStack is the RCS Virtual Machine (VM) Infrastructure as a Service (IaaS) offering for University of Calgary Researchers in order to allow them to quickly deploy VMs to support short term projects. It is intended as a proof of concept or for prototyping environments, where resources are quickly spun up as required. Once the project matures, the expectation would be to transition to a more permanent environment such as I.T. VMware.<br /> <br /> Researchers have a great degree of freedom in how they use CloudStack. As such, they are solely responsible for maintaining any VMs they deploy. The expectation is that the owners of the VMs will be patching and doing other maintenance work on a regular basis.<br /> <br /> Note, as the CloudStack deployment is a research environment, it is not intended for long term resilient service provision.<br /> <br /> ==Release Date==<br /> We expect that CloudStack will be available for UofC researchers at the end of March.<br /> ==Requesting Access==<br /> Requests to use CloudStack are done though I.T's Service Now request software. The request form is being finalized and further instruction will be forthcoming.<br /> ==End User Agreement==<br /> Please see &lt;https://rcs.ucalgary.ca/CloudStack_End_User_Agreement&gt;</div>

Fridman https://rcs.ucalgary.ca/index.php?title=CloudStack&diff=1656 2022-01-18T18:06:25Z

<p>Fridman: </p> <hr /> <div>Apache CloudStack is the RCS Virtual Machine (VM) Infrastructure as a Service (IaaS) offering for University of Calgary Researchers in order to allow them to quickly deploy VMs to support short term projects. It is intended as a proof of concept or for prototyping environments, where resources are quickly spun up as required. Once the project matures, the expectation would be to transition to a more permanent environment such as I.T. VMware.<br /> <br /> Researchers have a great degree of freedom in how they use CloudStack. As such, they are solely responsible for maintaining any VMs they deploy. The expectation is that the owners of the VMs will be patching and doing other maintenance work on a regular basis.<br /> <br /> Note, as the CloudStack deployment is a research environment, it is not intended for long term resilient service provision.<br /> <br /> ==Release Date==<br /> We expect that CloudStack will be available for UofC researchers at the end of March.<br /> ==Requesting Access==<br /> Requests to use CloudStack are done though I.T's Service Now request software. The request form is being finalized and further instruction will be forthcoming.</div>

Fridman https://rcs.ucalgary.ca/index.php?title=CloudStack&diff=1655 2022-01-18T18:02:15Z

<p>Fridman: </p> <hr /> <div>Apache CloudStack is the RCS Virtual Machine (VM) Infrastructure as a Service (IaaS) offering for University of Calgary Researchers in order to allow them to quickly deploy VMs to support short term projects. It is intended as a proof of concept or for prototyping environments, where resources are quickly spun up as required. Once the project matures, the expectation would be to transition to a more permanent environment such as I.T. VMware.<br /> <br /> Researchers have a great degree of freedom in how they use CloudStack. As such, they are solely responsible for maintaining any VMs they deploy. The expectation is that the owners of the VMs will be patching and doing other maintenance work on a regular basis.<br /> <br /> Note, as the CloudStack deployment is a research environment, it is not intended for long term resilient service provision.</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1623 2022-01-07T19:26:58Z

<p>Fridman: /* Winter 2022 */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> <br /> = Information =<br /> <br /> * TALC Terms of Use: https://rcs.ucalgary.ca/TALC_Terms_of_Use<br /> <br /> * TALC Guide: https://rcs.ucalgary.ca/TALC_Cluster<br /> <br /> <br /> == A New Course Checklist ==<br /> <br /> At least '''three (3) weeks''' before the course.<br /> * Please contact us about the course you are going to teach using TALC.<br /> <br /> * Please provide us with the '''course ID''', '''name of the course''', and the '''name of the instructor''' for the course.<br /> <br /> <br /> '''Soon after''':<br /> * Request '''accounts''' on TALC for yourself, and TA who is going to be helping with the course.<br /> <br /> <br /> '''Please note that'''<br /> : it is '''instructor's responsibility''' to make sure that the '''software''' required for the course is '''available''' and '''works''' on TALC. <br /> : if '''special software''' is required for the course it is recommended that the instructor contacts us '''well before the course starts''' to allow enough time for installation and testing of the software.<br /> <br /> <br /> * If you are planning to share data with the students, request a '''shared directory for the course'''. <br /> : There will also be a '''unix group''' on TALC to control access to the shared directory.<br /> : The directory name and the unix group will probably have the same name, like &quot;course601-21&quot;. <br /> : Your and TA's accounts have to be added to the access group.<br /> <br /> <br /> * The data for courses run on TALC is deleted once the course is over. So that the '''shared directory''' and '''software''' for a course have to be setup every time the course is run.<br /> : If you need to build / have some '''specific software''' on TALC for the course, please start early. <br /> : It needs to be '''installed and tested''' well before the course start, so that the solution can be found in case something does not work.<br /> <br /> <br /> * If you '''need help with setting up the software''' for the course, let the RCS support know at support@hpc.ucalgary.ca .<br /> <br /> <br /> * During the course, if students have '''difficulties with using TALC''', the TAs are expected to help the students.<br /> : If the TA cannot solve the issue, we prefer that the TA contact us rather than the student who has the problem.<br /> : If TAs need training, this has to be arranged before the course with us (RCS);<br /> <br /> <br /> * As soon as you have a '''list of students''' who are going to take the course, please send it to support@hpc.ucalgary.ca .<br /> : The list has to have students' '''names''' as well as associated '''UofC email addresses'''.<br /> : The accounts will be created and added to the access group.<br /> <br /> <br /> * If you have any concerns or questions about running a course on TALC please let us know.<br /> <br /> = Courses =<br /> <br /> == Current Courses ==<br /> <br /> === Winter 2022 ===<br /> * MDSC 301 - Introduction to Bioinformatics, Tatiana Maroilley<br /> * GLGY 605 - Groundwater Flow and Reactive Transport Modelling, Benjamin Tutolo (requested Nov 2021)<br /> <br /> * MDSC 201 - Applied Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * MDSC 519 - Advanced Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * DATA 608 - Developing Big Data Applications, Leanne Wu (requested Nov 2021).<br /> <br /> * BMEN 415 - Sensor Systems and Data Analytics, Ethan MacDonald (requested Dec 2021).<br /> <br /> * CPSC 601.04 - High Performance Scientific Computing and Visualization, Usman Alim (late request, Jan 2022);<br /> <br /> * ENEL 645 - Data Mining &amp; Machine Learning, Roberto Souza (late request, Jan 2022);<br /> <br /> == Past Courses ==<br /> <br /> === Fall 2021 ===<br /> <br /> * ENSF 619.01 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> * ENSF 619.02 - Roberto Medeiros de Souza<br /> * ENSF 612 - Gias Uddin and Ajoy Das<br /> <br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> === 2020 Winter ===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> === 2020 Winter Block Week ===<br /> * MDSC 395 D Anderson<br /> <br /> = Resources =<br /> <br /> == Academic Schedule ==<br /> * See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1622 2022-01-07T19:25:10Z

<p>Fridman: /* Winter 2022 */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> <br /> = Information =<br /> <br /> * TALC Terms of Use: https://rcs.ucalgary.ca/TALC_Terms_of_Use<br /> <br /> * TALC Guide: https://rcs.ucalgary.ca/TALC_Cluster<br /> <br /> <br /> == A New Course Checklist ==<br /> <br /> At least '''three (3) weeks''' before the course.<br /> * Please contact us about the course you are going to teach using TALC.<br /> <br /> * Please provide us with the '''course ID''', '''name of the course''', and the '''name of the instructor''' for the course.<br /> <br /> <br /> '''Soon after''':<br /> * Request '''accounts''' on TALC for yourself, and TA who is going to be helping with the course.<br /> <br /> <br /> '''Please note that'''<br /> : it is '''instructor's responsibility''' to make sure that the '''software''' required for the course is '''available''' and '''works''' on TALC. <br /> : if '''special software''' is required for the course it is recommended that the instructor contacts us '''well before the course starts''' to allow enough time for installation and testing of the software.<br /> <br /> <br /> * If you are planning to share data with the students, request a '''shared directory for the course'''. <br /> : There will also be a '''unix group''' on TALC to control access to the shared directory.<br /> : The directory name and the unix group will probably have the same name, like &quot;course601-21&quot;. <br /> : Your and TA's accounts have to be added to the access group.<br /> <br /> <br /> * The data for courses run on TALC is deleted once the course is over. So that the '''shared directory''' and '''software''' for a course have to be setup every time the course is run.<br /> : If you need to build / have some '''specific software''' on TALC for the course, please start early. <br /> : It needs to be '''installed and tested''' well before the course start, so that the solution can be found in case something does not work.<br /> <br /> <br /> * If you '''need help with setting up the software''' for the course, let the RCS support know at support@hpc.ucalgary.ca .<br /> <br /> <br /> * During the course, if students have '''difficulties with using TALC''', the TAs are expected to help the students.<br /> : If the TA cannot solve the issue, we prefer that the TA contact us rather than the student who has the problem.<br /> : If TAs need training, this has to be arranged before the course with us (RCS);<br /> <br /> <br /> * As soon as you have a '''list of students''' who are going to take the course, please send it to support@hpc.ucalgary.ca .<br /> : The list has to have students' '''names''' as well as associated '''UofC email addresses'''.<br /> : The accounts will be created and added to the access group.<br /> <br /> <br /> * If you have any concerns or questions about running a course on TALC please let us know.<br /> <br /> = Courses =<br /> <br /> == Current Courses ==<br /> <br /> === Winter 2022 ===<br /> * MDSC 301 - Tatiana Maroilley <br /> * GLGY 605 - Groundwater Flow and Reactive Transport Modelling, Benjamin Tutolo (requested Nov 2021)<br /> <br /> * MDSC 201 - Applied Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * MDSC 519 - Advanced Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * DATA 608 - Developing Big Data Applications, Leanne Wu (requested Nov 2021).<br /> <br /> * BMEN 415 - Sensor Systems and Data Analytics, Ethan MacDonald (requested Dec 2021).<br /> <br /> * CPSC 601.04 - High Performance Scientific Computing and Visualization, Usman Alim (late request, Jan 2022);<br /> <br /> * ENEL 645 - Data Mining &amp; Machine Learning, Roberto Souza (late request, Jan 2022);<br /> <br /> == Past Courses ==<br /> <br /> === Fall 2021 ===<br /> <br /> * ENSF 619.01 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> * ENSF 619.02 - Roberto Medeiros de Souza<br /> * ENSF 612 - Gias Uddin and Ajoy Das<br /> <br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> === 2020 Winter ===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> === 2020 Winter Block Week ===<br /> * MDSC 395 D Anderson<br /> <br /> = Resources =<br /> <br /> == Academic Schedule ==<br /> * See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1617 2022-01-06T22:04:05Z

<p>Fridman: /* Time limits */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu16<br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit <br /> ---------- ----------- -------------------- --------- <br /> normal 1-00:00:00 <br /> cpulimit cpu=48 <br /> gpucpulim+ cpu=18 <br /> gpulimit cpu=2,gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=cpu16<br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br /> AllocNodes=ALL Default=YES QoS=cpulimit<br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<br /> MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br /> Nodes=n[1-36]<br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO<br /> OverTimeLimit=NONE PreemptMode=OFF<br /> State=UP TotalCPUs=576 TotalNodes=36 SelectTypeParameters=NONE<br /> JobDefaults=(null)<br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br /> <br /> <br /> <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo<br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br /> cpu12 up 1-00:00:00 3 idle t[1-3]<br /> cpu16 up 1-00:00:00 36 idle n[1-36]<br /> bigmem up 1-00:00:00 2 idle bigmem[1-2]<br /> gpu up 1-00:00:00 3 idle t[1-3]<br /> <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1616 2022-01-06T22:02:29Z

<p>Fridman: /* Time limits */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu16<br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit <br /> ---------- ----------- -------------------- --------- <br /> normal 1-00:00:00 <br /> cpulimit cpu=48 <br /> gpucpulim+ cpu=18 <br /> gpulimit cpu=2,gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=cpu16<br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br /> AllocNodes=ALL Default=YES QoS=cpulimit<br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<br /> MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br /> Nodes=n[1-36]<br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO<br /> OverTimeLimit=NONE PreemptMode=OFF<br /> State=UP TotalCPUs=576 TotalNodes=36 SelectTypeParameters=NONE<br /> JobDefaults=(null)<br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br /> <br /> <br /> <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1615 2022-01-06T22:01:27Z

<p>Fridman: /* Partition limitations */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu16<br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit <br /> ---------- ----------- -------------------- --------- <br /> normal 1-00:00:00 <br /> cpulimit cpu=48 <br /> gpucpulim+ cpu=18 <br /> gpulimit cpu=2,gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=cpu16<br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br /> AllocNodes=ALL Default=NO QoS=cpu16<br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<br /> MaxNodes=UNLIMITED MaxTime=22:10:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br /> Nodes=n[1-36]<br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO<br /> OverTimeLimit=NONE PreemptMode=OFF<br /> State=UP TotalCPUs=576 TotalNodes=36 SelectTypeParameters=NONE<br /> JobDefaults=(null)<br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br /> <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1614 2022-01-06T19:41:16Z

<p>Fridman: /* Winter 2022 */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> <br /> = Information =<br /> <br /> * TALC Terms of Use: https://rcs.ucalgary.ca/TALC_Terms_of_Use<br /> <br /> * TALC Guide: https://rcs.ucalgary.ca/TALC_Cluster<br /> <br /> <br /> == A New Course Checklist ==<br /> <br /> At least '''four weeks''' before the course.<br /> * Please contact us about the course you are going to teach using TALC.<br /> <br /> * Please provide us with the '''course ID''', '''name of the course''', and the '''name of the instructor''' for the course.<br /> <br /> <br /> '''Soon after''':<br /> * Request '''accounts''' on TALC for yourself, and TA who is going to be helping with the course.<br /> <br /> <br /> '''Please note that'''<br /> : it is '''instructor's responsibility''' to make sure that the '''software''' required for the course is '''available''' and '''works''' on TALC. <br /> : it is recommended that if special software is required for the course '''well before the course starts''' to allow enough time for installation and testing of the software.<br /> <br /> <br /> * If you are planning to share data with the students, request a '''shared directory for the course'''. <br /> : There will also be a '''unix group''' on TALC to control access to the shared directory.<br /> : The directory name and the unix group will probably have the same name, like &quot;course601-21&quot;. <br /> : Your and TA's accounts have to be added to the access group.<br /> <br /> <br /> * The data for courses run on TALC is deleted once the course is over. So that the '''shared directory''' and '''software''' for a course have to be setup every time the course is run.<br /> : If you need to build / have some '''specific software''' on TALC for the course, please start early. <br /> : It needs to be '''installed and tested''' well before the course start, so that the solution can be found in case something does not work.<br /> <br /> <br /> * If you '''need help with setting up the software''' for the course, let the RCS support know at support@hpc.ucalgary.ca .<br /> <br /> <br /> * During the course, if students have '''difficulties with using TALC''', the TAs are expected to help the students.<br /> : If the TA cannot solve the issue, we prefer that the TA contact us rather than the student who has the problem.<br /> : If TAs need training, this has to be arranged before the course with us (RCS);<br /> <br /> <br /> * As soon as you have a '''list of students''' who are going to take the course, please send it to support@hpc.ucalgary.ca .<br /> : The list has to have students' '''names''' as well as associated '''UofC email addresses'''.<br /> : The accounts will be created and added to the access group.<br /> <br /> <br /> * If you have any concerns or questions about running a course on TALC please let us know.<br /> <br /> = Courses =<br /> <br /> == Current Courses ==<br /> <br /> === Winter 2022 ===<br /> <br /> * GLGY 605 - Groundwater Flow and Reactive Transport Modelling, Benjamin Tutolo (requested Nov 2021)<br /> <br /> * MDSC 201 - Applied Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * MDSC 519 - Advanced Bioinformatics, David Anderson (requested Nov 2021)<br /> <br /> * DATA 608 - Developing Big Data Applications, Leanne Wu (requested Nov 2021).<br /> <br /> * BMEN 415 - Sensor Systems and Data Analytics, Ethan MacDonald (requested Dec 2021).<br /> <br /> * CPSC 601.04 - High Performance Scientific Computing and Visualization, Usman Alim (late request, Jan 2022);<br /> <br /> * ENEL 645 - Data Mining &amp; Machine Learning, Roberto Souza (late request, Jan 2022);<br /> <br /> == Past Courses ==<br /> <br /> === Fall 2021 ===<br /> <br /> * ENSF 619.01 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> * ENSF 619.02 - Roberto Medeiros de Souza<br /> * ENSF 612 - Gias Uddin and Ajoy Das<br /> <br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> === 2020 Winter ===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> === 2020 Winter Block Week ===<br /> * MDSC 395 D Anderson<br /> <br /> = Resources =<br /> <br /> == Academic Schedule ==<br /> * See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=HPC_Systems&diff=1613 2022-01-06T18:28:15Z

<p>Fridman: </p> <hr /> <div>RCS manages various high performance computing clusters for the University of Calgary.<br /> <br /> {{Clear}}<br /> &lt;div class=&quot;row&quot;&gt;<br /> {{Cluster Box<br /> |letter=A<br /> |color=cef2e0<br /> |link=ARC Cluster Guide<br /> |title=ARC - Advanced Research Computing<br /> |description=ARC is a general purpose cluster for University of Calgary researchers.<br /> |content=<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[ARC Cluster Guide|ARC Cluster Guide]]<br /> * [[Managing software on ARC]]<br /> * [[Gaussian on ARC]] -- How to use Gaussian 16 on ARC.<br /> * [[Apache Spark on ARC]]<br /> }}<br /> {{Cluster Box<br /> |letter=T<br /> |color=ddcef2<br /> |link=TALC Cluster Guide<br /> |title=TALC - Teaching And Learning Computing<br /> |description=TALC is a cluster to support academic courses and workshops.<br /> |content=<br /> * [[TALC Cluster Guide|TALC Cluster Guide]]<br /> * [[TALC Terms of Use]]<br /> * [[List of courses on TALC]] <br /> }}<br /> {{Clear}}<br /> {{Cluster Box<br /> |letter=M<br /> |color=f2cece<br /> |link=MARC Cluster Guide<br /> |title=MARC - Medical Advanced Research Computing<br /> |description=MARC is a specialized cluster for processing level 4 data.<br /> |content=<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[MARC Cluster Guide]]<br /> * [[MARC accounts]]<br /> }}<br /> {{Cluster Box<br /> |letter=G<br /> |color=cedff2<br /> |title=GLaDOS<br /> |description=GLaDOS is a researcher-owned compute cluster<br /> |content=<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[GLaDOS Guide | GLaDOS cluster]]<br /> }}<br /> {{Clear}}<br /> {{Cluster Box<br /> |letter=H<br /> |color=f2ecce<br /> |title=Helix <br /> |description=Helix is a specialized cluster for Cumming School of Medicine projects<br /> |content=<br /> * [[General Cluster Guidelines and Policies]]<br /> * [[Helix Guide | Helix cluster]]<br /> }}<br /> <br /> &lt;/div&gt;<br /> {{Clear}}</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Terms_of_Use&diff=1612 2022-01-06T18:22:53Z

<p>Fridman: /* Guidelines and Regulations */</p> <hr /> <div>The [[TALC Cluster Guide|Teaching and Learning Cluster (TALC)]] is a computing resource provided by Research Computing Services (RCS) to support approved courses and workshops. Usage of the cluster is<br /> subject to certain conditions as outlined below and detailed on this page.<br /> <br /> {{Message Box<br /> |title=Questions or Concerns?<br /> |message=Please send all questions and inquiries to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Guidelines and Regulations==<br /> * Usage must not violate Municipal, Provincial and Federal laws<br /> * Usage must not violate the University's Policies and Procedures outlined in the [https://www.ucalgary.ca/legal-services/university-policies-procedures/acceptable-use-electronic-resources-and-information-policy Acceptable Use of Electronic Resources and Information Policy]<br /> * This account is for your use only. It is made available to you specifically for the course that requires the use of TALC. You must not share your password or let anyone use your account.<br /> * Commercial use of TALC, including digital currency mining, is strictly prohibited.<br /> * TALC is configured for Level 1 and Level 2 data as set forth in the [https://www.ucalgary.ca/legal-services/sites/default/files/teams/1/Standards-Legal-Information-Security-Classification-Standard.pdf University of Calgary Information Security Classification Standard] and is not suitable for Level 3 or Level 4 data.<br /> * RCS reserves the right to examine files, programs and any other material used on RCS systems at any time without warning.<br /> * Evidence of inappropriate use of TALC may result in immediate loss of access to your TALC account and may result in academic misconduct.<br /> * Please note that no backups are performed for data stored on TALC. It is your responsibility to copy data you need to save elsewhere. By default, your account and data will be deleted one week after the last course associated with the account has finished.<br /> * To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to contact RCS several months prior to the start of the course. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Terms_of_Use&diff=1611 2022-01-06T18:21:15Z

<p>Fridman: </p> <hr /> <div>The [[TALC Cluster Guide|Teaching and Learning Cluster (TALC)]] is a computing resource provided by Research Computing Services (RCS) to support approved courses and workshops. Usage of the cluster is<br /> subject to certain conditions as outlined below and detailed on this page.<br /> <br /> {{Message Box<br /> |title=Questions or Concerns?<br /> |message=Please send all questions and inquiries to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Guidelines and Regulations==<br /> * Usage must not violate Municipal, Provincial and Federal laws<br /> * Usage must not violate the University's Policies and Procedures outlined in the [https://www.ucalgary.ca/legal-services/university-policies-procedures/acceptable-use-electronic-resources-and-information-policy Acceptable Use of Electronic Resources and Information Policy]<br /> * This account is for your use only. It is made available to you specifically for the course that requires the use of TALC. You must not share your password or let anyone use your account.<br /> * Commercial use of TALC, including digital currency mining, is strictly prohibited.<br /> * TALC is configured for Level 1 and Level 2 data as set forth in the [https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf University of Calgary Information Security Classification Standard] and is not suitable for Level 3 or Level 4 data.<br /> * RCS reserves the right to examine files, programs and any other material used on RCS systems at any time without warning.<br /> * Evidence of inappropriate use of TALC may result in immediate loss of access to your TALC account and may result in academic misconduct.<br /> * Please note that no backups are performed for data stored on TALC. It is your responsibility to copy data you need to save elsewhere. By default, your account and data will be deleted one week after the last course associated with the account has finished.<br /> * To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to contact RCS several months prior to the start of the course. <br /> <br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1610 2022-01-06T18:02:53Z

<p>Fridman: /* Time limits */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu16<br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=cpu16<br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br /> AllocNodes=ALL Default=NO QoS=cpu16<br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<br /> MaxNodes=UNLIMITED MaxTime=22:10:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br /> Nodes=n[1-36]<br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO<br /> OverTimeLimit=NONE PreemptMode=OFF<br /> State=UP TotalCPUs=576 TotalNodes=36 SelectTypeParameters=NONE<br /> JobDefaults=(null)<br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br /> <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1609 2022-01-06T17:58:13Z

<p>Fridman: /* Working interactively */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu16<br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1608 2022-01-06T17:57:41Z

<p>Fridman: /* /home: Home file system */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory will be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with others on the TALC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All TALC nodes run a version of Rocky Linux. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> TALC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. CPU intensive workloads on the login node should be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu24 <br /> <br /> <br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> TALC has a Jupyterhub server which runs a Jupyter server on one of the TALC compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a TALC account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu16 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are refer to the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores<br /> |64 GB<br /> |62 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support for parallel processing using Message Passing Interface (MPI), OpenMP, etc. For example, MPI for parallel processing can distribute memory across multiple nodes, so that per-node memory requirements could be lower. Whereas OpenMP or single process serial code that is restricted to one node would require a higher memory node.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== CPU only jobs ====<br /> To select the &lt;code&gt;cpu16&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu16<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu16<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1607 2022-01-06T17:33:54Z

<p>Fridman: /* Hardware */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz (2012)<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with other researchers on the ARC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All ARC nodes run the latest version of CentOS 7 with the same set of base software packages. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> ARC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. We suggest CPU intensive workloads on the login node be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu24 <br /> <br /> <br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> Talc has a Jupyterhub server which runs a Jupyter server on one of the Talc compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a Talc account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu24 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are backed by the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores<br /> |256 GB<br /> |254 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support parallel processing using Message Passing Interface (MPI), OpenMP, etc.<br /> ** Eg. MPI for parallel processing can distribute memory across multiple nodes, per-node memory requirements could be lower. Whereas, OpenMP or single process code that is restricted to one node would require a higher memory node.<br /> ** Note: MPI code running on hardware with Omni-Path networking should be compiled with Omni-Path networking support. This is provided by loading the &lt;code&gt;openmpi/2.1.3-opa&lt;/code&gt; or &lt;code&gt;openmpi/3.1.2-opa&lt;/code&gt; modules prior to compiling.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== Bigmem and compute-only jobs ====<br /> To select the &lt;code&gt;cpu24&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu24<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu24<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1606 2022-01-06T17:33:02Z

<p>Fridman: /* Hardware */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel Xeon Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu16<br /> |General Purpose Compute<br /> |36<br /> |16 cores, 2x Eight-Core Intel Xeon CPU E5-2650 @ 2.00GHz<br /> |64 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with other researchers on the ARC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All ARC nodes run the latest version of CentOS 7 with the same set of base software packages. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> ARC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. We suggest CPU intensive workloads on the login node be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu24 <br /> <br /> <br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> Talc has a Jupyterhub server which runs a Jupyter server on one of the Talc compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a Talc account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu24 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are backed by the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores<br /> |256 GB<br /> |254 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support parallel processing using Message Passing Interface (MPI), OpenMP, etc.<br /> ** Eg. MPI for parallel processing can distribute memory across multiple nodes, per-node memory requirements could be lower. Whereas, OpenMP or single process code that is restricted to one node would require a higher memory node.<br /> ** Note: MPI code running on hardware with Omni-Path networking should be compiled with Omni-Path networking support. This is provided by loading the &lt;code&gt;openmpi/2.1.3-opa&lt;/code&gt; or &lt;code&gt;openmpi/3.1.2-opa&lt;/code&gt; modules prior to compiling.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== Bigmem and compute-only jobs ====<br /> To select the &lt;code&gt;cpu24&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu24<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu24<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1605 2022-01-06T17:22:43Z

<p>Fridman: /* Getting Support */</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca.<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores, 4x Six-Core AMD Opteron(tm) Processor 8431 (2009)<br /> |256 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with other researchers on the ARC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All ARC nodes run the latest version of CentOS 7 with the same set of base software packages. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> ARC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. We suggest CPU intensive workloads on the login node be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu24 <br /> <br /> <br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> Talc has a Jupyterhub server which runs a Jupyter server on one of the Talc compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a Talc account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu24 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are backed by the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores<br /> |256 GB<br /> |254 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support parallel processing using Message Passing Interface (MPI), OpenMP, etc.<br /> ** Eg. MPI for parallel processing can distribute memory across multiple nodes, per-node memory requirements could be lower. Whereas, OpenMP or single process code that is restricted to one node would require a higher memory node.<br /> ** Note: MPI code running on hardware with Omni-Path networking should be compiled with Omni-Path networking support. This is provided by loading the &lt;code&gt;openmpi/2.1.3-opa&lt;/code&gt; or &lt;code&gt;openmpi/3.1.2-opa&lt;/code&gt; modules prior to compiling.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== Bigmem and compute-only jobs ====<br /> To select the &lt;code&gt;cpu24&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu24<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu24<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=TALC_Cluster_Guide&diff=1604 2022-01-06T17:13:14Z

<p>Fridman: minor gramatical edits</p> <hr /> <div>{{Message Box<br /> |icon=Security Icon.png<br /> |title=Cybersecurity awareness at the U of C<br /> |message=Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. Please visit https://it.ucalgary.ca/it-security for more information, tips on secure computing, and how to report suspected security problems.}}<br /> <br /> This guide gives an overview of the Teaching and Learning Cluster (TALC) at the University of Calgary and is intended to be read by new account holders getting started on TALC. This guide covers topics as the hardware and performance characteristics, available software, usage policies and how to log in and run jobs. <br /> <br /> ==Introduction==<br /> TALC is a cluster of computers created by Research Computing Services (RCS) in response to requests for a central computing resource to support academic courses and workshops offered at the University of Calgary. It is a complement to the Advanced Research Computing (ARC) cluster that is used for research, rather than educational purposes. The software environment in the TALC and ARC clusters is very similar and workflows between the two clusters are identical. What students learn about using TALC will have direct applicability to using ARC should they go on to use ARC for research work. <br /> <br /> If you are the instructor for a course that could benefit from using TALC, please review this guide and the [[TALC Terms of Use]] and then contact us at support@hpc.ucalgary.ca to discuss your requirements. <br /> <br /> Please note that in order to ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> <br /> If you are a student in a course using TALC, please review this guide for basic instructions in using the cluster. Questions should first be directed to the teaching assistants or instructor for your course.<br /> <br /> ===Obtaining an account===<br /> TALC account requests are expected to be submitted by the course instructor rather than from individual students. You must have a University of Calgary IT account in order to use TALC. If you do not have a University of IT account or email address, please register for one at https://itregport.ucalgary.ca/. In order to ensure TALC is provisioned in time for a course start date, the instructor should submit the initial list of @ucalgary.ca accounts needed for the course 2 weeks before the start date.<br /> <br /> === Getting Support ===<br /> {{Message Box<br /> |title=Need Help or have other TALC Related Questions?<br /> |message='''Students''', please send TALC-related questions to your course instructor or teaching assistants.&lt;br /&gt;<br /> '''Course instructors and TAs''', please report system issues to support@hpc.ucalgary.ca).<br /> |icon=Support Icon.png}}<br /> <br /> ==Hardware==<br /> The TALC cluster is comprised of repurposed research clusters that are a few generations old. As a result, individual processor performance will not be comparable to the latest processors but should be sufficient for educational purposes and course work. <br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !CPU Cores, Model, and Year<br /> !Installed Memory<br /> !GPU<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores, 2x Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz (2019)<br /> |192 GB<br /> |5x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores, 4x Six-Core AMD Opteron(tm) Processor 8431 (2009)<br /> |256 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores, 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (2015)<br /> |1024 GB<br /> |N/A<br /> |40 Gbit/s InfiniBand<br /> |}<br /> <br /> ===Storage===<br /> {{Message Box<br /> | title=No Backup Policy!<br /> | message=You are responsible for your own backups. Since accounts on TALC and related data are removed shortly after the associated course has finished, you should download anything you need to save to your own computer before the end of the course.<br /> }}<br /> <br /> TALC is connected to a network disk storage system. This storage is split across the &lt;code&gt;/home&lt;/code&gt; and &lt;code&gt;/scratch&lt;/code&gt; file systems. <br /> ====&lt;code&gt;/home&lt;/code&gt;: Home file system====<br /> Each user has a directory under /home and is the default working directory when logging in to TALC. Each home directory has a per-user quota of 500 GB. This limit is fixed and cannot be increased.<br /> <br /> Note on file sharing: Due to security concerns, permissions set using &lt;code&gt;chmod&lt;/code&gt; on your home directory to allow other users to read/write to your home directory be automatically reverted by an automated system process unless an explicit exception is made. If you need to share files with other researchers on the ARC cluster, please write to support@hpc.ucalgary.ca to ask for such an exception.<br /> <br /> ====&lt;code&gt;/scratch&lt;/code&gt;: Scratch file system for large job-oriented storage====<br /> Associated with each job, under the &lt;code&gt;/scratch&lt;/code&gt; directory, a subdirectory is created that can be referenced in job scripts as &lt;code&gt;/scratch/${SLURM_JOB_ID}&lt;/code&gt;. You can use that directory for temporary files needed during the course of a job. Up to 30 TB of storage may be used, per user (total for all your jobs) in the &lt;code&gt;/scratch&lt;/code&gt; file system. <br /> <br /> Data in &lt;code&gt;/scratch&lt;/code&gt; associated with a given job will be deleted automatically, without exception, five days after the job finishes.<br /> <br /> == Software ==<br /> {{Message Box<br /> | title=Software Package Requests<br /> | message=Course instructors or teaching assistants should write to support@hpc.ucalgary.ca if additional software is required for their course.<br /> }}<br /> <br /> All ARC nodes run the latest version of CentOS 7 with the same set of base software packages. For your convenience, we have packaged commonly used software packages and dependencies as modules available under &lt;code&gt;/global/software&lt;/code&gt;. If your software package is not available as a module, you may also try Anaconda which allows users to manage and install custom packages in an isolated environment.<br /> <br /> For a list of available packages that have been made available, please see [[ARC Software pages]]. <br /> <br /> === Modules ===<br /> The setup of the environment for using some of the installed software is through the &lt;code&gt;module&lt;/code&gt; command.<br /> <br /> Software packages bundled as a module will be available under &lt;code&gt;/global/software&lt;/code&gt; and can be listed with the &lt;code&gt;module avail&lt;/code&gt; command.<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module avail<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To enable Python, load the Python module by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module load python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To unload the Python module, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module remove python/anaconda-3.6-5.1.0<br /> &lt;/syntaxhighlight&gt;<br /> <br /> To see currently loaded modules, run:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ module list<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==Using TALC==<br /> {{Message Box<br /> |title=Usage subject to [[TALC Terms of Use]]<br /> |message=Please review the [[TALC Terms of Use]] prior to using TALC.<br /> |icon=Support Icon.png}}<br /> <br /> ===Logging in===<br /> To log in to TALC, connect using SSH to talc.ucalgary.ca. Connections to TALC are accepted only from the University of Calgary network (on campus) or through the University of Calgary General VPN (off campus).<br /> <br /> See [[Connecting to RCS HPC Systems]] for more information.<br /> <br /> ===Working interactively===<br /> &lt;!-- original chunk --&gt;<br /> ARC uses the Linux operating system. The program that responds to your typed commands and allows you to run other programs is called the Linux shell. There are several different shells available, but, by default you will use one called bash. It is useful to have some knowledge of the shell and a variety of other command-line programs that you can use to manipulate files. If you are new to Linux systems, we recommend that you work through one of the many online tutorials that are available, such as the [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html UNIX Tutorial for Beginners (external link)] provided by the University of Surrey. The tutorial covers such fundamental topics, among others, as creating, renaming and deleting files and directories, how to produce a listing of your files and how to tell how much disk space you are using. For a more comprehensive introduction to Linux, see [http://linuxcommand.sourceforge.net/tlcl.php The Linux Command Line (external link)].<br /> <br /> The TALC login node may be used for such tasks as editing files, compiling programs and running short tests while developing programs. We suggest CPU intensive workloads on the login node be restricted to under 15 minutes as per [[General Cluster Guidelines and Policies|our cluster guidelines]]. For interactive workloads exceeding 15 minutes, use the '''[[Running_jobs#Interactive_jobs|salloc command]]''' to allocate an interactive session on a compute node.<br /> <br /> The default &lt;code&gt;salloc&lt;/code&gt; allocation is 1 CPU and 1 GB of memory. Adjust this by specifying &lt;code&gt;-n CPU#&lt;/code&gt; and &lt;code&gt;--mem Megabytes&lt;/code&gt;. You may request up to 5 hours of CPU time for interactive jobs.<br /> salloc --time 5:00:00 --partition cpu24 <br /> <br /> <br /> <br /> ===Running non-interactive jobs (batch processing)===<br /> Production runs and longer test runs should be submitted as (non-interactive) batch jobs, in which commands to be executed are listed in a script (text file). Batch jobs scripts are submitted using the &lt;code&gt;sbatch&lt;/code&gt; command, part of the Slurm job management and scheduling software. #SBATCH directive lines at the beginning of the script are used to specify the resources needed for the job (cores, memory, run time limit and any specialized hardware needed).<br /> <br /> Most of the information on the Running Jobs page on the Compute Canada web site is also relevant for submitting and managing batch jobs and reserving processors for interactive work on TALC. One major difference between running jobs on the TALC and Compute Canada clusters is in selecting the type of hardware that should be used for a job. On TALC, you choose the hardware to use primarily by specifying a partition, as described below.<br /> <br /> ===Using JupyterHub on Talc===<br /> Talc has a Jupyterhub server which runs a Jupyter server on one of the Talc compute nodes and provides all the necessary encryption and plumbing to deliver the notebook to your computer. To access this service you must have a Talc account. Point your browser at http://talc.ucalgary.ca and login with your usual UC account. As of this writing, the job that runs the jupyter notebook is 1 cpu and 10GiB of memory on a cpu24 node.<br /> <br /> ===Selecting a partition===<br /> TALC currently has the following partitions available for use. The &lt;code&gt;gpu&lt;/code&gt; and &lt;code&gt;cpu12&lt;/code&gt; partitions are backed by the same nodes. The &lt;code&gt;cpu12&lt;/code&gt; partition was created to only expose the CPUs on the GPU hardware for general purpose use. Each GPU node has 5 Tesla T4 GPUs installed, but you may only request one per job within the TALC environment.<br /> {| class=&quot;wikitable&quot;<br /> !Partition<br /> !Description<br /> !Nodes<br /> !Cores<br /> !Memory <br /> !Memory Request Limit<br /> !Time Limit<br /> !GPU Request per Job<br /> !Network<br /> |-<br /> |gpu<br /> |GPU Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |1x NVIDIA Corporation TU104GL [Tesla T4]<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu12<br /> |General Purpose Compute<br /> |3<br /> |12 cores<br /> |192 GB<br /> |190 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |cpu24<br /> |General Purpose Compute<br /> |15<br /> |24 cores<br /> |256 GB<br /> |254 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |-<br /> |bigmem<br /> |General Purpose Compute<br /> |2<br /> |32 cores<br /> |1024 GB<br /> |1022 GB<br /> |24 hours<br /> |None<br /> |40 Gbit/s InfiniBand<br /> |}<br /> There are some aspects to consider when selecting a partition including:<br /> * Resource requirements in terms of memory and CPU cores<br /> * Hardware specific requirements, such as GPU or CPU Instruction Set Extensions<br /> * Partition resource limits and potential wait time<br /> * Software support parallel processing using Message Passing Interface (MPI), OpenMP, etc.<br /> ** Eg. MPI for parallel processing can distribute memory across multiple nodes, per-node memory requirements could be lower. Whereas, OpenMP or single process code that is restricted to one node would require a higher memory node.<br /> ** Note: MPI code running on hardware with Omni-Path networking should be compiled with Omni-Path networking support. This is provided by loading the &lt;code&gt;openmpi/2.1.3-opa&lt;/code&gt; or &lt;code&gt;openmpi/3.1.2-opa&lt;/code&gt; modules prior to compiling.<br /> <br /> Since resources that are requested are reserved for your job, please request only as much CPU and memory as your job requires to avoid reducing the cluster efficiency. If you are unsure which partition to use or the specific resource requests that are appropriate for your jobs, please contact us at [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] and we would be happy to work with you.<br /> <br /> === Using a partition ===<br /> <br /> ==== Bigmem and compute-only jobs ====<br /> To select the &lt;code&gt;cpu24&lt;/code&gt; partition, include the following line in your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=cpu24<br /> &lt;/syntaxhighlight&gt;You may also start an interactive session with &lt;code&gt;salloc&lt;/code&gt;:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p cpu24<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== GPU jobs ====<br /> In TALC, you are limited to exactly 1 GPU per job. Jobs that request for 0 GPUs or 2 or more GPUs will not be scheduled.<br /> <br /> To submit a job using the &lt;code&gt;gpu&lt;/code&gt; partition with one GPU request, include the following to your batch job script:&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> #SBATCH --partition=gpu<br /> #SBATCH --gpus-per-node=1<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Like the previous example, you may also request interactive sessions with GPU nodes using &lt;code&gt;salloc&lt;/code&gt;. Just specify the &lt;code&gt;gpu&lt;/code&gt; partition and the number of GPUs required. &lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ salloc --time 1:00:00 -p gpu -n 1 --gpus-per-node 1 <br /> &lt;/syntaxhighlight&gt;You may verify that a GPU was assigned to your job or interactive session by running &lt;code&gt;nvidia-smi&lt;/code&gt;. This command will show you the status of the GPU that was assigned to you.&lt;syntaxhighlight lang=&quot;text&quot;&gt;<br /> $ nvidia-smi<br /> +-----------------------------------------------------------------------------+<br /> | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |<br /> |-------------------------------+----------------------+----------------------+<br /> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br /> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br /> | | | MIG M. |<br /> |===============================+======================+======================|<br /> | 0 Tesla T4 Off | 00000000:3B:00.0 Off | 0 |<br /> | N/A 36C P0 14W / 70W | 0MiB / 15109MiB | 5% Default |<br /> | | | N/A |<br /> +-------------------------------+----------------------+----------------------+<br /> <br /> +-----------------------------------------------------------------------------+<br /> | Processes: |<br /> | GPU GI CI PID Type Process name GPU Memory |<br /> | ID ID Usage |<br /> |=============================================================================|<br /> | No running processes found |<br /> +-----------------------------------------------------------------------------+<br /> &lt;/syntaxhighlight&gt;<br /> <br /> ==== Partition limitations ====<br /> In addition to the hardware limitations of the nodes within the partition, please be aware that there may also be policy limits imposed on your account for each partition. These limits restrict the number of cores, nodes, or GPUs that can be used at any given time. Since the limits are applied on a partition-by-partition basis, using resources in one partition should not affect the available resources you can use in another partition.<br /> <br /> These limits can be listed by running:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sacctmgr show qos format=Name,MaxWall,MaxTRESPU%20,MaxSubmitJobs<br /> Name MaxWall MaxTRESPU MaxSubmit<br /> ---------- ----------- -------------------- ---------<br /> normal 1-00:00:00 mem=127000M <br /> cpu24 1-00:00:00 mem=127G <br /> bigmem 1-00:00:00 <br /> gpu gres/gpu=1 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> === Time limits ===<br /> Use the &lt;code&gt;--time&lt;/code&gt; directive to tell the job scheduler the maximum time that your job might run. For example:<br /> #SBATCH --time=hh:mm:ss<br /> <br /> You can use &lt;code&gt;scontrol show partitions&lt;/code&gt; or &lt;code&gt;sinfo&lt;/code&gt; to see the current maximum time that a job can run.<br /> &lt;syntaxhighlight lang=&quot;bash&quot; highlight=&quot;6&quot;&gt;<br /> $ scontrol show partitions<br /> PartitionName=single <br /> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL <br /> AllocNodes=ALL Default=NO QoS=single <br /> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO <br /> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED <br /> Nodes=cn[001-168] <br /> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO <br /> OverTimeLimit=NONE PreemptMode=OFF <br /> State=UP TotalCPUs=1344 TotalNodes=168 SelectTypeParameters=NONE <br /> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED <br /> &lt;/syntaxhighlight&gt;<br /> <br /> Alternatively, with &lt;code&gt;sinfo&lt;/code&gt; under the &lt;code&gt;TIMELIMIT&lt;/code&gt; column:<br /> &lt;syntaxhighlight lang=&quot;bash&quot;&gt;<br /> $ sinfo <br /> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST <br /> single up 7-00:00:00 1 drain* cn097 <br /> single up 7-00:00:00 1 maint cn002 <br /> single up 7-00:00:00 4 drain* cn[001,061,133,154] <br /> ...<br /> &lt;/syntaxhighlight&gt;<br /> [[Category:TALC]]<br /> [[Category:Guides]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1484 2021-09-08T20:51:34Z

<p>Fridman: /* Current Courses */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> __NOTOC__<br /> <br /> == Current Courses ==<br /> <br /> * ENSF 619.01 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> * ENSF 619 - Roberto Medeiros de Souza<br /> <br /> == Previous Courses ==<br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> ===2020 Winter===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> ===2020 Winter Block Week===<br /> * MDSC 395 D Anderson<br /> <br /> == See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. ==<br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1483 2021-09-07T20:40:15Z

<p>Fridman: /* Current Courses */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> __NOTOC__<br /> <br /> == Current Courses ==<br /> <br /> * ENSF61901 - Ethan MacDonald<br /> * MDSC 523 - David Anderson<br /> <br /> == Previous Courses ==<br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> ===2020 Winter===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> ===2020 Winter Block Week===<br /> * MDSC 395 D Anderson<br /> <br /> == See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. ==<br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1482 2021-09-07T19:03:25Z

<p>Fridman: /* Previous Courses */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> __NOTOC__<br /> <br /> == Current Courses ==<br /> <br /> * ENSF61901 - Ethan MacDonald<br /> <br /> == Previous Courses ==<br /> === 2021 Winter ===<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> ===2020 Winter===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> ===2020 Winter Block Week===<br /> * MDSC 395 D Anderson<br /> <br /> == See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. ==<br /> [[Category:TALC]]</div>

Fridman https://rcs.ucalgary.ca/index.php?title=List_of_courses_on_TALC&diff=1308 2021-04-14T16:31:13Z

<p>Fridman: /* 2020 Winter Block Week */</p> <hr /> <div>{{Message Box<br /> |title=Interested in using TALC for your course?<br /> |message='''If you are the instructor for a course that could benefit from using TALC, please contact us at support@hpc.ucalgary.ca to discuss your requirements.'''<br /> &lt;br &gt;To ensure that the appropriate software is available, student accounts are in place, and appropriate training has been provided for your teaching assistants, it is best to start this discussion several months prior to the start of the course.<br /> |icon=Support Icon.png}}<br /> __NOTOC__<br /> == Current Courses ==<br /> * GLGY 605 - Benjamin Tutolo<br /> * BMEN 415 - Ethan MacDonald<br /> * MDSC 201 - David Anderson<br /> <br /> == Previous Courses ==<br /> === 2020 Spring ===<br /> * Bioinformatics workshop Q Zhang<br /> * Bioinformatics workshop Q Zhang<br /> <br /> ===2020 Winter===<br /> * DATA 623 R Walker<br /> * GLGY 605 B Tutolo<br /> * DATA 608 P Federl<br /> * ENSF 612 J Kaur<br /> <br /> ===2020 Winter Block Week===<br /> * MDSC 395 D Anderson<br /> <br /> == See https://www.ucalgary.ca/pubs/calendar/current/academic-schedule.html for the UofC 's academic schedule. ==<br /> [[Category:TALC]]</div>

Fridman