Tensorflow on ARC

Background

• Website: https://www.tensorflow.org/

Tensorflow is a tool for evaluating dataflow graphs that represent both the computations and model state in a machine learning algorithm. It enables distributed evaluation and explicit communication across a large number of computing devices (e.g. numerous CPUs or GPUs). The core tools of tensorflow consist of shared C libraries for constructing dataflow graphs and executing computations (typically linear algebra on tensors). This constitutes a fairly low level language for setting up data preprocessing, model training, and inference infrastructure for online machine learning applications. For an overview of tensorflow's design principles and its relationship with other popular machine learning technologies (like Caffe, Torch, and MXNet), Abadi's 2016 article is a good starting point.

However, in its most common usage, Tensorflow's implementation of common deep neural network operations are exposed through the Python API. Direct access to all of this functionality from a C++ API is under development but most of the core deep neural network functionality that users are familiar with are still only available through specific client languages with Python being the most developed. The high-level Tensorflow API is consistent with the modelling standards established by Keras (that support multiple alternate backend evaluation engines). The higher level abstractions do not necessarily offer the same flexibility of parallel computation as the core Tensorflow tool. However, they generally are more accessible to users that are primarily familiar with the modelling techniques that are common to artificial neural networks. Consequently, Keras has been integrated directly into the core utility libraries of the Tensorflow Python API.

Tensorflow documentation and tutorials

• https://www.tensorflow.org/api_docs/python

• Compute Canada article is not directly applicable on ARC but contains a lot of good information:

https://docs.computecanada.ca/wiki/TensorFlow

Installing Tensorflow

You will need a working local Conda install in your home directory first. If you do not have it yet, plaese follow these instructions to have it isntalled.

Once you have your own Conda, activate it with

$ ~/software/init-conda

Here you have two choices how to install Tensorflow (1) using Conda or (2) how it is explained on the Tensorflow site, using pip.

Installing with Conda

We will install Tensorflow-GPU into its own conda environment.

It is very important to create the environment with python and tensorflow-gpu in the same command. This way conda can select the best tensorflow-gpu and python combination.

$ conda create -n tensorflow python tensorflow-gpu

Once it is done, activate your tensorflow ennvironment:

$ conda activate tensorflow

Installing with PIP

If you want to follow the Tensorflow installation instructions (https://www.tensorflow.org/install/pip) and use pip, you still need a separate Conda environment, and you have to install required version of Python and Pip.

(base) $ conda create -n tensorflow 
....

(base) $ conda activate tensorflow
(tensorflow) $ conda install python=3.11.7 pip
....

If the installation has been successful, you now have the required versions of Python and Pip installed. \ You can now follow the instructions from the manual.

(tensorflow) $ pip install tensorflow[and-cuda]
....

After this installation is finished the Tensorflow is ready for testing.

Testing

You can test with the tensorflow-test.py script shown below. Copy and paste the text into a file and run if from the command line:

(tensorflow) $ python tensorflow-test.py

If you try this on the login node, it should tell you that GPUs are not available. It is normal, as the login node does not have any. You will need a GPU node to test the GPUs.

Once you know that your tensorflow environment is working properly, you can add more packages to the environment using conda.

To deactivate the environment use the

(tensorflow) $ conda deactivate 

command. And leave Conda

(base) $ conda deactivate

Test script

tensorflow-test.py:

#! /usr/bin/env python3
# ------------------------------------------
import os
import tensorflow as tf

os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices'
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
sep = "=" * 80
# ------------------------------------------
print(sep)
print("Checking for available GPUs...\n")
gg = tf.config.list_physical_devices("GPU")
print("GPUs found: %d" % len(gg))

for g in gg:
    print("\t%s: %s" % (g.device_type, g.name))

# ------------------------------------------
print(sep)
print("Testing Tensorflow...\n")
try:
    x = tf.reduce_sum(tf.random.normal([1000, 1000]))
except:
    print("FAILED!")
    sys.exit()

print("Tensorflow works properly. SUCCESS!")
# ------------------------------------------
print()

The --tf_xla_enable_xla_devices option allows GPUs to be also seen as XLA capable devices as well.

See here: https://www.tensorflow.org/xla

It is not critical and can be omitted.

Using Tensorflow on ARC

Requesting GPU Resources for Tensorflow Jobs

For interactive use see this How-To: How to request an interactive GPU on ARC.

An example of the job script tensorflow_job.slurm:

#! /bin/bash
# ====================================
#SBATCH --job-name=tf-test
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16GB
#SBATCH --time=0-04:00:00
#SBATCH --gres=gpu:1
#SBATCH --partition=gpu-v100
# ====================================

source ~/software/init-conda
conda activate tensorflow

python tensorflow-test.py

Links

ARC Software pages