Anaconda is a data science platform which includes Python and R.



Multiple versions of Anaconda are available on Bridges and on Bridges-2.  In addition, environments tailored for AI applications which include Anaconda and other popular AI/Machine Learning/Big Data packages, such as TensorFlow, Theano, Keras, pandas, opencv, scikit-learn, and more are set up for you to use.

General instructions

To see what versions of Anaconda are available, type

module avail anaconda     # on Bridges
module spider anaconda    # on Briges-2

Note that anaconda2 modules use python2 and anaconda3 modules use python3.

To see what other modules are needed, what commands are available and how to get additional help type

module help anaconda-version    # on Bridges
module spider anaconda-version  # on Bridges-2

To use Anaconda, include a command like this in your batch script or interactive session to load the Anaconda module:

module load anaconda-version

Be sure you also load any other modules needed, as listed by the module help or module spider anaconda command.


Storing your Anaconda environments

Your home directory has limited space, and Anaconda environment files can be very large.  We recommend working in your pylon5 space ($SCRATCH) on Bridges or your ocean space ($PROJECT) on Bridges-2 as your quota there is much larger.

By default, Anaconda environments are created in your home directory space. If you have already created some there, they should be moved to $SCRATCH or $PROJECT.   Check if you have environments in your home directory by looking for a .conda subdirectory.  Because the subdirectory name begins with a “.”, you need the -a flag to the ls command to see it:

[username@login005]$ ls -a
. .. .conda batch_defaults.out examples file.err 

To move your .conda directory and its contents to $SCRATCH or $PROJECT, use the appropriate command:

mv $HOME/.conda $SCRATCH/.conda
mv $HOME/.conda $PROJECT/.conda

This may take a while to complete.  Once it has, create a symbolic link to direct all new Anaconda environments to be created with the appropriate command:

ln -s $SCRATCH/.conda $HOME/.conda
ln -s $PROJECT/.conda $HOME/.conda

Adding to an environment

If you need additional software, you can create a new environment and install it there.  Steps to do this are:

  1. Get an interactive session with the interact command. On Bridges, you must use the --egress option  so that you will be able to access the anaconda website.
  2. Move to your pylon5 space ($SCRATCH)  or your ocean space ($PROJECT) to allow enough file space.
  3. Load the anaconda module you want. Type module avail anaconda on Bridges or module spider anaconda on Bridges-2 to see what versions are available.
  4. Use the conda create command to create your new environment.
  5. Activate your new environment.
  6. Install the software you need.

This example session on Bridges creates a new environment and installs fitsio.

[username@login006 ~]$ interact --egress
  A command prompt will appear when your session begins "Ctrl+d" or "exit" will end your session
 srun: job 5897598 queued and waiting for resources
 srun: job 5897598 has been allocated resources
 [username@r003 ~]$ module load anaconda3
 [username@r003 ~]$ conda create --name new-env
 Solving environment: done   ==> WARNING: A newer version of conda exists. <==   current version: 4.5.4
   latest version: 4.6.14  Please update conda by running      $ conda update -n base conda    
## Package Plan ##    environment location: /home/username/.conda/envs/new-env
   Proceed ([y]/n)? y
  Preparing transaction: done
 Verifying transaction: done
 Executing transaction: done
 # # To activate this environment, use: 
# > source activate new-env
# To deactivate an active environment, use:
 # > source deactivate
[username@r003 ~]$ source activate new-env (new-env)
 [username@r003 ~]$ conda install -c conda-forge fitsio
 Solving environment: done   ==> WARNING: A newer version of conda exists. <==   current version: 4.5.4   latest version: 4.6.14  Please update conda by running      $ conda update -n base conda    ## Package Plan ##    environment location: /home/username/.conda/envs/new-env    added / updated specs:     - fitsio   The following packages will be downloaded:
(a list of packages to be downloaded and installed will be displayed)
Proceed ([y]/n)? y
 Preparing transaction: done
 Verifying transaction: done
 Executing transaction: done
 (new-env) [username@r003 ~]$

At this point, a directory named /pylon5/groupname/username/.conda/envs/new-env will have been created for you.  The /bin subdirectory  will contain your new binaries.



PSC created environments for AI applications

We have built some environments that include software for AI, Big Data and machine learning applications.

The AI environments are built for the GPU nodes on Bridges and Bridges-2.  Be sure to use one of the GPU partitions.  See the Running jobs section of the Bridges User Guide  or the Bridges-2 User  Guide for information on Bridges and Bridges-2 partitions and how to choose one to use.


module avail AI/anaconda    # on Bridges
module spider AI/anaconda # on Bridges-2

to see the AI environments which you can use.

For additional help, type

module help AI/anaconda-version

Note that AI/anaconda2 environments use python2, while AI/anaconda3 environments use python3.

See what the PSC defined AI environment contains

To see the full list of software included in a given environment, first load the module and activate the environment with these commands:

module load AI/anaconda-version
source activate $AI_ENV

To see what is included in $AI_ENV, type

conda list

Customize the PSC defined AI environment

If you need software that is not in the pre-built environment, you can create a new environment by cloning the PSC defined one and then customizing it. First load the module and activate the PSC defined environment, as above, then  clone it with

conda create --name your-new-environment-name --clone $AI_ENV

Then you can activate the new environment and proceed with your customization. See the example below for more detail.



Example sessions are shown below for common uses of anaconda.

Use Tensorflow and python3

Use the following commands:

interact -gpu
module load AI/anaconda3-5.1.0_gpu   # choose an available AI/anaconda3 module
source activate $AI_ENV
python import tensorflow


Use Tensorflow and python2

Use the following commands.

interact -gpu
module load AI/anaconda2-5.1.0_gpu   # choose an available AI/anaconda2 module
source activate $AI_ENV
python import tensorflow

Customize the PSC defined environment

In this example, the user installs the h5py package in a new environment they are creating. Use the following commands.  Note:

The conda list command shows what packages are currently installed. Check to see if what you need is already available.  The conda list command also shows the version number of the installed packages.

The conda create command clones $AI_ENV to create a new environment.  This can take a long time, so ask for an hour of time with the interact command.

Here, the new environment is named clone-env-1, and is stored in the user’s pylon5 or ocean directory.  The –prefix flag names the full path to the where the environment will be stored. You can name the environment anything you like and store it in any directory you like.

interact -gpu --egress -t 01:00:00
 module load AI/anaconda3-5.1.0_gpu
 source activate $AI_ENV
 conda list 
 conda create --name clone-env-1 --clone $AI_ENV
 conda activate clone-env-1
 conda install h5py

Conda install will install the newest version of the package. If you want to install a version of the package not available in the public installations use the –revision option to the conda install command.


Use the additional packages you installed in a subsequent session

You can use any additional packages you install in the same session in which you install them. To access them in subsequent sessions, follow these steps.  In this example, the user created the environment clone-env-1 and installed h5py in it in a previous session.

interact -gpu
 module load AI/anaconda3-5.1.0_gpu
 source activate clone-env-1
 python import h5py