Anaconda is a data science platform which includes Python and R.
Multiple versions of Anaconda are available on Bridges-2. In addition, environments tailored for AI applications which include Anaconda and other popular AI/Machine Learning/Big Data packages, such as TensorFlow, Theano, Keras, pandas, opencv, scikit-learn, and more are set up for you to use.
To see what versions of Anaconda are available, type
module spider anaconda
Note that anaconda2 modules use python2 and anaconda3 modules use python3.
To see what other modules are needed, what commands are available and how to get additional help type
module spider anaconda-version
To use Anaconda, include a command like this in your batch script or interactive session to load the Anaconda module:
module load anaconda-version
Be sure you also load any other modules needed, as listed by the
module help or module spider anaconda command.
Storing your Anaconda environments
Your home directory has limited space, and Anaconda environment files can be very large. We recommend working in your ocean space ($PROJECT) on Bridges-2 as your quota there is much larger.
By default, Anaconda environments are created in your home directory space. If you have already created Anaconda environments there, they should be moved to $PROJECT. Check if you have environments in your home directory by looking for a .conda subdirectory. Because the subdirectory name begins with a “.”, you need the
-a flag to the
ls command to see it:
[username@login005]$ ls -a . .. .conda batch_defaults.out examples file.err [username@login005]$
To move your .conda directory and its contents to $PROJECT, use this command:
mv $HOME/.conda $PROJECT/.conda
This may take a while to complete.
If you didn’t already have a .conda directory, you can create one with
Once you have a $PROJECT/.conda directory, create a symbolic link to direct all new Anaconda environments to be created in $PROJECT with this command:
ln -s $PROJECT/.conda $HOME/.conda
Creating a new environment
If you need additional software, you can create a new environment and install it there. Steps to do this are:
- Get an interactive session with the
- Move to your ocean space ($PROJECT) to allow enough file space.
- Load the anaconda module you want. Type
module spider anacondaon Bridges-2 to see what versions are available.
- Use the
conda createcommand to create your new environment.
- Activate your new environment.
- Install the software you need.
This example session on Bridges-2 creates a new environment and installs fitsio.
[user@bridges2-login013 ~]$ interact A command prompt will appear when your session begins "Ctrl+d" or "exit" will end your session --partition RM-small,RM-shared salloc -J Interact --partition RM-small,RM-shared salloc: Pending job allocation 357591 salloc: job 357591 queued and waiting for resources salloc: job 357591 has been allocated resources salloc: Granted job allocation 357591 salloc: Waiting for resource configuration salloc: Nodes r001 are ready for job Due to MODULEPATH changes, the following have been reloaded: 1) allocations/1.0 [user@r001 ~]$ module load anaconda3 [user@r001 ~]cd $PROJECT [user@r001 ~]$ conda create --name march5-test Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /ocean/projects/groupname/username/.conda/envs/march5-test Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate march5-test # # To deactivate an active environment, use # # $ conda deactivate [user@r001 ~]$ source activate march5-test (march5-test) [user@r001 ~]$ conda install -c conda-forge fitsio Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /ocean/projects/groupname/username/.conda/envs/march5-test added / updated specs: - fitsio The following packages will be downloaded: package | build ---------------------------|----------------- _libgcc_mutex-0.1 | conda_forge 3 KB conda-forge _openmp_mutex-4.5 | 1_gnu 22 KB conda-forge certifi-2020.12.5 | py39hf3d152e_1 143 KB conda-forge fitsio-1.1.4 | py39hf50ba5c_0 713 KB conda-forge . . . ------------------------------------------------------------ Total: 72.1 MB The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu certifi conda-forge/linux-64::certifi-2020.12.5-py39hf3d152e_1 fitsio conda-forge/linux-64::fitsio-1.1.4-py39hf50ba5c_0 . . . Proceed ([y]/n)? y Downloading and Extracting Packages libev-4.33 | 104 KB | ##################################### | 100% c-ares-1.17.1 | 109 KB | ##################################### | 100% libblas-3.9.0 | 11 KB | ##################################### | 100% . . . Preparing transaction: done Verifying transaction: done Executing transaction: done (march5-test) [user@r001 ~]
At this point, a directory named $PROJECT/.conda/envs/new-env will have been created for you. The /bin subdirectory will contain your new binaries.
Use the additional packages you installed in a subsequent session
You can use any additional packages you install in the same session in which you install them. To access them in subsequent sessions, follow these steps. In this example, the user created the environment clone-env-1 and installed h5py in it in a previous session.
interact -gpu module load AI # loads the default AI module source activate clone-env-1 python import h5py
Example sessions are shown below for common uses of anaconda.
Use Tensorflow and python3
Use the following commands:
interact -gpu module load AI/anaconda3-5.1.0_gpu # choose an available AI/anaconda3 module source activate $AI_ENV python import tensorflow
Use Tensorflow and python2
Use the following commands.
interact -gpu module load AI/anaconda2-5.1.0_gpu # choose an available AI/anaconda2 module source activate $AI_ENV python import tensorflow
PSC created environments for AI applications
PSC has built some environments which provide a rich, unified, Anaconda-based environment for AI, Machine Learning, and Big Data applications. Each environment includes several popular AI/ML/BD packages, selected to work together well.
The AI environments are built for the GPU nodes on Bridges-2. Be sure to use one of the GPU partitions. See the Bridges-2 User Guide for information on Bridges-2 partitions and how to choose one to use.
- the python documentation, for a description of the ways to use python on Bridges-2
- the Anaconda modules, for information on creating, editing and storing anaconda environments on Bridges-2
Using the AI environments on Bridges-2
module spider AI/anaconda
to see the available AI environments.
For additional help, type
module help AI/anaconda-version
Note that AI/anaconda2 environments use python2, while AI/anaconda3 environments use python3.
See what the PSC defined AI environment contains
To see the full list of software included in a given environment, first load the module and activate the environment with these commands:
module load AI/anaconda-version source activate $AI_ENV
To see what is included in $AI_ENV, type
Customize the PSC defined AI environment
If you need software that is not in the pre-built environment, you can create a new environment by cloning the PSC defined one and then customizing it. First load the module and activate the PSC defined environment, as above, then clone it with
conda create --name your-new-environment-name --clone $AI_ENV
Then you can activate the new environment and proceed with your customization.
In this example, the user installs the h5py package in a new environment they are creating. Use the following commands.
conda listcommand shows what packages are currently installed. Check to see if what you need is already available. The conda list command also shows the version number of the installed packages.
conda createcommand clones $AI_ENV to create a new environment. This can take a long time, so ask for an hour of time with the
- Here, the new environment is named clone-env-1, and is stored in the user's ocean directory. The --prefix flag names the full path to the where the environment will be stored. You can name the environment anything you like and store it in any directory you like.
interact -gpu -t 01:00:00 module load AI # loads the default AI module source activate $AI_ENV conda list conda create --name clone-env-1 --clone $AI_ENV conda activate clone-env-1 conda install h5py
conda install command will install the newest version of the package. If you want to install a version of the package not available in the public installations use the --revision option to the conda install command.