Software

Introduction

The HPC cluster uses Lmod to install all of the HPC major software. It also allows us to install multiple versions of the same software.

Lmod

Lmod is an environment module system that provides simple commands for manipulating your environment for tool and library selections. A set of "modules", which are really just text files, are written to modify $PATH, $LD_LIBRARY_PATH, and $MANPATH and possibly create other needed environment variables for the specific tool or library. Lmod allows you to easily load, unload, swap, and purge modules to change your environment dynamically.

How to list the available modules

To see what modules are available to be loaded, you can use the module avail command, which lists all the modules in your current module path. You can use various options to filter or sort the output, such as:

  • -t which shows a terse listing of module names
  • -l which shows a long listing of module names with descriptions
  • -r which shows a reverse listing of module names
  • -d which shows a listing of module names with default versions marked
  • <string> which shows a listing of module names that contain

The output of module avail will show you information such as:

  • The name of the module (e.g. gcc, R, etc.)
  • The version of the module (e.g. 7.5.0, 3.8.5, etc.)
  • The default version of the module (marked with "(D)" if any)
  • The description of the module (if available)
  • The module path where the module is located (e.g. /opt/apps/modulefiles/Core)

For more details on how to use module avail, see this page.

How to load and unload modules

To load a module and modify your environment accordingly, you can use the module load command, followed by the name of the module. You can also specify the version of the module if you want a specific one. For example:

module load r

will load the default version of r (if any), or

module load r/4.2.3

will load the version 4.2.3 of r.

To unload a module and restore your environment to its previous state, you can use the module unload command, followed by the name of the module. You do not need to specify the version of the module if you want to unload it. For example:

module unload r

will unload any version of R that is currently loaded.

You can also load or unload multiple modules at once by separating them with spaces. For example:

module load miniconda3 r/4.2.3

For more details on how to use module load and module unload, see this page.

How to swap and purge modules

To swap one module for another, you can use the module swap command, followed by the name of the module you want to unload and the name of the module you want to load. You can also specify the versions of the modules if you want specific ones. For example:

module swap r miniconda3

will unload any version of gcc that is currently loaded and load the default version of intel (if any), or

module swap r/4.2.2 r/4.2.3

will unload r/4.2.2 and load r/4.2.3.

The module swap command is equivalent to doing a module unload followed by a module load.

To purge all modules from your environment and start from scratch, you can use the module purge command, which does not take any arguments. For example:

module purge

will unload all modules that are currently loaded.

For more details on how to use module swap and module purge, see this page.

How to get help and information about modules

To get help and information about a specific module, you can use the module help command, followed by the name of the module. You can also specify the version of the module if you want a specific one. For example:

module help r

will show you a help message for r (if available), or

module help r/4.2.2

will show you a help message for r/4.2.2 (if available).

The help message may contain information such as:

  • The description of what the module does
  • The usage instructions for how to load or unload the module
  • The environment variables that are set or modified by the module
  • The dependencies or conflicts with other modules
  • The contact information for support or feedback

To get more detailed information about a specific module, such as what commands it executes when it is loaded or unloaded, you can use the module show command, followed by the name of the module. You can also specify the version of the module if you want a specific one. For example:

module show r

will show you what commands are executed when r is loaded or unloaded, or

module show r/4.2.2

will show you what commands are executed when r/4.2.2 is loaded or unloaded.

The output of module show may contain information such as:

  • The prepend-path commands that add directories to your environment variables (e.g.prepend-pathPATH /opt/apps/gcc/7/bin)
  • The append-path commands that append directories to your environment variables (e.g append-path LD_LIBRARY_PATH /opt/apps/gcc/7/lib64)
  • The setenv commands that set new environment variables or overwrite existing ones (e.g. setenv CC /opt/apps/gcc/7/bin/gcc)
  • The unsetenv commands that remove existing environment variables (e.g. unsetenv CXX)
  • The conflict commands that prevent loading conflicting modules (e.g. conflict intel)
  • The prereq commands that require loading prerequisite modules (e.g. prereq mkl)
  • The family commands that group related modules together (e.g. family compiler)
  • The inherit commands that inherit properties from parent modules (e.g. inherit Core)
  • The comment lines that start with "#" (e.g. # This is a comment line)

For more details on how to use module help and module show, see this page.

How to create your own personal modules

If you have a set of modules that you frequently use together, you can create your own personal collection of modules using the module save command, followed by a name for your collection. For example:

$ module list

Currently Loaded Modules:
  1) miniconda3    2) r

$ module save mycollection

Saved current collection of modules to: "mycollection"

will save your current collection of five modules under the name "mycollection". You can then restore your collection at any time using the module restore command, followed by the name of your collection. For example:

$ module purge

$ module list

No modules loaded

$ module restore mycollection

Restoring modules from user's mycollection

$ module list

Currently Loaded Modules:
  1) miniconda3    2) r

will restore your collection after purging all modules.

You can also make one of your collections as your default collection by using "default" as its name when saving it. For example:

$ module save default

Saved current collection of modules to: "default"

will make your current collection as your default collection.

Then, whenever you log in or start a new session, your default collection will be automatically loaded for you.

You can also list all your saved collections using the --list-collections option with any module command (such as load, unload, etc.). For example:

$ module --list-collections

Named collections:
  1) default    2) mycollection    3) anothercollection

For more details on how to use module save and module restore, see this page.

Cryosparc

Installation

This guide will guide on how to download and install Cryosparc

1. Create a directory for CryoSPARC:

mkdir cryosparc
cd cryosparc

2. Set your license ID:

export LICENSE_ID="xxxx-xxxx

3. Download CryoSPARC master and worker packages:

curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID -o cryosparc_master.tar.gz
curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID -o cryosparc_worker.tar.gz

4. Extract the downloaded packages:

tar -xf cryosparc_master.tar.gz cryosparc_master
tar -xf cryosparc_worker.tar.gz cryosparc_worker

5. Switch to a GPU node for installation:

srun -p gpu --gres=gpu:1 --pty bash -l

6. Navigate to the CryoSPARC master directory:

cd ~/cryosparc/cryosparc_master

7. Set your license ID again:

export LICENSE_ID="xxxx-xxxx"

8. Set your user details:

EMAIL="your_email_here"
FIRST_NAME="first_name_here"
LAST_NAME="last_name_here"
PASSWORD="password_for_this_program"

9. Choose a port number:

  • Make sure to select a different port compared to your colleague to avoid conflicts.
  • Increment the port number by 10 for each user. For example, if the default port is 3900, the next user should use 3910, and so on.
PORT=39000

10. Run the installation script:

./install.sh --standalone --yes --license $LICENSE_ID \
--worker_path ~/cryosparc/cryosparc_worker \
--nossd \
--initial_email "$EMAIL" \
--initial_username "$USER" \
--initial_firstname $FIRST_NAME \
--initial_lastname $LAST_NAME \
--initial_password "$PASSWORD" \
--port $PORT

11. Remove the scheduler target node:

./bin/cryosparcm cli "remove_scheduler_target_node('$HOSTNAME')"

12. Stop CryoSPARC:

./bin/cryosparcm stop

13. Exit the session:

exit

Login to https://ondemand.hpc.pomona.edu and run Cryosparc application.
Make sure to close the browser so that the program can close cleanly.
Ending the job before closing the browser can lead to database corruption.