Pittsburgh Supercomputing Center 

Advancing the state-of-the-art in high-performance computing,
communications and data analytics.

GAUSSIAN

Gaussian Inc. is working to fix problems that have been reported with their D.01 executable on Blacklight.

There are instances when Gaussian09 V. D01 produces the following error message on Blacklight:

Internal consistency failure #1 in GetIJB

Version C.01 does not produce this error. If you encounter this error and your research allows it, a temporary solution is to run the previous version, C01. To do that, load the C.01 version in your script with:

module load gaussian09/c01

Gaussian is an electronic structure program, used by chemists, chemical engineers, biochemists, physicists and others for research in established and emerging areas of chemical interest.

Starting from the basic laws of quantum mechanics, Gaussian predicts the energies, molecular structures, and vibrational frequencies of molecular systems, along with numerous molecular properties derived from these basic computation types. It can be used to study molecules and reactions under a wide range of conditions, including both stable species and compounds which are difficult or impossible to observe experimentally such as short-lived intermediates and transition structures.

Access is restricted. Complete the PSC Gaussian User Agreement to obtain access.

Installed on blacklight and salk.

Versions

Gaussian 09, Revision D.01 is available on blacklight.

Running Gaussian efficiently

There are many variables that affect the performance of your Gaussian job. We have found that there are some things which always help, and others that depend on your application.

Be sure to read this document carefully for tips that can help you. If you have questions or need additional help, please contact PSC User Services.

Please see:

Following the performance recommendations, find instructions for running Gaussian on any PSC platform.

You can also check the section on efficiency considerations in the Gaussian 03 Online Manual for ways to improve your particular job's performance.

Improving Gaussian performance on blacklight

You should always do these things when using Gaussian:

      1. Use omplace in your batch script

        Gaussian is an Open MP code. Because of this, omplace should always be used to run your Gaussian executable. As an example, to run Gaussian 09:

        omplace -nt $OMP_NUM_THREADS g09 < inputfile > outputfile
        
      2. Use the right number of threads (%nprocs)

        Some Gaussian jobs will perform much better with %nprocs=16 than with %nprocs=1, and some will perform worse. Most Gaussian jobs, however, cannot use more than %nprocs=16 effectively. Do some tests to see what is best for your job.

        If you are using g03, you cannot use more than 8 processors on blacklight. This means that in your Gaussian input file, you should set

        %nproc=8

        In your job script, set

        #PBS -l ncpus=16   ! this is the minimum you can use

        but set OMP_NUM_THREADS to 8 (or less) (See #3 below)

      3. Set $OMP_NUM_THREADS in your batch script

        In your batch script, set the environment variable $OMP_NUM_THREADS to the same value that %nprocs is set to in the Gaussian input file:

        export OMP_NUM_THREADS=16   ! bash shell
        setenv OMP_NUM_THREADS 16   ! C shell
        
      4. Request the right amount of memory

        Only a small amount of memory is allocated by default. If you do not request enough memory, your job may be killed when it reaches the memory limit with a memory pressure error.

        In your Gaussian input file, set %mem to at least 1 GB, but beyond that, use the smallest amount of memory required to run your job. Caveat: you may benefit from using more memory if a larger amount will enable Gaussian to store large data structures in memory rather than writing them to disk.

        If you are still receiving the memory pressure error another approach you can try is to increase the number of cores you request in your job to 32 using the commands

                 #PBS -l ncpus=32

                  setenv OMP_NUM_THREADS 32

         You also need to change the following Gaussian input parameter

                   %Nproc=32

        If this does not remove the memory pressure error you can try asking for only 16 cores in your Gaussian input

                    %Nproc=16

           while still asking for 32 cores from PBS.

      5. (Almost always) Set GAUSS_SCRDIR to $SCRATCH_RAMDISK

        Gaussian creates checkpoint files while it is running, which can be quite large and can exceed the $HOME directory quota. To avoid this, it is best to run Gaussian from one of your scratch directories so that the checkpoint files are created there.

        In your script, set the environment variable GAUSS_SCRDIR to point to $SCRATCH_RAMDISK before calling Gaussian:

        export GAUSS_SCRDIR=$SCRATCH_RAMDISK   # for the bash shell
        setenv GAUSS_SCRDIR $SCRATCH_RAMDISK   # for the C shell
        

        This will cause Gaussian to use the fast memory filesystem $SCRATCH_RAMDISK for its temporary files (which are not needed after your job completes).

        The only reason to ignore this suggestion is if it causes your job to run out of memory. If this happens, contact PSC User Services for recommendations.

        Do not use $SCRATCH_RAMDISK for files that you will need later. Instead, use $SCRATCH. More information about $SCRATCH_RAMDISK is available in the blacklight document.

      6. (Almost) Never request more than 16 cores in your batch script

        If your job requires more than 124 GB of memory, you will need to ignore this suggestion. On blacklight, you have access to 124 GB of memory on every 16-core blade. (Some memory is used by the operating system.)

      7. Consider packing small jobs together

        Running many small jobs places a great burden on the scheduler and is probably inconvenient for you. Consider packing your small jobs into one. For information about packing jobs, see the  blacklight document.

Improving performance on salk

You should always do these things when using Gaussian:

      1. Use omplace in your batch script

        Gaussian is an Open MP code. Because of this, omplace should always be used to run your Gaussian executable. As an example, to run Gaussian 09:

        omplace -nt $OMP_NUM_THREADS g09 < inputfile > outputfile
        
      2. Set $OMP_NUM_THREADS in your batch script

        In your batch script, set the environment variable $OMP_THREADS to the same value that %nprocs is set to in the Gaussian input file:

        export OMP_NUM_THREADS=16   ! bash shell
        setenv OMP_NUM_THREADS 16   ! C shell
        
      3. Use scratch space to write files.

        Gaussian creates checkpoint files while it is running, which can be quite large and can exceed the $HOME directory quota. To avoid this, it is best to run Gaussian from your $SCRATCH directory so that the checkpoint files are created there.

        $SCRATCH is also only temporary storage, so files that need to be kept long-term should be moved to the archiver. However, $SCRATCH is available from the salk login node, so you may be able to retrieve files from it after your job ends. Be aware that copying checkpoint files to $HOME automatically in the job script may exceed the quota on $HOME.

        To use $SCRATCH, set the environment variable GAUSS_SCRDIR to point to it:

        export GAUSS_SCRDIR=$SCRATCH   # for the bash shell
        setenv GAUSS_SCRDIR $SCRATCH   # for the C shell
        
      4. Request the right amount of memory

        Only a small amount of memory is allocated by default. If you do not request enough memory, your job may be killed when it reaches the memory limit.

        In your Gaussian input file, set %mem to at least 2 GB, but beyond that, use the smallest amount of memory required to run your job. Caveat: you may benefit from using more memory if a larger amount will enable Gaussian to store large data structures in memory rather than writing them to disk.

        The maximum memory available on salk is 2*nprocs -1 GB. As an example, to request the maximum for a 4 processor job, use these lines in the Gaussian input file:

        %nproc=4
        %mem=7gb

File systems available

blacklight salk
$SCRATCH_RAMDISK  
$HOME and $SCRATCH $HOME and $SCRATCH   
The archival system The archival system

Running Gaussian

      1. Prepare your Gaussian input file. Among the other parameters, be sure to:
        1. Set %nprocs to the number of threads that you want to use. Insert the following as the first line in your Gaussian input file:
          %nproc = x
          
          where x is the number of processors to use. Choose the value that gives your program the best performance.
        2. Consider your memory usage.
          You may wish to override the small amount of memory allocated by default by requesting most of the memory available to your job. Do this by using the %mem parameter in the input file. See the discussion on  in the performance section for details.
      2. Create a job file which:
        1. Sets up the use of the module command in a batch job.
        2. Loads the correct Gaussian module for the version you wish to run.

          See documentation on the module software to determine

          To load the default version, type

          module load gaussian

          Note: Only one of the gaussian modules should be loaded at any time. For more information about the module package, see the PSC module documentation.

        3. Defines OMP_NUM_THREADS to be the same as ncpus requested
          export OMP_NUM_THREADS=value-of-ncpus   ! for the bash shell
          setenv OMP_NUM_THREADS value-of-ncpus   ! for the c shell
          
        4. Invokes Gaussian with:
          omplace -nt $OMP_NUM_THREADS g03 < inputfile >& outputfile
          or
          omplace -nt $OMP_NUM_THREADS g09 < inputfile >& outputfile
      3. Submit the job file with the qsub command.

Checkpoint files

You can use the chkchk utility to check what the content of a checkpoint file is. This can also tell you if a checkpoint file is corrupted.

For example, this checkpoint file is corrupted:

joeuser@tg-login1:~/gaussian chkchk
/usr/users/8/joeuser/gaussian/water.chk
Checkpoint file /usr/users/8/joeuser/water.chk:
No title or route information found.
AO basis is present.
SCF restart data present.

As you can see, there is no title or route info found. Compare this to the ouput from an uncorrupted checkpoint file. If all the information is there, the output should be similar to:

joeuser@tg-login1:~/gaussian chkchk
/usr/users/8/joeuser/gaussian/methane.chk
Checkpoint file /usr/users/8/joeuser/gaussian/methane.chk:
Title: DFT Calculation of (10,0) NT at B3LYP/6-31G(d) from B3LYP/3-21
G starting point
Route: #P opt B3LYP/6-31g(d) geom=check guess=read scf=(maxcyc=200)
Atomic coordinates present.
AO basis is present.
SCF restart data present.
MO coefficients present.
This file appears to be from the middle of a restartable job.
Internal force constants may be present.

See the chkchk page of the Gaussian 03 Online Manual for details on chkchk.

Utilities

Gaussian provides other utilities. For more information, see the Utility Programs page from the Gaussian 03 Online Manual.