Parallel execution of Abaqus is implemented using two different schemes: threads and message passing. Threads share the same memory pool. Thread-based parallelization is readily available on all shared memory platforms.
Parallelization with message passing uses multiple analysis processes that communicate with each other via the Message Passing Interface (MPI).

The following Abaqus/Standard features can be executed in parallel: analysis input preprocessing, the direct sparse solver, the iterative solver, and element operations. Analysis input preprocessing uses only MPI-based parallelization and will not be executed in parallel if only data checking is performed. For Abaqus/Explicit all of the computations other than those involving the analysis input preprocessor and the packager can be executed in parallel. Each of the features that are available for parallel execution has certain limitations (see documentation).

Abaqus/Standard and Abaqus/Explicit can be executed in parallel on shared memory computers by using threads or the MPI. When the MPI is available, Abaqus runs all available parallel features with MPI-based parallelization and activates thread-based parallel implementations for cases where an equivalent MPI-based implementation does not exist (e.g., direct sparse solver).

User subroutines
User subroutines can be used when running jobs in parallel. In a distributed run, the entire model is decomposed into separate domains (partitions). Each domain is serviced by a separate MPI process. Abaqus provides well-defined synchronization points at which it is possible to exchange information across all MPI ranks, using the MPI communications facilities. All native MPI calls are supported, in both Fortran and C++. In addition, for cases of hybrid execution, user subroutines and any subroutines called by them must be thread safe. This precludes the use of common blocks, data statements, and save statements.

Parallel execution in Abaqus/Standard:

• reduces run time for large analyses;
• is available for shared memory computers and computer clusters for the element operations, direct sparse solver, and iterative linear equation solver;
• can use compute-capable GPGPU hardware on shared memory computers for the direct sparse solver.

The direct sparse solver (Direct linear equation solver) supports both shared memory computers and computer clusters for parallelization. On shared memory computers or a single node of a computer cluster, thread-based parallelization is used for the direct sparse solver, and high-end graphics cards that support general processing (GPGPUs) can be used to accelerate the solution. On multiple compute nodes of a computer cluster, a hybrid MPI and thread-based parallelization is used.

The direct sparse solver cannot be used on multiple compute nodes of a computer cluster if:

• the analysis also includes an eigenvalue extraction procedure, or
• the analysis requires features for which MPI-based parallel execution of element operations is not supported.

In addition, the direct sparse solver cannot be used on multiple nodes of a computer cluster for analyses that include any of the following:

• multiple load cases with changing boundary conditions (Multiple load case analysis), and
• the quasi-Newton nonlinear solution technique (Convergence criteria for nonlinear problems).

MPI-based parallelization is used between the machines in the host list. Thread-based parallelization is used within a host machine if more than one processor is available on that machine in the host list and if the model does not contain cavity radiation using parallel decomposition (see Decomposing large cavities in parallel). For example, if the environment file has the following:

cpus=8
mp_host_list=[['maple',4],['pine',4]]

Abaqus/Standard will use four processors on each host through thread-based parallelization. A total of two MPI processes (equal to the number of hosts) will be run across the host machines so that all eight processors are used by the parallel direct sparse solver.

Models containing parallel cavity decomposition use only MPI-based parallelization. Therefore, MPI is used on both shared memory parallel computers and distributed memory compute clusters. The number of processes is equal to the number of CPUs requested during job submission. Element operations are executed in parallel using MPI-based parallelization when parallel cavity decomposition is enabled.

Parallel execution in Abaqus/Explicit:

• reduces run time for analyses that require a large number of increments;
• reduces run time for analyses that contain a large number of nodes and elements;
• produces analysis results that are independent of the number of processors used for the analysis;
• is available for shared memory computers using a thread-based loop level or thread-based domain decomposition implementation; and
• is available for both shared memory computers and computer clusters using an MPI-based domain decomposition parallel implementation.

The command to set the Abaqus environment is:
module load intel impi abaqus/2017 for the Intel Fortran + Intel MPI
module load intel mvapich2 abaqus/2017 for the Intel Fortran + MVAPICH2 MPI

It's important to write also the /2017 because this select the version number (there is also Abaqus 6.14). If there are other compiler and/or other MPI loaded, you will receive an error message asking to unload the other modules before activating the full stack of Intel compiler and MPI. These messages will suggest also the command to give to unload the modules. The best practice is to cleanup the environment by writing the command:

module purge that will unoad all modules

Then you can safely give the command to load the compiler, the MPI and abaqus 2017. The order matters: before the compiler, then MPI and finally abaqus.

The link to the documentation is: http://masternode.chem.polimi.it/v2017

Abaqus command line

#!/bin/bash
#
# Set Job execution shell
#PBS -S /bin/bash
 
# Set Job name: <jobname>=jobabaqus 
#PBS -N jobabaqus
 
# Set the execution queue: <queue name> is one of 
# gandalf, merlino, default, morgana, covenant
#PBS -q <queue name>
 
# Set mail addresses that will receive mail from PBS about job
# Can be a list of addresses separated by commas (,)
#PBS -M <polimi.it or mail.polimi.it email address only>
 
# Set events for mail from PBS about job
#PBS -m abe
 
# Job re-run (yes or no)
#PBS -r n
 
# Set standard output file 
#PBS -o jobabaqus.out
 
# Set standard error file 
#PBS -e jobabaqus.err
 
# Set request for N nodes,C (cores) per node
#PBS -l select=N:ncpus=C
 
# Pass environment to job
#PBS -V
 
# Change to submission directory
cd $PBS_O_WORKDIR
 
# Command to launch application and it's parameters
# Before you unload all modules and load the Intel Fotran + MPI + Abaqus 2017 environment
module purge
module load intel impi abaqus/2017
 
echo "---------------------------------------------------------------------"
echo  "---Starting job at: `date`"
echo
echo "------Current working directory is `pwd`"
np=$(wc -l < $PBS_NODEFILE)
echo "------Running on ${np} processes (cores) on the following nodes:"
cat $PBS_NODEFILE
echo "----Parallel abaqus run"
 
abaqus job=jobabaqus input=<my_input_file_name>.inp cpus=<N*C> interactive mpi_mode=mpi user=<name-of-user-subroutine>
 
echo "-----job finished at `date`"
echo "---------------------------------------------------------------------"

Please report new interesting or broken links to clusterhpc-dcmc@polimi.it

Quick guides and papers: