User Tools

Site Tools



icme:lammpsoncc

Running LAMMPS on CC servers

This tutorial assumes prior knowledge about LAMMPS LAMMPS and deals with running LAMMPS scripts on the compute servers at the computer center in IIT Madras. But, the procedure for building and using the serial and parallel version should work on any system with small modifications here and there. Further, it assumes basic experience using a command line interface, especially bash on a Linux terminal.

Installing LAMMPS

As a first step download the LAMMPS tarbal from the downloads page on the website and transfer it to the home directory of your account on the server, using winSCP from windows or sftp from a Linux terminal.

Once this is done login to your account. You should find the file you just transferred if you execute the 'ls' command. Next, extract the contents of the tarbal using the following command.

 tar xvf lammps-stable.tar 

The stable version at the time of writing is from 30th July and hence you should a new folder named 'lammps-30Jul16' or something like 'lammps-date' in general. Move into the 'src' folder within this folder using the following:

 cd lammps-30Jul16/src

The 'src' folder has the source code for LAMMPS written in C++ (mostly). The following command gives a bare minimum serialized version of the code:

 make serial 

'make' also works in parallel and you can use multiple cores on the CPU to do this, using:

 make -j 16 serial 

To get the parallel version of LAMMPS with MPI, execute this command on the same folder:

 make -j 16 mpi 

This requires openMPI or MPICH already installed and configured for your compiler. You can execute the above on GNR/VIRGO out of the box.

Next, create a 'soft link' of the executables to a 'bin' folder in your home directory. Add the 'bin' folder to your bash search path ('$PATH'). Make this permanent by adding a line to the '.bashrc' file in the home directory which sets the environment for bash. Execute the following commands one by one on the terminal to do this.

 mkdir $HOME/bin/ 
 ln -s `readlink -f lmp_serial` $HOME/bin/
 ln -s `readlink -f lmp_mpi` $HOME/bin/
 echo "export PATH=\$PATH:\$HOME/bin/" >> $HOME/.bashrc
 source $HOME/.bashrc

NOTE

  1. The 'readlink -f …' part of the command should be within backticks (The symbol below tilde on the key below 'Esc' key on standard keyboards). On shell, a part of a command which is within backticks is executed by the shell and the output from this part is substituted in the whole command. If the backtick does not get copied from the pdf file, type the command out
  2. Run all this from the 'src' folder

Now, executing 'lmp_serial' from any folder should print this on the terminal :

 LAMMPS (30 Jul 2016) 

This shows that the 'build' process was successful. Kill ('ctrl-c') this process for now.

Executing LAMMPS scripts on head node

From the 'src' directory we move to a directory with an example file for simulating crack propagation. You can use any LAMMPS input file in any directory (within your home folder) for this and replace/modify the commands appropriately.

 cd ../examples/crack/ 

Listing the files should show the presence of a file named 'in.crack'. Run this simulation using the serial version of LAMMPS using:

 lmp_serial < in.crack

This should run and output some details of the simulation, with the last line being:

Total wall time: 0:00:06

The time may vary according to the load on the system. This shows that the simulation completed successfully.

Now, in order to use the parallel version of LAMMPS we need to setup some environment variables for getting the libraries right. For this open the '.bashrc' file from the home directory in 'vi'.

 vi $HOME/.bashrc

If using GNR, paste the following line at the end of the file and save it.

 source /Apps/intel-2016-up3/bin/compilervars.sh intel64

If using VIRGO, paste the following line at the end of the file and save it.

 source /IITM_GPFS_FS1/sware/intel2016/bin/compilervars.sh intel64

'Source' the environment file so that the changes are reflected in the current session too.

 source $HOME/.bashrc

Now executing the next statement will give a similar output as using 'lmp_serial'.

 mpirun -np 16 lmp_mpi < in.crack

But, there should be a significant reduction in the total wall time. The previous statement indicates that 16 threads should be used to run the simulation.

Job submission to a compute node on GNR

While you can use the compute node to get your input script error free, it is never advisable to actually run your jobs on it. For this you should 'queue' your jobs on the scheduler. For this purpose, GNR uses the PBS job scheduling software. So, you should write a 'job submission script' and provide it to the software.

Serial execution:

Create and open a file called 'job_serial.cmd' using 'vim'

 vi job_serial.cmd

Paste the following in that file and save it.

 #! /bin/bash
 #PBS -o logfile.log
 #PBS -e errorfile.err
 #PBS -l cput=10:00:00
 #PBS -l select=1:ncpus=1
 tpdir=`echo $PBS_JOBID | cut -f 1 -d .`
 tempdir=$HOME/work/job$tpdir
 mkdir -p $tempdir
 cd $tempdir
 cp -R $PBS_O_WORKDIR/* .
 lmp_serial < in.crack
 mv ../job$tpdir $PBS_O_WORKDIR/.

From the shell execute this:

 qsub job_serial.cmd

This should give you an output of the form 'job-id.gnr'. Running 'qstat' should show that the job is running against that job-id. Once the job finishes the system moves the directory where the job executed, named 'jobjob-id' to the current folder along with two other files. The errorfile.err displays any error with execution and the logfile.log shows the 'stdout' when the LAMMPS script executed. The folder should have all your outputs.

Parallel execution:

For parallel execution the procedure is very similar except for small modifications to the job submission scripts.

Create a file named 'job_parallel.cmd' and paste the following in the file:

 #! /bin/bash
 #PBS -o logfile.log
 #PBS -e errorfile.err
 #PBS -l cput=10:00:00
 #PBS -l select=1:ncpus=16
 tpdir=`echo $PBS_JOBID | cut -f 1 -d .`
 tempdir=$HOME/work/job$tpdir
 mkdir -p $tempdir
 cd $tempdir
 cp -R $PBS_O_WORKDIR/* .
 mpirun -np 16 lmp_mpi < nve.in
 mv ../job$tpdir $PBS_O_WORKDIR/.

Execute with:

 qsub job_parallel.cmd

You should see nearly same output, but in a shorter time.

Job submission to a compute node on VIRGO

While GNR uses PBS for job scheduling, VIRGO uses Load Leveler for this and we need to write LL batch job scripts.

Serial execution:

Create and open a file called 'job_serial.cmd' using vim

 vi job_serial.cmd

Paste the following in that file and save it.

 #!/bin/bash
 #@ output= test.out
 #@ error= test.err
 #@ job_type= serial
 #@ class= Medium
 #@ environment = COPY_ALL 
 #@ queue
 Jobid=`echo $LOADL_STEP_ID | cut -f 6 -d .`
 tmpdir=$HOME/scratch/job$Jobid
 mkdir -p $tmpdir; cd $tmpdir
 cp -R $LOADL_STEP_INITDIR/* $tmpdir
 lmp_serial < in.crack
 mv ../job$Jobid $LOADL_STEP_INITDIR 

From the shell execute this:

 llsubmit job_serial.cmd

This should give you an output of the form 'c1hn1'.job-id Running 'llq job-id' should show that the job is running against that number. Once the job finishes the system moves the directory where the job executed, named 'jobjob-id' to the current folder.

Parallel execution:

For parallel execution the procedure is very similar except for small modifications to the job submission scripts.

Create a file named 'job_parallel.cmd' and paste the following in the file:

 #!/bin/bash
 #@ output= test.out
 #@ error= test.err
 #@ job_type= MPICH
 #@ class= Medium
 #@ node = 1
 #@ tasks_per_node = 16
 #@ environment = COPY_ALL 
 #@ queue
 Jobid=`echo $LOADL_STEP_ID | cut -f 6 -d .`
 tmpdir=$HOME/scratch/job$Jobid
 mkdir -p $tmpdir; cd $tmpdir
 cp -R $LOADL_STEP_INITDIR/* $tmpdir
 mpirun -np 16 lmp_mpi < in.crack
 mv ../job$Jobid $LOADL_STEP_INITDIR

Execute with:

 llsubmit job_parallel.cmd

Logging into the CC website and looking at the Virgo info under the hpce tab in the right side bar will give you more options you can use with the Load Leveller.

Useful tools

Some tools which will make life easier for you (and others):

  1. Google: Straight forward. Any error you see, google it up. Someone would have encountered the same error and posted on stack overflow or on the forums for the software. A person with more experience or the software developers would have solved it. Google indexes all of this and so you have you solution then and there. This works most of the times.
  2. GNU Screen
  3. Bash scripting

There are numerous tutorials for 2 and 3 online and should be easily accessible with minimal Linux experience. Bash scripting especially will help your productivity. You can have one script for creating input scripts, executing it and post processing data. Go figure!

PDF version

This is a pdf version of this same tutorial. Please use it if that'd be preferable.

Contact

This tutorial was made by Srihari Sundar. Contact him at sriharisundar95@gmail.com for any queries, suggestions and corrections.

Key Words

Mulitscale Modelling of Materials, MM5015, LAMMPS, Anand Kanjarla, Srihari Sundar

icme/lammpsoncc.txt · Last modified: 2016/09/15 06:46 by Anand Kanjarla