Mistral specific

Start Jupyter session on Mistral

The easiest way to start a Jupyter session on mistral is like follows:

  1. Connect by ssh to mistralpp using port forwording use a <port> which is 8000<port<8887:
ssh -L 8888:localhost:<port> <username>@mistralpp.dkrz.de
  1. Activate an appropriate python environment and start Jupyter using the above <port>:
source pyicon/tools/conda_act_mistral_pyicon_env.sh
jupyter notebook --no-browser --port=<port>
  1. Finally open you web browser on you local computer and enter the address:
localhost:8888

To use Jupyter on the DKRZ cluster, you can find valuable information here.

DKRZ also provides a script to easily setup a Jupyter session.

Download this script and use it with:

./start-jupyter -u username -i /path/to/pyicon/tools/conda_act_mistral_pyicon_env.sh

JupyterLab session can be started with -c lab option (note that pyicon_view does not work with JupyterLab):

./start-jupyter -u username -c lab -i /path/to/pyicon/tools/conda_act_mistral_pyicon_env.sh

Grid files on Mistral

At the moment, a collection of grid files and ckdtrees for different ICON grids can be found here:

/mnt/lustre01/work/mh0033/m300602/icon/grids

In general, ocean triangular grid files are archived here:

/pool/data/ICON/oes/input/

and atmospheric grid files are archived here:

/pool/data/ICON/grids/public/mpim/

An even larger collection of ocean grids can be found here:

/pool/data/ICON/oes/grids/

E.g. ICON SMT grids can be found in:

/pool/data/ICON/oes/grids/OceanOnly

Parallel computing with python on Mistral

By using the mpi4py module, it is possible to run tasks of a script in parallel. To make use of simple parallelization e.g. for creating animations, a python script needs to be modified in the following way (!todo! improve example):

# header of python script
import sys
import matplotlib
if len(sys.argv)>1 and sys.argv[1]=='--no_backend':
  print('apply: matplotlib.use(\'Agg\')')
  matplotlib.use('Agg')

...

# === mpi4py ===
try:
  from mpi4py import MPI
  comm = MPI.COMM_WORLD
  rank = comm.Get_rank()
  npro = comm.Get_size()
except:
  print('::: Warning: Proceeding without mpi4py! :::')
  rank = 0
  npro = 1
print('proc %d/%d: Hello world!' % (rank, npro))

# ajust `steps` index list to only contain reduced number of indices
list_all_pros = [0]*npro
for nn in range(npro):
  list_all_pros[nn] = steps[nn::npro]
steps = list_all_pros[rank]

Such a python script (assuming the name parallel_animation.py) can be run in parallel using the following slurm-script:

#! /bin/bash
#SBATCH --job-name=pyhur
#SBATCH --time=02:00:00
#SBATCH --output=log.o-%j.out
#SBATCH --error=log.o-%j.out
#SBATCH --ntasks=4
#SBATCH --partition=compute,compute2
#SBATCH --account=mh0033
set -x

#module load python/2.7.12
module list

source /home/mpim/m300602/bin/myactcondenv.sh
which python

startdate=`date +%Y-%m-%d\ %H:%M:%S`
mpirun -np 4 python -u parallel_animation.py --no_backend
enddate=`date +%Y-%m-%d\ %H:%M:%S`
echo "Started at ${startdate}"
echo "Ended at   ${enddate}"