7. Scripts to run combine and post processing#
7.1. Build the POST processing routines#
Copy the buildit script from the repo, as it was corrected to use CMAQv533 rather than CMAQv532
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/combine/scripts
cp /shared/cyclecloud-cmaq/run_scripts/bldit_combine.csh .
Run the bldit script for combine.
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/combine/scripts
./bldit_combine.csh gcc |& tee ./bldit_combine.gcc.log
Copy the bldit script from the repo, as it was corrected to use CMAQv533 rather than CMAQv532.
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/calc_tmetric/scripts
cp /shared/cyclecloud-cmaq/run_scripts/bldit_calc_tmetric.csh .
Run the bldit script for calc_tmetric
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/calc_tmetric/scripts
./bldit_calc_tmetric.csh gcc |& tee ./bldit_calc_tmetric.gcc.log
Copy the bldit script from the repo.
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/hr2day/scripts
cp /shared/cyclecloud-cmaq/run_scripts/bldit_hr2day.csh .
Run the bldit script for hr2day
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/hr2day/scripts
./bldit_hr2day.csh gcc |& tee ./bldit_hr2day.gcc.log
Copy the bldit script from the repo.
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/bldoverlay/scripts
cp /shared/cyclecloud-cmaq/run_scripts/bldit_bldoverlay.csh .
Run the bldit script for bldoverlay.
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/bldoverlay/scripts
./bldit_bldoverlay.csh gcc |& tee ./bldit_bldoverlay.gcc.log
8. Scripts to post-process CMAQ output#
8.1. Note, the post-processing analysis should be done on the head node#
Ideally the post-processing would be done on the HTC queue, but the R packages were installed to the head node system library and were not acccessible to the compute nodes. Installing the R software and packages to the /shared volume will be investigated in future work.
Verify that the compute nodes are no longer running if you have completed all of the benchmark runs
squeue
You should see that no jobs are running.
Show compute nodes
scontrol show nodes
8.2. Edit, Build and Run the POST processing routines#
You need to run the post processing scripts for every benchmark case.
cp /shared/cyclecloud-cmaq/run_scripts/run_combine_conus.csh .
Examine the run script
cat run_combine_conus.csh
The post processing scripts are set up for a specific case, example:
setenv APPL 2016_CONUS_10x18pe
note, you will need to change the sed command to a different configuration if you ran another case, example:
setenv APPL 2016_CONUS_12x9pe
If you used the CMAQ Benchmark Option 1, with the pre-loaded software, then these scripts have already been modified.
Run the following scripts
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/combine/scripts
sbatch run_combine_conus.csh
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/calc_tmetric/scripts
sbatch run_calc_tmetric_conus.csh
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/hr2day/scripts
sbatch run_hr2day_conus.csh
cd /shared/build/openmpi_gcc/CMAQ_v533/POST/bldoverlay/scripts
sbatch run_bldoverlay_conus.csh
If you used the CMAQ Bechmark Option 2 to install CMAQ yourself, you will need to save and modify the scripts using the instructions below.
setenv DIR /shared/build/openmpi_gcc/CMAQ_v533/
cd $DIR/POST/combine/scripts
sed -i 's/v532/v533/g' bldit_combine.csh
cp run_combine.csh run_combine_conus.csh
sed -i 's/v532/v533/g' run_combine_conus.csh
sed -i 's/Bench_2016_12SE1/2016_CONUS_10x18pe/g' run_combine_conus.csh
sed -i 's/intel/gcc/g' run_combine_conus.csh
./bldit_combine.csh gcc |& tee ./bldit_combine.gcc.log
sed -i 's/2016-07-01/2015-12-22/g' run_combine_conus.csh
sed -i 's/2016-07-14/2015-12-23/g' run_combine_conus.csh
setenv CMAQ_DATA /shared/data
./run_combine_conus.csh
cd $DIR/POST/calc_tmetric/scripts
./bldit_calc_tmetric.csh gcc |& tee ./bldit_calc_tmetric.gcc.log
cp run_calc_tmetric.csh run_calc_tmetric_conus.csh
sed -i 's/v532/v533/g' run_calc_tmetric_conus.csh
sed -i 's/Bench_2016_12SE1/2016_CONUS_10x18pe/g' run_calc_tmetric_conus.csh
sed -i 's/intel/gcc/g' run_calc_tmetric_conus.csh
sed -i 's/201607/201512/g' run_calc_tmetric_conus.csh
setenv CMAQ_DATA /shared/data
./run_calc_tmetric_conus.csh
cd $DIR/POST/hr2day/scripts
sed -i 's/v532/v533/g' bldit_hr2day.csh
./bldit_hr2day.csh gcc |& tee ./bldit_hr2day.gcc.log
cp run_hr2day.csh run_hr2day_conus.csh
sed -i 's/v532/v533/g' run_hr2day_conus.csh
sed -i 's/Bench_2016_12SE1/2016_CONUS_10x18pe/g' run_hr2day_conus.csh
sed -i 's/intel/gcc/g' run_hr2day_conus.csh
sed -i 's/2016182/2015356/g' run_hr2day_conus.csh
sed -i 's/2016195/2015357/g' run_hr2day_conus.csh
sed -i 's/201607/201512/g' run_hr2day_conus.csh
setenv CMAQ_DATA /shared/data
./run_hr2day_conus.csh
cd $DIR/POST/bldoverlay/scripts
sed -i 's/v532/v533/g' bldit_bldoverlay.csh
./bldit_bldoverlay.csh gcc |& tee ./bldit_bldoverlay.gcc.log
cp run_bldoverlay.csh run_bldoverlay_conus.csh
sed -i 's/v532/v533/g' run_bldoverlay_conus.csh
sed -i 's/Bench_2016_12SE1/2016_CONUS_10x18pe/g' run_bldoverlay_conus.csh
sed -i 's/intel/gcc/g' run_bldoverlay_conus.csh
sed -i 's/2016-07-01/2015-12-22/g' run_bldoverlay_conus.csh
sed -i 's/2016-07-02/2015-12-23/g' run_bldoverlay_conus.csh
setenv CMAQ_DATA /shared/data
./run_bldoverlay_conus.csh