Contents Menu Expand Light mode Dark mode Auto light/dark mode
azure-cmaq documentation
Logo
azure-cmaq documentation

Contents:

  • 1. Introductory Tutorial
    • 1.1. Create an Azure Account
    • 1.2. Sign up for a Developer Azure Support Plan
    • 1.3. Use Azure CLI to examine your quota
    • 1.4. Create a virtual machine
    • 1.5. Login to Virtual Machine
    • 1.6. Delete the virtual machine and all of the associated resources by deleting the resource group.
  • 2. System Requirements
    • 2.1. System Requirements for a Single Virtual Machine or Cycle Cloud Cluster
    • 2.2. Software Requirements for CMAQ on Single VM or CycleCloud Cluster
    • 2.3. Storage Options
    • 2.4. Recommended Cycle Cloud Configuration for CONUS Domain 12US2
  • 3. Intermediate Tutorial
    • 3.1. Create a HB120rs_v2 Virtual Machine
    • 3.2. Login to the Virtual Machine
    • 3.3. Mount the disk on the server as /shared using the instructions on the following link:
    • 3.4. Alternatively, you can create an nvme stripped disk that has faster performance.
    • 3.5. Obtain the Cyclecloud-cmaq code from github
    • 3.6. Change shell to use tcsh
    • 3.7. Create Environment Module for Libraries
    • 3.8. Install and Build CMAQ
    • 3.9. Copy the run scripts from the repo to the run directory
    • 3.10. Download the Input data from the S3 Bucket
    • 3.11. Run CMAQ interactively using the following command:
  • 4. Advanced Tutorial
    • 4.1. Create Cyclecloud CMAQ Cluster
    • 4.2. Modify Cyclecloud CMAQ Cluster
    • 4.3. Install CMAQv533 and pre-requisite libraries on linux
    • 4.4. Configuring selected storage and obtaining input data
    • 4.5. Copy the run scripts from the CycleCloud repo
    • 4.6. Run the CONUS Domain on 180 pes
    • 4.7. Check the status in the queue
    • 4.8. check the timings while the job is still running using the following command
    • 4.9. When the job has completed, use tail to view the timing from the log file.
    • 4.10. Check whether the scheduler thinks there are cpus or vcpus
  • 5. CMAQv5.3.3 Benchmark on HBv3_120 compute nodes and Lustre
    • 5.1. Use Cycle Cloud pre-installed with CMAQv5.3.3 software and 12US2 Benchmark data.
    • 5.2. This method relies on obtaining the code and data from blob storage.
    • 5.3. Update Cycle Cloud
    • 5.4. Log into the new cluster
    • 5.5. Verify Software
    • 5.6. Verify Input Data
    • 5.7. Examine CMAQ Run Scripts
    • 5.8. To run on the Shared Volume a code modification is required.
    • 5.9. Build the code by running the makefile
    • 5.10. Submit Job to Slurm Queue to run CMAQ on Lustre
    • 5.11. Submit a run script to run on the shared volume
    • 5.12. Submit a minimum of 2 benchmark runs
  • 6. CMAQv5.4+ Benchmark on HBv3_120 compute nodes and Lustre
    • 6.1. Use Cycle Cloud pre-installed with CMAQv5.4+ software and 12US1 Benchmark data.
    • 6.2. This method relies on obtaining the code and data from blob storage.
    • 6.3. Update Cycle Cloud
    • 6.4. Log into the new cluster
    • 6.5. Verify Software
    • 6.6. Download the input data from the AWS Open Data CMAS Data Warehouse.
    • 6.7. Verify Input Data
    • 6.8. Examine CMAQ Run Scripts
    • 6.9. To run on the Shared Volume a code modification is required.
    • 6.10. Build the code by running the makefile
    • 6.11. Submit Job to Slurm Queue to run CMAQ on Lustre
    • 6.12. Submit a run script to run on the shared volume
    • 6.13. Submit a minimum of 2 benchmark runs
  • 7. Scripts to run combine and post processing
  • 8. Scripts to post-process CMAQ output
  • 9. Install R, Rscript and Packages
  • 10. QA CMAQ
    • 10.1. Quality Assurance Checks for Successful CMAQ Run on CycleCloud
  • 11. Compare Timing of CMAQ Routines
    • 11.1. Parse timings from the log file
  • 12. Copy Output to S3 Bucket
    • 12.1. Copy Output Data and Run script logs to S3 Bucket
  • 13. Logout and Delete CycleCloud
    • 13.1. Link to Azure Instructions on how to logout and delete cyclecloud
  • 14. Performance Optimization
    • 14.1. Right-sizing Compute Nodes for a Single Virtual Machine.
    • 14.2. An explanation of why a scaling analysis is required for Single Node
    • 14.3. Benchmark Scaling Plots using Single Virtual Machine HBv120
    • 14.4. Right-sizing Compute Nodes for the CycleCloud
    • 14.5. An explanation of why a scaling analysis is required for Multinode or Parallel MPI Codes
    • 14.6. Slurm Compute Node Provisioning
    • 14.7. Benchmark Scaling Plots using CycleCloud
  • 15. Additional Resources
    • 15.1. Cycle Cloud Resources
    • 15.2. Help Resources for CMAQ
    • 15.3. Resources from Azure for diagnosing issues with running the Cycle Cloud
    • 15.4. Frequently Asked Questions
    • 15.5. Computing on the Cloud References
  • 16. Future Work
    • 16.1. List of ideas for future work
  • 17. Contribute to this Tutorial
    • 17.1. Contribute to Azure-cmaq Documentation
  v: latest
Versions
latest
cmaqv5.33
Downloads
html
On Read the Docs
Project Home
Builds
Back to top
Edit this page

15.5. Computing on the Cloud References#

WRF Cloud Computing Paper

NOAA Cloud-based Warn-on-Forecast

Next
16. Future Work
Previous
15.4. Frequently Asked Questions
Copyright © 2022, CMAS Center
Made with Sphinx and @pradyunsg's Furo