18.1.4. Configuring selected storage and obtaining input data#
Install AWS CLI to obtain data from AWS S3 Bucket#
see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
cd /shared/build
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
edit .cshrc file to add /usr/local/bin to path#
vi ~/.cshrc`
add /usr/local/bin to the set path line
Run csh at the command line
Verify you can run the aws command#
aws --help
If not, you may need to logout and back in.
Set up your credentials for using s3 copy (you can skip this if you do not have credentials)
aws configure
Change ownership#
sudo chown azureuser /shared/data
ls /shared/data
df -h
Output:
/dev/mapper/vg_cyclecloud_builtinshared-lv0 1000G 66G 935G 7% /shared
Use the following aws cp command to copy the CONUS input data from the CMAS s3 bucket#
Modify the script if you want to change where the data is saved to. Script currently uses /shared/data
cd /shared/data
aws s3 --no-sign-request --region=us-east-1 cp --recursive s3://cmas-cmaq/CMAQv5.4_2018_12US1_Benchmark_2Day_Input .
check that the resulting directory structure matches the run script
Note, this input data requires 85 GB of disk space
cd /shared/data/2018_12US1
du -sh
output:
85G .
CMAQ Cycle Cloud is configured to have 1 Terrabytes of space on the /shared filesystem, to allow multiple output runs to be stored.