A collection of useful AWS CLI commands and snippets I’ve gathered over time. These cover everything from S3 and Glacier operations to ECS, Lightsail, CloudFront, and Athena queries.

S3 and Glacier Link to heading

Iterate through an S3 Glacier vault and delete contents Link to heading

This multi-step process retrieves an inventory of a Glacier vault and then deletes all archives before removing the vault itself.

# To get each archive out of the vault:
aws --region us-east-1 glacier initiate-job \
  --vault-name arq_14396D48-D24A-4A8F-8EF7-B55F66B2BF68_78E9B43F-FF42-4594-9FE8-E877F458197E \
  --account-id 655943246657 \
  --job-parameters '{"Type": "inventory-retrieval"}'

# This command checks on the status of the job:
aws --region us-east-1 glacier describe-job \
  --vault-name arq_14396D48-D24A-4A8F-8EF7-B55F66B2BF68_78E9B43F-FF42-4594-9FE8-E877F458197E \
  --account-id 655943246657 \
  --job-id ppoQZmzIwiHzCgBWLy6L3nrnr6BMHeVxJ6U6_Z6kMdCnXyfgTXvG5k4TFoFz8zkPY-FdquZi_Lg6RKzRzxsyFnLGlNRh

# Once that's done I save the job output to a file:
aws --region us-east-1 glacier get-job-output \
  --vault-name arq_14396D48-D24A-4A8F-8EF7-B55F66B2BF68_78E9B43F-FF42-4594-9FE8-E877F458197E \
  --account-id 655943246657 \
  --job-id ppoQZmzIwiHzCgBWLy6L3nrnr6BMHeVxJ6U6_Z6kMdCnXyfgTXvG5k4TFoFz8zkPY-FdquZi_Lg6RKzRzxsyFnLGlNRh \
  1.json

# I then iterate through that file, deleting all of the archives:
for x in $(cat 1.json | jq ".ArchiveList[].ArchiveId"); do 
  echo "Deleting ${x}"
  aws --region us-east-1 glacier delete-archive \
    --vault-name arq_14396D48-D24A-4A8F-8EF7-B55F66B2BF68_78E9B43F-FF42-4594-9FE8-E877F458197E \
    --account-id 655943246657 \
    --archive-id ${x}
done

# To delete the vaults:
aws glacier --region us-east-1 delete-vault \
  --vault-name arq_14396D48-D24A-4A8F-8EF7-B55F66B2BF68_78E9B43F-FF42-4594-9FE8-E877F458197E \
  --account-id 655943246657

Enable intelligent tiering on all S3 buckets Link to heading 

for x in $(aws s3 ls | awk '{ print $3 }'); do 
  echo ${x}
  aws s3api put-bucket-intelligent-tiering-configuration \
    --cli-input-json "{\"Bucket\":\"${x}\",\"Id\":\"${x}\",\"IntelligentTieringConfiguration\":{\"Id\":\"${x}\",\"Status\":\"Enabled\",\"Tierings\":[{\"Days\":365,\"AccessTier\":\"DEEP_ARCHIVE_ACCESS\"}]}}"
done

Change storage class of all objects in a bucket Link to heading 

for x in $(aws s3 ls | grep 'dmp-00' | egrep -v '00014|00015' | awk '{ print $3 }'); do 
  echo ${x}
  aws s3 cp s3://${x}/ s3://${x}/ --recursive --storage-class INTELLIGENT_TIERING
done

S3 bucket policy for federated / SSO role Link to heading 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AcctMigrationDeveloper_asfasfafsafasf"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::ergon-test-chroma-datastage/*"
        }
    ]
}

Find largest objects in an S3 bucket Link to heading 

aws s3api list-objects-v2 --bucket BUCKET_NAME --query 'sort_by(Contents, &Size)[-10:].{Key:Key,Size:Size}' --output table

Sync S3 buckets across regions Link to heading 

aws s3 sync s3://source-bucket s3://destination-bucket --source-region us-east-1 --region us-west-2

Enable versioning on an S3 bucket Link to heading 

aws s3api put-bucket-versioning --bucket BUCKET_NAME --versioning-configuration Status=Enabled

List all S3 buckets with their creation dates Link to heading 

aws s3api list-buckets --query 'Buckets[].{Name:Name,Created:CreationDate}' --output table

Lightsail Link to heading

List, modify, and put firewall rules for Lightsail instances Link to heading 

aws lightsail get-instance-port-states --instance-name INSTANCE_NAME --output json | grep -v "state" > firewall.json
aws lightsail put-instance-public-ports --instance-name INSTANCE_NAME --cli-input-json file://firewall.json

ECS Link to heading

Get latest Amazon Linux ECS optimized AMI Link to heading 

aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2/recommended

List all ECS clusters Link to heading 

aws ecs list-clusters --output table

List all services in an ECS cluster Link to heading 

aws ecs list-services --cluster my-cluster --output table

Update ECS service to force new deployment Link to heading 

aws ecs update-service --cluster my-cluster --service my-service --force-new-deployment

Stop a running ECS task Link to heading 

aws ecs stop-task --cluster my-cluster --task TASK_ARN --reason "Manual stop"

Get logs from ECS container using task ID Link to heading 

TASK_ARN=$(aws ecs list-tasks --cluster my-cluster --service my-service --desired-status RUNNING --query 'taskArns[0]' --output text)
aws ecs describe-tasks --cluster my-cluster --tasks $TASK_ARN --query 'tasks[0].containers[0].name' --output text

IAM and Security Link to heading

List all IAM users Link to heading 

aws iam list-users --query 'Users[*].[UserName,CreateDate]' --output table

Find IAM users without MFA enabled Link to heading 

aws iam get-credential-report
aws iam generate-credential-report
sleep 5
aws iam get-credential-report --query 'Content' --output text | base64 -d | awk -F, '$4 == "false" {print $1}'

List all IAM access keys older than 90 days Link to heading 

aws iam list-users --query 'Users[].UserName' --output text | while read user; do
  aws iam list-access-keys --user-name $user --query "AccessKeyMetadata[?CreateDate<='$(date -d '90 days ago' -Iseconds)'].[UserName,AccessKeyId,CreateDate]" --output text
done

Assume a role and export credentials Link to heading 

CREDS=$(aws sts assume-role --role-arn arn:aws:iam::123456789012:role/MyRole --role-session-name my-session)
export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r '.Credentials.SessionToken')

Minimum requirements for Session Manager Link to heading 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:GetConnectionStatus",
                "ssm:StartSession"
            ],
            "Resource": "arn:aws:ec2:::instance/*"
        }
    ]
}

Root login notifications Link to heading

Terraform code for setting up CloudWatch event rule to monitor root account logins:

resource "aws_cloudwatch_event_rule" "root-login-event" {
  count = var.enable_events_root_login ? 1 : 0

  name        = "root-account-login"
  description = "Root account login"

  event_pattern = <<EOF
{
    "detail-type": [
      "AWS Console Sign In via CloudTrail"
    ],
    "detail": {
      "userIdentity": {
        "type": [
          "Root"
        ]
      }
    }
}
EOF
}

CloudFront Link to heading

Find the distribution / account that has your CNAME Link to heading

Useful for resolving CNAMEAlreadyExists errors. See AWS documentation for more details.

aws --profile santa_dev cloudfront list-conflicting-aliases \
  --distribution-id EMZM3HV48HKBK \
  --alias dev.muh-website.co \
  --no-cli-pager

Example output:

{
    "ConflictingAliasesList": {
        "MaxItems": 100,
        "Quantity": 1,
        "Items": [
            {
                "Alias": "dev.muh-website.co",
                "DistributionId": "*******9I2761G",
                "AccountId": "******673587"
            }
        ]
    }
}

EC2 Link to heading

Install atop on Amazon Linux Link to heading 

amazon-linux-extras install -y epel
yum -y install atop
sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atop
systemctl enable atop.service
systemctl restart atop.service

List all EC2 instances with name, instance ID, type, and state Link to heading 

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value|[0],InstanceId,InstanceType,State.Name]' --output table

Stop all running EC2 instances with a specific tag Link to heading 

aws ec2 describe-instances --filters "Name=tag:Environment,Values=dev" "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n 1 aws ec2 stop-instances --instance-ids

Find unattached EBS volumes Link to heading 

aws ec2 describe-volumes --filters Name=status,Values=available --query 'Volumes[*].{ID:VolumeId,Size:Size,Type:VolumeType,Created:CreateTime}' --output table

Get instance metadata from within an EC2 instance Link to heading 

# Get instance ID
curl -s http://169.254.169.254/latest/meta-data/instance-id

# Get availability zone
curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone

# Get public IP
curl -s http://169.254.169.254/latest/meta-data/public-ipv4

# Get IAM role
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/

Create an AMI from a running instance Link to heading 

aws ec2 create-image --instance-id i-1234567890abcdef0 --name "My-AMI-$(date +%Y%m%d-%H%M%S)" --description "Automated backup"

RDS Link to heading

List all RDS instances with status Link to heading 

aws rds describe-db-instances --query 'DBInstances[*].[DBInstanceIdentifier,DBInstanceClass,Engine,DBInstanceStatus]' --output table

Create a manual snapshot of an RDS instance Link to heading 

aws rds create-db-snapshot --db-instance-identifier mydb --db-snapshot-identifier mydb-snapshot-$(date +%Y%m%d-%H%M%S)

Find RDS instances without encryption Link to heading 

aws rds describe-db-instances --query 'DBInstances[?StorageEncrypted==`false`].[DBInstanceIdentifier,Engine,DBInstanceClass]' --output table

List all RDS snapshots older than 30 days Link to heading 

aws rds describe-db-snapshots --query "DBSnapshots[?SnapshotCreateTime<='$(date -d '30 days ago' -Iseconds)'].[DBSnapshotIdentifier,SnapshotCreateTime]" --output table

Lambda Link to heading

List all Lambda functions with runtime and memory Link to heading 

aws lambda list-functions --query 'Functions[*].[FunctionName,Runtime,MemorySize,LastModified]' --output table

Invoke a Lambda function Link to heading 

aws lambda invoke --function-name my-function --payload '{"key":"value"}' response.json
cat response.json

Update Lambda function code from a zip file Link to heading 

aws lambda update-function-code --function-name my-function --zip-file fileb://function.zip

Get Lambda function logs (recent 10) Link to heading 

aws logs tail /aws/lambda/FUNCTION_NAME --follow

CloudWatch Logs Link to heading

Tail CloudWatch logs in real-time Link to heading 

aws logs tail /aws/lambda/my-function --follow --format short

Search CloudWatch logs for a specific pattern Link to heading 

aws logs filter-log-events --log-group-name /aws/lambda/my-function --filter-pattern "ERROR" --start-time $(date -d '1 hour ago' +%s)000

Delete old log streams Link to heading 

aws logs describe-log-streams --log-group-name /aws/lambda/my-function --order-by LastEventTime --descending --max-items 100 | jq -r '.logStreams[] | select(.lastEventTimestamp < (now - 2592000) * 1000) | .logStreamName' | xargs -I {} aws logs delete-log-stream --log-group-name /aws/lambda/my-function --log-stream-name {}

Export logs to S3 Link to heading 

aws logs create-export-task --log-group-name /aws/lambda/my-function --from $(date -d '7 days ago' +%s)000 --to $(date +%s)000 --destination my-logs-bucket --destination-prefix lambda-logs/

Route 53 Link to heading

List all hosted zones Link to heading 

aws route53 list-hosted-zones --query 'HostedZones[*].[Name,Id,ResourceRecordSetCount]' --output table

List all records in a hosted zone Link to heading 

aws route53 list-resource-record-sets --hosted-zone-id Z1234567890ABC --output table

Create a simple A record Link to heading 

aws route53 change-resource-record-sets --hosted-zone-id Z1234567890ABC --change-batch '{
  "Changes": [{
    "Action": "CREATE",
    "ResourceRecordSet": {
      "Name": "example.com",
      "Type": "A",
      "TTL": 300,
      "ResourceRecords": [{"Value": "192.0.2.1"}]
    }
  }]
}'

VPC Link to heading

List all VPCs with their CIDR blocks Link to heading 

aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,CidrBlock,Tags[?Key==`Name`].Value|[0]]' --output table

Find unused security groups Link to heading 

aws ec2 describe-security-groups --query 'SecurityGroups[?IpPermissions==`[]` && IpPermissionsEgress==`[]`].[GroupId,GroupName]' --output table

List all subnets with available IP counts Link to heading 

aws ec2 describe-subnets --query 'Subnets[*].[SubnetId,CidrBlock,AvailableIpAddressCount,AvailabilityZone]' --output table

Show all internet gateways not attached to a VPC Link to heading 

aws ec2 describe-internet-gateways --query 'InternetGateways[?Attachments==`[]`].[InternetGatewayId]' --output table

Cost and Billing Link to heading

Get current month’s AWS costs by service Link to heading 

aws ce get-cost-and-usage --time-period Start=$(date +%Y-%m-01),End=$(date +%Y-%m-%d) --granularity MONTHLY --metrics BlendedCost --group-by Type=DIMENSION,Key=SERVICE --output table

Find most expensive resources in the last 30 days Link to heading 

aws ce get-cost-and-usage --time-period Start=$(date -d '30 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) --granularity MONTHLY --metrics UnblendedCost --group-by Type=DIMENSION,Key=RESOURCE_ID --output json | jq '.ResultsByTime[].Groups | sort_by(.Metrics.UnblendedCost.Amount | tonumber) | reverse | .[0:10]'

SNS Link to heading

List all SNS topics Link to heading 

aws sns list-topics --query 'Topics[*].TopicArn' --output table

Publish a message to an SNS topic Link to heading 

aws sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic --message "Hello from AWS CLI" --subject "Test Message"

List subscriptions for a topic Link to heading 

aws sns list-subscriptions-by-topic --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic --output table

SQS Link to heading

List all SQS queues Link to heading 

aws sqs list-queues --output table

Send a message to an SQS queue Link to heading 

aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue --message-body "Test message from CLI"

Receive and delete messages from SQS queue Link to heading 

# Receive messages
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue --max-number-of-messages 10

# Delete a message
aws sqs delete-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue --receipt-handle "MESSAGE_RECEIPT_HANDLE"

Purge all messages from a queue Link to heading 

aws sqs purge-queue --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue

Get queue attributes (message count, etc.) Link to heading 

aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue --attribute-names All

DynamoDB Link to heading

List all DynamoDB tables Link to heading 

aws dynamodb list-tables --output table

Describe a DynamoDB table Link to heading 

aws dynamodb describe-table --table-name my-table --query 'Table.[TableName,TableStatus,ItemCount,TableSizeBytes]' --output table

Scan a DynamoDB table (get all items) Link to heading 

aws dynamodb scan --table-name my-table --output json

Query DynamoDB table with a key condition Link to heading 

aws dynamodb query --table-name my-table --key-condition-expression "id = :id" --expression-attribute-values '{":id":{"S":"12345"}}' --output json

Put an item into DynamoDB Link to heading 

aws dynamodb put-item --table-name my-table --item '{"id":{"S":"12345"},"name":{"S":"John Doe"},"age":{"N":"30"}}'

Delete an item from DynamoDB Link to heading 

aws dynamodb delete-item --table-name my-table --key '{"id":{"S":"12345"}}'

Create a backup of a DynamoDB table Link to heading 

aws dynamodb create-backup --table-name my-table --backup-name my-table-backup-$(date +%Y%m%d)

Athena Link to heading

Create an Athena table of a TSV file in S3 Link to heading

This example works with gzipped files and skips the header row:

CREATE EXTERNAL TABLE IF NOT EXISTS `imdb-name`.`name_dir` (
  `nconst` string,
  `primaryName` string,
  `birthYear` int,
  `deathYear` int,
  `primaryProfession` string,
  `knownForTitles` string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('field.delim' = '\t')
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://imdb-example-dataset/name/'
TBLPROPERTIES (
  "skip.header.line.count"="1"
);

Join tables and run a select Link to heading 

SELECT
    tc.tconst,
    tc.originalTitle,
    rd.averageRating AS rating,
    rd.numVotes AS numberOfVotes
FROM
    title_dir AS tc
LEFT JOIN
    ratings_dir AS rd
ON
    tc.tconst = rd.tconst
WHERE
    tc.titleType = 'movie' AND rd.numVotes > 10000
ORDER BY
    rd.averageRating DESC,
    rd.numVotes DESC
LIMIT 10;