AWS CLI S3 Cheatsheet

It's much faster to interface with S3 files through the CLI than it is to use the web browser. Here is a cheatsheet of useful S3 based CLI commands. In order to use these commands, you will need to have configured your AWS credentials.

Related Posts

• Getting Started With The AWS CLI

Cheats

Remove All Files In A Bucket

aws s3 rm --recursive s3://my-bucket

One cannot use the * character as a wildcard. S3 considers this a valid character for a filename. Hence the use of the --recursive flag.

Copy A Bucket To Another Bucket

aws s3 sync \
  s3://my-bucket \
  s3://my-second-bucket

• Delete Flag - Add the --delete flag if you want to make sure the destination bucket does not contain any files that are in the source bucket.
• Move Between Accounts - One can use this to copy files between two different accounts, as long as one of the account's buckets is publicly accessible!
• Efficient/Fast - The files go direct from server to server (e.g. not through you) so this is super fast!
• Empty Folder Handling - Sync will not copy across empty folders/directories. Only folders that have files somewhere inside them will show up on the destination.

CP Alternative

Alternatively, one could use cp instead of sync as shown below. However, this will copy all files, even if they already exist in the target, which is obviously more costly in terms of performance/networking.

aws s3 cp --recursive \
  s3://my-bucket \
  s3://my-second-bucket

Move A Bucket's Contents

Same as before but with "mv" instead of "cp"

Upload To Bucket

aws s3 cp \
  /path/to/local/file.txt \
  s3://my-bucket/sub-folder/file.txt

Download File From Bucket

aws s3 cp \
  s3://my-bucket/sub-folder/file.txt \
  /path/to/local/file.txt

Syncing Two Buckets

You can sync from one bucket to another with:

aws s3 sync \
   s3://BucketFrom \
   s3://BucketTo

If the buckets are across two different accounts, you will need one of the two buckets to be public, and have your credentials configured for the private one.

Get The Size Of An S3 Bucket

aws s3 ls \
  --summarize \
  --human-readable \
  --recursive \
  s3://bucket-name/

Get The Size Of An S3 Bucket Folder

aws s3 ls \
  --summarize \
  --human-readable \
  --recursive \
  s3://bucket-name/directory

Get Size Of S3 Bucket (Faster)

The above commands get the size of a bucket by looping through every file which takes a long time if you have a large bucket.

Instead, do the following:

sudo apt-get install s4cmd
s4cmd du --recursive s3://my-bucket-name

You will need to sum up the integers and plug in "total x 512 bytes" into Google to get a human readable number.