Copy all Files in S3 Bucket to Local with AWS CLI

The AWS CLI makes working with files in S3 very easy. However, the file globbing available on most Unix/Linux systems is not quite as easy to use with the AWS CLI. S3 doesn’t have folders, but it does use the concept of folders by using the “/” character in S3 object keys as a folder delimiter.

To copy all objects in an S3 bucket to your local machine simply use the aws s3 cp command with the --recursive option.

For example aws s3 cp s3://big-datums-tmp/ ./ --recursive will copy all files from the “big-datums-tmp” bucket to the current working directory on your local machine. If there are folders represented in the object keys (keys containing “/” characters), they will be downloaded as separate directories in the target location.

The command aws s3 cp s3://big-datums-tmp/myFolder/ ./ --recursive is almost the same as the one above, but this command will only copy files from “myFolder” folder (objects with keys starting with “myFolder/”).

For using wildcards and patterns to copy only certain files, refer to Using Wildcards with AWS CLI on how to correctly use the --include and --exclude options.

One thought on “Copy all Files in S3 Bucket to Local with AWS CLI”

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">