Using UNIX Wildcards with AWS S3 (AWS CLI)

Currently AWS CLI doesn’t provide support for UNIX wildcards in a command’s “path” argument. However, it is quite easy to replicate this functionality using the --exclude and --include parameters available on several aws s3 commands.

The wildcards available for use are:
"*" – Matches everything
"?" – Matches any single character
"[]" – Matches any single character between the brackets
"[!]" – Matches any single character not between the brackets

A few things to remember about using --include and --exclude with the aws s3 command:

  • You may use any number of --include and --exclude parameters.
  • Parameters passed later take precedence over parameters passed earlier (in the same command).
  • All files and objects are “included” by default, so in order to include only certain files you must use “exclude” then “include”.
  • --recursive must be used in conjunction with --include and --exclude or else commands will only perform single file/object operations.


    9 thoughts on “Using UNIX Wildcards with AWS S3 (AWS CLI)”

    1. Pingback: Use AWS CLI to Copy all Files in S3 Bucket to Local Machine - Big Datums

    2. Pingback: Copy all Files in S3 Bucket to Local with AWS CLI - Big Datums

    3. Pingback: How to Copy local files to S3 with AWS CLI - Big Datums

    4. Rakesh123

      How can I rename this ?

      aws s3 mv s3://test/ACCOUNT_OTR*.txt s3://test/ACCOUNT_OTR.TXT

      (This is my file name- ACCOUNT_OTR_20180205_223059.txt . In the above command I have used wildcard. Using include exclude how we write it ?)

    5. mahendra

      I was trying to delete more than 2000000 files with prefix MyData-* with below command .

      aws s3 rm s3://bucket2040/data/ –recursive –exclude “*” –include “MyData-*”

      Above command exit without any message or error.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">