Set AWS Credentials in Cloudera Quickstart Docker Container

Cloudera’s Quickstart Image is a fantastic way to get started quickly with the big data ecosystem. With software such as Hadoop, Spark, Hive, Pig, Impala, and Hue already set up, this Docker image is a must in your big data toolkit.

One thing the Cloudera Quickstart container is lacking however, is an easy way to set up AWS credentials to allow software like Hadoop and Spark to use S3 for job inputs and outputs.

Set AWS Credentials in Dockerfile

The cloudera-quickstart-aws Docker image is a good example of how to set AWS credentials in a Cloudera Quickstart container. This image builds off of the Cloudera Quickstart image and sets AWS Credentials as well as installs the AWS Command Line Interface.

The cloudera-quickstart-aws image has two major parts; the shell script, and the Dockerfile.

The shell script sets AWS keys to be exported as environment variables in the root user’s .bashrc file (root is default Docker user). The script also uses sed to add the s3a and s3n AWS properties to Hadoop’s core-site.xml configuration file (don’t necessarily need both s3a and s3n properties set).


The Dockerfile is what is actually used to build the Docker image. The cloudera-quickstart-aws Dockerfile copies the script mentioned above and executes it. The Dockerfile also installs the AWS Command Line Interface, which is an extremely useful tool for working with AWS services.

Build Image & Run Container

To understand how to build this image you should read the documentation on the cloudera-quickstart-aws GitHub page. But in short, the commands to to build this image locally and run a Docker container are:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">