Copy all Files in S3 Bucket to Local with AWS CLI

The AWS CLI makes working with files in S3 very easy. However, the file globbing available on most Unix/Linux systems is not quite as easy to use with the AWS CLI. S3 doesn’t have folders, but it does use the concept of folders by using the “/” character in S3 object keys as a folder

read more

How to Sum Values in Awk

Knowing how to sum values in Awk can be extremely useful when working with data. Fortunately, it is very easy. Using variables in Awk will feel different than in other languages. There are no data types in Awk. A variable will either be considered a number or string depending on the context in which it

read more

How to Remove Last N Columns of File in Linux

A great way to remove the last n columns of a file in Linux is to use the combination of the cut and rev commands. The cut command is used to cut/remove sections/columns of text files. The rev command is used the reverse the content of lines in text files. In order to use these

read more

3 ways to get the Nth Line of a File in Linux

The need to get/print a particular line of a file on the Linux shell is a common task. Luckily there are various ways to do this. Below are three great ways to get the nth line of a file in Linux. 1. head / tail Simply using the combination of the head and tail commands

read more

Using UNIX Wildcards with AWS S3 (AWS CLI)

Currently AWS CLI doesn’t provide support for UNIX wildcards in a command’s “path” argument. However, it is quite easy to replicate this functionality using the –exclude and –include parameters available on several aws s3 commands. The wildcards available for use are: “*” – Matches everything “?” – Matches any single character “[]” – Matches any

read more

How to Enable the ‘root’ user on Ubuntu

By default, the root user is locked in Ubuntu. To unlock it, simply run the command below, and provide the new password for the root user when prompted.

Now you should be able to log in as the root user using the su command:

If for some reason you want to lock the

read more

Count Unique Elements in Text File with AWK

Counting the distinct/unique elements of text file is a common task.  Below is an example of doing this is AWK, using sample_data_1.txt.

Here is what is happening above: cat sample_data_1.txt – reading the file piping the data to AWK BEGIN{FS=”\t”} – specifying the field separators of the file NR>1 – Only executing the following

read more