Big Datums - A Data Blog

Create an MD5 Hash Code from a String in Java

Creating a hash codes from strings is a common programming task. MD5 is a very popular and commonly used hashing algorithm. Creating an MD5 hash code from a String in Java can be easily done, and is shown in the code below:

package com.bigdatums.hashing;

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

public class BdHashingUtils {

    public static String md5OfString(String str){
        StringBuilder sb = new StringBuilder();

        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            md.update(str.getBytes());
            byte[] digest = md.digest();

            for(byte dByte : digest){
                sb.append(String.format("%02x", dByte & 0xff));
            }
        } catch(NoSuchAlgorithmException e) {
            System.out.println(e);
        } catch(NullPointerException e){
            System.out.println("NullPointerException: Please provide a valid string");
        }

        return sb.toString();
    }

    public static void main(String[] args){
        System.out.println(BdHashingUtils.md5OfString("this is a test string"));
        System.out.println(BdHashingUtils.md5OfString(null));
    }

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

package com.bigdatums.hashing;

import java.security.MessageDigest;

import java.security.NoSuchAlgorithmException;

public class BdHashingUtils {

public static String md5OfString(String str){

StringBuilder sb = new StringBuilder();

try {

MessageDigest md = MessageDigest.getInstance("MD5");

md.update(str.getBytes());

byte[] digest = md.digest();

for(byte dByte : digest){

sb.append(String.format("%02x", dByte & 0xff));

}

} catch(NoSuchAlgorithmException e) {

System.out.println(e);

} catch(NullPointerException e){

System.out.println("NullPointerException: Please provide a valid string");

}

return sb.toString();

}

public static void main(String[] args){

System.out.println(BdHashingUtils.md5OfString("this is a test string"));

System.out.println(BdHashingUtils.md5OfString(null));

}

The code above is does several things: Creates a StringBuilder object to

Generating Fake Data in Java with jFairy

By Big Datums

Generating fake data can be a common need when developing applications or loading test data into a database. jFairy is a great fake data generator library built in Java that is very easy to use. jFairy allows you to build data sets containing diverse types of data including names, addresses, telephone numbers, dates, large integers,

How to Create a Fat Jar with Maven

By Big Datums

A fat jar or uber jar is a jar that contains the classes of your current project as well as all of the classes on which it depends. For example, if your application requires Joda-Time, your jar file will contain all the classes of your current project, as well as all the classes of Joda-Time.

Get List of Objects in S3 Bucket with Java

By Big Datums

Often when working with files in S3, you need information about all the items in a particular S3 bucket. Below is an example class that extends the AmazonS3Client class to provide this functionality. For the most part this class has been adapted from the sample in this AWS post. Aside from some additional methods, one

Export data from Postgres as CSV

By Big Datums

Exporting data from Postgres as a CSV or tab delimited file can be fairly simple using the COPY command. Examples below. Export an entire table as CSV:

psql -c "COPY my_table TO STDOUT WITH CSV" > output.csv

1	psql -c "COPY my_table TO STDOUT WITH CSV" > output.csv

Export an entire table as CSV with Header:

psql -c "COPY my_table TO STDOUT WITH CSV HEADER" > output.csv

1	psql -c "COPY my_table TO STDOUT WITH CSV HEADER" > output.csv

Export query as CSV:

psql -c "COPY (SELECT first_name, last_name FROM my_table) TO STDOUT WITH CSV" > output.csv

1	psql -c "COPY (SELECT first_name, last_name FROM my_table) TO STDOUT WITH CSV" > output.csv

Export query as data tab delimited text file:

psql -c "COPY (SELECT first_name, last_name FROM my_table) TO STDOUT" > output.txt

1	psql -c "COPY (SELECT first_name, last_name FROM my_table) TO STDOUT" > output.txt

Export data directly to

Using UNIX Wildcards with AWS S3 (AWS CLI)

By Big Datums

Currently AWS CLI doesn’t provide support for UNIX wildcards in a command’s “path” argument. However, it is quite easy to replicate this functionality using the –exclude and –include parameters available on several aws s3 commands. The wildcards available for use are: “*” – Matches everything “?” – Matches any single character “[]” – Matches any

Create a Docker group Linux

By Big Datums

When working with Docker in Linux, you may find yourself using sudo before every Docker command. This is because the docker daemon binds to a Unix socket (which by default is owned by the root user) rather than a TCP port. Thus the reason for always needing to use sudo. A good way to avoid

How to Enable the ‘root’ user on Ubuntu

By Big Datums

By default, the root user is locked in Ubuntu. To unlock it, simply run the command below, and provide the new password for the root user when prompted.

sudo passwd root
[sudo] password for nik: 
Enter new UNIX password: 
Retype new UNIX password:

1

2

3

4

sudo passwd root

[sudo] password for nik:

Enter new UNIX password:

Retype new UNIX password:

Now you should be able to log in as the root user using the su command:

$ su -

1

$ su -

If for some reason you want to lock the

Count Unique Elements in Text File with AWK

By Big Datums

Counting the distinct/unique elements of text file is a common task. Below is an example of doing this is AWK, using sample_data_1.txt.

 cat sample_data_1.txt \
| awk 'BEGIN{FS="\t"} NR>1{names[$2]=1} END{print length(names)}'

1 2	cat sample_data_1.txt \ \| awk 'BEGIN{FS="\t"} NR>1{names[$2]=1} END{print length(names)}'

Here is what is happening above: cat sample_data_1.txt – reading the file piping the data to AWK BEGIN{FS=”\t”} – specifying the field separators of the file NR>1 – Only executing the following