Selected lessons in bash scripting

The masterpiece

Let us first have a look at a more or less complicated bash script that employs a variety of features needed in bash scripting. The following script reads an input csv file line by line, stores each csv entry into a field in an array and then writes each value into a postgres database.

#!/bin/bash
# requires "list_of_ids" and "historical_Homer_final.csv" in same dir

declare -a TASKIDS=(`awk -F"|" '{printf "%s ",$1}' list_of_ids`)
declare -a DESCIDS=(`awk -F"|" '{printf "%s ",$2}' list_of_ids`)

while read line
do
 IFS=',' read -r -a array <<< "$line"
 ORIGINALDATE=`date --universal '+%s' -d "${array[0]}"`
 let ORIGINALDATE+=341712000

 for index in ${!array[*]}
 do
  if [[ $index -ge 1  && ${TASKIDS[$index-1]} -ge 1 ]]
  then
   #write value to DB
   VALUE="${array[$index]}"
   TASKID=${TASKIDS[$index-1]}
   DESCID=${DESCIDS[$index-1]}
   sudo -u postgres psql database --command "insert into history.measure (value, aggregation_counter, measurement_time, task_id, descriptor_id) values ($VALUE, 1, to_timestamp($ORIGINALDATE), $TASKID, $DESCID)"
  fi
 done
done < historical_data.csv 

Arrays

We can declare an array by using declare -a followed by the variable and assign operator and then the respective list of white space separated values.

Another way to declare an array is in the line IFS=',' read -r -a array <<< "$line". Here we use the comma field separator and the read command in order to read from the varibale $line.

Once we have an array, we can access individual fields by specifying an index like this: ${array[0]} or by iterating over the array with an index variable. Which we can get like this: index in ${!array[*]}

Here, using the index allows me to combine fields from several ordered arrays with similar length and pass them into my postgres command.

Date Calculations

Another awesome thing we can do with bash is calculate a date. While many recommend to use the -d "+10 days" parameter to add 10 days to the current date, I wanted to add 3955 days to a date I got from my input file. I was not able to calculate this via the date command but used seconds since unix time instead.

By specifying '%s' I converted the input date into seconds and then by using let I was able to execute simple arithmetics and then, later on, insert the seconds as a timestamp into the postgresql db by using the to_timestamp() function.

AWK

AWK, named after its creators Aho, Weinberger and Kernighan has been very helpful to me for over 10 years now. It is a specific scripting language and tool for string manipulation. The command awk -F"|" '{printf "%s ",$1}' list_of_ids will set the field separator to | and then print the first field followed by a white space and nothing else - just as we need it in order to use this within an array.

Input from File

Within a bash script we can input lines from a comma separated file by using the read command and stream manipulation by using the < symbol.

Sweet!