Automate your daily, weekly, and monthly operations using shell scripts and [crontab](crontab) schedules.

Macro recipes help:

  1. you save time running your automations each week by no longer needing to remember and type each specific command

  2. other people operate your project for you while you’re on vacation.

  3. you promote consistency between your Pipelines and command line sessions.

  4. you quickly promote understanding of your recipe by using the Summary Detail technique.

Macro recipes contain both conventional extension-less instructions and custom .sh extension instructions and you can use to define your project schedules, commands, automations, endpoints, and environments.

Specifics

Macros recipes are a set of instruction files defined in a top level project recipe path named macro/ .

A few conventional instructions have no filename extension, but the rest of the custom instructions typically have a .sh extension.

Conventional extension-less macro instructions:

  • macro/macpack : convenient bash functions to promote operational consistency across all of your projects

  • macro/crontab : crontab instructions that define automation schedules and commands to run on schedule.

  • macro/gitlab : configuration file that defines the Gitlab location and Gitlab project ID

Custom .sh extension macro instructions:

  • Custom .sh instructions are a series of commands that would you would typically type into a command line interactive prompt that executes a sequences of processes that work together to accomplish a goal, like building a Model for auditing or sending data to a partner SFTP site.

  • Typically, there are two variations of custom .sh instructions:

  • Schedule instructions perform a sequence of multiple commands, each with exit code testing (exit 1)

  • Command instructions each invoke flatmap via a single ‘fm’ command.

Schedule Custom .sh instructions:

  • Custom .sh instructions that run combinations of commands on a schedule are defined in the top-level macro path. These schedule .sh instructions will perform attaching and committing using (macpack)[macpack] functions as well as invoking environment specific instructions.

  • Schedule .sh instructions that run on a schedule follow a file naming convention to indicate which schedule, environment, and recipe they pertain to:

macro/<schedule>-<environment>-<recipe>-<extra>.sh

1: schedule ( daily | weekly | day-of-week | monthly |  <blank> meaning runs as-needed)
2: environment, destination  ( albert | staging | pdf | production)
3: recipe (data-rewards)
4: extra/display (pdf)
5: .sh extension

Command Custom .sh instructions are:

  • Custom .sh instructions that run specific commands ad-hoc via command line or Gitlab Pipelines are defined within enviornment-specific sub-paths.

  • Command .sh instructions that you run off-schedule, ad-hoc, as-needed in a command terminal or as Gitlab pipelines are often placed below an environment specific path:

macro/<environment>/<environment>-<recipe>-<extra>.sh

2: environment, destination  ( albert | staging | pdf | production)
3: recipe (data-rewards)
4: extra/display (pdf)
5: .sh extension

Samples

Sample Friday Production Data Macro

Tips:

  • Be sure to make the macro .sh files executable!
  • Use   exit 1 after each subcommand to ensure the script exits on internal failure. 
$ chmod +x macro/friday-production-data-pay-manager.sh

#!/bin/sh
# 20.5 macpack 1liner
. $(dirname $0)/./macpack || exit 1

echo "## Macro $0"
echo ""

# Acquire two Census Files and Commit

fm data-acquire e=census acquire || exit 1

attachCommit "Acquire Census"

# Process Three data recipes by calling production macros

./macro/production/production-data-pay.sh || exit 1

./macro/production/production-data-manager.sh || exit 1

./macro/production/production-data-manager-permissions.sh || exit 1

echo "Macro Success"

Sample Macpack Instruction

Copy the ExploreX example macpack into your own project, version 20.5.3:

$ cp ~/albert/git/explorex-examples-2020/macro/macpack ./macro/macpack
$ head -1 !$
# Macpack 20.5.3
$ cksum !$
973701664 3327 ./macro/macpack

Sample Gitlab Instruction

Set the gitlab project ID to ‘explorex-examples-2020’, so that we can use gitlab environment variables in our macro .sh instructions.

macro/gitlab

# 20.5 Macpack gitlabEnvironment
CI_PROJECT_URL=https://albert-20.albertlogin.us/ExploreX/explorex-examples-2020
CI_PROJECT_ID=291

Sample Crontab Instruction

Run daily production data-demographic each morning at 7:17AM local time

macro/crontab

07 17 * * 1-5 $HOME/albert/git/explorex-examples-2021q1/macro/daily-production-data-demographic.sh

On your data node, use the system crontab editor to copy this instruction content into your users’s crontab:

crontab -e

Sample well-formed Custom .sh Names

A few examples of well-formed macro instruction filenames:

  • macro/daily-production-data-demographic.sh : Daily in production, automate transmission of a demographic file.

  • macro/tuesday-production-data-eligibility.sh : Weekly, every Tuesday, in production, automate transmission of an eligibility file.

  • macro/pdf/albert-rewards-pdf-batch.sh : As-needed command to batch all of the packaged Rewards Statement PDF pages.

  • macro/production/production-data-rewards-pdf.sh : As-needed command to package & unload all Rewards Statement PDFs to production

  • macro/staging/staging-email-401k-notice.sh : As-needed command to package and unload all emails for 401(k) notices.