Sweep

Tidy up your recipe filesystem at the end of at flatmap run. Specify which paths and files should be retained by moving and optionally encrypting into an attach dated-path. Supports writing a .commit instruction!

Screenshots

Console Output After Sweeping a SimpleCsv Model

Specifics

Sweep is available in Flatmap Version 2020.4

Default

Sweep command can be invoked with zero configuration.

Example usage:

fm data acquire join model receipt sweep

The zero configuration is:

• sweep.created=true, Files created since the most recent join/plan will be included

• sweep.updated=true, Files updated since the most recent join/plan will be included

• sweep.overwrite=true, If a file with the same name already exists in the destination path, overwrite

• sweep.require=false, If zero files are found to sweep, do not throw an exception

• sweep.attach-path=””, The destination is an attach dated path in the format YYYY-MM-DD/

• no encryption, Files will not be encrypted when moved

• no commit, A commit instruction will not be written

• no retry, Sweep will not attempt to wait and retry

Retry

Sweep retry is designed to ‘watch’ for arrival of certain files into paths.

If all paths and filesRegex are all satisfied within the retry duration, then the sweep runs and completes succesfully.

Otherwise, if not all of the paths and filesRegex are satisfied, then the sweep command fails with exception after the retry.duration.

Encryption

Best Key Default Behavior

flatmap join model encrypt sweep.encrypt=true

If configuration encrypt=true or pgp.encrypt=true is specified without the configuration pgp.encrypt.publicKeyFile, then the project path is scanned recursively to find the most recently created public key, fails if no keys are found.

Samples

Configuration Sample - Unit Test Configuration Showing Possible values

sweep {

  created = true
  updated = true
  overwrite = true
  require = true

  fileRegex = "*.md"
  filesRegex = [ "Model/.*.csv" ],

  path = "**"
  paths = [ "Receipt" ],

  retry { duration = 5.m, frequency = 30.s },

  pgp {
    encrypt = true
    publicKeyFile = "test.key"
  }

  commit {
    author { name: "Unit Test", email: "unit@test.com" }
    message = "Unit Test Commit Message"
    alert { emails: [ "taylor.brockman@brainpowersoftware.com" ] }
  }
}

Configuration Case Classes

/*
 * 2020.4 Sweep - Relocate files, great for moving models and receipts into attach!
 *
 * Second activity configured via the automapper,
 * Deciding to make everything optional with the configuration to support a 'zero' config invocation option
 *
 */

case class SweepActivityConfig(created: Option[Boolean] = Some(true), // Autosweep all 'new' files since fm start
                               updated: Option[Boolean] = Some(true), // Autosweep all 'updated' files fm start
                               overwrite: Option[Boolean] = Some(true), // Enable Overwrite of destination file by default
                               require: Option[Boolean] = Some(false), // Require that at least 1 file is swept, Throw exception if true
                               fileRegex: Option[String], // Classic File Regex Method
                               filesRegex: Option[Seq[String]], // Classic File Regex Method
                               path: Option[String], // Modern Pathspec
                               paths: Option[Seq[String]], // Modern Pathspec array

                               attachPath: Option[String] = None, // Calculated an attachPath dated destination
                               destinationPath: Option[String] = None, // Alternatively, explicitly set a desitnation path

                               retry: Option[SweepRetryConfig] = None, // Enable a sweep to block and continue to re-check until all requirements satisfied

                               encrypt: Option[Boolean] = None, // Top Level Encrypt, which defaults to finding a pgp key
                               pgp: Option[PgpCommonConfig] = None, // Encrypt on Sweep

                               commit: Option[SweepCommitConfig] = None // Write a commit instruction to be picked up by reactor

                               // Future Ideas:
                               // addExtension : Option[String],

                               )
  extends Activity19Config[SweepActivityConfig]




case class SweepRetryConfig(duration: Duration, frequency: Duration ) {

}



// 2020.4 Sweep Supports writing a .commit instruction for pick up by reactor
//
case class SweepCommitAuthorConfig( name: String, email: String )

// The lack of key information is intentional!  Tokens + keys are configured by the process (Reactor) that responds to commit instructions
case class SweepCommitAlertConfig(email: Option[String],
                                  emails: Seq[String],
                                  slackPostUrl: Option[String],
                                  gitlabPostUrl: Option[String],
                                  v1WebhookPostUrl: Option[String] )

case class SweepCommitConfig( message: String, // Commit Message
                              author: SweepCommitAuthorConfig, // The author information of woh should do the commit
                              alert: Option[SweepCommitAlertConfig] // Send alerts/messages upon succesful commit
                            )