Find and copy ingredient files into your recipe filesystem at the beginning of your flatmap run.

Specify which paths and file patterns should be scanned, defaults to the attach/ path, but can look anywhere in your project. Supports pgp decryption, time range filters, and multiple sorting/prioritization schemes.

Screenshots

Console Output After Attaching Multiple Files

Specifics

Attach is available in Flatmap Version 20.3

Default

Attach requires a minimum configuration with attach.file-regex=’.csv’ to be specified.

The ‘attach’ top-level recipe path is scanned, this can be overriden with attach.source-path

Regular Expressions

Files are matched using case-sensitive Regular Expression patterns.

  • By default the Regexes match any part of the filename.

  • Use ^ at the front of your Regex if you want to match the beginning of the filename, example: “^Check”

  • Use $ at the end of your Regex if you want match the end of the filename, example: “[.]txt$”

Attach Dated Paths

  • By default, attach looks in the top-level attach/ recipe path, recusively scanning all paths.

  • By default, attach gives priority to files located in dated attach paths, such as: /attach/2020-10-08/

  • Files located outside of dated paths are considered as lower priority, such as: /attach/uploads/

Multiple

  • To match multiple files, which is really helpful to put many things into a Combine path, use the config attach.multiple=true

Decryption

  • To decrypt files at rest as they are copied, specify the attach.pgp.decrypt=true and the corresponding attach.pgp.privateKeyFile and attach.pgp.passphrase

  • It is best practice to never check-in your passphrase, but instead use environment variables to provide it at runtime.

  • It is best practice to always encrypt your private key with a strong passphrase if you are going to check the key into your project.

Configuration Sample

attach {
  multiple = true
  source-path = "src" # ovverides attach/ default
  ## Example of specifying many files-regex
  file-regex = "COMP.+[.]txt$"
  files-regex = [
    "PAY.+[.]txt$",
    "CHECK.+[.]txt$"
  ]
}

Configuration Case Classes

/*
 * 2020.3 Attach - Copy files into your data recipes
 * 
 */

case class AttachActivityConfig(fileRegex: Option[String],
                                filesRegex: Option[Seq[String]],
                                addExtension : Option[String],
                                maxAgeDays: Option[Int],
                                maxAge: Option[Duration],
                                // TODO Enable filtering to only look at certain attach extras, the part after the date.
                                attachPath: Option[String] = None,
                                // Enable Override of the ./attach recipe singleton.
                                sourcePath: Option[String] = None,
                                pgp: Option[PgpCommonConfig] = None,
                                // 2020.2 Enable Sorting by Alpha as the default
                                sortBy: Option[String],
                                removeExisting: Option[Boolean], // 2020.2 Remove exisitng files in path before attach
                                multiple: Option[Boolean], // 2020.4 Support Multiple Attach
                                requirePathDate: Option[Boolean] = None // 2020.4 Kill switch to ignore files outside of dated paths
) extends Activity19Config[AttachActivityConfig]  {