delimited

Use the delimited type for your format if you need to read or write a comma, pipe, or other formats into the Transformer Engine for processing. An example of a delimited format is shown below.

Reading Data

-
    name: csv
    type: delimited
    properties:
      delimiter: ','
      header: true
      inferSchema: true
  • delimiter: this property should equal the delimiter used witin your delimited file (ie, comma (,), pipe (|), colon (:), etc.)
  • header: set to true if the delimited file contains a header row (names of columns) or false if it does not
  • inferSchema: set to true to automatically detect schema of the file. If inferSchema is false, all data will be string data and require casting in SQL statements. (defaults to True)

Writing Data

-
    name: csv
    type: delimited
    properties:
      quoteAllFields: true
      delimiter: ','
      header: true
  • quoteAllFields: set to true if your .csv file has all data in quotes, or false if not
  • delimiter: this property should equal the delimiter used witin your .csv file (ie, comma (,), pipe (|), colon (:), etc.)
  • header: set to true if the .csv file contains a header row (names of columns) or false if it does not