Lingk's fundamental design philosophy is related to big data architecture. Lingk tools make it easy to access big data capabilities. Here are some key concepts to understand:

  1. Data is always memory
    Data is cleared after processing is complete unless explicitly written.
  2. Data is always processed as a set
    Think in terms of managing groups of data not individual rows ( with FOR loops, for example).
  3. The full power of SQL is available to data sets
    For conditional logic and filtering in Lingk Query Language (LQL), you can utilize the full capabilities of SQL. (see LQL statements)
  4. Data is always processed in the JSON format 
    Nested structures and arrays have unique support in the LQL dialects due to the power of Spark SQL
  5. Data can be automatically unioned from multiple API calls or files
    Creating sets of data from different data sources is made connectors that automatically join similar data sets together. (See ParameterizedBy)
  6. Create tables of data from other tables of data
    Similar to other SQL dialects, a powerful data processing technique is to create temp tables. In Lingk, these table are automatically dropped after execution. (see Recipe Execution Lifecycle)
  7. All data can be diffed
    Data coming from any provider or format can be cached using the data diff post processor.  (see Data Diff)
  8. Null values require schema to be properly handled
    When dealing with null values, schema enables subsequent LQL operations referencing the field to find the column in the set and succeed.  (see Schema)