Refresh

- statement: refresh canvasEnrollments # defer collection the data for enrollments until this step
- statement: (enrollments) => select * from canvasEnrollments

By default, a recipe reads all the data from connectors before executing a recipe. If a recipe has a parameterizedBy property, each row in the associated table will equate to a single operation in the connector. Once all the operations are complete, all the data will be unioned into a single table, enabling easy analysis and transformation.

When to use REFRESH:

  • When needed to get the latest information into an existing data table (reread) from a connector AFTER a previous statement has inserted or updated data.
  • When using an API and there is no easy way to get a collection of data other than making a number of calls with different query string parameters.

Example Recipe using Refresh for Parameterized Operations

connectors:
-
    name: canvasCourses
    type: canvasReader
    properties:
      path: api/v1/accounts/1/courses?completed=false
      pageSize: 100

-
    name: canvasEnrollments
    type: canvasReader
    parameterizedBy: canvasCourses
    properties:
      path: api/v1/courses/{{var.id}}/enrollments
      pageSize: 100   

statements:

# Courses
- statement: (courses) => select * from canvasCourses
- statement: print courses
- statement: (coursesCount) => select count(*) from canvasCourses
- statement: print coursesCount

# Enrollments
- statement: refresh canvasEnrollments
- statement: (enrollments) => select * from canvasEnrollments
- statement: print enrollments
- statement: (enrollmentsCount) => select (*) from canvasEnrollments
- statement: print enrollmentsCount