Lingk supports both batch and streaming data. Performance can vary based on the connector and the data use case, so the information provided here is tied to customer use cases on mature connectors.

Batch Performance

Batch data works in microbatches and relies on Spark Reliable Streaming.

Starting a Spark batch job:
3-5 seconds

Reading >800,000 PeopleSoft records from the Lingk Adapter (with page size of 10,000), transforming and writing to Salesforce:
~20 minutes

Streaming Performance

Streaming data works in microbatches and relies on Spark Reliable Streaming. Like batch data, streaming data is processed in memory.

Starting a Spark streaming worker for recipe:
20 seconds

Microbatch interval:
20 seconds (default)