Performance Expectations and Characteristics of Lingk
Lingk supports both batch and streaming data. Performance can vary based on the connector and the data use case, so the information provided here is tied to customer use cases on mature connectors.
Batch data works in microbatches and relies on Spark Reliable Streaming.
Starting a Spark batch job:
Reading >800,000 PeopleSoft records from the Lingk Adapter (with page size of 10,000), transforming and writing to Salesforce:
Streaming data works in microbatches and relies on Spark Reliable Streaming. Like batch data, streaming data is processed in memory.
Starting a Spark streaming worker for recipe:
20 seconds (default)