Skip to content

Schedulers

Most integrations are not real-time, meaning that data is not processed immediately after it is received. Instead, data is processed at specific times according to a schedule. For example, an integration may be set up to fetch data from a source system every 5 minutes and send it to a target system.

This has many advantages, such as:

  • 'Built-in' ability to retry tasks if they fail
  • Ability to process data in batches, which is more efficient than processing data one-by-one
  • Ability to process data at times when the systems are not busy

Schedulers are used to manage the timing of incoming and outgoing configurations, ensuring that data is processed according to a customized schedule.

Schedulers have a few types:

  • Run an Incoming Configuration
  • This is the most common type of scheduler. It runs an Incoming Configuration, which will create a Task.
  • Run an Outgoing Configuration of a Route
  • This runs the tasks with the setting 'New' (or 'Retry') for that particular route.
  • Repair Stuck Tasks
  • This resets the status to a defined status (i.e. 'New') for tasks that are stuck in a certain status (i.e. 'Processing'). These have additional settings to define the criteria.
  • Prune Tasks
  • This deletes tasks that are older than a certain date, or when there are too many tasks, etc. These have additional settings to define the criteria.

Settings

  • Number of Tasks to Run: This is only for the Run an Outgoing Configuration of a Route. This is important to set: this decides how many (maximum) tasks to execute in that particular 'run' of the scheduler. By default this is on 'unlimited', but that might cause issues when there are a lot of tasks in the 'New' status. The particular amount depends on the particular route. When working with Stock where every stock-task takes 500ms, you can easily set this to 1.000, but if it takes 10s, you shouldn't set it to more than 100.
  • Priority: If the system is overloaded, this decides which scheduler to run first.
  • Cron Expression: This is the most important setting. This is the schedule that the scheduler will run on. You can use a cron expression to define the schedule. For example, a cron expression can be set to trigger a job every 5 minutes or at specific times of the day. See more info on cron expressions here.