Tasks

In Alumio's Incoming-Route-Outgoing system, Tasks are the heart. The Incoming-Route combination delivers a dataset (i.e. a single product in JSON) and saves this to Alumio's database as a 'new' Task. When the Task is picked up by a scheduled process, it will go through the Outgoing configuration (defined in the Route) before reaching the state 'Finished'.

This way there is transparency in the actions that the connections do, and what data the integrations have to work with. Also, you can track the progress of a single entity (i.e. the product) and see whether it has been queued up ('New'), is currently being processed ('Processing'), has failed ('Failed'), has been filtered out ('Skipped') or has succeeded ('Finished').

The Task itself is also used to retrieve all logs connected to that entity. On the Import Logs side you can see the logs that led to the creation of this task while it went through the Incoming & Route. On the Export Logs side, you can see what happened when the Task was processed by the Outgoing configuration.

Tasks have two user-defined attributes: the Entity Data (created by the Incoming & Route, which is the full dataset of the entity) and the Entity Identifier. The Entity Identifier is used to identify and search for the Task in the Task Log/Queue. You can define the Entity Identifier in the Incoming or Route by setting an Entity Schema - then, within the Entity Schema, you can define the path(s) to the identifier.

Task Log/Queue

The Task Log/Queue allows you to search the Tasks, retry or skip certain tasks and, naturally, look at the details of a single Task.

Amount of Tasks

Note that there is a maximum amount of Tasks that Alumio will store. The exact maximum depends on the environment (it's pricing/plan based), but is generally in the range of 100.000-500.000. We also generally have 'prune' schedulers running which delete Tasks when they are older than x-date. Large amounts of Tasks have an impact on the performance of the system.

Large Tasks

Be careful in creating Tasks that are too large. Although it is always preferable to use the Tasks in the exact way that they are meant (including being retry-able), there are cases where the Task would have a very large 'body' (>10kb). Because Tasks are saved into a MySQL database, it's not great for the system to have very large Tasks. In those cases, it might be best to store the data into a Storage, then just keeping an identifier within the Task and getting the stored data from the Storage inside the Outgoing configuration.