Skip to content

Incoming

An incoming configuration in Alumio is a setup that defines how data should be retrieved from an external source. It specifies the connectivity details, such as the source type (e.g., REST API, database, FTP), the request method (e.g., GET, POST), and the endpoint URL.

To create an Incoming Configuration, you go to Connections -> Incoming and add a new Incoming.

A subscriber, on the other hand, is a technical component of the Incoming configuration that performs the communication with the external source. It is responsible for retrieving the data specified. The subscriber can be configured to handle different types of sources, such as HTTP subscribers, SOAP subscribers, database subscribers, and more.

The configuration options for an incoming differ wildly based on the particular Subscriber chosen. An Incoming configuration always has the option to add Entity Transformers. These are executed on the data provided by the Subscriber. Note that the Entity Transformers might be executed more than once: sometimes a Subscriber will 'branch' to items within the dataset, meaning that Entity Transformers run on each item in the dataset. This will also happen when the Subscriber paginates (i.e. when an API has a limit of, for example, 500 items per page). You can also make this happen within the Entity Transformers, by using the Get Branches transformer.

Some of the available subscribers are:

  1. HTTP Subscriber: Used for connecting to a REST API and can be configured with request parameters such as URI, parameters, body, headers, etc.

The HTTP Subscriber also supports Input Transformers. This is a place to put Entity Transformers that run before executing the Subscriber. This is used, for example, to retrieve a timestamp from a storage to use for the actual call the Subscriber makes.

The HTTP Subscriber supports pagination. This means that when certain conditions apply, it will automatically call the next 'page' of items from the API.

The HTTP Subscriber also supports branching to items itself, by utilizing the Decoder functionality. This is generally used to optimize memory usage, since running all subsequent Entity Transformer on just one item is generally better for memory usage than running it on the entire dataset.

  1. SOAP Subscriber: Used for connecting to a SOAP API and requires specifying the SOAP method and request data.

The SOAP Subscriber supports most of the features the HTTP Subscriber supports.

  1. Database Subscriber: Used for retrieving data from a database source and can be configured with predefined forms or raw queries.

  2. Filesystem: Listing Subscriber: Used for retrieving a list of files from a filesystem.

The 'entities' retrieved in this case will be the metadata of a file. Note that it will foreach through the files, you will not get a list.

  1. Filesystem: File Subscriber: Used for reading individual files from a filesystem.

The data received depends on how you decode/deserialize the file retrieved. Many options are available, among them JSON, XML & CSV. It also support decompressing ZIP files beforehand.

  1. Storage Subscriber: retrieves Storage Entities from the specified storage.

The data received is a single entity from the specified storage. This is often used to create 'queues' within Alumio, where several processes fill the queue and a single incoming actually starts the sync process.