Jobs

A job is a pipeline in three fixed phases, run in order:

1. Before refresh data — actions and stored procedures to run first, for example scaling a database up.
2. Refresh Source Data — dataset refreshes.
3. After refresh data — transformations and report refreshes.

Each phase holds activities, and an activity holds items: actions, stored procedures, or dataset refreshes. Items of an activity run in parallel by default; activities run one after another.

Jobs in the sidebar lists jobs with Name, Last run outcome, Last run completed, Enabled and actions: Run now (Stop run while it runs), Lineage view, View log, Duplicate (a new job prefilled from this one) and Copy API trigger URL (Standard and Enterprise). Rows are tinted by the last outcome; the tint disappears once the job is edited after that run.

The job form

• Name (unique), Description, Enabled.
• Time limit (hours) — how long the job may run in total before it is stopped, default 4. It counts every step together, so raise it for a big first full load. A schedule's own limit is the largest limit among its jobs.
• An existing job opens View only; the pencil switches to edit mode. Save keeps the editor open; Close leaves it. A job edited while it runs applies to the next run.
• The footer walks the phases with Next: …, ending in Next: Schedule, which offers Create new schedule with this job or Add to existing schedule.

Activities. Add activity (or Add dataset refresh in the middle phase) adds an activity with a name and a gear, Activity settings:

Setting	Default	Meaning
Run items in parallel	on	Items run concurrently up to Max parallel; off, they run one after another in list order
Max parallel	inherited from Setup → Runtime → Default max parallel at save time	How many items run at once. Set one for a rate-limited API or a heavy load
Continue on error	off	A failed item is logged, the activity's remaining items still run, and later activities and phases are not stopped
Run after error	off	Off, the activity is skipped once an earlier activity has failed. On, it runs regardless: cleanup such as scaling a database back down

Items run in the order they were added and cannot be reordered.

Items. In the first and last phase, Add item offers:

• Action — pick an action, give the call a Label for the log, and fill the action's Parameters, which may reference variables and job variables. A stored-procedure action can Run against another export system than its default.
• Stored procedure — an inline EXEC against a SQL Server export system, with its parameters, and no outputs. Offered only when the environment has a SQL Server export system.

In Refresh Source Data, an item is a dataset refresh: Source System, Instance (or all), Dataset (or all), Load Type (Full or Incremental), and Sync to, where unticking a destination skips it every time this job runs, for example while reworking a dataset. The item shows how many loads it will enqueue, and notes when it will be skipped because something it points at is disabled.

Notifications. Failure email recipients receive an email when the job ends in failure, standalone or inside a schedule, on top of the global recipients from Settings. Webhook subscriptions (Enterprise) POST the job's Start, End — Success, End — Failure and Still running events to a URL; see Outbound webhook subscriptions.

Failure handling and outcomes

Without Continue on error, the first failed item fails the activity and the run ends failed; later activities are skipped unless they have Run after error. With it, the job carries on and ends completed with failures, shown as a warning, and the failure recipients get a warning email listing the steps that did not complete. The action that supplies a job variable is exempt from continue-on-error: if it fails, the run fails.

A job never runs twice at once: a second start queues until the first run has finished. A run that passes the Long-running job warning threshold (see The emails Cloud Connector sends) sends an email and carries on. A run that hits its time limit is stopped with a message naming the limit.

Job variables

Job variables are run-scoped values shared by every action in the job, referenced as {{j:name}} in any action's URL, headers, body, command or parameter value. The panel under the phases lists them with their Kind (Single or List), Value from and Run on preview. Add job variable opens the dialog:

• A value I type — fixed for every run. Press Enter after a value to turn the field into a list.
• An action's output — the value is one of an action's captured outputs (see HTTP request). The producer either runs once before the job starts (Action + Output), so every activity can use it, or is an action already in this job, in which case only activities after it can read the value. A list output can be narrowed with a filter of allowed values, chosen from a live Preview of the action.
• Run on preview — lets the lineage view's What runs next execute this action to work out the plan. Leave it off for anything with a side effect; the preview then shows the items that use it as written.

A list job variable referenced as the whole value of an action parameter ({"id": "{{j:ids}}"}) fans the action out into one invocation per element, up to 1000, within the activity's max parallel; a reference inside a longer string sends the list as text and runs once. Two variables that read the same output are two independent selections. The form warns when a reference comes before its producer or names an undeclared variable, and a job's run log names each fanned-out invocation by its value.

The system values {{j:system:jobName}} and {{j:system:activityName}} are always available.

Run a job

Run now opens a dialog: From the beginning, or Start at a phase with a Phase and optionally an Activity. Earlier steps do not run; everything from there to the end does. Job variables are still produced, but values the skipped actions would have captured are not available. The run's log gets a note row naming what was skipped. The same dialog is used from the lineage view, so every manual run means the same thing.

Stop run stops the run; steps in progress are stopped, and work already handed off (a stored procedure, a dataset load already executing) may finish in the background. Stopping a job that runs inside a schedule stops the whole schedule run.

Lineage view

Lineage view draws the job as phases, activities and items, with a status dot per node from the last run (hover for status, date and duration). The job form shows a small copy of it; clicking a node there jumps to it in the form. Drag to pan, use the controls to zoom.

What runs next switches from the job as written to a plan of what the next run would do: fan-out items become one pill per value, and items whose inputs are missing read "Would not run" or "Would fail". The plan runs the producer actions whose job variables have Run on preview on; everything else is shown as written.