Skip to main content

Source systems

A source system groups everything about one data source: its schema (the namespace loaded data lands in), its instances, its endpoints and its datasets, and where they export to.

The Source Systems page​

The page lists source systems with Source System, Schema, Status (the enabled switch) and actions. When more than one schema exists, tabs All and one per schema filter the list. The header has Import configuration package and New Source System (disabled at the plan's source-system cap, with the tooltip saying how many are used).

Row actions:

  • Refresh now — refresh every enabled dataset across every enabled instance. Disabled while the system is disabled, during an instance rename, or when it has no enabled instance or dataset.
  • Copy API trigger URL — the API call that does the same.
  • Edit.
  • On a Business Central source system: Data model and Custom endpoints in Business Central (see Business Central).

Clicking a row expands it into three sections: Instances, Datasets and Data catalogue. The first two are covered in Source System Instances and Datasets; the catalogue in Data Catalogue.

The Refresh now prompt offers Incremental or Full. See How a load works for what each does.

Create or edit a source system​

New Source System shows the fields below on one page; Edit Source System splits them over the tabs Edit source system, Configurations and Export.

Edit source system tab​

  • Source System — the name, unique, at most 50 characters.
  • Schema — the namespace loaded data lands in: the Mongo collection is <schema>.<dataset> and the SQL table is [<schema>].[<dataset>]. Pick an existing schema from the suggestions or type a new one. Several source systems may share a schema; dataset names must then be unique across them. On edit the field is locked behind a padlock, because changing it renames every loaded collection under the schema and moves the exported tables.
  • System kind (create only) — Cloud (HTTPS API), loaded by the Cloud Connector service, or On-premises (Gateway push), pushed by a Gateway installed on the customer's Windows host. The kind cannot be changed afterwards.
  • Response format (cloud) — JSON (default), XML, CSV (with CSV Delimiter and, for files without a header row, CSV Column Names) or Excel (.xlsx). It is the default for every endpoint under the system; an endpoint can override it. For Excel, the sheet and column names are set per endpoint.

There is no description or enabled field in the dialog. A new source system starts disabled; switch it on with the Status switch on the list page.

Configurations tab​

  • Load concurrency — the switch Run loads one at a time per instance is for rate-limited APIs that answer 429. It holds each instance to a single load at a time, overriding a job's parallel setting, and caps the requests within each load at 45 per minute. Leave it off unless you see 429 errors; it makes loads slower. Independently of the switch, every request that gets a 429 or 503 is retried, honouring Retry-After.
  • Single instance — a single-instance source system is one you will only ever connect to once: one company, one tenant, one database. With Single-instance system on, record ids are stored as the plain business key (1 instead of <instance>-1), exactly one instance is allowed, and the setting locks once a dataset has loaded. Decide before the first load: switching it later does not re-key rows that are already loaded.
  • Color — a soft background tint that groups the system's rows on the Datasets page and tints its row on the Source Systems page.

Export tab​

  • Export to (optional) — the export systems every dataset of this source system writes to after each successful load. Linking a destination is what turns export on; a dataset can override the list or opt out. See Choosing where a dataset exports to.
  • Deleted records — Inherit from destination, Keep and flag or Remove from destination, for every dataset of the system unless a dataset overrides it. See Deleted records.
  • Progressive loads — with the switch on, full loads write rows to the SQL destination as they arrive instead of replacing the table when the load completes, and a failed load keeps what it wrote. Also for incremental loads extends that to incremental loads, for large change sets; a dataset with a delete feed exports the normal way regardless. New source systems start with the switch on. Only SQL Server destinations are affected; a Fabric Lakehouse always merges in place.

Delete a source system​

At the bottom of Edit source system the Danger zone shows what the deletion covers: the instances, datasets, endpoints and credential rows, the estimated number of loaded records, and any jobs that reference the system and would be left with a step pointing at nothing. Tick I understand this permanently deletes the configuration and all loaded data, press Delete source system and all its data, and confirm.

The deletion drops every loaded collection first; if one cannot be dropped, nothing is deleted. It then removes credentials, headers, parameters, load logs, endpoints, datasets, instances, the system's entries in Data access roles, and the system itself. Catalogue tables of an on-prem system are unbound, not deleted. Exported tables in your warehouse are left in place, and jobs are not edited.