Skip to main content

Export systems

An export system is a destination for loaded data. After each successful load, the dataset's rows are written there under <schema>.<dataset>. Export failures are non-fatal: a failed export never prevents the load from completing, and the next export catches up.

Kinds​

  • MS SQL Server — Azure SQL Database, SQL Managed Instance, or SQL Server on a VM or on-premises. An on-premises server is reached through an Azure Relay tunnel that Datawareness sets up during onboarding; the export system then points at 127.0.0.1,<relay port>.
  • Fabric Lakehouse (Delta) — a Microsoft Fabric lakehouse, written with a service principal. Standard and Enterprise.

Export Systems in the sidebar (page title Export to) lists them with Name, Type, Server, Database, Auth, Status (the enabled switch) and actions.

Create or edit an export system​

  • Name — unique, at most 100 characters. Type. Enabled.
  • SQL Server: Server (host, or host,port), Database, and Trust server certificate for a server on an internal VM or on-premises host with a self-signed certificate (encryption stays on; only the certificate check is skipped; leave it off for Azure SQL). Auth Type: SQL Authentication (username + password) or App Registration (service principal) with Tenant ID, Client ID and Client Secret.
  • Fabric: Workspace ID, Lakehouse ID, Schemas enabled (on for lakehouses created since 2024, where tables land under /Tables/<schema>/<table>/; off for the older kind, which lands them as <schema>_<table>; it must match how the lakehouse was created), and the service principal fields.

Every field accepts {{variables}}, and that is where a password or secret belongs: a value typed straight into these fields is stored as typed. Reference an encrypted or Key Vault variable instead; it also lets one export system definition be promoted between environments with different credentials.

Destination database setup. When creating a SQL Server destination the dialog asks you to prepare the database one of two ways and tick that you have:

  • Option A — Server-level Entra admin via group: make an Entra group the server's Microsoft Entra admin and put your admin and the export's service principal in it. This enables the bulk-load path, roughly ten to fifty times faster on large exports.
  • Option B — db_owner only: create the login in the database and add it to db_owner. Slower, but no server-level change.

Test Connection runs a probe (a SELECT 1, or a probe table on a lakehouse). In create mode the export system is saved on the first test so that variables resolve, and Create is enabled only after a successful test.

Deleting an export system removes it from every source system and dataset. A dataset left with no destinations of its own is switched to Never export.

Configuration: include deleted records​

The Configuration tab (edit mode) holds Include deleted records, the destination's default for what happens when a record is deleted at the source. See Deleted records.

Choosing where a dataset exports to​

Export systems are linked on the source system (Export tab, Export to (optional)), and every dataset under it inherits that list. The dataset's Export data tab overrides it:

  • Inherit from Source System — the default.
  • Export to specific systems — always export, to the destinations picked here. The list replaces the inherited one, so to send a dataset to the inherited destination and another one, tick both; the picker is prefilled with the inherited list.
  • Never export — skip this dataset even when the source system exports.

The Load Data dialog's Sync to boxes skip a destination on one run only, and the Refresh Source Data item of a job can exclude destinations every time that job runs.

Deleted records​

When a record is deleted at the source, the destination can do one of two things. The default is set per destination (Include deleted records on the export system) and overridden per source system and per dataset on their Export tabs (Deleted records):

  • Remove from destination (the default) — the row is deleted from the destination table. The table always matches the source, but nothing downstream can learn the row existed without reloading everything and comparing keys.
  • Keep and flag — the row stays, with a _deleted column set to 1, and its _updatedAt moves when it is deleted, so a downstream incremental load picks the deletion up on the same watermark it already uses for changes.

Choose Keep and flag for anything feeding an incremental warehouse load; choose Remove from destination when the table just needs to mirror the source.

Two readings of one table. On SQL Server, Keep and flag splits the destination into two objects:

ObjectHoldsUse it for
<schema>.<dataset>_allevery row, live and deleted, with _deletedan incremental warehouse load that needs the deletions
<schema>.<dataset>live rows only (a view)reports, semantic models, ad-hoc queries

The dataset's own name is the view, so SELECT * FROM <schema>.<dataset> stays correct without a WHERE clause, and reading the deletions means naming the _all table deliberately. Both follow the table when a column is added or the dataset is renamed. A Fabric Lakehouse gets the _deleted column but no view.

Two things worth knowing:

  • Flagged rows are never removed from the destination, so _all grows over time. The retention setting under Setup → Runtime purges them from Cloud Connector's own storage only, and a deletion older than that retention never reaches a destination that has not exported in the meantime.
  • A destination that kept deleted records under the plain table name before the view existed is left as it is, because a warehouse load may be reading those flagged rows from that name. Tables set up before current-state views on the export system's Configuration tab reports which tables would move (Check for tables to convert) and converts them (Convert N table(s)) once you have repointed anything reading them at _all.

What the SQL export writes​

Table and columns. The table is [<schema>].[<dataset>]; the schema is created when missing, and a rename of the dataset or the schema moves the table. Every stored field becomes a column, plus:

ColumnWhenType
_idthe dataset has a business keyNVARCHAR(450), clustered primary key: <instance>-<key>, or the bare key on a single-instance source system
_instancealwaysNVARCHAR(450), the instance name
_updatedAtalwaysDATETIME2, UTC
_deleteddeleted records are keptTINYINT 1/0
_urlthe _url option is onthe request URL
_parentIdnested datasetsNVARCHAR(450)

A source field whose name differs from _id or _deleted only in case is written as _id_src or _deleted_src.

Types are inferred from the first rows that carry a column: booleans become BIT; integers INT or BIGINT; decimals NUMERIC(38,20); dates and ISO-8601 date strings DATETIME2; text NVARCHAR(n) sized with room to grow, or NVARCHAR(MAX) for long or mixed values. A later row that does not fit widens the column in place (NVARCHAR to a larger size, INT to BIGINT, integer to NUMERIC); columns are never narrowed. A text that is not a number in a numeric column, or not a date in a date column, is written as NULL.

Nested objects and lists are written as JSON text in one column; query them with OPENJSON or JSON_VALUE, or turn them into tables of their own with Nested Datasets.

Merge or rebuild. An export after an incremental load merges the rows changed since the destination's last successful export. The export rebuilds the whole table on the first export to a destination, when the destination table is missing or empty, after a full load, and when more rows changed than a merge can handle. Without progressive loads, a rebuild loads a shadow table and swaps it in at the end, so the live table stays readable until the swap; a duplicate business key aborts the swap. With progressive loads, the rows are merged into the live table chunk by chunk and a failed export resumes. Keep anything you add to a destination table yourself (extra indexes, grants) in a script you can re-run after a rebuild.

Speed. On Azure SQL Database with the server-level Entra admin group (Option A), large exports go through bulk loading from the deployment's storage account; otherwise rows are inserted in batches. A failed chunk is retried once on a fresh connection; a failed export ends the load as warning and the next export catches up from the last success.

The Dataset log shows each export as its own export-sync row under the load, one per destination; a dataset with several instances exports once, after the last of them.

Stored procedures​

A SQL Server export system is also where a job's Stored procedure items and SQL stored procedure actions run. A Fabric Lakehouse cannot run procedures. See Actions and Jobs.