Data catalogue
The Data catalogue is the section of an expanded source system that lists what can still become a dataset. It has one layout for every kind of source: a search box, then a list with a Preview or Assign per row. What it lists depends on the source system:
| Source system | What the catalogue lists | How a row becomes a dataset |
|---|---|---|
| Cloud (REST API) | Endpoints not assigned to a dataset | Preview → pick unpack and key → Create dataset |
| On-premises | Tables the Gateway discovered and not yet assigned, per instance | Discover (columns), then Assign |
| Business Central with BI Endpoints | The entity sets the app publishes that are not datasets yet | Assign, or click in the data model |
| SharePoint Online | Sites, libraries, workbooks, sheets, tables and files | Preview → Create dataset |
For cloud systems the section is collapsed by default and remembers whether you opened it. A Business Central or SharePoint catalogue is read from the source only when the section is open.
Cloud endpoints
The list shows Method, Value (with the description underneath), Load Type and actions Preview, Edit and Delete, with tick boxes for Delete selected. The header has Import endpoints and Add Endpoint (see Unassigned endpoints and Import endpoints).
Preview → Create dataset opens Assign 1 endpoint to dataset:
- Create new dataset (one endpoint at a time) with Dataset name
(proposed from the path, letters, digits and
_kept, case unchanged, at most 50 characters), Description (prefilled from the endpoint's documentation), Business key (the picked fields joined with&&), Unpack, and Add request URL on each row (_url). - Assign to existing dataset — the endpoint joins a dataset that already exists, for example as the incremental sibling of a full endpoint. The picker lists the first 100 datasets of the source system.
- Run now queues a full load of a new dataset, or an incremental load of an existing one.
The assigned endpoint leaves the catalogue and appears under its dataset.
On-prem tables
The on-prem catalogue is always open and has one tab per instance, because tables are discovered and bound per instance (the same table on several instances can feed the same dataset). Discover catalogue asks the Gateway to read the database's tables; they appear within seconds as the Gateway walks the source, by name only.
Each row shows Schema.Table, Columns, Key (discovered or overridden) and Change detection (Change Tracking, watermark (column) · N windows, or snapshot). A new table has no columns yet: Discover on the row fetches them (the Gateway polls about every 30 seconds), after which the button becomes Assign.
Assign 1 table to dataset names the dataset after the table and shows the source's primary key as the business key, read-only: Change Tracking delete events carry only key values, so the key cannot be changed here. A table with no primary key asks you to pick Business key column(s) instead.
The pencil, Manage source table integration, sets how the Gateway reads the table:
- Watermark column — the column an incremental sync resumes from, one that changes whenever the row does. Use discovered keeps the Gateway's guess; Snapshot only reloads the whole table each time. A table with Change Tracking uses that instead.
- Read in windows (ODBC sources, no Change Tracking) — one query per range of the watermark column, Number of windows between 2 and 1000, progress saved after each. A failed load continues where it stopped. For very large tables; the column needs an index.
- Use discovered key, or Key columns, in order — the order is part of the composed business key, and a bound dataset's key follows a change.
Changes take effect on the Gateway's next sync. After assigning, the table is listed under the dataset's Source data section, where Remove sends it back to the catalogue and stops the Gateway loading it; data already loaded is kept. See On-Prem Gateway for the Gateway itself.