Skip to main content

Endpoints and pagination

An endpoint is one request a dataset makes: a method, a path appended to the instance's base URL, optionally a body, and the rules for paging through the response. Endpoints belong to the source system and are shared by all of its instances; what differs per instance (base URL, credentials, headers, parameters) lives on the instance.

An endpoint is either assigned to a dataset, listed in the dataset's expanded Endpoints section, or unassigned, listed in the source system's Data catalogue until a preview turns it into a dataset (see Data Catalogue).

Create or edit an endpoint​

In the dataset's Endpoints section, Add Endpoint opens the endpoint dialog; each row has Preview, Copy (to add an incremental sibling to a full endpoint: the copy defaults to the opposite load type) and Edit. The dialog has the tabs Endpoint and Configurations.

Endpoint tab​

  • Method — GET or POST.
  • Load Type — Full, Incremental or Delete. A full load reads the full endpoints; an incremental load runs the delete endpoints first and then the incremental ones. A delete endpoint should return records carrying the dataset's business-key columns; matching stored rows are marked deleted. See How a load works for what happens when a dataset has only one kind.
  • URL path — appended to the instance's base URL, up to 8000 characters. Use {{variableName}} for your variables and {{system:today-1}} or {{system:lastLoadedAt}} for system variables; typing {{ opens the picker. A path starting with ? or & is appended as-is.
  • Body (POST) — Raw / JSON or Form data, merged over the instance's body.
  • Description — carried to the dataset when the endpoint is assigned.
  • Output format (advanced) — override the source system's response format: JSON, XML, CSV (delimiter, column names) or Excel (.xlsx) with Sheet (a name or a 0-based index; empty for the first sheet), Table (a named Excel table, read instead of the whole sheet) and Excel Column Names.
  • Response mapping (advanced) — Unpack path (override) and Business key mapping (override) for this endpoint alone, when its response is shaped differently from the dataset's other endpoints. An override must have the same number and order of && parts as the dataset's key, or deletes will not match the stored rows.

Save & Preview saves and opens the preview; Delete removes the endpoint without further confirmation.

Configurations tab​

  • Pagination (NextLink) — see Pagination.
  • Column filter (load only) — keep only the listed fields from each response object. Empty means load everything.
  • Parameters — query parameters sent on this endpoint, overriding the instance's same-key parameter. An empty value deletes the row.
  • Headers — headers sent on this endpoint, overriding the instance's same-name header. A bearer, session-bearer or Microsoft token always sets Authorization, whatever a header row says.

Parameters and headers can be added once the endpoint is saved.

Unassigned endpoints​

Data catalogue → Add Endpoint on a cloud source system creates an endpoint that is not yet a dataset: method, Load Type (Full or Incremental), URL path (500 characters here), body and description. Paging, filters, parameters and headers become available once it is assigned. The lastLoadedAt system variables need a dataset and cannot be used in an unassigned endpoint's path.

Preview an endpoint​

Preview makes one live request to the source, with no paging, and stores nothing. An integer-loop variable is filled with its start value; {{system:lastLoadedAt}} resolves as it would for a real load of the chosen instance.

  • URL / path can be edited in place: Run with edits tries the edit once, Save & Run stores it.
  • Instance appears when the source system has more than one; with exactly one the preview runs at once.
  • The result toolbar shows the HTTP status, a record count, an N headers chip that reveals the response headers, the request URL, and for a dataset that renames fields a columns as stored chip.
  • Picker shows the response as a tree. Click the target icon next to the array of records to choose what to unpack, then click the field(s) that identify a record to pick the business key; a field on a parent object can be picked too and is copied onto each record. Raw shows the JSON with search.
  • From the catalogue, Create dataset is enabled once an unpack path and at least one key field are picked. From a dataset's own endpoint, Edit mapping saves a new unpack path and key to the dataset, and the Map columns warning saves an override on the endpoint.
  • Suggest business key asks the AI to propose a key from a sample of rows and checks it for duplicates. It needs the Standard or Enterprise plan and AI configured on the deployment.

Pagination​

Look In on the endpoint's Configurations tab, or on the instance for a default every endpoint inherits:

  • None (no pagination) — on an endpoint this means inherit the instance's setting; on the instance it means no paging.
  • Body — the next page's URL or token is a field in the response: Nested path (empty when the field is at the root, && between segments) and Field (for example @odata.nextLink or nextPage).
  • Header — the next page's URL or token is in a response header (Field, for example Link), optionally extracted with a Regex Pattern whose first group is the value.
  • Cursor (request body) — for GraphQL-style paging: read the cursor from the response (Nested path (response), Cursor field, optional Has-more field) and write it into the request body for the next page (Write to (request body), for example variables&&after). Needs a raw JSON POST body. Paging stops when the has-more field is false, the cursor is missing, or it repeats.

For APIs that return a token rather than a URL, the URL template turns it into the next request: <value> is replaced by the token, a template starting with ? replaces the query string (repeat any fixed parameters in it), one starting with & appends to it. A next link that is an absolute URL is used as-is; ?… replaces the current query; &… appends to it; anything else is appended to the base URL. Paging stops when the next URL equals the current one.

Parameters are sent on the first page only. After that the loader follows the next link exactly as the server gave it, so a fixed parameter the API needs on every page (an API key, a page size) belongs in the URL path or in the URL template rather than in a Parameters row.

Page-numbered APIs use an Integer Loop variable instead; see Integer loop.

Import endpoints​

Data catalogue → Import endpoints creates unassigned endpoints in bulk:

  • AI-assisted (URL, JSON, Excel, or any text) — upload a file or paste a URL, JSON, or any text describing the API, and press Extract with AI. The import runs in the background and creates GET endpoints, skipping duplicates; placeholders like {id} become {{id}}, which then needs a variable of that name. Needs the Standard or Enterprise plan and AI configured on the deployment.
  • Postman collection (no AI) — upload or paste a Postman v2.1 collection and tick the requests to create. GET requests are ticked by default, POST requests are not. Only the method, path and raw body are kept; headers and parameters from the collection are not.

Where variables resolve​

The URL path resolves every kind of variable: static, Key Vault, global, date, system, data reference (one request per value) and integer loop. Every other field (base URL, credentials, headers, parameters, bodies, action fields) resolves static, Key Vault, global and date variables and {{system:today}} / {{system:now}}. {{system:lastLoadedAt}} is resolved in the URL path only; put an incremental filter in the path, for example ?$filter=modifiedAt ge {{system:lastLoadedAt}}. Integer-loop tokens also work in query parameters (page={{loop}}).