Architecture
Cloud Connector is deployed from the Azure Marketplace as an Azure Managed Application: a locked resource group in your subscription that you can read, and that Datawareness can reach for support and maintenance. Your data is loaded, stored and exported inside that deployment; it does not pass through a shared Datawareness service.
At a glance
Your Azure subscription — managed resource group
┌───────────────────────────────────────────────────────────────────────────────┐
│ │
│ Browser ──► UI (React) ──► Backend API (Node.js) ◄──► Service (Python) │
│ │ ▲ │ │
│ │ │ │ scheduler, │
│ configuration │ │ triggers, webhooks │ jobs, loads, │
│ and loaded data│ │ Data API │ exports │
│ ▼ │ │ │
│ Azure DocumentDB (MongoDB vCore) │ │
│ per environment: configuration + data │ │
│ │ │
│ Storage account (export staging, AI import payloads) │ │
│ Virtual network, private endpoints, optional NAT Gateway ──┼── outbound IP │
└──────────────────────────────────────────────────────────────┼────────────────┘
│
┌──────────────────────────┬─────────────────────┬──────┴───────────────┐
▼ ▼ ▼ ▼
Cloud REST APIs Business Central SharePoint Online SQL Server / Fabric
(JSON, XML, CSV, Excel) (BI Endpoints app) (Microsoft Graph) Lakehouse, Power BI
Your network: Cloud Connector Gateway ── outbound HTTPS only ──► Backend API
(SQL Server, ODBC)
What the deployment contains
| Resource | Purpose |
|---|---|
| Container Apps | The UI, the backend API and the Python service. |
| Azure DocumentDB (MongoDB vCore) cluster | Configuration and loaded data. Every environment has a configuration database and a loaded-data database of its own. On Enterprise you can bring your own MongoDB instead. |
| Storage account | Staging for bulk loads into Azure SQL Database, and payloads for AI-assisted endpoint import. |
| Virtual network with private endpoints | Keeps the database and storage off the public internet. |
| NAT Gateway (optional) | A single, fixed outbound IP for sources and destinations that use IP allow-lists. See Allow-list Cloud Connector's outbound IP. |
| Managed identity | Sends notification email through Microsoft 365 and reads Key Vault secrets, with no stored password. |
| Azure OpenAI deployment (Standard and Enterprise) | Powers the AI features. Provisioned by Datawareness during activation. |
On the Enterprise advanced management plan you bring the database, storage account, Key Vault, managed identity, Azure OpenAI resource and subnet yourself, and the deployment creates only the Container Apps inside your subnet. See Plans.
The components
- UI (React) — the web interface. Users sign in with Microsoft Entra ID through the Cloud Connector app registration; no passwords are stored.
- Backend API (Node.js) — the REST API the UI talks to and the keeper of all configuration. It also receives trigger calls, inbound webhooks and Gateway pushes, and serves the Data API.
- Service (Python) — the scheduler and worker. Runs cron-driven schedules, executes jobs, calls source endpoints and writes to export systems.
- Cloud Connector Gateway (optional) — a Windows service in your network that discovers tables and pushes rows from SQL Server and ODBC sources over outbound HTTPS. See Install and connect the Gateway.
- BI Endpoints (optional) — the Datawareness app in Business Central whose catalogue Cloud Connector reads. See Business Central.
How a load flows
- A schedule fires on its cron expression, or someone presses Run now, or an external system calls a trigger URL.
- The job runs its three phases in order: Before refresh data, Refresh Source Data and After refresh data.
- In Refresh Source Data, every dataset refresh is queued per enabled instance. The service calls the source endpoints, follows paging, and writes each row by its business key, so a row seen again is updated, not duplicated. A Gateway-fed dataset is read by the Gateway instead and pushed in batches.
- After a successful load the rows are exported to the dataset's destinations: merged when few rows changed, rebuilt after a full load. An export failure never fails the load; the next export catches up.
- After refresh data runs stored procedures and HTTP actions, typically transformations and a Power BI refresh.
- Every load and run is logged, and failures send email to the configured recipients. See Monitoring.
The details of full, incremental and delete loads are in How a load works.
Who calls what
| Caller | Target | Authenticates with |
|---|---|---|
| User in the browser | UI and backend API | Microsoft Entra ID sign-in through the Cloud Connector app registration |
| Backend API ↔ Service | Each other | Client credentials for the Cloud Connector and Cloud Connector Service app registrations |
| External system triggering loads, jobs or schedules | Backend API | The environment's API key in X-API-Key, or a Cloud Connector sign-in token. See API keys. |
| External system pushing a webhook | Backend API | The environment key or a webhook key. See Inbound webhooks. |
| Cloud Connector Gateway | Backend API | An agent key bound to one on-prem instance |
| Power BI, Excel, scripts | Data API | Tokens for the Cloud Connector Data app registration, plus a Data access role. See Data API. |
| Cloud Connector | Microsoft 365 mail, Key Vault | The deployment's managed identity |
| Cloud Connector | Sources and destinations | What you configure on each instance and export system, ideally through encrypted or Key Vault variables |
The three app registrations are described step by step in App registrations.
Network
- Sources and destinations are called from the deployment's outbound IP. With a NAT Gateway the IP is stable; without one it comes from a shared Azure pool and may change. Setup → Access shows the IP and a ready-made Azure SQL firewall script.
- The Gateway only makes outbound HTTPS calls on port 443. No inbound port, VPN or jump host is needed.
- An on-premises SQL Server destination is reached through an Azure Relay tunnel that Datawareness sets up during onboarding.