Headless agent

The Fabric Desktop by KratosBI agent runs your metadata project on a server, unattended. It runs on Windows or Linux, signs in as a service principal or managed identity, and writes to OneLake exactly as the desktop app does. Develop with Fabric Desktop by KratosBI. Execute on your infrastructure. Store and consume in Microsoft Fabric.

The agent is the first milestone (M1) of the enterprise local execution roadmap. It runs one machine's worth of work. Several agents sharing one queue comes in M3.

Status: built and tested against local Delta tables on Linux and Windows CI. It has not yet run against a live Fabric tenant. Before you trust it with Prod, do the M0 checks in Verify on your tenant.

What it does#

schedules in agent.toml ─┐
fabric-desktop agent enqueue ─┼─> durable queue (SQLite) ─> worker ─> metadata engine ─> OneLake
                          │        retries, crash recovery     │        (the same code the app runs)
                          │                                    └─> a write lease per table (LH_Meta)
                          └─ JSON logs, run_log rows with worker, host, attempt and job id

• Same engine as the app. A job runs an environment's entities through fabric_desktop.meta.Engine, in dependency order, with the same checks, quarantine, SCD, surrogate keys and run log. Runs show in the app's Monitor as Agent <name> on <host>, with the job id and try in their details.
• No one at the keyboard. The agent signs in as an app identity. Nothing ever opens a sign-in window.
• Durable queue. Jobs are kept in queue.sqlite3 in the agent's home folder. A job is queued, then running, then succeeded, failed or dead. A running job holds a lease that its worker renews; if the worker dies (crash, reboot, killed service), the lease runs out and the job is put back in the queue on the next start.
• Retries that make sense. Network errors, throttling (HTTP 429 and 5xx), a lost Delta commit race and a table busy with another writer are retried, waiting backoff_seconds, then twice that, up to max_backoff_seconds, for max_attempts tries in all. A retry runs only the tables that didn't finish. A job that still fails is dead and posts to Teams. A table that fails its checks is failed straight away, since running it again would fail the same way.
• Safe concurrent writes. The engine writes a table by reading it, merging new rows and committing the result. Before it starts an entity, the agent takes a write lease on that table: a small file in the environment's LH_Meta lakehouse under Files/_fabric_desktop/leases/, created with If-None-Match: *, renewed with If-Match, and deleted when done. A second writer waits for it (up to table_lease_seconds), then retries later. An abandoned lease expires after table_lease_seconds. For a local environment the lease files are in its .local folder, and taking one over is a single step across every process on the PC, so two writers never both win an expired lease. A worker that can't renew its lease for a whole table_lease_seconds (OneLake unreachable, say) treats it as lost and reports the run that way. Every write, from the agent, the desktop app or a run in Fabric, also commits against the Delta version it read: if anyone committed to the table since, Delta refuses the commit and the engine re-reads the table and merges again (up to 3 times), so no writer's rows are lost even without a lease.
• Schedules. [[schedule]] entries fire in the machine's local time: every N minutes, daily at set times, or weekly. A schedule that comes due while its last run is still queued or running doesn't add a second job. A schedule missed while the agent was down runs once when it starts.
• Fabric pipelines can ask for runs. With [pipeline] turned on, a pipeline writes a run request to the environment's LH_Meta and waits for the agent's answer next to it. See Run from a Fabric pipeline.
• Guard rails. The agent runs only the environments listed in agent.environments, writes only to the workspaces in agent.workspaces when you set it, and never creates lakehouses: an environment is set up by promoting to it from the app.
• Logs. One JSON object per line on stdout (journald, docker logs) and in <home>/logs/agent.log (20 MB, 10 files kept). Every line has the worker and host; job lines add job_id, env, attempt and source. Every run_log row the agent writes has worker, host, attempt and job_id.

Before you start#

You need:

1. A metadata project in a folder the agent can read, usually a Git checkout of the project you develop in the app. The agent reads layers.toml, environments.toml and the entity files from it, so check out the version you promoted.
2. Environments set up in Fabric. Promote to each environment from the app first. The agent checks that the environment's deployed metadata version matches the project, as the app does.
3. An identity for the agent. One of:
   • Service principal with a certificate (any server). Recommended for on-premises.
   • Managed identity (Azure VM, or an Arc-enabled server). Recommended on Azure: there is no secret on the machine.
   • user: the desktop sign-in cached for the account the agent runs as. For trying it out only; the refresh token expires.

Set up the identity#

1. Create an app registration for the agent (keep it separate from the desktop app's registration): Entra admin center > App registrations > New registration, no redirect URI.

2. Give it a certificate. For example, on Linux:

   openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=fabric-desktop-agent" \
     -keyout key.pem -out cert.pem
   cat key.pem cert.pem > /etc/fabric-desktop/agent.pem && chmod 600 /etc/fabric-desktop/agent.pem
   

   Upload cert.pem under Certificates & secrets > Certificates. A .pfx works too (certificate_path ending .pfx, with certificate_password_env naming the variable that holds its password). A client secret works (client_secret_env), but a certificate is safer.

3. Let service principals use Fabric. In the Fabric admin portal, Tenant settings > Developer settings > Service principals can use Fabric APIs: enable it for a security group, and add the agent's app (or managed identity) to that group.

4. Add the identity to each workspace it runs, as Contributor or higher: the environment's workspace (its layer lakehouses and LH_Meta) and, if the project uses surrogate keys, the Keys workspace (LH_Keys). It needs no Entra API permissions: an app identity gets what its workspace roles allow.

The full list, in alphabetical order, is in permissions.md.

Configure#

Copy deploy/agent.example.toml to:

Set FABRIC_DESKTOP_AGENT_CONFIG or pass --config to use another file; set FABRIC_DESKTOP_HOME or agent.home for another home.

Secrets never go in agent.toml, and the agent refuses to start if one does. A certificate is a path; a client secret, certificate password or Teams webhook address is the name of an environment variable (client_secret_env, certificate_password_env, teams_webhook_env).

Then check it. This validates the file and the project and gets a token for Fabric and for OneLake:

fabric-desktop agent check

Install#

Linux (systemd)#

sudo useradd --system --home /var/lib/fabric-desktop fabric-desktop
sudo python3.12 -m venv /opt/fabric-desktop/venv
sudo /opt/fabric-desktop/venv/bin/pip install ./sidecar          # from a checkout of this repository
sudo mkdir -p /etc/fabric-desktop && sudo cp deploy/agent.example.toml /etc/fabric-desktop/agent.toml
sudo chown root:fabric-desktop /etc/fabric-desktop/agent.toml /etc/fabric-desktop/agent.pem
sudo chmod 640 /etc/fabric-desktop/agent.toml /etc/fabric-desktop/agent.pem
sudo cp deploy/linux/fabric-desktop-agent.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now fabric-desktop-agent
journalctl -u fabric-desktop-agent -f

Variables named by *_env settings go in /etc/fabric-desktop/agent.env (NAME=value lines, mode 600). The unit file is deploy/linux/fabric-desktop-agent.service.

Linux (container)#

docker build -f deploy/linux/Dockerfile -t fabric-desktop-agent .
docker run -d --name fd-agent --restart unless-stopped --stop-timeout 900 \
  -v /etc/fabric-desktop:/etc/fabric-desktop:ro \
  -v /srv/fabric/sales-metadata:/srv/fabric/sales-metadata:ro \
  -v fd-agent-home:/var/lib/fabric-desktop \
  fabric-desktop-agent
docker exec fd-agent fabric-desktop agent status

See deploy/linux/Dockerfile.

Windows#

From an elevated PowerShell in a checkout of this repository, with Python 3.12 installed:

.\deploy\windows\install-agent.ps1 -Account 'CONTOSO\fd-agent$'
notepad C:\ProgramData\FabricDesktop\agent.toml
& "C:\Program Files\FabricDesktopAgent\venv\Scripts\fabric-desktop.exe" agent check
Start-ScheduledTask -TaskPath '\FabricDesktop\' -TaskName Agent

The script installs the agent into a virtual environment and registers the Task Scheduler task \FabricDesktop\Agent. The task starts at boot with no one signed in and restarts every minute if it stops. A group managed service account (gMSA) is best, since there is no password to manage. Set *_env variables as machine environment variables.

Operate#

fabric-desktop agent status                          # queue counts, next schedule times, the last 20 jobs
fabric-desktop agent enqueue prod                     # run every Prod entity now
fabric-desktop agent enqueue prod silver.orders gold.fact_sales
fabric-desktop agent enqueue prod silver.orders --cascade --backfill full   # rebuild it and everything downstream

enqueue and status work while the service runs; they share the queue file. Stopping the service (SIGTERM, systemctl stop, docker stop, ending the task) stops taking new jobs and waits for running ones. A job cut off anyway runs again at the next start.

Status records#

Every agent.status_seconds, the agent writes agents/<name>.json to each environment it runs, in LH_Meta/Files/_fabric_desktop/ (a local environment keeps it in its .local/<env>/_control folder). Anyone who can read the environment's LH_Meta can see which agents serve it, whether they are alive and what they are doing, without access to the agent's machine. A record describes only that environment's jobs. Fabric Desktop by KratosBI's workload views and Monitor > Operations > Agents (user guide) read these files.

Fields may be added within contract 1; readers should ignore fields they don't know. Agents built before these fields were added leave out owner, identity, credential and each schedule's entities and when; read a missing field as unknown, never as "all" or "none". A record that can't be written (OneLake unreachable, LH_Meta missing) is logged and tried again at the next beat; it never stops the agent.

Incidents (Preview)#

With [incidents] enabled = true, every incidents.seconds the agent reads each environment's run log (the last 2,000 runs) and its entity files, and keeps one incident per problem in incidents/incidents.json in the same control area as the status records. The desktop app evaluates the same way when someone asks it to and writes to the same file, so the app and any number of agents never open two incidents for one problem: every change is a conditional write that only lands if nobody wrote in between, and is redone otherwise.

"Can't evaluate" is about reading, never about a workload: while it is open, workload incidents in that environment stay as they were instead of being marked recovered. The same holds for agent records: if agents/ can't be listed, agent-offline incidents stay as they are. An agent never reports itself offline; another agent or the app has to notice it. An agent stopped cleanly (state = "stopped") isn't offline.

A failed run's incident carries a likely category matched from the error text: Access, Throttled or capacity, Network, Table busy, Source missing, Code or schema, Data quality, Code, or Unrecognized, with whether a retry alone usually helps. It is a hint for where to look, never a diagnosis; the run's own error is the evidence. A new failure of a table with an open incident counts as another occurrence of it. Someone who resolves an incident by hand doesn't see it again for the same failed run, only for a new one.

With alerts.teams_webhook_env set, each change goes to Teams: one message when an incident opens or opens again, one when it recovers, and one digest when an evaluation changes more than incidents.digest_over. A message Teams doesn't take waits in alerts.sqlite3 in the agent's home and goes with the next evaluation, marked Delayed with when it happened; after three days it is dropped. The agent's own "a job didn't finish" alert is unchanged.

Routes send some incidents to their own channel. Each [[incidents.route]] has a name, webhook_env (the name of the environment variable holding the webhook; the address itself never goes in the file), and any of teams, owners, criticality, kinds and envs, each a list; an empty filter matches everything, and matching ignores case. A change goes to every route that matches it, and to alerts.teams_webhook_env when none does. A route whose variable isn't set is logged and skipped.

[incidents]
enabled = true
quiet_hours = "22:00-06:00"
escalate_after_minutes = 60
escalate_to = "on-call"

[[incidents.route]]
name = "sales"
webhook_env = "SALES_TEAMS_WEBHOOK"
teams = ["sales"]

[[incidents.route]]
name = "on-call"
webhook_env = "ONCALL_TEAMS_WEBHOOK"
criticality = ["critical"]

During quiet hours, messages for incidents whose criticality isn't in quiet_except wait in the outbox and go out when the hours end, marked Held during quiet hours since when they happened. Escalations are never held. An incident still open (nobody acknowledged it) escalate_after_minutes after it opened is marked escalated once, with a note, however many agents evaluate; the escalation route gets one Escalated message. Escalation doesn't page anyone by itself: it is a Teams post to the channel you name.

The file keeps every open incident and the last 200 resolved ones. Each has id, key (kind, workload, environment), kind, workload, name, env, title, text, evidence (run id and start time, or the freshness or objective numbers), state (open, acknowledged or resolved), opened_at, last_seen_at, occurrences, opened_by, owner (the entity's declared owner), team, criticality, classification (failed runs), escalated_at and escalated_to, who acknowledged and resolved it and when, resolution (recovered, cleared or resolved by hand) and notes. Contract 1; fields may be added.

Run from a Fabric pipeline#

Preview. Built and tested against local folders and a mocked OneLake. It has not yet run on a live tenant; see step 6 of Verify on your tenant.

A Fabric pipeline can ask the agent to run an environment and wait for the result, so the agent's runs fit into pipelines you already orchestrate in Fabric (after a Copy job lands files, before a semantic model refresh). The pipeline and the agent talk through files in the environment's LH_Meta lakehouse. Nothing listens on a network port, and the agent needs no inbound firewall rule.

pipeline ── writes ──> LH_Meta/Files/_fabric_desktop/requests/<id>.json
agent    ── claims, queues and runs it, then writes ──> status/<id>.json
pipeline ── reads status/<id>.json until it is final; writes cancel/<id>.json to cancel

Turn it on in agent.toml:

[pipeline]
enabled = true
entities = ["silver.*", "gold.*"]   # optional: what pipelines may run

Add the sample notebook. Import examples/pipeline/run_on_agent.ipynb into the environment's workspace as a Python notebook and set LH_Meta as its default lakehouse. In your pipeline, add a Notebook activity for it with these base parameters:

The activity succeeds when the run succeeds and fails otherwise, with the reason in its error. Its output (notebookutils.notebook.exit) is the final status, so later activities can read the run ids and row counts.

The contract (version 1)#

A request is a JSON object in requests/<id>.json. The id is 1 to 64 letters, digits, ., _ or -. Write the file under another name first and rename it, as the sample does; the agent gives a file it can't read 60 seconds before rejecting it.

The agent answers in status/<id>.json:

A final status never changes again. Fields may be added within version 1; readers should ignore fields they don't know.

How it stays safe:

• Each request runs once. The agent claims a request by creating its status file with If-None-Match: *, so when several agents read one mailbox only one takes it. The request id is the job's dedupe key in the agent's queue, so an agent that crashes after claiming queues it when it starts again, and never twice.
• Only the environment's own people can ask. A request runs the environment whose LH_Meta it is written to, and only identities that can write that lakehouse can write it. A Test pipeline can't run Prod. The agent also checks pipeline.environments, pipeline.entities and pipeline.allow_backfill, and that each entity exists in the project.
• Cancelling. Write any file to cancel/<id>.json. A request that hasn't started is cancelled. A run already going finishes its tables (stopping mid-write could leave a table half done) and isn't retried if it fails.
• Deadlines. A request not started by its deadline, including one waiting for a retry, is expired and its job is cancelled.
• Housekeeping. Finished requests and their status and cancel files are removed after pipeline.retention_days.

Shared schedules (Preview)#

Give the same schedule to several agents and set shared = true on it. Each due time then runs once, on whichever agent claims it, rather than once per agent. If one agent is down, the others still run the schedule. The design, and what is still to come, is in Multiple workers.

[pipeline]
enabled = true
environments = ["prod"]
claim = "when_free"          # optional: a busy agent leaves new requests for one with a free worker

[[schedule]]
id = "prod-hourly"           # the same id, env, entities and times on every agent that shares it
env = "prod"
entities = ["silver.orders"]
every_minutes = 60
shared = true

• How it works. When a shared schedule is due, each agent tries to create requests/sched-<id>-<slot>.json in the environment's mailbox with If-None-Match: *. Only the first create lands. The request is then claimed like any pipeline request, so one agent runs it. The <slot> comes from the due time in UTC, for example sched-prod-hourly-20270115T0900Z.
• Same times on every agent. An every-N-minutes shared schedule runs on multiples of its interval, counted in UTC (every hour on the hour, for example), so every agent computes the same due times. Daily and weekly times are each machine's local time, so agents sharing them must use the same time zone.
• No late runs. A slot's request has a deadline: its next slot, at most 6 hours on. An agent that comes up after that doesn't create it. An unclaimed request past its deadline is expired. With claim = "when_free", if every agent stays busy until the deadline, that due time expires without running. That is by design, not a lost run: the next due time runs as normal.
• Requirements. A shared schedule needs [pipeline] enabled = true for its environment, its id is at most 40 characters, and an every-N-minutes shared schedule uses whole minutes, at least 1. With pipeline.entities, it must name entities the allowlist takes. A shared backfill needs pipeline.allow_backfill.
• Not yet. A request claimed by an agent that then stops reporting waits for that agent, unless the others have take_over = true (below). Monitor > Operations > Agents doesn't warn about a schedule that agents share this way. A late or expired shared due time doesn't open an incident yet.

Taking over from a lost agent (Preview)#

With [pipeline] take_over = true, an agent takes over a pipeline request (a shared schedule's, or any other) that another agent accepted and then stopped reporting on. That agent counts as lost once its status record in agents/<name>.json is past its stale_after, plus a minute for clock differences.

• One agent takes it. The takeover replaces the request's status file only if it hasn't changed since it was read (If-Match), so when several agents try at once, one wins. The status then names the new agent, taken_over_from, taken_over_at, and agents, every agent that has held it.
• The whole request runs again. Tables the lost agent finished are run again. A request whose deadline has passed ends expired instead of starting again. Before it writes each table, the new agent takes that table's write lease, so it waits for a lease the lost agent still holds to run out.
• The lost agent stands down. An agent that comes back after its request was taken over (it was paused, or its network was down) cancels its own copy. A run it is still in stops before its next table, without writing or recording it, and its audit log says handed_over. A table write already under way when it comes back can't be stopped; the table lease is what keeps the two writers apart then.
• Not forever. A request that retry.max_attempts agents have held ends failed, with each of them named, so a request that crashes every agent doesn't go round them all for ever.
• What isn't taken over. An agent without a status record, or with one that can't be read, keeps its requests, because nothing shows it is lost. An agent doesn't take over a request it would refuse as new (its pipeline.entities, allow_backfill or environments). With claim = "when_free", it takes one over only while a worker is free.

Verify on your tenant#

Nothing here has run against a live Fabric tenant yet. Before Prod, check these on a Dev workspace (roadmap M0):

1. fabric-desktop agent check succeeds with your service principal (tokens for Fabric and OneLake).

2. fabric-desktop agent enqueue dev writes the Dev tables, and they show in the lakehouse after a refresh.

3. A lease file appears in LH_Meta/Files/_fabric_desktop/leases/ while a table runs and is gone after. OneLake's support for If-None-Match and If-Match on these requests is the key thing to confirm.

4. Two agents, or the agent and the app, running the same table at once: one waits for the other and no rows are lost.

5. Status records: LH_Meta/Files/_fabric_desktop/agents/<name>.json appears within status_seconds of starting the agent, its heartbeat_at moves on, and state is stopped after systemctl stop (or stopping the Windows service).

6. Pipeline requests: with [pipeline] enabled = true, run a pipeline whose Notebook activity runs run_on_agent.ipynb with entities set to one bronze entity. The activity succeeds, status/<RunId>.json in LH_Meta/Files/_fabric_desktop/ ends succeeded with the agent's job_id, and the Monitor shows the run. Then run it with a pipeline identity that can't write LH_Meta and check the activity fails to write the request. Then write cancel/<id>.json before the agent's next poll and check the status ends cancelled.

7. Incidents: with [incidents] enabled = true and a table that fails in Dev, LH_Meta/Files/_fabric_desktop/incidents/incidents.json appears within incidents.seconds with one open incident, Teams gets one message, and a second agent (or the app) evaluating at the same time doesn't add another. After a good run the incident is resolved as recovered. Turn the network off for one evaluation and check Teams gets the messages marked Delayed afterwards. Stop a second agent with Task Manager (not cleanly) and check the first opens "Agent … stopped reporting" after its stale_after, and recovers it when the second is back. With a route and quiet_hours covering now, check a non-critical failure waits and a critical one doesn't.

8. Shared schedules (Preview): give two agents the same shared = true schedule every 15 minutes. Over an hour, requests/ gets one sched-… file per quarter hour, and each runs once, on one agent. Stop one agent and check the other runs every slot.

9. Takeover (Preview): with take_over = true on two agents, end one agent's process with Task Manager or kill -9 while it runs a pipeline request. After its stale_after and one poll, the other agent's name is in status/<id>.json, agents lists both, and the run ends succeeded. Start the first agent again and check its log says the request was taken over and it writes nothing for it. This is the first time If-Match replace on OneLake is used for takeover.

Limits in this version#

• Pipeline requests don't record who wrote the request file, only the submitted_by text the pipeline gives. Whether OneLake reports a file's writer to the agent is still to verify.
• One agent per queue. Two agents on different machines each keep their own queue. Table leases still stop them writing the same table at once, but nothing stops both from running the same schedule, unless it is a shared schedule (Preview, untested on a tenant). Otherwise, give each environment's schedules to one agent.
• The desktop app and runs in Fabric don't take write leases. Their writes are still safe (each commits against the version it read and merges again on a conflict), but a busy table means retries instead of waiting in turn.
• Notebooks and job files don't run in the agent yet; it runs metadata environments.
• Agent environments run on the machine, whatever their Runs on setting in the app.
• Lease expiry uses the holder's clock. Keep agent clocks in sync (NTP).
• Incidents (Preview) cover metadata tables and agents only. The app shows them in Monitor > Operations > Incidents (user guide). The app leaves an environment to the agent while a running agent checks its incidents; when no agent does, the app checks it while open and posts to its own webhook. Quiet hours use the agent machine's clock, and routes are set per agent: two agents with different routes post the same incident to different channels.
• The agent doesn't update itself. Install a new version with pip install and restart the service.