Optimize and the central lakehouse

Optimize keeps optimization results in one Microsoft Fabric lakehouse that every Fabric Desktop by KratosBI installation in your organization can write to and read from. That includes findings, recommendations, applied fixes, measurements, local runs and savings. The lakehouse is the system of record; nothing about it lives only on one PC.

Status: Preview. None of this has been tried on a live Fabric tenant yet. The plan is in plan.md.

One lakehouse for the organization#

• One lakehouse is enough for a whole organization. You don't need one per developer, project or workspace.
• If you need separate boundaries, such as business units, regulated data or separate environments, use separate optimization lakehouses. Each installation is connected to one at a time and can switch between the ones it knows.
• The default name is FabricDesktop_Central: it is Fabric Desktop's central lakehouse, where other features keep their organization-wide data too (Table families). It is a separate item from the platform's LH_Meta, which is per environment. A lakehouse created before this release under the old default name, FabricDesktop_Optimization, keeps working as it is.

Where it is in the app#

Optimize in the left rail (Preview), in the place the Fabric page had; the Fabric portal is now a tab of Workspaces. It has three tabs: Overview, Opportunities and Cost and savings (user guide). The lakehouses this PC uses are in Settings > Optimize lakehouses.

Setting it up#

The first person creates the lakehouse: they pick a workspace and a name, and the app does the rest. It creates the lakehouse, creates the tables, records the schema version and default settings, checks it can read and write, and remembers the connection on that PC. Nobody runs SQL or creates tables by hand.

Everyone else connects to the existing lakehouse, and nothing is created again. When the app connects to a lakehouse, the outcome depends on what is already in it:

Disconnecting only removes the lakehouse from this PC's list. The lakehouse and its data stay as they are, and connecting again picks up where you left off. The app never deletes an optimization lakehouse.

Permissions#

Optimize signs in as you, the same way as the rest of the app (see Permissions). It needs no new permissions and no Fabric administrator role.

The app finds out what you can do by trying it. It reads the schema version, then registers this installation (a row it would write anyway). With read access only, it opens the lakehouse read-only and says so. It doesn't read the workspace's role assignments, which would need admin rights.

What each installation writes#

Each installation writes rows into the shared tables. Every row records which installation wrote it, when, which app version wrote it and which schema version it was written against.

Analysis runs add the Fabric items they looked at, the problems found and the recommendations made; later releases add measurements and local run records. Those rows hold Fabric workspace and item ids, and the workspace, lakehouse, table and item names they are about. Names are identifying: anyone who can read the lakehouse can read them. SQL text and code are never written. Only fingerprints of them are stored (the store_sql_text setting is never).

Who can see what#

Optimize relies on Fabric's own security. It adds no access layer of its own.

• Anyone who can read the optimization lakehouse can read every row in it, including which installation wrote what. OneLake security roles can limit access by table or folder (OneLake security model). Optimize does not rely on row-level filtering for people reading the Delta files directly.
• When groups mustn't see each other's diagnostics, give each group its own optimization lakehouse in a workspace only they can reach.
• Access tokens are never written to the lakehouse.

How several installations write safely#

• Insert only. No installation updates or deletes a shared row. A change, such as a recommendation being accepted, is a new row, and the latest row wins.
• No duplicates. Every row has a record_id worked out from what it records. Rows are written with an insert-only Delta MERGE on it, so a retry or a second sync adds nothing.
• Bounded batches. Rows go in batches of at most 5,000, one commit each. A batch that fails after its retries stops the write; the batches before it stay, and writing again adds only what's missing.
• Bounded retries. A commit that loses a race with another installation, throttling or a network blip is retried up to five times with growing waits. A permission error is never retried.
• Race-safe setup. Two installations setting up the same lakehouse at once end up with one set of tables and one schema version row.
• Recoverable migrations. A migration only adds tables or columns, and each step can safely run again. Its version is recorded only after every step succeeded, so a migration that stopped part way finishes the next time.

Findings and recommendations#

Every installation adds what its analysis finds to the shared lakehouse, and everyone connected sees the combined picture.

• One finding per problem. A problem is identified by the rule that found it, the Fabric item (by workspace and item id) and what in it the rule looked at, such as a table. When several PCs find the same problem it shows once, with when it was first and last seen, in how many analysis runs and by how many installations.
• A run counts once it's complete. An analysis run's own row is written after its findings. If a PC loses its connection part way, nothing from that run shows until it's recorded again, and recording it again adds only the rows that are missing.
• One recommendation per problem and kind of fix, opened by whichever installation sees it first. Accepting, applying, dismissing or reopening it is recorded as a change with who made it, when and why, and everyone sees the latest. A dismissed recommendation stays dismissed when the problem is seen again; someone has to reopen it.
• Allowed changes. Open can become accepted, applied, dismissed or superseded. Accepted can become applied, dismissed or open again. Applied can become verified or open again. Dismissed and verified can only be reopened. Superseded is final.
• Only items you can open. Findings and recommendations about items in workspaces you can't open in Fabric aren't shown to you. This is a courtesy, not security: anyone who can read the lakehouse can read every row (Who can see what).

The tables#

All are Delta tables in dbo (or directly under Tables in a lakehouse without schemas), named fdopt_ so they can't collide with yours: installations, analysis_runs, artifacts, findings, recommendations, actions, measurements, cu_usage, savings, local_execution, dependencies, rules, audit, config and schema_version. Columns are in sidecar/fabric_desktop/optimize/schema.py. You never need to query them yourself.

Table families#

The central lakehouse holds more than Optimize's tables. Each feature that keeps organization-wide data there has its own family of tables, with its own prefix, schema version and upgrades:

Every family follows the same rules: insert-only rows with a record_id, provenance on every row, upgrades that only add, and a newer schema opening that family read-only (the others keep working). A family never writes to or migrates another family's tables; reading them, for example a view that combines two families, is fine. Each family has one owner, so two features that store rows of the same shape still use two families (and can share the column definitions). The first time a feature writes to a connected central lakehouse, it asks you, then adds its own tables beside the others; nothing that is already there changes. Tables that name people, such as the access cache, also get their own OneLake security role at table level, set up by the feature that adds them.

Tenant settings history#

A Fabric administrator's read of Fabric's tenant settings (GET /v1/admin/tenantsettings) is kept in three fdten_ tables, in the same row shape as FinOps' capacity settings (fdcap_), so one view reads both:

• What counts as a value. A setting's value is everything Fabric returns for it except its name, title and group, with security groups and properties in a fixed order. Microsoft renaming a setting isn't a change.
• When and by whom. A change is only ever "between these two reads". Fabric's tenant settings API says neither when a setting changed nor who changed it, so the history doesn't claim either.
• People. read_by holds the reader's Microsoft Entra object id, never a name or email.
• Only reads. It never changes a setting. A read by someone who isn't a Fabric administrator is recorded as unreadable, so a gap in the history shows as a gap, not as "nothing changed".
• What's built. The engine calls tenant_settings_read and tenant_settings_history, untested on a live tenant. The page that shows the history comes after a mockup.

For developers: a family is a schema.Family registered with schema.register at import of the module that owns it. Its prefix is fd, 2 to 8 lowercase letters and _; its migrations are numbered from 1, and the first creates schema_version. OptimizeMixin.central_family(family, approve=...) returns the connected lakehouse with the family's tables in place; it creates or upgrades tables only with approve=True, passed after the person confirmed, and otherwise raises with what it would do. Write with setup.write(..., family=family). A family's default_config keys are stored in fdopt_config with the family's prefix.

For IT admins#

Set this in %ProgramData%\FabricDesktop\policy.json (Configure Azure and Fabric):

Optimize is separate from usage data. Its rows go to a lakehouse in your own tenant, never to KratosBI, so the usage data switch doesn't affect it.