Notebooks and SQL

Notebooks in Fabric Desktop by KratosBI look and work like Fabric's, but every cell runs in a Python kernel on your PC. Reading and writing lakehouse data goes straight to OneLake. A run on this PC starts no Fabric job, but its OneLake reads and writes still use the workspace's capacity. This page covers how to write, run, publish and schedule notebooks, which Fabric notebook code works here, and the SQL query tab.

Your first notebook#

1. Create > Notebook (or New notebook on Home, or Ctrl+K "New notebook"). The notebook opens with one empty code cell.

2. Add data items in the ribbon opens a OneLake catalog of every lakehouse you can reach. Select one to see its tables and files on the right, then Attach (or double-click it). The attached lakehouse shows in the Explorer on the left. New lakehouse at the bottom creates one and attaches it. A running cell keeps the lakehouse it started with: starting a cell from another notebook, or an external tool's run against another lakehouse, waits or is refused until it finishes.

3. In the Explorer, click a table to insert code that loads it, or a file to insert code that reads it.

4. Run the cell with Shift+Enter. Output shows below it.

5. Write a result back:

   import polars as pl
   from fabric_desktop import lakehouse
   
   sales = lakehouse.read_csv("Files/raw/sales.csv")          # from the attached lakehouse
   by_region = sales.group_by("region").agg(pl.col("amount").sum())
   lakehouse.write_table(by_region, "dbo.sales_by_region")     # a Delta table in OneLake
   

6. The notebook saves itself a second or two after each change. Ctrl+S saves right away. Rename it by clicking its name in the breadcrumb at the top.

Create > Notebook from a template (Preview) starts from a template instead. Pick a group or search, answer a few questions (the tables, the columns that identify a row, the table to write), check the cells it will write, and the notebook opens. The cells are plain code from a fixed template, nothing generated by AI, and they read and write through the lakehouse module, so the notebook runs here and in Fabric. The wizard also saves the columns that identify a row of each table it writes in the notebook, and Prove uses them to match rows.

Notebooks are .ipynb files in %LOCALAPPDATA%\FabricDesktop\notebooks. They stay on your PC until you publish them.

Each cell keeps a stable id (the standard id of notebook format 4.5) through edits, moves and saves; a copied cell gets its own. Comparisons and run receipts use it to point at a cell after the notebook changes. When you save a notebook that came from Fabric or Jupyter, the metadata Fabric Desktop by KratosBI doesn't use (a Fabric environment, widget state, other tools' cell settings) is kept as it was. Cell outputs aren't saved in the file.

The lakehouse module#

from fabric_desktop import lakehouse is the notebook's way to the attached lakehouse.

Table names work as they do in Fabric: table, schema.table, or either with the lakehouse's name in front (Sales_Lakehouse.dbo.orders). A lakehouse created without schemas has only dbo, so a name such as tpch.region fails there, in Fabric too; use tpch_region instead, or check lakehouse.has_schemas() to pick the name.

Every write:

• is checked against the Fabric-safe Delta profile (reader version 1, writer version 2, classic checkpoints, no deletion vectors or column mapping), so the SQL analytics endpoint and Direct Lake can read it. An existing table outside the profile (for example one where Fabric Spark turned on deletion vectors) is refused before anything is written, so the table is left exactly as it was;
• writes the data files first and the Delta log last, so Fabric never sees a half-written version;
• shows in Monitor as a notebook write, with rows, mode and Delta version.

A table Fabric hasn't seen before can show under Unidentified in the lakehouse until you refresh the lakehouse in Fabric.

Tables managed by the metadata framework can't be written from a notebook; you'll get an error. Use a scratch_ table name for experiments next to managed tables.

Fabric notebook code on this PC#

Most notebooks written for Fabric run here unchanged.

spark is ready in every cell. It is DuckDB's PySpark-compatible API, wired to the attached lakehouse. There's no Spark cluster; everything runs on this PC.

df = spark.read.table("dbo.sales")
df = spark.sql("SELECT region, SUM(amount) AS total FROM sales GROUP BY region")
raw = spark.read.csv("Files/raw/sales.csv", header=True)
df.write.mode("overwrite").saveAsTable("dbo.sales_by_region")
df.createOrReplaceTempView("v_sales")
from pyspark.sql import functions as F

• spark.read.csv/parquet/json/load("Files/..."), spark.read.table, spark.sql, saveAsTable, temp views and pyspark.sql.functions work.
• %%sql cells use the same session, so they see temp views. CREATE TABLE ... AS SELECT and INSERT INTO write lakehouse tables. As in Fabric, CREATE TABLE stops if the table already exists (IF NOT EXISTS leaves it as it is); use CREATE OR REPLACE TABLE or INSERT OVERWRITE to replace it.
• Writing tables follows Delta's rules, as in Fabric. An overwrite or append whose columns don't match the table, or that would change a column's type (decimals into a whole-number column, numbers into a text column), stops with nothing written, with or without mergeSchema. A narrower type of the same kind, such as INT into a BIGINT column, still writes. Add .option("overwriteSchema", "true") to an overwrite to replace the schema, or .option("mergeSchema", "true") to add new columns; mergeSchema never changes an existing column's type. insertInto writes to an existing table, matches columns by position, and appends unless you pass overwrite=True; SQL INSERT matches by position too.
• Writer options: overwriteSchema, mergeSchema and, for CSV files, header are honored. optimizeWrite, maxRecordsPerFile and compression only change file layout and are skipped with a note. Any other option stops the cell before anything is written. In particular, replaceWhere, partitionOverwriteMode, INSERT ... PARTITION and INSERT ... REPLACE WHERE replace part of a table, and ignoring them would replace all of it, so they run in Fabric only.
• read.schema(...) takes a StructType. CSV columns get its names and types by position; Parquet and JSON columns are matched by name. A value that doesn't fit the type becomes null, as in Spark's default mode. A schema on a table, a Delta path or text files stops the cell.
• Not available on this PC: RDDs, UDFs, streaming, Scala and R cells, and MERGE, UPDATE or DELETE in SQL. A notebook that uses them runs in Fabric only (see Where a notebook runs).

notebookutils (and mssparkutils) cover:

Paths can be written the Fabric way: Files/..., /lakehouse/default/Files/..., or the attached lakehouse's abfss:// path.

fs changes follow Fabric's rules: rm removes a folder that isn't empty only with recurse=True; Tables/ can be read but not changed through fs (write tables with write_table or saveAsTable); mv keeps the source until the copy has every file at its size; and a run that can't write the lakehouse (an external AI tool's run, or a proof) can't change files either.

Magics: %run Other notebook runs another notebook in this one's namespace. %pip lines are skipped with a note, because the engine's packages are fixed. %%configure does nothing here.

Running cells#

Printed output shows while a cell runs. Code stuck inside a library call can't always be interrupted; the app then offers Restart kernel.

Settings > Preview features > Run notebooks in their own process (Preview) runs cells in a Python process of their own instead of inside the app's engine. Then Restart kernel on a stuck cell ends only that process and any processes its cells started, a crash in a library ends only that process (the cell says so and the next cell starts a fresh one), and the rest of the app keeps answering meanwhile. Switching it clears variables.

With that setting on, two more Preview settings appear under it.

• Cell time limit. A cell that runs longer is interrupted, and variables are kept. If it doesn't stop within 30 seconds, for example because it's waiting on one long call, its process is ended and variables are cleared. The cell says which happened and offers Run again without the limit. Proofs and side by side use the same limit, and a proof's receipt records the limits in force.
• Notebook memory limit. If the notebook's process uses more memory than this, it's ended, so one notebook can't slow the whole PC. Variables are cleared and every cell is marked stale. Memory is checked every second, so a sudden spike can briefly go past the limit before it's seen. Fabric Desktop by KratosBI counts private bytes on Windows and resident memory on Linux, for that process only; processes it starts itself aren't counted.

Your organization can set either limit in the policy file (azure-setup.md); Settings then shows it as set by your organization.

Cell dependencies and stale outputs (Preview)#

View > Dependencies shows under each code cell what it depends on, read from its code: the variables it uses and which cell defined them, the tables and Files/ paths it reads and which cell wrote them, what it writes, and which later cells use it. Names no cell defines are listed, so a typo or a missing import shows before you run. A dependency you know about but the code doesn't show can be declared in the cell's metadata as depends_on (a list of cell ids).

A cell gets a Stale tag when its output may not match the code above it: its code changed since it ran, a cell it depends on is stale or ran again after it, or the kernel restarted. Cells that use exec, %run or from x import * can depend on anything, so they're stale after any change above them. Cells that haven't run aren't counted.

Run > Run stale cells runs the stale cells again, in order. If any of them writes a table or file, installs packages or calls the network, it asks first and lists what each does, and offers to run only the cells before the first one.

Dependencies come from reading the code, not from watching it run. A cell can still depend on something reading can't see (a value read from a file another program changed, a global set inside a function from a library), so Run all remains the way to be sure.

Variable history, watches and hidden state (Preview)#

The Variables pane (View > Variables) has three more parts for cells that run on this PC:

• Watches. Type an expression, such as len(events) or df["amount"].sum(), and it's shown with its value after every cell, with its last few values so you can see which cell changed it. A watch runs as notebook code in the kernel, after the cell and as part of its run, so Cancel and the cell time limit cover it. Keep watches to reads: one that changes data changes your notebook's state too. Only expressions are accepted (not x = 1), up to 10 per notebook. A watch that takes longer than 2 seconds is paused until you click Resume. The expressions are saved in the notebook file; their values aren't. Proofs and side by side don't run watches, so they can't change what a proof compares.
• History. For each cell run: the variables it added, changed and removed, with their type and size. Values aren't kept. History stays in memory for the current kernel only; it's cleared when the kernel restarts and isn't saved. Cell numbers are as they were when the cell ran.
• Hidden state. After a cell runs, Fabric Desktop by KratosBI checks the names it uses against the cells in the notebook (the same reading of the code as Dependencies). If the cell uses a name that no cell defines, or that only a cell further down defines, but the kernel still has it from an earlier run, a warning shows under the cell: the cell works now but will fail in Fabric, in a proof and after a restart. If that name was set by a cell you've since deleted or changed in this session, Insert that cell back above puts its code back. Ignore for this notebook stops the warning for that name and is saved in the notebook file.

None of these run for notebooks that run in Fabric.

Failure replay bundles (Preview)#

When a cell fails on this PC, Save replay bundle under its error saves a file someone else (or you, later) can use to run the notebook again up to that cell, against the same inputs. The dialog shows exactly what goes in before anything is saved:

• Always included: the code of the cells up to the failed one; each table the cells read, with the Delta version they read; each file under Files/ they read, with its SHA-256; Python, the versions of the packages the cells import, and whether the kernel runs in its own process with which limits; and the error, with anything that looks like a secret and your home folder's path removed.
• Workspace and lakehouse IDs: included unless you untick them. Untick them to share the bundle outside your organization. Unticked, they're also taken out of the error, the traceback, the notes and the cells' code: OneLake addresses become abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<lakehouse>/... and any other ID becomes <id>. If the code had them, put your own back before the replay reaches those cells. The bundle still has the lakehouse's name.
• A sample of the input data: off unless you tick it. Up to 1,000 rows of each table, read at the version the cells read. The bundle then holds your data, so share it only with people allowed to see it. Your organization can turn this option off (allow_replay_samples in the policy file).

The bundle is a .zip saved to your Downloads folder as a new file each time (bundle.json, plus samples/ when you include a sample). A table's version is recorded the first time a cell reads it since the kernel started; for a table the notebook writes before reading, it's the version before that write. Code that looks like it holds a secret is removed from the bundle, so a replay may fail there.

A bundle also lists what it can't promise: cells that hadn't run, ran in a different order or ran again after the failure; a file that changed on this PC after a cell read it (it has no SHA-256); and reads that didn't go through lakehouse (read_table, sql, read_csv and the like), which aren't recorded. Replaying a bundle. Create > Open replay bundle lists the bundles in your Downloads folder, or choose a bundle file someone sent you. It opens the bundle's cells as a new notebook, replay, with a banner at the top. Nothing runs until you click Replay to cell N. A replay runs the bundle's code on this PC with your sign-in, so read the cells first when the bundle came from someone else.

• Replay needs Run notebooks in their own process (Settings > Preview features), and the bundle's lakehouse attached. A bundle saved without workspace and lakehouse IDs replays against the lakehouse you attach; check it's the same one.
• Replay to cell N starts a fresh kernel process in which every table is read at the bundle's version, and every write goes to a sandbox under Files/_fabric_desktop/proofs/replay-<id>/pc/ in the attached lakehouse, never its tables, as in a proof. Then the cells up to the failed one run as usual, so their output, variables and watches are there to debug. The banner then says whether the cell failed the same way (reproduced), failed with the same kind of error but another message, failed differently or earlier, or didn't fail.
• If a table's version can no longer be read (vacuumed, or its log cleaned up), the banner says so and replay waits for you to choose: the bundle's sample (when it has one) or the table's latest version. Either is marked in the result as not the same input. Files are read as they are now; a file whose SHA-256 differs from the bundle's is listed.
• The banner also lists how this PC's Python and packages differ from the bundle's.
• While the replay runs, the kernel runs nothing else: other notebooks' cells wait until you click End replay. Restarting the kernel also ends it. A replay notebook's cells don't run outside a replay, and it can't be proven, run side by side or published, until you click Keep as a normal notebook. After that its cells read and write the lakehouse's tables like any notebook's.

What output looks like:

• A cell's last expression shows below it. display(x) shows a value mid-cell.
• Data frames (Spark, Polars, pandas, Arrow, DuckDB) show the first 1,000 rows with three views: Table (search, click a heading to sort), Chart (bar, line or scatter, with the X axis, values and aggregate you pick) and Inspect (missing, distinct, min, max, mean and a histogram per column). display(df, summary=True) opens Inspect.
• Matplotlib charts show as images. Objects with an HTML view (Plotly, %%html cells) render in a sandboxed frame.

Editing#

• Cells have line numbers, bracket matching, folding, find and replace (Ctrl+F), Ctrl+/ to comment, and autocomplete (Ctrl+Space) from the running kernel's variables, Python names, and the attached lakehouse's table names inside strings and %%sql cells.
• Each cell has a language picker (Python, Spark SQL, HTML, Markdown) and ... for more actions.
• Command mode (press Esc): A and B add a cell above or below, DD deletes, Z undoes a delete, M and Y switch between Markdown and code, C, X and V copy, cut and paste cells, O hides output, Shift+L toggles line numbers, and Enter goes back to editing.

The ribbon has four tabs:

A Parameter cell is the cell that pipelines and notebookutils.notebook.run pass values into, as in Fabric. A frozen cell can't be edited, run or deleted until you unfreeze it.

Publish to OneLake#

Publish to OneLake saves the notebook as a notebook item in a Fabric workspace, so it shows in Fabric and can run there.

1. The first time, it asks which workspace (starting from the attached lakehouse's). The choice is saved in the notebook, so later publishes go straight there.
2. The arrow next to Publish shows where it goes and has Change workspace....
3. The attached lakehouse stays the notebook's default lakehouse in Fabric, even when it lives in another workspace.

How it is published:

• A notebook that uses spark or %%sql is published as a Spark notebook, where spark and %%sql are Fabric's own.
• A notebook that uses fabric_desktop.lakehouse is published as a Fabric Python notebook with a setup cell first. That cell defines the same lakehouse module from libraries Fabric's Python notebooks already have (Polars, deltalake, DuckDB), so the notebook runs there unchanged. Its %%sql cells become lakehouse.sql(...). The setup cell is only in the published copy, not in your notebook.
• Other notebooks are published as they are.

Where a notebook runs#

Runs on in the ribbon picks where Run all runs the notebook. The choice is saved in the notebook.

A note above the cells tells you when a notebook can't run where it is set to, and which cell is the reason:

• Spark features this PC doesn't have (RDDs, UDFs, streaming, Scala or R cells, %%sql that MERGEs, UPDATEs or DELETEs) mean Fabric only.
• A notebook with no lakehouse attached, or one that reads a path on your PC like C:\data, means this PC only.

The note has a button to switch when the other place would work.

Run in Fabric and schedules (Preview)#

Run in Fabric publishes the notebook, then starts it on the workspace's Fabric capacity with Fabric's job scheduler. The status bar, the bell and Monitor follow it until it finishes. Attach a lakehouse first; the notebook runs in that lakehouse's workspace (or the workspace you publish to).

Schedule opens one dialog for both places:

• On this PC: Windows Task Scheduler runs the saved notebook, every cell in order, with its attached lakehouse. It runs even when the app is closed, as long as the PC is on and you're signed in to Windows. Save your changes first: it runs what's saved.
• In Fabric: publishes what's on screen and adds a Fabric schedule. It runs on capacity even when your PC is off.

For either, choose Repeat (Every day, Some days of the week, or Every few minutes), the Times of day (24-hour, comma between, for example 06:00, 18:00), the Days, or Minutes between runs, and the Time zone. The same dialog lists the notebook's schedules and removes them. All schedules are also in Monitor > Schedules.

Side by side (Preview)#

Turn on Side by side in the ribbon and Run all runs the notebook on this PC, then in Fabric, and shows the time of each and whether both wrote the same tables (rows, columns and a checksum of every row). Both runs write your real tables, so it lists the tables it will overwrite and asks first; Prove is the sandboxed way. The TPC-H benchmark notebooks test it at scale. See Side by side: this PC vs Fabric.

Prove this notebook (Preview)#

Prove in the ribbon opens a panel beside the cells. It checks the notebook, then runs it on this PC and in Fabric from the same pinned table versions, with both runs' writes kept in a sandbox folder, and shows whether the results match:

• Before a run: what blocks a proof (each reason names the cell), warnings such as clock or web calls, the packages on each side, and how results are compared. Change rules sets the columns that identify a row and the tolerances; they're saved in the notebook.
• While it runs: each step, a link to the Fabric run, and Cancel proof. Other cells wait until it finishes.
• The result: Same, Differs, Inconclusive or Failed, and each table's result. Click a table to see the rows that differ, side by side, and to verify again under other rules without running anything. Full scorecard shows time, resources and capacity, each marked Measured, Estimate or Not available. Receipt shows the proof's tamper-evident record and saves it to Downloads as an HTML or Markdown report (leaving out your name, locations or row values) or as the receipt file itself.

See How a proof works.

SQL query#

The SQL query tab works like Fabric's. Open it with SQL query on a lakehouse in OneLake or Workspaces, or New SQL query on a lakehouse page.

1. The Explorer lists the lakehouse's tables. Click a table to run SELECT * FROM <table> LIMIT 100. Click ▸ next to it to see its columns; click a column to type its name where the cursor is.
2. Pick the SQL you are writing in the dialect list next to Run (see below), then write it using the names Fabric uses: SELECT * FROM sales, or schema.table for schemas other than dbo.
3. Run, or Ctrl+Enter. Select some text first to run just that.
4. Results show the first page of rows (1,000 by default), with how many rows there were and how long it took. Next › and ‹ Previous move through the rest without running the query again; Stop ends a query that is still running. See Running SQL queries on this PC for states, limits and what a query may do.

How it works: sqlglot translates your Spark SQL to DuckDB SQL, and DuckDB reads only the columns and files it needs from OneLake. Dates and times are worked out in UTC, as in Fabric's Spark, whatever this PC's time zone, so year() and date_format() give the same answer here as in Fabric. Each query runs in a process of its own with a time and memory limit (Settings › SQL queries). Queries only read; nothing in the lakehouse changes unless you Save as table. A query that would change a lakehouse table, a file on this PC or Fabric is refused before anything runs, and one that changes only its own session (a temp view) asks first. Query tabs, with their SQL and dialect, are kept per lakehouse between launches.

Dialects#

Each Fabric engine has its own SQL, so each query tab has a dialect. It sets the highlighting, the suggestions, how Format SQL reads the query and which engine the compatibility check uses.

A query that runs here is not proof that it runs, or gives the same answer, in a Fabric Warehouse or SQL database: DuckDB is a different engine. That's why T-SQL isn't run on this PC.

Compatibility (preview)#

As you type, the query is checked against the engine picked in the dialect list, without running it: things that engine doesn't support are underlined, and the Compatibility panel beside the editor lists every finding with why, what to do, its confidence and its Microsoft source. Export JSON saves the report. See SQL compatibility check.

Editor#

• Line numbers, current-line highlight, bracket matching, folding (click the arrow in the gutter), and highlighting of keywords, functions, strings, numbers, comments, variables (@name in T-SQL) and operators. Table names the lakehouse has are underlined, so a misspelled table stands out.
• Suggestions as you type, or Ctrl+Space: keywords and functions for the dialect, the lakehouse's schemas and tables, and the columns of the tables the statement names. After s. where s is an alias or a table, you get that table's columns. Names that need it are quoted for the dialect (`Region Name`, [Region Name] or "Region Name"). Tables come from the list the Explorer already loaded, and a table's columns are read once, the first time a statement names it, so typing doesn't call Fabric. Refresh reads them again.
• Ctrl+Enter runs the selection, or everything. Ctrl+S saves the query tabs (they are also saved as you type). Shift+Alt+F formats.
• Ctrl+F finds, with replace. Ctrl+/ comments or uncomments lines. Tab and Shift+Tab indent. Ctrl+Z and Ctrl+Y undo and redo.
• More than one cursor: Ctrl+D selects the next match of the selection, Ctrl+click adds a cursor, and Alt+drag selects a rectangle.

Format SQL#

Format SQL uses sqlglot to read the query in the tab's dialect and print it back laid out. It never changes what the query does: before replacing your text it checks that the formatted query reads the same, and that every identifier, string, number, function name and comment is still there. Only keywords, function names and date parts such as 'month' change case. If sqlglot would change anything else (rename a function, drop a comment, read something differently) or can't read the query, your SQL stays as it was and a message says why. Formatting is one step for Ctrl+Z. Comments are kept; a comment can move to the end of the line next to the code it was attached to.

Job files#

For repeatable transforms without a notebook, a job file runs DuckDB SQL or a Polars script and publishes the result. See Job files in the README. From a terminal: fabric-desktop preview job.toml shows the result without writing it, and fabric-desktop run job.toml writes it. With a metadata project open, run refuses a table the project manages and any workspace Test or Prod uses.