Skip to main content

Kernel, cells and hidden state

A notebook shows its cells in page order, but the kernel behind it remembers them in the order they were run. Build a kernel from a dictionary and exec, run three cells three ways, read a notebook file with the json module, and learn the one test that catches hidden state.
P02.9.1L1 Core25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: P01.4 Functions and scope P01.7 Strings and files

What you will be able to do
  • Explain what a notebook’s kernel is and why it remembers runs in time order, not page order.
  • Predict the values of x and y after the cells x = 10, x = x + 1 and y = x * 2 are run in any order.
  • Build a toy kernel in plain Python from one shared dictionary and exec.
  • Open a .ipynb file with the json module and find each cell’s source, outputs and execution count.
  • Use Restart and Run All as the test of a notebook, and say what execution counts can and cannot prove.

The question

A notebook on your screen says y is 24. You send it to a colleague, they run it from the top, and they get 22. Neither of you changed a single character of code. Where did the difference come from, and how can you tell, before you share a notebook, that its numbers will come out the same for everyone?

Intuition

A notebook is a file of cells. A code cell holds a few lines of Python; a text cell holds words. You run one cell at a time, in any order you like, and the output appears under it.

The code does not run inside the file. It runs in the kernel: a separate Python process that stays alive between cells. Every name a cell defines is kept in the kernel’s memory, and the next cell you run starts from whatever is there. The kernel does not know where a cell sits on the page. It only knows the order in which cells arrived.

Two panels. Left, titled The file: cells in page order, shows three code cells, x = 10, x = x + 1 and y = x * 2, with the execution counts 1, 3 and 4 in orange beside them. Right, titled The kernel: runs in time order, lists four runs as numbered yellow badges: x = 10 giving x = 10, x = x + 1 giving x = 11, x = x + 1 again giving x = 12, and y = x * 2 giving x = 12 and y = 24. Green arrows lead from each cell to its runs, two of them from the middle cell. A table below shows the kernel holds x = 12 and y = 24. A sentence underneath says that the same cells run once, top to bottom, would leave x = 11 and y = 22.
P02.9.1-F1

The page shows three cells; the kernel saw four runs, because the middle cell was run twice. What the kernel holds depends on the runs, and the numbers in square brackets are all the file keeps of them.

That gap between the page and the runs is hidden state: values in the kernel that you cannot work out by reading the cells from top to bottom. It is the reason a notebook can show one result and rerun to another.

Build it up

Step 1. A cell is a piece of text. Before it runs, the cell x = x + 1 is a string of nine characters. Running it means handing that string to Python.

Step 2. The kernel’s memory is a dictionary. Python keeps the global names of a program in a dictionary from name to value (P01.4.4). A kernel is a program whose dictionary lives as long as the kernel does.

Step 3. Running a cell reads and writes that dictionary. The built-in exec(source, kernel) runs a string of code with the dictionary kernel as its global names. x = 10 writes the key x. x = x + 1 reads x, adds one and writes it back. y = x * 2 reads x and writes y.

Step 4. Run the three cells top to bottom in a fresh kernel. The dictionary grows one run at a time.

Four panels in a row joined by green arrows. The first, fresh kernel, has an empty table of names and values. The second, after run 1, shows the cell x = 10 on a yellow strip and a table with x equal to 10. The third, after run 2, shows the cell x = x + 1 and x equal to 11. The fourth, after run 3, shows the cell y = x * 2 and a table with x equal to 11 and y equal to 22.
P02.9.1-F2

A clean run. Each cell reads the dictionary that the runs before it left behind, and writes its own names into it.

Step 5. Every run gets a number. The kernel counts the runs it has done: 1, 2, 3 and so on. After a run, the cell shows that number as its execution count, written [2]: beside it. A cell run again shows its newest number, so the count records the last run of each cell, not all of them.

The formal statement

NoteDefinition: kernel state and a clean run

Number the code cells \(1, 2, \dots, n\) in page order. The state of the kernel is its dictionary of names. Running cell \(i\) turns a state \(S\) into a new state \({c_i(S)}\).

A run of the cells \(i_1, i_2, \dots, i_k\), starting from the empty state \(S_0\), ends in

\[ \begin{aligned} S_1 &= c_{i_1}(S_0), \\ S_2 &= c_{i_2}(S_1), \\ &\;\;\vdots \\ S_k &= c_{i_k}(S_{k-1}). \end{aligned} \]

The clean run is the sequence \(1, 2, \dots, n\) from \(S_0\): every cell once, in page order, in a fresh kernel.

The end state depends on the whole sequence, not only on which cells ran. That is the formal face of hidden state. Two consequences follow.

Order matters. Cells that read a name another cell writes do not commute. Here cell 3 reads x and cell 2 writes it, so running 2 then 3 differs from running 3 then 2.

Repeats matter. Cell 2 reads x and writes it, so running it twice is not the same as running it once: \({c_2(c_2(S))} \neq {c_2(S)}\).

What the counts prove. A clean run leaves the counts \(1, 2, \dots, n\) down the page. So any other pattern, a gap or a number smaller than the one above it, proves that the saved outputs came from some other sequence. The reverse is not true. A cell edited after its last run keeps its old count and its old output, so even a perfect \(1, 2, \dots, n\) is a clue, not a proof.

The test. The only way to know what the cells produce is to make the clean run happen: restart the kernel, which empties its dictionary and sets the counter back to zero, and run every cell from the top. Jupyter calls this Restart and Run All. Rule et al. (2019, Rule 7) recommend it as a habit while working and as the final test of a notebook’s results.

The file that holds a notebook is the second half of the picture. A .ipynb file is a JSON document (P01.7.4): a dictionary whose key "cells" holds a list, one dictionary per cell.

Left, titled On screen: a code cell print(x, y) with the execution count 5 in orange, and its printed output 12 24 in vermillion below it. Right, titled In the file, report.ipynb (JSON): the text of the file, a dictionary with the key cells holding a list. The cell's entry reads cell_type code, then execution_count 5 in orange, source print(x, y) in black, and outputs in vermillion, a list holding one stream output whose name is stdout and whose text is 12 24 with a newline. Below come metadata and nbformat 4, nbformat_minor 5. Thin lines join the count, the code and the output on screen to the keys that store them.
P02.9.1-F5

A code cell on screen and in the file. The count, the source and the outputs are stored side by side, so the file keeps what the last runs printed, whatever order they ran in.

Worked example

Run x = 10, x = x + 1 and y = x * 2 in three orders, each in a fresh kernel.

Top to bottom, cells 1, 2, 3. Cell 1 sets \(x = 10\). Cell 2 reads 10 and sets \(x = 10 + 1 = 11\). Cell 3 reads 11 and sets \(y = {11 \times 2} = 22\). The result is \({(x, y)} = {(11, 22)}\), and the counts read 1, 2, 3.

Middle cell twice, cells 1, 2, 2, 3. Cell 1 sets \(x = 10\). Cell 2 sets \(x = 11\). Cell 2 again reads 11 and sets \(x = 12\). Cell 3 sets \(y = {12 \times 2} = 24\). The result is \({(12, 24)}\). The middle cell’s count is 3, its last run; count 2 has vanished from the page.

Last two swapped, cells 1, 3, 2. Cell 1 sets \(x = 10\). Cell 3 runs next and sets \(y = {10 \times 2} = 20\). Cell 2 then sets \(x = 11\), but \(y\) was already computed and nothing recomputes it. The result is \({(11, 20)}\): an x and a y that no reading of the page can explain, since the page says \(y = {x \times 2}\) and \({11 \times 2} = 22\).

Three panels, one per order of runs. Top to bottom runs cells 1, 2, 3: x = 10 gives x = 10, x = x + 1 gives x = 11, y = x * 2 gives x = 11 and y = 22; a yellow box reads (x, y) = (11, 22) and the file shows the counts 1, 2, 3. Middle cell twice runs cells 1, 2, 2, 3 and ends at (12, 24), with counts 1, 3, 4. Last two swapped runs cells 1, 3, 2: after y = x * 2 the state is x = 10, y = 20, and after x = x + 1 it is x = 11, y = 20; the box reads (11, 20) and the counts are 1, 3, 2.
P02.9.1-F3

One page, three histories, three answers. Only the counts under each panel hint at what happened.

Code

A toy kernel: one dictionary, and exec to run each cell’s text in it. Positions in code count from 0, so the cells 1, 2, 3 of the text are 0, 1, 2 here.

CELLS = ["x = 10", "x = x + 1", "y = x * 2"]

def run(order, cells=CELLS):
    """Run cells by position, in the given order, in one fresh shared dictionary."""
    kernel = {}                       # the kernel's memory: name -> value
    counts = [None] * len(cells)      # the execution count each cell shows
    for n, i in enumerate(order, start=1):
        exec(cells[i], kernel)        # the cell reads and writes kernel
        counts[i] = n                 # a cell keeps the number of its last run
    names = {k: v for k, v in kernel.items() if not k.startswith("__")}
    return names, counts

for order in ([0, 1, 2], [0, 1, 1, 2], [0, 2, 1]):
    print(order, *run(order))
[0, 1, 2] {'x': 11, 'y': 22} [1, 2, 3]
[0, 1, 1, 2] {'x': 12, 'y': 24} [1, 3, 4]
[0, 2, 1] {'x': 11, 'y': 20} [1, 3, 2]

exec puts a key __builtins__ into the dictionary, which is why the function leaves names that start with two underscores out of what it returns.

A deleted cell is hidden state too. Its names stay in the kernel until a restart.

kernel = {}
exec("threshold = 25", kernel)                    # a cell, later deleted from the page
exec("hot = [t for t in [21, 27, 29] if t > threshold]", kernel)
print("old kernel:", kernel["hot"])

fresh = {}                                        # Restart: a new, empty dictionary
try:
    exec("hot = [t for t in [21, 27, 29] if t > threshold]", fresh)
except NameError as error:
    print("fresh kernel:", type(error).__name__)
old kernel: [27, 29]
fresh kernel: NameError

Now a notebook file. The function below runs cells in a given order, catches what each one prints, and stores the result in the layout of nbformat 4, the format that Jupyter writes.

import contextlib
import io
import json
import pathlib
import tempfile

def run_and_record(order, cells):
    """Run cells in `order` in one kernel; return each cell's last count and printed text."""
    kernel, counts, printed = {}, [None] * len(cells), [""] * len(cells)
    for n, i in enumerate(order, start=1):
        buffer = io.StringIO()
        with contextlib.redirect_stdout(buffer):
            exec(cells[i], kernel)
        counts[i], printed[i] = n, buffer.getvalue()
    return counts, printed

def notebook(cells, counts, printed):
    """A dictionary in the layout of an .ipynb file (nbformat 4.5), code cells only."""
    def one(k):
        out = [{"output_type": "stream", "name": "stdout", "text": printed[k]}] if printed[k] else []
        return {"cell_type": "code", "execution_count": counts[k], "id": f"cell-{k + 1}",
                "metadata": {}, "outputs": out, "source": cells[k]}
    return {"cells": [one(k) for k in range(len(cells))], "metadata": {},
            "nbformat": 4, "nbformat_minor": 5}

SHOW = CELLS + ["print(x, y)"]
counts, printed = run_and_record([0, 1, 1, 2, 3], SHOW)     # the messy run
folder = tempfile.TemporaryDirectory()
path = pathlib.Path(folder.name) / "report.ipynb"
path.write_text(json.dumps(notebook(SHOW, counts, printed), indent=1), encoding="utf-8")

saved = json.loads(path.read_text(encoding="utf-8"))        # open it as plain JSON
for c in saved["cells"]:
    texts = [o["text"] for o in c["outputs"]]
    print(c["execution_count"], repr(c["source"]), texts)
1 'x = 10' []
3 'x = x + 1' []
4 'y = x * 2' []
5 'print(x, y)' ['12 24\n']

The file kept 12 24. Restart and Run All is now a few lines: take the sources, run them once in page order in a fresh dictionary, and compare what they print with what the file stored. A real file may store a source as a list of lines, so join them first.

def restart_and_run_all(nb):
    """Rerun the code cells in page order in a fresh kernel; list the cells whose output changes."""
    code = [c for c in nb["cells"] if c["cell_type"] == "code"]
    sources = ["".join(c["source"]) for c in code]
    _, fresh = run_and_record(range(len(sources)), sources)
    stored = ["".join("".join(o.get("text", "")) for o in c["outputs"]) for c in code]
    return [(k + 1, s, f) for k, (s, f) in enumerate(zip(stored, fresh)) if s != f]

print(restart_and_run_all(saved))
[(4, '12 24\n', '11 22\n')]

Cell 4 stored 12 24; a clean run prints 11 22. This site builds every lesson page the same way: a fresh process runs the cells top to bottom and the assertions check every stated number.

Explore

P02.9-X1

Run-order sandbox

Run the three cells in any order, as often as you like, and watch the counts and the kernel’s names. Then press Restart and Run All and read whether the screen you had matched a clean run. Try running x = x + 1 first in a fresh kernel.

Three cells, x = 10, x = x + 1 and y = x * 2, with execution counts 1, 3 and 4, beside the kernel's runs in time order and its names x = 12 and y = 24, which a clean run would make 11 and 22.

Pitfalls

Two notebooks of four cells: x = 10, x = x + 1, y = x * 2 and print(x, y). Left, marked with a cross and titled Saved after a messy run: the counts are 1, 3, 4 and 5, and the printed and saved output is 12 24. Right, marked with a tick and titled After Restart and Run All: the counts are 1, 2, 3 and 4, and the output is 11 22.
P02.9.1-F4

Wrong beside right. The same four cells, saved after two different histories. Only the right-hand file can be reproduced by anyone who runs it.

“The output is right there in the file, so that is what the code produces.”

The file stores what the last runs printed, in whatever order they happened. In the left-hand notebook above, 12 24 is genuine output, yet no one who runs the cells from the top will ever see it. Before trusting or sharing saved output, restart the kernel and run all.

“I deleted that cell, so its variable is gone.”

Deleting a cell removes its text from the page, not its names from the kernel. Later cells go on working until the next restart, when they fail with NameError. The same holds for a cell you edit: the kernel keeps the values from the old version until you run the new one. This is the notebook form of the global-state trap of P01.4.4.

“The counts read 1, 2, 3, so the notebook is clean.”

Counts in order are necessary, not sufficient. Change x = 10 to x = 5 after running it, save, and the file still shows [1]: beside it with outputs computed from 10. In machine learning this is how a training notebook ends up reporting an accuracy that its own code no longer produces.

Check yourself

Recall

1. What is the kernel of a notebook, and what does it keep between cells?

Show answer

A separate Python process that runs the cells. It keeps every name the cells have defined, in a dictionary that lasts until the kernel is restarted.

2. Starting from a fresh kernel, which values do the cells x = 10, x = x + 1, y = x * 2 give when the middle cell is run twice?

Show answer

\(x = 12\) and \(y = 24\), where a clean run gives \(x = 11\) and \(y = 22\).

3. What does a cell’s execution count record?

Show answer

The number of that cell’s most recent run, counted from the last restart. Earlier runs of the same cell leave no trace on the page.

4. Which three things does a .ipynb file store for a code cell, and in what format is the file written?

Show answer

Its source, its outputs and its execution count (with a cell type, an id and metadata). The file is JSON, so the json module can read it.

5. What is the test of a notebook, and why does it work?

Show answer

Restart the kernel and run every cell from the top. The restart empties the kernel’s dictionary, so no hidden state is left, and the run in page order is the clean run that anyone else will make.

Exercises

1. ★ In a fresh kernel, run the cells 1, 2, 3, 2 (the middle cell last). Give the final \(x\) and \(y\) and the counts the page shows.

Show solution

Cell 1 sets \(x = 10\), cell 2 sets \(x = 11\), cell 3 sets \(y = 22\), and cell 2 again sets \(x = 12\). So \({(x, y)} = {(12, 22)}\). The counts are 1 for cell 1, 4 for cell 2 (its last run) and 3 for cell 3, read down the page as 1, 4, 3. run([0, 1, 2, 1]) returns the same.

2. ★★ In a fresh kernel, the first cell you run is x = x + 1. What happens, and which single change to the page would make the code end with the same \(x\) and \(y\) however often, and in whatever order, you run it?

Show solution

It fails with NameError: the kernel’s dictionary has no x yet, so x + 1 cannot be computed. For the second part, merge the cells into one, x = 10 then x = x + 1 then y = x * 2. A cell runs its lines in order and starts by setting x afresh, so every run, the first or the tenth, ends with \({(11, 22)}\). Fewer, longer cells that each start from what they need leave less room for hidden state.

3. ★★★ Write looks_clean(nb), which returns True when the code cells of a notebook dictionary show the counts \(1, 2, \dots, n\) down the page. Apply it to the messy file of the Code section, and explain why True would still not prove the outputs reproducible.

Show solution
def looks_clean(nb):
    counts = [c["execution_count"] for c in nb["cells"] if c["cell_type"] == "code"]
    return counts == list(range(1, len(counts) + 1))

The messy file has counts 1, 3, 4, 5, so looks_clean returns False. A file can pass and still mislead: if a cell’s text was edited after its last run, the file keeps the old output beside the new code, and the counts do not change. Only rerunning, as restart_and_run_all does, compares the code with its outputs.

Summary card

The kernel remembers runs in time order; the page shows cells in page order.

The anchor figure again: three cells with counts 1, 3 and 4 on the left, and on the right the kernel's four runs in time order, ending with x = 12 and y = 24 where a clean run gives 11 and 22.
P02.9.1-F1

Page order beside run order.

  • A cell’s text runs in the kernel’s one dictionary: exec(source, kernel).
  • End state \(S_k = {c_{i_k}(\cdots c_{i_1}(S_0))}\) depends on the order and on repeats.
  • Clean run: cells \(1, 2, \dots, n\) in a fresh kernel. Counts \(1, 2, \dots, n\) are a clue, not a proof.
  • A .ipynb file is JSON: each cell stores source, outputs and execution_count.
  • The test: Restart and Run All.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Project Jupyter, Kernels (Programming Languages), Jupyter documentation: kernels are processes, specific to one language, that run independently and talk to the Jupyter applications; ipykernel, built on IPython, is the reference kernel.
  • Jupyter developers, The Notebook file format, nbformat 5 documentation. Top-level structure: a notebook is a dictionary with the keys metadata, nbformat, nbformat_minor and cells, and multi-line strings may be stored as a list of strings. Code cells carry an execution_count, an integer or null; Cell ids are strings of 1 to 64 characters; Code cell outputs lists stream, display_data, execute_result and error. The nbformat home page describes notebooks as JSON documents.
  • Python Software Foundation, exec, Built-in functions: with only a globals dictionary, it serves as both the global and the local names, and __builtins__ is inserted when it is missing.
  • Adam Rule and coauthors, Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks, PLOS Computational Biology 15(7), e1007007, published in 2019. Rule 7, Build a pipeline, advises restarting the kernel and rerunning all cells regularly, and calls it a good final test of results.

Next. P02.9.2 Randomness you can repeat meets a second kind of hidden state: the state of a random number generator, which every call moves on.

This lesson unlocks

P02.9.2 Randomness you can repeat

Your progress is saved in this browser only. No account, no tracking.
Back to top