Skip to content
niri-computer-use

Architecture

These drawings follow the code. The full account, module by module, is docs/architecture.md in the repository.

Processes

Section titled “Processes”

Your agent starts niri-computer-use serve over stdio, one per session. By default every serve on one niri relays to a single engine, and starts that engine when none is running. With shared = false or NIRI_COMPUTER_USE_SHARED=0, serve runs the engine itself. The engine holds the lease and talks to niri, the accessibility bus and Noctalia. It runs grim, notify-send and loginctl through one runner, each with a deadline. The crash guardian is the same binary watching the engine: if the engine dies while its virtual pointer or keyboard holds a button or key, the guardian sends the release.

Processes and what they talk toAn MCP client starts niri-computer-use serve over stdio. In shared mode serve is a bridge to the one engine of the niri instance, over engine.sock, and starts that engine when none listens; standalone, serve runs the engine itself. serve reads policy.toml at start and, in shared mode, sends it in the hello. The engine holds the desk with the lease, the event stream, the AT-SPI connection, the audit writer and the runner. It reaches files in the runtime directory beside niri's socket and the audit log; niri's IPC socket and Wayland display; the AT-SPI bus, Noctalia's IPC socket and logind; and its children: grim, notify-send, loginctl, the crash guardian and a paste keeper per paste. You reach the runtime directory with stop, resume and recover.MCP clientClaude Code, Codex, Pistdioniri-computer-use serveone per client: a bridge,or the engine itselfengine.sockpolicy.tomlread at startyou, in a terminalstop · resume · recoverengineone per niri instancedesklease · action mutex · refsevent streamone connection to niriAT-SPI connectionmade on first useaudit writerone line per callrunnerevery subprocessfilesruntime dir beside niri's socket:engine.sock · lease · lease.jsonstop · input-dirtystate dir: audit.jsonlniriIPC socket: requests, event streamWayland: virtual devices, data-controldesktop servicesAT-SPI bus · Noctalia IPC socket · logindchildrengrim · notify-send · loginctlguard: releases held input if the engine diespaste-keeper: one per paste

scroll sideways for the rest →

The lease, the stop flag and the input-dirty marker are files in a directory beside niri’s socket. Every server on the same niri sees the same files, in either mode.

One action

Section titled “One action”

Looking needs no lease. Every action passes the same gate first, under the action mutex, so one server’s actions run one at a time. A refusal at the gate is an error, and nothing was sent. Once the request is on its way, a stop can’t take it back: the result is uncertain, with accepted true if niri answered and null if it didn’t. Otherwise the result has accepted, observed, changes when niri saw something change, and a screenshot when the agent asked for one or the outcome is in doubt. Every call writes one line to the audit log, refusals included.

One action's lifeA tools/call runs raced against the client's cancel. The gate, under the action mutex, checks the stop flag, the input-dirty marker and the lease, asks the readiness report about niri's version, the policy and the lock state, then checks the stop flag, the marker and the lease file again. The tool checks its arguments against niri's state, the deny list and expect. It sends the request once the lock state was read at most 250 ms before and the desk is told. accepted is true once niri answers. observed comes from niri's events, up to five seconds. Then a screenshot if asked or in doubt, the changes, and one audit line. A refusal or stop before sending is an error with nothing sent; a stop or lost lease after sending makes the outcome uncertain with no screenshot and no changes.gate · Desk::actstop flag · input-dirty · leasereadiness: niri version, policy, lockstop flag and marker again · lease filetools/callraced against the client's canceltool's checksarguments against niri · deny list · expectsendlock read ≤ 250 ms old · Desk::sendingacceptedtrue once niri answers, null if the reply is lostobservedeach event applied to the waiter, up to 5 sscreenshotsettled if asked, at once if in doubtchangesniri's scene before and after, once quietaudit lineone JSON line per call, refusals included→ result: accepted · observed · changes · screenshotrefused or stopped here:an error, nothing sentstopped or lease lost here:uncertain, no screenshot,no changespast the stop race,action mutex still held

scroll sideways for the rest →

The drawing follows a niri action such as focus. Keys and text go to the server’s own virtual keyboard, clicks to a virtual pointer, and element actions to the app over the accessibility bus, all behind the same gate.

Lease, stop and recovery

Section titled “Lease, stop and recovery”
Lease, stop and recovery statesFree: acquire_desktop, with no stop flag, no marker and the policy allowing it, locks the lease and is pending; with confirm_lease it asks you. Granted, after Allow or with no prompt, the lease is held by one session; refused, declined, stopped or with the client gone, it is free again. release_desktop, the client going, the lease file moving or the server dying frees it. The stop flag, set while held, makes the watcher give the lease up; stop sets it any time and resume clears it, but not while input-dirty is set. Input that may be stuck sets the input-dirty marker, which refuses every action until the holder releases and a human runs recover.acquire_desktopno stop flag, no marker,policy allowsrefused, declined,stopped or client gonegrantedAllow, or no promptrelease_desktop · client gonelease file moved · server diedstop flag set,lease given upinputmay bestuckstopresume,not while input-dirtyrelease_desktop, then a human runs recoverfreeno lease heldpendinglocked, not grantedheldby one sessionstoppedstop flag setrecovery requiredinput-dirty setconfirm_lease asks you

scroll sideways for the rest →

acquire_desktop locks the lease file before it asks anything slow, so an agent on another server gets lease_held at once. Another session of the same engine waits until the first acquire_desktop is done, which with confirm_lease can take up to a minute, and then gets lease_held, or the lease if the first was refused. With confirm_lease on, a notification asks you to Allow or Deny before the lease is granted. niri-computer-use stop sets the stop flag: the running action is cancelled and the lease is taken back. resume clears the flag, except while the input-dirty marker says a key or button may be stuck. Then recover releases what the marker names and asks you to check before it clears the marker.

Coordinates

Section titled “Coordinates”

A click names a pixel of a screenshot, by its screenshot_ref, or an element from elements. Each space is its own type in the code, so a pixel can’t be passed where a layout point is expected. Before anything moves, the server maps the final point back through niri’s own formula and refuses it if it lands more than 0.002 logical pixels away.

Coordinate spacesAn ImagePx, a pixel of a screenshot_ref, becomes a LayoutPt in niri's layout: its centre divided by the capture scale plus the captured rectangle's origin. An element_ref enters as a LayoutPt too, the centre of its box from niri's layout and its AT-SPI extents, checked again when used. The ref must be under 60 seconds old, on the same event stream, with its output unchanged, and the point inside the capture; then the output's origin is subtracted to give an OutputLocalPt. Undoing the output's transform and scaling by 1000 gives a ProtocolPt, the arguments of motion_absolute, which go to a virtual pointer bound to the screenshot's output once niri's own formula maps them back within 0.002 logical pixels.ImagePxa pixel of a screenshot_refLayoutPtniri's layout, logical pxOutputLocalPton the output as shownProtocolPtmotion_absolute argumentspixel centre ÷ capture scale+ captured rectangle's originref under 60 s, same event stream,output unchanged, point inside thecapture; then − the output's originundo the output's transform,× 1000, clamped inside the outputelement_refits box: niri + AT-SPI extentschecked againvirtual pointerbound to the screenshot's outputniri's formulalands ≤ 0.002 px

scroll sideways for the rest →

A screenshot ref expires after 60 seconds, when niri’s event stream reconnects, or when its output’s position, size, scale or transform changes. An element ref is checked against the app again each time it is used, and the pointer aims at one only while its window is floating and in view.