Home

Software · desktop companion · Windows

Jarvis Desk

A small companion that lives beside my cursor. It reads the window I'm looking at, points at what I need, draws its answers next to the text and talks back in a British voice. Private things stay on the laptop.

Default braina local model
Leaves the laptopnothing, by default
Reading a window31 controls in 0.1 s
Notes on a confocal microscope in Notepad. A formula in the text is circled, and the companion answers beside it: the resolution is about 178 nm, with the worked calculation and a badge that says Local, nothing left this computer.
I circled a formula in my notes and asked. The answer sits next to it, with the working, and the badge says which brain answered.

Part of Work · What I'm doing

Most assistants live in their own window. You copy the question over and read the answer somewhere else. I wanted one that comes to the page I'm already reading.

Jarvis Desk sits next to the mouse pointer on Windows. I ask by voice or by typing, and it answers in a small bubble. When a picture says it better, it draws one.

01 · What it does

It points, draws and remembers

Pointing and walkthroughs

Ask how to do something and it rings the first button, reads the step aloud and waits for my click before it moves on. It knows where the button is because it reads the window, not because it guesses from a picture.

A two-step walkthrough in Notepad. The Edit menu is ringed in amber, a label reads Step 1: press Edit, and a card with Back and Next sits beside the companion.
Step one of two: the button is ringed, and the next step waits for my click.

Drawing beside what I'm reading

It has one drawing tool with many kinds of picture: bar and line charts, timelines, tables, flow charts, worked maths, geometry, molecules, small 3D models with moving parts, maps, quizzes and chess positions. It can also underline words on the screen itself. Each drawing gets a short name, like d3, so I can say "change d3" or "clear them".

Beside the notes: a bar chart of lateral resolution for four objectives, a line chart of resolution against wavelength, and a timeline of confocal microscopy.
Charts and a timeline, drawn from the numbers in my notes.
A flow chart of the four steps to take a stack, a table of three objectives with their numerical aperture and resolution, and worked steps that reach about 178 nm.
The steps as a flow, the numbers as a table, and the working.
A cone of light explaining numerical aperture, the structure of fluorescein drawn from its formula, and a multiple-choice question about the pinhole.
A cone of light, a dye molecule and a quick question to check I understood.
A 3D piston on its crank, a map of three European synchrotrons, and a chess position.
A 3D piston that turns, a map and a chess position.

Circle to ask

I hold one key, talk, and circle something with the mouse while I talk. It looks at that part of the screen and answers about it. Dragging a box and typing the question works too.

Two lines of the notes are framed by amber corner marks labelled This area, while the companion's bubble says Thinking.
The circled area is framed while it thinks. The answer is at the top of this page.
Two terms in the notes are underlined in amber with a handwritten note, these two set the resolution, and a labelled picture of the window marks what it does, resolution and the steps.
Underlines on the page itself, and a labelled picture of the window.

Reminders and memory

Reminders are set in plain words: "every weekday at 9", "every 2 hours", "friday 5pm", "in 1h 30m". It also remembers preferences and facts I mention. Everything it remembers is listed in its panel, and one click forgets it.

02 · The characters

Four characters, each with its own voice

Jarvis is an optical aperture, Hargreaves a line-drawn butler with a monocle, Galahad a chess knight and Ada a small orrery. Each has a British voice and its own way of moving. They breathe when idle, open up while I talk, turn while they think, ripple while they speak and snap into focus when they point. With reduced motion switched on in Windows they keep still and only the colours change.

Jarvis, an aperture with six blades, answering a question about lateral resolution.
Jarvis, the aperture.
Hargreaves, a line-drawn butler with a monocle, saying good evening and giving the pinhole size.
Hargreaves, the butler.
Galahad, a chess knight, giving the chess drill for today.
Galahad, the knight.
Ada, a small orrery, explaining a step in the notes.
Ada, the orrery.
The four characters, each shown listening with a red dot and thinking with a blue dot.
Listening (red dot, the microphone is live) and thinking (blue), for each of the four. They come in four sizes, down to a tiny one that rides the mouse pointer.

03 · How it works

A small app that reads the screen like a person would

  1. The app. Tauri, with the core in Rust and the interface in TypeScript. It installs for one user, needs no administrator rights and starts without a console window.
  2. Reading the screen. When I ask, it captures the window in front of me, reads its controls through Windows UI Automation (the names and positions of buttons, menus and fields) and runs Windows OCR on the picture for everything else. Each control and each line of text gets a short id, so when it points, it points at the real thing. On a test window it read 31 controls in about 0.1 seconds.
  3. The brains. A local model on the laptop answers by default, and then nothing leaves the machine. For hard questions, like a long derivation or a plan, a stronger cloud model can answer, only when I choose. Questions that mention passwords, banking, health or salary always stay local. Under every answer a small badge says which brain answered and what left the laptop.
  4. Voice. Speech recognition and the voice both run on the laptop. The voice downloads once, about 90 MB, and then works offline. The recording is deleted after use.

Safe by default

  • Reading runs straight away. Anything that changes something first shows a card with "Allow" and "Not now". No answer in 60 seconds means no.
  • Clicking, typing, opening apps, editing files and running commands are all off until I switch them on, and each one still asks.
  • It never sends, shares, submits or deletes anything for me.
  • A hands-off list keeps it out of password managers, banking and terminals.
  • The companion and its panel are hidden from screen capture and screen sharing, so they don't show up in recordings or meetings.
  • No analytics, no telemetry, no crash reports and no update check.
The settings panel: doing things for you, all off until you switch them on, and a hands-off list of password managers, terminals and banks that are never clicked or typed in.
Actions start switched off. The hands-off list is in plain text, so I can add to it.

Tested

65 Rust tests cover the safety rules, memory, geometry and the tools, and 41 TypeScript tests cover the interface logic.

Rust · Tauri · TypeScript · Windows UI Automation · Windows OCR · KaTeX · local speech in and out

Status: a personal tool on my own laptop. Not released yet.

See my assistant, Argus← All my work