dsh-gui-browser

An operable, live view of the Session's built-in Chromium in the GUI right Sidebar — the same browser the model's mcp__playwright-mcp__* tools drive.

Requirements

• DSH 0.2.0-rc.2 or a later 0.2.x. Declared twice — engines.dsh and the @deepseek-ai/dsh-* peers — so the plugin manager refuses the install on an older runtime and boot denies this row instead of failing halfway through; dsh plugin allow-version is the deliberate, per-version override.
• Node ^22.19 || >=24.
• A profile built on @deepseek-ai/dsh-base and @deepseek-ai/dsh-web-app: the pane is a Web right-Sidebar tab.
• A browser-use bundle. This package has no browser of its own. It depends on the provider packages (@deepseek-ai/dsh-browser-use and @deepseek-ai/dsh-experimental-browser-use-playwright-mcp) so the provider rows resolve, and its patch overrides the Playwright MCP provider's row to point it at the launcher — but the rows themselves, and the isolated headless Chromium behind them, come from a browser-use bundle enabled earlier in the layer order (this repository's sibling dsh-browser-use-playwright is exactly those two rows). A profile admits one provider at a time: @deepseek-ai/dsh-browser-use owns the single registration slot and rejects a second one, so never enable two. Without any, the pane stays in its idle state and says so.

Install

dsh plugin --profile <profile> add dsh-gui-browser
dsh --profile <profile> --dump-config   # the layer is listed as dsh-gui-browser

The package's postinstall (launcher/install.mjs) puts the platform launcher in the registry directory; see The launcher. pnpm ≥10 runs no dependency lifecycle script until it is allowed, so the install reports a pending build script and the plugin manager asks for it — approve it, because without that step there is no launcher.sh/launcher.exe for the provider to launch. The step is idempotent and never fails an install: on a machine with no C# compiler it warns loudly and leaves the Windows launcher absent rather than breaking the package.

dsh plugin --profile <profile> remove dsh-gui-browser removes the dependency and its layer.

Installing into a running Harness needs a restart. The plugin manager tries to activate a newly installed bundle through HMR, but a Host answers plugin specifiers from the package table it built at startup: a package name it has not seen before is reported as failed to import and mounts only after a restart (measured — the same process imports a pre-boot name, and the resolution a fresh boot computes points at the right directory). The pane then appears on its own, and the browser it mirrors keeps running across the restart because the browser's debugging port is recorded in the registry directory rather than held by the plugin. Replacing a package in place has the same requirement.

To publish from a checkout, npm pack builds the tarball; its contents are exactly files in package.json — sources, the launcher's .cs/.mjs/.sh, and never a compiled binary.

What it does

• Same pixels as the model sees. The Host half attaches to the browser Chromium was launched with (--remote-debugging-port, added by wrapper.sh) and pushes Page.startScreencast frames of the page target. Nothing is re-rendered, re-laid-out, or resized: the viewport, the page state, and the focus are whatever the browser has. One thing is corrected: a page Chromium created on its own (the new tab after a browser's last page closes, or a popup a site opened) is born at Chromium's unsized 800x600 default, which is narrower than most pages and arrives cropped, so any page still at exactly that size is set to the browser's own viewport (1280x720). Any other size — the model's own resize, a site's popup size — is left alone.
• Frames only while somebody is watching. Encoding a frame is the most expensive thing here, so frame production follows demand: it starts when a viewer attaches, stops when the last one detaches, and stops again while a viewer says its window is hidden. A hidden pane is sent no frames at all.
• One shown page at a time. A browser with several tabs paints all of them; casting every one is what once made the pane alternate between tabs. Only the shown page is cast, and it follows what a browser would do: a page that appears and lands — including one born about:blank and named a moment later — takes the view, and a page chosen by hand from the page menu keeps it until another page appears or that one closes.
• Chrome's furniture. Tabs live in a strip above the toolbar, oldest on the left, each with its own ✕, and a + opens a new tab page and puts the cursor in the address bar; the strip scrolls sideways with the wheel. The address bar shows the shown page's URL and can be typed into — a scheme is taken at its word, a bare host gets https://, and anything else becomes a Bing search, the same three-way guess Chrome makes. Everything the pane does to a page is reported to the model; the frame-quality rows in the ⋮ menu are the viewer's own preference and are not.
• Back and Forward mean what they mean in a browser. They step inside the shown page while it has history of its own, and otherwise across the pages the viewer has been shown: a tab opened by a link holds a history of exactly one entry (measured on bilibili: index=0/0), so Back there restores the page the link was clicked from instead of doing nothing. Alt+Left and Alt+Right are the same two actions while the pane has focus, and the toolbar dims a direction that would do nothing.
• Frames are scaled to the pane, not to the page. A screencast frame is always the whole page, so every viewer tells the Host how many device pixels its pane draws the page at (fit mode: the fitted size; 1:1: the page at the display's density, capped at the page's own pixels). The widest pane wins, because a frame can be scaled down but never back up, and the cast is restarted — debounced by 200 ms, so a drag that resizes the pane settles into one restart per pause — when that width no longer matches. A pane is therefore never sent pixels it scales away again: measured on a 1280x720 page being scrolled, casting at 160x90 instead of full size took the same browser from 1.69 cores / 81 Mbit/s to 0.76 / 1.2, and a pane that reports nothing at all still gets the full page.
• Low delay, not high frame rate. Frames are pushed, never polled, and each viewer acknowledges a frame as it arrives and decodes it behind that acknowledgement, so the Host encodes the next frame while the previous one is still decoding instead of waiting for the draw — the two costs no longer add up. At most one frame can ever be waiting, so a viewer that falls behind drops frames instead of accumulating a queue. Measured page change → viewer paint is ~20 ms locally.
• Real interaction. Mouse (move, drag, click, right/middle click, wheel) and keyboard travel as CDP input events into the page. Typing goes through an invisible editable surface, so the client's IME (Microsoft Pinyin, Japanese, dead keys, emoji panel) composes locally and its committed text is inserted upstream, exactly the way a remote-desktop client handles IME — including the key that commits a candidate, which the page must never also receive. The Host measures the page's real caret and publishes it, so the platform's candidate window appears over the field being typed into. Ctrl+C also copies the page's selection into the client's clipboard.
• Reported to the model. What the viewer does is coalesced into a minimal record (clicks, type"...", keys, scrolls, navigation) enriched with the page's URL, title, and focused element, and contributed as one runtime-context line to that Session's next model step.

Routes

Loopback callers only.

Route	Meaning
GET /gui-browser/list	every live browser's state
GET /gui-browser/status?port=	one browser's state
GET /gui-browser/frame.jpg?port=	newest frame as a still image
GET /gui-browser/activity?session=	the exact text the model would receive
GET /gui-browser/stream?port=	MJPEG fallback for a non-WebSocket viewer
POST /gui-browser/input?port=&session=	the same commands the socket takes
WS /gui-browser/ws	frames and state downstream; input, caret, page choice, frame quality, and pane visibility upstream

Files

• host.js — stable entry; imports engine.js?rev=<mtime> so a Host-process module cache cannot pin an old generation for the life of the process.
• engine.js — registry watcher, CDP plumbing, WebSocket transport, input translation, and the model-facing activity record.
• client.js — the gui-browser right-Sidebar tab: canvas view, socket, pointer/keyboard/IME handling, zoom, auto-open.
• wrapper.sh — the POSIX Chromium launcher: allocates a free debugging port per launch and publishes it under ~/.dsh/run/gui-browser/.
• launcher/launcher.cs, launcher/install.mjs — the same launcher for Windows and the install step that puts either one in place.

The launcher

Playwright is pointed at a launcher through the provider's executablePath, because that is the only hook a bundle has for adding --remote-debugging-port beside Playwright's own control channel. The patch names it as dshHomePath('run', 'gui-browser', 'launcher.exe' | 'launcher.sh'), so it never has to know where the package was installed, and the bundle's postinstall (launcher/install.mjs) puts the right launcher for the platform in that directory:

• POSIX: wrapper.sh copied to launcher.sh, mode 755.
• Windows: launcher.cs compiled to launcher.exe by the csc.exe that every .NET Framework install provides, with PowerShell Add-Type as the fallback. Nothing prebuilt ships in the package, so there is nothing to sign or build per architecture; a machine with no C# compiler is reported loudly and otherwise ignored, and the model keeps its browser either way.

Two things about the Windows launcher are load-bearing. Playwright spawns executablePath without a shell, so a .cmd cannot be used at all (spawn EINVAL); and it hands the browser its DevTools pipe as CRT file descriptors 3 and 4, which Windows passes in STARTUPINFO.lpReserved2 — a per-process descriptor table the child's CRT reads at startup. The launcher therefore calls CreateProcess itself, forwards that block unchanged, and marks the handles it names inheritable; Process.Start would lose the block and Chromium would refuse to start with "Remote debugging pipe file descriptors are not open". launcher.exe --gui-browser-handles prints the inherited table, which is how that failure is told apart from a broken browser.

Limits

• Ctrl+wheel (page zoom) is not forwarded: it would desync the pane's geometry from the browser's viewport.

• A browser showing this GUI itself is view-only, so the nested panel cannot forward its own clicks forever.

• The viewer's clipboard reaches the page on paste, and the page's selection reaches the viewer on Ctrl+C; richer clipboard flavours are not bridged.

• Chromium's frame pacing is left alone by default. Measured here: the default 60 Hz frame clock gives 60 fps repaint, 59.3 fps capture, 60 fps delivery with no dropped frames — the pipeline is not the limit, the frame clock is. --disable-frame-rate-limit --disable-gpu-vsync raises capture to ~114 fps but drives requestAnimationFrame to ~3240 Hz, changing every animation's timing for the model as well. To opt in for the next launch, create a file named uncapped in the registry directory (~/.dsh/run/gui-browser/); delete it to go back. Page restarts are needed, so the browser's current tabs are lost.

• Lossless frames are a per-viewer choice, off by default, because they cost the frame rate the pane exists for. Measured at 1280x720 on bilibili while it is being scrolled and repainted, from the browser's own process tree:

  encoding	frames/s	KB/frame	Mbit/s	cores
  JPEG 88 (default)	55.1	184	83	1.17
  JPEG 100	52.3	485	208	1.75
  PNG (lossless)	17.5	768	110	1.84

  A static page emits no frames at all in either mode, and the model's own screenshots are a separate on-demand capture: at the same moment a full-quality screenshot of that page measured 203 KB as JPEG and 654 KB as PNG, so the mirror's encoding never touches what the model sees. Those figures are for a full-size cast; because the cast now follows the pane, lossless costs a narrow pane far less than the table suggests — measured the same way, a 160x90 screencast held 60 fps at 2 KB a frame and 1.2 Mbit/s.