dsh-local-vision

Give a text-only DeepSeek Harness agent local "eyes". This plugin registers a model-callable local_vision tool that describes an image using a local vision model through Ollama. The image never leaves your machine — no cloud, no uploads, no API keys.

Requirements

• Ollama running (default http://localhost:11434).

• At least one vision-capable model pulled, e.g.:

  ollama pull qwen2.5vl:3b        # small, fast — text / colors / summary
  ollama pull gemma4:12b-it-qat   # larger — thorough description
  

Install

# from npm (recommended)
npm install -g @deepseek-ai/dsh
dsh plugin --profile web add dsh-local-vision

# or from this repository (use `file:` so @deepseek-ai/dsh-tools resolves)
dsh plugin --profile web add file:/path/to/deepseek-harness-plugins/dsh-local-vision

Restart the DSH session so the new bundle is composed. The package then appears in Settings → Plugins → Plugin list and the tool local_vision is available.

Usage

local_vision(image: "/abs/path/screenshot.png", mode: "fast")
local_vision(image: "/abs/path/mockup.png", mode: "detailed", prompt: "Describe the navigation and buttons")
local_vision(image: "screen", mode: "detailed")   # capture the current display (macOS screencapture)

Modes

mode	intent	default model
fast	transcribe visible text, name dominant colors, one-line summary	qwen2.5vl:3b
detailed	full factual description: layout, UI elements, all text, colors, state	gemma4:12b-it-qat

Parameters

Param	Type	Notes
image	string (required)	Absolute path to a PNG/JPEG/WebP/GIF, or the literal screen.
mode	"fast" | "detailed"	Defaults to fast.
prompt	string	Optional focused question. Defaults per mode.
model	string	Optional override of the Ollama model id for this call.

Configuration

Optional, at $DSH_HOME/local-vision.json (default ~/.dsh/local-vision.json):

{
  "host": "http://localhost:11434",
  "models": { "fast": "qwen2.5vl:3b", "detailed": "gemma4:12b-it-qat" },
  "temperature": 0.1,
  "timeoutMs": 300000
}

All keys are optional; defaults are shown above. The plugin sends the image to Ollama's OpenAI-compatible /v1/chat/completions endpoint as a base64 data URI.

Notes & limitations

• The first call after the model is cold loads it into memory (a 7 GB model takes a while); subsequent calls are fast.
• image: "screen" requires macOS and Screen Recording permission for the terminal.
• The tool calls Ollama directly (not through the DSH sandbox executor).

License

MIT