Skip to content

dsh-ab-knowledge

Verified

dsh-ab-knowledge Β· v0.1.1 Β· MIT

A **self-contained knowledge-base service** for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).

Install

dsh plugin add dsh-ab-knowledge

Confirm the layer applied with dsh --profile default --dump-config β€” see the install guide.

Source

Published to npm without a public repository. Inspect the package contents before installing.

Tags

Readme

dsh-ab-knowledge

A self-contained knowledge-base service for DeepSeek Harness.

It reimplements WeKnora's knowledge-base capabilities locally β€” chunk, embed, index, hybrid (vector + keyword) retrieval with Reciprocal Rank Fusion, and cited question-answering β€” using only Node 22's built-in node:sqlite (FTS5) and a local Ollama model.

No WeKnora required. The agent no longer needs a running WeKnora instance, a Go backend, Qdrant, Postgres/pgvector, or any other external service. The whole knowledge pipeline runs in-process inside the harness.


Why this plugin exists

WeKnora ships an official DSH plugin (@wxg-prc-cpg/dsh-weknora) that exposes a thin client over a live WeKnora server. That keeps the agent dependent on a Docker Compose stack (postgres + redis + docreader + app + frontend) just to answer questions over documents.

This plugin instead extracts the knowledge-base engine itself out of WeKnora's source and runs it locally. The read-side tool shapes (list_knowledge_bases / search / read_document / ask) are kept byte-for-byte compatible with the official client plugin, so existing tool-calling behaviour transfers; the write/manage tools (create_knowledge_base / add_document / add_documents_from_dir / delete_document / delete_knowledge_base) make ingestion and lifecycle fully local β€” no upload UI, no server, no API key.

Capabilities reimplemented (faithful to WeKnora)

Capability WeKnora source This plugin
Recursive character chunking internal/infrastructure/chunker src/chunker.ts
Default 512 chars / 80 overlap DefaultChunkSize=512, DefaultChunkOverlap=80 chunkSize, chunkOverlap
Separators \n\n > \n > 。 strategy.go separators
Semantic (vector) embedding Ollama /api/embed src/embed.ts
Keyword (FTS5 bm25) retrieval chunks_fts src/store.ts
Hybrid fusion knowledgebase_search_fusion.go fuse() + fuseWithRRF
RRF k=60, vector 0.7 / keyword 0.3 retrieval_config.go rrfK, rrfVectorWeight, rrfKeywordWeight
Over-retrieval pool TopK=50 DefaultRetrievalTopK=50 retrievalTopK
Optional rerank (TopK=10, thr 0.2) RerankTopK, RerankThreshold rerankTopK (threshold via cosine)
Cited knowledge-QA /knowledge-chat src/qa.ts
Locate a document by name findDocuments findDocumentsByName
Document parsing (bytes β†’ markdown) docreader/parser/* src/parsers.ts (routed by extension, like registry.py)

Local document parsing β€” the docreader, without WeKnora

WeKnora's docreader microservice turns uploaded bytes into markdown text (base_parser.parse_into_text); the Go app then chunks and embeds it. This plugin reimplements that first step locally in src/parsers.ts, routed by file extension exactly like WeKnora's registry.py. Chunking, embedding, retrieval and QA are already the engine's job (table above), so the boundary matches 1:1.

Pass a file via knowledge_add_document's file_path (absolute, local path) and it is parsed before chunking β€” no upload UI, no server, no API key.

To ingest a whole folder at once, use knowledge_add_documents_from_dir with dir_path (plus optional recursive β€” default true, extensions to narrow formats, and title_prefix). Every supported file is run through the same parse β†’ chunk β†’ embed β†’ index pipeline; the result reports scanned / indexed / skipped counts with a per-file breakdown, so one unreadable file is reported under skipped without aborting the rest of the batch.

Format Parser Notes
.txt .md .csv .json .log native read as UTF-8 text
.pdf pdf-parse (or PaddleOCR-VL OCR) digital PDFs use the text layer; scanned/image-only PDFs are OCR-ed when ocrProvider: paddleocr_vl is set (see below) - otherwise they yield little text
.docx mammoth β†’ turndown DOCX β†’ HTML β†’ Markdown (headings/tables preserved)
.xlsx .xls xlsx (SheetJS) each sheet β†’ a Markdown table
.pptx .ppt officeparser slide text
.html .htm .mhtml turndown script/style stripped, converted to Markdown

Scanned / image-only PDFs are handled by an optional, self-hosted PaddleOCR-VL OCR step (see Local PaddleOCR-VL OCR below). When ocrProvider: paddleocr_vl is configured with your local endpoint, every PDF - digital or scanned - is sent through it and the returned markdown (OCR text + layout) is what gets chunked. With ocrProvider: none (default) we fall back to the pdf-parse text layer, which is empty for scanned pages.

Parser libraries are loaded lazily and bundled by tsdown, so the plugin stays self-contained β€” no docreader container, no MinIO, no WeKnora. The only runtime dependency beyond Node is a local Ollama for embeddings/QA.


Requirements

  • Node β‰₯ 22.19 (the plugin uses the built-in node:sqlite + FTS5 virtual tables β€” bundled since Node 22.5 for in-tree builds, GA in 22.19 / Node 24).
  • Ollama reachable at the configured embedBaseUrl and qaBaseUrl (default http://localhost:11434):
    • an embedding model, default qwen3-embedding:4b (2560-dim; pull it with ollama pull qwen3-embedding:4b); nomic-embed-text (768-dim) also works - keep embedDim in sync;
    • a chat model for QA, e.g. qwen2.5:7b.
  • Both providers can be set to none to degrade gracefully:
    • embedProvider: none β†’ keyword-only (FTS5) retrieval, fully offline;
    • qaProvider: none β†’ ask returns the retrieved passages as references only.

Install

# from the harness checkout / profile
dsh plugin --profile web add dsh-ab-knowledge

The bundle patch (cordis.patch.yml) contributes the plugin row id: knowledge and points the harness at the built code. Build with:

pnpm install
pnpm build        # tsc -p tsconfig.json && tsdown

Tools

All read tools are prefixed with knowledge_ by default (configurable via toolPrefix).

Tool Kind Purpose
knowledge_create_knowledge_base write Create a named collection of documents. Returns its id.
knowledge_list_knowledge_bases read List existing bases with ids.
knowledge_add_document write Ingest text or a single local file (file_path); the file is parsed (PDF/DOCX/XLSX/PPT/HTML/MD/TXT/CSV/JSON) then chunk β†’ embed β†’ index.
knowledge_add_documents_from_dir write Bulk-ingest every supported file in a directory (recursive) through the same pipeline as add_document. Returns scanned / indexed / skipped counts plus a per-file breakdown.
knowledge_search read Hybrid retrieval; verbatim passages + by-name document matches.
knowledge_read_document read Page through a document's stored chunks in order.
knowledge_ask read Retrieve + compose a cited answer; multi-turn via session_id.
knowledge_delete_document write Remove a document and its chunks/FTS/embeddings.
knowledge_delete_knowledge_base write Remove a base and all its documents.

Tool output shapes

The four read tools mirror @wxg-prc-cpg/dsh-weknora exactly:

  • search β†’ { query, knowledge_base_ids, count, results[], documents[] } where each results[] item carries rank, chunk_id, knowledge_id, document, chunk_index, score, content, truncated; each documents[] item carries knowledge_id, title, knowledge_base_id, knowledge_base.
  • ask β†’ { answer, session_id, pipeline, tool_calls[], references[] } with [n] citations into the referenced passages.
  • read_document β†’ { knowledge_id, title, summary, page, page_size, total_chunks, returned_chunks, has_more, truncated, content }.
  • list_knowledge_bases β†’ { count, knowledge_bases[] }.

Configuration

Every tunable lives in one place: the config: block of the bundle patch (cordis.patch.yml), validated against src/config.ts's Config schema. A deployment edits behaviour there and never in the TypeScript source.

Key Default Meaning
dataDir knowledge-base Where knowledge.sqlite lives (relative to harness cwd).
embedProvider ollama ollama or none (keyword-only).
embedBaseUrl http://localhost:11434 Ollama embedding endpoint.
embedModel qwen3-embedding:4b Embedding model id (default; 2560-dim).
embedDim 2560 Embedding dimensionality (must match model).
embedTimeoutMs 8000 Per-embed-request ceiling.
qaProvider ollama ollama or none (references only).
qaBaseUrl http://localhost:11434 Ollama chat endpoint.
qaModel qwen2.5:7b Chat model for cited answers.
qaTemperature 0.1 Sampling temperature.
qaMaxTokens 2048 Max answer tokens.
qaTimeoutMs 300000 Per-QA-request ceiling.
defaultKnowledgeBaseIds [] Scope used when a call names none (empty = all).
chunkSize 512 Chunk size in chars (WeKnora default).
chunkOverlap 80 Overlap in chars (WeKnora default; capped at half size).
separators ['\n\n','\n','。'] Recursive splitter order (WeKnora default).
rrfK 60 RRF smoothing constant (WeKnora default).
rrfVectorWeight 0.7 Vector retriever weight (WeKnora default).
rrfKeywordWeight 0.3 Keyword retriever weight (WeKnora default).
retrievalTopK 50 Over-retrieval pool before the final cut (WeKnora default).
rerankTopK 0 Optional rerank depth by query↔chunk cosine (0 = off).
ingestConcurrency 4 Max concurrent file ingestions for add_documents_from_dir (bounds host I/O and Ollama embed calls; 1–16).
maxResults 8 Passages a search/ask returns.
maxChunkChars 1200 Per-chunk truncation budget in tool output.
requestTimeoutMs 30000 Non-streaming tool timeout.
chatTimeoutMs 300000 QA call timeout.
toolPrefix knowledge Tool-name prefix.
ocrProvider none none or paddleocr_vl (local PaddleOCR-VL OCR for scanned PDFs).
ocrEndpoint http://localhost:18080 Base URL of your self-hosted PaddleOCR-VL serving (e.g. http://localhost:18080).
ocrTimeoutMs 600000 Per-OCR-request ceiling (scanning PDFs are slow).
ocrUseSealRecognition true PaddleOCR-VL: detect and OCR seals/stamps.
ocrUseChartRecognition false PaddleOCR-VL: convert charts into markdown tables.
ocrMinPixels 147384 PaddleOCR-VL VLM minimum input pixel count.
ocrMaxPixels 2822400 PaddleOCR-VL VLM maximum input pixel count.
persistMarkdown true Persist the OCR'd / parsed markdown to disk (one .md per source file) before vectorizing; the file keeps its <!-- page: N --> page markers for verifiability, then the markers are stripped from the in-memory copy that is chunked/embedded. false skips the sidecar.
markdownDir knowledge-base/markdown Directory for the persisted .md files; relative paths resolve against the harness workspace root (DSH_HOME, else cwd).
tools all true Per-tool registration switches.

How retrieval works

query
  β”‚
  β”œβ”€β–Ί FTS5 bm25 over chunks_fts  ──────────────┐
  β”‚                                            β”œβ”€β–Ί fuse()  (WeKnora RRF:
  └─► Ollama /api/embed β†’ cosine vs stored β”€β”€β”€β”€β”˜          vW/(k+vRank) + kW/(k+kRank))
        embeddings (JSON column)                            β”‚
                                                          β”œβ”€β–Ί optional rerank by query↔chunk cosine
                                                          β”‚
                              search hits β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                  β”‚
                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
              search tool                       ask tool
        (verbatim passages)              (QA model composes a cited answer)
  • Vector-only results keep their raw cosine scores; keyword-only bm25 is rescaled to [0,1] (prevents BM25 saturation from dominating the fused rank, matching WeKnora's fuseWithRRF).
  • Embedding is an accelerator, not a hard dependency: if Ollama is unreachable, ingestion still completes (keyword retrieval keeps working) and a warning is logged.
  • All SQL is parameterised β€” no string interpolation of user input.

Local PaddleOCR-VL OCR (scanned PDFs)

Scanned / image-only PDFs have no text layer, so pdf-parse returns almost nothing for them. WeKnora routes those through a self-hosted PaddleOCR-VL service on the Go side; this plugin reproduces that exact client contract (src/ocr.ts -> POST {endpoint}/layout-parsing, the same request/response shape WeKnora's paddleocr_vl_converter.go uses) so you can OCR locally without WeKnora and without any cloud.

1. Deploy a local PaddleOCR-VL serving endpoint

PaddleOCR-VL is served by PaddleX; the paddlex CLI exposes a /layout-parsing REST endpoint whose protocol this plugin already speaks (src/ocr.ts -> POST {endpoint}/layout-parsing).

Prerequisites (read first):

  • OS: PaddleOCR-VL does not run on native Windows. Run every command below inside WSL2 (Ubuntu) or a Docker container with --gpus all. (You are on Windows, so Git Bash / CMD will not work for these steps.)
  • GPU: an NVIDIA card, CUDA 12.x, compute capability >= 7 (RTX 3060 / A10 / A100 and up). CPU inference is technically possible but impractically slow.
  • The OCR service runs entirely on your machine - no data leaves your network.
# Inside WSL2 Ubuntu (or the GPU Docker container):

# 1. PaddlePaddle GPU wheel (match your CUDA; 3.3.0 / cu126 shown)
python -m pip install paddlepaddle-gpu==3.3.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
# 2. PaddleX with the OCR pipeline (NOT "paddleocr[ocr_vl]")
python -m pip install "paddlex[ocr]"
# 3. serving plugin
paddlex --install serving

# 4. Generate the full pipeline config (first run only)
paddlex --get_pipeline_config PaddleOCR-VL -o D:/ai-service/paddleocr_vl/pipeline_full.yaml

# 5. Merge in our Serving overrides (visualize off + lift the PDF page cap)
cd D:/ai-service/paddleocr_vl && python3 merge_config.py

# 6. Start the service on :18080 (GPU strongly recommended)
paddlex --serve --pipeline D:/ai-service/paddleocr_vl/paddleocr_vl.serving.yaml --host 0.0.0.0 --port 18080

Quick path (skip the config merge): paddlex --serve --pipeline PaddleOCR-VL-1.6 starts the service on :18080 with defaults. Use the merged config only if you need to lift the PDF page cap or disable annotated output images.

The endpoint is now http://localhost:18080. Verify it is up:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/layout-parsing
# -> 405 (method not allowed on GET) means the route exists; 200/422 on a real POST means ready

Port note for WSL2: if the service binds inside WSL2, reach it from Windows-side tooling via http://localhost:18080 (WSL2 forwards localhost). If you set ocrEndpoint inside the DSH process running on Windows, use the same localhost:18080.

Docker (recommended on Windows)

A single-container image that runs the PaddleX native backend and serves POST /layout-parsing on :18080 (no separate vLLM container required). The entrypoint generates + merges the config on first boot, so you only run one command.

Prerequisites: Docker Desktop with the WSL2 backend, an NVIDIA GPU with the current driver (WSL2 shares the Windows driver), and the NVIDIA Container Toolkit enabled. Confirm GPU passthrough first:

docker run --rm --gpus all nvidia/cuda:12.6.0-runtime-ubuntu22.04 nvidia-smi
# -> should print your GPU; if it errors, fix the toolkit before continuing

Build & run (from D:/ai-service/paddleocr_vl):

cd D:/ai-service/paddleocr_vl
docker compose up -d --build
docker compose logs -f           # wait for "Uvicorn running on http://0.0.0.0:8080"
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/layout-parsing
# -> 405 means the route exists and the service is up

The first boot downloads the PaddleOCR-VL model into the paddleocr_vl_data volume; subsequent starts are fast. Stop / restart with docker compose down / docker compose up -d. To rebuild after config changes: docker compose up -d --build.

2. Point the plugin at it

In cordis.patch.yml (or your profile's config: block):

ocrProvider: paddleocr_vl
ocrEndpoint: http://localhost:18080
ocrTimeoutMs: 600000          # scanning PDFs are slow
ocrUseSealRecognition: true
ocrUseChartRecognition: false
ocrMinPixels: 147384
ocrMaxPixels: 2822400

3. Use it

Just knowledge_add_document with a scanned file_path. The plugin detects ocrProvider: paddleocr_vl, sends the PDF to your local endpoint, and chunks the returned markdown. No code changes needed.

Note: PaddleOCR-VL needs a GPU for practical speed. The endpoint protocol (base64 file + fileType + recognition params -> result.layoutParsingResults[].markdown) is the same one WeKnora's paddleocr_vl_converter.go implements, so this is a drop-in local replacement for WeKnora's scanning-PDF OCR.

Persisted markdown sidecar (parse β†’ persist .md β†’ strip markers β†’ vectorize)

When persistMarkdown: true (default), ingesting a file β€” via knowledge_add_document or knowledge_add_documents_from_dir β€” writes the OCR'd / parsed markdown to disk before chunking and embedding, and then reads that file back to drive vectorization:

  1. The source file is parsed/OCR-ed into coherent, page-delimited markdown. Each page is prefixed with an explicit <!-- page: N --> marker so the .md is verifiably page-ordered and page-bounded.
  2. The markdown is written to markdownDir as one .md per source file; the file name is derived from the source (e.g. report.pdf β†’ report.md). A name clash is disambiguated with a short id rather than overwriting. The persisted .md keeps the page markers, so the on-disk artifact stays page-coherent and human-inspectable.
  3. The tool reads the .md back and strips the <!-- page: N --> markers before chunking/embedding. The markers are pure noise for the chunker and embedder (an HTML comment that would otherwise pollute every chunk it lands in), so the vector store reflects clean content while the disk artifact stays page-verifiable.

The directory resolves against the harness workspace root (DSH_HOME, else cwd), so the sidecars never land at an absolute path baked into the plugin. Content-only ingestion (no file_path) has no source to name from, so it skips the sidecar.


Files

dsh-ab-knowledge/
β”œβ”€β”€ cordis.patch.yml     # bundle patch: contributes the `knowledge` plugin row
β”œβ”€β”€ package.json         # deps, build scripts, dsh.bundle.patch pointer
β”œβ”€β”€ tsconfig.json        # TS build (allowImportingTsExtensions, strict)
β”œβ”€β”€ tsdown.config.ts     # bundle config (neverBundle @deepseek-ai/*)
β”œβ”€β”€ pnpm-workspace.yaml  # standalone workspace root (prevents pnpm escape)
β”œβ”€β”€ README.md
└── src/
    β”œβ”€β”€ index.ts         # function-plugin entry: name/inject/Config/apply; self-aware
    β”‚                    #   systemPrompt gate so disabled tools are never advertised
    β”œβ”€β”€ config.ts        # Config interface + schemastery schema (every tunable lives here)
    β”œβ”€β”€ chunker.ts       # recursive character splitter (WeKnora-faithful)
    β”œβ”€β”€ embed.ts         # Ollama embed client + cosine + rrfFuse
    β”œβ”€β”€ store.ts         # node:sqlite engine: KB/doc/chunk/FTS/embed, hybrid search
    β”œβ”€β”€ qa.ts            # cited QA via Ollama /api/chat
    β”œβ”€β”€ parsers.ts       # local docreader: bytes -> markdown, routed by extension
    β”œβ”€β”€ ocr.ts           # local PaddleOCR-VL OCR client (scanned PDFs, no WeKnora/cloud)
    β”œβ”€β”€ markdown.ts       # persisted-markdown sidecar: one .md per source file, page markers
    β”œβ”€β”€ shims.d.ts       # ambient types for untyped parser deps
    β”œβ”€β”€ render.ts        # clip / formatScore / describeScope helpers
    └── tools/           # pure core (helpers) split from the defineTool registration
        β”œβ”€β”€ helpers.ts   # arg helpers, config->runtime adapters, dir walker (no Cordis)
        β”œβ”€β”€ manage.ts    # create/list/delete knowledge base tools
        β”œβ”€β”€ document.ts  # add / add_documents_from_dir (bounded concurrency) / read / delete
        β”œβ”€β”€ retrieval.ts # search + ask (in-process multi-turn sessions)
        └── index.ts     # createTools(config): composes the above into one ordered list

# NOTE: the PaddleOCR-VL serving deployment lives OUTSIDE this plugin, at
#   D:/ai-service/paddleocr_vl/   (Dockerfile, entrypoint.sh, docker-compose.yml,
#   pipeline_config.yaml, merge_config.py). The plugin only speaks its /layout-parsing
#   protocol via src/ocr.ts; the OCR service is a separate, self-hosted deployment.

License

MIT.