dsh-ab-knowledge
Verifieddsh-ab-knowledge Β· v0.1.1 Β· MIT
A **self-contained knowledge-base service** for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
Install
dsh plugin add dsh-ab-knowledge Confirm the layer applied with dsh --profile default --dump-config β see the install guide.
Source
Published to npm without a public repository. Inspect the package contents before installing.
Tags
Readme
dsh-ab-knowledge
A self-contained knowledge-base service for DeepSeek Harness.
It reimplements WeKnora's knowledge-base capabilities locally β chunk, embed,
index, hybrid (vector + keyword) retrieval with Reciprocal Rank Fusion, and
cited question-answering β using only Node 22's built-in node:sqlite (FTS5)
and a local Ollama model.
No WeKnora required. The agent no longer needs a running WeKnora instance, a Go backend, Qdrant, Postgres/pgvector, or any other external service. The whole knowledge pipeline runs in-process inside the harness.
Why this plugin exists
WeKnora ships an official DSH plugin (@wxg-prc-cpg/dsh-weknora) that exposes a
thin client over a live WeKnora server. That keeps the agent dependent on a
Docker Compose stack (postgres + redis + docreader + app + frontend) just to
answer questions over documents.
This plugin instead extracts the knowledge-base engine itself out of WeKnora's
source and runs it locally. The read-side tool shapes
(list_knowledge_bases / search / read_document / ask) are kept
byte-for-byte compatible with the official client plugin, so existing
tool-calling behaviour transfers; the write/manage tools
(create_knowledge_base / add_document / add_documents_from_dir /
delete_document / delete_knowledge_base) make ingestion and lifecycle fully
local β no upload UI, no server, no API key.
Capabilities reimplemented (faithful to WeKnora)
| Capability | WeKnora source | This plugin |
|---|---|---|
| Recursive character chunking | internal/infrastructure/chunker |
src/chunker.ts |
| Default 512 chars / 80 overlap | DefaultChunkSize=512, DefaultChunkOverlap=80 |
chunkSize, chunkOverlap |
Separators \n\n > \n > γ |
strategy.go |
separators |
| Semantic (vector) embedding | Ollama /api/embed |
src/embed.ts |
| Keyword (FTS5 bm25) retrieval | chunks_fts |
src/store.ts |
| Hybrid fusion | knowledgebase_search_fusion.go |
fuse() + fuseWithRRF |
RRF k=60, vector 0.7 / keyword 0.3 |
retrieval_config.go |
rrfK, rrfVectorWeight, rrfKeywordWeight |
Over-retrieval pool TopK=50 |
DefaultRetrievalTopK=50 |
retrievalTopK |
Optional rerank (TopK=10, thr 0.2) |
RerankTopK, RerankThreshold |
rerankTopK (threshold via cosine) |
| Cited knowledge-QA | /knowledge-chat |
src/qa.ts |
| Locate a document by name | findDocuments |
findDocumentsByName |
| Document parsing (bytes β markdown) | docreader/parser/* |
src/parsers.ts (routed by extension, like registry.py) |
Local document parsing β the docreader, without WeKnora
WeKnora's docreader microservice turns uploaded bytes into markdown text
(base_parser.parse_into_text); the Go app then chunks and embeds it. This
plugin reimplements that first step locally in src/parsers.ts, routed by file
extension exactly like WeKnora's registry.py. Chunking, embedding, retrieval
and QA are already the engine's job (table above), so the boundary matches 1:1.
Pass a file via knowledge_add_document's file_path (absolute, local path)
and it is parsed before chunking β no upload UI, no server, no API key.
To ingest a whole folder at once, use knowledge_add_documents_from_dir with
dir_path (plus optional recursive β default true, extensions to narrow
formats, and title_prefix). Every supported file is run through the same
parse β chunk β embed β index pipeline; the result reports scanned / indexed
/ skipped counts with a per-file breakdown, so one unreadable file is reported
under skipped without aborting the rest of the batch.
| Format | Parser | Notes |
|---|---|---|
.txt .md .csv .json .log |
native | read as UTF-8 text |
.pdf |
pdf-parse (or PaddleOCR-VL OCR) |
digital PDFs use the text layer; scanned/image-only PDFs are OCR-ed when ocrProvider: paddleocr_vl is set (see below) - otherwise they yield little text |
.docx |
mammoth β turndown |
DOCX β HTML β Markdown (headings/tables preserved) |
.xlsx .xls |
xlsx (SheetJS) |
each sheet β a Markdown table |
.pptx .ppt |
officeparser |
slide text |
.html .htm .mhtml |
turndown |
script/style stripped, converted to Markdown |
Scanned / image-only PDFs are handled by an optional, self-hosted
PaddleOCR-VL OCR step (see
Local PaddleOCR-VL OCR below). When
ocrProvider: paddleocr_vl is configured with your local endpoint, every PDF -
digital or scanned - is sent through it and the returned markdown (OCR text +
layout) is what gets chunked. With ocrProvider: none (default) we fall back to
the pdf-parse text layer, which is empty for scanned pages.
Parser libraries are loaded lazily and bundled by tsdown, so the plugin stays
self-contained β no docreader container, no MinIO, no WeKnora. The only
runtime dependency beyond Node is a local Ollama for embeddings/QA.
Requirements
- Node β₯ 22.19 (the plugin uses the built-in
node:sqlite+ FTS5 virtual tables β bundled since Node 22.5 for in-tree builds, GA in 22.19 / Node 24). - Ollama reachable at the configured
embedBaseUrlandqaBaseUrl(defaulthttp://localhost:11434):- an embedding model, default
qwen3-embedding:4b(2560-dim; pull it withollama pull qwen3-embedding:4b);nomic-embed-text(768-dim) also works - keepembedDimin sync; - a chat model for QA, e.g.
qwen2.5:7b.
- an embedding model, default
- Both providers can be set to
noneto degrade gracefully:embedProvider: noneβ keyword-only (FTS5) retrieval, fully offline;qaProvider: noneβaskreturns the retrieved passages as references only.
Install
# from the harness checkout / profile
dsh plugin --profile web add dsh-ab-knowledge
The bundle patch (cordis.patch.yml) contributes the plugin row
id: knowledge and points the harness at the built code. Build with:
pnpm install
pnpm build # tsc -p tsconfig.json && tsdown
Tools
All read tools are prefixed with knowledge_ by default (configurable via
toolPrefix).
| Tool | Kind | Purpose |
|---|---|---|
knowledge_create_knowledge_base |
write | Create a named collection of documents. Returns its id. |
knowledge_list_knowledge_bases |
read | List existing bases with ids. |
knowledge_add_document |
write | Ingest text or a single local file (file_path); the file is parsed (PDF/DOCX/XLSX/PPT/HTML/MD/TXT/CSV/JSON) then chunk β embed β index. |
knowledge_add_documents_from_dir |
write | Bulk-ingest every supported file in a directory (recursive) through the same pipeline as add_document. Returns scanned / indexed / skipped counts plus a per-file breakdown. |
knowledge_search |
read | Hybrid retrieval; verbatim passages + by-name document matches. |
knowledge_read_document |
read | Page through a document's stored chunks in order. |
knowledge_ask |
read | Retrieve + compose a cited answer; multi-turn via session_id. |
knowledge_delete_document |
write | Remove a document and its chunks/FTS/embeddings. |
knowledge_delete_knowledge_base |
write | Remove a base and all its documents. |
Tool output shapes
The four read tools mirror @wxg-prc-cpg/dsh-weknora exactly:
searchβ{ query, knowledge_base_ids, count, results[], documents[] }where eachresults[]item carriesrank,chunk_id,knowledge_id,document,chunk_index,score,content,truncated; eachdocuments[]item carriesknowledge_id,title,knowledge_base_id,knowledge_base.askβ{ answer, session_id, pipeline, tool_calls[], references[] }with[n]citations into the referenced passages.read_documentβ{ knowledge_id, title, summary, page, page_size, total_chunks, returned_chunks, has_more, truncated, content }.list_knowledge_basesβ{ count, knowledge_bases[] }.
Configuration
Every tunable lives in one place: the config: block of the bundle patch
(cordis.patch.yml), validated against src/config.ts's Config schema. A
deployment edits behaviour there and never in the TypeScript source.
| Key | Default | Meaning |
|---|---|---|
dataDir |
knowledge-base |
Where knowledge.sqlite lives (relative to harness cwd). |
embedProvider |
ollama |
ollama or none (keyword-only). |
embedBaseUrl |
http://localhost:11434 |
Ollama embedding endpoint. |
embedModel |
qwen3-embedding:4b |
Embedding model id (default; 2560-dim). |
embedDim |
2560 |
Embedding dimensionality (must match model). |
embedTimeoutMs |
8000 |
Per-embed-request ceiling. |
qaProvider |
ollama |
ollama or none (references only). |
qaBaseUrl |
http://localhost:11434 |
Ollama chat endpoint. |
qaModel |
qwen2.5:7b |
Chat model for cited answers. |
qaTemperature |
0.1 |
Sampling temperature. |
qaMaxTokens |
2048 |
Max answer tokens. |
qaTimeoutMs |
300000 |
Per-QA-request ceiling. |
defaultKnowledgeBaseIds |
[] |
Scope used when a call names none (empty = all). |
chunkSize |
512 |
Chunk size in chars (WeKnora default). |
chunkOverlap |
80 |
Overlap in chars (WeKnora default; capped at half size). |
separators |
['\n\n','\n','γ'] |
Recursive splitter order (WeKnora default). |
rrfK |
60 |
RRF smoothing constant (WeKnora default). |
rrfVectorWeight |
0.7 |
Vector retriever weight (WeKnora default). |
rrfKeywordWeight |
0.3 |
Keyword retriever weight (WeKnora default). |
retrievalTopK |
50 |
Over-retrieval pool before the final cut (WeKnora default). |
rerankTopK |
0 |
Optional rerank depth by queryβchunk cosine (0 = off). |
ingestConcurrency |
4 |
Max concurrent file ingestions for add_documents_from_dir (bounds host I/O and Ollama embed calls; 1β16). |
maxResults |
8 |
Passages a search/ask returns. |
maxChunkChars |
1200 |
Per-chunk truncation budget in tool output. |
requestTimeoutMs |
30000 |
Non-streaming tool timeout. |
chatTimeoutMs |
300000 |
QA call timeout. |
toolPrefix |
knowledge |
Tool-name prefix. |
ocrProvider |
none |
none or paddleocr_vl (local PaddleOCR-VL OCR for scanned PDFs). |
ocrEndpoint |
http://localhost:18080 |
Base URL of your self-hosted PaddleOCR-VL serving (e.g. http://localhost:18080). |
ocrTimeoutMs |
600000 |
Per-OCR-request ceiling (scanning PDFs are slow). |
ocrUseSealRecognition |
true |
PaddleOCR-VL: detect and OCR seals/stamps. |
ocrUseChartRecognition |
false |
PaddleOCR-VL: convert charts into markdown tables. |
ocrMinPixels |
147384 |
PaddleOCR-VL VLM minimum input pixel count. |
ocrMaxPixels |
2822400 |
PaddleOCR-VL VLM maximum input pixel count. |
persistMarkdown |
true |
Persist the OCR'd / parsed markdown to disk (one .md per source file) before vectorizing; the file keeps its <!-- page: N --> page markers for verifiability, then the markers are stripped from the in-memory copy that is chunked/embedded. false skips the sidecar. |
markdownDir |
knowledge-base/markdown |
Directory for the persisted .md files; relative paths resolve against the harness workspace root (DSH_HOME, else cwd). |
tools |
all true |
Per-tool registration switches. |
How retrieval works
query
β
βββΊ FTS5 bm25 over chunks_fts βββββββββββββββ
β βββΊ fuse() (WeKnora RRF:
βββΊ Ollama /api/embed β cosine vs stored βββββ vW/(k+vRank) + kW/(k+kRank))
embeddings (JSON column) β
βββΊ optional rerank by queryβchunk cosine
β
search hits ββββββββββββββββββ
β
βββββββββββββββββ΄βββββββββββββββββ
search tool ask tool
(verbatim passages) (QA model composes a cited answer)
- Vector-only results keep their raw cosine scores; keyword-only bm25 is
rescaled to
[0,1](prevents BM25 saturation from dominating the fused rank, matching WeKnora'sfuseWithRRF). - Embedding is an accelerator, not a hard dependency: if Ollama is unreachable, ingestion still completes (keyword retrieval keeps working) and a warning is logged.
- All SQL is parameterised β no string interpolation of user input.
Local PaddleOCR-VL OCR (scanned PDFs)
Scanned / image-only PDFs have no text layer, so pdf-parse returns almost
nothing for them. WeKnora routes those through a self-hosted PaddleOCR-VL
service on the Go side; this plugin reproduces that exact client contract
(src/ocr.ts -> POST {endpoint}/layout-parsing, the same request/response
shape WeKnora's paddleocr_vl_converter.go uses) so you can OCR locally
without WeKnora and without any cloud.
1. Deploy a local PaddleOCR-VL serving endpoint
PaddleOCR-VL is served by PaddleX; the
paddlex CLI exposes a /layout-parsing REST endpoint whose protocol this plugin
already speaks (src/ocr.ts -> POST {endpoint}/layout-parsing).
Prerequisites (read first):
- OS: PaddleOCR-VL does not run on native Windows. Run every command below
inside WSL2 (Ubuntu) or a Docker container with
--gpus all. (You are on Windows, so Git Bash / CMD will not work for these steps.) - GPU: an NVIDIA card, CUDA 12.x, compute capability >= 7 (RTX 3060 / A10 / A100 and up). CPU inference is technically possible but impractically slow.
- The OCR service runs entirely on your machine - no data leaves your network.
# Inside WSL2 Ubuntu (or the GPU Docker container):
# 1. PaddlePaddle GPU wheel (match your CUDA; 3.3.0 / cu126 shown)
python -m pip install paddlepaddle-gpu==3.3.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
# 2. PaddleX with the OCR pipeline (NOT "paddleocr[ocr_vl]")
python -m pip install "paddlex[ocr]"
# 3. serving plugin
paddlex --install serving
# 4. Generate the full pipeline config (first run only)
paddlex --get_pipeline_config PaddleOCR-VL -o D:/ai-service/paddleocr_vl/pipeline_full.yaml
# 5. Merge in our Serving overrides (visualize off + lift the PDF page cap)
cd D:/ai-service/paddleocr_vl && python3 merge_config.py
# 6. Start the service on :18080 (GPU strongly recommended)
paddlex --serve --pipeline D:/ai-service/paddleocr_vl/paddleocr_vl.serving.yaml --host 0.0.0.0 --port 18080
Quick path (skip the config merge):
paddlex --serve --pipeline PaddleOCR-VL-1.6starts the service on:18080with defaults. Use the merged config only if you need to lift the PDF page cap or disable annotated output images.
The endpoint is now http://localhost:18080. Verify it is up:
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/layout-parsing
# -> 405 (method not allowed on GET) means the route exists; 200/422 on a real POST means ready
Port note for WSL2: if the service binds inside WSL2, reach it from Windows-side tooling via
http://localhost:18080(WSL2 forwards localhost). If you setocrEndpointinside the DSH process running on Windows, use the samelocalhost:18080.
Docker (recommended on Windows)
A single-container image that runs the PaddleX native backend and serves
POST /layout-parsing on :18080 (no separate vLLM container required). The
entrypoint generates + merges the config on first boot, so you only run one
command.
Prerequisites: Docker Desktop with the WSL2 backend, an NVIDIA GPU with the current driver (WSL2 shares the Windows driver), and the NVIDIA Container Toolkit enabled. Confirm GPU passthrough first:
docker run --rm --gpus all nvidia/cuda:12.6.0-runtime-ubuntu22.04 nvidia-smi
# -> should print your GPU; if it errors, fix the toolkit before continuing
Build & run (from D:/ai-service/paddleocr_vl):
cd D:/ai-service/paddleocr_vl
docker compose up -d --build
docker compose logs -f # wait for "Uvicorn running on http://0.0.0.0:8080"
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/layout-parsing
# -> 405 means the route exists and the service is up
The first boot downloads the PaddleOCR-VL model into the paddleocr_vl_data
volume; subsequent starts are fast. Stop / restart with
docker compose down / docker compose up -d. To rebuild after config changes:
docker compose up -d --build.
2. Point the plugin at it
In cordis.patch.yml (or your profile's config: block):
ocrProvider: paddleocr_vl
ocrEndpoint: http://localhost:18080
ocrTimeoutMs: 600000 # scanning PDFs are slow
ocrUseSealRecognition: true
ocrUseChartRecognition: false
ocrMinPixels: 147384
ocrMaxPixels: 2822400
3. Use it
Just knowledge_add_document with a scanned file_path. The plugin detects
ocrProvider: paddleocr_vl, sends the PDF to your local endpoint, and chunks the
returned markdown. No code changes needed.
Note: PaddleOCR-VL needs a GPU for practical speed. The endpoint protocol (base64
file+fileType+ recognition params ->result.layoutParsingResults[].markdown) is the same one WeKnora'spaddleocr_vl_converter.goimplements, so this is a drop-in local replacement for WeKnora's scanning-PDF OCR.
Persisted markdown sidecar (parse β persist .md β strip markers β vectorize)
When persistMarkdown: true (default), ingesting a file β via knowledge_add_document
or knowledge_add_documents_from_dir β writes the OCR'd / parsed markdown to disk
before chunking and embedding, and then reads that file back to drive vectorization:
- The source file is parsed/OCR-ed into coherent, page-delimited markdown. Each page
is prefixed with an explicit
<!-- page: N -->marker so the.mdis verifiably page-ordered and page-bounded. - The markdown is written to
markdownDiras one.mdper source file; the file name is derived from the source (e.g.report.pdfβreport.md). A name clash is disambiguated with a short id rather than overwriting. The persisted.mdkeeps the page markers, so the on-disk artifact stays page-coherent and human-inspectable. - The tool reads the
.mdback and strips the<!-- page: N -->markers before chunking/embedding. The markers are pure noise for the chunker and embedder (an HTML comment that would otherwise pollute every chunk it lands in), so the vector store reflects clean content while the disk artifact stays page-verifiable.
The directory resolves against the harness workspace root (DSH_HOME, else cwd), so
the sidecars never land at an absolute path baked into the plugin. Content-only
ingestion (no file_path) has no source to name from, so it skips the sidecar.
Files
dsh-ab-knowledge/
βββ cordis.patch.yml # bundle patch: contributes the `knowledge` plugin row
βββ package.json # deps, build scripts, dsh.bundle.patch pointer
βββ tsconfig.json # TS build (allowImportingTsExtensions, strict)
βββ tsdown.config.ts # bundle config (neverBundle @deepseek-ai/*)
βββ pnpm-workspace.yaml # standalone workspace root (prevents pnpm escape)
βββ README.md
βββ src/
βββ index.ts # function-plugin entry: name/inject/Config/apply; self-aware
β # systemPrompt gate so disabled tools are never advertised
βββ config.ts # Config interface + schemastery schema (every tunable lives here)
βββ chunker.ts # recursive character splitter (WeKnora-faithful)
βββ embed.ts # Ollama embed client + cosine + rrfFuse
βββ store.ts # node:sqlite engine: KB/doc/chunk/FTS/embed, hybrid search
βββ qa.ts # cited QA via Ollama /api/chat
βββ parsers.ts # local docreader: bytes -> markdown, routed by extension
βββ ocr.ts # local PaddleOCR-VL OCR client (scanned PDFs, no WeKnora/cloud)
βββ markdown.ts # persisted-markdown sidecar: one .md per source file, page markers
βββ shims.d.ts # ambient types for untyped parser deps
βββ render.ts # clip / formatScore / describeScope helpers
βββ tools/ # pure core (helpers) split from the defineTool registration
βββ helpers.ts # arg helpers, config->runtime adapters, dir walker (no Cordis)
βββ manage.ts # create/list/delete knowledge base tools
βββ document.ts # add / add_documents_from_dir (bounded concurrency) / read / delete
βββ retrieval.ts # search + ask (in-process multi-turn sessions)
βββ index.ts # createTools(config): composes the above into one ordered list
# NOTE: the PaddleOCR-VL serving deployment lives OUTSIDE this plugin, at
# D:/ai-service/paddleocr_vl/ (Dockerfile, entrypoint.sh, docker-compose.yml,
# pipeline_config.yaml, merge_config.py). The plugin only speaks its /layout-parsing
# protocol via src/ocr.ts; the OCR service is a separate, self-hosted deployment.
License
MIT.