Skip to content
dsh marketplace

dsh-llm-rate-limiter

Verified

@leaf233/dsh-llm-rate-limiter ยท v0.2.0 ยท MIT ยท Web UI

Per-model LLM call rate limiter for DeepSeek Harness with queue support and a live status panel

Install

dsh plugin add @leaf233/dsh-llm-rate-limiter

Confirm the layer applied with dsh --profile default --dump-config โ€” see the install guide.

Source

Tags

Creators

Readme

dsh-llm-rate-limiter

npm License: MIT DSH 0.1.x Cordis 4.x Tests

Per-model LLM call rate limiter for DeepSeek Harness with queue/reject support and interactive GUI configuration.

Features

  • Per-model rate limiting โ€” independent concurrency, RPM, and burst limits for each provider/model
  • Two algorithms โ€” Token Bucket (allows bursts) or Sliding Window (smooth, strict RPM)
  • Queue mode โ€” throttled requests wait in queue and are released when a slot opens
  • Reject mode โ€” throttled requests fail immediately (integrates with dsh-llm-retry for auto-backoff)
  • Interactive GUI โ€” collapsible card in DSH Settings โ†’ Plugins โ†’ Configurable
  • Live status panel โ€” real-time counters, per-model progress bars and an event log (v0.2.0)
  • Hot-reload โ€” settings changes take effect immediately, no restart needed
  • Every request checked โ€” intercepts llm/stream waterfall, covering every LLM call in every agent turn

Status Panel (v0.2.0)

Expanding the card shows a live panel at the top of its body:

๐Ÿ“Š ๅฎžๆ—ถ็Šถๆ€                          โ— ๅฎžๆ—ถ   [ๆธ…้›ถ]
 ่ฏทๆฑ‚ 42 ยท ้€š่ฟ‡ 38 ยท ๆ‹’็ป 2 ยท ่ถ…ๆ—ถ 1 ยท ไธญๆญข 1 ยท ๅนณๅ‡็ญ‰ๅพ… 214ms
 deepseek/deepseek-chat   [ไปค็‰Œๆกถ]  โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–‘โ–‘โ–‘ 7.5/10   ๅนถๅ‘ 2/5   ๆŽ’้˜Ÿ 1
 openai/gpt-4o            [ๆป‘ๅŠจ็ช—ๅฃ] โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“โ–“ 3/3 rpm  ๅนถๅ‘ 1/5
 12:00:03  timeout   openai/gpt-4o              ็ญ‰ๅพ… 1m
 12:00:01  rejected  openai/gpt-4o
 12:00:00  granted   deepseek/deepseek-chat     ็ญ‰ๅพ… 4.2s
Aspect Behaviour
Data channel Framework connection.rpc channel /llm-rate-limiter โ€” authenticated (401/403 fence), POST+JSON, auto-cleaned with the plugin fiber
Cadence 1 s polling while the card is expanded; backs off 2 s โ†’ 4 s โ†’ 8 s after failures
Collapsed card The panel unmounts, so no polling runs at all
Endpoints snapshot (live counters) and reset (zero the statistics)
Without a channel Shows "็Šถๆ€้€š้“ไธๅฏ็”จ" and leaves the rest of the card fully functional
Counters requests / granted / rejected / timeouts / aborted / totalWaitMs, plus the last 8 events (ring buffer of 64)
Progress bars Token bucket shows tokens/burstSize; sliding window shows countInWindow/maxRpm; both turn amber as the limit approaches

๐Ÿง  From Hindsight memory (dsh-context-host-client) โ€” the channel idiom is dsh-context's: ctx.inject(["connection"]) โ†’ conn.rpc.handle(channel, handler), with the browser side resolving ctx.get("connection")?.rpc.call defensively so a missing service degrades instead of throwing.

Installation

Option 1: npm (recommended)

dsh plugin add <your-profile> @leaf233/dsh-llm-rate-limiter
# or, inside the profile directory:
pnpm add @leaf233/dsh-llm-rate-limiter

Option 2: local path (development)

dsh plugin add <your-profile> ./path/to/dsh-llm-rate-limiter
# or
dsh plugin add ./path/to/dsh-llm-rate-limiter   # default profile

The plugin must be added as a dependency in the profile's package.json. The bundle entry (cordis.patch.yml) is auto-detected by reconcilePlugins.

Option 3: from GitHub

dsh plugin add <your-profile> github:Leafyezi233/dsh-llm-rate-limiter

โš ๏ธ Important: Git-hosted plugins are blocked by pnpm's allowBuilds restriction on first install. If the install fails, check the error message for the exact key pnpm suggests, then add it to your profile's pnpm-workspace.yaml:

pnpm:
  allowBuilds:
    - '@leaf233/dsh-llm-rate-limiter'

Then re-run the install command.

Configuration

Via GUI

  1. Open DSH Web UI (dsh web)
  2. Go to Settings โ†’ Plugins
  3. Find โš™ LLM ่ฐƒ็”จ้™้€Ÿ card โ€” click to expand
  4. Configure defaults, per-model overrides, and throttle behavior

Via file

Edit the profile's settings.yaml or use the GUI โ€” changes are persisted to the DSH settings store:

llm-rate-limiter:
  enabled: true
  strategy: token-bucket       # "token-bucket" | "sliding-window"
  defaults:
    maxConcurrent: 5
    maxRpm: 60
    burstSize: 10               # token-bucket only
    refillRate: 1               # token-bucket only (tokens/sec)
  models:
    "deepseek/deepseek-chat":
      maxConcurrent: 8
      maxRpm: 120
    "openai/gpt-4o":
      maxConcurrent: 2
      maxRpm: 10
      burstSize: 3
    "anthropic/claude-3-5-sonnet":
      enabled: false            # skip rate limiting for this model
  onThrottled: queue            # "queue" | "reject"
  maxQueueWaitMs: 60000

Settings Reference

Field Default Description
enabled true Global on/off switch. When off, zero overhead bypass.
strategy "token-bucket" "token-bucket" (allows bursts) or "sliding-window" (smooth, strict RPM)
defaults.maxConcurrent 5 Max simultaneous requests per model
defaults.maxRpm 60 Max requests per minute per model
defaults.burstSize 10 Token bucket capacity โ€” how many requests can burst at once
defaults.refillRate 1 Tokens refilled per second (token-bucket). Auto-derived from maxRpm / 60 if not set.
models.<key>.maxConcurrent โ€” Per-model concurrency override
models.<key>.maxRpm โ€” Per-model RPM override
models.<key>.burstSize โ€” Per-model burst capacity override
models.<key>.refillRate โ€” Per-model refill rate override
models.<key>.enabled โ€” Set false to skip rate limiting for this specific model
onThrottled "queue" What happens when a request hits the limit: "queue" (wait) or "reject" (fail immediately)
maxQueueWaitMs 60000 Max time (ms) a request waits in queue before being rejected

Note: When a model overrides maxRpm without explicitly setting refillRate, the refill rate is automatically derived as maxRpm / 60 (tokens per second). This ensures "set maxRpm=3" actually limits to 3 requests per minute.

Algorithm Comparison

Token Bucket Sliding Window
Burst Yes (controlled by burstSize) No โ€” strictly smooth
Recovery Tokens refill at refillRate/sec Window slides continuously
Best for Tolerating request spikes APIs with hard per-minute limits
GUI label ไปค็‰Œๆกถ (Token Bucket) ๆป‘ๅŠจ็ช—ๅฃ (Sliding Window)

How It Works

Agent Turn
  โ†’ LLM Call (e.g. deepseek/deepseek-chat)
    โ†’ ctx.on("llm/stream") interceptor
      โ†’ Resolve rate limiter for this provider/model
      โ†’ Token bucket: has tokens + concurrency room?
      โ†’ If YES: consume token, acquire slot, forward to API
      โ†’ If NO (reject mode): return RATE_LIMIT error immediately
      โ†’ If NO (queue mode): park in waiters[], wait for token refill
    โ†’ Request completes โ†’ release slot โ†’ drain waiting requests
  โ†’ dsh-llm-retry catches RATE_LIMIT โ†’ exponential backoff โ†’ retry

Development

# Clone
git clone https://github.com/Leafyezi233/dsh-llm-rate-limiter.git
cd dsh-llm-rate-limiter

# Install deps
pnpm install

# Run tests (19 tests)
node test-strategies.mjs

# Run E2E rate-limit test
node test-3rpm.mjs

# Install into a DSH profile for testing
dsh plugin add <your-profile> .

The plugin uses a live symlink when installed via link: โ€” edits to lib/ take effect on browser hard-refresh (Ctrl+Shift+R) without reinstalling.

Compatibility

DSH Version Status Notes
0.1.x (RC) โœ… Tested Verified against 0.1.2-rc.1, cordis 4.0.2
0.2.x โš ๏ธ Untested May need API adjustments
Cordis 5+ โš ๏ธ Untested Major version change likely requires rewrite

See COMPATIBILITY.md for detailed API dependency analysis.

License

MIT