Changelog

All notable changes to Cortex are documented here, organized by date.

July 28, 2026

Deep Research quality no longer depends on the answer model volunteering to reflect between search rounds. Models that skip that step used to degrade into repetitive query volleys — burning every iteration, drifting off-topic, and answering from a diluted context — while models that reflect naturally converged in a few targeted rounds. The researcher loop now enforces the reflect-then-search rhythm structurally and stops searching once results converge, which brings weaker-reflecting models (measured on Qwen 3.6 35B A3B) up to the research quality previously seen only from naturally reflective ones (Gemma 4 26B), at roughly half the wall-clock. A new default time budget also guarantees the answer starts writing even when the LLM provider is having a slow moment. And once the research was good, the write-up itself had to survive: Deep Research answers on thinking models were being cut off mid-sentence — sometimes coming back empty — because the model's hidden reasoning was spending the very token budget meant for the answer. That budget now goes to the answer alone, and a truncated answer can no longer pass itself off as a finished one.

Deep Research: enforced reflection & convergence

Forced reflection — RESEARCHER_FORCE_REFLECTION (default true): a search round that arrives without a reasoning step (neither a reasoning tool call nor prose alongside the tool calls) is followed by one forced micro-call pinned to the reasoning tool. The model's analysis of what it found — and the specific gap to target next — lands in the conversation and steers the following round. Models that already reflect never trigger it; providers that reject a named tool_choice disarm it gracefully for the run.
Convergence stop — RESEARCHER_NOVELTY_MIN_NEW_RATIO (default 0.35) + RESEARCHER_NOVELTY_STALE_ROUNDS (default 2): the loop tracks how many previously-unseen sources each search round returns; consecutive stale rounds end research early instead of burning the remaining iterations on re-fetched ground (which dilutes the writer context). Set either to 0 to disable.
Novelty-aware iteration notes — each round, the agent is told what share of its last round's sources were new, with a standing directive to reason about the gap or call done — so a well-behaved model stops on its own before the hard stop fires.
Evidence-based stopping criterion — the Deep Research prompt now says to finish as soon as searches stop surfacing new sources (instead of "exhaust all research avenues"), and warns against pivoting to entities that merely appear in results rather than bearing on the question.
Prose reflections stream — models that reflect as text alongside their tool calls (instead of the reasoning tool) now have those thoughts surfaced as thinking events too.

Deep Research: latency guardrails

Research time budget on by default — RESEARCHER_WALL_CLOCK_SECONDS default changed 0 → 120. Healthy gathering finishes in 30-60s; the budget exists for provider queue spikes (measured live: a 2-token completion taking 107s), where it breaks to the writer so the answer always starts instead of stalling silently for minutes. The forced reflection call is skipped when less than 20s of budget remain. Raise the value — or set 0 to restore unlimited — for slow self-hosted inference.
Honest streaming status — a Planning the next research step status event now precedes each researcher LLM call, so during a slow provider call the UI no longer shows a stale "Searching the knowledge base" from the previous round.

Measured on the same knowledge graph before/after (Qwen 3.6 35B A3B, hardest test question): 13 search calls → 8, first answer token 92s → 56s, and the answer went from a generic non-answer to a specific, fully cited list — matching the Gemma 4 baseline at half its total wall-clock. No API or schema breakage: all events are additive, and every behavior change is env-gated with the flags above.

Deep Research: answers no longer cut off mid-sentence

On thinking models, the hidden reasoning trace is billed against the same output-token budget as the visible answer. The Deep Research writer ran with reasoning enabled, so a long report could spend most of its 4,000-token allowance thinking and then stop mid-word — with no error, no warning, and a completed-looking response in the UI. Measured on Qwen 3.6 35B A3B with a 15-source research context, the trace ran 2,700-4,300 tokens against that cap; in the worst case it consumed the entire budget and the user received an empty answer.

Writer reasoning suppressed — the final writer call now runs with reasoning off in Deep Research too (it already did in chat). The writer only composes prose from context the researcher already gathered, so hidden reasoning bought nothing while consuming the answer's budget. The same 15-source report now completes in ~1,300 output tokens — a 3× margin under the cap — and the first token lands sooner. The researcher loop is unchanged and still reasons in Deep Research; only the write-up step changed.
Larger answer budget — WRITER_MAX_TOKENS_QUALITY default 4000 → 8000, giving long reports room and protecting against providers that ignore the disable-thinking flag. WRITER_MAX_TOKENS_SPEED is unchanged at 1200.
Truncation is never silent — if an answer does hit the output limit, the response now ends with a visible note saying it was cut short, and the backend logs a warning naming the mode, the cap, and which env var to raise. Previously the stream simply ended and the turn was marked complete, making a mid-word cut indistinguishable from a finished answer in both the UI and the logs.

Verified end-to-end against a live instance: the question that previously broke off mid-sentence now returns a complete, fully cited answer over 15 sources, and an artificially lowered cap produces the labeled-truncation path instead of a silent cut. No API or schema breakage — the truncation note arrives as ordinary answer content, and both limits stay env-overridable.

July 23, 2026

Cortex Chat — the standalone multi-tenant chat frontend — now has first-class documentation. Until now it was a single row in the ecosystem table; operators had to read the app's repository to understand what it does and how to deploy it against an instance.

Documentation: Cortex Chat

New docs page — Cortex Chat explains the app end to end: the chat experience (streaming Ask AI, Deep Research, inline citations, conversation memory, cross-device history, support link, default chat mode), the multi-tenant model (roles, groups, self-registration with approval, password reset over SMTP, bulk xlsx user import, content roles), the admin library console (Documents / Processing / Collections tabs that drive the backend's knowledge-graph pipeline from inside the chat app), the analytics dashboard and <cortexchatanalytics> injection, the one-admin-key/minted-scoped-keys security design, resilience behaviors, and full setup — a complete environment reference (required, optional SMTP/registration/GlitchTip knobs, deprecated aliases), Docker/Coolify deployment, and a first-run walkthrough.
New handbook chapter — Chapter 25 covers the same ground for handbook readers, and Chapter 24 (Apps) now distinguishes in-instance apps from standalone apps like Cortex Chat that run outside the instance and connect via the REST API with scoped keys.
No product changes — documentation only.

July 22, 2026

Web Import stopped producing a scatter of one-off pages. A crawl now folds every page from the same site into one document per domain, completes the moment those documents are staged (processing continues in the background, where you can watch it), and names each document by its domain instead of whatever title the crawler guessed. Crawls driven through a front-end can also carry a contributor tag, so a site imported via the MOCA Library is attributed like any other community submission.

Web Import: one document per site

Per-domain aggregation — all URLs from the same host in a job become a single markdown document (example.com.md, titled # example.com) with one ## section per page, each keeping its own > Source: line. Retrieval and the knowledge graph read a site's subpages as one related work — an artist named only on an About page now ties to the exhibition and press pages that never repeat the name. A job spanning several domains produces one document per domain.
Import completes on staging, not on full processing — the job finishes (and the modal's completion screen appears) as soon as the documents are added, then hands the extract/embed/graph pass to a normal batch_processing task. The Documents page shows them processing in the background — no more watching a progress bar through the whole pipeline for a large crawl.
Documents named by domain — crawl4ai often falls back to the URL path for a page title (e.g. o.html), which used to become the document's filename and every citation. The aggregated document is now titled and filed by its domain, so sources read clearly.
Contributor attribution — POST /api/web-import accepts an optional submitted_by; when set, each document's source becomes crawl:<domain> community:<id> (mirroring the upload path's community:<addr>), so crawls submitted through a front-end are attributed to the submitter. Omitted, the source stays crawl:<domain>.

July 21, 2026

The app platform learned to speak to cloud storage: the task engine gained WebDAV listings, binary file transfer, and runtime OAuth credentials — the pieces a folder-sync app needs — and the first wave ships with it. Nextcloud Sync and Dropbox Sync are live in the registry (now browseable on the web at registry.cortex.eco), and Paperless Sync 0.4.0 switches from re-rendering OCR text to ingesting the actual documents. Shipping these against real accounts also surfaced fixes that land here: a config-masking gap after app upgrades, clipboard access inside the app sandbox, and a Knowledge Graph page that kept offering Regenerate Graph while a sync-driven build was already running.

Cloud-sync task engine extensions

The declarative task DSL (platform apps' server-side step-queues) gained the vocabulary for cloud-storage sync — all behind the existing security gates (host allowlist, SSRF guard, endpoint allowlist), with the manifest schema unchanged:

webdav step — a PROPFIND folder listing through the same gates as http steps, with the multistatus XML parsed server-side into plain items (href, name, etag, lastModified, size, contentType, isDir, fileId); depth 0/1/infinity and a files/dirs filter. ETags come normalized, ready to diff against stored state — the change-detection primitive for WebDAV sync. The interactive platform http envelope accepts PROPFIND too, so app UIs can drive folder browsers.
Binary passthrough uploads — cortex steps accept multipart: {fromUrl: …}: the instance fetches the file server-side and streams the bytes straight into /api/upload. PDFs, Office files, and images survive intact, and the bytes never enter the task context or a browser. Capped by the existing 20 MB transfer limit.
Dynamic per-request auth — http/webdav/fromUrl steps accept auth: {bearer: …} (or basic) resolved from the run context, so a task can refresh an OAuth token at the start of a run and use it on every call. Config secrets remain untemplatable, so static credentials can't leak through this path.
Multi-listing fan-out — items.fromEach concatenates several listing steps (one per synced folder) into one work pool, and a new ext template filter enables extension-based type filtering for APIs whose listings carry no MIME type.
Redirect-following binary fetches — fromUrl accepts followRedirect: true for APIs that serve file bytes via a 302 to a pre-authenticated URL on a variable host (Microsoft Graph /content). Exactly one hop, https + public targets only, and no request headers are forwarded to the redirect target — safe here because the bytes stream straight into the instance's own upload, never into app-readable context.
Delta-cursor plumbing — paginate output gains last (the final page's body, where cursor APIs put their continuation token), and reference paths handle keys that contain dots (OData's @odata.deltaLink). WebDAV listings now always carry a fileId (the server's stable id, or a deterministic href hash on servers without one).
Google service-account auth without OAuth — auth_header templates may carry google_sa_token(VAR, scopes): the instance mints short-lived access tokens from a service-account JSON key (config secret) server-side — RS256-signed, exchanged at Google's pinned token endpoint, cached until near-expiry. The Google Drive Sync app builds on this: users share folders with the service account's email; no consent screens, no app verification, no tokens in tasks or browsers.
Microsoft app-only auth for SharePoint — the sibling ms_graph_token(TENANT, CLIENT_ID, SECRET) transform mints app-only Graph tokens via the client-credentials grant (endpoint pinned to Microsoft, tenant segment sanitized, cached). Pairs with the Sites.Selected permission: the SharePoint Sync app sees only sites an admin explicitly grants — no user sign-in, no consent screens, no tenant-wide access.
Basic auth without hand-encoding — auth_header templates can now reference any config var and wrap parts in base64(...), evaluated server-side: Authorization: Basic base64(NC_USER:NC_APP_PASSWORD). Users paste a plain login and app password; nobody base64-encodes credentials by hand (a field-tested error magnet).

Cloud-sync apps in the registry

The registry catalog is browseable on the web at registry.cortex.eco — the same checksum-pinned listings the in-instance Browse Registry panel installs from.

Nextcloud Sync 0.1.0 — pick folders in a built-in WebDAV browser (with a live "~N files across M folders will sync" projection), and a scheduled task keeps them in the graph: ETag-based change detection, edited files replace their previous document instead of duplicating, sub-folders included. Auth is a plain Nextcloud app password; Basic auth is assembled server-side.
Dropbox Sync 0.1.0 — OAuth 2 + PKCE with no app secret and no redirect server (copy/paste code flow, granted scopes verified at connect time with actionable errors), then per-folder delta cursors (list_folder/continue) drive incremental sync. The refresh token lives in the app's private storage; access tokens are minted per run.
Paperless Sync 0.4.0 — now ships the full documents: each sync downloads the archived PDF (or original upload) server-side and streams it into ingestion, so Docling parses the real document — layout, tables, images — instead of a markdown re-render of paperless's OCR text. Requires the platform extensions above (minVersion 2.1.0).

Fixes from shipping against real accounts

Orphaned app config keys no longer surface after upgrades — app upgrades preserve config.json, but a var that was secret-typed in the old manifest lost its masking the moment a new manifest stopped declaring it: the stale secret came back as plaintext in both the admin config view and the app-facing platform/config endpoint. Both now expose only vars declared by the current manifest (regression-tested).
Copy buttons work inside the app sandbox — the launcher and share-shell iframes now delegate clipboard-write (the async Clipboard API is permission-gated in the sandbox's opaque origin), so in-app copy buttons (auth generators, OAuth links) actually copy.
Queued documents read as Queued, not Processing — a burst of API ingests (e.g. a sync app uploading hundreds of files) is capped at BATCH_PROCESSING_CONCURRENCY, but every waiting document displayed as Processing — 342 uploads looked like 342 concurrent pipelines. Waiting documents now carry an additive processing_queued flag: the Knowledge Graph Step 1 tile gains a Queued count, the Documents page shows a Queued badge, and Processing means exactly the documents actually holding a slot. No behavior change to the cap itself.
Knowledge Graph page: Abort during any live build — the top CTA kept showing Regenerate Graph while ingestion started elsewhere (a sync app's uploads, another tab) was running, because per-document processing creates no batch task record and the page only checked for running tasks once on mount. The CTA now also keys on document-derived activity (the same signal as the Step 1 tile), running-task detection is continuous, and an idle 15-second refresh heartbeat notices externally-started ingestion in the first place. Aborting genuinely stops per-document processing — documents are kept.
No API or schema breakage: all task-DSL additions are opt-in step fields, the manifest schema is unchanged (all three registry/template/instance validators stay in lockstep), and instances without ENABLE_APPS are unaffected.

July 20, 2026

Restarting the backend or the LLM endpoint mid-processing used to throw away everything a large document had already paid for — conversion, embeddings, and hours of extraction — and start over from the first chunk. Processing is now checkpointed at sub-document granularity and pauses gracefully through LLM endpoint outages, so a litellm restart at 95% of a 3,000-chunk book costs the remaining 5%, not the whole document. Separately, two silent login foot-guns reported by self-hosters are fixed: the session cookie's Secure flag gets a runtime switch for plain-HTTP setups, and a missing ADMIN_EMAIL now fails loudly instead of silently substituting a default. This release also introduces Apps — self-contained web apps that install into an instance from a zip and run sandboxed against a proxied, least-privilege slice of the Cortex API.

Ingest checkpoint & resume

Stored work is reused, not redone — each run records a fingerprint (file hash + extraction config). When an interrupted or reprocessed document still matches it, the stored chunks and embeddings are loaded from the database instead of re-running document conversion and a full re-embed. Reprocessing keeps resumable chunks instead of deleting them.
Entity extraction checkpoints per batch — every settled extraction batch persists its entities immediately and advances a chunk-range watermark on the document. A resumed run skips completed ranges, and the progress bar continues where it stopped (e.g. 2850/3000 chunks) instead of resetting.
Per-chunk relationship extraction checkpoints too — chunks whose relationship pass settled are flagged and skipped on resume (a chunk can legitimately yield zero relationships, so completion is tracked explicitly).
Controlled by ENABLE_INGEST_RESUME (default true); set false for the legacy restart-from-zero behavior. Checkpoints are invalidated automatically when the file bytes or extraction config change.

LLM endpoint outage handling

Batches are no longer silently dropped when the endpoint is down — connection failures (endpoint restart, gateway 502/503) previously fell into the generic error path, which discarded every remaining batch within seconds and let the document "complete" with silent entity loss. They are now classified separately: the affected batch is requeued and re-probed with backoff until the endpoint returns.
Bounded by LLM_OUTAGE_MAX_WAIT_SECONDS (default 900) — if the endpoint stays unreachable past the budget, the document is marked failed with its checkpoint intact, and the error message says so: reprocessing resumes from the checkpoint once the endpoint is back.
The document's progress message shows the outage live (LLM endpoint unreachable — retrying in 20s...), and run telemetry gains a connection_retries counter.

Paused & Interrupted states in the UI

The outage and checkpoint machinery is now fully visible in the frontend, so operators see exactly what a document is doing instead of a generic spinner or failure:

Paused badge (amber) — a processing document whose LLM endpoint is temporarily unreachable shows a pause badge, an amber status line with the wait details ("LLM endpoint unreachable — retrying automatically (waited 40s of 900s)"), and an amber progress bar in the phase stepper — on both the Documents page and Knowledge Graph Step 1. Processing continues automatically; no action needed.
Interrupted badge (amber) — a document that failed because the outage outlasted the wait budget is labeled Interrupted instead of Failed, with an inline note that progress is checkpointed; its reprocess button becomes Resume from checkpoint.
Backed by additive document fields (processing_paused, paused_reason, resume_available) returned by the document endpoints — not a new processing status, so filters, integrations, and API clients are unaffected.
No API or schema breakage: new document/chunk properties are additive, and behavior with ENABLE_INGEST_RESUME=false matches the previous release.

Serving the dashboard over plain HTTP (e.g. a LAN self-host without TLS termination) used to break login silently: the session cookie was sent with the Secure flag hard-wired at image-build time, browsers drop Secure cookies over HTTP per spec, and the login form simply bounced back with no error. There was no runtime override — only rebuilding the image could change it.

New SESSION_COOKIE_SECURE env var (true/false, read at container runtime) — overrides the session cookie's Secure flag; set false for TLS-less HTTP setups, leave unset behind HTTPS. Default unchanged (Secure in production builds).
Mapped through the frontend service in all compose files (local, prod, Coolify, Dokploy) — a .env edit plus container restart is enough; no image rebuild, no compose patches.
Documented in the configuration reference and the authentication guide.
No API or schema breakage; behavior with the variable unset is identical to the previous release.

Missing `ADMIN_EMAIL` fails loudly instead of silently defaulting

When the frontend container started without ADMIN_EMAIL (env file pointed at the wrong path, secrets not wired into a standalone deployment), the code silently substituted admin@example.com — so logging in with the correct credentials answered "Invalid email or password", indistinguishable from a typo'd password, and the real cause (env loading) left no trace.

No more silent default — a missing/empty ADMIN_EMAIL or ADMIN_PASSWORD now answers "Admin authentication not configured" at login (the same actionable error the password path already used) and logs a startup warning naming exactly which variable is missing.
The compose files are unaffected: they already supply admin@example.com as a visible env-level fallback (${ADMIN_EMAIL:-admin@example.com}). Correctly configured deployments see no change.

Apps — in-instance app hosting

Cortex can now host self-contained web apps built from the Cortex App Template. An admin uploads a packaged app (a zip of a static bundle plus a manifest) in Settings → Apps; it is served sandboxed under /apps/{id}/ and reaches the Cortex API only through a proxy that enforces the manifest's endpoint allowlist. Off by default (ENABLE_APPS=true). See the Apps feature guide.

Least-privilege by construction — each install mints a dedicated API key scoped to exactly what the manifest declares (read or read-write, optionally restricted to chosen collections at install time via a collection picker). The browser never holds a real key: apps receive a short-lived, auto-renewing token, and the proxy attaches the minted key server-side. Uninstalling revokes the key.
Sandboxed — apps run in an iframe without allow-same-origin (opaque origin, no cookie access), each served with a per-app Content-Security-Policy. Config values (if the app declares any) use the same encrypted-secret storage as skills.
Platform capabilities — a type: "platform" app can make external API calls through a server-side http capability: the call executes from the backend with credentials injected from the app's encrypted config, so the external service needs no CORS configuration and secrets never reach the browser.
Background tasks that survive a closed tab — platform apps can declare a tasks capability and submit declarative step-queues: JSON programs of http / cortex / llm / store / template steps that Cortex executes server-side in its async loop, with per-item error isolation, pause/resume/cancel/retry controls, and automatic resume after a server restart. A task with "schedule": {"everyMinutes": N} re-runs on its own — turning an integration app into a true sync daemon (e.g. pulling newly added paperless-ngx documents into Cortex every hour with no browser open). Every step passes the same security gates as the interactive paths: http steps are host-allowlisted and SSRF-guarded, cortex steps are checked against the manifest's endpoint allowlist, and llm steps run the instance's configured model, metered like any other completion, with built-in chunking and output-validation policies (length-ratio + word-overlap guards) for long-text workloads.
Per-app storage — a storage capability gives an app its own quota-capped, strictly namespaced key/value store (per-app SQLite in the apps volume), usable both from the app's UI and from task steps; it also powers the task engine's built-in fan-out deduplication (skipIfStored). Storage and task state survive app upgrades — reinstalling a new version keeps dedup keys, cursors, and schedules intact.
Host-scoped credentials for multi-host apps — a config var may declare auth_host (literal hostname or ${CONFIG_VAR} reference) so its auth_header is injected only on calls to that host. Without it, an app declaring several external hosts would send every credential to all of them (e.g. a Venice API key on YouTube calls). Single-host apps need no change. The platform http envelope also accepts extra request headers now (auth/framing names stripped server-side; app-authored Cookie values allowed — needed for e.g. YouTube's EU consent bypass).
Share links — for apps that allow it, an admin mints revocable share links (/a/{id}?g=…) so people who are not Cortex users can open an app without logging in; a share visitor's token validates only at that app's proxy and never grants access to the rest of Cortex. Share-link visitors are read-only for platform state: storage writes and task control require an owner or editor token.
New environment variables (all optional; documented in the configuration reference): ENABLE_APPS (default false), APPS_DIR (default .agents/apps, persisted via the new apps_data volume), APP_MAX_PACKAGE_MB (default 50), APP_TOKEN_TTL_SECONDS (default 900), APP_PROXY_UPSTREAM, APP_HTTP_TIMEOUT (default 30); for storage and tasks: APP_STORAGE_MAX_MB (default 50), APP_STORAGE_MAX_VALUE_KB (default 1024), APP_TASK_MAX_ITEMS (default 2000), APP_TASK_MAX_CONCURRENCY (default 4), APP_TASKS_GLOBAL_CONCURRENCY (default 8), APP_TASK_MIN_SCHEDULE_MINUTES (default 15), APP_TASK_LLM_CALLS_PER_RUN (default 500), APP_TASK_STEP_OUTPUT_MAX_KB (default 2048), APP_TASK_MAX_PER_APP (default 50).
Task oversight in the admin panel — each installed app's expanded view now shows its background tasks (status, schedule, progress counts, errors) with pause/resume/cancel/retry/run-now controls, so a failing scheduled sync is visible without opening the app.
Install from the registry — a new Browse Registry panel (Settings → Apps) lists the public Cortex Registry catalog with each app's declared scope, endpoints, and capabilities. Installs are checksum-verified end to end: the instance downloads the release artifact and verifies its catalog-pinned sha256 (and exact size) before unpacking anything; updates surface in place and preserve config, keys, share links, storage, and schedules. Catalog location: APP_REGISTRY_URL (default: the official registry; empty disables browsing). Publishing an app = a GitHub release + one PR — the registry's CI re-downloads and re-verifies every artifact.
No API or schema breakage: the subsystem is filesystem-backed with no Neo4j schema changes, and every route is inert unless ENABLE_APPS=true.

July 17, 2026

Cortex can now sell pay-per-query access to its retrieval endpoints via the open x402 payment standard. Free member keys keep working exactly as before; in parallel, admins can mint monetized public keys that agents pay per query in stablecoins, with revenue flowing straight to a wallet the owner controls — a way to earn on domain-specific knowledge and subsidize members' inference.

x402 Payments

One env var, everything else at runtime — X402_ENABLED=true (default false) unlocks a new Settings → x402 Payments section; recipient wallet, facilitator URL, network, and asset are configured there, stored in Neo4j (survive redeploys, excluded from library export and system reset). Facilitator auth headers are encrypted at rest via ENCRYPTION_KEY.
Verification before money moves — a one-click verify suite checks wallet address format (EIP-55 checksum / Solana base58), facilitator reachability, and scheme+network support, reported check-by-check. Priced keys can only be created — and paid requests only served — while the saved config is verified; any payment-relevant change re-requires verification.
Vendor-agnostic by design — Cortex implements the x402 v2 facilitator interface (/verify, /settle, /supported) over plain HTTP: any spec-compliant facilitator works, no vendor SDK, no chain RPC, no private keys on the server.
Monetized public keys are locked down at the API level — read-only by construction (a price can never be combined with manage), restricted to the four retrieval endpoints (/api/search + the Ask AI endpoints; raw document/file access and stats are off-limits), and still collection-scopable — sell exactly the slice of knowledge you choose. New cortex_pub_ key prefix.
Settle-before-serve payment flow — unpaid requests get HTTP 402 + PAYMENT-REQUIRED (base64 requirements per the x402 HTTP transport); paid requests are verified and settled through the facilitator before the query runs, with the settlement receipt returned in PAYMENT-RESPONSE. Facilitator outages fail closed (503, never served free).
Earnings dashboard — settled totals overall and per key in the admin section, every payment recorded with its on-chain transaction hash for independent auditability.
Honest analytics for paid keys — HTTP 402 challenges are excluded from per-key error-rate tracking: a challenge is the normal first leg of every x402 handshake, not a failure.
Two-tier pricing: deep research costs more — each monetized key carries a research_multiplier (default 10, set at key creation with a live dual-price preview): agentic deep-research queries are quoted at price × multiplier in the 402 challenge itself, so agents see the true cost before signing; quick asks and search stay at the flat rate. 0 disables deep research on a key entirely (403). Requests that would fail validation are rejected before payment — a payer can never settle for a guaranteed 422.
Paid queries still consume MAX_QUERIES_PER_MONTH units — the quota meters inference cost regardless of who pays.
No API or schema breakage: existing keys, clients, and flows are unaffected while the flag is off (default) and for all non-priced keys when it is on.

July 13, 2026

The duplicate-entity scan behind Manage → Deduplicate was the least reliable surface on large graphs: requests timed out at the proxy while the server kept computing, every retry stacked another CPU-heavy scan on top, and on graphs beyond roughly 20k entities the scan could exhaust the container's memory and crash outright. The scan has been rebuilt end to end.

Memory-bounded, faster matching

The similarity computation no longer materializes full n×n score matrices (4 GB at 32k entities — an instant out-of-memory in a standard 4 GiB container). Matching now runs in fixed-size row blocks (~128 MB peak) over the upper triangle only, so peak memory stays flat no matter how many entities the graph holds and redundant pair comparisons are halved. A 40k-entity graph scans in ~35 seconds well within a 4 GiB container.
Scan results are now fully deterministic: repeated scans over an unchanged graph return identical groups and canonical suggestions.
Grouping semantics are unchanged (typo matching, word reordering, Person-aware name-prefix gating, star clustering) and locked by a new parity test suite.

No more proxy timeouts: single-flight scan with progress

One scan runs at a time; identical requests join the running scan instead of stacking new ones — a stampede of retries can no longer saturate the CPU.
If a scan outlasts DEDUP_SCAN_WAIT_SECONDS (default 25s, deliberately below typical proxy read timeouts), GET /api/entities/duplicates answers 202 {"status": "running", "progress": 0.42} and the client polls the same URL until it returns "status": "complete". The web UI does this automatically and shows live progress on the scan button.
Completed results are cached server-side for DEDUP_SCAN_CACHE_TTL_SECONDS (default 600s) per parameter set — reopening the page is instant. Entity merges invalidate the cache; refresh=true forces a fresh scan.

Ingestion injection scan is now experimental & opt-in

The ingestion-time prompt-injection scan is reclassified as experimental and disabled by default. On a default instance the feature is completely absent: no scan runs at ingestion (not even the free heuristic), the toggle no longer appears under Admin → System Configuration → Features & Security, and the "Injection Flagged" document filter is only offered when at least one document is actually flagged.

New env var ENABLE_INGESTION_INJECTION_SCAN (default false) — the master flag that activates the feature. Once set, behavior is unchanged from before: heuristic + optional LLM classifier, flag-only (never blocks), with the classifier admin-toggleable at runtime via the existing INGESTION_INJECTION_SCAN default. With the master flag off, PATCH /api/admin/config rejects the ingestion_injection_scan field with 400.
Query-time defenses (regex detection, Prompt Guard, output filtering, and untrusted-content fencing of retrieved chunks) are unaffected and remain on by default.

July 11, 2026

Security hardening pass

A round of defense-in-depth tightenings across configuration defaults, outbound requests, and access scoping. Correctly-configured deployments are unaffected; the changes mainly raise the floor for the default and misconfigured cases. No API changes.

Production hardening is now the default on the deploy paths. The docker-compose.prod.yml, coolify, and dokploy compose files set ENVIRONMENT=${ENVIRONMENT:-production} (still overridable), so a standard deploy runs with secret enforcement, response-error sanitization, and API docs disabled without an extra step.
Stronger secret hygiene at startup. .env.recommended now ships clearly marked placeholders (generate real values with openssl rand -hex 32) rather than sample values, and startup — both the backend validator and the Next.js frontend — refuses known-placeholder or too-short signing secrets in production. Empty admin credentials still fail closed as before.
Outbound-request safety (SSRF guard). A shared guard (services/ssrf_guard.py) validates the target of requests influenced by user or agent input — the skill http_request tool, Web Import, and the git connector — resolving the host and blocking loopback/link-local/metadata addresses (and, for the agent tool, private ranges by default), re-checked on each redirect hop. Per-host allowlists and SKILL_HTTP_ALLOW_PRIVATE / WEB_IMPORT_ALLOW_PRIVATE cover legitimate internal targets; self-hosted git on a private network keeps working.
Tighter collection scoping. Restricted API keys are now checked against a document's source collection (not just the target) when moving or adding documents, so a key stays within its granted collections. No effect on admin/unrestricted keys.
Safer library import. Imported-archive skill identifiers are validated before any files are written, keeping restored content inside the skills directory.
Rate-limit and frontend hardening. The opt-in rate limiter now keys only on trusted identity with an always-applied per-IP bucket; the login redirect is restricted to same-site paths; and the frontend sends baseline security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy).

Generate Graph chain survives restarts

Fixed: a server restart or redeploy mid-run no longer strands the Generate Graph pipeline after Step 1. The 3-step chain (extraction → relationship analysis → community detection) is orchestrated in-process, so a restart used to lose the remaining chain — the startup auto-resume restarted Step 1 without it and the pipeline visibly stopped once Step 1 finished. Every pipeline task now persists its resume parameters (remaining chain, concurrency, scope) alongside its task record; on boot, Cortex resumes the interrupted step with its chain — Step 1 continues into Steps 2 and 3, an interrupted Step 2 or 3 restarts directly.
Also covered: a run killed between queueing and its first document (documents sat at pending forever with nothing to resume them) and the case where an upload was mid-processing while a Step 2/3 task was interrupted (Step 1 resumes first, then the interrupted step). All resume paths remain quota-guarded behind AUTO_RESUME_PENDING_ON_STARTUP.

July 10, 2026

Two days of live extraction-tuning (July 9-10) distilled into shipped defaults plus a new low-friction onboarding path — a fresh install now runs the production-validated configuration without setting a single tuning knob.

Tuned defaults — the validated values now ship in code

GRAPH_EXTRACTION_MAX_CONTEXT / EXTRACTION_MAX_OUTPUT_TOKENS now default to 16000 / 16000 (were 0-inherit-clamped-at-48k and 12000). The pairing is production-validated: zero truncated batches, zero dropped entities across multi-day live runs including entity-dense documents (indexes, bibliographies — dense-batch output peaked ~13.4k under the 16k cap, versus ~25% truncation at the old settings). Explicit values are still honored unclamped; set 0 explicitly to restore the inherit chain.
OPENAI_MAX_CONTEXT defaults to 256000 (was 32768), matching the recommended primary (Gemma4 26B A4B) — it's the retrieval agent's working context and deliberately does not flow into extraction.
EMBEDDING_MODEL defaults to text-embedding-3-small (bare id, was the LiteLLM-style openai/text-embedding-3-small) — the string is sent verbatim to the embeddings API, so the bare id is correct for direct OpenAI and Venice-class gateways alike.
Extraction is decode-bound — output scales with input, so the context budget is a batch-size / graph-density dial sized to provider decode speed, not the model's window. The terse-description extraction prompt (short ~12-word entity descriptions; enrichment restores depth) is what bounds output-per-entity and makes the matched 16k/16k ceiling hold on dense docs.
Default-config deployments are flagged for a graph refresh after upgrading (the extraction fingerprint changes — intended, the new defaults are the tested values). Deployments that pin these vars in their env see no change.

Zero-config onboarding: `.env.recommended`

.env.recommended rewritten as the recommended starting point: cp .env.recommended .env, fill the secrets block + one API key, done. Secrets first, then the bench-validated model stack — Gemma4 26B A4B (google-gemma-4-26b-a4b-it) as the primary agent model, Qwen3.6 27B (qwen3-6-27b) for knowledge-graph generation and vision, text-embedding-3-small/1536 embeddings — with commented *_API_BASE/*_API_KEY escape hatches per tier (any model from any OpenAI-compatible API works) and explanatory notes at the bottom.
ENCRYPTION_KEY joins the standard secrets block — it encrypts git connector PATs and secret-typed skill config at rest (empty = plaintext with a startup warning; setting it later auto-encrypts on the next boot). Generation + rotation guidance included.
The docs Quickstart, the setup skill (including the autonomous agent-install flow, which now generates an ENCRYPTION_KEY automatically), and the hermes cortex.sh setup helper all lead with this path and write the same stack — one consistent story across every install door.
Docs accuracy sweep: .env.example audited programmatically against config.py (every stated default verified), stale 24000/12000 budget advice removed everywhere, EMBEDDING_SEND_DIMENSIONS docs now name fixed-dimension models (e.g. Qwen/Qwen3-VL-2B-Instruct) that require false.

Self-healing image analysis on restart

A restart used to strand image analysis permanently: it runs as in-process background futures after a document's text pipeline completes, so the document was already completed when the futures died — the counters froze at current < total, Step 1 showed "finishing image analysis" forever, and the startup recovery (which only matches in-flight statuses) never saw it. Observed in the field: 34 documents stuck at 8/2023 images with zero LLM traffic.

On boot, Cortex now detects completed documents with unfinished image analysis and resumes them automatically (AUTO_RESUME_IMAGE_ANALYSIS, default true; quota-guarded, visible as an image_analysis_resume task).
Already-paid work is never redone: images are re-extracted via local Docling re-conversion (CPU only — no LLM cost), and only images whose analysis chunk isn't stored yet are sent to the vision model. Text chunks, entities, and relationships are untouched.
Irrecoverable documents (source file deleted, vision model no longer configured, re-conversion finds no images) get their counters closed with an explanatory message instead of reading as in-flight forever; conversion failures leave them queued for retry on the next startup.

Abort Graph Generation

New "Abort Generation" control on the Knowledge Graph (/extract) page. A running generation can now be stopped from the UI — and the abort actually stops the work: it cancels the reprocess batch and the Step-2/3 chain, kills in-flight document conversions (a docling worker subprocess previously survived cancellation and kept pinning CPU/RAM), and resets stranded documents back to pending so on-page counts reflect reality.
The button appears whenever any pipeline task is running; when idle, the primary action reads Generate Graph (no entities yet) or Regenerate Graph (entities exist).

Extraction transport overhaul — no more retry storms

Run telemetry from a 7-hour field ingest showed ~40% of extraction calls "timing out at 361s" — actually three 120s SDK attempts silently re-sending the same prompt into a saturated endpoint.

LLM_REQUEST_TIMEOUT_SECONDS now truly governs extraction — tier clients no longer hardcode 120s; one attempt gets the full window.
No SDK retries where splitting is the retry — entity batch and per-chunk relationship calls run one-shot (max_retries=0); timeouts go straight to split-retry, 429s requeue with bounded exponential backoff.
Timeout-adaptive batch sizing — a proven-surviving batch size is remembered per extraction config, so later documents skip the split cascade (the field log showed 15 planned batches ballooning to 77 calls / 4.8 hours).

Extraction observability

Batch-start progress (Finding entities: 120/741 chunks (batch 8/22)…) so multi-minute calls no longer read as a hang; the denominator grows as batches split, making retry churn visible as forward motion.
Per-batch telemetry (tokens, duration, finish reason), one-shot budget/timeout diagnoses naming the exact knobs to turn, a run health summary persisted on the document (extraction_stats), and repetition-loop detection for model degeneration floods (observed: 288 "entities" from one 5-sentence chunk).

Branding

Footer now reads "Built by MOCA" with the Museum of Crypto Art logo and Twitter link; the README carries the same links with a theme-aware logo.

July 8, 2026

A deep reliability pass over the entire document-ingestion pipeline, driven by live debugging of large-PDF ingestion (multi-hundred-page books) against a hosted inference provider: conversion no longer runs out of memory, embedding and extraction failures self-heal instead of silently degrading the graph, a new compact extraction output format cuts structured-output overhead, and the UI finally shows truthfully what the pipeline is doing at every stage. Plus full-stack error tracking via GlitchTip. No API or schema breakage.

Large-document conversion & memory

Docling worker heap release — the conversion worker returns freed page-image memory to the OS after every page chunk (gc + malloc_trim). Previously the allocator retained it, and conversions of 300+ page PDFs were OOM-killed inside the container's memory limit. Books of any length now convert reliably within a bounded footprint.
Live conversion progress — the worker's per-page-chunk progress streams into the document's progress state (Converting document: page 150/320...) instead of surfacing only after the process exits; a queued document now says Waiting for a conversion slot... instead of falsely claiming to convert.
pypdf's per-object warnings on malformed PDFs no longer flood the logs.

Embedding input guards & recovery

Token-accurate input capping — oversized chunks are sub-split against real token counts (tiktoken, char-heuristic fallback) instead of a chars-per-token approximation that undercounted punctuation-dense text (tables of contents, indexes, bibliographies). Zero content loss; EMBEDDING_MAX_INPUT_TOKENS now defaults to 5400 to stay under gateway-side token validators (see the new "Provider Notes: Venice" section in Configuration).
Per-chunk recovery — a batched embed request rejected by the provider no longer strips embeddings from every chunk in the batch: affected chunks are re-embedded individually (rate-limit aware, with automatic halving for inputs the provider still refuses). Image-derived chunks get the same treatment.
Empty/whitespace-only chunks are dropped before embedding, and a failed embed pass is retried once before failing the document.

Extraction self-healing & budgets

Timeout and truncation split-retry — an extraction batch that times out or overflows its output budget is split in half and retried instead of being silently dropped. Previously an oversized batch could lose all of its entities without any error surfaced on the document.
Inherited context clamp — when GRAPH_EXTRACTION_MAX_CONTEXT is unset, the value inherited from OPENAI_MAX_CONTEXT is now clamped at 48000: a chat model's large context window must not dictate extraction batch sizing, since hosted providers cannot decode a fully-packed prompt inside a request timeout. Explicitly set values are honored as-is.
Token estimation for extraction batching uses tiktoken's cl100k_base for non-OpenAI models instead of a chars/4 heuristic.
Entity extraction reports per-batch progress (Finding entities: 240/1511 chunks...) instead of a static message for the whole phase.

Compact extraction output format

Entity and per-chunk relationship extraction now instruct the model to emit a compact line protocol (ENT|Name|Type|Description, REL|Source|Target|TYPE|weight|confidence|Description) instead of XML elements — the XML scaffolding consumed a large share of every response's output tokens, and structured-output latency on hosted providers is decode-bound, so scaffolding directly inflated wall time and pushed dense batches past request timeouts. Stored graph content (names, types, descriptions, weights) is unchanged.
XML parsing is retained as an automatic fallback for prompt drift and older models; the batched per-chunk path keeps its <chunk index> grouping with compact lines inside.
The extraction prompt version is part of the reprocess fingerprint, so upgraded instances re-extract documents on their next reprocess instead of delta-skipping them.

Ingestion progress UI

Phase stepper on the Documents page: processing documents show the pipeline phases (Convert → Chunk & Embed → Store → Extract) with live counts, an overall bar, and a within-phase bar — replacing a single opaque message. Image analysis renders as a parallel track, matching how it actually runs.
Per-document breakdown on the Knowledge Graph page while Step 1 runs, with compact steppers and per-document image-analysis bars — replacing the aggregate "Processing N documents..." banner.
Pending documents distinguish "queued by the system — waiting for a processing slot" from plain "Unprocessed"; stale image counters from interrupted runs are no longer displayed.

Prompt-injection scan accuracy

The ingestion-scan heuristics are now word-bounded and require instruction-domain anchors, eliminating false flags on ordinary book-scale prose ("firmly rooted", "reflects all aspects", "encoded … instructions", "show only the physical configuration") while the full attack corpus remains detected. Documents previously flagged by the over-broad patterns are cleared on their next scan.
The jailbreak pattern joined the same tightening: DAN is matched case-sensitively and all trigger words are word-bounded, so prose like "the Main–Danube canal — the modest 1992 waterway" no longer trips the flag; "escape" now requires an instruction-domain anchor ("press Escape at the prompt" is fine, "escape your instructions" is not).
Heuristic hits are no longer final verdicts on documents. When the LLM classifier layer is enabled, a regex hit only escalates: the classifier re-judges the matched region and the document is flagged solely on its confirmation. Regex-only flagging remains only when the classifier is toggled off or unreachable. This ends the class of complaints where a clearly safe document carried an "Injection Flagged" badge because a chat-input regex fired on long-form prose.

Provider notes: Venice (docs)

New Configuration section with measured tuning guidance for running the ingestion stack against api.venice.ai: extraction batch sizing against serving speed, embed-input validation behavior, rate-limit and concurrency-slot characteristics, and the HTTP-200 error-envelope quirk. The previous guidance recommending full-window extraction budgets was corrected accordingly.

Error tracking with GlitchTip

Full-stack crash and error reporting via a self-hosted GlitchTip instance (Sentry-protocol compatible). Backend and frontend report to separate GlitchTip projects with readable stack traces on both sides — Python events carry source-context lines out of the box, and production frontend builds upload source maps so browser errors show original TypeScript instead of minified chunks. Entirely opt-in and env-driven: without a DSN the images run exactly as before.

Backend error tracking

SENTRY_DSN (default: empty) activates sentry-sdk at startup — unhandled endpoint exceptions and ERROR-level logs from any code path (API, background ingestion, the Docling conversion worker) become GlitchTip events. The client-facing 5xx sanitization is unchanged; capture happens before it.
Log correlation — every event is tagged with the same request_id that appears in server logs and X-Request-ID response headers, plus a service tag (backend / docling-worker).
Privacy deny-by-default — request bodies are never attached (SENTRY_MAX_REQUEST_BODY_SIZE=never) and PII capture is off (SENTRY_SEND_DEFAULT_PII=false), mirroring the Langfuse content-masking posture. SENTRY_TRACES_SAMPLE_RATE=0 keeps it errors-only by default.

Frontend error tracking

@sentry/nextjs wired for browser, Server Components, and proxy errors (instrumentation-client.ts, instrumentation.ts + onRequestError, global-error.tsx), enabled by NEXT_PUBLIC_SENTRY_DSN/SENTRY_DSN (empty = disabled).
Readable production stack traces — when SENTRY_URL, SENTRY_ORG, SENTRY_PROJECT, and SENTRY_AUTH_TOKEN are provided at image build time, next build uploads source maps as debug-ID artifact bundles (GlitchTip ≥ 4.2, verified against Turbopack builds) and deletes the .map files afterward so they are never served publicly.
Deploy wiring — dev/prod/Dokploy/Coolify compose files map SENTRY_DSN_BACKEND/SENTRY_DSN_FRONTEND onto each container so the two apps report to their own projects; all values optional with empty defaults.

July 7, 2026

A resilience-focused release: every change is about keeping an instance healthy under real-world failure — provider hangs, database restarts, oversized requests, redeploys mid-ingestion — and giving operators a compliance-grade audit trail plus a backup story that is verified on every run and proven restorable. It also closes the failure modes that stay invisible until the day they matter: backups that silently produce nothing, background work that dies without a trace, and a failing git sync that hammers the provider forever. No API breakage.

Backups you can actually restore

Backups on every deploy path — the Coolify and Dokploy stacks now include the nightly backup sidecar that previously shipped only with the standalone prod overlay. Chat data is included where the chat service runs.
Server-side graph export — the backup sidecar no longer streams the APOC export through a client-side quoting round-trip (which could silently produce an empty or subtly corrupted graph.cypher). The neo4j service now writes the export itself (NEO4J_apoc_export_file_enabled=true + the backups volume mounted at its import directory): byte-exact output, compressed to graph.cypher.gz.
Verified before it counts — exported row counts are checked against the live database, a SHA256SUMS manifest is written, and the run is stamped .complete + LAST_SUCCESS only after all checks pass. Retention now runs only after a verified success and never deletes the newest complete backup — consecutive failures can no longer rotate away the last good copy.
Failures are visible — the sidecar has a compose healthcheck that goes unhealthy when the newest verified backup is older than 2× the interval. The first backup runs ~2 minutes after start instead of a full interval later.
A tested restore path — new /restore.sh <timestamp>: verifies checksums, wipes, replays, and reports counts; validated end-to-end (backup → restore → byte-identical data, including adversarial newline/quote/backslash content). The handbook and deployment guide now document this runbook instead of a neo4j-admin dump procedure that could not restore sidecar backups.
Coolify fix — the Coolify backup sidecar referenced an env var the stack never sets (NEO4J_PASSWORD instead of SERVICE_PASSWORD_NEO4J) and could not authenticate; corrected.

Self-healing background work

Stranded-document sweep — if a document's failure-status write itself loses Neo4j past its retries, the document no longer sits in "processing" until the next restart: an hourly sweep resets transient-state documents that have no live in-process task and no status heartbeat for 30+ minutes.
Dead-task reaper — a background task whose coroutine died without reporting completion is marked failed after 2 hours without a progress heartbeat, instead of living in memory forever, blocking new syncs for its git connection and pinning safe_to_redeploy: false until a restart.
Abandoned import uploads — stale chunked library-import upload sessions are now swept hourly (previously only when the next upload started).

Git connector

Event-loop protection — the sync engine's per-file database and disk work now runs off the event loop; syncing a multi-hundred-file repository no longer stalls concurrent requests (including /health) for the duration of the ingest.
Failure backoff — a connection whose scheduled sync keeps failing (revoked PAT, oversized repo, network) now backs off exponentially (capped at 24h) instead of being re-cloned on every 5-minute scheduler tick, and reports sync_status: error. A successful sync resets the backoff.

Health checks that actually gate

GET /health now answers HTTP 503 when the instance is degraded (Neo4j unreachable or schema not yet confirmed) instead of 200 with a degraded body. Compose healthchecks, depends_on: service_healthy gates, and health-aware proxies key off the status code, so a booting or broken instance is no longer reported healthy at the transport level. The backend healthchecks gained a 60-second start_period to cover schema initialization at startup.

Quieter outages

Background loops (task persistence, usage-meter flush, schema-init retry, git scheduler) now rate-limit their failure warnings to one per 5 minutes per source, with a suppressed-repeat count — a Neo4j outage no longer produces tens of identical log lines per minute.

Resilience & data safety

Neo4j transient-error retry — idempotent reads and status upserts retry automatically (3 attempts with backoff) across a Neo4j restart instead of failing the request or stranding a document in failed.
No more phantom 401s — API-key validation previously turned any Neo4j hiccup (restart, brief lock contention) into 401 Unauthorized, which clients rightly treat as "your key is invalid" and surface as auth errors. Validation now retries transient errors, answers 503 + Retry-After when the auth store is genuinely unreachable (401 is reserved for keys that were checked and rejected), caches successful validations in-process for API_KEY_CACHE_TTL_SECONDS (default 30s, invalidated instantly on key changes) so a page-load burst validates once instead of a dozen times, and no longer fails a request when the best-effort last_used_at bookkeeping write fails. Per-key usage recording moved off the event loop.
Cross-tenant DNS misrouting on shared proxy networks — on hosts running multiple cortex stacks, internal server-side calls to http://backend:8000 could resolve to another tenant's backend: every stack auto-aliases backend on the shared Dokploy/Coolify proxy network, and Docker DNS aggregates same-name aliases across networks — the wrong tenant then rejects the caller's keys, surfacing as random 401s in chat and frontend that no amount of backend-side resilience can prevent. The deploy composes now give the backend a cortex-backend-internal alias on the stack-private network and all internal consumers dial that alias, making cross-tenant resolution impossible.
Server-side query timeout — the deploy composes set a Neo4j transaction timeout (CORTEX_NEO4J_TX_TIMEOUT, default 300s) so a runaway query can no longer pin a connection and an API worker forever.
Schema init that can't be skipped silently — constraint/index creation now retries with backoff at startup and keeps retrying in the background; /health reports the new schema_initialized field and stays degraded until the schema is confirmed, so deploy gates wait instead of routing traffic to an instance with broken dedup and dead vector search.
Auto-resume after redeploys — documents stranded mid-processing by a restart are not just reset to pending but automatically resumed (AUTO_RESUME_PENDING_ON_STARTUP, quota-guarded). Bulk uploads parked deliberately with start_processing=false stay parked.
LLM transport limits — every LLM client the backend builds now carries an explicit timeout (LLM_REQUEST_TIMEOUT_SECONDS, default 360s) instead of the SDK's 600s default, so one hung provider connection can't pin an ingestion slot for 10 minutes.
Task state survives restarts — background task records (batch processing, relationship analysis, community detection, library export/import) are now write-through persisted to Neo4j. After a redeploy, polling a task no longer returns 404: interrupted tasks report failed with "interrupted by server restart", terminal states are preserved for 7 days, and stale records are pruned automatically every hour.
Free-disk guardrail — uploads, reprocessing, and library-import sessions are refused with 507 Insufficient Storage when accepting the payload would leave the uploads filesystem below MIN_FREE_DISK_MB (default 500) — disk-full corrupts Neo4j checkpoints, so refusing new data early is strictly safer. Operators get cortex_disk_free_bytes / cortex_disk_total_bytes gauges and a rejection counter in /metrics, plus disk_free_mb / disk_total_mb on GET /api/stats.

Request hygiene

Request-body ceilings — a new middleware rejects oversized bodies with 413 before they can pressure the container's memory: MAX_REQUEST_BODY_MB (default 32) globally, MAX_FILE_SIZE_MB + slack on upload routes, and MAX_IMPORT_BODY_MB (default 2048) on library import, which streams to disk. The reprocess upload path now enforces its size cap mid-stream instead of buffering the whole file first.
Event-loop protection — the hybrid-search endpoint, topic-hint generation, and the git-connector endpoints moved their blocking database/embedding work off the event loop; deleting a git connection with document purge now uses one batched delete. Under load these paths can no longer stall every concurrent request (including health checks).
Sanitized server errors — in production, 5xx responses return a generic message plus the request ID; the full detail stays in server logs. Exception internals (connection URIs, provider error bodies, file paths) no longer reach clients.

Audit trail (new)

ENABLE_AUDIT_LOG=true activates an append-only JSONL audit log: authentication failures, key-attributed mutating requests (uploads, deletions, configuration changes, key CRUD), and search/ask activity — each event carrying the acting API key, outcome, and request ID. Metadata only: document content and query text are never written, consistent with the observability masking stance. Fail-open by design and size-rotated. See the Security guide for details.

July 6, 2026

A security-focused release that hardens Cortex against prompt injection on both the paths an attacker can reach: the questions users ask, and the content you ingest. No single layer is trusted on its own — detection, a model-based classifier, output filtering, untrusted-content fencing, and an ingestion-time scan each narrow what the others miss. All layers are best-effort by design and fail safe (a defense being unavailable never breaks a question or an ingest).

Query-time detection & output filtering

Obfuscation-resistant detection — user questions are normalized before pattern matching: NFKC unicode folding defeats fullwidth/homoglyph tricks (e.g. ｉｇｎｏｒｅ) and zero-width/soft-hyphen stripping defeats keyword-splitting (ig<ZWSP>nore). The special-character heuristic was retuned so legitimate code/markup questions no longer trip it. On the chat and research entry points a detected injection returns a safe refusal, not an error.
Real streaming output filter — token-streamed answers now pass through a sliding-window filter (filter_stream) that redacts verbatim system-prompt phrases and structural role tags before they reach the client, catching leaks that span multiple chunks. Wired into both the fast and standard chat loops.
Research pipeline gated — the deep-research entry point now validates its own input, so every question path is covered.

Prompt Guard — query-time classifier (new)

A dedicated prompt-injection classifier model (PIGuard, MIT-licensed) can now screen each question before retrieval, as a second gate on top of the regex layer. It catches jailbreak/injection phrasings patterns miss, and is specifically tuned to avoid over-flagging benign questions that merely contain trigger words.

Zero per-instance cost — the model is hosted once per machine by the cortex-helper service (alongside the reranker) and called over HTTP, so no backend loads it into its own memory.
Opt-in, fail-open — active only when PROMPT_GUARD_SERVICE_URL points at a helper; if the helper is unreachable, questions proceed unguarded (availability over strictness). Flagged questions get the same safe refusal.
Runtime toggle with a clear cost — admins flip Prompt Guard in System Configuration → Features & Security without a restart. Each guarded question costs one extra query unit (metered toward the monthly quota) and appears as a prompt_guard.classify entry in Langfuse tracing; the toggle lets you trade that cost off. New settings: PROMPT_GUARD_SERVICE_URL, PROMPT_GUARD, PROMPT_GUARD_THRESHOLD.
Self-host without a helper — set PROMPT_GUARD_LOCAL=true to load the classifier in-process instead (mirrors the local reranker fallback). Off by default, since it adds resident memory; the shared-helper path stays the recommendation for multi-tenant hosts.

Untrusted-content fencing (indirect injection)

Retrieved chunks, merged graph context, and researcher tool results (knowledge search, http_request, skill outputs) are now wrapped in data-boundary markers, and the model is instructed to treat everything inside as data, not instructions — a defense against second-order injection planted in your own documents or fetched web/tool content.
A phrase-only heuristic scan annotates fenced content with an inline caution on a hit; content is logged but never dropped.

Ingestion-time document scan

Every ingested document (uploaded, web-crawled, or git-synced) is scanned once for planted injection instructions. A free heuristic always runs; an optional LLM classifier additionally scans the text when enabled.
Detected documents are flagged, never blocked — they stay ingested and answerable (and fenced at query time), and surface as an "Injection Flagged" badge/filter in the document list.
First runtime-editable setting — the LLM scan is on by default (INGESTION_INJECTION_SCAN) and admin-toggleable at runtime (persisted as a setting override, effective for new ingestions without a restart). Turning it off keeps the free heuristic but skips the per-document classifier query.

Deployment note (AaaS / shared-helper hosts): Prompt Guard is served from cortex-helper. Deploy the updated helper (it pre-caches the classifier), then provision PROMPT_GUARD_SERVICE_URL (= your helper URL) and PROMPT_GUARD into tenants. Self-hosters without a helper are unaffected by default — the guard stays inert (or set PROMPT_GUARD_LOCAL=true to run it in-process), and all other layers work unchanged.

July 4, 2026

The monthly quota now measures what actually costs money, silently broken documents are now visible and fixable from the UI, and the knowledge-graph build pipeline got a pass focused on API-call economy and robustness on large libraries.

Unit-denominated usage metering

MAX_QUERIES_PER_MONTH is now denominated in units — internal LLM completions — instead of HTTP requests, so quota consumption tracks inference cost directly. Every successful model call counts one unit: each step of a Q&A answer (typically a handful per question) and each extraction/analysis call during document processing (a few per file). Embeddings are free.

Document processing is now metered — previously, imports and graph rebuilds consumed unlimited model calls outside the quota. Now uploads, custom inputs, reprocessing, web imports, git syncs, and the graph-build steps all draw from the same monthly pool.
In-flight work always finishes — the quota blocks starting new work (429 + Retry-After until the next UTC month). A batch import stops cleanly between documents; skipped documents stay pending and resume next month.
Usage meter in the admin UI — the Settings page's Statistics panel shows a monthly usage bar with a queries vs. processing breakdown, turning amber at 80% and red when exhausted. GET /api/stats exposes monthly_usage_used, monthly_usage_limit, monthly_usage_query, monthly_usage_processing.
Fleet orchestration surface — GET /api/instance/status (admin) now also reports library size (document_count, entity_count, collection_count) and the same monthly usage fields, so deployment automation can read plan consumption from the endpoint it already polls for redeploy safety.

Failed & degraded documents

A document can complete processing yet be silently broken — e.g. an extraction timeout leaving it with 0 entities, or chunks missing embeddings (invisible to semantic search). These are now flagged as degraded:

New Degraded status filter and amber warning badge on the Documents page, with the reason shown on the card ("0 entities extracted" / "N chunks missing embeddings").
Combined failed/degraded banner with one-click Select all for bulk reprocess.
Reprocessing a degraded document automatically bypasses the "content unchanged" skip.
Signals (entity_count, unembedded chunk count) are persisted at processing time and backfilled once at startup for existing libraries.

Graph builds: leaner extraction & rebuild hardening

Graph content and quality semantics are unchanged; builds complete faster and make substantially fewer LLM calls, especially under provider rate limits.

Entity extraction skips redundant summary calls — the per-document summary is now generated as extraction context only when a document spans multiple extraction batches. Single-batch documents (the common case) already carry their full text in the extraction prompt, so no separate summary call is made for them.
Entity extraction is protected against output truncation — a batch whose LLM response hits the output-token cap is automatically split in half and retried, so entity-dense documents keep their tail entities. A single chunk that still truncates logs a warning naming EXTRACTION_MAX_OUTPUT_TOKENS.
Batched extraction accepts explicit empty results — when the model returns an explicitly empty <chunk> block in a batched relationship call, that is accepted as a deliberate "no relationships" answer; only chunks missing from a (truncated or partial) response are retried individually. This avoids re-asking sparse chunks one by one.
Batched graph writes and fulltext dedup prefilter are now the default — ENABLE_BATCHED_KG_WRITES=true stores entities/links/relationships via UNWIND batches (a handful of Neo4j round trips per document), and ENTITY_DEDUP_PREFILTER=true scores the top fulltext-index candidates instead of scanning every entity during Levenshtein dedup. Both preserve the per-item dedup semantics (locked by parity tests) and can be set to false per stack.
Relationship confidence is persisted — both relationship writers now store the model's confidence score alongside description and weight (previously only library imports carried it through), making it available for filtering and analysis.
Robust to embedding-model switches — entities carrying vectors from a previously configured embedding model are re-embedded when merged into and treated as missing by Step 2's automatic backfill, and the kNN candidate scan skips incompatible vectors instead of aborting. Changing EMBEDDING_MODEL/EMBEDDING_DIMENSION no longer quietly degrades semantic dedup or targeted relationship discovery.
Graph-wide deletes scale with library size — "delete all entities", "delete all relationships", and the rebuild's batch-relationship cleanup now run in bounded batches (CALL {} IN TRANSACTIONS), matching the system-reset behavior, instead of a single whole-graph transaction that can exceed Neo4j's transaction memory limit on large graphs.

Other changes

BATCH_PROCESSING_CONCURRENCY default raised from 2 to 3 — the bench-validated, Venice-safe setting from the June recalibration is now the shipped default. Deployments that pinned the variable explicitly are unaffected; unset deployments process one more document in parallel during batch ingestion.

July 3, 2026

Both knowledge-graph build steps got cheaper, faster defaults — substantially fewer LLM calls at equivalent extraction quality, with the previous behaviors preserved behind flags.

Step 1: relationship descriptions & batched extraction

Relationship descriptions are now reliably captured in Step 1 — the per-chunk extraction prompts now pin the exact output format, so models consistently include a short evidence-grounded description with every relationship (previously most per-chunk relations were stored without one). The parser also accepts the common <relation> tag variant. Descriptions appear in the Explore relationship browser and enrich RAG context.
Chunk-batched relationship extraction is now the default — ENABLE_BATCHED_CHUNK_RELATIONSHIPS (now default true, with RELATIONSHIP_CHUNKS_PER_CALL=4) packs several chunks into one extraction call: several times fewer Step 1 LLM calls at equivalent extraction quality, which speeds up graph generation considerably when your LLM provider enforces request-rate limits. Set it to false to restore the previous one-call-per-chunk behavior.

Step 2: targeted relationship discovery

Deep relationship analysis (Step 2) got a new default engine that scales much better on large graphs. Candidate entity pairs are now identified directly in Neo4j and the LLM focuses on verifying pairs with actual signal, which substantially reduces analysis time and token cost as libraries grow.

Candidates without the LLM — the new default engine (RELATIONSHIP_DISCOVERY_MODE=targeted) generates candidate entity pairs from the Neo4j entity_embedding vector index (kNN; entities missing embeddings are backfilled automatically first) and from document co-mention (unconnected pairs mentioned together in ≥2 documents, hub entities excluded). Pairs are scored, deduplicated, and capped per entity and in total.
LLM as verifier, not scanner — the relationship model only verifies/classifies the ranked pairs in small batched calls (~40 pairs per call) with a bounded chunk-context budget, instead of scanning every entity batch against a near-full context window.
Single pass — targeted mode does no multi-round discovery; RELATIONSHIP_TARGET_RATIO and RELATIONSHIP_MAX_ROUNDS now only apply to the legacy mode. RELATIONSHIP_MAX_HOURS, RELATIONSHIP_MAX_PER_ENTITY, PARALLEL_RELATIONSHIP_BATCHES, and the confidence ≥ 0.5 filter apply to both modes. Relationships are still stored with extraction_method='cross_collection'; Step 1 per-chunk relations and the Knowledge Graph UI are unchanged (the ERR indicator still displays in both modes).
New env vars (defaults): RELATIONSHIP_KNN_K=8, RELATIONSHIP_KNN_MIN_SIMILARITY=0.80, RELATIONSHIP_MIN_SHARED_DOCS=2, RELATIONSHIP_DOC_FREQ_CAP=30, RELATIONSHIP_MAX_CANDIDATE_PAIRS=15000, RELATIONSHIP_CANDIDATES_PER_ENTITY=10, RELATIONSHIP_PAIRS_PER_CALL=40, RELATIONSHIP_PAIR_CONTEXT_TOKENS=3000. The admin config endpoint now also exposes relationship_discovery_mode.
Legacy path preserved — set RELATIONSHIP_DISCOVERY_MODE=llm_scan to restore the previous two-phase full-batch scan with multi-round discovery.

July 2, 2026

Two operations that worked fine on small instances but failed on grown ones are now size-independent, and a structural latency pass over the agentic ask pipeline cut average chat time-to-first-token by ~65% on the reference deployment.

System reset & library import at scale

System reset no longer runs out of Neo4j memory — deleting the knowledge graph now happens in bounded batches (CALL {} IN TRANSACTIONS) instead of one whole-graph transaction, which on large libraries exceeded Neo4j's transaction memory limit (MemoryPoolOutOfMemoryError). Orphaned chunks are swept too.
Library import uploads are chunked — the web UI uploads export archives in 8MB sequential requests (/api/admin/import/upload/*) with live upload progress, then starts the import task. Single-request uploads of large archives were cut off mid-body by reverse proxies with body-read timeouts (Traefik v3 on Coolify defaults to 60s → HTTP 502). The single-request POST /api/admin/import remains for API use.
Long admin calls survive the frontend proxy — the Next.js rewrite proxy timeout was raised from its 30s default to 300s (it previously returned a bare HTTP 500 for a reset that took ~33s, even though the reset completed).

AskAI v2 — the pipeline

A chat turn now costs 2 LLM calls instead of 3 (plus one eliminated helper call per search), and long Deep Research questions that previously hit gateway limits now complete. Answer quality and citation behavior are unchanged — the writer stage is untouched. All changes below are default-on, each individually revertible via env flag:

Chat early-write (RESEARCHER_SPEED_EARLY_WRITE) — once a search iteration has produced sources (and no skill/git action ran), the answer is written immediately instead of spending one more full LLM round-trip asking the agent to confirm it is done.
Concurrent searches (RESEARCHER_PARALLEL_TOOL_CALLS) — read-only tool calls issued in one agent turn (knowledge / community / entity searches) now execute in parallel; skill and git actions stay sequential.
Entity hints (RESEARCHER_TOOL_ENTITY_HINTS) — the agent passes the entity names from its own search queries, eliminating the separate query entity-extraction LLM call on every search.
Search dedup (RESEARCHER_SEARCH_DEDUP) — an identical repeated search within one question is answered from cache with a nudge to try a different angle.
Faster turn completion (EMIT_DONE_BEFORE_MEMORY) — the SSE done event now fires as soon as the last answer token streams, before the post-answer memory compaction; the memory_update blob follows 1–4 s later and then the stream closes. The done frame carries pending_memory: true when an update is still coming. Integrator note: consume the stream to its actual end — a client that stops reading at done loses memory continuity. Set EMIT_DONE_BEFORE_MEMORY=false to restore the legacy order. See the Ask AI event reference.
Connection reuse for LLM calls, skill/git metadata caching, and event-loop hygiene fixes reduce per-request overhead further; SSE responses now send X-Accel-Buffering: no so customer-managed reverse proxies can't buffer the stream.

AskAI v2 — skills

The MAX_SKILL_INSTRUCTIONS_TOKENS budget is now actually enforced (with an explicit truncation marker) — previously skill instructions grew the system prompt without bound as more skills were enabled.
Deployments with no skills installed no longer receive skill instructions referencing a tool that doesn't exist (which occasionally provoked hallucinated tool calls on small models).
Truncated skill API responses are now labeled as truncated with pagination guidance, so the model paginates instead of confidently answering from cut data; the answer stage caps oversized API payloads.
Action requests ("create a ticket for that") can no longer be swallowed by the memory fast-path — they always run the full pipeline.

AskAI v2 — chat UI

Live citations — sources and [src_N] badges are clickable while the answer is still streaming, and survive pressing Stop.
Conversation memory wired end-to-end — the web UI now round-trips the conversation-memory blob, so memory-answerable follow-ups ("summarize that") skip retrieval entirely.
Conversations survive navigation — switching tabs or reloading no longer destroys the chat (per-tab persistence); Clear properly aborts an in-flight stream.
Smoother streaming — token rendering is batched per animation frame and completed messages no longer re-render on every incoming token, keeping long research sessions responsive.
Stopping mid-thought no longer leaks raw <think> text into the answer; the composer resets and refocuses after sending.

June 30, 2026

The bench-validated default primary for Q&A / research / chat is now Gemma4 26B A4B (google-gemma-4-26b-a4b-it on Venice) — a 26B/4B-active MoE that benches even faster than MiniMax-M3 at similar answer quality, making it an ideal fit for fast data retrieval from Cortex. MiniMax M2.7 / M3 were comparatively slower on this workload.

What changed

OPENAI_MODEL default changed from openai/minimax-m3 to google-gemma-4-26b-a4b-it.
OPENAI_MAX_CONTEXT recommended value is now 256000 (Gemma4 26B A4B's full 256K input window).
Recommended-stack blocks updated across .env.recommended, .env.example, README.md, the handbook, and the docs; the admin Primary Model panel reflects the new recommendation.

Existing deployments are unaffected — any explicit OPENAI_MODEL you set still takes precedence; this only changes the documented default and recommendation.

June 29, 2026

A round of reliability and UX hardening across the backend pipeline and the chat experience.

Reliability

Startup recovery of orphaned documents — on startup, any document left mid-processing (processing/extracting) by a prior shutdown or crash is reset to pending so it rejoins the queue instead of spinning forever (and /api/instance/status no longer reports safe_to_redeploy: false permanently after a redeploy). A WARNING log lists the reset ids.
DOCLING_CONVERSION_TIMEOUT (default 600 s) bounds a hung local Docling subprocess conversion — on timeout the worker is killed and the document is marked failed with a clear message instead of hanging on a large or corrupt file. Does not apply to the remote DOCLING_SERVICE_URL path.
SSE / chat errors no longer leak raw exception text — clients get a generic message plus a request id, with the full error logged server-side.
Streaming uploads are rejected with HTTP 413 once they exceed MAX_FILE_SIZE_MB, instead of being fully buffered first.
Clearer messages for partial / zero-match delete and move operations, and for task lookups after a restart.

Chat UX

Multi-line composer (Enter sends, Shift+Enter inserts a newline).
A Stop button to cancel generation mid-stream.
Graceful handling of interrupted streams — no endless typing cursor after a redeploy.
Real backend errors are now surfaced to the user.

General UX

Error + retry states where failures were previously silent (documents, search, collections, entity/community detail).
Escape-to-close and focus trap on modals.
Stale-response guards and poller cleanup.
Clipboard fallback on non-HTTPS origins.

June 26, 2026

Two admin-surface improvements: Langfuse traces now redact all authored content by default, and the System Configuration panel gained a curated view.

Langfuse content masking — privacy by default

Langfuse tracing now redacts all prompt, completion, tool, embedding, and vision text by default. A new LANGFUSE_LOG_EXTENDED flag (default false) wires a client-side mask hook so authored content never leaves the app — only structure reaches the Langfuse server: message roles, model + parameters, tool names + argument keys, parameter keys, token usage, cost, latency, and tags. This protects user content (we don't store what users prompt) and sharply cuts trace storage.

Set LANGFUSE_LOG_EXTENDED=true to log full content for local debugging (the previous behavior). Masking is deny-by-default and fail-closed — any unclassifiable text is redacted, never leaked. The Admin → System Configuration panel now has a Privacy section showing "Prompt & Content Redaction" and "LLM Tracing" status, so an operator (or a customer auditing a hosted instance) can verify at a glance that prompt content never leaves the box. No API or schema breakage — purely additive, and tracing stays fully env-gated (no keys = no tracing). See the Configuration guide.

Admin System Config — curated view

The Admin → System Configuration panel can now show a curated view. A new DISPLAY_FULL_SYSTEM_CONFIG flag (default false) hides advanced tuning knobs — output-token budgets, concurrency counts, chunking parameters, hybrid-search weights, graph hops, community sizes, and similarity thresholds — leaving models, API bases, context windows, dimensions, and feature toggles. Set it to true to reveal every knob.

This is display-only — settings, the /api/admin/config response shape, and ingestion/RAG behavior are unchanged. No API or schema breakage. See the Configuration guide.

June 23, 2026

Two additive operator features: optional LLM tracing and cost tracking via a self-hosted Langfuse instance, and a status endpoint deploy automation can poll before restarting an instance.

LLM observability — Langfuse tracing

Optional, env-driven LLM tracing and cost tracking via a self-hosted Langfuse instance. When LANGFUSE_BASE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY are set, every LLM, embedding, and vision call is traced (cost, tokens, latency, errors), and each agentic Q&A is grouped into one inspectable trace — across Venice, OpenRouter, and any OpenAI-compatible provider.

Activation is fully env-gated: with no keys the same image runs identically untraced (plain OpenAI client, zero overhead). Coverage spans the whole stack — the agentic researcher/writer loop, graph extraction, the Haystack embedders, and the vision analyzer. LANGFUSE_TRACING_ENABLED (master off-switch) and LANGFUSE_SAMPLE_RATE (0.0–1.0) tune it.

For accurate USD cost, add your Venice/OpenRouter models under the Langfuse project's Models settings (token counts are tracked regardless). No API or schema breakage — purely additive. See the Configuration guide.

Instance status endpoint — redeploy safety

A new admin endpoint, GET /api/instance/status, gives deploy automation a single call to decide whether an instance can be safely restarted or upgraded.

It returns safe_to_redeploy (with a reasons list of any active blockers) plus the supporting detail: documents currently processing/extracting, running background tasks, in-flight AskAI/research queries, and last-activity timestamps (last_query_at, last relationship analysis, community detection, entity merge).

safe_to_redeploy is false while any destructible work is in flight — documents mid-processing, background tasks (held in an in-memory store a restart would lose), or an active AskAI query (a restart kills the stream). Pending documents persist in Neo4j and resume after a restart, so they're reported but never block. AskAI activity is tracked across /api/ask, /api/ask/stream, and /api/ask/stream/thinking.

Requires manage permission. No API or schema breakage — the endpoint is additive. See the Deployment guide for the redeploy-safety polling pattern.

June 22, 2026

Two additions: Web Import harvests web pages into the knowledge base as clean markdown, and a systematic QA pass expanded the backend test suite and fixed four defects.

Web Import — MDHarvest powered by Crawl4ai

A new way to get web content into your knowledge base: Web Import harvests web pages into clean markdown and ingests them exactly like uploaded documents — chunked, embedded, and run through entity/relationship extraction, so they show up in Search, Ask AI, and the knowledge graph. It supersedes the standalone mdharvest tool.

Cortex never embeds a browser. It calls an open-source crawl4ai service over HTTP — you run crawl4ai once and a single instance can serve many Cortex deployments (in hosted setups, one per server, shared across tenants). There is no local crawler fallback, which keeps the per-instance footprint small.

Using it

On the Documents page, the Upload button now has a ▾ menu beside it with a Web Import option (it appears only when the feature is enabled). It opens a modal where you:

pick a collection,
paste URLs (one per line) — or use Discover links to crawl a page and pick same-site links from a checklist,
choose a content filter (Readable / Full page / Relevance-ranked),
and click Import from Web.

A progress bar tracks the crawl and processing; when it finishes you get a clear "imported N of M — close this window to see your documents" summary, and the document list refreshes automatically. Each imported page carries a provenance header recording its source URL and the extraction date.

Enabling it

Code
 
ENABLE_WEB_CRAWL=true
CRAWL_SERVICE_URL=http://crawl4ai:11235   # your own, or a shared crawl4ai
# CRAWL_SERVICE_TOKEN=...                  # only if crawl4ai requires a token

With the flag off or no service URL set, the feature stays hidden. See the Web Import guide for self-hosting crawl4ai and the full tuning reference (CRAWL_CONTENT_FILTER, CRAWL_CONCURRENCY, CRAWL_MAX_URLS_PER_JOB, …).

Privacy on shared crawl4ai instances: when several deployments share one crawler, no crawl history is retained and no deployment can see another's crawls — storage is ephemeral, only request-scoped endpoints are used (never the async job API), and every request bypasses the cache.

No API or schema breakage — the feature is additive and off by default.

QA pass — test coverage, fixes & docs hardening

A systematic quality pass: a canonical feature/defect inventory (qa/cortex_qa_master.ods — Features/Defects/Summary sheets), a large expansion of the backend test suite, a live end-to-end harness, and four fixes.

New config

EXPOSE_API_DOCS (default auto) — gates the interactive API docs (/docs, /redoc, /openapi.json). auto keeps them on in development and off in production, so a directly-reachable backend no longer discloses its full API schema to anonymous callers. Set true/false to force. (Closes D-004 below.)

Fixes

D-001 (Medium) — the git connector's extension check (GitConnectorService._supported) instantiated a full DocumentProcessor (building the embedder) just to read a class-level constant; it now reads the class attribute directly. This also restored hermetic test isolation on machines with a real .env.
D-003 (Low) — API-usage endpoint categorization used dict-order prefix matching, mislabeling /api/custom-inputs/{id} as upload instead of documents; now longest-prefix-wins (analytics data integrity).
D-004 (Low) — interactive API docs were served unauthenticated; now gated by EXPOSE_API_DOCS (see above).

Testing

Backend suite expanded from 322 to 458 hermetic tests (new modules for auth, API-usage, vision image-prep, context-curator memory, library-transfer NDJSON round-trip, researcher-agent helpers, skill-service parsing, RRF fusion, and FastAPI endpoint contracts).
New live end-to-end harness (backend/tests/test_live_e2e*.py) exercises real journeys (health, auth boundary, collections CRUD, hybrid search, streaming chat, full document ingestion→extraction) against a running stack; auto-skips when none is reachable. Authenticated runs read the key from CORTEX_E2E_API_KEY.

No API or schema breakage — EXPOSE_API_DOCS is additive; the only behavior change is that interactive docs default off under ENVIRONMENT=production (re-enable with EXPOSE_API_DOCS=true).

June 18, 2026

A major overhaul of the chat/answer path. Reasoning-capable models (especially on Venice) were returning empty or 30–90 s answers: the chat path never suppressed model "thinking", so hidden chain-of-thought streamed in a separate reasoning_content channel the answer stream ignores — adding seconds before the first token and, across the multi-iteration agent loop, frequently exhausting the budget into empty/timeout answers. This update makes chat answers fast, grounded, concise, and deterministic.

Snappy chat answers

Reasoning suppressed on the chat path by default. DEFAULT_REASONING_MODE now defaults to off and is actually applied: the speed-mode researcher loop, the answer writer, and the non-agentic + fast-search streaming writers are all routed through safe_chat_completion (per-provider reasoning suppression + a 400-fallback), replacing an ad-hoc single-provider hack. On Venice this sends disable_thinking, cutting model time-to-first-token to under 1 s (a direct probe measured 13.6 s → 0.57 s for gemma). Deep-research (quality) mode is unaffected — it stays AUTO and keeps full reasoning.
Fewer loop iterations. RESEARCHER_MAX_ITERATIONS_SPEED lowered 5 → 3 — with thinking off, the agent loop (not the model) was the dominant latency. Three rounds keep conversational answers snappy (one search + answer) while leaving room for a skill flow (activate → call → answer) to complete without truncation.
Rewritten speed-writer prompt for concise, direct, cited answers.
Result: empty-answer rate dropped to zero on previously-failing models; chat time-to-first-token fell from ~27–60 s to ~8–14 s.
Caveat (OpenAI): on GPT-5 / o-series, off maps to reasoning_effort none/minimal, which can disable parallel tool calls in the agent loop. Operators on those models should set DEFAULT_REASONING_MODE=auto.

Structured timeouts & a non-blocking agent loop on `/api/ask`

New ASK_DEADLINE_SECONDS (default 28) — the non-streaming POST /api/ask now wraps its work in a deadline and returns a clean 504 JSON {detail} on expiry, instead of the edge proxy (Coolify/Traefik) cutting the silent socket at ~30 s and emitting a bare plain-text 500. Keep it just below the proxy read timeout and raise both in lockstep; 0 disables. Does not apply to /api/ask/stream (SSE heartbeats keep that alive).
Agentic requires streaming — a non-streaming use_agentic request now fails fast with a 400 {error: "agentic_requires_streaming", use_endpoint: "/api/ask/stream"} rather than racing the proxy timeout (deep research routinely runs much longer than 30 s and only survives over SSE).
Cleaner error surfacing — 4xx/504 responses (and the existing collection-access 403) are no longer swallowed and re-wrapped as 500.
Event-loop unblock — the last two synchronous Neo4j tool calls in the researcher (community_search, entity_lookup) now run off the event loop, so they no longer stall concurrent requests during agentic runs.

Docs

New handbook chapter "The Cortex Standard" — state-of-the-art knowledge graphs, the open-systems/portable-memory ethos, and the fast/grounded/concise answer principles. Chapter 10 (Ask AI), Chapter 4 (Configuration), environment.md, configuration.mdx, and the README were synced for the new reasoning default.

Behavior change to note: the chat/answer path now suppresses model reasoning by default (DEFAULT_REASONING_MODE=off). Restore provider-default thinking on chat with DEFAULT_REASONING_MODE=auto (recommended on OpenAI GPT-5/o-series). No API or schema breakage; everything else is additive.

June 10, 2026

Opt-in flags that cut the LLM and database cost of graph building, leaner instance memory with an optional shared per-machine model service, and a broad production-hardening pass. All changes are additive or flag-gated — existing .env files and deployments keep working unchanged.

LLM-stack efficiency (flag-gated, default off)

A set of opt-in flags cuts the LLM and database cost of knowledge-graph building without changing any API or output semantics. Each flag ships default-off and is meant to be enabled after an A/B run against your bench baseline (bench/BASELINE.md):

ENABLE_BATCHED_KG_WRITES — entities, chunk links, and relationships are written in UNWIND batches: a handful of Neo4j round trips per document instead of one per item (hundreds→~10 on extraction-heavy docs), while preserving the exact dedup/merge semantics of the per-item path (locked by parity tests).
ENABLE_BATCHED_CHUNK_RELATIONSHIPS (+ RELATIONSHIP_CHUNKS_PER_CALL, default 4) — per-chunk relationship extraction packs several chunks into one LLM call (÷~4 calls), with a per-batch safety net that re-dispatches through the single-chunk path on parse failure.
ENABLE_PHASEB_CHECKPOINTING — cross-document relationship analysis persists per-batch progress: a crash or redeploy resumes instead of re-paying every batch, and multi-round runs reuse round 1's candidate scan (~50% fewer Phase 1 calls).
ENABLE_REPROCESS_DELTA — reprocessing a document whose file bytes and extraction config are unchanged is skipped entirely; git re-syncs of unchanged files cost ~zero.
ENTITY_DEDUP_PREFILTER — entity dedup scores the top-50 fulltext candidates instead of scanning every entity (big win on 10k+ entity graphs).
Prompt caching: the researcher system prompt is now byte-stable across loop iterations (RESEARCHER_STABLE_PROMPT, default on — provider prefix caches hit from iteration 2), and ENABLE_PROMPT_CACHE_CONTROL adds Anthropic cache_control breakpoints when routed via OpenRouter. Skill catalog/instructions are TTL-cached per process instead of being reloaded (Neo4j + filesystem + decryption) on every request.

Production hardening

Resilient helper transport: calls to the shared cortex-helper now retry with backoff behind a circuit breaker; HELPER_STRICT_REMOTE=true keeps docling out of tenant containers even on helper outages. The helper itself gained per-tenant fair queuing (CONVERT_PER_TENANT, CONVERT_QUEUE_TIMEOUT → 503 + Retry-After) and a shutdown drain.
Graceful shutdown: rolling restarts drain in-flight requests (uvicorn --timeout-graceful-shutdown + compose stop_grace_period); SSE streams receive a terminal event: shutdown frame so clients can reconnect instead of seeing a dead socket.
Observability: optional JSON logging with X-Request-ID correlation end-to-end (chat → backend → helper), and an admin-protected Prometheus GET /metrics endpoint (HTTP latency/route, SSE streams, document outcomes, conversion timing, helper breaker state). LOG_FORMAT=plain (default) keeps logs byte-identical.
Per-key rate limiting (RATE_LIMIT_QPM, default off) on ask/upload endpoints — a burst guardrail beneath the monthly quota, returning 429 + Retry-After.
Memory caps everywhere: Neo4j, frontend, and nginx now carry compose mem_limits (tunable via NEO4J_MEM_LIMIT etc.) so one stack's blowup can't OOM a neighbor; the Neo4j driver pool is configurable; nginx gained a dedicated unbuffered SSE location with a 1-hour read timeout.
Slim backend image: Dockerfile.prod --build-arg INSTALL_LOCAL_ML=false builds a torch-free image (~1.2 GB vs ~7 GB dev image) for stacks that fully offload to cortex-helper — the biggest per-tenant footprint lever yet. CI smoke-builds it on every PR.
Backups: an opt-in compose overlay (docker-compose.backup.yml) runs nightly online graph exports (APOC, works on Community) + file-volume archives with retention, plus a documented restore runbook (ops/backup/backup.sh).
Fixes: skill HTTP calls now honor SKILL_HTTP_TIMEOUT (was hardcoded 15s); vector-index health is verified at startup and silent semantic-dedup degradation is surfaced in /api/stats (vector_search_failures); inbound conversation-memory blobs are size-clamped.
CORS is now allowlist-driven via CORS_ALLOWED_ORIGINS (default * allows any origin but with credentials disabled, since auth is header-based). Set an explicit allowlist for production.
Fail-fast secrets: ENVIRONMENT=production refuses to boot on weak/default secrets — an empty/password123 NEO4J_PASSWORD, or a SESSION_SECRET under 32 characters when ADMIN_PASSWORD is set.
CI: a GitHub Actions workflow now runs backend tests + lint and frontend type-check + lint on every PR.

Leaner instances: lazy models + a shared per-machine model service

Backend startup memory dropped from ~2.2 GB to ~1 GB by deferring heavy model loads. The cross-encoder reranker no longer loads at startup (RERANKER_PRELOAD=false by default) — its ~7 s cold start is now hidden behind the query-analysis and search work that precedes reranking, and the model unloads after an idle period (RERANKER_IDLE_TTL_SECONDS, default 1800; 0 = never) to reclaim ~1 GB. Docling is no longer imported at startup (it ran in a subprocess anyway), saving ~244 MB per instance.

New cortex-helper service. For deployments running many isolated stacks on one host, the cross-encoder reranker and Docling converter can now be hosted once per physical machine and shared by all stacks via RERANKER_SERVICE_URL / DOCLING_SERVICE_URL / HELPER_SERVICE_TOKEN. Models stay warm (Docling conversions skip the per-document model reload — ~0.04 s vs ~4.5 s), and both clients fall back to the built-in local path automatically if the service is unset or unreachable. Single-instance and on-prem users are unaffected.

Upgrade notes

No database migration and no API changes — endpoints, request/response shapes, and the Neo4j schema are unchanged, so existing API-key consumers and cortex-chat are unaffected. Rebuild the backend image and redeploy; the defaults preserve current behavior closely. A few things to decide per deployment:

Reranker is now lazy + idle-unloading. Previously it was warmed at startup and stayed resident; now the first reranked query after a restart (or after an idle period) pays a ~7 s in-memory load (no re-download — the model is baked into the image). To restore always-warm behavior, set RERANKER_PRELOAD=true and RERANKER_IDLE_TTL_SECONDS=0. Recommended for latency-sensitive single-tenant deployments; leave the new defaults for multi-tenant density.
CORS default disables credentials with the wildcard. Harmless because auth is header-based (X-API-Key), and browsers reject wildcard-with-credentials anyway. If a browser client genuinely needs credentialed CORS, set an explicit CORS_ALLOWED_ORIGINS allowlist (which re-enables credentials).
ENVIRONMENT=production fail-fast is opt-in. Deployments that don't set it are unaffected. But once set, startup aborts if NEO4J_PASSWORD is empty/password123 or SESSION_SECRET is under 32 characters (with ADMIN_PASSWORD set) — fix those secrets before enabling the flag.
Adopting the shared cortex-helper service is optional and fails safe. Leave RERANKER_SERVICE_URL / DOCLING_SERVICE_URL unset to keep the built-in local path; when set, both clients fall back to local automatically if the service is unreachable. Ensure the stacks can reach the helper (a shared external Docker network for permanence), that HELPER_SERVICE_TOKEN matches the helper's HELPER_TOKEN, and that the helper's RERANKING_MODEL matches expectations.
Memory: with full offload, backend idle drops to ~1 GB and no longer spikes — BACKEND_MEM_LIMIT can be lowered. Without offload (or docling-only), a local rerank still spikes to ~2 GB, so keep mem_limit at ≥ ~2.5 GB; budget the helper's own ~0.6–4 GB once per host.

June 8, 2026

Snappier chat: live status, heartbeats, and a Deep Research toggle

The chat no longer feels stuck in the seconds before the first token. The streaming endpoint now emits structured status events at each pipeline stage (analyzing → searching → reranking → generating, in every mode including standard Chat) and SSE heartbeats during silent windows. The chat UI replaces the static "Thinking…" dots with a live indicator — a blinking accent dot, the current stage label, a running elapsed-seconds counter, and a reassurance line after 12 s. Gated by STREAM_REASONING_STEPS.

Deep Research is now an in-chat toggle. Chat is the default; an Erlenmeyer-flask button flips on Deep Research (multi-step depth) at any time — even mid-conversation — instead of being a separate destination.

Conversation Memory — a multi-bucket context curator for the agent

The research agent gains an opt-in, client-carried conversation memory that replaces blunt history truncation (the agent previously kept only the last 6 messages, silently forgetting older turns). The backend stays stateless: send an opaque conversation_memory blob on the request, read the updated one back from a new memory_update SSE event, and replay it next turn. Omit the field and nothing changes — it's fully backward-compatible.

A new context curator rebuilds a small, fixed context each turn from several memory buckets instead of re-feeding ever-growing history:

transcript — recent turns verbatim + a rolling summary of older ones (no more amnesia past a few turns).
facts / open_questions / intent — durable knowledge, unresolved threads, and the user's goal + preferred answer style/language, extracted automatically.
source_ledger — every cited source gets a conversation-stable sid (now on each sources event), so a citation in turn 2 keeps its identity in turn 5.
kg_context — a knowledge-graph snapshot for grounding follow-ups.

Faster, cheaper follow-ups. A memory fast-path lets questions answerable from memory alone ("summarize that", "why?", "in German") skip retrieval entirely — a single lightweight pass instead of the full research loop. Memory updates (summary folding + bucket extraction) run after the answer streams using a cheap fast model, so they add no user-visible latency. See Ask AI → Conversation Memory. Tunable via CONVERSATION_MEMORY_* and ENABLE_MEMORY_FAST_PATH.

Skill API failures are now surfaced instead of silently swallowed

When the research agent calls an external API through a skill's http_request tool and the call fails — an error status (401, 403, 422, 5xx) or a timeout — that failure is now visible to you rather than glossed over.

Previously a failed call (for example, a Zammad ticket creation returning 403 Forbidden) was only fed back to the model internally: nothing in the UI distinguished it from a success, and the assistant could narrate the action as if it had completed. Now:

A red skill step appears in the chat's research process showing the failing method, URL, and HTTP status (e.g. API call failed: POST https://.../tickets → HTTP 403).
The final answer explicitly reports the failure — it states the attempted action did not succeed and explains why, using the API's error message, and will not imply the action worked. This is driven by a dedicated "Failed Actions" instruction passed to the answer writer, so it holds even for small/fast models.

This is most important for write actions (creating tickets, opening PRs, posting data). Common causes of a failed call: a missing or expired token (configure it in the skill's Setup Wizard), insufficient permissions on the token's account, or a payload the target API rejects as invalid. See Agent Skills.

June 3, 2026

Git integration — connect GitHub, GitLab & Gitea repositories as a living knowledge source

Cortex can now connect directly to git repositories and keep their knowledge in sync. This is a major capability expansion: a connected repo becomes a first-class, bidirectional interface — Cortex ingests the repo's files and wiki into the knowledge graph (read), and the research agent can act on the repo by opening pull requests (read/write). Works with GitHub, GitLab, and Gitea (including self-hosted) behind a single provider abstraction. Enable with ENABLE_GIT_INTEGRATION=true.

Two surfaces, one connection. Each connection carries an access_level:

Read → ingestion. Cortex clones the repo and feeds matching files (and optionally the wiki) through the normal pipeline — chunking, embedding, entity/relationship extraction — so repo content is searchable and shows up in the graph alongside your other documents.
Read/write → the agent gains a git_repo tool. It can read live file contents and propose changes as pull requests. Writes always land on a fresh cortex/agent-* branch and open a PR/MR for human review — never a direct push to the default branch. Write actions are rejected server-side on read-only connections, so the toggle is a hard guarantee.

Incremental, versioning-aware sync. Re-syncing doesn't re-ingest everything. Cortex stores the last-synced commit and runs git diff --name-status -M to classify each change — Added (new document), Modified (re-extract in place), Deleted (flag the document for review, never auto-delete), Renamed (remap the path). If history was rewritten (force-push) or the filters changed, it self-heals by falling back to a full-tree reconcile that compares every file's blob SHA to what's stored. Documents carry git provenance (git_connection_id, git_path, git_blob_sha, git_commit_sha) so the sync key is the repo path, not a filename. After a sync that changed anything, the graph is flagged stale so you know to re-run relationship analysis and community detection.

Curated "documents only" default. New connections default to ingesting .pdf and .md files only — a single checkbox in the connect form, checked by default. Uncheck it to reveal custom include/exclude globs (gitignore-style). Code files (.py, .ts, .go, …) and markdown ingest through a new Docling-free fast path (running Docling on source code is wasteful); PDFs and Office documents route through Docling as usual. Images and audio are deliberately excluded from repo sync so a connection never OCRs every logo.

Manual + scheduled. Sync on demand with a "Sync now" button, or set a per-connection interval and a background poller keeps the repo current. No webhooks required (no public endpoint needed). GitHub wikis are cloned via repo.wiki.git; GitLab/Gitea wikis use their Wikis API.

Setup UX. A new Git Integration card on the Settings page walks you through it: pick a provider, paste a personal access token, and an inline, vendor-specific guide tells you exactly which token to create (always recommending the least-privilege option) with a direct link to the right settings page. "Test" verifies the token, then you pick the repo, access level, and (optionally) advanced filters. Existing connections are fully editable — access level, branch, schedule, globs, wiki, and token rotation.

Credentials. Single-tenant, personal-access-token per connection. The token is injected server-side into every git and API call — the agent/LLM never sees it — masked in all API responses, scrubbed from logs and errors, and never persisted in .git/config.

Bonus correctness fix (applies to all documents). While building the sync's re-extraction path we fixed a pre-existing bug: relationships carry a source_document_id but previously survived reprocess/delete unless an endpoint entity became fully orphaned, leaving stale RELATES_TO edges. A new delete_relationships_by_source_document now runs in delete_document_chunks and delete_document, so reprocessing or deleting any document (uploads and custom inputs included) cleans up its relationships.

New env vars: ENABLE_GIT_INTEGRATION, GIT_WORK_DIR, GIT_CLONE_DEPTH, GIT_MAX_REPO_SIZE_MB, GIT_SYNC_MAX_FILE_SIZE_MB, GIT_SYNC_POLL_INTERVAL, GIT_HTTP_TIMEOUT, GIT_HTTP_INSECURE_HOSTS. The backend image now bundles git; pathspec was added for glob matching. New endpoints under /api/integrations/git/* (admin-gated). New backend: services/git_providers/ (GitHub/GitLab/Gitea adapters) and services/git_connector_service.py; new frontend: components/admin/GitIntegrations.tsx; new docs: handbook/22-git-integration.md, documentation/pages/features/git-integration.mdx, and .claude/domain/git-integration.md.

Knowledge Graph wording: "relations" and "cross-document relations"

The Knowledge Graph page (and docs) now use simpler terminology: Step 1 counts are labeled relations (formerly "within-document relationships") and Step 2 counts are labeled cross-document relations (formerly "cross-document relationships"). All of these are relations — only the cross-document ones need the qualifier. No behavior change; per_chunk_relationship_count and other API fields are unchanged.

June 1, 2026

Library export AND import stream to/from disk instead of buffering — fixes OOM crash on large instances

Exporting a large instance (~1,300 documents / 22K entities / 49K relationships, all carrying embedding vectors) crashed the backend container with no logs and left the subsequent redeploy failing. Root cause was memory, not storage: the export built a full Python list of every serialized record and then did "\n".join(lines), holding the entire payload twice in RAM. Chunks and entities both carry embedding vectors, so the two heaviest sections alone could hit 1–2 GB transient — enough to trip the kernel OOM killer, which sent an uncatchable SIGKILL (hence no logs) and took Neo4j and the redeploy down with it.

Export now streams: a new _write_ndjson helper writes each NDJSON entry one JSON line at a time via zf.open(name, "w", force_zip64=True), and the embedding-heavy sections pull 500-row batches from Neo4j and stream them straight into the zip. New batched query methods in neo4j_service.py: export_entity_count/export_all_entities_batched (ORDER BY e.name) and export_relationship_count/export_all_entity_relationships_batched (ORDER BY elementId(r) for stable SKIP/LIMIT pagination).

Import got the mirror treatment so restoring a large archive doesn't OOM the target instance: _iter_ndjson reads each heavy entry through an io.TextIOWrapper one line at a time, _iter_ndjson_batches feeds the existing import_*_batch inserts (chunks, entities, chunk mentions), and relationships stream one-at-a-time. The pre-import limit guards (MAX_FILES, MAX_ENTITIES) now use _count_ndjson — which counts lines without parsing or buffering — instead of loading the whole entities-with-embeddings file into RAM just to call len(). Progress totals are read from manifest.stats.

Peak RAM on both sides is now ~one 500-record batch regardless of corpus size. The archive format, endpoints, and on-disk output are unchanged — existing exports/imports remain fully compatible. Touched backend/app/services/{library_transfer_service.py, neo4j_service.py} and .claude/domain/admin-features.md.

May 22, 2026

Image downscaling + JPEG recompression before vision API call

PDF pages are rendered at 2× DPI for OCR-grade text legibility (images_scale=2.0), so a typical page becomes a ~2400×1700 PIL image. Encoded as PNG with our original _pil_to_data_url, that's several MB of base64 — and some hosted vision deployments (LiteLLM-wrapped vLLM, custom OpenAI-compatible endpoints) tokenize the base64 data-URL payload as text instead of using tile-based image accounting. One customer instance hit 184K input tokens against a 192K context cap on a single image as a result.

Two new env knobs in backend/app/services/vision_analyzer._pil_to_data_url:

VISION_MAX_IMAGE_SIDE (default 1568) — caps the longer side via Image.thumbnail with Lanczos resampling, preserves aspect ratio, original PIL untouched. 1568 matches Claude's recommended max side: high enough for OCR text legibility, low enough to keep JPEG payloads under ~700 KB. Set 0 to disable.
VISION_JPEG_QUALITY (default 85) — JPEG quality for opaque content. Images with alpha (mode RGBA) still go out as PNG; everything else uses JPEG with optimize=True for 5–10× smaller payloads at visually-near-lossless quality.

Surfaced in the admin Settings page (LLM Configuration → Vision Model) and documented in .env.example, .claude/environment.md, and handbook/14-image-analysis.md.

Venice perf-tuning numbers recalibrated to verifiably-stable defaults

The previously documented Venice tuning block (BATCH_PROCESSING_CONCURRENCY=5, CONCURRENT_EXTRACTIONS=10, CONCURRENT_RELATIONS=5, VISION_MAX_CONCURRENT=5) was too aggressive — a 52-doc ingestion run logged 1012 × HTTP 429 "Too Many Requests" against Venice's embeddings endpoint (tripping their 20-failure / 30-second penalty), and 34 entity-dedup batches fell back to Levenshtein because the rate-limit cascade killed embedding calls used for semantic resolution. Recalibrated to:

Code
 
BATCH_PROCESSING_CONCURRENCY=3
CONCURRENT_EXTRACTIONS=4
CONCURRENT_RELATIONS=4
VISION_MAX_CONCURRENT=4

A subsequent 52-doc batch on the new values processed clean: 0 × 429s, 0 × Levenshtein fallbacks, 0 × event-loop block warnings, 52/52 succeeded. Updated in .env.recommended, .env.example, README.md, .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, handbook/{03-getting-started.md, 04-configuration.md} — eight locations total. The old values remain reasonable on bigger self-hosted vLLM endpoints where you control the rate-limit policy; they were just too hot for Venice's gateway.

Recommended stack standardizes on MiniMax-M3 + Qwen3-Embedding-8B native context

The bench-validated primary is MiniMax-M3 (192K window). The recommended block in .env.recommended, .env.example, README.md, .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, and handbook/{03-getting-started.md, 04-configuration.md} reads OPENAI_MODEL=minimax-m3 / OPENAI_MAX_CONTEXT=196608. Keep MiniMax confined to the primary Q&A/research tier — it inlines <think> tokens on ingestion paths despite disable_thinking, so extraction/vision stay on Qwen3.

`EMBEDDING_MAX_INPUT_TOKENS` env var + post-chunker token-cap sub-splitter

New env knob EMBEDDING_MAX_INPUT_TOKENS (default 8192) — matches the cap Venice and OpenAI enforce at the API gateway (8192 regardless of the underlying model's native context). Two-layer protection now sits between the chunker and the embed call:

Sub-splitter — _enforce_embed_token_cap in backend/app/services/document_processor.py walks the chunker output after URL restoration. Any chunk exceeding EMBEDDING_MAX_INPUT_TOKENS × ~2.8 chars/token gets recursively split (paragraph → line → sentence → space → hard char slice) into safe pieces with the parent's meta preserved. Zero content loss; new pieces slot into chunk_index via the existing enumerate loop. Catches Docling-extracted tables and custom-input pastes without sentence boundaries — the exact inputs that previously emitted single oversize chunks.
Truncation safety net — _truncate_for_embedding (~2.8 chars/token, deliberately conservative for markdown/code/CJK) runs immediately before each OpenAIDocumentEmbedder.run() call as belt-and-suspenders. With the sub-splitter in front, it should fire essentially never on normal docs.

Together they stop the HTTP 400 "Input text exceeds the maximum token limit of N tokens" (OpenAI-direct) and litellm.ContextWindowExceededError (proxied) failures that silently dropped embeddings. Self-hosted vLLM users running Qwen3-Embedding-8B can lift the cap to 32768 to use the model's full native context; managed-provider users should leave the default.

`VISION_MIN_IMAGE_SIDE` skips sub-threshold images before the vision call

Default 64. PDFs expose ~10–40 px bullets, icons, and dingbats as Docling PictureItems; Venice (and most hosted vision APIs) reject any image with a side under ~64 px as HTTP 400 "Supplied image did not pass validation checks". Direct API probing confirmed the cutoff: 31×31, 32×32, 48×48, 56×56 all fail; 64×64 passes. Cortex now short-circuits in vision_analyzer.analyze_image_with_vision_model when min(width, height) < VISION_MIN_IMAGE_SIDE — saves API spend, eliminates the 3-retry cascade per tiny image, and removes the noisy fallback storage entries from the graph. Set 0 to disable. Format (PNG vs JPEG) was not the cause — both behave identically.

Stats endpoint no longer multiplies file_size by chunk count

GET /api/stats was reporting total_size ~70× larger than reality (23 GB shown for a 320 MB corpus). Root cause in neo4j_service.py:get_stats: the unscoped query did MATCH (d:Document) OPTIONAL MATCH (d)-[:HAS_CHUNK]->(c:Chunk) WITH … sum(d.file_size), fanning out one row per chunk so each document's file_size was summed once per chunk per doc. Fixed by hoisting the total_size aggregation before the chunk join. The scoped (per-collection) variant got a matching WITH DISTINCT d defensive fix in case docs appear in multiple collections.

Image entities now use the same embedding-based dedup as text entities

Previously, entities extracted from image descriptions always went through fuzzy-only Levenshtein 85% deduplication, even with ENABLE_SEMANTIC_ENTITY_RESOLUTION=true enabled. Text entities benefited from embedding-first dedup (catching "MOCA" ↔ "Museum of Crypto Art"); image entities did not. This left a cross-source dedup gap — an "MOCA" entity extracted from a chart caption couldn't merge with an existing "Museum of Crypto Art" text entity.

backend/app/services/neo4j_service.py:store_graph_extraction() gains an optional entity_embeddings parameter. When provided and the semantic resolution flag is on, each entity routes through store_entity_with_embedding() (embedding-first vector match, Levenshtein fallback for typos). Otherwise it falls back to the existing fuzzy-only path — no behavior change for callers that don't pass embeddings.
backend/app/services/document_processor.py:process_single_image() now batch-embeds each image's extracted entities via generate_entity_embeddings_batch_async() before storing, mirroring what the per-document text path has been doing. One embeddings call per image-with-entities (typically <10 names, very cheap). On embedding failure, the image still stores via Levenshtein with a logged warning — never aborts the document.
Image-derived entities now populate the same entity_embedding Neo4j vector index that text entities populate. Cross-source duplicates collapse at write time; the /deduplicate page's manual workflow becomes more effective because semantic matches happen during ingestion instead of being deferred. Verified end-to-end: a test document with 32 images stored 190 image entities, 100% carrying 4096-dim embeddings, zero fallback warnings.
Docs updated: .claude/domain/entities.md, .claude/domain/document-pipeline.md, handbook/13-deduplication.md, handbook/14-image-analysis.md, documentation/pages/features/knowledge-graph.mdx.

One-click Generate Graph from /documents

Clicking Generate Graph on the Documents page previously navigated to /extract and then required a second click to actually start the pipeline. The button now appends ?autostart=1; the Knowledge Graph page detects the param on mount, waits for its initial data fetch, fires handleRegenerateGraph() exactly once, and router.replaces the URL clean so a refresh won't re-fire.

frontend/src/components/DocumentList.tsx — router.push("/extract?autostart=1") instead of plain /extract.
frontend/src/app/extract/page.tsx — new useEffect reads useSearchParams().get("autostart"), guards via a hasAutoStarted ref, fires once when loading === false && documents.length > 0 && !isRegenerating.
Trigger code stays in one place on the destination page; the documents page just emits the intent via query param. The destructive-action confirm dialog inside handleRegenerateGraph is preserved — auto-start does not bypass it when entities already exist.

May 21, 2026

Reasoning Suppression Control (Provider-Agnostic)

Modern reasoning models (GPT-5/5.1, Claude 4.x, Qwen3, DeepSeek-R1, GLM, Kimi, MiniMax M3) over-think on structured-extraction tasks, inline <think> tokens that corrupt XML output, and burn token budget on chain-of-thought that the parser never sees. A new provider-agnostic switch forces reasoning OFF (or any other level) on every LLM call inside the knowledge-graph pipeline while leaving the researcher/writer Q&A path on AUTO.

New backend/app/services/reasoning_config.py — ReasoningMode enum, regex-based model-family classifier, and a per-backend dispatch table. Each backend gets the correct kwargs:
- OpenAI → reasoning_effort (none/minimal/low/…)
- OpenRouter → extra_body.reasoning.effort
- Venice → extra_body.venice_parameters.disable_thinking
- Anthropic → extra_body.thinking={"type":"disabled"} (omitted on Opus 4.7+ adaptive thinking)
- vLLM → extra_body.chat_template_kwargs.enable_thinking=false
safe_chat_completion / safe_chat_completion_sync wrappers handle a runtime fallback: if a model 400s on the reasoning param, the wrapper retries without it and caches the (base_url, model) pair so future calls skip the param upfront — one wasted call per misclassified model, then nothing.
All 14 LLM call sites in graph_extractor.py route through the wrappers — entity extraction, document summaries, community naming, entity enrichment, candidate scan, gleaning pass, per-chunk + Phase-2 relationship extraction.
Vision-model image analysis uses the same switch via a flatten_reasoning_body() helper that converts OpenAI-SDK-style extra_body into a raw-HTTP body dict (vision uses httpx, not the OpenAI client). One-shot 400-fallback works the same way.
New env vars: EXTRACTION_REASONING_MODE (default off), RELATIONSHIP_REASONING_MODE (default off), VISION_REASONING_MODE (default off), DEFAULT_REASONING_MODE (default auto — Q&A stays on provider default). Per-model escape hatch via REASONING_MODEL_OVERRIDES=model1:mode1,model2:mode2.
Defaults are no-ops for pure-instruct models (Mistral, Llama, GPT-OSS) — zero behavior change for users not on reasoning models.

Token & Context Budget Inheritance Chain

The hardcoded max_output_tokens=2000 default in per-chunk relationship extraction was collapsing Qwen3-family output (Run 03 stored 13 relationships across 77 candidate pairs because the model truncated mid-XML). Budgets are now configurable across the whole stack via a fallback chain so a 3-model deployment can be configured with just two env vars.

Inheritance chain (output tokens):

Code
 
OPENAI_MAX_OUTPUT_TOKENS=8000      (new primary default — covers Qwen3 verbose XML)
     ↓ (when EXTRACTION_*=0)
EXTRACTION_MAX_OUTPUT_TOKENS
     ↓ (when RELATIONSHIP_*=0)
RELATIONSHIP_MAX_OUTPUT_TOKENS      ← per-chunk + candidate scan
     ↓ (when VISION_*=0)
VISION_MAX_OUTPUT_TOKENS

RELATIONSHIP_BATCH_MAX_OUTPUT_TOKENS=16000   (standalone — Phase 2 batch only, NOT in chain)

Inheritance chain (input context):

Code
 
OPENAI_MAX_CONTEXT=32768
     ↓ (when EXTRACTION_*=0)
GRAPH_EXTRACTION_MAX_CONTEXT
     ↓ (when RELATIONSHIP_*=0)
RELATIONSHIP_MAX_CONTEXT

0 is the inherit sentinel for ints (consistent with MAX_FILES=0 = unlimited convention).
EXTRACTION_MAX_CONTEXT → renamed to GRAPH_EXTRACTION_MAX_CONTEXT for prefix consistency with GRAPH_EXTRACTION_MODEL / GRAPH_EXTRACTION_API_BASE. Legacy name honored as a deprecated alias for one release; a one-shot startup WARN from app.config._warn_deprecated_env_aliases nudges migration.
RELATIONSHIP_MAX_OUTPUT_TOKENS semantic split: the env var now drives per-chunk + candidate-scan in the chain; the Phase 2 batch budget moved to the new standalone RELATIONSHIP_BATCH_MAX_OUTPUT_TOKENS=16000. Users who explicitly set RELATIONSHIP_MAX_OUTPUT_TOKENS=16000 for Phase 2 should rename to the new var; the legacy value still works (per-chunk gets harmless 16000-token headroom).
Entity-extraction call sites at graph_extractor.py:821, 1014, 1840 now read settings.extraction_max_output_tokens instead of hardcoded 3000/8000. Per-chunk relationship caller in document_processor.py:1461 passes settings.relationship_max_output_tokens. Phase 2 batch caller switched to settings.relationship_batch_max_output_tokens. Vision call (vision_analyzer.py:304) reads settings.vision_max_output_tokens.
Utility-tier hardcoded budgets stay hardcoded (50/300/500/1000 for community names, entity descriptions, query-time extraction) — those are task-tuned and would harm functionality if scaled with the primary.
17-test backend/tests/test_budget_fallback.py covers all chain combinations, mid-chain overrides, Phase 2 isolation, legacy alias loading, and the deprecation WARN trigger.

Neo4j 5.26 Upgrade (4096-dim Vector Indexes)

Neo4j 5.15-community caps vector-index dimensions at 2048, blocking native use of Qwen3-Embedding-8B (4096) and other modern embedding models. Bumped to Neo4j 5.26 LTS across all compose files.

image: neo4j:5.26-community in docker-compose.yml, coolify/docker-compose.coolify.yml, dokploy/docker-compose.dokploy.yml; neo4j:5.26-enterprise in docker-compose.prod.yml.
In-place data file migration (within Neo4j 5.x the named volume neo4j_data upgrades automatically on first boot).
Python neo4j driver pin (>=5.17.0) is already forward-compatible with the new server — no driver changes needed.
APOC plugin auto-installs the matching APOC version via NEO4J_PLUGINS=["apoc"] — no separate APOC version bump required.
Existing deployments at 1536/2048-dim embeddings keep their corpora intact: Cortex preserves the vector index when EMBEDDING_DIMENSION matches the stored one. Only bumping the dim triggers an auto drop-and-recreate of the index (requires re-embedding).
New deployments can set EMBEDDING_DIMENSION=4096 directly for native Qwen3-Embedding-8B fidelity.

Recommended Minimal Stack + `.env.recommended`

The model + budget inheritance chains compress a 3-model 12-env-var stack down to a 2-model 6-env-var stack. New .env.recommended template at the project root captures the bench-validated Venice setup:

Code
 
OPENAI_API_KEY=...
OPENAI_API_BASE=https://api.venice.ai/api/v1
OPENAI_MODEL=minimax-m3
OPENAI_MAX_CONTEXT=196608

GRAPH_EXTRACTION_MODEL=qwen3-6-27b
GRAPH_EXTRACTION_MAX_CONTEXT=256000

VISION_MODEL=qwen3-6-27b

EMBEDDING_MODEL=text-embedding-qwen3-8b
EMBEDDING_DIMENSION=4096

BATCH_PROCESSING_CONCURRENCY=5
CONCURRENT_EXTRACTIONS=10
CONCURRENT_RELATIONS=5
VISION_MAX_CONCURRENT=5

The same canonical block now appears in .env.example (top-of-file Quick Start), README.md (Quick Setup), .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, and handbook/{03-getting-started.md, 04-configuration.md} — eight locations total.
Companion performance tuning (Venice-validated) block documents the concurrency knobs and warns that CONCURRENT_EXTRACTIONS is the biggest multiplier to dial down on smaller providers.
Relationship + (optionally) Vision sub-tier models inherit GRAPH_EXTRACTION_MODEL automatically; embedding API base/key inherit from OPENAI_* so no separate config is needed when Venice serves both.
Vision exception: VISION_MODEL does NOT inherit — set it explicitly or Cortex falls back to Docling's built-in SmolVLM (no API call).

Autonomous LLM-Stack Benchmark Harness

New bench/ directory — a self-contained orchestrator that cycles a fixed dataset through arbitrary LLM-model combinations against the Cortex ingestion pipeline, measuring extraction yield, relationship density, and runtime per combo. Includes a model registry, automated .env backup/restore safety, and per-run heuristics. Not yet publicly documented; CLI is internal-only for now.

Admin Settings Page Enhancements

Settings page (/admin) didn't expose the new budget knobs. The LLM Configuration section now shows:

Primary Model: added Context Window (OPENAI_MAX_CONTEXT) and Output Tokens (OPENAI_MAX_OUTPUT_TOKENS) rows
Extraction Model: added Output Tokens (EXTRACTION_MAX_OUTPUT_TOKENS)
Relationship Model: split into Output Tokens (per-chunk) and Output Tokens (batch) — reflecting the inheritance-chain + standalone split
Vision Model: added Output Tokens (VISION_MAX_OUTPUT_TOKENS)

Each new field includes a tooltip explaining its inheritance path and where it sits in the budget chain. Backend SystemConfigResponse model + /api/admin/config endpoint extended with the corresponding fields.

Instance Limits

New MAX_ENTITIES env var (default 0 = unlimited) caps the total entity count across the graph. Enforced at upload-time and custom-input creation: new ingestion is rejected once get_stats()["entity_count"] is at or above the cap. A single in-flight document can push the post-extraction count slightly above the cap (accepted tradeoff).
MAX_QUERIES_PER_MONTH instance-wide cap on chat-style queries — applies to POST /api/search, POST /api/ask, POST /api/ask/stream, POST /api/ask/stream/thinking. Returns 429 Too Many Requests with a Retry-After header (seconds until next UTC month) when exceeded.
Comprehensive test coverage in backend/tests/ for MAX_FILES, MAX_ENTITIES, MAX_COLLECTIONS enforcement across upload, custom input, collection creation, and library import endpoints.

Skill `http_request` Tool Improvements

Config variable substitution in request body: _substitute_variables() previously ran on the URL only, so skills that referenced configured values in a JSON body (e.g. "group": "ZAMMAD_GROUP_NAME") shipped the literal placeholder string instead of the substituted value. Now applied to URL + body.
Default Content-Type injection: the tool previously only injected Authorization headers from skill config schemas. Skills POST/PUT-ing JSON bodies without explicitly setting Content-Type were getting text/plain and being rejected by strict APIs. Default is now application/json when a body is present, overridable per request.
Improved researcher-agent logging: tool call inputs/outputs are now logged at INFO level with payload truncation for easier debugging of skill-driven research flows.

Design System & UI Kit

MOCA Library UI Kit: complete UI kit for the MOCA Library web app, including the ManageScreen for document management and the Shell component for top-level navigation chrome.
Design system preview files + tokens: HTML previews for drop zone, glass surface, icons, input, logo, motion, nav pill, radii, shadows, spacing scale, stats card, and other primitives. Tokens published as the canonical reference for in-product surfaces and partner integrations.

Minor Fixes

Restart-vs-recreate gotcha documented: docker compose restart does NOT re-read env_file: (env vars are baked at container create time). Added a callout to handbook/03-getting-started.md and a dedicated section in handbook/20-troubleshooting.md pointing users to docker compose up -d --force-recreate backend after any .env change.
Build resilience: logo download via curl no longer fails the frontend build when the Directus URL is unreachable — logs a warning and continues with the default logo (81d99af).
Code structure cleanup across .claude/architecture.md, .claude/design-system.md, and several services (172c1db).

April 8, 2026

Collection-Based Auth System

A comprehensive security audit remediation that closes every gap in the collection-scoped API key system. Previously, restricted keys could silently bypass collection boundaries on graph, stats, community, and task endpoints. All endpoints now enforce collection scope consistently.

Foundation fixes (neo4j_service.py)

Fixed get_document(), get_document_content(), and get_documents_file_paths() to include the Collection join — they were returning collection_id: null, causing every downstream collection access check to silently pass
Added allowed_collection_ids parameter to all graph query methods, implementing the 4-hop scoping pattern (Collection→Document→Chunk→Entity) throughout: get_graph_visualization_data(), list_entities_paginated(), get_entity_relationships(), get_graph_subgraph(), get_entity_types(), get_relationship_types(), find_entities_by_name(), list_relationships_paginated(), suggest_duplicate_entities()
Added allowed_collection_ids to community methods using a 5-hop pattern (Community→Entity→Chunk→Document→Collection): list_communities_paginated(), get_community(), search_communities_by_content()
Updated get_stats() to accept allowed_collection_ids — restricted keys now see document/entity/community counts scoped to their allowed collections only

Full endpoint authentication coverage (main.py)

All endpoints now require an X-API-Key header. Previously unprotected endpoints now enforce appropriate permission levels:

Read — GET /api/tasks, GET /api/tasks/{id}, GET /api/tasks/{id}/result
Manage — POST /api/documents/{id}/reprocess (+ per-document collection validation), POST /api/documents/reprocess (+ per-document collection check in loop), POST /api/documents/process-pending, POST /api/cleanup/orphaned-entities, PATCH /api/graph/entity/{name}, POST /api/entities/merge, POST /api/graph/relationships/analyze (+ collection_id validated if supplied), DELETE /api/graph/relationships, DELETE /api/graph/entities, POST /api/graph/communities/detect (+ collection_id validated if supplied), DELETE /api/graph/communities/{id}, DELETE /api/graph/communities, POST /api/graph/communities/summarize, DELETE /api/tasks/{id}, POST /api/tasks/cleanup, POST /api/custom-input/generate-topic Collection scoping propagated to all read endpoints

Restricted keys now receive automatically filtered results across every read endpoint — no endpoint leaks cross-collection data:

/api/stats and /api/graph/status — counts scoped to allowed collections
/api/graph/entities, /api/graph/entity-types, /api/graph/visualization, /api/graph/entity/{name}, /api/graph/entity/{name}/relationships, /api/graph/search, /api/graph/subgraph — all scoped via 4-hop pattern
/api/graph/relationships, /api/graph/relationship-types — scoped via source/target entity 4-hop filter
/api/graph/communities, /api/graph/communities/{id}, /api/graph/communities/search — scoped via 5-hop pattern
/api/entities/duplicates — entity candidates scoped to allowed collections
Researcher agent (researcher_agent.py) and query processor (document_processor.py) — allowed_collection_ids propagated through run_research_pipeline(), graph_search_async(), _execute_knowledge_search(), hybrid_search_with_graph(), and semantic_search() so Ask AI and Deep Research respect collection boundaries end-to-end

Bug fixes

Removed dead get_stats() method (neo4j_service.py) that was shadowed by the new scoped version — the dead method returned a stripped-down dict missing community_count, collection_count, pending_count and other fields, making it a silent trap for future callers
Fixed reprocess_documents and reprocess_document endpoints incorrectly using collection_id or "default" when checking collection access for documents without a collection assignment — can_access_collection(None) already returns True for restricted keys (global queries are allowed), so the fallback was incorrectly blocking access to uncollected documents

April 1, 2026

Agent Skills: Setup Wizard, http_request Tool, and Reliable Skill Execution

Setup Wizard: After installing a skill, the primary LLM analyzes the SKILL.md to extract required configuration variables (API tokens, URLs, etc.). A dynamic modal prompts the user for values — stored persistently in config.json in the skill directory. No more .env editing or container restarts to configure skills.
http_request built-in tool: The researcher agent can now call external APIs described in skill instructions. The LLM provides only method and url — authentication headers are injected server-side from the config schema's auth_header template. The LLM never touches tokens or API keys.
Auto-activation: Enabled skills are now automatically loaded at the start of every research/chat session. The full SKILL.md body (with auth-related lines stripped) is injected into the system prompt. Replaces the previous on-demand activate_skill/list_skills pattern.
Server-side auth injection: Auth headers are built from the config schema's auth_header field (e.g. Authorization: Bearer API_TOKEN, X-API-Key: API_KEY) — works for any auth pattern, any skill.
Config API endpoints: POST /api/admin/skills/{id}/analyze (LLM analysis), GET /api/admin/skills/{id}/config (schema + masked values), PUT /api/admin/skills/{id}/config (save values with mask preservation)
Config status: Skills show "Needs setup" badge when required config is missing. "Configure" button in expanded skill details on Settings page.
Skills directory persistence: Added skills_data Docker named volume to prod and Coolify compose files. Fixed path resolution bug (4→3 parent levels) that was writing skills to ephemeral container storage.
Smart API response truncation: JSON responses are intelligently slimmed (truncate long strings, flatten nested objects) to keep all entries within context budget instead of cutting mid-object.
Source dedup fix: _deduplicate_sources() no longer drops skill API responses (sources without chunk_id). Skill API data is now passed to the writer as a priority source.
Chat mode improvements: Agentic chat (ENABLE_AGENT_CHAT) now defaults to true. Speed mode gains the reasoning tool when skills are active. Max iterations bumped from 2 to 5. Retry with tool_choice=required when the model skips tools on first iteration.
Frontend: New SkillConfigModal component, "View SKILL.md" button in skill details, removed "Powered by Neo4j + Haystack" footer, chat/research views use full viewport height.
tools.json deprecated: Not part of the agentskills.io standard. The standard approach is instruction skills with API documentation + the built-in http_request tool.

Document Source Tracking

Added source field to documents — tracks the origin of each document (e.g. upload, custom_input, or a custom identifier like youtube-transcriber)
Upload endpoint (POST /api/upload) accepts optional source query parameter (defaults to upload)
Custom input endpoint (POST /api/custom-input) accepts optional source field in body (defaults to custom_input)
Document list and detail endpoints return the source field
Frontend shows source label on document cards when source is not the default upload
Source filter dropdown appears automatically in the Documents page toolbar when documents have multiple distinct sources
Existing documents are automatically backfilled on startup: upload for regular uploads, custom_input for custom inputs
Library export/import preserves the source field

March 30, 2026

Agent Skills in Library Transfer

Agent Skills are now included in library export/import: Skill nodes, SKILL.md files, and tools.json are bundled in the ZIP archive with directory_path remapping on import
Full system reset now cleans up skill nodes and skill directories on disk
Fixed vector index dimension mismatch: Neo4j vector indexes are now auto-detected and recreated when embedding dimension config changes (fixes broken search after importing a library with different dimensions)
Step 2 progress now shows only newly discovered cross-document relationships instead of total relationship count

March 28, 2026

Image Extraction Integration Improvements

Image-extracted relationships are now tagged as extraction_method="per_chunk" so they count toward within-document relationships in the Step 1 display
Image entities now use store_entity_with_resolution() for the same fuzzy dedup as text entities, instead of basic store_entity()
Page number and caption are now stored in image chunk metadata for document position tracking
Image-extracted entities are cross-linked to text chunks via fuzzy MENTIONS matching
Entity provenance (source_document_id) is now passed through store_graph_extraction() for image entities

Graph Generation Pipeline Resilience

Added retry with exponential backoff (up to 5 attempts) on all poll functions, preventing the entire 3-step pipeline from aborting on the first transient network error
Added visibilitychange listener to immediately resume polling when a browser tab becomes active (browsers throttle setTimeout in background tabs)
Active poll state is tracked via ref for reliable resume across tab switches

March 27, 2026

Agent Skills System

Integrated the open AgentSkills standard into Deep Research and Chat flows
Skills are SKILL.md files with YAML frontmatter (name, description, license, metadata) supporting two types: instruction skills (modify researcher behavior) and tool-providing skills (include tools.json with HTTP/script execution)
On-demand activation: researcher agent receives a compact skill catalog and activate_skill/list_skills tools — full instructions and tools are loaded only when the agent decides they're relevant to the query
Skills discoverable from .agents/skills/ directory on startup, installable from direct URLs or the skills.sh registry
Added SkillsManager component on Settings page for installing, enabling, disabling, and deleting skills with registry search
Skill tool calls and activations rendered in chat with Puzzle icon
New environment variables: ENABLE_SKILLS (default: true), SKILLS_DIR, ENABLE_SKILL_SCRIPTS (default: false), SKILL_SCRIPT_TIMEOUT, SKILL_HTTP_TIMEOUT, MAX_SKILL_TOOLS, MAX_SKILL_INSTRUCTIONS_TOKENS
Admin-only CRUD endpoints: GET/POST/PATCH/DELETE /api/admin/skills/*

Library Import/Export

Added full instance migration via Settings page → Data Management section
Export (POST /api/admin/export) runs as a background task building a ZIP64 archive containing 12 NDJSON data files (documents, chunks with embeddings, entities, relationships, communities, collections, merge history, system meta, and more) plus original document files
Import (POST /api/admin/import) accepts multipart ZIP upload with clean mode (requires empty instance) or replace mode (auto-wipes first)
Validates manifest, checks embedding model/dimension compatibility, remaps file paths, restores all nodes and edges including dynamic APOC relationship types
Concurrency guard prevents simultaneous export/import operations (409 conflict)
Frontend shows Export card (stats summary + progress bar + download) and Import card (mode selector + drag-and-drop upload + progress bar + result summary with warnings)

Three-Tier LLM Architecture

Added dedicated relationship model (RELATIONSHIP_EXTRACTION_MODEL, RELATIONSHIP_EXTRACTION_API_BASE, RELATIONSHIP_EXTRACTION_API_KEY) with its own API endpoint and rate limit for all relationship work (Step 1 per-chunk + Step 2 batch analysis)
Three-tier model separation: Primary (reasoning, for Q&A/research), Extraction (instruction-following, for entity extraction and community summarization), Relationship (instruction-following, for all relationship discovery)
Added get_relationship_llm_config() with fallback chain: relationship model → extraction model → primary model
Added CONCURRENT_RELATIONS (default: 3) for per-chunk relationship extraction concurrency, separate from entity extraction
Changed PARALLEL_RELATIONSHIP_BATCHES default from 0 to 5
Community summarization moved from primary model to extraction model for more reliable structured output
Added Relationship Model section to Settings page LLM Configuration between Extraction and Vision

Per-Chunk Relationship Extraction Fixes

Added tenacity retry with exponential backoff (4 attempts, 2–30s wait) for 429 rate-limit errors during per-chunk extraction
Original-to-canonical entity name mapping is now tracked during dedup, and relationship source/target are remapped before storing — fixes silent storage failures when entity names were merged during fuzzy resolution
Self-referential relationships (source == target) are now filtered at both extraction and storage
Fixed stats query to count all Entity→Entity edges, not just RELATED_TO type
Fixed per_chunk_relationship_count missing from stats API response
Step 2 rebuild now only deletes batch-analysis relationships via delete_batch_relationships(), preserving per-chunk relationships from Step 1

Knowledge Graph Page UX Improvements

Step 1 renamed to "Entity Extraction & Relationship Discovery" — now shows entity count and within-document relationship count
Step 2 renamed to "Deep Relationship Analysis" — now shows only cross-document relationship count
"Re-analyze" button renamed to "Find more"; rebuild warning clarifies that per-chunk relationships are preserved
ERR (Entity-Relationship Ratio) indicator now shows 2 decimal places for better incremental visibility
Fresh instance warning on Step 1 (0 entities) recommends "Generate Graph" instead of "Extract Entities"

March 26, 2026

Relationship Analysis Pipeline Overhaul

Two-phase per-batch analysis: Phase 1 scans entity batches for candidate pairs, Phase 2 confirms and classifies with structured XML output including confidence scores (0.0–1.0) — relationships with confidence < 0.5 are filtered before storage
Co-occurrence batching via Union-Find clustering: entities sharing chunks are grouped into the same batch for richer context; replaces O(n²) greedy BFS sort that was freezing on 32k entities; new algorithm is O(n·chunks), handles 100k+ entities
Dynamic chunk context filling: token budget split 60/40 between entities and source text; chunk context fetcher accepts a token budget and fills up to it, replacing the old fixed 10 chunks × 500 chars (~1,250 tokens out of ~44k available)
Multi-round discovery: initial analysis runs up to RELATIONSHIP_MAX_ROUNDS (default: 3) rounds, stopping early if target ERR reached or RELATIONSHIP_MAX_HOURS exhausted; "Find more" always does 1 round
Removed 5,000 entity fetch cap from get_all_entities_for_collection() — was silently ignoring 27k of 32k entities
Removed 500 existing-relationship fetch cap between rounds
Added Entity-Relationship Ratio (ERR) metric shown on Knowledge Graph page with color-coded indicator (green ≥ 0.69, yellow ≥ 0.29, red < 0.29) and tooltip
New config variables: RELATIONSHIP_TARGET_RATIO (default: 1.0), RELATIONSHIP_MAX_ROUNDS (default: 3), RELATIONSHIP_MAX_HOURS (default: 0), RELATIONSHIP_MAX_OUTPUT_TOKENS bumped from 8k to 16k

Relationship Quality and Anti-Hub Protections

Removed MENTIONS from allowed relationship types (now 14 types) — was being used as a lazy catch-all for co-occurrence
Added per-chunk relationship extraction during entity processing: chunks with 2+ entities get an LLM call using chunk text as direct evidence, stored with extraction_method='per_chunk'
Added few-shot good/bad examples to Phase 1 candidate scan and Phase 2 extraction prompts
Added anti-hub negative instructions: "If no clear relationship exists, do not create one. Co-occurrence is not a relationship."
Added RELATIONSHIP_MAX_PER_ENTITY config (default: 50) — soft cap that skips storage when both endpoints are saturated
Capped existing relationships shown to LLM at 20 per entity between rounds to prevent hub reinforcement
Reduced batch overlap from 15% to 5% with degree-aware selection (excludes entities already in 2+ batches, prefers low-connection entities)
Relationship type fuzzy-matching via rapidfuzz (80% threshold, fallback to RELATED_TO) with plaintext arrow format fallback parser

Entity Deduplication Improvements

Wired embedding-based entity dedup into the main storage path, catching semantic matches (e.g., "Museum of Crypto Art" / "MOCA") that Levenshtein misses; controlled by ENABLE_SEMANTIC_ENTITY_RESOLUTION (default: true), falls back to Levenshtein

Graph Visualization Diversity

Replaced ORDER BY mention_count visualization query with a diversity score: mention_count / (1 + log(1 + degree)) — prevents hub entities from dominating the default 100-node view
Stronger d3 force charge and link distance for better graph spacing

Bug Fixes

Fixed multi-round progress: cumulative batch counts across all rounds with global ETA instead of per-round reset
Fixed rebuild flag not passed through to analyze_collection_relationships, causing rebuild mode to run only 1 round instead of RELATIONSHIP_MAX_ROUNDS
Fixed login white screen by moving redirect() out of server action into client-side useEffect

March 25, 2026

Bulk Document Download

Added POST /api/documents/download-zip endpoint that accepts a list of document IDs, builds a ZIP64-enabled archive of original files with duplicate filename disambiguation, and streams the response in 1MB chunks
Added Download button to the bulk actions toolbar on the Documents page — select documents and download them as a single ZIP file
Handles 1000+ files with ZIP64 support

March 20, 2026

Parallel Image Analysis Within Documents

Images within a single document are now analyzed concurrently using asyncio.gather, replacing the previous sequential for-loop
Added VISION_MAX_CONCURRENT environment variable (default: 3) to control the maximum number of concurrent vision API calls system-wide
Thread pool sizes for image processing now scale automatically with VISION_MAX_CONCURRENT
Fixed "Event loop is closed" errors during concurrent image analysis by using thread-local HTTP clients
For a document with 200 images at ~30s per image: processing time reduced from ~100 minutes (sequential) to ~10 minutes (with VISION_MAX_CONCURRENT=10)

March 18, 2026

Entity Deduplication

Added GET /api/entities/duplicates endpoint to find duplicate entity candidates using fuzzy name similarity, with configurable threshold (0.5-1.0) and limit parameters
Added POST /api/entities/merge endpoint to merge duplicate entities into a canonical entity, transferring all relationships, chunk mentions, and community memberships; merged entity names are stored as aliases on the canonical entity
Added GET /api/entities/merge-history endpoint to view a chronological log of past merge operations
Updated Python client class with find_duplicate_entities(), merge_entities(), and get_merge_history() methods
Added cURL examples for all three entity deduplication endpoints
Updated OpenAPI specification with new endpoints and schemas

March 17, 2026

Two-Phase Graph Extraction Pipeline

Refactored entity extraction to work per-document instead of per-chunk, reducing LLM calls from N (one per chunk) to 1-4 (batched by token budget) per document
Added dedicated graph extraction model configuration (GRAPH_EXTRACTION_MODEL, GRAPH_EXTRACTION_API_BASE, GRAPH_EXTRACTION_API_KEY) to allow using a smaller/faster model for entity extraction
Entity-to-chunk linking now uses fuzzy string matching (rapidfuzz) to create MENTIONS relationships
Added entity provenance tracking: source_documents, extraction_count, last_extracted_at properties on Entity nodes
New Phase B relationship analysis endpoint: POST /api/graph/relationships/analyze - discovers cross-document relationships using the main (large) model
Relationship analysis runs as a background task with progress tracking via /api/tasks/{id}
Added relationship provenance: extracted_at, extraction_method, source_document_id properties on relationships
New auto-trigger options: AUTO_RELATIONSHIP_ANALYSIS_AFTER_BATCH and AUTO_COMMUNITY_DETECTION_AFTER_BATCH for automated pipeline after batch processing

Community Deletion and Graph Cleanup

Added DELETE /api/graph/communities/{id} endpoint to delete a specific community (entities are unlinked but preserved)
Added DELETE /api/graph/communities endpoint to delete all communities at once
Enhanced POST /api/cleanup/orphaned-entities to also clean orphaned communities in one call
Added delete buttons (individual and "Delete All") to the Explore page communities panel and the Collections page community section
StatsBar now shows an orphaned data warning banner with a cleanup button when entities/relationships exist but no documents are present
Frontend API client now includes deleteCommunity(), deleteAllCommunities(), and cleanupOrphanedEntities() methods

March 16, 2026

Ask AI API Endpoint Update

Renamed API endpoints from /api/rag to /api/ask across all examples and documentation for improved clarity and consistency
Updated configuration settings for document processing and community detection endpoints

March 5, 2026

Large PDF Processing Improvements

Added memory optimizations for handling large PDF files, including chunked processing and backend unloading to prevent out-of-memory errors
Introduced PAGE_CHUNK_SIZE and MAX_PAGES_PER_CHUNK environment variables for configuring chunked PDF processing
Integrated pypdf for lightweight PDF page counting
Added fallback to PyPdfium for large files when the primary converter encounters memory constraints
Fixed page index conversion in the Docling worker to correctly handle 1-based page indices

Image Analysis Execution Optimization

Implemented a synchronous version of the image analysis method in VisionAnalyzer to allow thread pool execution, preventing the main event loop from being blocked
Updated document converter to support image format options for more flexible input handling
Updated Dockerfile with necessary system dependencies for PDF/image processing, including X11 libraries and Tesseract OCR

March 4, 2026

Docling Integration and Advanced Document Processing

Integrated Docling as the primary document conversion engine, replacing the previous processing pipeline
Added a dedicated thread pool for image analysis to optimize performance and prevent competition with document processing tasks
Implemented subprocess-based document conversion for improved efficiency with large files
Added event loop watchdog to monitor and log potential blocking issues
Enhanced VisionAnalyzer with retry logic for API calls
New fields for image analysis progress in the Document model, with frontend components to display processing status
Added configuration options for vision models in the environment setup
Increased upload body size limit to 100MB

Logging and Robustness Improvements

Suppressed Neo4j notification warnings by adjusting the logging level
Added a function to clean image placeholders in document processing
Conditionally configured OCR and image description based on vision model availability
Enhanced GraphExtractor to handle potential None responses from document summary generation
Refactored Neo4jService queries to use coalesce for better handling of missing data
Refined default analysis prompt in VisionAnalyzer for improved clarity and document retrieval purposes

March 2, 2026

Image Extraction and Vision Analysis

Added a new vision analyzer service that extracts images from documents (PDF, DOCX, PPTX) and analyzes them using configurable vision models
Supports OpenAI-compatible APIs with fallback to Docling's built-in image descriptions
Image analyses are integrated into the RAG pipeline for searchable visual content
Enabled Docling's built-in image description generation (do_picture_description) as an additional source
Added EasyOCR for enhanced image text extraction
Updated Dockerfile with Tesseract OCR and additional libraries for improved document processing

Enhanced File Upload Support

Expanded allowed file extensions to support a wider range of document types including additional document and media formats
Updated frontend FileUpload and UploadZone components to reflect the expanded file type support
Updated file type icons for better visual representation of supported formats

February 27, 2026

Document Processing Engine Overhaul

Updated Dockerfile to include necessary system dependencies for Docling, enabling PDF and image processing
Added docling-haystack dependency for enhanced document handling
Expanded allowed file extensions in configuration to support a wider range of document types
Refactored document processor to utilize the new Docling converter, streamlining handling of various document formats

February 25, 2026

Customizable Branding and Theming

Added support for custom logo configuration via the CUSTOM_LOGO_URL environment variable, allowing white-label deployments
Introduced custom accent color support via the CUSTOM_ACCENT_COLOR environment variable for theme customization
Added configurable branding options (site title, description) through environment variables
Increased logo dimensions for improved sharpness on high-DPI screens
Documented all frontend customization variables in the deployment guide

February 6, 2026

API Key Usage Tracking and Analytics

Introduced a dedicated API usage tracking service that records every request per API key, categorized by endpoint type (ask, search, upload, documents, graph, collections, admin, etc.)
Added middleware to automatically track API requests and errors for all authenticated endpoints
Admin API key records are now persisted in Neo4j for long-term usage statistics
New backend models for API key usage statistics, usage data points, and usage history

Admin Dashboard Improvements

Added an API Key Analytics panel with interactive usage charts showing request volume over time and endpoint breakdown
New API Key Manager component for creating, viewing, and managing API keys directly from the admin UI
Introduced a System Reset Modal with granular deletion options (documents, uploaded files, custom inputs, collections, API keys) and a confirmation safeguard
Admin page now displays system configuration overview including API key usage settings
Updated Cortex logo assets (dark and light variants)

Resource Limits

Added MAX_FILES and MAX_COLLECTIONS environment variables to cap the number of documents and collections (set to 0 for unlimited)
Backend now enforces these limits on file uploads, custom input creation, and collection creation, returning a 403 error when limits are exceeded

Collection-Scoped AI Queries

The Ask AI feature now supports scoping queries to a specific collection via collection_id, so answers are drawn only from documents in that collection
Collection scoping works across all ask modes: standard, streaming, streaming with thinking, and fast search
Ask AI settings (streaming, agentic mode, fast search, selected collection) are now persisted to localStorage so preferences survive page reloads

Skill Library Restructure

Renamed the skill package from openclaw-library to library
Added sync scripts (Python, Shell, and JavaScript) for automated skill file synchronization
Removed obsolete OpenClaw library skill files

February 5, 2026

Document Deletion with Task Cancellation

Deleting a document now automatically cancels any active processing task for that document before removal, preventing orphaned background jobs
Introduced proper task tracking for active document processing with cancellation flag support
Added cancel_multiple_documents and improved cancel_processing_task methods in the DocumentProcessor with deadlock-safe locking
Processing tasks are now started with cancellation flags initialized upfront, ensuring graceful shutdown from the very first checkpoint

Knowledge Graph Cleanup on Deletion

Document deletion now triggers a comprehensive cleanup of the knowledge graph, removing orphaned entities and communities that are no longer referenced by any remaining document
API responses for deletion endpoints now include detailed cleanup statistics (entities removed, communities removed, etc.)
Updated documentation pages for Document Upload and Knowledge Graph features to explain the new deletion and cleanup behavior

API Parameter Changes

collection_id and start_processing parameters on the upload endpoint are now passed as URL query parameters instead of form data fields
Updated API examples in the documentation (cURL and Python) to reflect the new parameter format
Skill documentation updated to emphasize that both api_key and base_url credentials are required
Skill version bumped to 1.2.0

February 4, 2026

Admin Authentication and API Key Management

Implemented a full admin authentication system with email/password login validated against environment variables (ADMIN_EMAIL, ADMIN_PASSWORD)
Added session management with encrypted JWT tokens stored in HTTP-only cookies
Introduced Next.js middleware to protect all routes (except /login) behind authentication
Created a dedicated login page with form validation and error handling, wrapped in React Suspense for smooth loading states

Permission-Based API Protection

All API endpoints are now protected with a three-tier permission system: read, manage, and admin
Read endpoints (search, ask, list documents, stats) require at minimum a valid API key with read permission
Write endpoints (upload, delete, create collections) require manage permission
Admin endpoints (API key CRUD, system management) require admin-level access
New backend services: auth_service for request authentication and api_key_service for key lifecycle management

Documentation Site Launch

Launched the full documentation site built with Zudoku, featuring:
- Getting Started guides (Introduction, Quickstart, Configuration)
- Core Features documentation (Document Upload, Search, Ask AI, Knowledge Graph, Collections, Communities)
- Guides (Deployment, Authentication, Security)
- Code examples (Python, cURL, Integration)
- OpenAPI-powered API Reference
- Pagefind-based full-text search
- LLM-friendly output (llms.txt and llms-full.txt)
Created comprehensive backend API documentation (BACKEND_API_DOCUMENTATION.md)
Cleaned up documentation pages by removing redundant section headers for a streamlined reading experience

January 29, 2026

Rebrand to Cortex

Renamed the project from MOCA Knowledge Base to Cortex
Updated the tagline to "The Agentic Knowledge Base for the AI Era"
Added a new "What is Cortex?" section to the README explaining the project's philosophy around portable, long-term AI memory
Introduced the memory hierarchy concept: Context (short-term) → Agent Memory Stack (mid-term) → Cortex (long-term)

Last modified on July 28, 2026