Changelog

All notable changes to Cortex are documented here, organized by date.

September 4, 2026

Cortex Chat catches up with the last month of backend releases: a compatibility fix for Cortex 1.2.0+, the September 3 answer-quality flags surfaced in the chat UI, and the August 15 no-content document state in the library console. Everything below is on cortex-chat main for the next Chat release; the stack pin stays where it is until that tag exists.

Cortex Chat: non-streaming chat against Cortex 1.2.0+

Fix: the chat's own session_id no longer reaches the backend. Cortex Chat tags every turn with its chat-session id (session_id, the relay target that lets project members watch a teammate's answer stream in). Since the August 10 server-side sessions work that name is a real ask-request field, so the non-streaming path — the one taken when a user switches off Stream responses in the chat settings — failed on every Cortex 1.2.0+ backend: 403 Sessions are disabled with ENABLE_SESSIONS off, 400 session_conflict with it on. The streaming proxy already stripped the chat-app fields before forwarding; the generic proxy now strips session_id, assistant_id and project_id ahead of POST /api/ask as well.

Cortex Chat: answer-quality flags in the UI

Refusals and cut-off answers are labeled. The chat reads the September 3 refused / truncated flags from the done frame (and from the non-streaming response) and shows a neutral notice under the answer: declined by the safety filter, rephrase as a plain question — or cut short at the length limit, regenerate or narrow the question. The flags persist with the message like the thumbs rating, so the notice survives reload and device switch. Against older backends the chat falls back to matching the canned refusal text.
The personality generator skips refused research answers. Its benign research questions ride /api/ask/stream; a safety-filter deflection is now dropped from the findings via the flag instead of a text heuristic alone. The research asks also stop sending an empty conversation_memory, which only bought a discarded post-answer compaction call per question.

Cortex Chat: no-content documents in the library console

Grey "No content" / "Protected" chips. The Documents tab reads the August 15 content_status / content_note fields: zero-byte, zero-page and password-protected sources get a grey chip beside their status with the backend's explanation as tooltip, and are excluded from the client-side Degraded heuristic — so an empty file never shows up as a broken pipeline.

September 3, 2026

Three fixes driven by field reports: an agent researching against a public Cortex found that a cut-off answer and a canned prompt-injection refusal both looked exactly like a complete answer, that a failed non-streaming ask came back as a bare 500, and that fifteen essays crawled one at a time from the same site all carried the same document name; a self-host operator hit a TLS error cloning from a plain-HTTP forge.

Ask AI: answer-quality flags and structured failures

Machine clients can now tell a real answer from a deflected or clipped one without parsing prose, and can fall back to the streaming endpoint automatically.

refused: true — when the prompt-injection defenses answer with the safe refusal instead of knowledge, the streaming content and done frames carry the flag, and the non-streaming POST /api/ask response has a top-level refused. The refusal text itself is unchanged.
truncated: true — when the writer hits its output-token cap (WRITER_MAX_TOKENS_SPEED / WRITER_MAX_TOKENS_QUALITY), the done frame carries the flag alongside the existing visible "cut short" note; the non-streaming response adds truncated plus the provider's finish_reason.
Structured 500 on POST /api/ask — a failure before an answer is produced now returns {"detail": {"error": "ask_failed", "message": …, "use_endpoint": "/api/ask/stream"}} (exception text included outside production only). Production error sanitization keeps deliberately structured 5xx bodies — those carrying an error code — so the documented 504 deadline_exceeded now also reaches clients in production instead of being replaced by the generic message. Free-text 5xx details are still sanitized; every 5xx still carries request_id.
No API or schema breakage — all new fields are additive and default to false/null; SSE frames keep their flat keys and type stamp.

Web Import: titled single-page documents

A one-page import whose crawl yields a real page title is now filed as host - Page Title.md with a matching # host - Page Title heading, so repeated single-page imports from one site (a newsletter's essays, a blog's posts) stay distinguishable in search results and citations. URL-fallback "titles" such as o.html keep the plain host.md, and multi-page site imports are unchanged: still one document per domain, titled by the domain.

Git Integration: plain-HTTP self-hosted forges

A self-hosted Gitea or GitLab reached over plain http:// (the normal case for a LAN install) could be connected and verified, but every sync then failed at git clone with GnuTLS, handshake failed: An unexpected TLS packet was received. The REST client honored the scheme of the connection's base_url, while the clone-URL builder kept only the host and re-attached a literal https://. Clone and wiki-clone URLs — and the tokenless remote written back after cloning — now take their scheme from base_url, defaulting to https when none is set, so GitHub cloud and existing https connections are unchanged. Reported by a self-host operator.

September 2, 2026

A recall pass over the retrieval that feeds every answer. Walking the hybrid-search path end to end turned up four places where candidates were silently lost before the reranker ever saw them — the graph leg only fired on an exact entity-name match, collection-scoped vector search returned a fraction of what it was asked for, a stray / or : in a query dropped the whole keyword leg, and the reranker was handed exactly as many candidates as it keeps. Each is fixed inside the search layer; the API surface, response shapes and environment variables are unchanged.

Hybrid search recall

Entity aliases now count at query time — graph traversal matches stored entity names exactly, while the names it receives come from an LLM (query extraction or the researcher's own hints). "polygon" or a merged alias never matched Polygon Network, so the graph leg of hybrid search — and the entities and relationships the writer sees — stayed empty. Query entities are now resolved first: exact name, then case-insensitive name or alias (the alias list that dedup, merge and rename already maintain), then a fulltext match on the name field that only accepts a stored name containing every word of the mention.
Collection-scoped vector search over-fetches — Neo4j vector indexes cannot pre-filter, so a scoped request for 15 candidates fetched the 15 nearest chunks across the whole library and then discarded the ones outside the collection, often leaving one or two. When a collection scope or filter applies, the index is now asked for 10× the candidates (capped at 200) before filtering. Unscoped queries are unchanged.
Keyword search is parse-safe — questions containing /, :, unbalanced parentheses, a leading - or a stray NOT threw inside the Lucene parser and the keyword leg returned nothing, with only a warning in the log. Queries are now reduced to their word tokens before reaching the index; the same guard covers community-summary search.
The reranker selects instead of reordering — each of a search's queries fetched 5 fused candidates, so three queries pooled at most 15, which is exactly what RERANK_TOP_K keeps. The per-query depth is now derived from RERANK_TOP_K (three queries → 10 each, floor 5, cap 12) so the pool is about twice what is kept, and duplicates across queries are collapsed before reranking so the same chunk never occupies two slots.
Graph traversal is ranked and about ten times faster — the old traversal walked two hops through any node, so a character's "neighbors" were every word co-mentioned in the same paragraph ("Age", "Mirror", "close"), and it pulled the full text of every passage of every neighbor only to keep the first ten in arbitrary order. The new traversal follows entity-to-entity relationships only (one hop plus a capped second ring), ranks passages by how many of the question's entities they mention — the entities you asked about weigh more than their neighbors — and applies the limit inside the database. On the reference graph a hub entity went from 350ms to 30ms and a two-entity question from 774ms to 21ms, while the neighbors became the people and factions the graph actually links. The entity detail endpoint keeps the previous traversal and is unchanged.
The three legs run concurrently — vector, keyword and graph traversal for one query were three sequential round-trips; they now run in parallel on a dedicated thread pool.

Deep research tools honor the collection scope

community_search and entity_lookup ran without the request's collection or the API key's collection restriction, unlike knowledge_search. Both now receive the same scope — the request's collection when set, otherwise the key's allowed collections. Before, out-of-scope community summaries could reach a scoped client through the graph_context event.

Operator visibility

One log line per knowledge_search (INFO) — queries, whether the researcher supplied entity hints, per-query depth, scope, per-query vector/keyword/graph hit counts, query vs. resolved entity counts, and pooled/unique/kept candidate counts. This is the way to see how often the graph leg actually contributes on a given library.
RERANK_TOP_K (default 15) keeps its meaning — candidates kept per search — and additionally drives the per-query fetch depth.
Revert switches — each behavior change can be backed out per instance without a redeploy: ENABLE_QUERY_ENTITY_RESOLUTION=false (exact-name matching only), VECTOR_SCOPED_OVERFETCH=1 (no over-fetch), ENABLE_RANKED_GRAPH_TRAVERSAL=false (legacy traversal), ENABLE_PARALLEL_SEARCH_LEGS=false (sequential legs). Entity resolution results are cached in-process for 60 seconds per name set, so a search's parallel queries and the next research iteration reuse them.
No API or schema breakage; the /api/graph/search and /api/graph/entity/{name} endpoints keep their existing ordering and output.

August 17, 2026

Dedicated writer model

The two halves of an answer have different demands: the researcher loop needs strong, reliable tool calling, while the writer only composes prose from the context the loop already gathered. Until now both ran on OPENAI_MODEL. A new optional tier lets them differ — keep a fast tool-caller researching and hand synthesis to a better prose stylist, or a cheaper model.

New WRITER_MODEL, WRITER_API_BASE, WRITER_API_KEY — each falls back independently to the primary config, so setting just WRITER_MODEL reuses the primary gateway and key (same idiom as the extraction and relationship tiers).
Applies everywhere an answer is written — the agent pipeline's writer stage (Chat and Deep Research) and the legacy non-agent streaming writer.
Everything else about the writer is unchanged — reasoning stays suppressed regardless of model, and the WRITER_MAX_TOKENS_* limits, citation guards and truncation notice apply as before. Unset, behavior is byte-identical to previous releases.

August 15, 2026

Two independent cleanups: documents that hold nothing to ingest stop being treated as failures, and deep research stops paying for hidden thinking it never uses.

Deep research runs non-thinking by default

Thinking-by-default models make hidden reasoning expensive in a way the researcher loop gains nothing from: the trace is discarded between chat-completion turns, so it is re-paid every iteration while accumulating nothing, and it competes with the visible output for the same max_tokens budget. Qwen3.8 ships reasoning_effort=xhigh by default — enough to spend an entire 16k completion cap on a trace that gets cut mid-thought. Deliberation in that loop is already explicit: the reasoning tool, which the forced-reflection micro-call pins tool_choice to, produces the thought with or without hidden thinking, and that thought persists in the message history and steers the next round.

New RESEARCH_REASONING_MODE (default off) — the deep-research (quality) researcher loop, previously hardcoded to auto. Set to auto to restore provider-default thinking if a model's tool-calling regresses without it. The chat path keeps its own DEFAULT_REASONING_MODE, and the final writer stays reasoning-off in both modes as before.
Retry paths no longer drop reasoning params — the researcher loop's empty-choices, tool_choice=required, nudge, and empty-response retries called the API directly, so on a thinking model the retry ran with thinking while the call it retried ran without it. All loop calls now go through one wrapper.
App-task llm steps run on the extraction tier but called the API bare, spending the step's whole token budget on hidden CoT and returning truncated text on a thinking model. They now carry EXTRACTION_REASONING_MODE like every other extraction-tier call.
No API or schema breakage; RESEARCH_REASONING_MODE=auto restores the previous behavior exactly.

Empty and password-protected documents get a terminal state

Documents that hold nothing to ingest are no longer failures. A zero-byte export, a PDF with no pages, or a password-protected PDF produces the identical result on every retry — so parking them in failed left large archive imports with permanently-red documents no reprocess could ever clear (reported from a self-hosted 3.6k-document Paperless library, where 21 of 24 failed documents were exactly this). They now finish as completed carrying a reason flag, and the UI badges them neutrally instead of red.

New document fields — content_status ("empty" | "encrypted", absent for normal documents) and content_note, a human-readable sentence. Returned by GET /api/documents and GET /api/documents/{id}.
Detected before conversion — a pre-flight probe catches zero-byte files (any type), PDFs with an empty page tree, and PDFs needing a password Cortex does not have, so Docling never pays a model load for a file it cannot read. Post-conversion empty output (including after the scanned-PDF OCR retry) lands in the same state.
Deliberately conservative — only a positive identification counts. A malformed PDF, or anything the probe cannot answer, takes the normal conversion path and a genuine conversion failure is still reported as failed.
Password-protected PDFs are called out separately — these often hold real content (a locked multi-page scan), so the note says what to do: re-upload a decrypted copy. PDFs with an owner password but an empty user password are unaffected and convert as before.
UI — grey No Content / Protected badge with the explanation inline on the card, a No Content status filter that appears once a library has any, and exclusion from both the Completed count and the failed/degraded needs-attention banner and its bulk-reprocess selection. Empty documents also no longer show up as Degraded.
Operator surfaces — GET /api/ingestion/status counts them in a separate no_content bucket; the cortex_documents_processed_total metric gains a status="no_content" label; the document.processed webhook carries content_status.
No migration needed — reprocessing an affected document reclassifies it, and the flag is cleared at the start of every run, so re-uploading a decrypted or non-empty replacement ingests normally. No API or schema breakage.

August 14, 2026

Cortex Slides joins the ecosystem as a beta release: a standalone app that turns the domain knowledge inside a Cortex instance into presentation decks — pitch, training or report, 6 to 30 slides — delivered as a .pptx plus a PDF. Slides are full-image: gpt-image-2 (via Venice.ai) paints each entire slide, typography included, from the layout's composition brief, the slide's exact approved text, its artwork concept and a deck-wide design system — maximally designed pages instead of a filled template. The one slide approved at the style gate then anchors the rest: it travels along as a reference image so every page shares a single visual system. The trade is explicit: on-slide text is painted pixels, so the pipeline approves every word before generation, tells you to proofread instead of pretending to verify pixels, and repaints any single slide for the price of one image. Like Cortex Chat, Cortex Trainings and Cortex Videogen it runs as its own deployment and connects purely through the REST API with a read-scoped key.

Cortex Slides (Beta): six steps, three human gates

Research — two-channel grounding against the instance: agentic Deep Research writes the argument, verbatim evidence excerpts from hybrid search keep exact names and numbers intact, and the graph's topic communities are appended as outline fuel. Sectioned decks get one pass per section; research can be scoped to a collection.
Outline (the content gate) — the whole deck across 8 layouts, each slide with a takeaway title, its text and an artwork concept, shown next to the research it is grounded in. Approve (optionally with per-slide hand edits, applied verbatim), regenerate with feedback, or re-run the research with steering — all free, unlimited rounds. Every word approved here is painted into pixels.
Style (the look gate) — three of ~12 design-system presets proposed with rationale, then one representative content slide painted in full as a paid sample (cached per preset — comparing styles never re-pays). What is approved is a finished page, not a swatch, and it becomes the deck's style anchor; approved styles save to a cross-project library.
Visuals (the cost gate) — every missing slide is quoted against live pricing and the run pauses with the total; the approved sample already counts as done. Optional subject reference images bind a product, mascot or character's identity on the slides that feature it, while each page stages it in its own pose and camera angle.
Assembly & QA — pure packaging into .pptx (one full-bleed image per slide) and PDF, then structural probes that end by naming what they cannot verify: the painted text. Post-delivery, a text or artwork fix quotes and repaints exactly one slide, as a fresh render rather than a touch-up.
Every artefact is cached — a crashed or resumed run never pays for the same slide twice. Reference cost: ~$0.50 per slide on gpt-image-2 at 2K/high — a 12-slide deck lands around $6.

Documentation: Cortex Slides

New docs page — Cortex Slides covers the full-image approach and its trade-offs, the pipeline with its three gates, setup (Node 22+, three env vars: VENICE_API_KEY, CORTEX_BASE_URL, CORTEX_API_KEY), and costs.
New handbook chapter — Chapter 29 covers the same ground for handbook readers; Chapter 24 (Apps), Chapter 27 (Cortex Trainings) and Chapter 28 (Cortex Videogen) now name Slides as the third sibling standalone app.
Access model — the app reads only, using a read-scoped (cortex_ro_…) key that is ideally collection-scoped; research relies on the instance's agentic retrieval (ENABLE_AGENTIC_RAG, ENABLE_AGENT_RESEARCH).
No changes to Cortex itself — the instance-side surface is unchanged, and everything above lives in the app at github.com/mocaOS/cortex-slides.

August 12, 2026

Release 1.2.1: the hardening pass, tagged

Cortex 1.2.1 is tagged and on GHCR — a patch release shipping the security hardening pass below on top of 1.2.0. No API or schema changes, no new configuration; the cortex-chat pin stays at 1.1.0. Existing self-host installs update with npx @mocaos/cortex update.

Release 1.2.0: everything above, installable

Cortex 1.2.0 is tagged and on GHCR. It ships the whole run of work since 1.1.1 as one tested stack: the API/SDK release of August 10 (the TypeScript SDK, server-side sessions, remote MCP, POST /api/context, the depth dial, webhooks, structured answers, the consolidation scheduler and the ergonomics pass) and the August 6 conversion fast path (anydoc routing, the >1000-chunk truncation fix, the scanned-PDF OCR retry). The cortex-chat pin is unchanged at 1.1.0. An existing self-host install gets all of it with npx @mocaos/cortex update.

New default primary model: Qwen3.6 35B A3B (replacing Gemma4 26B A4B) — qwen3-6-35b-a3b on Venice, qwen/qwen3.6-35b-a3b:nitro on OpenRouter, 262K-native context. This is the default for new installs; an existing .env with an explicit OPENAI_MODEL keeps whatever it names.

Security hardening: skill/app egress guards + import robustness

A pass over an external security report closed the remaining unguarded server-side fetch paths and made library imports resilient to crafted archives. No API or schema breakage; existing deployments need no changes unless a skill legitimately calls an internal API.

Skill HTTP tools + install-from-URL now SSRF-guarded — a skill's tools.json execution.url and the admin install-from-URL fetch now resolve through the same ssrf_guard policy as the agent's built-in http_request tool: private/loopback/link-local/metadata (169.254.169.254) targets are blocked by default. Operator escape hatches unchanged: SKILL_HTTP_ALLOW_PRIVATE (default false) and SKILL_HTTP_ALLOWED_HOSTS (default empty) — a skill that legitimately calls an intranet API needs one of these set.
App registry artifact fetch guarded — the release-zip download validates the catalog URL (and every redirect hop) with private targets always refused; registry artifacts are public HTTPS by design, so nothing changes for the default registry.
Git clone re-validated at sync time — the clone/fetch target is re-resolved through the SSRF guard on every sync, narrowing the DNS-rebinding window (git still resolves DNS itself, so container-level egress filtering remains the complete answer for the clone path).
Library import survives crafted archives — a hostile file_path (..) no longer aborts the whole import: the document node imports file-less with a warning. And two documents sharing a basename no longer silently overwrite each other on disk — restored files are named {doc_id}_{basename} with each copy failure-isolated to a warning. Previously imported libraries are unaffected.
Weak app-token secret warning — with Apps enabled, boot logs a loud warning when the app-token signing secret (SESSION_SECRET, falling back to ENCRYPTION_KEY/ADMIN_API_KEY) is shorter than 32 characters outside production.

August 10, 2026

Cortex gets an official TypeScript SDK. Until today, every integration talked to Cortex through its own hand-rolled client — a bash helper here, a Python plugin there, a TypeScript wrapper in the MCP server — each independently rediscovering the same choreography: which endpoint honors deep research, how to parse the SSE stream, when the memory frame arrives, how collections resolve. @mocaos/cortex-client ends that era: one typed, dependency-free package that owns the entire integration surface, informed by everything live agent-harness testing taught us about where clients go wrong. Alongside it, the API itself was reshaped so the SDK launches on a clean surface instead of wrapping workarounds — one depth dial instead of boolean flags, webhooks instead of polling, structured answers, and a knowledge graph that keeps itself fresh.

The SDK: `@mocaos/cortex-client`


Code
 
const cortex = new CortexClient({ baseUrl, apiKey });

const quick = await cortex.ask("What did we decide about the auth rewrite?");
const deep  = await cortex.deepResearch("Compare every SSE approach we tried", {
  onContent: (token) => process.stdout.write(token),
});

const thread = cortex.thread("auth-review");        // multi-turn memory
await thread.ask("How does Cortex validate API keys?");
await thread.ask("Expand on the caching part");      // follow-ups just work

What it owns so integrators never have to:

Unified ask — depth: "fast" | "standard" | "deep", with deep transparently running over SSE (the only place the backend executes agentic research). Version-tolerant: it sends the new dial and the agreeing legacy flags, so one client works against current and older instances alike.
Streaming done right — typed SSE events, keep-alive handling, graceful-restart detection, and an aggregator that reads past the done frame for the late conversation-memory update (the subtle contract every hand-rolled client got wrong at least once).
Conversation threads — history plus the server-curated memory blob carried across asks, with pluggable persistence; the file store shares its on-disk format with the agent skill's thread state, so a CLI agent and an SDK app can continue each other's conversations.
The rest of the surface — upload-and-wait, collection find-or-create, server-side document listing, ingestion status, knowledge-graph reads, webhook administration, and typed errors carrying the backend's machine-readable codes.

Node ≥ 18 and modern browsers; 11 kB packed; zero runtime dependencies.

The MCP server rides on it. @mocaos/cortex-mcp was rebuilt as a thin wrapper over the SDK and is now a one-liner install — npx @mocaos/cortex-mcp — instead of clone-and-build. Same tools, now with conversation-thread support in ask_question.

Server-side sessions: conversations the backend remembers (opt-in)

Cortex's multi-turn machinery has always been client-carried: send the full history plus an opaque memory blob every turn, catch the update frame, replay both. That contract stays — it's the right stateless default — but with ENABLE_SESSIONS=true there is now an alternative: POST /api/sessions mints a session_id, and passing it on ask endpoints makes the backend keep the conversation itself. Follow-ups work from a bare curl; the request size stops growing with conversation length; and a conversation started on one machine can be continued from another.

Sessions belong to the API key that created them (anything else sees 404), inherit its collection scoping, and can be seeded at creation with existing history + memory — the migration path for clients already using the blob contract.
Passing session_id together with client-carried state is a clean 400 — one source of truth per request.
History is capped per session (oldest turns trimmed, with the memory blob's internal index adjusted so the rolling summary stays coherent), sessions are quota-capped per key, and idle sessions expire on a configurable TTL.
Off by default, deliberately: enabling means the instance stores conversation content on behalf of API callers — a data-retention decision the operator makes explicitly. Clients can feature-detect via GET /api/features.

POST /api/context: bring the knowledge to your own prompt

Every ask endpoint has Cortex write the answer. The new context endpoint inverts that: one call returns a token-budgeted context bundle — reranked chunks, entity/relationship context, and community summaries — both structured and as a ready-to-inject text block with [src_N] citations. It's for agents and pipelines that compose their own prompts: RAG loops, prefetch layers, anything that wants Cortex as pure retrieval. Budget policy keeps chunks whole (graph and community enrichment capped at ~30% of the budget), and the response reports exactly how the budget was spent. Collection-scoped like every read; available to monetized keys at the base per-query rate. Also exposed in the SDK (cortex.getContext(query, { max_tokens })) and as a get_context MCP tool.

Remote MCP: every instance is now an MCP server

With ENABLE_REMOTE_MCP=true, the instance itself speaks the Model Context Protocol at /mcp (streamable HTTP, stateless) — point Claude Desktop, Claude Code, or Cursor at https://your-instance/mcp with an API key and you're connected, no npm install, no local process. Thirteen tools (search, ask with deep research, context assembly, documents, knowledge graph, collections, communities, uploads, stats) dispatch through the instance's own REST API, so authentication, collection scoping, quotas, and usage analytics treat MCP traffic exactly like API traffic. Long deep-research calls stream with keep-alives so proxies don't cut them. Off by default; when disabled the endpoint doesn't exist. The npm package (npx @mocaos/cortex-mcp) remains the right choice for stdio-based setups and conversation threads.

One dial for research depth: `depth`

The chat/deep-research split used to live in two boolean flags with non-obvious interactions. Ask requests now take a single depth parameter — fast (vector-only quick answer), standard (the default researched answer), deep (agentic deep research, streaming endpoints only). The legacy use_agentic/use_fast_search flags remain permanently supported as aliases; sending a flag that contradicts depth is a clean 400. Pay-per-query instances price by the same resolution: depth: "deep" quotes the research rate in the 402 challenge, and malformed or contradictory requests are rejected before any payment moves.

Webhooks: stop polling for "is it done?"

With ENABLE_WEBHOOKS=true, admins can register webhook endpoints (POST /api/admin/webhooks) that receive signed POSTs when work finishes: document.processed, document.failed, task.completed, task.failed. Deliveries carry a Stripe-style HMAC signature (X-Cortex-Signature: t=…,v1=…) so receivers can verify authenticity and reject replays; the signing secret is generated server-side, shown once at creation, and stored encrypted. Bounded retries with backoff, per-endpoint delivery bookkeeping, and a "send test event" endpoint round it out. Off by default — when disabled the feature costs nothing.

Alongside it, GET /api/ingestion/status answers "how busy is the pipeline?" in one call: document counts by status (including queued-for-a-slot), the documents currently inside a pipeline with live progress, and an overall idle flag — no more fetching the whole document list to count statuses.

Consolidation scheduler: the graph stays fresh unattended

Long-running instances accumulate staleness: documents keep arriving, entities get merged, and the community layer silently drifts until an operator clicks regenerate. With ENABLE_CONSOLIDATION_SCHEDULER=true, Cortex now refreshes itself — when the instance has been genuinely idle (no queries, no pipelines, configurable quiet window), the previous run's cooldown has passed, and there is actual work (stale communities or ≥ N new documents), the standard community-detection + summarization task launches on its own. It shows up in the task list like any operator-started run, emits the task.completed webhook, and deliberately never auto-merges entities — deduplication stays a human-reviewed operation. Off by default.

Structured answers: `response_format` on POST /api/ask

Pass a JSON Schema (root type: "object") as response_format and the non-streaming ask endpoint generates the answer as JSON conforming to it — parsed into a new structured response field (raw text always remains in answer). Provider-native structured output is used when available, with a graceful fallback cascade for OpenAI-compatible backends that lack it. Built for apps and agent harnesses that consume answers programmatically. Streaming endpoints and agentic mode reject it with a clear 400 pointing at the right call.

API ergonomics (all additive — nothing breaks)

Live end-to-end testing of agent integrations (the Hermes/OpenClaw skill, the memory plugin, the MCP server) showed every external client re-implementing the same workarounds. This release removes those frictions at the source — everything is additive, and no existing integration changes behavior.

GET /api/documents grows query parameters — collection_id, status, sort (upload_date|filename|file_size|chunk_count|processing_status|entity_count, - prefix for descending), limit + offset. Filtering, sorting and pagination now happen server-side; total reports the filtered count before pagination. With no parameters the response is byte-compatible with before. Previously every API client fetched the entire corpus and filtered locally.
Response fields normalized via aliases — search responses carry total alongside total_results; upload responses carry id alongside document_id; search/ask/stream sources now populate document_title (from the filename) instead of leaving it null; POST /api/ask echoes the collection_id that was actually applied instead of always null. The old names all remain — existing clients are untouched. Clients that deserialize with "reject unknown fields" enabled will see new keys.
SSE frames carry a type discriminator — every data: frame on the streaming endpoints now names itself (content, status, sources, done, memory_update, error, …) while keeping the flat keys, so new clients switch on type instead of key-sniffing and old clients notice nothing.
Uniform collection_id placement — POST /api/search accepts a top-level collection_id (equivalent to filters.collection_id; a disagreement between the two is a clean 400), and POST /api/upload accepts collection_id/start_processing/source as multipart form fields alongside the query params.

Conversation threads for every integration surface

Cortex has shipped a client-carried conversation-memory contract for a while (conversation_memory blob in, memory_update SSE event out) — but only the built-in frontend and Cortex Chat spoke it. Now the whole integration stack does:

Skill (cortex.sh) — cortex.sh --thread <name> ask "…" carries the full history plus the server-curated memory blob across calls, so follow-ups like "expand on the second point" finally work from a harness. cortex.sh thread list|show|clear manages threads; state lives beside the existing sources state, chmod 600.
Hermes memory plugin — cortex_ask gains thread (same state files as the skill — the two interoperate) and deep (agentic deep research, previously unreachable from the plugin) parameters, with a stdlib SSE reader that correctly handles the memory frame arriving after done.
MCP server — ask_question gains a thread parameter with the same semantics, wired through the SDK.

Skills distribution: versions + integrity manifest

Every skill on cortexskills.org now carries a frontmatter version, and the site publishes a machine-readable manifest at /index.json — per-skill version, entry point, and a sha256 + byte size for every distributable file. Agents that installed a skill from a single URL can now detect updates, verify what they fetched, and discover the companion files (scripts/, references/, plugin/) that a one-URL install misses.

August 6, 2026

Document conversion gets a second engine. Cortex now routes text-based PDFs and office documents through anydoc — an in-process, pure-Rust converter with no ML models — and reserves Docling for the documents that genuinely need its layout analysis and OCR: scans, image-rich PDFs, standalone images, and audio. The practical difference is dramatic: a 462-page book that previously ran into the 600-second conversion timeout now converts in under a second, at a fraction of the memory. Nothing about chunking, embedding, or extraction changes — only how the markdown gets made.

Conversion fast path (anydoc)

Routing, not replacement — every eligible file tries anydoc first; anything it declines falls through to the exact Docling paths that existed before (shared cortex-helper service or local subprocess). Worst case equals previous behavior plus a few milliseconds.
Eligible formats — PDF, Word (.doc/.docx), Excel (.xls/.xlsx), PowerPoint (.ppt/.pptx), and EPUB. HTML, images (OCR), audio (ASR), LaTeX, and XML keep their existing paths.
Scanned and hybrid PDFs stay on Docling — image-only PDFs are refused by anydoc with a typed "OCR is required" error, and PDFs whose text layer averages fewer than ANYDOC_PDF_MIN_CHARS_PER_PAGE (default 200) characters per page are treated as scans; both route to Docling's OCR pipeline automatically.
Vision analysis is protected — anydoc cannot extract images from PDFs, so when a vision model is active, PDFs with more than ANYDOC_PDF_MAX_IMAGES_PER_PAGE (default 0.5) embedded images per page keep the Docling path and their figure analysis. Embedded images in Word/PowerPoint/EPUB files are extracted on the fast path and flow into image analysis as before.
Master switch — ENABLE_ANYDOC (default true). Setting it false restores pre-router behavior exactly.
Per-document recourse — if a specific document converts badly on the fast path, POST /api/documents/{id}/reprocess?engine=docling forces a full Docling re-conversion for that run (bypassing the content-unchanged skip).
Slim image benefit — the anydoc wheel is ~8 MB and torch-free, so slim (INSTALL_LOCAL_ML=false) deployments without a cortex-helper can now convert office documents and text-based PDFs locally instead of failing.
Changing the engine configuration alters the reprocess fingerprint, so a reprocess after flipping ENABLE_ANYDOC performs a real re-run instead of reporting "Content unchanged". No API or schema breakage — the reprocess engine query parameter is the only new surface.

Fixed: documents with more than 1000 chunks were silently truncated

Image chunks used chunk_index 1000+ as their namespace, and two queries defined "text chunk" as chunk_index < 1000. Any document that legitimately chunked past 1000 — which the anydoc fast path makes routine, since full-length books now convert instead of timing out — had its chunk count clamped at 1000 in the UI, and worse: a resumed or re-run extraction only saw the first 1000 chunks, so up to two-thirds of a long book's content silently got no entities or relationships.

Text and image chunks are now distinguished by the image chunk's deterministic id ({doc_id}_image_{idx}), never by index; new image chunks sort at index 1,000,000+ so they can't interleave with text chunks. Existing image chunks keep their old indices — nothing filters on the number anymore.
The reprocess fingerprint carries a chunk-namespace:v2 marker, so reprocessing a previously-affected document performs a real re-run (full extraction over all chunks) instead of skipping as "Content unchanged". Documents under 1000 chunks were never affected.

Fixed: scanned PDFs silently produced empty documents on vision-enabled instances

Routing scans deliberately through Docling exposed a long-standing bug: with a vision model configured, Docling runs with OCR disabled (vision replaces its picture descriptions) — but a scan's text lines are layout-classified as text regions, not pictures, so with no text layer and no OCR the conversion returned neither markdown nor images and the document failed as "No content extracted".

OCR retry — when a PDF conversion on a vision-enabled instance returns no text and no images, Cortex now retries it exactly once with OCR forced on. Born-digital documents never pay for this; scans pay one extra conversion and come back with their text.
cortex-helper update required — the helper's /convert endpoint gained a force_ocr field (warm converters are cached per mode). A helper predating the field ignores it: the document then fails exactly as before, never worse — but deploy the updated helper together with the app to get the fix fleet-wide.
Verified live: a scanned page that previously failed empty now OCRs (EasyOCR, English + German) and proceeds through chunking, embedding, and extraction normally.

August 4, 2026

Cortex Videogen joins the ecosystem as a beta release: a standalone app that turns the domain knowledge inside a Cortex instance into short marketing videos — 15 to 120 seconds, landscape (16:9) or native vertical (9:16), delivered as a publish-ready MP4 master plus a lighter social copy. Every organization running Cortex already owns the hardest part of video production — the substance — and Videogen collapses the remaining distance to a form and two approvals: it researches the instance for grounded claims (the generation models are explicitly forbidden to invent facts), directs choreographed AI footage around them, and assembles the finished video with gapless narration, burned-in karaoke captions and a locally rendered call-to-action card — for single-digit dollars per video and roughly half an hour end to end. Like Cortex Chat and Cortex Trainings it runs as its own deployment and connects purely through the REST API with a read-scoped key.

Cortex Videogen (Beta): six steps, two human gates

Research — agentic Deep Research against the instance with two-channel grounding: the synthesized brief plus verbatim evidence excerpts from hybrid search, so exact names and numbers survive summarization. Research can be scoped to a collection.
Storyboard (the content gate) — a retention-first storyboard (hook in the first seconds, few long choreographed one-take shots, narration written for the ear) is shown next to the research it is grounded in. Approve, regenerate with feedback, or re-run the research with steering — all free, as many rounds as needed.
Voiceover — per-sentence TTS produces an exact timeline; shot lengths derive from the narration, so speech flows gaplessly across cuts.
Video generation (the cost gate) — every shot and start frame is quoted against Venice.ai's live pricing and the run pauses with the total; nothing is generated until confirmed. Optional character/product and style reference images condition every shot's start frame, so identity and aesthetic stay consistent across the film.
Assembly & QA — a deterministic offline render composites footage, a uniform color grade, karaoke captions (the only text layer — most feed video plays muted) and the brand-accent CTA end card, then the file is probed for duration, orientation and deliverables before the run reports done.
Every artefact is cached — a crashed or resumed run never pays for the same shot twice, and regenerating one shot costs one shot. Reference cost: a 30-second video is roughly $5–7 of footage plus ~$0.35 per start frame on the default MiniMax model at 2K.

Documentation: Cortex Videogen

New docs page — Cortex Videogen covers the opportunity, the pipeline with its two gates, setup (Node 22+, three env vars: VENICE_API_KEY, CORTEX_BASE_URL, CORTEX_API_KEY), and costs.
New handbook chapter — Chapter 28 covers the same ground for handbook readers; Chapter 24 (Apps) and Chapter 27 (Cortex Trainings) now name Videogen as the sibling standalone app.
Access model — the app reads only, using a read-scoped (cortex_ro_…) key that is ideally collection-scoped; research relies on the instance's agentic retrieval.
No changes to Cortex itself — the instance-side surface is unchanged, and everything above lives in the app at github.com/mocaOS/cortex-videogen.

July 31, 2026

Cortex Trainings joins the ecosystem as a beta release: a standalone app that turns a knowledge base into interactive training units. Where Cortex Chat answers the questions people think to ask, trainings cover the ones they don't know to ask — onboarding, compliance, process knowledge — generated from documents the organization already has, instead of written once by hand and left to go stale. The app also learns to look like you: upload reference images of your own character and of the visual style you want, and both carry through every scene of the finished course.

Documentation: Cortex Trainings (Beta)

New docs page — Cortex Trainings explains the app end to end: the two-part workflow (a researched, cited curriculum you approve as a plain document, then media production behind that gate), what a training unit contains, why the choice between generated video and locally rendered animation dominates cost, and the access model.
New handbook chapter — Chapter 27 covers the same ground for handbook readers, and Chapter 24 (Apps) now names Cortex Trainings alongside Cortex Chat as an example of a standalone app — one that runs outside the instance and connects through the REST API with scoped keys.
Access model — the app reads only, using a read-only (cortex_ro_…) key that is ideally collection-scoped, and requires ENABLE_AGENTIC_RAG and ENABLE_AGENT_RESEARCH on the instance because the research phase relies on agentic retrieval.
No changes to Cortex itself — the instance-side surface is unchanged, and everything below lives in the app at github.com/mocaOS/cortex-trainings.

Cortex Trainings: bring your own character and visual style

Until now every training used a generated abstract guide — an orb, a crystal, a cube — in the deployment's accent colour. A course can now carry your character and your aesthetic, so it reads as your material rather than as generic AI output.

Reference images, analysed once — upload up to three images of a character and up to three that define a visual style. A vision model turns each set into a precise English description at upload time; every prompt downstream, from the curriculum to the last film shot, is written against those descriptions. They can be replaced or removed until production starts, at which point they lock, because the plan was extracted from them.
Your character, not a paraphrase of it — the guide-character anchor image is generated from the uploaded images themselves through image-to-image editing, so identifying details survive that no text description would reliably reproduce. The uploaded images then ride along as extra consistency references on every shot that features the character.
Your character keeps its own colours. The single-accent rule still governs the interface, animations and highlights, but nothing is recoloured to match it.
The storyboard decides where the character appears. Roughly half of a film's shots — establishing shots, object details, concept imagery — are deliberately character-free, and the opening shot almost always is. Conditioning every shot on the character produced a mascot parade: the guide turned up in shots written without it, and because the anchor image is a hero portrait, every film opened on that same portrait. Character-free shots are anchored on the style references instead, so losing the character never means losing the look.
Reference images are instructions about everything, not just identity. A hero-framed character reference makes a video model copy that framing into every clip; style references bleed their subjects. Prompts now state explicitly what may be taken from a reference — identity only, or palette and lighting only — and the negative prompt names the portrait framing directly.

Cortex Trainings: film edges

Model-baked padding is cropped before assembly. Video models compose to their own taste and pad the result to the requested aspect ratio — one composes at roughly 2:1 and fills the remainder with near-white bars, inside the returned frame. Left alone those bars reach the learner looking like a rendering bug. Detection samples several frames per clip, because the padding is not constant within a clip and propagates into a chained shot through its start frame.
A hard-won lesson: the first fix cropped the white bars and then fitted the clip into 1920×1080, which re-letterboxed the now-2:1 picture with black bars — trading the model's bars for our own. Clips are now scaled to fill the frame. A second attempt also tried to detect dark padding by flatness and cropped 283 pixels off a shot that had none, because in near-black footage a genuine edge column is flat black too; only bright padding is detected, and filling handles the rest.

July 30, 2026

Cortex Chat grows from a chat frontend into a collaborative AI workspace. This release adds personalities — portable SOUL.md personas, including a generator that researches your knowledge base and writes one for you — shared projects where teams converse in the same threads and watch each other's answers stream in live, voice in and out, Single Sign-On against any OpenID Connect identity provider, and a broad chat-UX pass (message actions, feedback, pinning, search, exports, deep links). On the Cortex side it ships one new endpoint, POST /api/llm/completions, so every LLM call the chat makes — including persona writing — runs through the instance's single model configuration, metered and traced like everything else. All Cortex Chat changes are additive; existing deployments migrate their database automatically on first boot.

Cortex Chat: personalities (SOUL.md)

Assistant personas layered over the same retrieval engine — each one is a portable SOUL.md file (identity, purpose, voice, behavioral directives, boundaries), injected server-side on every turn and never exposed to the browser or stored in chat history.

Three tiers — built-ins ship with the app (Research Analyst, Support Assistant, Sales Companion; admins can remove or restore them, and releases can add more), admins curate global or per-group personalities, and every user keeps up to 20 personal ones.
One-click picker on the empty chat screen; each personality brings its own starter questions and optional default collection. The choice is fixed per conversation and survives reloads.
Portability is the point — export any personality as its verbatim SOUL.md; import by pasting, uploading, or from a URL. Files signed by soulweaver are EIP-191 signature-verified on import and carry a verified-signer badge.

Cortex Chat: the personality generator

Describe the assistant you need in one paragraph and the app writes the SOUL.md — grounded in what your knowledge base actually contains.

Research first, write second. The generator runs hybrid searches and plain questions against the knowledge base — every step visible in a live research log, exactly like Deep Research's thinking panel — then writes the file with the instance's own primary model, findings inlined. Drafts stream in, are hand-editable, and support refine rounds ("less slang, more formal") that visibly land.
The architecture is a hard-won lesson: the first implementation sent the entire author meta-prompt as a Deep Research question. On instances with prompt security enabled, the injection classifier — correctly — deflected it with a canned response, every time. The generator now never sends instruction-shaped text as a query: Cortex answers benign research questions, and the writing happens through the new raw completion endpoint. Research asks additionally ride the streaming ask path (the one every chat exercises daily), and the research log reports real failure modes — an HTTP error, a refused answer with a preview, an empty answer — instead of a mysterious "skipped".

Cortex Chat: shared projects & multi-user conversations

A project groups conversations and carries defaults every chat inside it inherits: an optional personality, a collection scope, and instructions — an invisible briefing sent with every question in the project.

Sharing is one modal — a single search field over groups and individual people. Members see every conversation in the project; moving an existing chat into a shared project asks for confirmation first, because it exposes that chat's full history.
Conversations are multi-user by default. Any member continues any project thread; every message is attributed to its author (server-stamped — clients cannot forge it), editing and regenerating are limited to your own turns, and chat administration stays with the creator.
It's live. When a teammate asks a question in a chat you have open, their question appears immediately and the answer streams into your view in real time — opening a chat mid-answer replays what you missed and picks up the live tokens. New chats, moves, deletions, and shares reach every member's sidebar within a second. All of it is Server-Sent Events through the app's own server — no websockets, no external broker, nothing to deploy.
Chats drag & drop between the personal list and project folders, and the active chat now lives in the URL (/?chat=<id>) — refresh keeps your place, back/forward walk your conversations, and chat links are shareable.

Cortex Chat: voice

Dictation and read-aloud, each gated on a pair of env vars pointing at any OpenAI-compatible audio API (a self-hosted speaches/LiteLLM router, Venice, OpenAI). Unset = the buttons don't exist; provider keys stay server-side behind proxy routes.
Read-aloud strips markdown, citations, and tables before synthesis. Dictation records in the browser and transcribes server-side.
A fix worth confessing: the app's own baseline security headers shipped Permissions-Policy: microphone=() — an explicit "this app never uses the microphone" that made every browser refuse dictation silently, with no permission prompt, regardless of user settings. The policy now allows same-origin microphone use, and the mic button explains the remaining real-world failure modes (insecure origin, OS-level block, no device) instead of doing nothing.

Cortex Chat: everyday chat quality

Message actions on hover: copy, regenerate the last answer, edit & resend (forks the conversation at that point), and thumbs up/down feedback — persisted per message and rolled up as a KPI on the admin dashboard.
Regeneration respects conversation memory: the app snapshots the memory blob each turn and replays the pre-turn state, so a regenerated answer doesn't "remember" the answer it replaces.
Sidebar: title search, pinned chats (their own group on top, never reordered by pinning), and per-chat Markdown export with numbered source footnotes.
Starter prompts — admin-curated suggested questions as clickable cards on the empty screen, combined with the selected personality's own starters.
Deep Research is now the default mode for new conversations (admin-overridable, and users still switch per conversation), and a new-chat button lives in the header — no sidebar trip required.

Cortex Chat: Single Sign-On (OIDC)

One SSO adapter that plugs into whatever identity stack a team already runs — Entra ID at an enterprise or a self-hosted Authentik at a five-person shop — with no per-vendor code, because it speaks the one protocol they all share: OpenID Connect with discovery.

Three env vars enable it (OIDC_ISSUER_URL, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET); unset, the feature is invisible — same gating philosophy as email and voice. The flow is Authorization Code + PKCE, and it terminates into the chat's own session model: SSO is simply a second way to mint a session, not a new auth system.
First-time users are JIT-provisioned into a configurable default group; a returning email/password user is linked to their IdP identity automatically — but only when the IdP asserts the email as verified, the guard against the classic SSO account-takeover bug. Linking also revokes every pre-existing session for the account, so an identity squatted before its real owner's first login is evicted the moment they sign in.
OIDC_ONLY=true turns the deployment fully IdP-managed: password login and self-registration disappear. The superadmin stays env-managed and excluded from SSO in both directions — break-glass access never depends on the IdP being up.
Works with Entra ID / Azure AD, ADFS 2016+, Okta, Auth0, Google Workspace, Keycloak, Authentik, Authelia, Zitadel, PocketID; legacy LDAP-only or SAML-only stacks bridge with a thin IdP (Authentik or Dex) in front. Login analytics distinguish SSO from password sign-ins, and the repo ships a preconfigured dev Keycloak for local testing.

Cortex: raw completion endpoint

POST /api/llm/completions — a raw chat completion on the instance's primary model: no retrieval, no prompt security, the caller owns the full prompt. Admin-key-gated precisely because prompt security is bypassed — minted read/manage keys and monetized public keys can never reach it. Streaming responses are OpenAI-compatible chunks over SSE with heartbeats; completions draw from the monthly quota and are unit-metered and Langfuse-traced through the same client factory as every other call. Built for trusted first-party services — Cortex Chat's personality generator is the first consumer — and the base for the app-platform completion capability on the roadmap.
SSE streams survive in-process restarts — the internal shutdown flag that ends active streams gracefully was never cleared on startup, so any in-process restart (test harnesses, embedded reloads) instantly terminated every subsequent stream with a shutdown frame. Real deployments were unaffected; it is fixed at the source regardless.

Verified end to end against a live instance: persona generation grounded in real knowledge-base content through the new endpoint, two-account realtime collaboration (live question + streaming answer on the watching side, mid-stream join with replay), and the full multi-user attribution model. No API or schema breakage anywhere: all chat SSE events are additive, the new endpoint is new surface, and Cortex Chat's SQLite migrations apply automatically on boot. The personality generator requires a Cortex release with the completion endpoint — older backends get a clear message naming the fix, and everything else keeps working.

Release 1.1.1: everything above, installable

Cortex Chat v1.1.0 is tagged and published to GHCR, and this release pins it in stack.json — so the workspace release and the completion endpoint it depends on ship as one tested set. An existing self-host install gets all of it with npx @mocaos/cortex update (chat updates only where it's enabled, of course); nothing else in the stack changed.

July 29, 2026

Cortex can now be installed and operated without cloning anything: npx @mocaos/cortex brings up the full stack — Cortex, Neo4j, a nightly backup sidecar, optionally Cortex Chat, and Caddy for automatic HTTPS in domain mode — from prebuilt, version-pinned images, and owns the lifecycle afterwards. This entry also carries a data-loss fix in graph restore that everyone self-hosting should read: restoring a backup wiped the graph and then failed to replay it, on every instance that had ever been started. Restore was verified end to end against a live stack for the first time as part of this work, which is how the bug surfaced.

Graph restore no longer destroys the graph it was restoring

The failure — restore.sh ran its DETACH DELETE wipe, then aborted while replaying the export, leaving an empty database. The backup was untouched and still checksum-verified, but nothing said so, and the graph was gone. Reproduced on a live stack: a backup holding 2 nodes left the graph at 0.
The cause — apoc.export.cypher.all({ifNotExists: true}) emits IF NOT EXISTS for constraints and plain indexes but not for fulltext ones, so the export contains a bare CREATE FULLTEXT INDEX chunk_content …. The backend creates that schema at startup, so the replay hit "An equivalent index already exists" and rolled back its entire transaction — after the wipe. Restore therefore only ever worked against a schema-less database, which is not the situation anyone restores in.
The fix — the wipe now drops constraints and indexes too, so the replay is authoritative for both the data and the schema it carries. Vector indexes are deliberately left alone: the logical export never carried them, the backend rebuilds them at startup, and dropping them would force a full re-embed of every chunk. Existing backups replay correctly with the fixed script — nothing needs re-taking.
A failed replay now says where you stand — that it wiped, that the graph is empty, that the backup is intact and verified, and that re-running the same command finishes the job. Previously it surfaced as a bare shell error.

Self-hosting: `npx @mocaos/cortex`

One command installs the stack — environment preflight (Docker, Compose v2, architecture, disk, RAM, ports), then two live LLM calls to validate your credentials before anything is written to disk or pulled, then pinned images and a health-gated start. A bad key fails in seconds rather than after a 1.6 GB pull.
Twelve verbs, not just install — status, logs, doctor, start/stop/restart, backup, restore, update, config, uninstall. doctor prints one pasteable diagnostic block.
Releases are pinned as a tested set — each release publishes a stack.json naming the exact backend, frontend, chat, Neo4j and Caddy versions that were tested together, plus the minimum installer those versions require. Images are multi-arch (linux/amd64 and linux/arm64).
It never touches data it did not create — the default project name is cortex, and if Docker already holds cortex_* volumes from an earlier install the wizard stops and offers rename, reuse or abort. Neo4j only applies NEO4J_AUTH when its volume is first created, so silently reusing a project name would write a password Neo4j never adopts. uninstall removes containers, and deletes volumes only behind a confirmation plus a typed phrase.
Secrets are generated by default — all five, unless you choose to set them yourself. .env is written 0600, and no secret is ever written to the installer's own state file.

Embeddings can come from a different provider than chat

Why it matters — the embedding probe is fatal, so a chat provider with no embedding endpoint did not merely warn: it aborted the install with "The embedding endpoint did not answer" and left no way through. Groq serves chat but no embeddings; Ollama users routinely embed elsewhere. Those setups could not be installed at all.
The wizard now asks whether embeddings come from that same provider and, if not, collects a second base URL and key. The model list, the probe, and the rendered .env all follow that second endpoint. --yes reads CORTEX_EMBEDDING_API_BASE and CORTEX_EMBEDDING_API_KEY.
Both or neither — a base URL on its own is rejected rather than paired with the chat provider's key, which would send one vendor's credential to another vendor's endpoint.
The dimension is always measured, never assumed, from whichever endpoint ends up serving embeddings — it is baked into the Neo4j vector index on first use, and changing it later means re-embedding the corpus.

Self-hosting fixes

update now reaches the backup sidecar. It is built locally from the release's ops/backup directory, and Compose never rebuilds an existing image just because its build context changed — so an update that shipped corrected backup or restore scripts wrote them to disk and kept running the old ones. This is exactly how the restore fix above would have failed to reach anyone already installed; update passes --build, and the stack manifest requires the newer installer so the two cannot separate.
The manual update runbook was applying releases only halfway. selfhost/README.md and the self-hosting guide told you to bump the three image tags and run pull && up -d — which re-pulls nothing for the build-only backup service, and never re-fetches the release's Compose files or ops/ scripts at all. Anyone updating by hand therefore got the new backend and frontend images and none of the stack changes, including the restore fix above. Both documents now clone the release tag, re-copy selfhost/ + ops/, and pass --build. Nothing to re-take or re-run beyond following the corrected steps once.
Preflight no longer reports a false disk failure on Docker Desktop. docker info reports DockerRootDir=/var/lib/docker, a path that exists only inside the VM, so df failed on every macOS and Windows host and doctor showed FAIL Disk: could not determine free disk space on a healthy machine. It now measures the first path it can resolve, preferring Docker's own root dir (on Linux that is frequently a separate, smaller partition and the right thing to measure).
GET /health reports the real release version instead of a hardcoded 1.0.0, and a release-time guard keeps it in step with the published version. It is the only thing an operator can query to confirm which release is actually running.

Two `npx` failures that aren't what they look like

could not determine executable to run — there is no install subcommand. npx already means "fetch and run", so npx install @mocaos/cortex asks npm to run a package literally named install. The command is npx @mocaos/cortex.
ENOVERSIONS — No versions available — the package is public and fine. An .npmrc with min-release-age set (increasingly common after the npm supply-chain compromises of recent months) hides every version published inside that window, so a release that is hours old has none left. Keep the setting — it is real protection — and override it for the single command: npx --min-release-age=0 @mocaos/cortex. npm has no way to exempt one package, so it is the flag or the wait.

Cortex Chat is now optional

Off by default. The installer asks whether to include Cortex Chat and defaults to no. Nothing in Cortex references it, so a knowledge base no longer arrives with a second web app attached. --yes opts in with CORTEX_ENABLE_CHAT=true.
One line to change your mind. The chat service sits behind a Compose profile, so COMPOSE_PROFILES=chat in .env plus a restart is the whole switch in localhost mode — the chat port and encryption key are written either way for exactly this reason. Domain mode additionally needs a chat domain and the Caddyfile.chat.template variant, because a Caddy site block with an unset address makes Caddy refuse to load at all.
Existing installs keep their chat. An install that predates the option was running chat, so update treats it as enabled and writes the profile line for you. minInstaller rises to 1.2.2 so an older installer cannot apply this stack and silently drop the service.
Requires installer 1.2.2: npx @mocaos/cortex.

See Self-hosting and handbook chapter 26 for the full picture.

July 28, 2026

Deep Research quality no longer depends on the answer model volunteering to reflect between search rounds. Models that skip that step used to degrade into repetitive query volleys — burning every iteration, drifting off-topic, and answering from a diluted context — while models that reflect naturally converged in a few targeted rounds. The researcher loop now enforces the reflect-then-search rhythm structurally and stops searching once results converge, which brings weaker-reflecting models (measured on Qwen 3.6 35B A3B) up to the research quality previously seen only from naturally reflective ones (Gemma 4 26B), at roughly half the wall-clock. A new default time budget also guarantees the answer starts writing even when the LLM provider is having a slow moment. And once the research was good, the write-up itself had to survive: Deep Research answers on thinking models were being cut off mid-sentence — sometimes coming back empty — because the model's hidden reasoning was spending the very token budget meant for the answer. That budget now goes to the answer alone, and a truncated answer can no longer pass itself off as a finished one.

Deep Research: enforced reflection & convergence

Forced reflection — RESEARCHER_FORCE_REFLECTION (default true): a search round that arrives without a reasoning step (neither a reasoning tool call nor prose alongside the tool calls) is followed by one forced micro-call pinned to the reasoning tool. The model's analysis of what it found — and the specific gap to target next — lands in the conversation and steers the following round. Models that already reflect never trigger it; providers that reject a named tool_choice disarm it gracefully for the run.
Convergence stop — RESEARCHER_NOVELTY_MIN_NEW_RATIO (default 0.35) + RESEARCHER_NOVELTY_STALE_ROUNDS (default 2): the loop tracks how many previously-unseen sources each search round returns; consecutive stale rounds end research early instead of burning the remaining iterations on re-fetched ground (which dilutes the writer context). Set either to 0 to disable.
Novelty-aware iteration notes — each round, the agent is told what share of its last round's sources were new, with a standing directive to reason about the gap or call done — so a well-behaved model stops on its own before the hard stop fires.
Evidence-based stopping criterion — the Deep Research prompt now says to finish as soon as searches stop surfacing new sources (instead of "exhaust all research avenues"), and warns against pivoting to entities that merely appear in results rather than bearing on the question.
Prose reflections stream — models that reflect as text alongside their tool calls (instead of the reasoning tool) now have those thoughts surfaced as thinking events too.

Deep Research: latency guardrails

Research time budget on by default — RESEARCHER_WALL_CLOCK_SECONDS default changed 0 → 120. Healthy gathering finishes in 30-60s; the budget exists for provider queue spikes (measured live: a 2-token completion taking 107s), where it breaks to the writer so the answer always starts instead of stalling silently for minutes. The forced reflection call is skipped when less than 20s of budget remain. Raise the value — or set 0 to restore unlimited — for slow self-hosted inference.
Honest streaming status — a Planning the next research step status event now precedes each researcher LLM call, so during a slow provider call the UI no longer shows a stale "Searching the knowledge base" from the previous round.

Measured on the same knowledge graph before/after (Qwen 3.6 35B A3B, hardest test question): 13 search calls → 8, first answer token 92s → 56s, and the answer went from a generic non-answer to a specific, fully cited list — matching the Gemma 4 baseline at half its total wall-clock. No API or schema breakage: all events are additive, and every behavior change is env-gated with the flags above.

Deep Research: answers no longer cut off mid-sentence

On thinking models, the hidden reasoning trace is billed against the same output-token budget as the visible answer. The Deep Research writer ran with reasoning enabled, so a long report could spend most of its 4,000-token allowance thinking and then stop mid-word — with no error, no warning, and a completed-looking response in the UI. Measured on Qwen 3.6 35B A3B with a 15-source research context, the trace ran 2,700-4,300 tokens against that cap; in the worst case it consumed the entire budget and the user received an empty answer.

Writer reasoning suppressed — the final writer call now runs with reasoning off in Deep Research too (it already did in chat). The writer only composes prose from context the researcher already gathered, so hidden reasoning bought nothing while consuming the answer's budget. The same 15-source report now completes in ~1,300 output tokens — a 3× margin under the cap — and the first token lands sooner. The researcher loop is unchanged and still reasons in Deep Research; only the write-up step changed.
Larger answer budget — WRITER_MAX_TOKENS_QUALITY default 4000 → 8000, giving long reports room and protecting against providers that ignore the disable-thinking flag. WRITER_MAX_TOKENS_SPEED is unchanged at 1200.
Truncation is never silent — if an answer does hit the output limit, the response now ends with a visible note saying it was cut short, and the backend logs a warning naming the mode, the cap, and which env var to raise. Previously the stream simply ended and the turn was marked complete, making a mid-word cut indistinguishable from a finished answer in both the UI and the logs.

Verified end-to-end against a live instance: the question that previously broke off mid-sentence now returns a complete, fully cited answer over 15 sources, and an artificially lowered cap produces the labeled-truncation path instead of a silent cut. No API or schema breakage — the truncation note arrives as ordinary answer content, and both limits stay env-overridable.

The agent no longer reads a connected repo it has already ingested

On instances with a git repository connected, the research agent was being handed its git_repo tool on every question — including read-only connections, whose entire purpose is to ingest the repo into the knowledge base. The result, seen live during an ordinary capability question: four consecutive read_file calls guessing at paths in the connected repo. The tool has no way to list or browse files, so a path the agent hasn't read somewhere is a guess; each guess costs a research iteration and usually returns an error, and even a successful read can't be cited in the answer — while the same content was already sitting in the knowledge base as a citable source.

RESEARCHER_GIT_TOOL (default auto) — the tool is now offered only when a repository is connected read/write, which is when its reason to exist (proposing pull requests) actually applies. This matches what the documentation has always described. Set always for the previous behavior on read-only connections, or off to withhold it entirely; repository ingestion is unaffected in every case.
The repository is named and scoped in the prompt — when the tool is offered, the agent is told which repository is connected, that its files are already indexed and searchable, that browsing via git_repo is not possible, and the two cases that justify using it: proposing a change, or re-reading one file whose exact path it already knows. The tool's own description carries the same framing.
Read-only instances get the latency back — a repo read also disabled the fast chat path (its output never reaches the writer) and ran sequentially, outside the parallel-search batch. Removing the spurious calls removes both costs.

Answering from the graph was always the better path for this: it returns the same file contents with a citation, in one search, alongside everything else the question touches.

July 23, 2026

Cortex Chat — the standalone multi-tenant chat frontend — now has first-class documentation. Until now it was a single row in the ecosystem table; operators had to read the app's repository to understand what it does and how to deploy it against an instance.

Documentation: Cortex Chat

New docs page — Cortex Chat explains the app end to end: the chat experience (streaming Ask AI, Deep Research, inline citations, conversation memory, cross-device history, support link, default chat mode), the multi-tenant model (roles, groups, self-registration with approval, password reset over SMTP, bulk xlsx user import, content roles), the admin library console (Documents / Processing / Collections tabs that drive the backend's knowledge-graph pipeline from inside the chat app), the analytics dashboard and <cortexchatanalytics> injection, the one-admin-key/minted-scoped-keys security design, resilience behaviors, and full setup — a complete environment reference (required, optional SMTP/registration/GlitchTip knobs, deprecated aliases), Docker/Coolify deployment, and a first-run walkthrough.
New handbook chapter — Chapter 25 covers the same ground for handbook readers, and Chapter 24 (Apps) now distinguishes in-instance apps from standalone apps like Cortex Chat that run outside the instance and connect via the REST API with scoped keys.
No product changes — documentation only.

July 22, 2026

Web Import stopped producing a scatter of one-off pages. A crawl now folds every page from the same site into one document per domain, completes the moment those documents are staged (processing continues in the background, where you can watch it), and names each document by its domain instead of whatever title the crawler guessed. Crawls driven through a front-end can also carry a contributor tag, so a site imported via the MOCA Library is attributed like any other community submission.

Web Import: one document per site

Per-domain aggregation — all URLs from the same host in a job become a single markdown document (example.com.md, titled # example.com) with one ## section per page, each keeping its own > Source: line. Retrieval and the knowledge graph read a site's subpages as one related work — an artist named only on an About page now ties to the exhibition and press pages that never repeat the name. A job spanning several domains produces one document per domain.
Import completes on staging, not on full processing — the job finishes (and the modal's completion screen appears) as soon as the documents are added, then hands the extract/embed/graph pass to a normal batch_processing task. The Documents page shows them processing in the background — no more watching a progress bar through the whole pipeline for a large crawl.
Documents named by domain — crawl4ai often falls back to the URL path for a page title (e.g. o.html), which used to become the document's filename and every citation. The aggregated document is now titled and filed by its domain, so sources read clearly.
Contributor attribution — POST /api/web-import accepts an optional submitted_by; when set, each document's source becomes crawl:<domain> community:<id> (mirroring the upload path's community:<addr>), so crawls submitted through a front-end are attributed to the submitter. Omitted, the source stays crawl:<domain>.

July 21, 2026

The app platform learned to speak to cloud storage: the task engine gained WebDAV listings, binary file transfer, and runtime OAuth credentials — the pieces a folder-sync app needs — and the first wave ships with it. Nextcloud Sync and Dropbox Sync are live in the registry (now browseable on the web at registry.cortex.eco), and Paperless Sync 0.4.0 switches from re-rendering OCR text to ingesting the actual documents. Shipping these against real accounts also surfaced fixes that land here: a config-masking gap after app upgrades, clipboard access inside the app sandbox, and a Knowledge Graph page that kept offering Regenerate Graph while a sync-driven build was already running.

Cloud-sync task engine extensions

The declarative task DSL (platform apps' server-side step-queues) gained the vocabulary for cloud-storage sync — all behind the existing security gates (host allowlist, SSRF guard, endpoint allowlist), with the manifest schema unchanged:

webdav step — a PROPFIND folder listing through the same gates as http steps, with the multistatus XML parsed server-side into plain items (href, name, etag, lastModified, size, contentType, isDir, fileId); depth 0/1/infinity and a files/dirs filter. ETags come normalized, ready to diff against stored state — the change-detection primitive for WebDAV sync. The interactive platform http envelope accepts PROPFIND too, so app UIs can drive folder browsers.
Binary passthrough uploads — cortex steps accept multipart: {fromUrl: …}: the instance fetches the file server-side and streams the bytes straight into /api/upload. PDFs, Office files, and images survive intact, and the bytes never enter the task context or a browser. Capped by the existing 20 MB transfer limit.
Dynamic per-request auth — http/webdav/fromUrl steps accept auth: {bearer: …} (or basic) resolved from the run context, so a task can refresh an OAuth token at the start of a run and use it on every call. Config secrets remain untemplatable, so static credentials can't leak through this path.
Multi-listing fan-out — items.fromEach concatenates several listing steps (one per synced folder) into one work pool, and a new ext template filter enables extension-based type filtering for APIs whose listings carry no MIME type.
Redirect-following binary fetches — fromUrl accepts followRedirect: true for APIs that serve file bytes via a 302 to a pre-authenticated URL on a variable host (Microsoft Graph /content). Exactly one hop, https + public targets only, and no request headers are forwarded to the redirect target — safe here because the bytes stream straight into the instance's own upload, never into app-readable context.
Delta-cursor plumbing — paginate output gains last (the final page's body, where cursor APIs put their continuation token), and reference paths handle keys that contain dots (OData's @odata.deltaLink). WebDAV listings now always carry a fileId (the server's stable id, or a deterministic href hash on servers without one).
Google service-account auth without OAuth — auth_header templates may carry google_sa_token(VAR, scopes): the instance mints short-lived access tokens from a service-account JSON key (config secret) server-side — RS256-signed, exchanged at Google's pinned token endpoint, cached until near-expiry. The Google Drive Sync app builds on this: users share folders with the service account's email; no consent screens, no app verification, no tokens in tasks or browsers.
Microsoft app-only auth for SharePoint — the sibling ms_graph_token(TENANT, CLIENT_ID, SECRET) transform mints app-only Graph tokens via the client-credentials grant (endpoint pinned to Microsoft, tenant segment sanitized, cached). Pairs with the Sites.Selected permission: the SharePoint Sync app sees only sites an admin explicitly grants — no user sign-in, no consent screens, no tenant-wide access.
Basic auth without hand-encoding — auth_header templates can now reference any config var and wrap parts in base64(...), evaluated server-side: Authorization: Basic base64(NC_USER:NC_APP_PASSWORD). Users paste a plain login and app password; nobody base64-encodes credentials by hand (a field-tested error magnet).

Cloud-sync apps in the registry

The registry catalog is browseable on the web at registry.cortex.eco — the same checksum-pinned listings the in-instance Browse Registry panel installs from.

Nextcloud Sync 0.1.0 — pick folders in a built-in WebDAV browser (with a live "~N files across M folders will sync" projection), and a scheduled task keeps them in the graph: ETag-based change detection, edited files replace their previous document instead of duplicating, sub-folders included. Auth is a plain Nextcloud app password; Basic auth is assembled server-side.
Dropbox Sync 0.1.0 — OAuth 2 + PKCE with no app secret and no redirect server (copy/paste code flow, granted scopes verified at connect time with actionable errors), then per-folder delta cursors (list_folder/continue) drive incremental sync. The refresh token lives in the app's private storage; access tokens are minted per run.
Paperless Sync 0.4.0 — now ships the full documents: each sync downloads the archived PDF (or original upload) server-side and streams it into ingestion, so Docling parses the real document — layout, tables, images — instead of a markdown re-render of paperless's OCR text. Requires the platform extensions above (minVersion 2.1.0).

Fixes from shipping against real accounts

Orphaned app config keys no longer surface after upgrades — app upgrades preserve config.json, but a var that was secret-typed in the old manifest lost its masking the moment a new manifest stopped declaring it: the stale secret came back as plaintext in both the admin config view and the app-facing platform/config endpoint. Both now expose only vars declared by the current manifest (regression-tested).
Copy buttons work inside the app sandbox — the launcher and share-shell iframes now delegate clipboard-write (the async Clipboard API is permission-gated in the sandbox's opaque origin), so in-app copy buttons (auth generators, OAuth links) actually copy.
Queued documents read as Queued, not Processing — a burst of API ingests (e.g. a sync app uploading hundreds of files) is capped at BATCH_PROCESSING_CONCURRENCY, but every waiting document displayed as Processing — 342 uploads looked like 342 concurrent pipelines. Waiting documents now carry an additive processing_queued flag: the Knowledge Graph Step 1 tile gains a Queued count, the Documents page shows a Queued badge, and Processing means exactly the documents actually holding a slot. No behavior change to the cap itself.
Knowledge Graph page: Abort during any live build — the top CTA kept showing Regenerate Graph while ingestion started elsewhere (a sync app's uploads, another tab) was running, because per-document processing creates no batch task record and the page only checked for running tasks once on mount. The CTA now also keys on document-derived activity (the same signal as the Step 1 tile), running-task detection is continuous, and an idle 15-second refresh heartbeat notices externally-started ingestion in the first place. Aborting genuinely stops per-document processing — documents are kept.
No API or schema breakage: all task-DSL additions are opt-in step fields, the manifest schema is unchanged (all three registry/template/instance validators stay in lockstep), and instances without ENABLE_APPS are unaffected.

July 20, 2026

Restarting the backend or the LLM endpoint mid-processing used to throw away everything a large document had already paid for — conversion, embeddings, and hours of extraction — and start over from the first chunk. Processing is now checkpointed at sub-document granularity and pauses gracefully through LLM endpoint outages, so a litellm restart at 95% of a 3,000-chunk book costs the remaining 5%, not the whole document. Separately, two silent login foot-guns reported by self-hosters are fixed: the session cookie's Secure flag gets a runtime switch for plain-HTTP setups, and a missing ADMIN_EMAIL now fails loudly instead of silently substituting a default. This release also introduces Apps — self-contained web apps that install into an instance from a zip and run sandboxed against a proxied, least-privilege slice of the Cortex API.

Ingest checkpoint & resume

Stored work is reused, not redone — each run records a fingerprint (file hash + extraction config). When an interrupted or reprocessed document still matches it, the stored chunks and embeddings are loaded from the database instead of re-running document conversion and a full re-embed. Reprocessing keeps resumable chunks instead of deleting them.
Entity extraction checkpoints per batch — every settled extraction batch persists its entities immediately and advances a chunk-range watermark on the document. A resumed run skips completed ranges, and the progress bar continues where it stopped (e.g. 2850/3000 chunks) instead of resetting.
Per-chunk relationship extraction checkpoints too — chunks whose relationship pass settled are flagged and skipped on resume (a chunk can legitimately yield zero relationships, so completion is tracked explicitly).
Controlled by ENABLE_INGEST_RESUME (default true); set false for the legacy restart-from-zero behavior. Checkpoints are invalidated automatically when the file bytes or extraction config change.

LLM endpoint outage handling

Batches are no longer silently dropped when the endpoint is down — connection failures (endpoint restart, gateway 502/503) previously fell into the generic error path, which discarded every remaining batch within seconds and let the document "complete" with silent entity loss. They are now classified separately: the affected batch is requeued and re-probed with backoff until the endpoint returns.
Bounded by LLM_OUTAGE_MAX_WAIT_SECONDS (default 900) — if the endpoint stays unreachable past the budget, the document is marked failed with its checkpoint intact, and the error message says so: reprocessing resumes from the checkpoint once the endpoint is back.
The document's progress message shows the outage live (LLM endpoint unreachable — retrying in 20s...), and run telemetry gains a connection_retries counter.

Paused & Interrupted states in the UI

The outage and checkpoint machinery is now fully visible in the frontend, so operators see exactly what a document is doing instead of a generic spinner or failure:

Paused badge (amber) — a processing document whose LLM endpoint is temporarily unreachable shows a pause badge, an amber status line with the wait details ("LLM endpoint unreachable — retrying automatically (waited 40s of 900s)"), and an amber progress bar in the phase stepper — on both the Documents page and Knowledge Graph Step 1. Processing continues automatically; no action needed.
Interrupted badge (amber) — a document that failed because the outage outlasted the wait budget is labeled Interrupted instead of Failed, with an inline note that progress is checkpointed; its reprocess button becomes Resume from checkpoint.
Backed by additive document fields (processing_paused, paused_reason, resume_available) returned by the document endpoints — not a new processing status, so filters, integrations, and API clients are unaffected.
No API or schema breakage: new document/chunk properties are additive, and behavior with ENABLE_INGEST_RESUME=false matches the previous release.

Serving the dashboard over plain HTTP (e.g. a LAN self-host without TLS termination) used to break login silently: the session cookie was sent with the Secure flag hard-wired at image-build time, browsers drop Secure cookies over HTTP per spec, and the login form simply bounced back with no error. There was no runtime override — only rebuilding the image could change it.

New SESSION_COOKIE_SECURE env var (true/false, read at container runtime) — overrides the session cookie's Secure flag; set false for TLS-less HTTP setups, leave unset behind HTTPS. Default unchanged (Secure in production builds).
Mapped through the frontend service in all compose files (local, prod, Coolify, Dokploy) — a .env edit plus container restart is enough; no image rebuild, no compose patches.
Documented in the configuration reference and the authentication guide.
No API or schema breakage; behavior with the variable unset is identical to the previous release.

Missing `ADMIN_EMAIL` fails loudly instead of silently defaulting

When the frontend container started without ADMIN_EMAIL (env file pointed at the wrong path, secrets not wired into a standalone deployment), the code silently substituted admin@example.com — so logging in with the correct credentials answered "Invalid email or password", indistinguishable from a typo'd password, and the real cause (env loading) left no trace.

No more silent default — a missing/empty ADMIN_EMAIL or ADMIN_PASSWORD now answers "Admin authentication not configured" at login (the same actionable error the password path already used) and logs a startup warning naming exactly which variable is missing.
The compose files are unaffected: they already supply admin@example.com as a visible env-level fallback (${ADMIN_EMAIL:-admin@example.com}). Correctly configured deployments see no change.

Apps — in-instance app hosting

Cortex can now host self-contained web apps built from the Cortex App Template. An admin uploads a packaged app (a zip of a static bundle plus a manifest) in Settings → Apps; it is served sandboxed under /apps/{id}/ and reaches the Cortex API only through a proxy that enforces the manifest's endpoint allowlist. Off by default (ENABLE_APPS=true). See the Apps feature guide.

Least-privilege by construction — each install mints a dedicated API key scoped to exactly what the manifest declares (read or read-write, optionally restricted to chosen collections at install time via a collection picker). The browser never holds a real key: apps receive a short-lived, auto-renewing token, and the proxy attaches the minted key server-side. Uninstalling revokes the key.
Sandboxed — apps run in an iframe without allow-same-origin (opaque origin, no cookie access), each served with a per-app Content-Security-Policy. Config values (if the app declares any) use the same encrypted-secret storage as skills.
Platform capabilities — a type: "platform" app can make external API calls through a server-side http capability: the call executes from the backend with credentials injected from the app's encrypted config, so the external service needs no CORS configuration and secrets never reach the browser.
Background tasks that survive a closed tab — platform apps can declare a tasks capability and submit declarative step-queues: JSON programs of http / cortex / llm / store / template steps that Cortex executes server-side in its async loop, with per-item error isolation, pause/resume/cancel/retry controls, and automatic resume after a server restart. A task with "schedule": {"everyMinutes": N} re-runs on its own — turning an integration app into a true sync daemon (e.g. pulling newly added paperless-ngx documents into Cortex every hour with no browser open). Every step passes the same security gates as the interactive paths: http steps are host-allowlisted and SSRF-guarded, cortex steps are checked against the manifest's endpoint allowlist, and llm steps run the instance's configured model, metered like any other completion, with built-in chunking and output-validation policies (length-ratio + word-overlap guards) for long-text workloads.
Per-app storage — a storage capability gives an app its own quota-capped, strictly namespaced key/value store (per-app SQLite in the apps volume), usable both from the app's UI and from task steps; it also powers the task engine's built-in fan-out deduplication (skipIfStored). Storage and task state survive app upgrades — reinstalling a new version keeps dedup keys, cursors, and schedules intact.
Host-scoped credentials for multi-host apps — a config var may declare auth_host (literal hostname or ${CONFIG_VAR} reference) so its auth_header is injected only on calls to that host. Without it, an app declaring several external hosts would send every credential to all of them (e.g. a Venice API key on YouTube calls). Single-host apps need no change. The platform http envelope also accepts extra request headers now (auth/framing names stripped server-side; app-authored Cookie values allowed — needed for e.g. YouTube's EU consent bypass).
Share links — for apps that allow it, an admin mints revocable share links (/a/{id}?g=…) so people who are not Cortex users can open an app without logging in; a share visitor's token validates only at that app's proxy and never grants access to the rest of Cortex. Share-link visitors are read-only for platform state: storage writes and task control require an owner or editor token.
New environment variables (all optional; documented in the configuration reference): ENABLE_APPS (default false), APPS_DIR (default .agents/apps, persisted via the new apps_data volume), APP_MAX_PACKAGE_MB (default 50), APP_TOKEN_TTL_SECONDS (default 900), APP_PROXY_UPSTREAM, APP_HTTP_TIMEOUT (default 30); for storage and tasks: APP_STORAGE_MAX_MB (default 50), APP_STORAGE_MAX_VALUE_KB (default 1024), APP_TASK_MAX_ITEMS (default 2000), APP_TASK_MAX_CONCURRENCY (default 4), APP_TASKS_GLOBAL_CONCURRENCY (default 8), APP_TASK_MIN_SCHEDULE_MINUTES (default 15), APP_TASK_LLM_CALLS_PER_RUN (default 500), APP_TASK_STEP_OUTPUT_MAX_KB (default 2048), APP_TASK_MAX_PER_APP (default 50).
Task oversight in the admin panel — each installed app's expanded view now shows its background tasks (status, schedule, progress counts, errors) with pause/resume/cancel/retry/run-now controls, so a failing scheduled sync is visible without opening the app.
Install from the registry — a new Browse Registry panel (Settings → Apps) lists the public Cortex Registry catalog with each app's declared scope, endpoints, and capabilities. Installs are checksum-verified end to end: the instance downloads the release artifact and verifies its catalog-pinned sha256 (and exact size) before unpacking anything; updates surface in place and preserve config, keys, share links, storage, and schedules. Catalog location: APP_REGISTRY_URL (default: the official registry; empty disables browsing). Publishing an app = a GitHub release + one PR — the registry's CI re-downloads and re-verifies every artifact.
No API or schema breakage: the subsystem is filesystem-backed with no Neo4j schema changes, and every route is inert unless ENABLE_APPS=true.

July 17, 2026

Cortex can now sell pay-per-query access to its retrieval endpoints via the open x402 payment standard. Free member keys keep working exactly as before; in parallel, admins can mint monetized public keys that agents pay per query in stablecoins, with revenue flowing straight to a wallet the owner controls — a way to earn on domain-specific knowledge and subsidize members' inference.

x402 Payments

One env var, everything else at runtime — X402_ENABLED=true (default false) unlocks a new Settings → x402 Payments section; recipient wallet, facilitator URL, network, and asset are configured there, stored in Neo4j (survive redeploys, excluded from library export and system reset). Facilitator auth headers are encrypted at rest via ENCRYPTION_KEY.
Verification before money moves — a one-click verify suite checks wallet address format (EIP-55 checksum / Solana base58), facilitator reachability, and scheme+network support, reported check-by-check. Priced keys can only be created — and paid requests only served — while the saved config is verified; any payment-relevant change re-requires verification.
Vendor-agnostic by design — Cortex implements the x402 v2 facilitator interface (/verify, /settle, /supported) over plain HTTP: any spec-compliant facilitator works, no vendor SDK, no chain RPC, no private keys on the server.
Monetized public keys are locked down at the API level — read-only by construction (a price can never be combined with manage), restricted to the four retrieval endpoints (/api/search + the Ask AI endpoints; raw document/file access and stats are off-limits), and still collection-scopable — sell exactly the slice of knowledge you choose. New cortex_pub_ key prefix.
Settle-before-serve payment flow — unpaid requests get HTTP 402 + PAYMENT-REQUIRED (base64 requirements per the x402 HTTP transport); paid requests are verified and settled through the facilitator before the query runs, with the settlement receipt returned in PAYMENT-RESPONSE. Facilitator outages fail closed (503, never served free).
Earnings dashboard — settled totals overall and per key in the admin section, every payment recorded with its on-chain transaction hash for independent auditability.
Honest analytics for paid keys — HTTP 402 challenges are excluded from per-key error-rate tracking: a challenge is the normal first leg of every x402 handshake, not a failure.
Two-tier pricing: deep research costs more — each monetized key carries a research_multiplier (default 10, set at key creation with a live dual-price preview): agentic deep-research queries are quoted at price × multiplier in the 402 challenge itself, so agents see the true cost before signing; quick asks and search stay at the flat rate. 0 disables deep research on a key entirely (403). Requests that would fail validation are rejected before payment — a payer can never settle for a guaranteed 422.
Paid queries still consume MAX_QUERIES_PER_MONTH units — the quota meters inference cost regardless of who pays.
No API or schema breakage: existing keys, clients, and flows are unaffected while the flag is off (default) and for all non-priced keys when it is on.

July 13, 2026

The duplicate-entity scan behind Manage → Deduplicate was the least reliable surface on large graphs: requests timed out at the proxy while the server kept computing, every retry stacked another CPU-heavy scan on top, and on graphs beyond roughly 20k entities the scan could exhaust the container's memory and crash outright. The scan has been rebuilt end to end.

Memory-bounded, faster matching

The similarity computation no longer materializes full n×n score matrices (4 GB at 32k entities — an instant out-of-memory in a standard 4 GiB container). Matching now runs in fixed-size row blocks (~128 MB peak) over the upper triangle only, so peak memory stays flat no matter how many entities the graph holds and redundant pair comparisons are halved. A 40k-entity graph scans in ~35 seconds well within a 4 GiB container.
Scan results are now fully deterministic: repeated scans over an unchanged graph return identical groups and canonical suggestions.
Grouping semantics are unchanged (typo matching, word reordering, Person-aware name-prefix gating, star clustering) and locked by a new parity test suite.

No more proxy timeouts: single-flight scan with progress

One scan runs at a time; identical requests join the running scan instead of stacking new ones — a stampede of retries can no longer saturate the CPU.
If a scan outlasts DEDUP_SCAN_WAIT_SECONDS (default 25s, deliberately below typical proxy read timeouts), GET /api/entities/duplicates answers 202 {"status": "running", "progress": 0.42} and the client polls the same URL until it returns "status": "complete". The web UI does this automatically and shows live progress on the scan button.
Completed results are cached server-side for DEDUP_SCAN_CACHE_TTL_SECONDS (default 600s) per parameter set — reopening the page is instant. Entity merges invalidate the cache; refresh=true forces a fresh scan.

Ingestion injection scan is now experimental & opt-in

The ingestion-time prompt-injection scan is reclassified as experimental and disabled by default. On a default instance the feature is completely absent: no scan runs at ingestion (not even the free heuristic), the toggle no longer appears under Admin → System Configuration → Features & Security, and the "Injection Flagged" document filter is only offered when at least one document is actually flagged.

New env var ENABLE_INGESTION_INJECTION_SCAN (default false) — the master flag that activates the feature. Once set, behavior is unchanged from before: heuristic + optional LLM classifier, flag-only (never blocks), with the classifier admin-toggleable at runtime via the existing INGESTION_INJECTION_SCAN default. With the master flag off, PATCH /api/admin/config rejects the ingestion_injection_scan field with 400.
Query-time defenses (regex detection, Prompt Guard, output filtering, and untrusted-content fencing of retrieved chunks) are unaffected and remain on by default.

July 11, 2026

Security hardening pass

A round of defense-in-depth tightenings across configuration defaults, outbound requests, and access scoping. Correctly-configured deployments are unaffected; the changes mainly raise the floor for the default and misconfigured cases. No API changes.

Production hardening is now the default on the deploy paths. The docker-compose.prod.yml, coolify, and dokploy compose files set ENVIRONMENT=${ENVIRONMENT:-production} (still overridable), so a standard deploy runs with secret enforcement, response-error sanitization, and API docs disabled without an extra step.
Stronger secret hygiene at startup. .env.recommended now ships clearly marked placeholders (generate real values with openssl rand -hex 32) rather than sample values, and startup — both the backend validator and the Next.js frontend — refuses known-placeholder or too-short signing secrets in production. Empty admin credentials still fail closed as before.
Outbound-request safety (SSRF guard). A shared guard (services/ssrf_guard.py) validates the target of requests influenced by user or agent input — the skill http_request tool, Web Import, and the git connector — resolving the host and blocking loopback/link-local/metadata addresses (and, for the agent tool, private ranges by default), re-checked on each redirect hop. Per-host allowlists and SKILL_HTTP_ALLOW_PRIVATE / WEB_IMPORT_ALLOW_PRIVATE cover legitimate internal targets; self-hosted git on a private network keeps working.
Tighter collection scoping. Restricted API keys are now checked against a document's source collection (not just the target) when moving or adding documents, so a key stays within its granted collections. No effect on admin/unrestricted keys.
Safer library import. Imported-archive skill identifiers are validated before any files are written, keeping restored content inside the skills directory.
Rate-limit and frontend hardening. The opt-in rate limiter now keys only on trusted identity with an always-applied per-IP bucket; the login redirect is restricted to same-site paths; and the frontend sends baseline security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy).

Generate Graph chain survives restarts

Fixed: a server restart or redeploy mid-run no longer strands the Generate Graph pipeline after Step 1. The 3-step chain (extraction → relationship analysis → community detection) is orchestrated in-process, so a restart used to lose the remaining chain — the startup auto-resume restarted Step 1 without it and the pipeline visibly stopped once Step 1 finished. Every pipeline task now persists its resume parameters (remaining chain, concurrency, scope) alongside its task record; on boot, Cortex resumes the interrupted step with its chain — Step 1 continues into Steps 2 and 3, an interrupted Step 2 or 3 restarts directly.
Also covered: a run killed between queueing and its first document (documents sat at pending forever with nothing to resume them) and the case where an upload was mid-processing while a Step 2/3 task was interrupted (Step 1 resumes first, then the interrupted step). All resume paths remain quota-guarded behind AUTO_RESUME_PENDING_ON_STARTUP.

July 10, 2026

Two days of live extraction-tuning (July 9-10) distilled into shipped defaults plus a new low-friction onboarding path — a fresh install now runs the production-validated configuration without setting a single tuning knob.

Tuned defaults — the validated values now ship in code

GRAPH_EXTRACTION_MAX_CONTEXT / EXTRACTION_MAX_OUTPUT_TOKENS now default to 16000 / 16000 (were 0-inherit-clamped-at-48k and 12000). The pairing is production-validated: zero truncated batches, zero dropped entities across multi-day live runs including entity-dense documents (indexes, bibliographies — dense-batch output peaked ~13.4k under the 16k cap, versus ~25% truncation at the old settings). Explicit values are still honored unclamped; set 0 explicitly to restore the inherit chain.
OPENAI_MAX_CONTEXT defaults to 256000 (was 32768), matching the recommended primary (Gemma4 26B A4B) — it's the retrieval agent's working context and deliberately does not flow into extraction.
EMBEDDING_MODEL defaults to text-embedding-3-small (bare id, was the LiteLLM-style openai/text-embedding-3-small) — the string is sent verbatim to the embeddings API, so the bare id is correct for direct OpenAI and Venice-class gateways alike.
Extraction is decode-bound — output scales with input, so the context budget is a batch-size / graph-density dial sized to provider decode speed, not the model's window. The terse-description extraction prompt (short ~12-word entity descriptions; enrichment restores depth) is what bounds output-per-entity and makes the matched 16k/16k ceiling hold on dense docs.
Default-config deployments are flagged for a graph refresh after upgrading (the extraction fingerprint changes — intended, the new defaults are the tested values). Deployments that pin these vars in their env see no change.

Zero-config onboarding: `.env.recommended`

.env.recommended rewritten as the recommended starting point: cp .env.recommended .env, fill the secrets block + one API key, done. Secrets first, then the bench-validated model stack — Gemma4 26B A4B (google-gemma-4-26b-a4b-it) as the primary agent model, Qwen3.6 27B (qwen3-6-27b) for knowledge-graph generation and vision, text-embedding-3-small/1536 embeddings — with commented *_API_BASE/*_API_KEY escape hatches per tier (any model from any OpenAI-compatible API works) and explanatory notes at the bottom.
ENCRYPTION_KEY joins the standard secrets block — it encrypts git connector PATs and secret-typed skill config at rest (empty = plaintext with a startup warning; setting it later auto-encrypts on the next boot). Generation + rotation guidance included.
The docs Quickstart, the setup skill (including the autonomous agent-install flow, which now generates an ENCRYPTION_KEY automatically), and the hermes cortex.sh setup helper all lead with this path and write the same stack — one consistent story across every install door.
Docs accuracy sweep: .env.example audited programmatically against config.py (every stated default verified), stale 24000/12000 budget advice removed everywhere, EMBEDDING_SEND_DIMENSIONS docs now name fixed-dimension models (e.g. Qwen/Qwen3-VL-2B-Instruct) that require false.

Self-healing image analysis on restart

A restart used to strand image analysis permanently: it runs as in-process background futures after a document's text pipeline completes, so the document was already completed when the futures died — the counters froze at current < total, Step 1 showed "finishing image analysis" forever, and the startup recovery (which only matches in-flight statuses) never saw it. Observed in the field: 34 documents stuck at 8/2023 images with zero LLM traffic.

On boot, Cortex now detects completed documents with unfinished image analysis and resumes them automatically (AUTO_RESUME_IMAGE_ANALYSIS, default true; quota-guarded, visible as an image_analysis_resume task).
Already-paid work is never redone: images are re-extracted via local Docling re-conversion (CPU only — no LLM cost), and only images whose analysis chunk isn't stored yet are sent to the vision model. Text chunks, entities, and relationships are untouched.
Irrecoverable documents (source file deleted, vision model no longer configured, re-conversion finds no images) get their counters closed with an explanatory message instead of reading as in-flight forever; conversion failures leave them queued for retry on the next startup.

Abort Graph Generation

New "Abort Generation" control on the Knowledge Graph (/extract) page. A running generation can now be stopped from the UI — and the abort actually stops the work: it cancels the reprocess batch and the Step-2/3 chain, kills in-flight document conversions (a docling worker subprocess previously survived cancellation and kept pinning CPU/RAM), and resets stranded documents back to pending so on-page counts reflect reality.
The button appears whenever any pipeline task is running; when idle, the primary action reads Generate Graph (no entities yet) or Regenerate Graph (entities exist).

Extraction transport overhaul — no more retry storms

Run telemetry from a 7-hour field ingest showed ~40% of extraction calls "timing out at 361s" — actually three 120s SDK attempts silently re-sending the same prompt into a saturated endpoint.

LLM_REQUEST_TIMEOUT_SECONDS now truly governs extraction — tier clients no longer hardcode 120s; one attempt gets the full window.
No SDK retries where splitting is the retry — entity batch and per-chunk relationship calls run one-shot (max_retries=0); timeouts go straight to split-retry, 429s requeue with bounded exponential backoff.
Timeout-adaptive batch sizing — a proven-surviving batch size is remembered per extraction config, so later documents skip the split cascade (the field log showed 15 planned batches ballooning to 77 calls / 4.8 hours).

Extraction observability

Batch-start progress (Finding entities: 120/741 chunks (batch 8/22)…) so multi-minute calls no longer read as a hang; the denominator grows as batches split, making retry churn visible as forward motion.
Per-batch telemetry (tokens, duration, finish reason), one-shot budget/timeout diagnoses naming the exact knobs to turn, a run health summary persisted on the document (extraction_stats), and repetition-loop detection for model degeneration floods (observed: 288 "entities" from one 5-sentence chunk).

Branding

Footer now reads "Built by MOCA" with the Museum of Crypto Art logo and Twitter link; the README carries the same links with a theme-aware logo.

July 8, 2026

A deep reliability pass over the entire document-ingestion pipeline, driven by live debugging of large-PDF ingestion (multi-hundred-page books) against a hosted inference provider: conversion no longer runs out of memory, embedding and extraction failures self-heal instead of silently degrading the graph, a new compact extraction output format cuts structured-output overhead, and the UI finally shows truthfully what the pipeline is doing at every stage. Plus full-stack error tracking via GlitchTip. No API or schema breakage.

Large-document conversion & memory

Docling worker heap release — the conversion worker returns freed page-image memory to the OS after every page chunk (gc + malloc_trim). Previously the allocator retained it, and conversions of 300+ page PDFs were OOM-killed inside the container's memory limit. Books of any length now convert reliably within a bounded footprint.
Live conversion progress — the worker's per-page-chunk progress streams into the document's progress state (Converting document: page 150/320...) instead of surfacing only after the process exits; a queued document now says Waiting for a conversion slot... instead of falsely claiming to convert.
pypdf's per-object warnings on malformed PDFs no longer flood the logs.

Embedding input guards & recovery

Token-accurate input capping — oversized chunks are sub-split against real token counts (tiktoken, char-heuristic fallback) instead of a chars-per-token approximation that undercounted punctuation-dense text (tables of contents, indexes, bibliographies). Zero content loss; EMBEDDING_MAX_INPUT_TOKENS now defaults to 5400 to stay under gateway-side token validators (see the new "Provider Notes: Venice" section in Configuration).
Per-chunk recovery — a batched embed request rejected by the provider no longer strips embeddings from every chunk in the batch: affected chunks are re-embedded individually (rate-limit aware, with automatic halving for inputs the provider still refuses). Image-derived chunks get the same treatment.
Empty/whitespace-only chunks are dropped before embedding, and a failed embed pass is retried once before failing the document.

Extraction self-healing & budgets

Timeout and truncation split-retry — an extraction batch that times out or overflows its output budget is split in half and retried instead of being silently dropped. Previously an oversized batch could lose all of its entities without any error surfaced on the document.
Inherited context clamp — when GRAPH_EXTRACTION_MAX_CONTEXT is unset, the value inherited from OPENAI_MAX_CONTEXT is now clamped at 48000: a chat model's large context window must not dictate extraction batch sizing, since hosted providers cannot decode a fully-packed prompt inside a request timeout. Explicitly set values are honored as-is.
Token estimation for extraction batching uses tiktoken's cl100k_base for non-OpenAI models instead of a chars/4 heuristic.
Entity extraction reports per-batch progress (Finding entities: 240/1511 chunks...) instead of a static message for the whole phase.

Compact extraction output format

Entity and per-chunk relationship extraction now instruct the model to emit a compact line protocol (ENT|Name|Type|Description, REL|Source|Target|TYPE|weight|confidence|Description) instead of XML elements — the XML scaffolding consumed a large share of every response's output tokens, and structured-output latency on hosted providers is decode-bound, so scaffolding directly inflated wall time and pushed dense batches past request timeouts. Stored graph content (names, types, descriptions, weights) is unchanged.
XML parsing is retained as an automatic fallback for prompt drift and older models; the batched per-chunk path keeps its <chunk index> grouping with compact lines inside.
The extraction prompt version is part of the reprocess fingerprint, so upgraded instances re-extract documents on their next reprocess instead of delta-skipping them.

Ingestion progress UI

Phase stepper on the Documents page: processing documents show the pipeline phases (Convert → Chunk & Embed → Store → Extract) with live counts, an overall bar, and a within-phase bar — replacing a single opaque message. Image analysis renders as a parallel track, matching how it actually runs.
Per-document breakdown on the Knowledge Graph page while Step 1 runs, with compact steppers and per-document image-analysis bars — replacing the aggregate "Processing N documents..." banner.
Pending documents distinguish "queued by the system — waiting for a processing slot" from plain "Unprocessed"; stale image counters from interrupted runs are no longer displayed.

Prompt-injection scan accuracy

The ingestion-scan heuristics are now word-bounded and require instruction-domain anchors, eliminating false flags on ordinary book-scale prose ("firmly rooted", "reflects all aspects", "encoded … instructions", "show only the physical configuration") while the full attack corpus remains detected. Documents previously flagged by the over-broad patterns are cleared on their next scan.
The jailbreak pattern joined the same tightening: DAN is matched case-sensitively and all trigger words are word-bounded, so prose like "the Main–Danube canal — the modest 1992 waterway" no longer trips the flag; "escape" now requires an instruction-domain anchor ("press Escape at the prompt" is fine, "escape your instructions" is not).
Heuristic hits are no longer final verdicts on documents. When the LLM classifier layer is enabled, a regex hit only escalates: the classifier re-judges the matched region and the document is flagged solely on its confirmation. Regex-only flagging remains only when the classifier is toggled off or unreachable. This ends the class of complaints where a clearly safe document carried an "Injection Flagged" badge because a chat-input regex fired on long-form prose.

Provider notes: Venice (docs)

New Configuration section with measured tuning guidance for running the ingestion stack against api.venice.ai: extraction batch sizing against serving speed, embed-input validation behavior, rate-limit and concurrency-slot characteristics, and the HTTP-200 error-envelope quirk. The previous guidance recommending full-window extraction budgets was corrected accordingly.

Error tracking with GlitchTip

Full-stack crash and error reporting via a self-hosted GlitchTip instance (Sentry-protocol compatible). Backend and frontend report to separate GlitchTip projects with readable stack traces on both sides — Python events carry source-context lines out of the box, and production frontend builds upload source maps so browser errors show original TypeScript instead of minified chunks. Entirely opt-in and env-driven: without a DSN the images run exactly as before.

Backend error tracking

SENTRY_DSN (default: empty) activates sentry-sdk at startup — unhandled endpoint exceptions and ERROR-level logs from any code path (API, background ingestion, the Docling conversion worker) become GlitchTip events. The client-facing 5xx sanitization is unchanged; capture happens before it.
Log correlation — every event is tagged with the same request_id that appears in server logs and X-Request-ID response headers, plus a service tag (backend / docling-worker).
Privacy deny-by-default — request bodies are never attached (SENTRY_MAX_REQUEST_BODY_SIZE=never) and PII capture is off (SENTRY_SEND_DEFAULT_PII=false), mirroring the Langfuse content-masking posture. SENTRY_TRACES_SAMPLE_RATE=0 keeps it errors-only by default.

Frontend error tracking

@sentry/nextjs wired for browser, Server Components, and proxy errors (instrumentation-client.ts, instrumentation.ts + onRequestError, global-error.tsx), enabled by NEXT_PUBLIC_SENTRY_DSN/SENTRY_DSN (empty = disabled).
Readable production stack traces — when SENTRY_URL, SENTRY_ORG, SENTRY_PROJECT, and SENTRY_AUTH_TOKEN are provided at image build time, next build uploads source maps as debug-ID artifact bundles (GlitchTip ≥ 4.2, verified against Turbopack builds) and deletes the .map files afterward so they are never served publicly.
Deploy wiring — dev/prod/Dokploy/Coolify compose files map SENTRY_DSN_BACKEND/SENTRY_DSN_FRONTEND onto each container so the two apps report to their own projects; all values optional with empty defaults.

July 7, 2026

A resilience-focused release: every change is about keeping an instance healthy under real-world failure — provider hangs, database restarts, oversized requests, redeploys mid-ingestion — and giving operators a compliance-grade audit trail plus a backup story that is verified on every run and proven restorable. It also closes the failure modes that stay invisible until the day they matter: backups that silently produce nothing, background work that dies without a trace, and a failing git sync that hammers the provider forever. No API breakage.

Backups you can actually restore

Backups on every deploy path — the Coolify and Dokploy stacks now include the nightly backup sidecar that previously shipped only with the standalone prod overlay. Chat data is included where the chat service runs.
Server-side graph export — the backup sidecar no longer streams the APOC export through a client-side quoting round-trip (which could silently produce an empty or subtly corrupted graph.cypher). The neo4j service now writes the export itself (NEO4J_apoc_export_file_enabled=true + the backups volume mounted at its import directory): byte-exact output, compressed to graph.cypher.gz.
Verified before it counts — exported row counts are checked against the live database, a SHA256SUMS manifest is written, and the run is stamped .complete + LAST_SUCCESS only after all checks pass. Retention now runs only after a verified success and never deletes the newest complete backup — consecutive failures can no longer rotate away the last good copy.
Failures are visible — the sidecar has a compose healthcheck that goes unhealthy when the newest verified backup is older than 2× the interval. The first backup runs ~2 minutes after start instead of a full interval later.
A tested restore path — new /restore.sh <timestamp>: verifies checksums, wipes, replays, and reports counts; validated end-to-end (backup → restore → byte-identical data, including adversarial newline/quote/backslash content). The handbook and deployment guide now document this runbook instead of a neo4j-admin dump procedure that could not restore sidecar backups.
Coolify fix — the Coolify backup sidecar referenced an env var the stack never sets (NEO4J_PASSWORD instead of SERVICE_PASSWORD_NEO4J) and could not authenticate; corrected.

Self-healing background work

Stranded-document sweep — if a document's failure-status write itself loses Neo4j past its retries, the document no longer sits in "processing" until the next restart: an hourly sweep resets transient-state documents that have no live in-process task and no status heartbeat for 30+ minutes.
Dead-task reaper — a background task whose coroutine died without reporting completion is marked failed after 2 hours without a progress heartbeat, instead of living in memory forever, blocking new syncs for its git connection and pinning safe_to_redeploy: false until a restart.
Abandoned import uploads — stale chunked library-import upload sessions are now swept hourly (previously only when the next upload started).

Git connector

Event-loop protection — the sync engine's per-file database and disk work now runs off the event loop; syncing a multi-hundred-file repository no longer stalls concurrent requests (including /health) for the duration of the ingest.
Failure backoff — a connection whose scheduled sync keeps failing (revoked PAT, oversized repo, network) now backs off exponentially (capped at 24h) instead of being re-cloned on every 5-minute scheduler tick, and reports sync_status: error. A successful sync resets the backoff.

Health checks that actually gate

GET /health now answers HTTP 503 when the instance is degraded (Neo4j unreachable or schema not yet confirmed) instead of 200 with a degraded body. Compose healthchecks, depends_on: service_healthy gates, and health-aware proxies key off the status code, so a booting or broken instance is no longer reported healthy at the transport level. The backend healthchecks gained a 60-second start_period to cover schema initialization at startup.

Quieter outages

Background loops (task persistence, usage-meter flush, schema-init retry, git scheduler) now rate-limit their failure warnings to one per 5 minutes per source, with a suppressed-repeat count — a Neo4j outage no longer produces tens of identical log lines per minute.

Resilience & data safety

Neo4j transient-error retry — idempotent reads and status upserts retry automatically (3 attempts with backoff) across a Neo4j restart instead of failing the request or stranding a document in failed.
No more phantom 401s — API-key validation previously turned any Neo4j hiccup (restart, brief lock contention) into 401 Unauthorized, which clients rightly treat as "your key is invalid" and surface as auth errors. Validation now retries transient errors, answers 503 + Retry-After when the auth store is genuinely unreachable (401 is reserved for keys that were checked and rejected), caches successful validations in-process for API_KEY_CACHE_TTL_SECONDS (default 30s, invalidated instantly on key changes) so a page-load burst validates once instead of a dozen times, and no longer fails a request when the best-effort last_used_at bookkeeping write fails. Per-key usage recording moved off the event loop.
Cross-tenant DNS misrouting on shared proxy networks — on hosts running multiple cortex stacks, internal server-side calls to http://backend:8000 could resolve to another tenant's backend: every stack auto-aliases backend on the shared Dokploy/Coolify proxy network, and Docker DNS aggregates same-name aliases across networks — the wrong tenant then rejects the caller's keys, surfacing as random 401s in chat and frontend that no amount of backend-side resilience can prevent. The deploy composes now give the backend a cortex-backend-internal alias on the stack-private network and all internal consumers dial that alias, making cross-tenant resolution impossible.
Server-side query timeout — the deploy composes set a Neo4j transaction timeout (CORTEX_NEO4J_TX_TIMEOUT, default 300s) so a runaway query can no longer pin a connection and an API worker forever.
Schema init that can't be skipped silently — constraint/index creation now retries with backoff at startup and keeps retrying in the background; /health reports the new schema_initialized field and stays degraded until the schema is confirmed, so deploy gates wait instead of routing traffic to an instance with broken dedup and dead vector search.
Auto-resume after redeploys — documents stranded mid-processing by a restart are not just reset to pending but automatically resumed (AUTO_RESUME_PENDING_ON_STARTUP, quota-guarded). Bulk uploads parked deliberately with start_processing=false stay parked.
LLM transport limits — every LLM client the backend builds now carries an explicit timeout (LLM_REQUEST_TIMEOUT_SECONDS, default 360s) instead of the SDK's 600s default, so one hung provider connection can't pin an ingestion slot for 10 minutes.
Task state survives restarts — background task records (batch processing, relationship analysis, community detection, library export/import) are now write-through persisted to Neo4j. After a redeploy, polling a task no longer returns 404: interrupted tasks report failed with "interrupted by server restart", terminal states are preserved for 7 days, and stale records are pruned automatically every hour.
Free-disk guardrail — uploads, reprocessing, and library-import sessions are refused with 507 Insufficient Storage when accepting the payload would leave the uploads filesystem below MIN_FREE_DISK_MB (default 500) — disk-full corrupts Neo4j checkpoints, so refusing new data early is strictly safer. Operators get cortex_disk_free_bytes / cortex_disk_total_bytes gauges and a rejection counter in /metrics, plus disk_free_mb / disk_total_mb on GET /api/stats.

Request hygiene

Request-body ceilings — a new middleware rejects oversized bodies with 413 before they can pressure the container's memory: MAX_REQUEST_BODY_MB (default 32) globally, MAX_FILE_SIZE_MB + slack on upload routes, and MAX_IMPORT_BODY_MB (default 2048) on library import, which streams to disk. The reprocess upload path now enforces its size cap mid-stream instead of buffering the whole file first.
Event-loop protection — the hybrid-search endpoint, topic-hint generation, and the git-connector endpoints moved their blocking database/embedding work off the event loop; deleting a git connection with document purge now uses one batched delete. Under load these paths can no longer stall every concurrent request (including health checks).
Sanitized server errors — in production, 5xx responses return a generic message plus the request ID; the full detail stays in server logs. Exception internals (connection URIs, provider error bodies, file paths) no longer reach clients.

Audit trail (new)

ENABLE_AUDIT_LOG=true activates an append-only JSONL audit log: authentication failures, key-attributed mutating requests (uploads, deletions, configuration changes, key CRUD), and search/ask activity — each event carrying the acting API key, outcome, and request ID. Metadata only: document content and query text are never written, consistent with the observability masking stance. Fail-open by design and size-rotated. See the Security guide for details.

July 6, 2026

A security-focused release that hardens Cortex against prompt injection on both the paths an attacker can reach: the questions users ask, and the content you ingest. No single layer is trusted on its own — detection, a model-based classifier, output filtering, untrusted-content fencing, and an ingestion-time scan each narrow what the others miss. All layers are best-effort by design and fail safe (a defense being unavailable never breaks a question or an ingest).

Query-time detection & output filtering

Obfuscation-resistant detection — user questions are normalized before pattern matching: NFKC unicode folding defeats fullwidth/homoglyph tricks (e.g. ｉｇｎｏｒｅ) and zero-width/soft-hyphen stripping defeats keyword-splitting (ig<ZWSP>nore). The special-character heuristic was retuned so legitimate code/markup questions no longer trip it. On the chat and research entry points a detected injection returns a safe refusal, not an error.
Real streaming output filter — token-streamed answers now pass through a sliding-window filter (filter_stream) that redacts verbatim system-prompt phrases and structural role tags before they reach the client, catching leaks that span multiple chunks. Wired into both the fast and standard chat loops.
Research pipeline gated — the deep-research entry point now validates its own input, so every question path is covered.

Prompt Guard — query-time classifier (new)

A dedicated prompt-injection classifier model (PIGuard, MIT-licensed) can now screen each question before retrieval, as a second gate on top of the regex layer. It catches jailbreak/injection phrasings patterns miss, and is specifically tuned to avoid over-flagging benign questions that merely contain trigger words.

Zero per-instance cost — the model is hosted once per machine by the cortex-helper service (alongside the reranker) and called over HTTP, so no backend loads it into its own memory.
Opt-in, fail-open — active only when PROMPT_GUARD_SERVICE_URL points at a helper; if the helper is unreachable, questions proceed unguarded (availability over strictness). Flagged questions get the same safe refusal.
Runtime toggle with a clear cost — admins flip Prompt Guard in System Configuration → Features & Security without a restart. Each guarded question costs one extra query unit (metered toward the monthly quota) and appears as a prompt_guard.classify entry in Langfuse tracing; the toggle lets you trade that cost off. New settings: PROMPT_GUARD_SERVICE_URL, PROMPT_GUARD, PROMPT_GUARD_THRESHOLD.
Self-host without a helper — set PROMPT_GUARD_LOCAL=true to load the classifier in-process instead (mirrors the local reranker fallback). Off by default, since it adds resident memory; the shared-helper path stays the recommendation for multi-tenant hosts.

Untrusted-content fencing (indirect injection)

Retrieved chunks, merged graph context, and researcher tool results (knowledge search, http_request, skill outputs) are now wrapped in data-boundary markers, and the model is instructed to treat everything inside as data, not instructions — a defense against second-order injection planted in your own documents or fetched web/tool content.
A phrase-only heuristic scan annotates fenced content with an inline caution on a hit; content is logged but never dropped.

Ingestion-time document scan

Every ingested document (uploaded, web-crawled, or git-synced) is scanned once for planted injection instructions. A free heuristic always runs; an optional LLM classifier additionally scans the text when enabled.
Detected documents are flagged, never blocked — they stay ingested and answerable (and fenced at query time), and surface as an "Injection Flagged" badge/filter in the document list.
First runtime-editable setting — the LLM scan is on by default (INGESTION_INJECTION_SCAN) and admin-toggleable at runtime (persisted as a setting override, effective for new ingestions without a restart). Turning it off keeps the free heuristic but skips the per-document classifier query.

Deployment note (AaaS / shared-helper hosts): Prompt Guard is served from cortex-helper. Deploy the updated helper (it pre-caches the classifier), then provision PROMPT_GUARD_SERVICE_URL (= your helper URL) and PROMPT_GUARD into tenants. Self-hosters without a helper are unaffected by default — the guard stays inert (or set PROMPT_GUARD_LOCAL=true to run it in-process), and all other layers work unchanged.

July 4, 2026

The monthly quota now measures what actually costs money, silently broken documents are now visible and fixable from the UI, and the knowledge-graph build pipeline got a pass focused on API-call economy and robustness on large libraries.

Unit-denominated usage metering

MAX_QUERIES_PER_MONTH is now denominated in units — internal LLM completions — instead of HTTP requests, so quota consumption tracks inference cost directly. Every successful model call counts one unit: each step of a Q&A answer (typically a handful per question) and each extraction/analysis call during document processing (a few per file). Embeddings are free.

Document processing is now metered — previously, imports and graph rebuilds consumed unlimited model calls outside the quota. Now uploads, custom inputs, reprocessing, web imports, git syncs, and the graph-build steps all draw from the same monthly pool.
In-flight work always finishes — the quota blocks starting new work (429 + Retry-After until the next UTC month). A batch import stops cleanly between documents; skipped documents stay pending and resume next month.
Usage meter in the admin UI — the Settings page's Statistics panel shows a monthly usage bar with a queries vs. processing breakdown, turning amber at 80% and red when exhausted. GET /api/stats exposes monthly_usage_used, monthly_usage_limit, monthly_usage_query, monthly_usage_processing.
Fleet orchestration surface — GET /api/instance/status (admin) now also reports library size (document_count, entity_count, collection_count) and the same monthly usage fields, so deployment automation can read plan consumption from the endpoint it already polls for redeploy safety.

Failed & degraded documents

A document can complete processing yet be silently broken — e.g. an extraction timeout leaving it with 0 entities, or chunks missing embeddings (invisible to semantic search). These are now flagged as degraded:

New Degraded status filter and amber warning badge on the Documents page, with the reason shown on the card ("0 entities extracted" / "N chunks missing embeddings").
Combined failed/degraded banner with one-click Select all for bulk reprocess.
Reprocessing a degraded document automatically bypasses the "content unchanged" skip.
Signals (entity_count, unembedded chunk count) are persisted at processing time and backfilled once at startup for existing libraries.

Graph builds: leaner extraction & rebuild hardening

Graph content and quality semantics are unchanged; builds complete faster and make substantially fewer LLM calls, especially under provider rate limits.

Entity extraction skips redundant summary calls — the per-document summary is now generated as extraction context only when a document spans multiple extraction batches. Single-batch documents (the common case) already carry their full text in the extraction prompt, so no separate summary call is made for them.
Entity extraction is protected against output truncation — a batch whose LLM response hits the output-token cap is automatically split in half and retried, so entity-dense documents keep their tail entities. A single chunk that still truncates logs a warning naming EXTRACTION_MAX_OUTPUT_TOKENS.
Batched extraction accepts explicit empty results — when the model returns an explicitly empty <chunk> block in a batched relationship call, that is accepted as a deliberate "no relationships" answer; only chunks missing from a (truncated or partial) response are retried individually. This avoids re-asking sparse chunks one by one.
Batched graph writes and fulltext dedup prefilter are now the default — ENABLE_BATCHED_KG_WRITES=true stores entities/links/relationships via UNWIND batches (a handful of Neo4j round trips per document), and ENTITY_DEDUP_PREFILTER=true scores the top fulltext-index candidates instead of scanning every entity during Levenshtein dedup. Both preserve the per-item dedup semantics (locked by parity tests) and can be set to false per stack.
Relationship confidence is persisted — both relationship writers now store the model's confidence score alongside description and weight (previously only library imports carried it through), making it available for filtering and analysis.
Robust to embedding-model switches — entities carrying vectors from a previously configured embedding model are re-embedded when merged into and treated as missing by Step 2's automatic backfill, and the kNN candidate scan skips incompatible vectors instead of aborting. Changing EMBEDDING_MODEL/EMBEDDING_DIMENSION no longer quietly degrades semantic dedup or targeted relationship discovery.
Graph-wide deletes scale with library size — "delete all entities", "delete all relationships", and the rebuild's batch-relationship cleanup now run in bounded batches (CALL {} IN TRANSACTIONS), matching the system-reset behavior, instead of a single whole-graph transaction that can exceed Neo4j's transaction memory limit on large graphs.

Other changes

BATCH_PROCESSING_CONCURRENCY default raised from 2 to 3 — the bench-validated, Venice-safe setting from the June recalibration is now the shipped default. Deployments that pinned the variable explicitly are unaffected; unset deployments process one more document in parallel during batch ingestion.

July 3, 2026

Both knowledge-graph build steps got cheaper, faster defaults — substantially fewer LLM calls at equivalent extraction quality, with the previous behaviors preserved behind flags.

Step 1: relationship descriptions & batched extraction

Relationship descriptions are now reliably captured in Step 1 — the per-chunk extraction prompts now pin the exact output format, so models consistently include a short evidence-grounded description with every relationship (previously most per-chunk relations were stored without one). The parser also accepts the common <relation> tag variant. Descriptions appear in the Explore relationship browser and enrich RAG context.
Chunk-batched relationship extraction is now the default — ENABLE_BATCHED_CHUNK_RELATIONSHIPS (now default true, with RELATIONSHIP_CHUNKS_PER_CALL=4) packs several chunks into one extraction call: several times fewer Step 1 LLM calls at equivalent extraction quality, which speeds up graph generation considerably when your LLM provider enforces request-rate limits. Set it to false to restore the previous one-call-per-chunk behavior.

Step 2: targeted relationship discovery

Deep relationship analysis (Step 2) got a new default engine that scales much better on large graphs. Candidate entity pairs are now identified directly in Neo4j and the LLM focuses on verifying pairs with actual signal, which substantially reduces analysis time and token cost as libraries grow.

Candidates without the LLM — the new default engine (RELATIONSHIP_DISCOVERY_MODE=targeted) generates candidate entity pairs from the Neo4j entity_embedding vector index (kNN; entities missing embeddings are backfilled automatically first) and from document co-mention (unconnected pairs mentioned together in ≥2 documents, hub entities excluded). Pairs are scored, deduplicated, and capped per entity and in total.
LLM as verifier, not scanner — the relationship model only verifies/classifies the ranked pairs in small batched calls (~40 pairs per call) with a bounded chunk-context budget, instead of scanning every entity batch against a near-full context window.
Single pass — targeted mode does no multi-round discovery; RELATIONSHIP_TARGET_RATIO and RELATIONSHIP_MAX_ROUNDS now only apply to the legacy mode. RELATIONSHIP_MAX_HOURS, RELATIONSHIP_MAX_PER_ENTITY, PARALLEL_RELATIONSHIP_BATCHES, and the confidence ≥ 0.5 filter apply to both modes. Relationships are still stored with extraction_method='cross_collection'; Step 1 per-chunk relations and the Knowledge Graph UI are unchanged (the ERR indicator still displays in both modes).
New env vars (defaults): RELATIONSHIP_KNN_K=8, RELATIONSHIP_KNN_MIN_SIMILARITY=0.80, RELATIONSHIP_MIN_SHARED_DOCS=2, RELATIONSHIP_DOC_FREQ_CAP=30, RELATIONSHIP_MAX_CANDIDATE_PAIRS=15000, RELATIONSHIP_CANDIDATES_PER_ENTITY=10, RELATIONSHIP_PAIRS_PER_CALL=40, RELATIONSHIP_PAIR_CONTEXT_TOKENS=3000. The admin config endpoint now also exposes relationship_discovery_mode.
Legacy path preserved — set RELATIONSHIP_DISCOVERY_MODE=llm_scan to restore the previous two-phase full-batch scan with multi-round discovery.

July 2, 2026

Two operations that worked fine on small instances but failed on grown ones are now size-independent, and a structural latency pass over the agentic ask pipeline cut average chat time-to-first-token by ~65% on the reference deployment.

System reset & library import at scale

System reset no longer runs out of Neo4j memory — deleting the knowledge graph now happens in bounded batches (CALL {} IN TRANSACTIONS) instead of one whole-graph transaction, which on large libraries exceeded Neo4j's transaction memory limit (MemoryPoolOutOfMemoryError). Orphaned chunks are swept too.
Library import uploads are chunked — the web UI uploads export archives in 8MB sequential requests (/api/admin/import/upload/*) with live upload progress, then starts the import task. Single-request uploads of large archives were cut off mid-body by reverse proxies with body-read timeouts (Traefik v3 on Coolify defaults to 60s → HTTP 502). The single-request POST /api/admin/import remains for API use.
Long admin calls survive the frontend proxy — the Next.js rewrite proxy timeout was raised from its 30s default to 300s (it previously returned a bare HTTP 500 for a reset that took ~33s, even though the reset completed).

AskAI v2 — the pipeline

A chat turn now costs 2 LLM calls instead of 3 (plus one eliminated helper call per search), and long Deep Research questions that previously hit gateway limits now complete. Answer quality and citation behavior are unchanged — the writer stage is untouched. All changes below are default-on, each individually revertible via env flag:

Chat early-write (RESEARCHER_SPEED_EARLY_WRITE) — once a search iteration has produced sources (and no skill/git action ran), the answer is written immediately instead of spending one more full LLM round-trip asking the agent to confirm it is done.
Concurrent searches (RESEARCHER_PARALLEL_TOOL_CALLS) — read-only tool calls issued in one agent turn (knowledge / community / entity searches) now execute in parallel; skill and git actions stay sequential.
Entity hints (RESEARCHER_TOOL_ENTITY_HINTS) — the agent passes the entity names from its own search queries, eliminating the separate query entity-extraction LLM call on every search.
Search dedup (RESEARCHER_SEARCH_DEDUP) — an identical repeated search within one question is answered from cache with a nudge to try a different angle.
Faster turn completion (EMIT_DONE_BEFORE_MEMORY) — the SSE done event now fires as soon as the last answer token streams, before the post-answer memory compaction; the memory_update blob follows 1–4 s later and then the stream closes. The done frame carries pending_memory: true when an update is still coming. Integrator note: consume the stream to its actual end — a client that stops reading at done loses memory continuity. Set EMIT_DONE_BEFORE_MEMORY=false to restore the legacy order. See the Ask AI event reference.
Connection reuse for LLM calls, skill/git metadata caching, and event-loop hygiene fixes reduce per-request overhead further; SSE responses now send X-Accel-Buffering: no so customer-managed reverse proxies can't buffer the stream.

AskAI v2 — skills

The MAX_SKILL_INSTRUCTIONS_TOKENS budget is now actually enforced (with an explicit truncation marker) — previously skill instructions grew the system prompt without bound as more skills were enabled.
Deployments with no skills installed no longer receive skill instructions referencing a tool that doesn't exist (which occasionally provoked hallucinated tool calls on small models).
Truncated skill API responses are now labeled as truncated with pagination guidance, so the model paginates instead of confidently answering from cut data; the answer stage caps oversized API payloads.
Action requests ("create a ticket for that") can no longer be swallowed by the memory fast-path — they always run the full pipeline.

AskAI v2 — chat UI

Live citations — sources and [src_N] badges are clickable while the answer is still streaming, and survive pressing Stop.
Conversation memory wired end-to-end — the web UI now round-trips the conversation-memory blob, so memory-answerable follow-ups ("summarize that") skip retrieval entirely.
Conversations survive navigation — switching tabs or reloading no longer destroys the chat (per-tab persistence); Clear properly aborts an in-flight stream.
Smoother streaming — token rendering is batched per animation frame and completed messages no longer re-render on every incoming token, keeping long research sessions responsive.
Stopping mid-thought no longer leaks raw <think> text into the answer; the composer resets and refocuses after sending.

June 30, 2026

The bench-validated default primary for Q&A / research / chat is now Gemma4 26B A4B (google-gemma-4-26b-a4b-it on Venice) — a 26B/4B-active MoE that benches even faster than MiniMax-M3 at similar answer quality, making it an ideal fit for fast data retrieval from Cortex. MiniMax M2.7 / M3 were comparatively slower on this workload.

What changed

OPENAI_MODEL default changed from openai/minimax-m3 to google-gemma-4-26b-a4b-it.
OPENAI_MAX_CONTEXT recommended value is now 256000 (Gemma4 26B A4B's full 256K input window).
Recommended-stack blocks updated across .env.recommended, .env.example, README.md, the handbook, and the docs; the admin Primary Model panel reflects the new recommendation.

Existing deployments are unaffected — any explicit OPENAI_MODEL you set still takes precedence; this only changes the documented default and recommendation.

June 29, 2026

A round of reliability and UX hardening across the backend pipeline and the chat experience.

Reliability

Startup recovery of orphaned documents — on startup, any document left mid-processing (processing/extracting) by a prior shutdown or crash is reset to pending so it rejoins the queue instead of spinning forever (and /api/instance/status no longer reports safe_to_redeploy: false permanently after a redeploy). A WARNING log lists the reset ids.
DOCLING_CONVERSION_TIMEOUT (default 600 s) bounds a hung local Docling subprocess conversion — on timeout the worker is killed and the document is marked failed with a clear message instead of hanging on a large or corrupt file. Does not apply to the remote DOCLING_SERVICE_URL path.
SSE / chat errors no longer leak raw exception text — clients get a generic message plus a request id, with the full error logged server-side.
Streaming uploads are rejected with HTTP 413 once they exceed MAX_FILE_SIZE_MB, instead of being fully buffered first.
Clearer messages for partial / zero-match delete and move operations, and for task lookups after a restart.

Chat UX

Multi-line composer (Enter sends, Shift+Enter inserts a newline).
A Stop button to cancel generation mid-stream.
Graceful handling of interrupted streams — no endless typing cursor after a redeploy.
Real backend errors are now surfaced to the user.

General UX

Error + retry states where failures were previously silent (documents, search, collections, entity/community detail).
Escape-to-close and focus trap on modals.
Stale-response guards and poller cleanup.
Clipboard fallback on non-HTTPS origins.

June 26, 2026

Two admin-surface improvements: Langfuse traces now redact all authored content by default, and the System Configuration panel gained a curated view.

Langfuse content masking — privacy by default

Langfuse tracing now redacts all prompt, completion, tool, embedding, and vision text by default. A new LANGFUSE_LOG_EXTENDED flag (default false) wires a client-side mask hook so authored content never leaves the app — only structure reaches the Langfuse server: message roles, model + parameters, tool names + argument keys, parameter keys, token usage, cost, latency, and tags. This protects user content (we don't store what users prompt) and sharply cuts trace storage.

Set LANGFUSE_LOG_EXTENDED=true to log full content for local debugging (the previous behavior). Masking is deny-by-default and fail-closed — any unclassifiable text is redacted, never leaked. The Admin → System Configuration panel now has a Privacy section showing "Prompt & Content Redaction" and "LLM Tracing" status, so an operator (or a customer auditing a hosted instance) can verify at a glance that prompt content never leaves the box. No API or schema breakage — purely additive, and tracing stays fully env-gated (no keys = no tracing). See the Configuration guide.

Admin System Config — curated view

The Admin → System Configuration panel can now show a curated view. A new DISPLAY_FULL_SYSTEM_CONFIG flag (default false) hides advanced tuning knobs — output-token budgets, concurrency counts, chunking parameters, hybrid-search weights, graph hops, community sizes, and similarity thresholds — leaving models, API bases, context windows, dimensions, and feature toggles. Set it to true to reveal every knob.

This is display-only — settings, the /api/admin/config response shape, and ingestion/RAG behavior are unchanged. No API or schema breakage. See the Configuration guide.

June 23, 2026

Two additive operator features: optional LLM tracing and cost tracking via a self-hosted Langfuse instance, and a status endpoint deploy automation can poll before restarting an instance.

LLM observability — Langfuse tracing

Optional, env-driven LLM tracing and cost tracking via a self-hosted Langfuse instance. When LANGFUSE_BASE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY are set, every LLM, embedding, and vision call is traced (cost, tokens, latency, errors), and each agentic Q&A is grouped into one inspectable trace — across Venice, OpenRouter, and any OpenAI-compatible provider.

Activation is fully env-gated: with no keys the same image runs identically untraced (plain OpenAI client, zero overhead). Coverage spans the whole stack — the agentic researcher/writer loop, graph extraction, the Haystack embedders, and the vision analyzer. LANGFUSE_TRACING_ENABLED (master off-switch) and LANGFUSE_SAMPLE_RATE (0.0–1.0) tune it.

For accurate USD cost, add your Venice/OpenRouter models under the Langfuse project's Models settings (token counts are tracked regardless). No API or schema breakage — purely additive. See the Configuration guide.

Instance status endpoint — redeploy safety

A new admin endpoint, GET /api/instance/status, gives deploy automation a single call to decide whether an instance can be safely restarted or upgraded.

It returns safe_to_redeploy (with a reasons list of any active blockers) plus the supporting detail: documents currently processing/extracting, running background tasks, in-flight AskAI/research queries, and last-activity timestamps (last_query_at, last relationship analysis, community detection, entity merge).

safe_to_redeploy is false while any destructible work is in flight — documents mid-processing, background tasks (held in an in-memory store a restart would lose), or an active AskAI query (a restart kills the stream). Pending documents persist in Neo4j and resume after a restart, so they're reported but never block. AskAI activity is tracked across /api/ask, /api/ask/stream, and /api/ask/stream/thinking.

Requires manage permission. No API or schema breakage — the endpoint is additive. See the Deployment guide for the redeploy-safety polling pattern.

June 22, 2026

Two additions: Web Import harvests web pages into the knowledge base as clean markdown, and a systematic QA pass expanded the backend test suite and fixed four defects.

Web Import — MDHarvest powered by Crawl4ai

A new way to get web content into your knowledge base: Web Import harvests web pages into clean markdown and ingests them exactly like uploaded documents — chunked, embedded, and run through entity/relationship extraction, so they show up in Search, Ask AI, and the knowledge graph. It supersedes the standalone mdharvest tool.

Cortex never embeds a browser. It calls an open-source crawl4ai service over HTTP — you run crawl4ai once and a single instance can serve many Cortex deployments (in hosted setups, one per server, shared across tenants). There is no local crawler fallback, which keeps the per-instance footprint small.

Using it

On the Documents page, the Upload button now has a ▾ menu beside it with a Web Import option (it appears only when the feature is enabled). It opens a modal where you:

pick a collection,
paste URLs (one per line) — or use Discover links to crawl a page and pick same-site links from a checklist,
choose a content filter (Readable / Full page / Relevance-ranked),
and click Import from Web.

A progress bar tracks the crawl and processing; when it finishes you get a clear "imported N of M — close this window to see your documents" summary, and the document list refreshes automatically. Each imported page carries a provenance header recording its source URL and the extraction date.

Enabling it

Code
 
ENABLE_WEB_CRAWL=true
CRAWL_SERVICE_URL=http://crawl4ai:11235   # your own, or a shared crawl4ai
# CRAWL_SERVICE_TOKEN=...                  # only if crawl4ai requires a token

With the flag off or no service URL set, the feature stays hidden. See the Web Import guide for self-hosting crawl4ai and the full tuning reference (CRAWL_CONTENT_FILTER, CRAWL_CONCURRENCY, CRAWL_MAX_URLS_PER_JOB, …).

Privacy on shared crawl4ai instances: when several deployments share one crawler, no crawl history is retained and no deployment can see another's crawls — storage is ephemeral, only request-scoped endpoints are used (never the async job API), and every request bypasses the cache.

No API or schema breakage — the feature is additive and off by default.

QA pass — test coverage, fixes & docs hardening

A systematic quality pass: a canonical feature/defect inventory (qa/cortex_qa_master.ods — Features/Defects/Summary sheets), a large expansion of the backend test suite, a live end-to-end harness, and four fixes.

New config

EXPOSE_API_DOCS (default auto) — gates the interactive API docs (/docs, /redoc, /openapi.json). auto keeps them on in development and off in production, so a directly-reachable backend no longer discloses its full API schema to anonymous callers. Set true/false to force. (Closes D-004 below.)

Fixes

D-001 (Medium) — the git connector's extension check (GitConnectorService._supported) instantiated a full DocumentProcessor (building the embedder) just to read a class-level constant; it now reads the class attribute directly. This also restored hermetic test isolation on machines with a real .env.
D-003 (Low) — API-usage endpoint categorization used dict-order prefix matching, mislabeling /api/custom-inputs/{id} as upload instead of documents; now longest-prefix-wins (analytics data integrity).
D-004 (Low) — interactive API docs were served unauthenticated; now gated by EXPOSE_API_DOCS (see above).

Testing

Backend suite expanded from 322 to 458 hermetic tests (new modules for auth, API-usage, vision image-prep, context-curator memory, library-transfer NDJSON round-trip, researcher-agent helpers, skill-service parsing, RRF fusion, and FastAPI endpoint contracts).
New live end-to-end harness (backend/tests/test_live_e2e*.py) exercises real journeys (health, auth boundary, collections CRUD, hybrid search, streaming chat, full document ingestion→extraction) against a running stack; auto-skips when none is reachable. Authenticated runs read the key from CORTEX_E2E_API_KEY.

No API or schema breakage — EXPOSE_API_DOCS is additive; the only behavior change is that interactive docs default off under ENVIRONMENT=production (re-enable with EXPOSE_API_DOCS=true).

June 18, 2026

A major overhaul of the chat/answer path. Reasoning-capable models (especially on Venice) were returning empty or 30–90 s answers: the chat path never suppressed model "thinking", so hidden chain-of-thought streamed in a separate reasoning_content channel the answer stream ignores — adding seconds before the first token and, across the multi-iteration agent loop, frequently exhausting the budget into empty/timeout answers. This update makes chat answers fast, grounded, concise, and deterministic.

Snappy chat answers

Reasoning suppressed on the chat path by default. DEFAULT_REASONING_MODE now defaults to off and is actually applied: the speed-mode researcher loop, the answer writer, and the non-agentic + fast-search streaming writers are all routed through safe_chat_completion (per-provider reasoning suppression + a 400-fallback), replacing an ad-hoc single-provider hack. On Venice this sends disable_thinking, cutting model time-to-first-token to under 1 s (a direct probe measured 13.6 s → 0.57 s for gemma). Deep-research (quality) mode is unaffected — it stays AUTO and keeps full reasoning.
Fewer loop iterations. RESEARCHER_MAX_ITERATIONS_SPEED lowered 5 → 3 — with thinking off, the agent loop (not the model) was the dominant latency. Three rounds keep conversational answers snappy (one search + answer) while leaving room for a skill flow (activate → call → answer) to complete without truncation.
Rewritten speed-writer prompt for concise, direct, cited answers.
Result: empty-answer rate dropped to zero on previously-failing models; chat time-to-first-token fell from ~27–60 s to ~8–14 s.
Caveat (OpenAI): on GPT-5 / o-series, off maps to reasoning_effort none/minimal, which can disable parallel tool calls in the agent loop. Operators on those models should set DEFAULT_REASONING_MODE=auto.

Structured timeouts & a non-blocking agent loop on `/api/ask`

New ASK_DEADLINE_SECONDS (default 28) — the non-streaming POST /api/ask now wraps its work in a deadline and returns a clean 504 JSON {detail} on expiry, instead of the edge proxy (Coolify/Traefik) cutting the silent socket at ~30 s and emitting a bare plain-text 500. Keep it just below the proxy read timeout and raise both in lockstep; 0 disables. Does not apply to /api/ask/stream (SSE heartbeats keep that alive).
Agentic requires streaming — a non-streaming use_agentic request now fails fast with a 400 {error: "agentic_requires_streaming", use_endpoint: "/api/ask/stream"} rather than racing the proxy timeout (deep research routinely runs much longer than 30 s and only survives over SSE).
Cleaner error surfacing — 4xx/504 responses (and the existing collection-access 403) are no longer swallowed and re-wrapped as 500.
Event-loop unblock — the last two synchronous Neo4j tool calls in the researcher (community_search, entity_lookup) now run off the event loop, so they no longer stall concurrent requests during agentic runs.

Docs

New handbook chapter "The Cortex Standard" — state-of-the-art knowledge graphs, the open-systems/portable-memory ethos, and the fast/grounded/concise answer principles. Chapter 10 (Ask AI), Chapter 4 (Configuration), environment.md, configuration.mdx, and the README were synced for the new reasoning default.

Behavior change to note: the chat/answer path now suppresses model reasoning by default (DEFAULT_REASONING_MODE=off). Restore provider-default thinking on chat with DEFAULT_REASONING_MODE=auto (recommended on OpenAI GPT-5/o-series). No API or schema breakage; everything else is additive.

June 10, 2026

Opt-in flags that cut the LLM and database cost of graph building, leaner instance memory with an optional shared per-machine model service, and a broad production-hardening pass. All changes are additive or flag-gated — existing .env files and deployments keep working unchanged.

LLM-stack efficiency (flag-gated, default off)

A set of opt-in flags cuts the LLM and database cost of knowledge-graph building without changing any API or output semantics. Each flag ships default-off and is meant to be enabled after an A/B run against your bench baseline (bench/BASELINE.md):

ENABLE_BATCHED_KG_WRITES — entities, chunk links, and relationships are written in UNWIND batches: a handful of Neo4j round trips per document instead of one per item (hundreds→~10 on extraction-heavy docs), while preserving the exact dedup/merge semantics of the per-item path (locked by parity tests).
ENABLE_BATCHED_CHUNK_RELATIONSHIPS (+ RELATIONSHIP_CHUNKS_PER_CALL, default 4) — per-chunk relationship extraction packs several chunks into one LLM call (÷~4 calls), with a per-batch safety net that re-dispatches through the single-chunk path on parse failure.
ENABLE_PHASEB_CHECKPOINTING — cross-document relationship analysis persists per-batch progress: a crash or redeploy resumes instead of re-paying every batch, and multi-round runs reuse round 1's candidate scan (~50% fewer Phase 1 calls).
ENABLE_REPROCESS_DELTA — reprocessing a document whose file bytes and extraction config are unchanged is skipped entirely; git re-syncs of unchanged files cost ~zero.
ENTITY_DEDUP_PREFILTER — entity dedup scores the top-50 fulltext candidates instead of scanning every entity (big win on 10k+ entity graphs).
Prompt caching: the researcher system prompt is now byte-stable across loop iterations (RESEARCHER_STABLE_PROMPT, default on — provider prefix caches hit from iteration 2), and ENABLE_PROMPT_CACHE_CONTROL adds Anthropic cache_control breakpoints when routed via OpenRouter. Skill catalog/instructions are TTL-cached per process instead of being reloaded (Neo4j + filesystem + decryption) on every request.

Production hardening

Resilient helper transport: calls to the shared cortex-helper now retry with backoff behind a circuit breaker; HELPER_STRICT_REMOTE=true keeps docling out of tenant containers even on helper outages. The helper itself gained per-tenant fair queuing (CONVERT_PER_TENANT, CONVERT_QUEUE_TIMEOUT → 503 + Retry-After) and a shutdown drain.
Graceful shutdown: rolling restarts drain in-flight requests (uvicorn --timeout-graceful-shutdown + compose stop_grace_period); SSE streams receive a terminal event: shutdown frame so clients can reconnect instead of seeing a dead socket.
Observability: optional JSON logging with X-Request-ID correlation end-to-end (chat → backend → helper), and an admin-protected Prometheus GET /metrics endpoint (HTTP latency/route, SSE streams, document outcomes, conversion timing, helper breaker state). LOG_FORMAT=plain (default) keeps logs byte-identical.
Per-key rate limiting (RATE_LIMIT_QPM, default off) on ask/upload endpoints — a burst guardrail beneath the monthly quota, returning 429 + Retry-After.
Memory caps everywhere: Neo4j, frontend, and nginx now carry compose mem_limits (tunable via NEO4J_MEM_LIMIT etc.) so one stack's blowup can't OOM a neighbor; the Neo4j driver pool is configurable; nginx gained a dedicated unbuffered SSE location with a 1-hour read timeout.
Slim backend image: Dockerfile.prod --build-arg INSTALL_LOCAL_ML=false builds a torch-free image (~1.2 GB vs ~7 GB dev image) for stacks that fully offload to cortex-helper — the biggest per-tenant footprint lever yet. CI smoke-builds it on every PR.
Backups: an opt-in compose overlay (docker-compose.backup.yml) runs nightly online graph exports (APOC, works on Community) + file-volume archives with retention, plus a documented restore runbook (ops/backup/backup.sh).
Fixes: skill HTTP calls now honor SKILL_HTTP_TIMEOUT (was hardcoded 15s); vector-index health is verified at startup and silent semantic-dedup degradation is surfaced in /api/stats (vector_search_failures); inbound conversation-memory blobs are size-clamped.
CORS is now allowlist-driven via CORS_ALLOWED_ORIGINS (default * allows any origin but with credentials disabled, since auth is header-based). Set an explicit allowlist for production.
Fail-fast secrets: ENVIRONMENT=production refuses to boot on weak/default secrets — an empty/password123 NEO4J_PASSWORD, or a SESSION_SECRET under 32 characters when ADMIN_PASSWORD is set.
CI: a GitHub Actions workflow now runs backend tests + lint and frontend type-check + lint on every PR.

Leaner instances: lazy models + a shared per-machine model service

Backend startup memory dropped from ~2.2 GB to ~1 GB by deferring heavy model loads. The cross-encoder reranker no longer loads at startup (RERANKER_PRELOAD=false by default) — its ~7 s cold start is now hidden behind the query-analysis and search work that precedes reranking, and the model unloads after an idle period (RERANKER_IDLE_TTL_SECONDS, default 1800; 0 = never) to reclaim ~1 GB. Docling is no longer imported at startup (it ran in a subprocess anyway), saving ~244 MB per instance.

New cortex-helper service. For deployments running many isolated stacks on one host, the cross-encoder reranker and Docling converter can now be hosted once per physical machine and shared by all stacks via RERANKER_SERVICE_URL / DOCLING_SERVICE_URL / HELPER_SERVICE_TOKEN. Models stay warm (Docling conversions skip the per-document model reload — ~0.04 s vs ~4.5 s), and both clients fall back to the built-in local path automatically if the service is unset or unreachable. Single-instance and on-prem users are unaffected.

Upgrade notes

No database migration and no API changes — endpoints, request/response shapes, and the Neo4j schema are unchanged, so existing API-key consumers and cortex-chat are unaffected. Rebuild the backend image and redeploy; the defaults preserve current behavior closely. A few things to decide per deployment:

Reranker is now lazy + idle-unloading. Previously it was warmed at startup and stayed resident; now the first reranked query after a restart (or after an idle period) pays a ~7 s in-memory load (no re-download — the model is baked into the image). To restore always-warm behavior, set RERANKER_PRELOAD=true and RERANKER_IDLE_TTL_SECONDS=0. Recommended for latency-sensitive single-tenant deployments; leave the new defaults for multi-tenant density.
CORS default disables credentials with the wildcard. Harmless because auth is header-based (X-API-Key), and browsers reject wildcard-with-credentials anyway. If a browser client genuinely needs credentialed CORS, set an explicit CORS_ALLOWED_ORIGINS allowlist (which re-enables credentials).
ENVIRONMENT=production fail-fast is opt-in. Deployments that don't set it are unaffected. But once set, startup aborts if NEO4J_PASSWORD is empty/password123 or SESSION_SECRET is under 32 characters (with ADMIN_PASSWORD set) — fix those secrets before enabling the flag.
Adopting the shared cortex-helper service is optional and fails safe. Leave RERANKER_SERVICE_URL / DOCLING_SERVICE_URL unset to keep the built-in local path; when set, both clients fall back to local automatically if the service is unreachable. Ensure the stacks can reach the helper (a shared external Docker network for permanence), that HELPER_SERVICE_TOKEN matches the helper's HELPER_TOKEN, and that the helper's RERANKING_MODEL matches expectations.
Memory: with full offload, backend idle drops to ~1 GB and no longer spikes — BACKEND_MEM_LIMIT can be lowered. Without offload (or docling-only), a local rerank still spikes to ~2 GB, so keep mem_limit at ≥ ~2.5 GB; budget the helper's own ~0.6–4 GB once per host.

June 8, 2026

Snappier chat: live status, heartbeats, and a Deep Research toggle

The chat no longer feels stuck in the seconds before the first token. The streaming endpoint now emits structured status events at each pipeline stage (analyzing → searching → reranking → generating, in every mode including standard Chat) and SSE heartbeats during silent windows. The chat UI replaces the static "Thinking…" dots with a live indicator — a blinking accent dot, the current stage label, a running elapsed-seconds counter, and a reassurance line after 12 s. Gated by STREAM_REASONING_STEPS.

Deep Research is now an in-chat toggle. Chat is the default; an Erlenmeyer-flask button flips on Deep Research (multi-step depth) at any time — even mid-conversation — instead of being a separate destination.

Conversation Memory — a multi-bucket context curator for the agent

The research agent gains an opt-in, client-carried conversation memory that replaces blunt history truncation (the agent previously kept only the last 6 messages, silently forgetting older turns). The backend stays stateless: send an opaque conversation_memory blob on the request, read the updated one back from a new memory_update SSE event, and replay it next turn. Omit the field and nothing changes — it's fully backward-compatible.

A new context curator rebuilds a small, fixed context each turn from several memory buckets instead of re-feeding ever-growing history:

transcript — recent turns verbatim + a rolling summary of older ones (no more amnesia past a few turns).
facts / open_questions / intent — durable knowledge, unresolved threads, and the user's goal + preferred answer style/language, extracted automatically.
source_ledger — every cited source gets a conversation-stable sid (now on each sources event), so a citation in turn 2 keeps its identity in turn 5.
kg_context — a knowledge-graph snapshot for grounding follow-ups.

Faster, cheaper follow-ups. A memory fast-path lets questions answerable from memory alone ("summarize that", "why?", "in German") skip retrieval entirely — a single lightweight pass instead of the full research loop. Memory updates (summary folding + bucket extraction) run after the answer streams using a cheap fast model, so they add no user-visible latency. See Ask AI → Conversation Memory. Tunable via CONVERSATION_MEMORY_* and ENABLE_MEMORY_FAST_PATH.

Skill API failures are now surfaced instead of silently swallowed

When the research agent calls an external API through a skill's http_request tool and the call fails — an error status (401, 403, 422, 5xx) or a timeout — that failure is now visible to you rather than glossed over.

Previously a failed call (for example, a Zammad ticket creation returning 403 Forbidden) was only fed back to the model internally: nothing in the UI distinguished it from a success, and the assistant could narrate the action as if it had completed. Now:

A red skill step appears in the chat's research process showing the failing method, URL, and HTTP status (e.g. API call failed: POST https://.../tickets → HTTP 403).
The final answer explicitly reports the failure — it states the attempted action did not succeed and explains why, using the API's error message, and will not imply the action worked. This is driven by a dedicated "Failed Actions" instruction passed to the answer writer, so it holds even for small/fast models.

This is most important for write actions (creating tickets, opening PRs, posting data). Common causes of a failed call: a missing or expired token (configure it in the skill's Setup Wizard), insufficient permissions on the token's account, or a payload the target API rejects as invalid. See Agent Skills.

June 3, 2026

Git integration — connect GitHub, GitLab & Gitea repositories as a living knowledge source

Cortex can now connect directly to git repositories and keep their knowledge in sync. This is a major capability expansion: a connected repo becomes a first-class, bidirectional interface — Cortex ingests the repo's files and wiki into the knowledge graph (read), and the research agent can act on the repo by opening pull requests (read/write). Works with GitHub, GitLab, and Gitea (including self-hosted) behind a single provider abstraction. Enable with ENABLE_GIT_INTEGRATION=true.

Two surfaces, one connection. Each connection carries an access_level:

Read → ingestion. Cortex clones the repo and feeds matching files (and optionally the wiki) through the normal pipeline — chunking, embedding, entity/relationship extraction — so repo content is searchable and shows up in the graph alongside your other documents.
Read/write → the agent gains a git_repo tool. It can read live file contents and propose changes as pull requests. Writes always land on a fresh cortex/agent-* branch and open a PR/MR for human review — never a direct push to the default branch. Write actions are rejected server-side on read-only connections, so the toggle is a hard guarantee.

Incremental, versioning-aware sync. Re-syncing doesn't re-ingest everything. Cortex stores the last-synced commit and runs git diff --name-status -M to classify each change — Added (new document), Modified (re-extract in place), Deleted (flag the document for review, never auto-delete), Renamed (remap the path). If history was rewritten (force-push) or the filters changed, it self-heals by falling back to a full-tree reconcile that compares every file's blob SHA to what's stored. Documents carry git provenance (git_connection_id, git_path, git_blob_sha, git_commit_sha) so the sync key is the repo path, not a filename. After a sync that changed anything, the graph is flagged stale so you know to re-run relationship analysis and community detection.

Curated "documents only" default. New connections default to ingesting .pdf and .md files only — a single checkbox in the connect form, checked by default. Uncheck it to reveal custom include/exclude globs (gitignore-style). Code files (.py, .ts, .go, …) and markdown ingest through a new Docling-free fast path (running Docling on source code is wasteful); PDFs and Office documents route through Docling as usual. Images and audio are deliberately excluded from repo sync so a connection never OCRs every logo.

Manual + scheduled. Sync on demand with a "Sync now" button, or set a per-connection interval and a background poller keeps the repo current. No webhooks required (no public endpoint needed). GitHub wikis are cloned via repo.wiki.git; GitLab/Gitea wikis use their Wikis API.

Setup UX. A new Git Integration card on the Settings page walks you through it: pick a provider, paste a personal access token, and an inline, vendor-specific guide tells you exactly which token to create (always recommending the least-privilege option) with a direct link to the right settings page. "Test" verifies the token, then you pick the repo, access level, and (optionally) advanced filters. Existing connections are fully editable — access level, branch, schedule, globs, wiki, and token rotation.

Credentials. Single-tenant, personal-access-token per connection. The token is injected server-side into every git and API call — the agent/LLM never sees it — masked in all API responses, scrubbed from logs and errors, and never persisted in .git/config.

Bonus correctness fix (applies to all documents). While building the sync's re-extraction path we fixed a pre-existing bug: relationships carry a source_document_id but previously survived reprocess/delete unless an endpoint entity became fully orphaned, leaving stale RELATES_TO edges. A new delete_relationships_by_source_document now runs in delete_document_chunks and delete_document, so reprocessing or deleting any document (uploads and custom inputs included) cleans up its relationships.

New env vars: ENABLE_GIT_INTEGRATION, GIT_WORK_DIR, GIT_CLONE_DEPTH, GIT_MAX_REPO_SIZE_MB, GIT_SYNC_MAX_FILE_SIZE_MB, GIT_SYNC_POLL_INTERVAL, GIT_HTTP_TIMEOUT, GIT_HTTP_INSECURE_HOSTS. The backend image now bundles git; pathspec was added for glob matching. New endpoints under /api/integrations/git/* (admin-gated). New backend: services/git_providers/ (GitHub/GitLab/Gitea adapters) and services/git_connector_service.py; new frontend: components/admin/GitIntegrations.tsx; new docs: handbook/22-git-integration.md, documentation/pages/features/git-integration.mdx, and .claude/domain/git-integration.md.

Knowledge Graph wording: "relations" and "cross-document relations"

The Knowledge Graph page (and docs) now use simpler terminology: Step 1 counts are labeled relations (formerly "within-document relationships") and Step 2 counts are labeled cross-document relations (formerly "cross-document relationships"). All of these are relations — only the cross-document ones need the qualifier. No behavior change; per_chunk_relationship_count and other API fields are unchanged.

June 1, 2026

Library export AND import stream to/from disk instead of buffering — fixes OOM crash on large instances

Exporting a large instance (~1,300 documents / 22K entities / 49K relationships, all carrying embedding vectors) crashed the backend container with no logs and left the subsequent redeploy failing. Root cause was memory, not storage: the export built a full Python list of every serialized record and then did "\n".join(lines), holding the entire payload twice in RAM. Chunks and entities both carry embedding vectors, so the two heaviest sections alone could hit 1–2 GB transient — enough to trip the kernel OOM killer, which sent an uncatchable SIGKILL (hence no logs) and took Neo4j and the redeploy down with it.

Export now streams: a new _write_ndjson helper writes each NDJSON entry one JSON line at a time via zf.open(name, "w", force_zip64=True), and the embedding-heavy sections pull 500-row batches from Neo4j and stream them straight into the zip. New batched query methods in neo4j_service.py: export_entity_count/export_all_entities_batched (ORDER BY e.name) and export_relationship_count/export_all_entity_relationships_batched (ORDER BY elementId(r) for stable SKIP/LIMIT pagination).

Import got the mirror treatment so restoring a large archive doesn't OOM the target instance: _iter_ndjson reads each heavy entry through an io.TextIOWrapper one line at a time, _iter_ndjson_batches feeds the existing import_*_batch inserts (chunks, entities, chunk mentions), and relationships stream one-at-a-time. The pre-import limit guards (MAX_FILES, MAX_ENTITIES) now use _count_ndjson — which counts lines without parsing or buffering — instead of loading the whole entities-with-embeddings file into RAM just to call len(). Progress totals are read from manifest.stats.

Peak RAM on both sides is now ~one 500-record batch regardless of corpus size. The archive format, endpoints, and on-disk output are unchanged — existing exports/imports remain fully compatible. Touched backend/app/services/{library_transfer_service.py, neo4j_service.py} and .claude/domain/admin-features.md.

May 22, 2026

Image downscaling + JPEG recompression before vision API call

PDF pages are rendered at 2× DPI for OCR-grade text legibility (images_scale=2.0), so a typical page becomes a ~2400×1700 PIL image. Encoded as PNG with our original _pil_to_data_url, that's several MB of base64 — and some hosted vision deployments (LiteLLM-wrapped vLLM, custom OpenAI-compatible endpoints) tokenize the base64 data-URL payload as text instead of using tile-based image accounting. One customer instance hit 184K input tokens against a 192K context cap on a single image as a result.

Two new env knobs in backend/app/services/vision_analyzer._pil_to_data_url:

VISION_MAX_IMAGE_SIDE (default 1568) — caps the longer side via Image.thumbnail with Lanczos resampling, preserves aspect ratio, original PIL untouched. 1568 matches Claude's recommended max side: high enough for OCR text legibility, low enough to keep JPEG payloads under ~700 KB. Set 0 to disable.
VISION_JPEG_QUALITY (default 85) — JPEG quality for opaque content. Images with alpha (mode RGBA) still go out as PNG; everything else uses JPEG with optimize=True for 5–10× smaller payloads at visually-near-lossless quality.

Surfaced in the admin Settings page (LLM Configuration → Vision Model) and documented in .env.example, .claude/environment.md, and handbook/14-image-analysis.md.

Venice perf-tuning numbers recalibrated to verifiably-stable defaults

The previously documented Venice tuning block (BATCH_PROCESSING_CONCURRENCY=5, CONCURRENT_EXTRACTIONS=10, CONCURRENT_RELATIONS=5, VISION_MAX_CONCURRENT=5) was too aggressive — a 52-doc ingestion run logged 1012 × HTTP 429 "Too Many Requests" against Venice's embeddings endpoint (tripping their 20-failure / 30-second penalty), and 34 entity-dedup batches fell back to Levenshtein because the rate-limit cascade killed embedding calls used for semantic resolution. Recalibrated to:

Code
 
BATCH_PROCESSING_CONCURRENCY=3
CONCURRENT_EXTRACTIONS=4
CONCURRENT_RELATIONS=4
VISION_MAX_CONCURRENT=4

A subsequent 52-doc batch on the new values processed clean: 0 × 429s, 0 × Levenshtein fallbacks, 0 × event-loop block warnings, 52/52 succeeded. Updated in .env.recommended, .env.example, README.md, .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, handbook/{03-getting-started.md, 04-configuration.md} — eight locations total. The old values remain reasonable on bigger self-hosted vLLM endpoints where you control the rate-limit policy; they were just too hot for Venice's gateway.

Recommended stack standardizes on MiniMax-M3 + Qwen3-Embedding-8B native context

The bench-validated primary is MiniMax-M3 (192K window). The recommended block in .env.recommended, .env.example, README.md, .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, and handbook/{03-getting-started.md, 04-configuration.md} reads OPENAI_MODEL=minimax-m3 / OPENAI_MAX_CONTEXT=196608. Keep MiniMax confined to the primary Q&A/research tier — it inlines <think> tokens on ingestion paths despite disable_thinking, so extraction/vision stay on Qwen3.

`EMBEDDING_MAX_INPUT_TOKENS` env var + post-chunker token-cap sub-splitter

New env knob EMBEDDING_MAX_INPUT_TOKENS (default 8192) — matches the cap Venice and OpenAI enforce at the API gateway (8192 regardless of the underlying model's native context). Two-layer protection now sits between the chunker and the embed call:

Sub-splitter — _enforce_embed_token_cap in backend/app/services/document_processor.py walks the chunker output after URL restoration. Any chunk exceeding EMBEDDING_MAX_INPUT_TOKENS × ~2.8 chars/token gets recursively split (paragraph → line → sentence → space → hard char slice) into safe pieces with the parent's meta preserved. Zero content loss; new pieces slot into chunk_index via the existing enumerate loop. Catches Docling-extracted tables and custom-input pastes without sentence boundaries — the exact inputs that previously emitted single oversize chunks.
Truncation safety net — _truncate_for_embedding (~2.8 chars/token, deliberately conservative for markdown/code/CJK) runs immediately before each OpenAIDocumentEmbedder.run() call as belt-and-suspenders. With the sub-splitter in front, it should fire essentially never on normal docs.

Together they stop the HTTP 400 "Input text exceeds the maximum token limit of N tokens" (OpenAI-direct) and litellm.ContextWindowExceededError (proxied) failures that silently dropped embeddings. Self-hosted vLLM users running Qwen3-Embedding-8B can lift the cap to 32768 to use the model's full native context; managed-provider users should leave the default.

`VISION_MIN_IMAGE_SIDE` skips sub-threshold images before the vision call

Default 64. PDFs expose ~10–40 px bullets, icons, and dingbats as Docling PictureItems; Venice (and most hosted vision APIs) reject any image with a side under ~64 px as HTTP 400 "Supplied image did not pass validation checks". Direct API probing confirmed the cutoff: 31×31, 32×32, 48×48, 56×56 all fail; 64×64 passes. Cortex now short-circuits in vision_analyzer.analyze_image_with_vision_model when min(width, height) < VISION_MIN_IMAGE_SIDE — saves API spend, eliminates the 3-retry cascade per tiny image, and removes the noisy fallback storage entries from the graph. Set 0 to disable. Format (PNG vs JPEG) was not the cause — both behave identically.

Stats endpoint no longer multiplies file_size by chunk count

GET /api/stats was reporting total_size ~70× larger than reality (23 GB shown for a 320 MB corpus). Root cause in neo4j_service.py:get_stats: the unscoped query did MATCH (d:Document) OPTIONAL MATCH (d)-[:HAS_CHUNK]->(c:Chunk) WITH … sum(d.file_size), fanning out one row per chunk so each document's file_size was summed once per chunk per doc. Fixed by hoisting the total_size aggregation before the chunk join. The scoped (per-collection) variant got a matching WITH DISTINCT d defensive fix in case docs appear in multiple collections.

Image entities now use the same embedding-based dedup as text entities

Previously, entities extracted from image descriptions always went through fuzzy-only Levenshtein 85% deduplication, even with ENABLE_SEMANTIC_ENTITY_RESOLUTION=true enabled. Text entities benefited from embedding-first dedup (catching "MOCA" ↔ "Museum of Crypto Art"); image entities did not. This left a cross-source dedup gap — an "MOCA" entity extracted from a chart caption couldn't merge with an existing "Museum of Crypto Art" text entity.

backend/app/services/neo4j_service.py:store_graph_extraction() gains an optional entity_embeddings parameter. When provided and the semantic resolution flag is on, each entity routes through store_entity_with_embedding() (embedding-first vector match, Levenshtein fallback for typos). Otherwise it falls back to the existing fuzzy-only path — no behavior change for callers that don't pass embeddings.
backend/app/services/document_processor.py:process_single_image() now batch-embeds each image's extracted entities via generate_entity_embeddings_batch_async() before storing, mirroring what the per-document text path has been doing. One embeddings call per image-with-entities (typically <10 names, very cheap). On embedding failure, the image still stores via Levenshtein with a logged warning — never aborts the document.
Image-derived entities now populate the same entity_embedding Neo4j vector index that text entities populate. Cross-source duplicates collapse at write time; the /deduplicate page's manual workflow becomes more effective because semantic matches happen during ingestion instead of being deferred. Verified end-to-end: a test document with 32 images stored 190 image entities, 100% carrying 4096-dim embeddings, zero fallback warnings.
Docs updated: .claude/domain/entities.md, .claude/domain/document-pipeline.md, handbook/13-deduplication.md, handbook/14-image-analysis.md, documentation/pages/features/knowledge-graph.mdx.

One-click Generate Graph from /documents

Clicking Generate Graph on the Documents page previously navigated to /extract and then required a second click to actually start the pipeline. The button now appends ?autostart=1; the Knowledge Graph page detects the param on mount, waits for its initial data fetch, fires handleRegenerateGraph() exactly once, and router.replaces the URL clean so a refresh won't re-fire.

frontend/src/components/DocumentList.tsx — router.push("/extract?autostart=1") instead of plain /extract.
frontend/src/app/extract/page.tsx — new useEffect reads useSearchParams().get("autostart"), guards via a hasAutoStarted ref, fires once when loading === false && documents.length > 0 && !isRegenerating.
Trigger code stays in one place on the destination page; the documents page just emits the intent via query param. The destructive-action confirm dialog inside handleRegenerateGraph is preserved — auto-start does not bypass it when entities already exist.

May 21, 2026

Reasoning Suppression Control (Provider-Agnostic)

Modern reasoning models (GPT-5/5.1, Claude 4.x, Qwen3, DeepSeek-R1, GLM, Kimi, MiniMax M3) over-think on structured-extraction tasks, inline <think> tokens that corrupt XML output, and burn token budget on chain-of-thought that the parser never sees. A new provider-agnostic switch forces reasoning OFF (or any other level) on every LLM call inside the knowledge-graph pipeline while leaving the researcher/writer Q&A path on AUTO.

New backend/app/services/reasoning_config.py — ReasoningMode enum, regex-based model-family classifier, and a per-backend dispatch table. Each backend gets the correct kwargs:
- OpenAI → reasoning_effort (none/minimal/low/…)
- OpenRouter → extra_body.reasoning.effort
- Venice → extra_body.venice_parameters.disable_thinking
- Anthropic → extra_body.thinking={"type":"disabled"} (omitted on Opus 4.7+ adaptive thinking)
- vLLM → extra_body.chat_template_kwargs.enable_thinking=false
safe_chat_completion / safe_chat_completion_sync wrappers handle a runtime fallback: if a model 400s on the reasoning param, the wrapper retries without it and caches the (base_url, model) pair so future calls skip the param upfront — one wasted call per misclassified model, then nothing.
All 14 LLM call sites in graph_extractor.py route through the wrappers — entity extraction, document summaries, community naming, entity enrichment, candidate scan, gleaning pass, per-chunk + Phase-2 relationship extraction.
Vision-model image analysis uses the same switch via a flatten_reasoning_body() helper that converts OpenAI-SDK-style extra_body into a raw-HTTP body dict (vision uses httpx, not the OpenAI client). One-shot 400-fallback works the same way.
New env vars: EXTRACTION_REASONING_MODE (default off), RELATIONSHIP_REASONING_MODE (default off), VISION_REASONING_MODE (default off), DEFAULT_REASONING_MODE (default auto — Q&A stays on provider default). Per-model escape hatch via REASONING_MODEL_OVERRIDES=model1:mode1,model2:mode2.
Defaults are no-ops for pure-instruct models (Mistral, Llama, GPT-OSS) — zero behavior change for users not on reasoning models.

Token & Context Budget Inheritance Chain

The hardcoded max_output_tokens=2000 default in per-chunk relationship extraction was collapsing Qwen3-family output (Run 03 stored 13 relationships across 77 candidate pairs because the model truncated mid-XML). Budgets are now configurable across the whole stack via a fallback chain so a 3-model deployment can be configured with just two env vars.

Inheritance chain (output tokens):

Code
 
OPENAI_MAX_OUTPUT_TOKENS=8000      (new primary default — covers Qwen3 verbose XML)
     ↓ (when EXTRACTION_*=0)
EXTRACTION_MAX_OUTPUT_TOKENS
     ↓ (when RELATIONSHIP_*=0)
RELATIONSHIP_MAX_OUTPUT_TOKENS      ← per-chunk + candidate scan
     ↓ (when VISION_*=0)
VISION_MAX_OUTPUT_TOKENS

RELATIONSHIP_BATCH_MAX_OUTPUT_TOKENS=16000   (standalone — Phase 2 batch only, NOT in chain)

Inheritance chain (input context):

Code
 
OPENAI_MAX_CONTEXT=32768
     ↓ (when EXTRACTION_*=0)
GRAPH_EXTRACTION_MAX_CONTEXT
     ↓ (when RELATIONSHIP_*=0)
RELATIONSHIP_MAX_CONTEXT

0 is the inherit sentinel for ints (consistent with MAX_FILES=0 = unlimited convention).
EXTRACTION_MAX_CONTEXT → renamed to GRAPH_EXTRACTION_MAX_CONTEXT for prefix consistency with GRAPH_EXTRACTION_MODEL / GRAPH_EXTRACTION_API_BASE. Legacy name honored as a deprecated alias for one release; a one-shot startup WARN from app.config._warn_deprecated_env_aliases nudges migration.
RELATIONSHIP_MAX_OUTPUT_TOKENS semantic split: the env var now drives per-chunk + candidate-scan in the chain; the Phase 2 batch budget moved to the new standalone RELATIONSHIP_BATCH_MAX_OUTPUT_TOKENS=16000. Users who explicitly set RELATIONSHIP_MAX_OUTPUT_TOKENS=16000 for Phase 2 should rename to the new var; the legacy value still works (per-chunk gets harmless 16000-token headroom).
Entity-extraction call sites at graph_extractor.py:821, 1014, 1840 now read settings.extraction_max_output_tokens instead of hardcoded 3000/8000. Per-chunk relationship caller in document_processor.py:1461 passes settings.relationship_max_output_tokens. Phase 2 batch caller switched to settings.relationship_batch_max_output_tokens. Vision call (vision_analyzer.py:304) reads settings.vision_max_output_tokens.
Utility-tier hardcoded budgets stay hardcoded (50/300/500/1000 for community names, entity descriptions, query-time extraction) — those are task-tuned and would harm functionality if scaled with the primary.
17-test backend/tests/test_budget_fallback.py covers all chain combinations, mid-chain overrides, Phase 2 isolation, legacy alias loading, and the deprecation WARN trigger.

Neo4j 5.26 Upgrade (4096-dim Vector Indexes)

Neo4j 5.15-community caps vector-index dimensions at 2048, blocking native use of Qwen3-Embedding-8B (4096) and other modern embedding models. Bumped to Neo4j 5.26 LTS across all compose files.

image: neo4j:5.26-community in docker-compose.yml, coolify/docker-compose.coolify.yml, dokploy/docker-compose.dokploy.yml; neo4j:5.26-enterprise in docker-compose.prod.yml.
In-place data file migration (within Neo4j 5.x the named volume neo4j_data upgrades automatically on first boot).
Python neo4j driver pin (>=5.17.0) is already forward-compatible with the new server — no driver changes needed.
APOC plugin auto-installs the matching APOC version via NEO4J_PLUGINS=["apoc"] — no separate APOC version bump required.
Existing deployments at 1536/2048-dim embeddings keep their corpora intact: Cortex preserves the vector index when EMBEDDING_DIMENSION matches the stored one. Only bumping the dim triggers an auto drop-and-recreate of the index (requires re-embedding).
New deployments can set EMBEDDING_DIMENSION=4096 directly for native Qwen3-Embedding-8B fidelity.

Recommended Minimal Stack + `.env.recommended`

The model + budget inheritance chains compress a 3-model 12-env-var stack down to a 2-model 6-env-var stack. New .env.recommended template at the project root captures the bench-validated Venice setup:

Code
 
OPENAI_API_KEY=...
OPENAI_API_BASE=https://api.venice.ai/api/v1
OPENAI_MODEL=minimax-m3
OPENAI_MAX_CONTEXT=196608

GRAPH_EXTRACTION_MODEL=qwen3-6-27b
GRAPH_EXTRACTION_MAX_CONTEXT=256000

VISION_MODEL=qwen3-6-27b

EMBEDDING_MODEL=text-embedding-qwen3-8b
EMBEDDING_DIMENSION=4096

BATCH_PROCESSING_CONCURRENCY=5
CONCURRENT_EXTRACTIONS=10
CONCURRENT_RELATIONS=5
VISION_MAX_CONCURRENT=5

The same canonical block now appears in .env.example (top-of-file Quick Start), README.md (Quick Setup), .claude/environment.md, documentation/pages/{configuration.mdx, quickstart.mdx}, and handbook/{03-getting-started.md, 04-configuration.md} — eight locations total.
Companion performance tuning (Venice-validated) block documents the concurrency knobs and warns that CONCURRENT_EXTRACTIONS is the biggest multiplier to dial down on smaller providers.
Relationship + (optionally) Vision sub-tier models inherit GRAPH_EXTRACTION_MODEL automatically; embedding API base/key inherit from OPENAI_* so no separate config is needed when Venice serves both.
Vision exception: VISION_MODEL does NOT inherit — set it explicitly or Cortex falls back to Docling's built-in SmolVLM (no API call).

Autonomous LLM-Stack Benchmark Harness

New bench/ directory — a self-contained orchestrator that cycles a fixed dataset through arbitrary LLM-model combinations against the Cortex ingestion pipeline, measuring extraction yield, relationship density, and runtime per combo. Includes a model registry, automated .env backup/restore safety, and per-run heuristics. Not yet publicly documented; CLI is internal-only for now.

Admin Settings Page Enhancements

Settings page (/admin) didn't expose the new budget knobs. The LLM Configuration section now shows:

Primary Model: added Context Window (OPENAI_MAX_CONTEXT) and Output Tokens (OPENAI_MAX_OUTPUT_TOKENS) rows
Extraction Model: added Output Tokens (EXTRACTION_MAX_OUTPUT_TOKENS)
Relationship Model: split into Output Tokens (per-chunk) and Output Tokens (batch) — reflecting the inheritance-chain + standalone split
Vision Model: added Output Tokens (VISION_MAX_OUTPUT_TOKENS)

Each new field includes a tooltip explaining its inheritance path and where it sits in the budget chain. Backend SystemConfigResponse model + /api/admin/config endpoint extended with the corresponding fields.

Instance Limits

New MAX_ENTITIES env var (default 0 = unlimited) caps the total entity count across the graph. Enforced at upload-time and custom-input creation: new ingestion is rejected once get_stats()["entity_count"] is at or above the cap. A single in-flight document can push the post-extraction count slightly above the cap (accepted tradeoff).
MAX_QUERIES_PER_MONTH instance-wide cap on chat-style queries — applies to POST /api/search, POST /api/ask, POST /api/ask/stream, POST /api/ask/stream/thinking. Returns 429 Too Many Requests with a Retry-After header (seconds until next UTC month) when exceeded.
Comprehensive test coverage in backend/tests/ for MAX_FILES, MAX_ENTITIES, MAX_COLLECTIONS enforcement across upload, custom input, collection creation, and library import endpoints.

Skill `http_request` Tool Improvements

Config variable substitution in request body: _substitute_variables() previously ran on the URL only, so skills that referenced configured values in a JSON body (e.g. "group": "ZAMMAD_GROUP_NAME") shipped the literal placeholder string instead of the substituted value. Now applied to URL + body.
Default Content-Type injection: the tool previously only injected Authorization headers from skill config schemas. Skills POST/PUT-ing JSON bodies without explicitly setting Content-Type were getting text/plain and being rejected by strict APIs. Default is now application/json when a body is present, overridable per request.
Improved researcher-agent logging: tool call inputs/outputs are now logged at INFO level with payload truncation for easier debugging of skill-driven research flows.

Design System & UI Kit

MOCA Library UI Kit: complete UI kit for the MOCA Library web app, including the ManageScreen for document management and the Shell component for top-level navigation chrome.
Design system preview files + tokens: HTML previews for drop zone, glass surface, icons, input, logo, motion, nav pill, radii, shadows, spacing scale, stats card, and other primitives. Tokens published as the canonical reference for in-product surfaces and partner integrations.

Minor Fixes

Restart-vs-recreate gotcha documented: docker compose restart does NOT re-read env_file: (env vars are baked at container create time). Added a callout to handbook/03-getting-started.md and a dedicated section in handbook/20-troubleshooting.md pointing users to docker compose up -d --force-recreate backend after any .env change.
Build resilience: logo download via curl no longer fails the frontend build when the Directus URL is unreachable — logs a warning and continues with the default logo (81d99af).
Code structure cleanup across .claude/architecture.md, .claude/design-system.md, and several services (172c1db).

April 8, 2026

Collection-Based Auth System

A comprehensive security audit remediation that closes every gap in the collection-scoped API key system. Previously, restricted keys could silently bypass collection boundaries on graph, stats, community, and task endpoints. All endpoints now enforce collection scope consistently.

Foundation fixes (neo4j_service.py)

Fixed get_document(), get_document_content(), and get_documents_file_paths() to include the Collection join — they were returning collection_id: null, causing every downstream collection access check to silently pass
Added allowed_collection_ids parameter to all graph query methods, implementing the 4-hop scoping pattern (Collection→Document→Chunk→Entity) throughout: get_graph_visualization_data(), list_entities_paginated(), get_entity_relationships(), get_graph_subgraph(), get_entity_types(), get_relationship_types(), find_entities_by_name(), list_relationships_paginated(), suggest_duplicate_entities()
Added allowed_collection_ids to community methods using a 5-hop pattern (Community→Entity→Chunk→Document→Collection): list_communities_paginated(), get_community(), search_communities_by_content()
Updated get_stats() to accept allowed_collection_ids — restricted keys now see document/entity/community counts scoped to their allowed collections only

Full endpoint authentication coverage (main.py)

All endpoints now require an X-API-Key header. Previously unprotected endpoints now enforce appropriate permission levels:

Read — GET /api/tasks, GET /api/tasks/{id}, GET /api/tasks/{id}/result
Manage — POST /api/documents/{id}/reprocess (+ per-document collection validation), POST /api/documents/reprocess (+ per-document collection check in loop), POST /api/documents/process-pending, POST /api/cleanup/orphaned-entities, PATCH /api/graph/entity/{name}, POST /api/entities/merge, POST /api/graph/relationships/analyze (+ collection_id validated if supplied), DELETE /api/graph/relationships, DELETE /api/graph/entities, POST /api/graph/communities/detect (+ collection_id validated if supplied), DELETE /api/graph/communities/{id}, DELETE /api/graph/communities, POST /api/graph/communities/summarize, DELETE /api/tasks/{id}, POST /api/tasks/cleanup, POST /api/custom-input/generate-topic Collection scoping propagated to all read endpoints

Restricted keys now receive automatically filtered results across every read endpoint — no endpoint leaks cross-collection data:

/api/stats and /api/graph/status — counts scoped to allowed collections
/api/graph/entities, /api/graph/entity-types, /api/graph/visualization, /api/graph/entity/{name}, /api/graph/entity/{name}/relationships, /api/graph/search, /api/graph/subgraph — all scoped via 4-hop pattern
/api/graph/relationships, /api/graph/relationship-types — scoped via source/target entity 4-hop filter
/api/graph/communities, /api/graph/communities/{id}, /api/graph/communities/search — scoped via 5-hop pattern
/api/entities/duplicates — entity candidates scoped to allowed collections
Researcher agent (researcher_agent.py) and query processor (document_processor.py) — allowed_collection_ids propagated through run_research_pipeline(), graph_search_async(), _execute_knowledge_search(), hybrid_search_with_graph(), and semantic_search() so Ask AI and Deep Research respect collection boundaries end-to-end

Bug fixes

Removed dead get_stats() method (neo4j_service.py) that was shadowed by the new scoped version — the dead method returned a stripped-down dict missing community_count, collection_count, pending_count and other fields, making it a silent trap for future callers
Fixed reprocess_documents and reprocess_document endpoints incorrectly using collection_id or "default" when checking collection access for documents without a collection assignment — can_access_collection(None) already returns True for restricted keys (global queries are allowed), so the fallback was incorrectly blocking access to uncollected documents

April 1, 2026

Agent Skills: Setup Wizard, http_request Tool, and Reliable Skill Execution

Setup Wizard: After installing a skill, the primary LLM analyzes the SKILL.md to extract required configuration variables (API tokens, URLs, etc.). A dynamic modal prompts the user for values — stored persistently in config.json in the skill directory. No more .env editing or container restarts to configure skills.
http_request built-in tool: The researcher agent can now call external APIs described in skill instructions. The LLM provides only method and url — authentication headers are injected server-side from the config schema's auth_header template. The LLM never touches tokens or API keys.
Auto-activation: Enabled skills are now automatically loaded at the start of every research/chat session. The full SKILL.md body (with auth-related lines stripped) is injected into the system prompt. Replaces the previous on-demand activate_skill/list_skills pattern.
Server-side auth injection: Auth headers are built from the config schema's auth_header field (e.g. Authorization: Bearer API_TOKEN, X-API-Key: API_KEY) — works for any auth pattern, any skill.
Config API endpoints: POST /api/admin/skills/{id}/analyze (LLM analysis), GET /api/admin/skills/{id}/config (schema + masked values), PUT /api/admin/skills/{id}/config (save values with mask preservation)
Config status: Skills show "Needs setup" badge when required config is missing. "Configure" button in expanded skill details on Settings page.
Skills directory persistence: Added skills_data Docker named volume to prod and Coolify compose files. Fixed path resolution bug (4→3 parent levels) that was writing skills to ephemeral container storage.
Smart API response truncation: JSON responses are intelligently slimmed (truncate long strings, flatten nested objects) to keep all entries within context budget instead of cutting mid-object.
Source dedup fix: _deduplicate_sources() no longer drops skill API responses (sources without chunk_id). Skill API data is now passed to the writer as a priority source.
Chat mode improvements: Agentic chat (ENABLE_AGENT_CHAT) now defaults to true. Speed mode gains the reasoning tool when skills are active. Max iterations bumped from 2 to 5. Retry with tool_choice=required when the model skips tools on first iteration.
Frontend: New SkillConfigModal component, "View SKILL.md" button in skill details, removed "Powered by Neo4j + Haystack" footer, chat/research views use full viewport height.
tools.json deprecated: Not part of the agentskills.io standard. The standard approach is instruction skills with API documentation + the built-in http_request tool.

Document Source Tracking

Added source field to documents — tracks the origin of each document (e.g. upload, custom_input, or a custom identifier like youtube-transcriber)
Upload endpoint (POST /api/upload) accepts optional source query parameter (defaults to upload)
Custom input endpoint (POST /api/custom-input) accepts optional source field in body (defaults to custom_input)
Document list and detail endpoints return the source field
Frontend shows source label on document cards when source is not the default upload
Source filter dropdown appears automatically in the Documents page toolbar when documents have multiple distinct sources
Existing documents are automatically backfilled on startup: upload for regular uploads, custom_input for custom inputs
Library export/import preserves the source field

March 30, 2026

Agent Skills in Library Transfer

Agent Skills are now included in library export/import: Skill nodes, SKILL.md files, and tools.json are bundled in the ZIP archive with directory_path remapping on import
Full system reset now cleans up skill nodes and skill directories on disk
Fixed vector index dimension mismatch: Neo4j vector indexes are now auto-detected and recreated when embedding dimension config changes (fixes broken search after importing a library with different dimensions)
Step 2 progress now shows only newly discovered cross-document relationships instead of total relationship count

March 28, 2026

Image Extraction Integration Improvements

Image-extracted relationships are now tagged as extraction_method="per_chunk" so they count toward within-document relationships in the Step 1 display
Image entities now use store_entity_with_resolution() for the same fuzzy dedup as text entities, instead of basic store_entity()
Page number and caption are now stored in image chunk metadata for document position tracking
Image-extracted entities are cross-linked to text chunks via fuzzy MENTIONS matching
Entity provenance (source_document_id) is now passed through store_graph_extraction() for image entities

Graph Generation Pipeline Resilience

Added retry with exponential backoff (up to 5 attempts) on all poll functions, preventing the entire 3-step pipeline from aborting on the first transient network error
Added visibilitychange listener to immediately resume polling when a browser tab becomes active (browsers throttle setTimeout in background tabs)
Active poll state is tracked via ref for reliable resume across tab switches

March 27, 2026

Agent Skills System

Integrated the open AgentSkills standard into Deep Research and Chat flows
Skills are SKILL.md files with YAML frontmatter (name, description, license, metadata) supporting two types: instruction skills (modify researcher behavior) and tool-providing skills (include tools.json with HTTP/script execution)
On-demand activation: researcher agent receives a compact skill catalog and activate_skill/list_skills tools — full instructions and tools are loaded only when the agent decides they're relevant to the query
Skills discoverable from .agents/skills/ directory on startup, installable from direct URLs or the skills.sh registry
Added SkillsManager component on Settings page for installing, enabling, disabling, and deleting skills with registry search
Skill tool calls and activations rendered in chat with Puzzle icon
New environment variables: ENABLE_SKILLS (default: true), SKILLS_DIR, ENABLE_SKILL_SCRIPTS (default: false), SKILL_SCRIPT_TIMEOUT, SKILL_HTTP_TIMEOUT, MAX_SKILL_TOOLS, MAX_SKILL_INSTRUCTIONS_TOKENS
Admin-only CRUD endpoints: GET/POST/PATCH/DELETE /api/admin/skills/*

Library Import/Export

Added full instance migration via Settings page → Data Management section
Export (POST /api/admin/export) runs as a background task building a ZIP64 archive containing 12 NDJSON data files (documents, chunks with embeddings, entities, relationships, communities, collections, merge history, system meta, and more) plus original document files
Import (POST /api/admin/import) accepts multipart ZIP upload with clean mode (requires empty instance) or replace mode (auto-wipes first)
Validates manifest, checks embedding model/dimension compatibility, remaps file paths, restores all nodes and edges including dynamic APOC relationship types
Concurrency guard prevents simultaneous export/import operations (409 conflict)
Frontend shows Export card (stats summary + progress bar + download) and Import card (mode selector + drag-and-drop upload + progress bar + result summary with warnings)

Three-Tier LLM Architecture

Added dedicated relationship model (RELATIONSHIP_EXTRACTION_MODEL, RELATIONSHIP_EXTRACTION_API_BASE, RELATIONSHIP_EXTRACTION_API_KEY) with its own API endpoint and rate limit for all relationship work (Step 1 per-chunk + Step 2 batch analysis)
Three-tier model separation: Primary (reasoning, for Q&A/research), Extraction (instruction-following, for entity extraction and community summarization), Relationship (instruction-following, for all relationship discovery)
Added get_relationship_llm_config() with fallback chain: relationship model → extraction model → primary model
Added CONCURRENT_RELATIONS (default: 3) for per-chunk relationship extraction concurrency, separate from entity extraction
Changed PARALLEL_RELATIONSHIP_BATCHES default from 0 to 5
Community summarization moved from primary model to extraction model for more reliable structured output
Added Relationship Model section to Settings page LLM Configuration between Extraction and Vision

Per-Chunk Relationship Extraction Fixes

Added tenacity retry with exponential backoff (4 attempts, 2–30s wait) for 429 rate-limit errors during per-chunk extraction
Original-to-canonical entity name mapping is now tracked during dedup, and relationship source/target are remapped before storing — fixes silent storage failures when entity names were merged during fuzzy resolution
Self-referential relationships (source == target) are now filtered at both extraction and storage
Fixed stats query to count all Entity→Entity edges, not just RELATED_TO type
Fixed per_chunk_relationship_count missing from stats API response
Step 2 rebuild now only deletes batch-analysis relationships via delete_batch_relationships(), preserving per-chunk relationships from Step 1

Knowledge Graph Page UX Improvements

Step 1 renamed to "Entity Extraction & Relationship Discovery" — now shows entity count and within-document relationship count
Step 2 renamed to "Deep Relationship Analysis" — now shows only cross-document relationship count
"Re-analyze" button renamed to "Find more"; rebuild warning clarifies that per-chunk relationships are preserved
ERR (Entity-Relationship Ratio) indicator now shows 2 decimal places for better incremental visibility
Fresh instance warning on Step 1 (0 entities) recommends "Generate Graph" instead of "Extract Entities"

March 26, 2026

Relationship Analysis Pipeline Overhaul

Two-phase per-batch analysis: Phase 1 scans entity batches for candidate pairs, Phase 2 confirms and classifies with structured XML output including confidence scores (0.0–1.0) — relationships with confidence < 0.5 are filtered before storage
Co-occurrence batching via Union-Find clustering: entities sharing chunks are grouped into the same batch for richer context; replaces O(n²) greedy BFS sort that was freezing on 32k entities; new algorithm is O(n·chunks), handles 100k+ entities
Dynamic chunk context filling: token budget split 60/40 between entities and source text; chunk context fetcher accepts a token budget and fills up to it, replacing the old fixed 10 chunks × 500 chars (~1,250 tokens out of ~44k available)
Multi-round discovery: initial analysis runs up to RELATIONSHIP_MAX_ROUNDS (default: 3) rounds, stopping early if target ERR reached or RELATIONSHIP_MAX_HOURS exhausted; "Find more" always does 1 round
Removed 5,000 entity fetch cap from get_all_entities_for_collection() — was silently ignoring 27k of 32k entities
Removed 500 existing-relationship fetch cap between rounds
Added Entity-Relationship Ratio (ERR) metric shown on Knowledge Graph page with color-coded indicator (green ≥ 0.69, yellow ≥ 0.29, red < 0.29) and tooltip
New config variables: RELATIONSHIP_TARGET_RATIO (default: 1.0), RELATIONSHIP_MAX_ROUNDS (default: 3), RELATIONSHIP_MAX_HOURS (default: 0), RELATIONSHIP_MAX_OUTPUT_TOKENS bumped from 8k to 16k

Relationship Quality and Anti-Hub Protections

Removed MENTIONS from allowed relationship types (now 14 types) — was being used as a lazy catch-all for co-occurrence
Added per-chunk relationship extraction during entity processing: chunks with 2+ entities get an LLM call using chunk text as direct evidence, stored with extraction_method='per_chunk'
Added few-shot good/bad examples to Phase 1 candidate scan and Phase 2 extraction prompts
Added anti-hub negative instructions: "If no clear relationship exists, do not create one. Co-occurrence is not a relationship."
Added RELATIONSHIP_MAX_PER_ENTITY config (default: 50) — soft cap that skips storage when both endpoints are saturated
Capped existing relationships shown to LLM at 20 per entity between rounds to prevent hub reinforcement
Reduced batch overlap from 15% to 5% with degree-aware selection (excludes entities already in 2+ batches, prefers low-connection entities)
Relationship type fuzzy-matching via rapidfuzz (80% threshold, fallback to RELATED_TO) with plaintext arrow format fallback parser

Entity Deduplication Improvements

Wired embedding-based entity dedup into the main storage path, catching semantic matches (e.g., "Museum of Crypto Art" / "MOCA") that Levenshtein misses; controlled by ENABLE_SEMANTIC_ENTITY_RESOLUTION (default: true), falls back to Levenshtein

Graph Visualization Diversity

Replaced ORDER BY mention_count visualization query with a diversity score: mention_count / (1 + log(1 + degree)) — prevents hub entities from dominating the default 100-node view
Stronger d3 force charge and link distance for better graph spacing

Bug Fixes

Fixed multi-round progress: cumulative batch counts across all rounds with global ETA instead of per-round reset
Fixed rebuild flag not passed through to analyze_collection_relationships, causing rebuild mode to run only 1 round instead of RELATIONSHIP_MAX_ROUNDS
Fixed login white screen by moving redirect() out of server action into client-side useEffect

March 25, 2026

Bulk Document Download

Added POST /api/documents/download-zip endpoint that accepts a list of document IDs, builds a ZIP64-enabled archive of original files with duplicate filename disambiguation, and streams the response in 1MB chunks
Added Download button to the bulk actions toolbar on the Documents page — select documents and download them as a single ZIP file
Handles 1000+ files with ZIP64 support

March 20, 2026

Parallel Image Analysis Within Documents

Images within a single document are now analyzed concurrently using asyncio.gather, replacing the previous sequential for-loop
Added VISION_MAX_CONCURRENT environment variable (default: 3) to control the maximum number of concurrent vision API calls system-wide
Thread pool sizes for image processing now scale automatically with VISION_MAX_CONCURRENT
Fixed "Event loop is closed" errors during concurrent image analysis by using thread-local HTTP clients
For a document with 200 images at ~30s per image: processing time reduced from ~100 minutes (sequential) to ~10 minutes (with VISION_MAX_CONCURRENT=10)

March 18, 2026

Entity Deduplication

Added GET /api/entities/duplicates endpoint to find duplicate entity candidates using fuzzy name similarity, with configurable threshold (0.5-1.0) and limit parameters
Added POST /api/entities/merge endpoint to merge duplicate entities into a canonical entity, transferring all relationships, chunk mentions, and community memberships; merged entity names are stored as aliases on the canonical entity
Added GET /api/entities/merge-history endpoint to view a chronological log of past merge operations
Updated Python client class with find_duplicate_entities(), merge_entities(), and get_merge_history() methods
Added cURL examples for all three entity deduplication endpoints
Updated OpenAPI specification with new endpoints and schemas

March 17, 2026

Two-Phase Graph Extraction Pipeline

Refactored entity extraction to work per-document instead of per-chunk, reducing LLM calls from N (one per chunk) to 1-4 (batched by token budget) per document
Added dedicated graph extraction model configuration (GRAPH_EXTRACTION_MODEL, GRAPH_EXTRACTION_API_BASE, GRAPH_EXTRACTION_API_KEY) to allow using a smaller/faster model for entity extraction
Entity-to-chunk linking now uses fuzzy string matching (rapidfuzz) to create MENTIONS relationships
Added entity provenance tracking: source_documents, extraction_count, last_extracted_at properties on Entity nodes
New Phase B relationship analysis endpoint: POST /api/graph/relationships/analyze - discovers cross-document relationships using the main (large) model
Relationship analysis runs as a background task with progress tracking via /api/tasks/{id}
Added relationship provenance: extracted_at, extraction_method, source_document_id properties on relationships
New auto-trigger options: AUTO_RELATIONSHIP_ANALYSIS_AFTER_BATCH and AUTO_COMMUNITY_DETECTION_AFTER_BATCH for automated pipeline after batch processing

Community Deletion and Graph Cleanup

Added DELETE /api/graph/communities/{id} endpoint to delete a specific community (entities are unlinked but preserved)
Added DELETE /api/graph/communities endpoint to delete all communities at once
Enhanced POST /api/cleanup/orphaned-entities to also clean orphaned communities in one call
Added delete buttons (individual and "Delete All") to the Explore page communities panel and the Collections page community section
StatsBar now shows an orphaned data warning banner with a cleanup button when entities/relationships exist but no documents are present
Frontend API client now includes deleteCommunity(), deleteAllCommunities(), and cleanupOrphanedEntities() methods

March 16, 2026

Ask AI API Endpoint Update

Renamed API endpoints from /api/rag to /api/ask across all examples and documentation for improved clarity and consistency
Updated configuration settings for document processing and community detection endpoints

March 5, 2026

Large PDF Processing Improvements

Added memory optimizations for handling large PDF files, including chunked processing and backend unloading to prevent out-of-memory errors
Introduced PAGE_CHUNK_SIZE and MAX_PAGES_PER_CHUNK environment variables for configuring chunked PDF processing
Integrated pypdf for lightweight PDF page counting
Added fallback to PyPdfium for large files when the primary converter encounters memory constraints
Fixed page index conversion in the Docling worker to correctly handle 1-based page indices

Image Analysis Execution Optimization

Implemented a synchronous version of the image analysis method in VisionAnalyzer to allow thread pool execution, preventing the main event loop from being blocked
Updated document converter to support image format options for more flexible input handling
Updated Dockerfile with necessary system dependencies for PDF/image processing, including X11 libraries and Tesseract OCR

March 4, 2026

Docling Integration and Advanced Document Processing

Integrated Docling as the primary document conversion engine, replacing the previous processing pipeline
Added a dedicated thread pool for image analysis to optimize performance and prevent competition with document processing tasks
Implemented subprocess-based document conversion for improved efficiency with large files
Added event loop watchdog to monitor and log potential blocking issues
Enhanced VisionAnalyzer with retry logic for API calls
New fields for image analysis progress in the Document model, with frontend components to display processing status
Added configuration options for vision models in the environment setup
Increased upload body size limit to 100MB

Logging and Robustness Improvements

Suppressed Neo4j notification warnings by adjusting the logging level
Added a function to clean image placeholders in document processing
Conditionally configured OCR and image description based on vision model availability
Enhanced GraphExtractor to handle potential None responses from document summary generation
Refactored Neo4jService queries to use coalesce for better handling of missing data
Refined default analysis prompt in VisionAnalyzer for improved clarity and document retrieval purposes

March 2, 2026

Image Extraction and Vision Analysis

Added a new vision analyzer service that extracts images from documents (PDF, DOCX, PPTX) and analyzes them using configurable vision models
Supports OpenAI-compatible APIs with fallback to Docling's built-in image descriptions
Image analyses are integrated into the RAG pipeline for searchable visual content
Enabled Docling's built-in image description generation (do_picture_description) as an additional source
Added EasyOCR for enhanced image text extraction
Updated Dockerfile with Tesseract OCR and additional libraries for improved document processing

Enhanced File Upload Support

Expanded allowed file extensions to support a wider range of document types including additional document and media formats
Updated frontend FileUpload and UploadZone components to reflect the expanded file type support
Updated file type icons for better visual representation of supported formats

February 27, 2026

Document Processing Engine Overhaul

Updated Dockerfile to include necessary system dependencies for Docling, enabling PDF and image processing
Added docling-haystack dependency for enhanced document handling
Expanded allowed file extensions in configuration to support a wider range of document types
Refactored document processor to utilize the new Docling converter, streamlining handling of various document formats

February 25, 2026

Customizable Branding and Theming

Added support for custom logo configuration via the CUSTOM_LOGO_URL environment variable, allowing white-label deployments
Introduced custom accent color support via the CUSTOM_ACCENT_COLOR environment variable for theme customization
Added configurable branding options (site title, description) through environment variables
Increased logo dimensions for improved sharpness on high-DPI screens
Documented all frontend customization variables in the deployment guide

February 6, 2026

API Key Usage Tracking and Analytics

Introduced a dedicated API usage tracking service that records every request per API key, categorized by endpoint type (ask, search, upload, documents, graph, collections, admin, etc.)
Added middleware to automatically track API requests and errors for all authenticated endpoints
Admin API key records are now persisted in Neo4j for long-term usage statistics
New backend models for API key usage statistics, usage data points, and usage history

Admin Dashboard Improvements

Added an API Key Analytics panel with interactive usage charts showing request volume over time and endpoint breakdown
New API Key Manager component for creating, viewing, and managing API keys directly from the admin UI
Introduced a System Reset Modal with granular deletion options (documents, uploaded files, custom inputs, collections, API keys) and a confirmation safeguard
Admin page now displays system configuration overview including API key usage settings
Updated Cortex logo assets (dark and light variants)

Resource Limits

Added MAX_FILES and MAX_COLLECTIONS environment variables to cap the number of documents and collections (set to 0 for unlimited)
Backend now enforces these limits on file uploads, custom input creation, and collection creation, returning a 403 error when limits are exceeded

Collection-Scoped AI Queries

The Ask AI feature now supports scoping queries to a specific collection via collection_id, so answers are drawn only from documents in that collection
Collection scoping works across all ask modes: standard, streaming, streaming with thinking, and fast search
Ask AI settings (streaming, agentic mode, fast search, selected collection) are now persisted to localStorage so preferences survive page reloads

Skill Library Restructure

Renamed the skill package from openclaw-library to library
Added sync scripts (Python, Shell, and JavaScript) for automated skill file synchronization
Removed obsolete OpenClaw library skill files

February 5, 2026

Document Deletion with Task Cancellation

Deleting a document now automatically cancels any active processing task for that document before removal, preventing orphaned background jobs
Introduced proper task tracking for active document processing with cancellation flag support
Added cancel_multiple_documents and improved cancel_processing_task methods in the DocumentProcessor with deadlock-safe locking
Processing tasks are now started with cancellation flags initialized upfront, ensuring graceful shutdown from the very first checkpoint

Knowledge Graph Cleanup on Deletion

Document deletion now triggers a comprehensive cleanup of the knowledge graph, removing orphaned entities and communities that are no longer referenced by any remaining document
API responses for deletion endpoints now include detailed cleanup statistics (entities removed, communities removed, etc.)
Updated documentation pages for Document Upload and Knowledge Graph features to explain the new deletion and cleanup behavior

API Parameter Changes

collection_id and start_processing parameters on the upload endpoint are now passed as URL query parameters instead of form data fields
Updated API examples in the documentation (cURL and Python) to reflect the new parameter format
Skill documentation updated to emphasize that both api_key and base_url credentials are required
Skill version bumped to 1.2.0

February 4, 2026

Admin Authentication and API Key Management

Implemented a full admin authentication system with email/password login validated against environment variables (ADMIN_EMAIL, ADMIN_PASSWORD)
Added session management with encrypted JWT tokens stored in HTTP-only cookies
Introduced Next.js middleware to protect all routes (except /login) behind authentication
Created a dedicated login page with form validation and error handling, wrapped in React Suspense for smooth loading states

Permission-Based API Protection

All API endpoints are now protected with a three-tier permission system: read, manage, and admin
Read endpoints (search, ask, list documents, stats) require at minimum a valid API key with read permission
Write endpoints (upload, delete, create collections) require manage permission
Admin endpoints (API key CRUD, system management) require admin-level access
New backend services: auth_service for request authentication and api_key_service for key lifecycle management

Documentation Site Launch

Launched the full documentation site built with Zudoku, featuring:
- Getting Started guides (Introduction, Quickstart, Configuration)
- Core Features documentation (Document Upload, Search, Ask AI, Knowledge Graph, Collections, Communities)
- Guides (Deployment, Authentication, Security)
- Code examples (Python, cURL, Integration)
- OpenAPI-powered API Reference
- Pagefind-based full-text search
- LLM-friendly output (llms.txt and llms-full.txt)
Created comprehensive backend API documentation (BACKEND_API_DOCUMENTATION.md)
Cleaned up documentation pages by removing redundant section headers for a streamlined reading experience

January 29, 2026

Rebrand to Cortex

Renamed the project from MOCA Knowledge Base to Cortex
Updated the tagline to "The Agentic Knowledge Base for the AI Era"
Added a new "What is Cortex?" section to the README explaining the project's philosophy around portable, long-term AI memory
Introduced the memory hierarchy concept: Context (short-term) → Agent Memory Stack (mid-term) → Cortex (long-term)

Last modified on September 4, 2026