Core Features

Web Import (MDHarvest powered by Crawl4ai)

Copy page

Web Import turns web pages into clean markdown in your knowledge base. Paste a list of URLs (or discover the links on a page and pick which ones to pull), and Cortex crawls each one, extracts the readable content, and ingests it — chunked, embedded, and run through entity/relationship extraction so it shows up in Search, Ask AI, and the knowledge graph. Pages from the same site are aggregated into one document per domain, so a site's subpages read as a single related work (e.g. an artist named only on the About page is tied to their exhibitions and press pages that never repeat the name).

The crawling itself is done by crawl4ai, an open-source, LLM-oriented web crawler. Cortex never embeds a browser of its own — it talks to a crawl4ai service over HTTP. That means you run crawl4ai once and point Cortex at it.

Enable it with ENABLE_WEB_CRAWL=true and a CRAWL_SERVICE_URL. A Web Import option then appears in the menu next to the Upload button on the Documents page. If either is unset, the feature stays hidden.

Using Web Import

1. Go to the Documents page and click the small ▾ arrow next to the Upload button, then choose Web Import. A modal opens.
2. Pick a collection to import into.
3. Paste one or more URLs (one per line) — or expand Discover links: enter a single page URL and Cortex returns the same-site links it finds, each with a checkbox, so you can add just the ones you want to the list.
4. Pick a content filter:
   • Readable (recommended) — the main article content, stripped of nav/ads/boilerplate.
   • Full page — the entire page converted to markdown.
   • Relevance-ranked — keeps the passages most relevant to a query you provide.
5. Click Import from Web. A progress bar tracks the crawl.
6. As soon as the pages are staged, the modal shows how many pages were imported, grouped into how many documents (and any that failed). Close it — the new documents are already on the Documents page, where they keep processing in the background (text extraction + knowledge-graph build). You don't have to wait on the modal; watch progress on the Documents page.

Each site becomes one document, titled and filed by its domain (e.g. example.com.md), with a provenance header and one section per page so you always know where each part came from:

Code
 
# example.com

> Source site: example.com
> Pages: 3
> Extracted: 2026-07-22

---

## Page title
> Source: https://example.com/the-page

…clean markdown…

(A single-page import uses a simpler header — # example.com + > Source: … — but is still filed by the domain.)

Running crawl4ai (self-hosting)

If you self-host Cortex, run your own crawl4ai container and point Cortex at it. The simplest setup:

Code
 
docker run -d --name crawl4ai \
  -p 11235:11235 \
  --shm-size=1g \
  -e CRAWL4AI_API_TOKEN=your-shared-token \
  unclecode/crawl4ai:0.9.0

Then set in your Cortex backend .env:

Code
 
ENABLE_WEB_CRAWL=true
CRAWL_SERVICE_URL=http://crawl4ai:11235     # or http://<host>:11235
CRAWL_SERVICE_TOKEN=your-shared-token       # required for crawl4ai >= 0.9.0

CRAWL_SERVICE_URL should be reachable from the Cortex backend container — use the service name if they share a Docker network, or the host's IP/port otherwise. For crawl4ai ≥ 0.9.0 the token is effectively mandatory: without CRAWL4AI_API_TOKEN crawl4ai binds its API to 127.0.0.1 only (unreachable from the Cortex container). Set the same value as CRAWL_SERVICE_TOKEN (Cortex sends it as Authorization: Bearer …). If you reach crawl4ai by a bare host IP rather than a Docker service name, add that IP to crawl4ai's trusted_hosts (its TrustedHostMiddleware returns 400 "Invalid host header" otherwise).

That single crawl4ai instance can serve as many Cortex backends as you like — it's stateless, so you don't need one per Cortex.

Tuning

A few backend settings control behaviour (see the Configuration reference):

Setting	Default	What it does
CRAWL_CONTENT_FILTER	fit	Default content filter (fit / raw / bm25).
CRAWL_CONCURRENCY	5	How many URLs in one job are crawled at once.
CRAWL_MAX_URLS_PER_JOB	100	Maximum URLs accepted per import.
CRAWL_HTTP_TIMEOUT	60	Per-page crawl timeout (seconds).
CRAWL_DISCOVER_MAX_LINKS	200	Cap on links returned by Discover.

Privacy on shared crawl4ai instances

If several independent Cortex deployments share one crawl4ai (e.g. in a hosted, multi-customer setup), crawl4ai is configured so that no crawl history is retained and no customer can see what another crawled: its storage is ephemeral, Cortex only uses request-scoped endpoints (never the async job API), and every request bypasses the cache. Operators should additionally keep crawl4ai on a private network with a bearer token and block crawls to internal addresses. See the cortex-helper repository for the hardened shared-host setup.