Core Features

Communities

Copy page

Cortex automatically detects communities - clusters of related documents and entities that share common themes. This enables topic discovery, document organization, and focused Q&A.

What Are Communities?

Communities are groups of entities (and their associated documents) that are densely connected in the knowledge graph. They represent natural topic clusters within your knowledge base.

Detection Algorithm

Cortex uses Leiden (preferred) or Louvain algorithm for community detection, with a BFS fallback when GDS is unavailable:

Key improvements in the detection pipeline:

• Old community cleanup before re-detection prevents stale data accumulation
• Undirected projection treats all relationships as bidirectional for community membership
• Relationship weights (0-10 scale) influence community membership — strongly weighted relationships pull entities together
• Co-mention edges connect entities that appear in the same chunk, providing structure even when direct entity-to-entity relationships are sparse
• Distribution monitoring logs warnings for pathological distributions (e.g., one mega-community containing >50% of entities)

API Usage

Detect Communities

Start community detection (runs as a background task):

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/detect" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "collection_id": "default",
    "min_community_size": 3
  }'

Response:

Code
 
{
  "task_id": "task_abc123",
  "status": "pending",
  "message": "Community detection started"
}

Track Detection Progress

Code
 
curl "http://localhost:8000/api/tasks/task_abc123" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "task_id": "task_abc123",
  "task_type": "community_detection",
  "status": "running",
  "progress_percent": 45.0,
  "message": "Analyzing graph structure..."
}

List Communities

Code
 
curl "http://localhost:8000/api/graph/communities" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "communities": [
    {
      "id": 1,
      "name": "Machine Learning Fundamentals",
      "description": "Core ML concepts including neural networks, optimization, and training methods",
      "document_count": 12,
      "entity_count": 156,
      "top_entities": ["Neural Network", "Deep Learning", "Gradient Descent"]
    },
    {
      "id": 2,
      "name": "Natural Language Processing",
      "description": "NLP techniques, transformers, and language models",
      "document_count": 8,
      "entity_count": 98,
      "top_entities": ["Transformer", "BERT", "Attention"]
    }
  ],
  "total": 23
}

Get Community Details

Community IDs are integers. The detail endpoint returns the community's summary, member entities, and key relationships (filtered to the API key's allowed collections):

Code
 
curl "http://localhost:8000/api/graph/communities/1" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "id": 1,
  "name": "Machine Learning Fundamentals",
  "description": "Core ML concepts including neural networks, optimization, and training methods",
  "summary": "This community covers the foundational aspects of machine learning, with a focus on neural network architectures, training algorithms, and optimization techniques...",
  "document_count": 12,
  "entity_count": 156,
  "entities": [
    {"name": "Neural Network", "type": "Concept", "mentions": 45},
    {"name": "Deep Learning", "type": "Concept", "mentions": 38}
  ],
  "documents": [
    {"id": "doc_1", "title": "Deep Learning Fundamentals.pdf"},
    {"id": "doc_2", "title": "Neural Network Training.pdf"}
  ]
}

Community Members

The detail endpoint above returns the community's member entities (and their key relationships). There is no per-community documents listing — browse documents per collection instead, e.g. GET /api/documents?collection_id=<collection_id>.

Summarize Communities

Generate or regenerate AI summaries for communities (batch endpoint; requires manage permission):

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/summarize" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "community_ids": [1],
    "force_regenerate": true
  }'

Omit community_ids to summarize all communities; omit force_regenerate (or set it false) to skip communities that already have a summary.

Code
 
{
  "results": [
    {
      "id": 1,
      "status": "summarized",
      "name": "Machine Learning Fundamentals",
      "summary": "This community covers the foundational aspects of machine learning, with a focus on neural network architectures, training algorithms, and optimization techniques."
    }
  ],
  "total_processed": 1
}

Community-Based Queries

Communities are not a scoping parameter on the search or ask endpoints:

• POST /api/search accepts no community filter — SearchRequest scopes by collection_id only. A community field in the body is ignored: it adds no community restriction, and the search remains scoped exactly as it would be without it — the supplied collection_id, or the API key's collection scope when no collection_id is supplied.
• collection_id on /api/search and /api/ask selects a Collection (Collection.id) only. A community identifier such as comm_1 matches no collection, and community IDs are integers — passing one as a string collection scope finds nothing, while a numeric value fails the string field with 422.

Use the supported paths below.

Search Community Summaries

Search communities by their summary content (results are filtered to the API key's allowed collections):

Code
 
curl "http://localhost:8000/api/graph/communities/search?query=backpropagation" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "query": "backpropagation",
  "results": [
    {
      "id": 1,
      "name": "Machine Learning Fundamentals",
      "summary": "Core ML concepts including neural networks, optimization, and training methods",
      "entity_count": 156,
      "score": 3.42
    }
  ]
}

Ask with Community Context

There is no community-scoped ask. On agentic (deep research) queries, the researcher agent can retrieve relevant community summaries into the answer context with its community_search tool, scoped to the request's collection_id (or the API key's allowed collections) — tool use is model-chosen per run, not a guaranteed inclusion. To focus an ask, scope it to a real collection ID:

Code
 
curl -N -X POST "http://localhost:8000/api/ask/stream" \
  -H "X-API-Key: your-api-key" \
  -H "Accept: text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What are the key topics covered in this collection?",
    "depth": "deep",
    "collection_id": "default"
  }'

(legacy equivalent: "use_agentic": true in place of depth).

To read one community's summary and members directly, use GET /api/graph/communities/{community_id} with the community's integer ID.

Regenerating Communities

Communities are not automatically updated when documents are added. To update:

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/detect" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "force_regenerate": true
  }'

Configuration

Code
 
# Enable community detection
ENABLE_COMMUNITY_DETECTION=true

# Minimum entities per community
MIN_COMMUNITY_SIZE=3

# Maximum communities to detect
MAX_COMMUNITIES=50

# Enable automatic summarization
# (names/summaries are generated by the extraction model, GRAPH_EXTRACTION_MODEL)
ENABLE_GRAPH_SUMMARIZATION=true

Use Cases

1. Topic Discovery - Find natural topic clusters in your documents
2. Content Organization - Use communities to organize collections
3. Focused Q&A - Ask questions about specific topic areas
4. Knowledge Mapping - Visualize how topics relate
5. Onboarding - Help new users understand the knowledge base