Core Features

Communities

Copy page

Cortex automatically detects communities - clusters of related documents and entities that share common themes. This enables topic discovery, document organization, and focused Q&A.

What Are Communities?

Communities are groups of entities (and their associated documents) that are densely connected in the knowledge graph. They represent natural topic clusters within your knowledge base.

Detection Algorithm

Cortex uses Leiden (preferred) or Louvain algorithm for community detection, with a BFS fallback when GDS is unavailable:

Key improvements in the detection pipeline:

• Old community cleanup before re-detection prevents stale data accumulation
• Undirected projection treats all relationships as bidirectional for community membership
• Relationship weights (0-10 scale) influence community membership — strongly weighted relationships pull entities together
• Co-mention edges connect entities that appear in the same chunk, providing structure even when direct entity-to-entity relationships are sparse
• Distribution monitoring logs warnings for pathological distributions (e.g., one mega-community containing >50% of entities)

API Usage

Detect Communities

Start community detection (runs as a background task):

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/detect" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "collection_id": "default",
    "min_community_size": 3
  }'

Response:

Code
 
{
  "task_id": "task_abc123",
  "status": "pending",
  "message": "Community detection started"
}

Track Detection Progress

Code
 
curl "http://localhost:8000/api/tasks/task_abc123" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "task_id": "task_abc123",
  "task_type": "community_detection",
  "status": "running",
  "progress_percent": 45.0,
  "message": "Analyzing graph structure..."
}

List Communities

Code
 
curl "http://localhost:8000/api/graph/communities" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "communities": [
    {
      "id": "comm_1",
      "name": "Machine Learning Fundamentals",
      "description": "Core ML concepts including neural networks, optimization, and training methods",
      "document_count": 12,
      "entity_count": 156,
      "top_entities": ["Neural Network", "Deep Learning", "Gradient Descent"]
    },
    {
      "id": "comm_2",
      "name": "Natural Language Processing",
      "description": "NLP techniques, transformers, and language models",
      "document_count": 8,
      "entity_count": 98,
      "top_entities": ["Transformer", "BERT", "Attention"]
    }
  ],
  "total": 23
}

Get Community Details

Code
 
curl "http://localhost:8000/api/graph/communities/comm_1" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "id": "comm_1",
  "name": "Machine Learning Fundamentals",
  "description": "Core ML concepts including neural networks, optimization, and training methods",
  "summary": "This community covers the foundational aspects of machine learning, with a focus on neural network architectures, training algorithms, and optimization techniques...",
  "document_count": 12,
  "entity_count": 156,
  "entities": [
    {"name": "Neural Network", "type": "Concept", "mentions": 45},
    {"name": "Deep Learning", "type": "Concept", "mentions": 38}
  ],
  "documents": [
    {"id": "doc_1", "title": "Deep Learning Fundamentals.pdf"},
    {"id": "doc_2", "title": "Neural Network Training.pdf"}
  ]
}

Get Community Documents

Code
 
curl "http://localhost:8000/api/graph/communities/comm_1/documents" \
  -H "X-API-Key: your-api-key"

Summarize a Community

Generate an AI summary of a community's content:

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/comm_1/summarize" \
  -H "X-API-Key: your-api-key"

Code
 
{
  "community_id": "comm_1",
  "summary": "This community contains 12 documents focused on machine learning fundamentals. Key topics include:\n\n1. Neural network architectures (CNNs, RNNs, Transformers)\n2. Training algorithms and optimization (SGD, Adam, learning rate scheduling)\n3. Regularization techniques (dropout, batch normalization)\n\nThe documents span from introductory material to advanced research papers.",
  "key_topics": ["Neural Networks", "Optimization", "Regularization"],
  "generated_at": "2024-01-15T10:30:00Z"
}

Community-Based Queries

Search Within a Community

Code
 
curl -X POST "http://localhost:8000/api/search" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "backpropagation algorithm",
    "community_id": "comm_1"
  }'

Ask About a Community

Code
 
curl -X POST "http://localhost:8000/api/ask" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What are the key topics covered in this community?",
    "collection_id": "comm_1"
  }'

Regenerating Communities

Communities are not automatically updated when documents are added. To update:

Code
 
curl -X POST "http://localhost:8000/api/graph/communities/detect" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "force_regenerate": true
  }'

Configuration

Code
 
# Enable community detection
ENABLE_COMMUNITY_DETECTION=true

# Minimum entities per community
MIN_COMMUNITY_SIZE=3

# Maximum communities to detect
MAX_COMMUNITIES=50

# Enable automatic summarization
# (names/summaries are generated by the extraction model, GRAPH_EXTRACTION_MODEL)
ENABLE_GRAPH_SUMMARIZATION=true

Use Cases

1. Topic Discovery - Find natural topic clusters in your documents
2. Content Organization - Use communities to organize collections
3. Focused Q&A - Ask questions about specific topic areas
4. Knowledge Mapping - Visualize how topics relate
5. Onboarding - Help new users understand the knowledge base