A company brain for AI agents
cortex
Company knowledge extraction platform that turns scattered data into AI-executable workflows.
- Automated tests
- 283
- Query latency
- ~32ms
- Ingestion
- Idempotent, multi-source
the problem
Company knowledge lives everywhere: docs, wikis, tickets, chat threads. When you want an AI agent to actually do something with that knowledge, raw text is not enough. The agent needs structured, verified workflows it can execute, and it needs them fast. Most RAG setups stop at retrieval and leave the hard part (turning retrieval into reliable action) unsolved.
the solution
Cortex ingests data from multiple sources with idempotent, deduplicated pipelines, then runs unsupervised extraction over the corpus. HDBSCAN clustering over vector embeddings surfaces the most relevant knowledge clusters upfront, so queries resolve in about 32ms through local semantic search instead of a round trip to a hosted vector service. Extracted workflows pass through a deterministic, agent-based safety validator that automatically flags anything risky that lacks an approval gate. The result is a knowledge base that agents can execute against, not just quote from.
tech stack
- Python + FastAPICore API and extraction pipeline
- PostgreSQLSource of truth for structured workflows and ingestion state
- ChromaDBLocal vector store for low-latency semantic search
- HDBSCANUnsupervised clustering over embeddings to surface knowledge clusters
- Groq / Llama 3.3LLM inference for extraction and workflow synthesis
- RedisMessage broker for Celery background ingestion and extraction jobs
- ReactFrontend for browsing clusters and reviewing flagged workflows
key engineering decisions
Local vector search over a hosted service
Keeping ChromaDB local removed a network hop from every query. That is most of how query latency landed around 32ms. The tradeoff is managing persistence ourselves, which the ingestion layer's idempotency made manageable.
Idempotent ingestion with cross-path deduplication
The same document often arrives through multiple sources. Every ingestion path computes stable content hashes, so re-runs and overlapping sources never create duplicate knowledge. This made the pipeline safe to re-trigger blindly, which simplified operations a lot.
Deterministic safety validation, not LLM-judged safety
Asking an LLM whether a workflow is safe gives you probabilistic answers to a question that needs a guaranteed one. The validator is rule-based and deterministic: workflows touching sensitive operations without an approval gate get flagged, every time.
Testing as a first-class deliverable
283 automated tests cover ingestion idempotency, clustering behavior, and validator rules. For a pipeline whose whole value is reliability, the test suite is the product as much as the code is.