Submit projectSubmit

TOPIC GUIDE

Vector databases and RAG infrastructure

Pinecone, Qdrant, Weaviate, Milvus, Chroma and turbopuffer, plus tools that prepare data for retrieval.

products
58
things to compare
3
updated
2026-10-31

Retrieval-augmented generation needs three pieces: getting clean text out of documents and web pages, storing embeddings, and retrieving the right passages. Start with data preparation, since retrieval quality is usually limited by messy inputs before the database matters.

What to compare

  1. Decide between a managed service and self-hosting before comparing features.
  2. Test retrieval quality on your own documents and questions, not benchmark datasets.
  3. Estimate cost at your expected data size and query volume, including minimum monthly charges.

This is a focused selection, not an exhaustive ranking. See each profile for evidence and review scope.

Projects to explore

Pinecone

A managed vector database and knowledge platform for semantic search and retrieval in AI applications.

Read the project profile

Qdrant

An open-source vector search engine and database, available self-hosted or as Qdrant Cloud.

Read the project profile

Weaviate

An AI-native database with vector and hybrid search, available as source code to self-host or as Weaviate Cloud.

Read the project profile

Milvus

An open-source vector database for large-scale similarity search, also offered as managed Zilliz Cloud.

Read the project profile

Chroma

Open-source search infrastructure for AI with vector, full-text and metadata search, available locally or as Chroma Cloud.

Read the project profile

turbopuffer

A vector and full-text search engine built on object storage, designed to be cheaper and highly scalable.

Read the project profile

Unstructured

A platform that turns PDFs, documents and 64+ other file types into clean, structured inputs for RAG and AI applications.

Read the project profile

Firecrawl

A web data API for AI agents that searches, scrapes and crawls websites into clean markdown or structured data.

Read the project profile

LlamaIndex

AI agents for document parsing (LlamaParse) and workflows, alongside its framework for building on your data.

Read the project profile

Mem0

A memory layer for AI agents and apps that stores and retrieves what users said before, available as open source or a hosted platform.

Read the project profile

Jina AI

Embeddings, rerankers and a Reader API that turns web pages into Markdown for LLM search and grounding; now part of Elastic.

Read the project profile

Cohere

Enterprise AI models for generation, embeddings, reranking and speech, plus North, a secure workspace for agents and search.

Read the project profile

Docling

An open-source toolkit that converts PDFs, Office files, HTML, images, audio and more into structured data for AI, started at IBM Research.

Read the project profile

Crawl4AI

An open-source, LLM-friendly web crawler that turns any URL into clean Markdown or typed JSON, also available as a hosted API.

Read the project profile

RAGFlow

An open-source RAG engine with an integrated agent platform for giving AI agents reliable context from your documents.

Read the project profile

Graphiti

Zep's open-source framework for building temporal knowledge graphs that track how facts change over time, as memory for AI agents.

Read the project profile

Marker

Datalab's open-source tool for converting PDFs to Markdown and JSON quickly and accurately, including tables and layout.

Read the project profile

MarkItDown

Microsoft's lightweight Python tool for converting PDFs, Office documents and other files to Markdown for LLMs and text analysis.

Read the project profile

Onyx

An open-source enterprise search and AI chat platform that connects to company data and runs in your own cloud.

Read the project profile

txtai

An all-in-one open-source framework for semantic search, LLM orchestration and language model workflows, built around an embeddings database.

Read the project profile

Unstract

An open-source, no-code platform for extracting structured data from unstructured documents with LLMs, deployed as APIs or ETL pipelines.

Read the project profile

MinerU

An open-source document parser from OpenDataLab that converts PDFs, Office files, images and more into Markdown and JSON for AI.

Read the project profile

PrivateGPT

An open-source project for private, local question answering over your documents, maintained by the team behind Zylon.

Read the project profile

Kapa

An AI knowledge retrieval service that connects documentation and support content to assistants through prebuilt agents, an API, or MCP.

Read the project profile

Cognee

An agent memory toolkit that organizes documents, code and past sessions into connected knowledge for later retrieval.

Read the project profile

PaperQA

A Python package that indexes scientific documents, retrieves supporting passages and answers questions with source citations.

Read the project profile

SurfSense

A desktop research workspace that answers questions over local documents with citations and turns source material into reports, slides and podcasts.

Read the project profile

Reducto

A document processing platform with APIs that parse layouts, extract structured fields and prepare content for AI applications.

Read the project profile

LanceDB

A multimodal data platform and embedded retrieval library for storing and searching vectors, text and other AI data.

Read the project profile

Morphik Core

A multimodal retrieval platform for ingesting, searching and extracting information from visually rich documents through an API, Python SDK and web console.

Read the project profile

Zep

A hosted context and memory layer that turns conversations, documents and business data into temporal context graphs for AI agents.

Read the project profile

Chonkie

A Python ingestion library that chunks documents, refines passages, adds embeddings and connects the results to vector databases for RAG pipelines.

Read the project profile

DocETL

A Python and YAML toolkit that uses LLM map, filter and reduce operations to turn document collections into structured tables.

Read the project profile

DataChain

A Python library for creating typed, versioned datasets from unstructured files and enriching them with models for search and agent workflows.

Read the project profile

ScrapeGraphAI

An AI web-scraping API that turns natural-language extraction requests into structured data, with a separate self-hosted Python library.

Read the project profile

LightRAG

A retrieval framework that combines knowledge graphs and vector embeddings, with a server, REST API and web interface for indexing and querying documents.

Read the project profile

Meilisearch

A search engine combining full-text and semantic retrieval, vector storage, filtering and faceting for application search and RAG.

Read the project profile

Vespa

A search platform combining text, vector and structured retrieval with machine-learned ranking, model inference and real-time data updates.

Read the project profile

FastEmbed

A Python library that uses ONNX Runtime to generate dense, sparse and image embeddings and rerank search results.

Read the project profile

Xberg

A document-intelligence engine with a Rust core, language bindings, OCR, structured extraction, embeddings and CLI, REST and MCP interfaces.

Read the project profile

DB-GPT

A self-hosted AI data assistant that connects to databases and files, generates SQL and Python analysis, and produces charts and reports.

Read the project profile

MemMachine

An open-source memory layer that stores conversational history and user preferences for AI agents across sessions.

Read the project profile

Hindsight

An agent memory system that extracts facts, retrieves relevant context and reasons over stored knowledge with retain, recall and reflect APIs.

Read the project profile

docTR

An Apache-2.0 Python OCR library on PyTorch that finds and reads words in PDFs and images, with optional layout detection and a key-information extraction predictor.

Read the project profile

llmware

An open-source Python framework for building local RAG and agent applications with small, specialized models that run on laptops and private servers.

Read the project profile

Extend

A document processing platform with APIs to parse, extract, classify, split and fill documents for AI agents and data pipelines.

Read the project profile

Supermemory

A memory API for AI agents and apps that extracts facts and user profiles from chats, files and URLs, stores them in a vector-graph database and injects relevant memories back into the model context.

Read the project profile

Spider

A web crawling, scraping and search API for AI agents and RAG that returns pages as markdown or JSON, backed by its own browser engine and a residential proxy network.

Read the project profile

MemOS

A memory operating system from MemTensor that gives LLM apps and agents long-term memory through a hosted cloud API, an open-source framework, a local-only Lite version and agent plugins.

Read the project profile

WeKnora

Tencent’s open-source knowledge framework that parses team documents into shared knowledge bases and works over them in three modes: cited RAG answers, a tool-using agent and an auto-organised wiki with a knowledge graph.

Read the project profile

MaxKB

An open-source platform from FIT2CLOUD for building enterprise knowledge-base Q&A apps and agents, combining a RAG pipeline, a visual workflow engine and MCP tool calls with support for both local and hosted models.

Read the project profile

Mixedbread

Managed search API that ingests PDFs, slides, images, audio, video and code, then lets apps and agents query it with semantic search, grep, reranking or an agentic search model that returns cited passages.

Read the project profile

Pathway Live Data Framework

Python framework, powered by a Rust engine, for building streaming ETL, real-time analytics and live LLM/RAG pipelines that stay in sync with 300+ data sources.

Read the project profile

Superlinked

SIE (Superlinked Inference Engine), an open-source server and hosted API that runs small open models for embeddings, reranking, OCR, extraction, classification, guardrails and chat behind one API.

Read the project profile

Apify

Web scraping and automation platform where you run ready-made or custom "Actors" in the cloud and expose them to AI agents through an MCP server.

Read the project profile

Surya

Datalab's open-source OCR toolkit for multilingual text recognition, layout analysis, reading order and table recognition.

Read the project profile

LangExtract

Google's open-source Python library that uses LLMs to extract structured information from unstructured text, maps each extraction back to its exact location in the source and renders the results as an interactive HTML review page.

Read the project profile

PaddleOCR

Open-source OCR and document-parsing toolkit from the PaddlePaddle team, with PP-OCRv6 text recognition, PP-StructureV3 layout parsing and the PaddleOCR-VL vision-language document parser, plus an MCP server.

Read the project profile

Go a little deeper