TOPIC GUIDE
Vector databases and RAG infrastructure
Pinecone, Qdrant, Weaviate, Milvus, Chroma and turbopuffer, plus tools that prepare data for retrieval.
- products
- 58
- things to compare
- 3
- updated
- 2026-10-31
Retrieval-augmented generation needs three pieces: getting clean text out of documents and web pages, storing embeddings, and retrieving the right passages. Start with data preparation, since retrieval quality is usually limited by messy inputs before the database matters.
What to compare
- Decide between a managed service and self-hosting before comparing features.
- Test retrieval quality on your own documents and questions, not benchmark datasets.
- Estimate cost at your expected data size and query volume, including minimum monthly charges.
This is a focused selection, not an exhaustive ranking. See each profile for evidence and review scope.
Projects to explore
Pinecone
A managed vector database and knowledge platform for semantic search and retrieval in AI applications.
Read the project profileQdrant
An open-source vector search engine and database, available self-hosted or as Qdrant Cloud.
Read the project profileWeaviate
An AI-native database with vector and hybrid search, available as source code to self-host or as Weaviate Cloud.
Read the project profileMilvus
An open-source vector database for large-scale similarity search, also offered as managed Zilliz Cloud.
Read the project profileChroma
Open-source search infrastructure for AI with vector, full-text and metadata search, available locally or as Chroma Cloud.
Read the project profileturbopuffer
A vector and full-text search engine built on object storage, designed to be cheaper and highly scalable.
Read the project profileUnstructured
A platform that turns PDFs, documents and 64+ other file types into clean, structured inputs for RAG and AI applications.
Read the project profileFirecrawl
A web data API for AI agents that searches, scrapes and crawls websites into clean markdown or structured data.
Read the project profileLlamaIndex
AI agents for document parsing (LlamaParse) and workflows, alongside its framework for building on your data.
Read the project profileMem0
A memory layer for AI agents and apps that stores and retrieves what users said before, available as open source or a hosted platform.
Read the project profileJina AI
Embeddings, rerankers and a Reader API that turns web pages into Markdown for LLM search and grounding; now part of Elastic.
Read the project profileCohere
Enterprise AI models for generation, embeddings, reranking and speech, plus North, a secure workspace for agents and search.
Read the project profileDocling
An open-source toolkit that converts PDFs, Office files, HTML, images, audio and more into structured data for AI, started at IBM Research.
Read the project profileCrawl4AI
An open-source, LLM-friendly web crawler that turns any URL into clean Markdown or typed JSON, also available as a hosted API.
Read the project profileRAGFlow
An open-source RAG engine with an integrated agent platform for giving AI agents reliable context from your documents.
Read the project profileGraphiti
Zep's open-source framework for building temporal knowledge graphs that track how facts change over time, as memory for AI agents.
Read the project profileMarker
Datalab's open-source tool for converting PDFs to Markdown and JSON quickly and accurately, including tables and layout.
Read the project profileMarkItDown
Microsoft's lightweight Python tool for converting PDFs, Office documents and other files to Markdown for LLMs and text analysis.
Read the project profileOnyx
An open-source enterprise search and AI chat platform that connects to company data and runs in your own cloud.
Read the project profiletxtai
An all-in-one open-source framework for semantic search, LLM orchestration and language model workflows, built around an embeddings database.
Read the project profileUnstract
An open-source, no-code platform for extracting structured data from unstructured documents with LLMs, deployed as APIs or ETL pipelines.
Read the project profileMinerU
An open-source document parser from OpenDataLab that converts PDFs, Office files, images and more into Markdown and JSON for AI.
Read the project profilePrivateGPT
An open-source project for private, local question answering over your documents, maintained by the team behind Zylon.
Read the project profileKapa
An AI knowledge retrieval service that connects documentation and support content to assistants through prebuilt agents, an API, or MCP.
Read the project profileCognee
An agent memory toolkit that organizes documents, code and past sessions into connected knowledge for later retrieval.
Read the project profilePaperQA
A Python package that indexes scientific documents, retrieves supporting passages and answers questions with source citations.
Read the project profileSurfSense
A desktop research workspace that answers questions over local documents with citations and turns source material into reports, slides and podcasts.
Read the project profileReducto
A document processing platform with APIs that parse layouts, extract structured fields and prepare content for AI applications.
Read the project profileLanceDB
A multimodal data platform and embedded retrieval library for storing and searching vectors, text and other AI data.
Read the project profileMorphik Core
A multimodal retrieval platform for ingesting, searching and extracting information from visually rich documents through an API, Python SDK and web console.
Read the project profileZep
A hosted context and memory layer that turns conversations, documents and business data into temporal context graphs for AI agents.
Read the project profileChonkie
A Python ingestion library that chunks documents, refines passages, adds embeddings and connects the results to vector databases for RAG pipelines.
Read the project profileDocETL
A Python and YAML toolkit that uses LLM map, filter and reduce operations to turn document collections into structured tables.
Read the project profileDataChain
A Python library for creating typed, versioned datasets from unstructured files and enriching them with models for search and agent workflows.
Read the project profileScrapeGraphAI
An AI web-scraping API that turns natural-language extraction requests into structured data, with a separate self-hosted Python library.
Read the project profileLightRAG
A retrieval framework that combines knowledge graphs and vector embeddings, with a server, REST API and web interface for indexing and querying documents.
Read the project profileMeilisearch
A search engine combining full-text and semantic retrieval, vector storage, filtering and faceting for application search and RAG.
Read the project profileVespa
A search platform combining text, vector and structured retrieval with machine-learned ranking, model inference and real-time data updates.
Read the project profileFastEmbed
A Python library that uses ONNX Runtime to generate dense, sparse and image embeddings and rerank search results.
Read the project profileXberg
A document-intelligence engine with a Rust core, language bindings, OCR, structured extraction, embeddings and CLI, REST and MCP interfaces.
Read the project profileDB-GPT
A self-hosted AI data assistant that connects to databases and files, generates SQL and Python analysis, and produces charts and reports.
Read the project profileMemMachine
An open-source memory layer that stores conversational history and user preferences for AI agents across sessions.
Read the project profileHindsight
An agent memory system that extracts facts, retrieves relevant context and reasons over stored knowledge with retain, recall and reflect APIs.
Read the project profiledocTR
An Apache-2.0 Python OCR library on PyTorch that finds and reads words in PDFs and images, with optional layout detection and a key-information extraction predictor.
Read the project profilellmware
An open-source Python framework for building local RAG and agent applications with small, specialized models that run on laptops and private servers.
Read the project profileExtend
A document processing platform with APIs to parse, extract, classify, split and fill documents for AI agents and data pipelines.
Read the project profileSupermemory
A memory API for AI agents and apps that extracts facts and user profiles from chats, files and URLs, stores them in a vector-graph database and injects relevant memories back into the model context.
Read the project profileSpider
A web crawling, scraping and search API for AI agents and RAG that returns pages as markdown or JSON, backed by its own browser engine and a residential proxy network.
Read the project profileMemOS
A memory operating system from MemTensor that gives LLM apps and agents long-term memory through a hosted cloud API, an open-source framework, a local-only Lite version and agent plugins.
Read the project profileWeKnora
Tencent’s open-source knowledge framework that parses team documents into shared knowledge bases and works over them in three modes: cited RAG answers, a tool-using agent and an auto-organised wiki with a knowledge graph.
Read the project profileMaxKB
An open-source platform from FIT2CLOUD for building enterprise knowledge-base Q&A apps and agents, combining a RAG pipeline, a visual workflow engine and MCP tool calls with support for both local and hosted models.
Read the project profileMixedbread
Managed search API that ingests PDFs, slides, images, audio, video and code, then lets apps and agents query it with semantic search, grep, reranking or an agentic search model that returns cited passages.
Read the project profilePathway Live Data Framework
Python framework, powered by a Rust engine, for building streaming ETL, real-time analytics and live LLM/RAG pipelines that stay in sync with 300+ data sources.
Read the project profileSuperlinked
SIE (Superlinked Inference Engine), an open-source server and hosted API that runs small open models for embeddings, reranking, OCR, extraction, classification, guardrails and chat behind one API.
Read the project profileApify
Web scraping and automation platform where you run ready-made or custom "Actors" in the cloud and expose them to AI agents through an MCP server.
Read the project profileSurya
Datalab's open-source OCR toolkit for multilingual text recognition, layout analysis, reading order and table recognition.
Read the project profileLangExtract
Google's open-source Python library that uses LLMs to extract structured information from unstructured text, maps each extraction back to its exact location in the source and renders the results as an interactive HTML review page.
Read the project profilePaddleOCR
Open-source OCR and document-parsing toolkit from the PaddlePaddle team, with PP-OCRv6 text recognition, PP-StructureV3 layout parsing and the PaddleOCR-VL vision-language document parser, plus an MCP server.
Read the project profile