🛠️ AI Tool Lab Daily updates · 500+ AI tools

What Is Milvus?

Milvus is an open-source vector database. That sounds technical, but the real-world job is simple: when someone types a question into an AI chatbot, Milvus finds the most relevant chunks of knowledge from your database and hands them to the LLM. It is the retrieval half of RAG (Retrieval-Augmented Generation), and it is what makes AI apps actually know things beyond their training data.

I started using Milvus in early 2025 when building a document Q&A system for a law firm. They had 200,000 case files in PDF format, and their lawyers were spending 3-4 hours a day searching through them. I embedded every document with OpenAI's text-embedding-3-large model, loaded the vectors into Milvus, and built a simple chat interface on top. The result: a lawyer types "find me similar cases to Smith v. Jones about breach of fiduciary duty in Delaware" and gets relevant excerpts in under 2 seconds. I charged $12,000 for that build.

By late 2025, the system had grown to 1.8 million document chunks. Milvus handled it without breaking a sweat. That is when I realized this was not just a database -- it was a client business in a box. Every company with a knowledge management problem is a potential Milvus consulting gig.

How Milvus Actually Works

1. Embedding Storage and Indexing

Milvus takes your embedding vectors (the 1536-dimension or 3072-dimension arrays that represent text/images/audio mathematically) and stores them in collections, similar to tables in a relational database. It then builds an index on top of these vectors so that similarity searches are fast, not brute-force scans of millions of numbers.

The index choice matters a lot. HNSW (Hierarchical Navigable Small World) is the default and works well for most cases -- fast, accurate, memory-intensive. IVF (Inverted File Index) trades some accuracy for speed and lower RAM usage. For truly massive datasets (100M+ vectors), DiskANN lets you offload indexes to SSD. Each project requires a different tuning strategy, and Milvus gives you the knobs.

2. Hybrid Search (The Feature That Sells Itself)

Pure vector search is not enough in production. A user searching for "Q3 2025 revenue projections by region -- North America" needs to match both the semantic meaning AND filter by metadata fields like quarter and region. Milvus handles this with hybrid search: it runs the vector similarity query and applies scalar/metadata filters simultaneously. It is the difference between a demo and a product someone will actually pay for.

I built a real estate search tool that combines property image similarity (visual vectors from CLIP embeddings) with structured filters (price range, bedroom count, lot size). The hybrid search in Milvus made this possible without running two queries and manually merging results -- a mess I have seen multiple startups try and fail at.

3. Distributed Architecture

Milvus separates compute and storage: query nodes handle search requests, data nodes manage indexing, and storage nodes (MinIO or S3) persist everything. This means you can scale query capacity independently of storage capacity, which is the only architecture that makes economic sense at production scale. When a client's user base doubles, you spin up more query nodes instead of ripping out the entire infrastructure.

Making Money With Milvus

Path 1: Enterprise RAG Consulting ($5K-$15K per project)

This is the bread and butter. Companies with 50K+ internal documents (legal firms, hospitals, financial institutions, engineering teams) have a search problem. Their existing tools -- SharePoint search, Google Drive search -- are terrible for anything beyond exact keyword matching. Build them a Milvus-backed document Q&A system and you solve a real pain point.

The typical project goes: document parsing (2-3 days for mixed formats) → embedding pipeline (1 day with batch processing) → Milvus deployment (1-2 days) → chat interface (2-3 days) → testing and refinement (2-3 days). At $150-$200/hr freelance rates, that is a $7K-$15K project that takes 10-15 days. The ongoing maintenance retainer ($300-$800/mo) covers index updates, monitoring, and occasional feature additions.

Path 2: Managed Milvus Hosting ($500-$2K/mo per client)

Not every company wants to run Kubernetes. Set up and manage Milvus clusters for 5-10 SMBs who need vector search but lack DevOps bandwidth. At $500-$2K/mo per client depending on scale, 5 clients = $2.5K-$10K monthly recurring. Your costs are cloud infrastructure ($80-$300/mo per cluster on AWS/GCP) plus maybe 2-3 hours per client per month for monitoring and updates. Margins run 70-80% once clusters are stable.

Path 3: Pinecone-to-Milvus Migration ($2K-$5K per migration)

Pinecone's pricing model is usage-based and gets expensive fast. A company with 5 million vectors and moderate query volume can easily hit $800-$2,000/mo on Pinecone. Self-hosted Milvus on a $200/mo cloud instance handles the same workload. The math is obvious, but the migration work (re-indexing, API changes, testing) is not trivial. Charge $2K-$5K per migration and the client breaks even within 2-4 months. I have done two of these and both clients are still running Milvus a year later.

Path 4: Specialized Search Products

Build a domain-specific search product on Milvus and sell it as SaaS. A medical literature search tool for doctors. An architectural pattern matcher for engineering firms. A video clip finder for video editors. The core search engine (Milvus) is free; you charge for the UI, the domain-specific preprocessing, and the ongoing data updates. Medical literature search at $49/mo per seat with 20 seats in a mid-size hospital = ~$12K/yr per client.

What Milvus Is Bad At

I need to be honest about the rough edges because they will bite you on client projects if you do not plan for them:

Dependency hell in production. Milvus depends on etcd (metadata), MinIO/S3 (object storage), and Pulsar (message logs). Each one is a distributed system with its own failure modes. When etcd runs out of disk space, your Milvus cluster silently stops accepting writes and the error messages are not helpful. I learned this at 11 PM on a Thursday during a client deployment.

Query node memory is expensive. For fast search, Milvus loads vector indexes into RAM on query nodes. A collection with 10 million 1536-dim vectors needs roughly 60-80GB of RAM for HNSW indexes. On AWS, that is a r6i.2xlarge at roughly $360/mo. Factor this into your pricing -- too many consultants quote based on storage size and forget about the RAM requirement.

Milvus Cloud vs self-hosted is a real decision. Zilliz Cloud (managed Milvus) removes the DevOps burden but costs significantly more at scale. A moderate deployment (5M vectors, 100 queries/second) runs about $400-$800/mo on Zilliz Cloud vs $150-$250 self-hosted on equivalent hardware. The trade-off is simple: pay with money or pay with time. For client projects where you charge a retainer, self-hosted often works in your favor.

SDK inconsistency between versions. The Python SDK has had breaking changes between major releases (1.x to 2.x was a rewrite). If you build a project on Milvus 2.3 and need to upgrade to 2.5, budget 1-2 days for migration testing. The Go SDK is more stable but has fewer examples and community resources.

Milvus vs Alternatives

Getting Started

• Start with Docker standalone mode. Do not touch Kubernetes on day one. docker-compose with standalone Milvus + Attu (web GUI) is all you need for prototyping and small projects under 100K vectors.
• Pick HNSW for your first index. It works well out of the box for 99% of use cases. Do not optimize before you measure.
• Normalize your embeddings. If embeddings are not normalized, cosine and inner product distances give different rankings. Normalize at ingestion time and use inner product (faster than cosine).
• Monitor query latency from day one. Milvus exposes Prometheus metrics. Set up a Grafana dashboard tracking p50/p95/p99 latency so you know when index rebuilds are needed before clients complain.
• Budget RAM, not disk. Vector indexes live in memory. Disk usage is rarely the bottleneck. When quoting infrastructure costs to a client, lead with RAM requirements first.
• Use batch inserts when loading initial data. Sending 1 million vectors one at a time through the REST API will take hours. Use milvus_client.insert() with batch sizes of 5K-10K rows.

Final Verdict

Milvus is not the easiest vector database and it is not the cheapest managed option. But it is the most scalable open-source choice, and for someone building client-facing AI applications where the data keeps growing, that matters more than anything else.

If you are doing small projects or learning RAG concepts, start with Chroma or pgvector. If you are building production systems for paying clients, invest the time to learn Milvus. The first deployment is a headache. The second one is manageable. By the fifth, you will wonder why anyone pays Pinecone bills.