Comparing Vector Search Solutions: Weaviate, OpenSearch, and pgvector

Introduction

When your application requires search capabilities, you might consider tools such as OpenSearch for full-text search or PostgreSQL with the pgvector extension for applications already utilizing relational data. Although both can manage and query vectors, they're not primarily designed for vector-based data structures. OpenSearch incorporates vector search through a plugin on its inverted-index engine. Meanwhile, pgvector integrates nearest-neighbor search as a SQL operator, suitable when vectors are part of a broader dataset but not for central semantic search purposes.

Weaviate approaches this differently by focusing on the HNSW (Hierarchical Navigable Small World) vector index as its core structure, with BM25 keyword search and hybrid search built on top, rather than as subsequent additions. This setup allows for semantic search, keyword search, and hybrid search to be executed via a single API call. Managed Weaviate services provide automated provisioning, backups, and patching without operational burdens.

To illustrate these differences, this article examines how the three technologies handle the same dataset—podcast transcripts—focusing on distinctions in search quality (vector vs. keyword vs. hybrid retrieval), the amount of search logic required in applications, ingest speed, and index size.

Key Takeaways

• Hybrid search offers the best quality across all engines, with Weaviate providing it through a single API call. The combination of vector and keyword retrieval outperforms either individually, with Weaviate achieving the highest hybrid score.
• Vector-search quality depends on the embeddings, not the database. Aligning the candidate list depth, all three scored similarly on the same vectors.
• Operational differences are notable. Weaviate ingests data fastest due to its single-pass index building, while pgvector boasts the most compact index size but requires careful query construction for keyword searches.

Methodology

The dataset used consists of podcast transcripts, totaling 4,886 unique episodes, chunked into approximately 500-token passages. This resulted in 100,000 documents, each containing passage text and metadata. Embeddings were generated using OpenAI's text-embedding-3-small and cached for consistency across systems.

The tests were conducted on managed database services, ensuring comparable environments. Each system was configured for English text processing, aligning vector search parameters and using similar text analysis techniques.

Ingest and Loading

Embeddings were pre-computed, and loading involved inserting passages and building vector and keyword indexes. The sequence of ingest and indexing varied:

• Weaviate: Simultaneous ingestion and indexing via gRPC, making the collection searchable immediately.
• Postgres: Separates loading from indexing, requiring explicit post-load index builds.
• OpenSearch: Interleaves indexing with loading, leading to repeated costs during ingestion.

Hybrid Search: One Call vs. Two

Hybrid search, combining semantic and keyword relevance, showcases architectural differences:

• Weaviate: Executes hybrid search in one call, using Reciprocal Rank Fusion (RRF) for ranking.

  results = collection.query.hybrid(query=query_text, vector=query_vector, alpha=0.5, limit=10)
  

• OpenSearch: Requires separate queries for vector and keyword searches, followed by client-side fusion.

  knn = client.search(index="passages", body={"size": 50, "query": {"knn": {"embedding": {"vector": query_vector, "k": 50, "method_parameters": {"ef_search": 100}}}}})
  bm25 = client.search(index="passages", body={"size": 50, "query": {"match": {"transcript_text": query_text}}})
  

def rrf(*result_lists, k=60): scores = for results in result_lists: for rank, hit in enumerate(results): scores[hit["_id"]] = scores.get(hit["_id"], 0) + 1 / (k + rank + 1) return sorted(scores, key=scores.get, reverse=True)[:10]

results = rrf(knn["hits"]["hits"], bm25["hits"]["hits"])

- **pgvector**: Uses SQL for merging query results.
```sql
WITH v AS (SELECT guid, row_number() OVER (ORDER BY embedding <=> %(qv)s) rk FROM passages ORDER BY embedding <=> %(qv)s LIMIT 50),
     k AS (SELECT guid, row_number() OVER (ORDER BY ts_rank(tsv, to_tsquery('english', %(q)s)) DESC) rk FROM passages WHERE tsv @@ to_tsquery('english', %(q)s) LIMIT 50)
SELECT p.guid, COALESCE(1.0/(60+v.rk),0) + COALESCE(1.0/(60+k.rk),0) AS rrf_score
FROM passages p LEFT JOIN v USING (guid) LEFT JOIN k USING (guid)
WHERE v.guid IS NOT NULL OR k.guid IS NOT NULL
ORDER BY rrf_score DESC LIMIT 10;

Search Quality: Vector vs Keyword vs Hybrid

Using 200 questions generated from the dataset, each engine was evaluated on whether the source passage was included in the top 10 results and on mean reciprocal rank (MRR).

• Hybrid search consistently outperforms vector and keyword searches alone.
• Vector quality remains consistent across all engines, as databases only fetch vectors without enhancing them.
• Weaviate delivers the top overall results due to its strong native fusion and BM25 implementation.

Index Size

After loading 100,000 documents with embeddings, the index sizes were:

• PostgreSQL + pgvector: 1.87 GB
• OpenSearch: 2.91 GB
• Weaviate: Index size not disclosed

pgvector is the most compact, advantageous when vectors are part of an existing table. OpenSearch's larger size results from storing raw embeddings along with its k-NN graph.

When to Use Each

• Weaviate: Ideal for applications where semantic search is central. It offers native hybrid search and high-quality retrieval with minimal maintenance.
• OpenSearch: Suitable for existing OpenSearch users needing to add vector search or those requiring advanced analytics and scaling capabilities. It requires more client-side orchestration for hybrid searches.
• PostgreSQL + pgvector: Best for scenarios where vectors complement a relational schema. It offers compactness but requires careful keyword query construction and hybrid logic implemented in SQL.

Conclusion

All three systems efficiently store and query vectors. Their differences lie in result quality and the amount of search logic required. Hybrid retrieval offers superior quality, with Weaviate providing the best performance with minimal code. For applications centered around search, Weaviate offers comprehensive solutions, while OpenSearch and pgvector require more configuration and maintenance.