←Back to Documentation

Vector Search

Build AI-powered semantic search with vector embeddings

Overview

entropyDB provides native support for vector embeddings and similarity search using HNSW (Hierarchical Navigable Small World) indexing. Store embeddings alongside your relational data for powerful hybrid search capabilities.

Key Features

  • Support for vectors up to 2048 dimensions
  • HNSW indexing for sub-millisecond search
  • Multiple distance metrics (cosine, L2, inner product)
  • Hybrid search combining vectors and metadata

Creating Vector Columns

Add vector columns to your existing tables:

-- Create table with vector column
CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  content TEXT,
  embedding VECTOR(1536),  -- OpenAI ada-002 dimension
  metadata JSONB,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Add vector column to existing table
ALTER TABLE products 
ADD COLUMN embedding VECTOR(768);  -- Custom dimension

Inserting Vector Data

From Array

INSERT INTO documents (content, embedding)
VALUES (
  'entropyDB is a multi-model database',
  ARRAY[0.1, 0.2, 0.3, ...]::VECTOR(1536)
);

Using Python

import openai
from entropydb import connect

# Generate embedding
response = openai.Embedding.create(
    input="Your text here",
    model="text-embedding-ada-002"
)
embedding = response['data'][0]['embedding']

# Insert into entropyDB
conn = connect("postgresql://user:pass@localhost/entropydb")
conn.execute(
    "INSERT INTO documents (content, embedding) VALUES (%s, %s)",
    ("Your text here", embedding)
)

Creating Vector Indexes

Build HNSW indexes for fast approximate nearest neighbor search:

-- Create HNSW index with cosine distance
CREATE INDEX idx_documents_embedding 
ON documents 
USING hnsw (embedding vector_cosine_ops)
WITH (
  m = 16,              -- Number of connections per layer
  ef_construction = 64 -- Size of dynamic candidate list
);

-- For L2 distance
CREATE INDEX idx_products_embedding 
ON products 
USING hnsw (embedding vector_l2_ops);

-- For inner product
CREATE INDEX idx_images_embedding 
ON images 
USING hnsw (embedding vector_ip_ops);

Index Parameters

  • m: Higher values = better recall, more memory
  • ef_construction: Higher values = better index quality, slower build

Similarity Search

Basic Search

-- Find 10 most similar documents
SELECT 
  id, 
  content,
  1 - (embedding <=> query_vector) as similarity
FROM documents
ORDER BY embedding <=> ARRAY[0.1, 0.2, ...]::VECTOR(1536)
LIMIT 10;

Hybrid Search

-- Combine vector similarity with metadata filters
SELECT 
  id,
  content,
  metadata->>'category' as category,
  1 - (embedding <=> query_vector) as similarity
FROM documents
WHERE 
  metadata->>'category' = 'technology'
  AND created_at > NOW() - INTERVAL '30 days'
ORDER BY embedding <=> query_vector
LIMIT 10;

Distance Operators

  • <=> Cosine distance
  • <-> L2 distance (Euclidean)
  • <#> Inner product

RAG Application Example

Complete example of a Retrieval-Augmented Generation system:

import openai
from entropydb import connect

def rag_query(question: str, limit: int = 5):
    # Generate query embedding
    response = openai.Embedding.create(
        input=question,
        model="text-embedding-ada-002"
    )
    query_embedding = response['data'][0]['embedding']
    
    # Search for relevant documents
    conn = connect("postgresql://localhost/entropydb")
    results = conn.execute("""
        SELECT content, 1 - (embedding <=> %s) as similarity
        FROM documents
        ORDER BY embedding <=> %s
        LIMIT %s
    """, (query_embedding, query_embedding, limit))
    
    # Build context
    context = "\n\n".join([row['content'] for row in results])
    
    # Generate answer with GPT
    completion = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "Answer based on the context."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ]
    )
    
    return completion.choices[0].message.content

# Usage
answer = rag_query("What is entropyDB?")

Performance Tuning

Query-Time Parameters

-- Increase search accuracy
SET hnsw.ef_search = 100;  -- Default: 40

-- Then run your query
SELECT * FROM documents 
ORDER BY embedding <=> query_vector 
LIMIT 10;

Best Practices

  • Use appropriate index parameters for your dataset size
  • Consider quantization for very large datasets
  • Batch insertions for better performance
  • Use hybrid search to reduce search space

Next Steps

→ Performance Tuning→ Python SDK Documentation