dynavec
Technical deep dive into 11 merged pull requests on dynavec, an open-source serverless hybrid vector database built on Amazon DynamoDB and Amazon S3 Vectors.
| PR | Issue | Domain | What Is the Issue? | How I Resolved It (Technical In-Depth) |
|---|---|---|---|---|
| PR #170 | Issue #80 | Distributed Systems | SPFresh-Style Incremental Hot Index Rebalancing Engine Static vector clustering (IVF/SPANN) requires expensive full offline index rebuilds when vectors are dynamically added or deleted. Hot clusters become skewed and centroids drift, degrading search recall and causing latency spikes. | • Dynamic 2-Means Bipartitioning: Designed and implemented SPFreshHotIndex which monitors partition sizes and splits overloaded clusters (N > N_max) via spherical 2-means with farthest-first seed initialization and unit-norm projection in <= 10 iterations. • Nearest-Centroid Merge: Automatically consolidates underflowed clusters (N < N_min) into closest neighbor centroids with overflow guards preventing infinite split/merge oscillation. • Drift Tracking & Boundary Migration: Implemented normalized centroid drift metric (Delta_drift = ||c_curr - c_anchor||_2 / ||c_anchor||_2) that dynamically migrates boundary vectors. • Non-Blocking Concurrency: Engineered per-partition RLock and atomic memory snapshots allowing parallel read queries alongside active background maintenance via SPFreshRebalancer. • Pre-Allocated Memory Buffer: Implemented chunk-doubling contiguous NumPy buffers for O(1) amortized insertion. • Testing: Wrote comprehensive 27-test suite covering 1536-D OpenAI embeddings, brute-force recall benchmarks (>85%), and 8-thread reader/writer stress testing. |
| PR #165 | Issue #65 | AI & Ecosystem | Ship FastMCP Server for Semantic and Knowledge Graph RAG AI coding assistants and agent frameworks (Claude Desktop, Cursor, Antigravity) lacked a standardized Model Context Protocol (MCP) server to query dynavec's vector search and Knowledge Graph traversal directly. | • FastMCP Server (dynavec.mcp.server): Built a standalone MCP server exposing dynavec_search (with hybrid filtering, metric rescoring, and reranking) and dynavec_graph_search (multi-hop entity graph traversal combined with vector ranking). • Zero-Config Environment Parser: Implemented client_from_env() automatically detecting AWS credentials and initializing bring-your-own-key embedders (OpenAI, Gemini, Voyage, Mistral, Bedrock, SentenceTransformers). • CLI Entrypoint & Transports: Added dynavec mcp CLI supporting both stdio and sse transports with configurable ports. • Lazy Loading Architecture: Guarded heavy MCP dependencies to keep core package installation lightweight (boto3 + numpy only). |
| PR #144 | Issue #87 | Distributed Systems | AWS Pre-Flight Diagnostic Health Check CLI (dynavec doctor) Users deploying dynavec into their AWS accounts frequently encountered silent runtime failures due to misconfigured STS credentials, missing IAM permissions, or non-existent S3 Vector buckets/DynamoDB tables. | • Diagnostic Engine (src/dynavec/cli.py): Engineered the dynavec doctor command performing non-destructive, read-only validation of cloud infrastructure. • Multi-Stage Validation Pipeline: Sequentially checks: (1) AWS STS identity and caller credentials, (2) S3 Vectors index accessibility, and (3) DynamoDB table read/write permissions. • Actionable UX: Formatted diagnostic reports with clear ANSI color-coded pass/fail indicators, explicit error remediation tips, and Unix exit codes (0 for healthy, 1 for failure). • Offline Testing: Authored credential-free unit tests in tests/test_cli.py mocking STS/boto3 failure modes. |
| PR #120 | Issue #63 | AI & Ecosystem | In-Run Content-Hash Document Deduplication During large batch ingestion jobs, identical text chunks across documents were repeatedly embedded and upserted, causing excessive LLM embedding API billing and database bloat. | • Content-Hashed Deduplication (src/dynavec/ingest.py): Added in-memory SHA-256 content hashing across document ingestion pipelines. • Cost & I/O Reduction: Evaluates chunk text hashes against an in-run hash set, silently skipping duplicate texts while preserving unique embeddings and metadata. • Integration Testing: Updated ingestion test suites in tests/test_ingest.py to assert accurate deduplication counts across multi-document runs. |
| PR #163 | Issue #105 | Reliability & Docs | Production FAQ & System Constraints Architecture Docs Developers and platform architects lacked clarity on AWS region availability, vector dimensionality limits, consistency models, metadata splitting rules, and serverless cost structures. | • Technical Documentation Page (opensource/dynavec/docs/faq.html): Authored a comprehensive production FAQ covering single-region latency optimizations, 4,096-dim vector ceiling, eventual consistency on ingest, and $3/mo cost breakdown. • Metadata Architecture Details: Documented the architectural split between S3 Vectors (filterable indexed metadata) vs DynamoDB (400 KB hydrated document payload). • Static Site Generator Integration: Integrated into tools/build_docs.py navigation matrix and regenerated global documentation. |
| PR #154 | Issue #111 | Reliability & Docs | PEP 561 Static Type Hint Compliance (py.typed) Downstream IDEs, mypy, and pyright were unable to discover dynavec's inline type hints because the package lacked PEP 561 compliance markers. | • PEP 561 Marker: Added src/dynavec/py.typed marker file. • Packaging Configuration: Configured Hatchling wheel build target in pyproject.toml (packages = ["src/dynavec"], include = ["src/dynavec/py.typed"]) ensuring type hint metadata is bundled into PyPI distribution wheels. • Verification: Validated type checker discovery across clean virtual environment builds. |
| PR #155 | Issue #34 | Reliability & Docs | Knowledge Graph Traversal Infinite-Cycle Regression Suite Cyclic relationships in Knowledge Graphs (a -> b -> c -> a) risked causing infinite recursive loops and stack overflows during multi-hop GraphRAG traversals. | • Cycle Detection Suite (tests/test_client_inmemory.py): Built regression tests constructing circular graph topologies. • Termination & Isolation Checks: Verified breadth-first traversal algorithms terminate safely within depth bounds and strictly return reachable nodes without including the origin root entity. |
| PR #116 | Issue #14 | Distributed Systems | S3 Vectors 100+ Key Chunked Batch Retrieval Testing AWS S3 Vectors limits get_vectors API calls to 100 keys per request. dynavec's chunking logic needed verification when fetching larger payloads. | • Batch Boundary Suite (tests/test_s3vectors.py): Designed test scenarios requesting 250 vector keys. • Chunk Pagination Validation: Validated that dynavec correctly partitions keys into consecutive chunks of 100, 100, and 50, and seamlessly merges multi-part responses without key drops or order corruption. |
| PR #119 | Issue #9 | AI & Ecosystem | Embedder Dimension Mismatch Runtime Validation If a custom or third-party embedder model produced vector dimensions differing from the index's configured dimension, writes could corrupt storage indexes without clear error tracing. | • Validation Guard (tests/test_client_inmemory.py): Added runtime dimension check verification confirming dynavec immediately intercepts dimension mismatches on write. • Deterministic Error Handling: Verified DimensionMismatchError is raised before executing network calls, protecting database integrity. |
| PR #2 | Foundational | Reliability & Docs | Retrieval & Chunker Edge-Case Hardening Edge conditions in text chunking (whitespace-only text) and Reciprocal Rank Fusion (RRF) (mismatched weights, empty candidate lists) lacked test coverage. | • Edge-Case Test Suite (tests/test_ingest.py, tests/test_retrieval.py): Validated whitespace-only chunk filtering, graceful handling of empty result lists in RRF, and weight vector dimension validation. |
| PR #1 | Foundational | Reliability & Docs | Python 3.9 Dependency Environment Markers uv and pip dependency resolution failed on Python 3.9 environments due to unconstrained Python 3.10+ sub-dependencies in pyproject.toml. | • Environment Markers (pyproject.toml): Added explicit PEP 508 environment markers (mistralai>=2.0; python_version >= '3.10', mcp>=1.0; python_version >= '3.10', crewai>=0.70; python_version >= '3.10'). • Cross-Platform Compatibility: Allowed Python 3.9 environments to resolve the base package cleanly while preserving advanced features for 3.10+ runtimes. |