By Mohit Palriwal
October 29, 2024
Google Cloud has introduced scalable vector search capabilities to its Memorystore for Valkey and Redis clusters, empowering developers to execute vector searches on billions of vectorswith ultra-low latency. This enhancement is particularly beneficial for applications reliant on generative AI, such as retrieval-augmented generation (RAG), recommendation systems, andsemantic search.
The update leverages the ability to partition vector indices across nodes within a cluster. Each node houses an index partition corresponding to its portion of the keyspace, enabling the cluster to handle billions of vectors while maintaining single-digitmillisecond latency and over 99% recall. This architecture not only linearly accelerates index build times when adding nodes but also optimizes search performance – achieving logarithmic optimization for hierarchical navigable small-world (HNSW) search and linear optimization forbrute-force search.
Developers can utilize these new features to scale their clusters up to 250 shards, storing billions of vectors within a single instance. This scalability is crucial for enterprise applications requiring semantic search on massive datasets.
Beyond scalability, the update introduces support for hybrid queries. Hybrid queries allow developers to combine vector searchwith filters on numeric and tag fields. This functionality is particularly useful for fine-tuning search results based on specific conditions. For instance, an online apparel retailer could use hybrid search to recommend similar items while filtering results based on clothing type and price range.
To implement hybrid queries, developers can create a new vector index incorporatingadditional fields for filtering:
FT.CREATE inventory_index SCHEMA embedding VECTOR HNSW 6 DIM 128 TYPE FLOAT32 DISTANCE_METRIC L2 clothing_type TAG clothing_price_usd NUMERIC
This creates an index ‘inventoryindex’ with a vector field’embedding’ for semantic embeddings of clothing items, a tag field ‘clothingtype’ for representing clothing categories (e.g., dress or hat), and a numeric field ‘clothingpriceusd’ for representing the price of the clothing item.
To perform a query on ‘inventory_index’ for a specific clothing type and price range:
FT.SEARCH inventory_index @embedding[KNN 10 @vector] FILTER clothing_type:dress clothing_price_usd:[10,50]
This query will retrieve the 10 nearest neighbors based on theprovided vector, filtering the results to only include dresses with a price between $10 and $50.
This new feature significantly enhances the capabilities of Google Cloud’s Memorystore for Valkey and Redis clusters, enabling developers to build more sophisticated and efficient AI-powered applications.
References:
*Google Cloud Blog: Memorystore for Redis Now Supports Scalable Vector Search
* Google Cloud Documentation: Memorystore for Redis
Views: 0