wolfgarbe/seekstorm_server

https://img.shields.io/crates/v/seekstorm.svg](https://crates.io/crates/seekstorm) https://img.shields.io/crates/d/seekstorm.svg?style=flat-square](https://crates.io/crates/seekstorm) 

https://github.com/SeekStorm/SeekStorm | Website | https://seekstorm.github.io/search-benchmark-game/ | Demo | https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#documentation | https://seekstorm.github.io/documentation/ | https://github.com/SeekStorm/SeekStorm/blob/main/src/seekstorm_server/README.md | https://github.com/SeekStorm/SeekStorm#roadmap | Blog | ***

docker run -ti -p "8000:80" -v "$(pwd)/seekstorm_storage:/seekstorm_index:z" wolfgarbe/seekstorm_server

https://seekstorm.github.io/documentation/, see master API key in console or in docker desktop / view details / logs .

SeekStorm is an open-source, sub-millisecond lexical & vector search library & multi-tenancy server implemented in Rust.

Development started in 2015, in production since 2020, Rust port in 2023, open sourced in 2024, work in progress.

SeekStorm is open source licensed under the https://github.com/SeekStorm/SeekStorm?tab=Apache-2.0-1-ov-file#readme

Blog Posts: SeekStorm is now Open Source and SeekStorm gets Faceted search, Geo proximity search, Result sorting

SeekStorm high-performance search library

• Full-text lexical search
• Vector search
• True real-time search, with negligible performance impact
• Incremental indexing
• Multithreaded indexing & search
• Unlimited field number, field length & index size
• Compressed document store: ZStandard
• Boolean queries: AND, OR, PHRASE, NOT
• BM25F and BM25F_Proximity ranking
• Field filtering
• https://github.com/SeekStorm/SeekStorm/blob/main/FACETED_SEARCH.md: Counting & filtering of String & Numeric range facets (with Histogram/Bucket & Min/Max aggregation)
• Result sorting by any field, ascending or descending, multiple fields com***ed by "tie-breaking".
• Geo proximity search, filtering and sorting.
• Tokenizer with Chinese word segmentation
• KWIC snippets, highlighting
• One-way and multi-way synonyms
• Billion-scale index
• Language independent
• API keys
• RESTful API with CORS
• Index either in RAM or memory mapped files
• Cross-platform (Windows, Linux, MacOS)
• SIMD (Single Instruction, Multiple Data) hardware acceleration support,
  both for x86-64 (AMD64 and Intel 64) and AArch64 (ARM, Apple Silicon).
• Single-machine scalability: serving thousands of concurrent queries with low latency from a single commodity server without needing clusters or proprietary hardware accelerators.

SeekStorm uses two separate, first-class, native index architectures, under one roof.

• Lexical search: sharded and leveled inverted index.
• Vector search: sharded and leveled IVF index for ANN or exhaustive search.
• Shared document store, shared document ID space.
• Both first-class engines are integrated at the query planner level.
• Query planner with QueryModes (Lexical, Vector, Hybrid…) and FusionTypes (RRF, …).

Vector Features

• Multi-Vector indexing: both from multiple fields and from multiple chunks per field.
• Integrated inference from any text document field or Import external embeddings.
• Variable dimensions and precisions: f32, i8.
• TurboQuant (TQ) and affine Scalar Quantization (SQ).
• Multiple similarity measures: Cosine similarity, Dot product, Euclidean distance.
• Chunking that respects sentence boundaries and Unicode segmentation for multilingual text.
• K-medoid clustering with actual data points as centers.
• Field filters are active during vector search, not just as post-search filtering step.
• True real-time search
• disk-based billion-scale vector search
• Sub-millisecond search for 1 million vectors on a laptop, CPU only:
  • Sift1M recall@10=95%, 0.2 ms | recall@10=99%, 0.3 ms

Query types

• OR disjunction union
• AND conjunction intersection
• "" phrase
• - NOT

Result types

• TopK
• Count
• TopKCount

SeekStorm multi-tenancy search server

• Index and search via https://github.com/SeekStorm/SeekStorm/blob/main/src/seekstorm_server#rest-api-endpoints.
• Ingest local data files in JSON, https://github.com/ndjson/ndjson-spec (ndjson), and Concatenated JSON formats via console command.
• Ingest local PDF files via console command (single file or all files in a directory).
• Multi-tenancy index management.
• API-key management.
• https://github.com/SeekStorm/SeekStorm/blob/main/src/seekstorm_server#open-embedded-web-ui-in-browser to search and display results from any index without coding.
• Code first OpenAPI generated REST API documentation
• Cross-platform: runs on Linux, Windows, and macOS (other OS untested).