OpenSearchCon Europe 2025

Many OS users, especially those new to Lucene-based search engines, struggle with sharding best practices. This often results in inefficient shard sizes, leading to performance issues and resource waste. Our talk introduces a novel approach: Dynamic Sharding Strategy.

We'll explore how this strategy adaptively determines the optimal primary shard count for indices, particularly beneficial for time-series data with fluctuating ingestion rates. By analyzing past ingestion rates, cluster resources, and workload patterns, our method predicts and applies suitable shard counts for each index rollover.

We'll discuss
* Common sharding pitfalls and their impact
* The idea behind our dynamic sharding approach
* Implementation as a Python script integrated with OpenSearch
* Real-world case studies and performance improvements
* Potential for native integration into OpenSearch's Index State Management

This talk aims to empower OpenSearch users with a practical solution for maintaining optimal cluster performance, reducing storage skewness, and minimizing manual interventions in shard management. Join us to learn how dynamic sharding can revolutionize your OpenSearch cluster efficiency