OpenSearchCon Europe 2025

Abstract:

Contributing to an open-source project like OpenSearch can seem daunting, especially without prior knowledge of the product. In this presentation, I will share my personal journey of starting as a complete beginner and becoming an active contributor to OpenSearch.

Description:

Open-source contributions are a journey of growth, collaboration and innovation. I started contributing to OpenSearch about 8 months back. I will discuss the strategies and resources that helped me to overcome the initial learning curve, including understanding the codebase, finding beginner-friendly issues, and engaging with the community. I contributed to 3 different projects so far (OpenSearch core, Security and ml-commons). I will go through some of these PRs and explain how and why I chose these tasks. I will talk about different meetups like triage meetings, community meetings, code walkthrough sessions and conferences and what we can learn from these.

Key Takeaways:

Practical steps to begin contributing to OpenSearch.
Overcoming the learning curve and finding beginner-friendly issues.
Leveraging community support and resources for successful contributions.

As a Maintainer of the OpenSearch Specification Repository, I’d like to share how we addressed a major challenge in maintaining OpenSearch clients (Ruby, Python, Java, Node.js).

Every OpenSearch update required manual replication of changes across all clients, consuming countless hours since version 1.0. By version 2.18, it became clear we needed a scalable solution. Our team built a Specification Repository that centralizes all possible request and response schemas, enabling automated client code generation.

We use OpenAPI specifications and a custom testing framework to ensure consistency, minimize human error, and accelerate development. This approach eliminates repetitive work, improves reliability, and scales better with OpenSearch's growth.

I will demonstrate how the repository works, share code generation examples, and discuss its impact on reducing maintenance overhead while improving client reliability. Additional resources showcasing real-world applications will be provided.

This solution not only streamlines client updates but also sets a foundation for innovation in OpenSearch's ecosystem.

Join us to explore how user research drives technical innovation in OpenSearch. Discover how our UX team rebuilt OpenSearch Dashboards' information architecture by starting with comprehensive user personas of Security and SRE teams and optimized for the workflows of real-time operational analytics.
We'll share our research insights, persona frameworks, and the resulting information architecture that transformed our UI approach.
Key topics include:
How security and observability personas shaped our core design decisions
Creating an information hierarchy that supports real-time analysis workflows
Mapping common patterns between security analytics and observability use cases
Design patterns that emerged from our persona-driven approach
Future extensibility opportunities for the community
Whether you're contributing to OpenSearch or building analytics tools, you'll gain practical insights into designing interfaces that scale across diverse use cases. We'll share our learnings from the recent redesign of the information architecture and invite community feedback on our proposed framework.

In today's rapidly evolving AI landscape, Agentic Retrieval-Augmented Generation (RAG) systems represent a significant leap forward in intelligent search capabilities. This presentation introduces how Agentic RAG enhances traditional RAG by incorporating autonomous agents that can both retrieve and act on information. We'll demonstrate a practical implementation using OpenSearch's vector database, showcasing how this combination enables agents to understand user queries, retrieve relevant information, and execute meaningful actions based on the context.

The presentation will focus on three key areas: the fundamental architecture of Agentic RAG, its integration with OpenSearch vector search capabilities, and practical scaling considerations for enterprise applications. Through demonstration steps, we'll show how this system handles real-world scenarios, highlighting the potential impact on business operations. We'll conclude with insights into deployment strategies and best practices for maintaining system performance at scale, providing attendees with actionable knowledge they can apply in their own organizations.

OpenSearch provides several tools for search and information retrieval, important in many stages of prompt processing. However, to be used in a varied range of industries and regulations such as healthcare, telco or finance, the solution must be proven to be secure, robust and highly available.

In this talk we will discuss how to build a secure conversational chat infra from the ground up to answer your prompts, using OpenSearch. We will explore how we can use existing features of OpenSearch to build the basis of this infra, e.g. vector search and RAG. We will discuss the latest technologies such as confidential computing and hardware acceleration to get the most out of the environment. We will also explore an OpenSearch VM operator that helps with the lifecycle of the cluster. It provides the features needed to scale, upgrade, specialize nodes in different roles according to the needs and manage security such as certificate rotation and user management.

Versioning OpenSearch indexes is critical for maintaining data integrity, ensuring seamless updates, and enabling rollback capabilities in modern CICD pipelines. This presentation explores how Terraform and Python can be combined to manage the versioning of OpenSearch indexes effectively. By leveraging Terraform’s infrastructure-as-code capabilities and Python’s scripting flexibility, teams can automate the lifecycle management of indexes, ensuring consistent reintegration and streamlined operations. Key benefits include enhanced automation, improved traceability, and reduced manual overhead.

Building a communication platform that handles diverse data types—emails, chats, attachments, and files—can be complex. Storing and organizing this data in a way that makes it searchable, optimizable, and queryable is key to providing valuable insights. Using OpenSearch as the backbone for data storage and search, this presentation explores how to structure multi-modal data and integrate it with Retrieval-Augmented Generation (RAG) applications powered by Large Language Models (LLMs).

The goal is to help users query and generate contextual replies based on a wide variety of input formats. We will share a use case showcasing how OpenSearch can be leveraged to create a seamless experience for managing and retrieving text-based and multimodal data efficiently. Tips and insights from our experience building such an application will be shared, covering best practices for managing data at scale, optimizing search performance, and implementing LLM-based feedback mechanisms to assist users in formulating contextually relevant responses.

Join us for an in-depth exploration of tiered caching in OpenSearch, a powerful search and analytics engine. Caching plays a crucial role in optimizing performance for performance-intensive applications like OpenSearch. This session delves into the limitations of on-heap caching and introduces tiered caching, a multi-layered caching approach that balances performance and capacity. Learn how tiered caching leverages both on-heap and disk-based caches to store larger datasets efficiently without consuming excessive heap memory. Discover the workings of tiered caching, its benefits, and when to implement it. Uncover performance test results comparing tiered caching with the default on-heap cache, and get insights into future enhancements in tiered caching technology. Don't miss this opportunity to enhance your understanding of caching strategies in OpenSearch for improved query response times and application performance.

In observability and analytics workloads, aggregation queries are fundamental but often become a performance bottleneck as OpenSearch aggregations scale linearly with document count, leading to unpredictable latencies and resource consumption.

This session presents a technical deep dive into how we improved performance of OpenSearch aggregations by implementing star tree index, a sophisticated multi-field indexing structure that delivers up to 100x faster aggregations while maintaining predictable query latencies.

We'll walk through the complete journey of implementing star tree index in OpenSearch, from initial design challenges to production ready feature development.

The presentation covers the technical architecture that enables pre-computing aggregations during indexing, the complex merge operations that maintain performance during segment merges, and the sophisticated query algorithms that leverage star nodes for efficient pruning.

Through real-world benchmarks from various analytical workloads, we'll demonstrate how star tree indexes achieve dramatic performance improvements while managing storage overhead to be less than 1% for typical use cases along with tradeoffs.

Who watches the watchmen?

Released several years ago, the open-source project Home Assistant is becoming increasingly popular for managing our smart homes. Its plugin architecture and the large number of supported protocols provide real versatility. But once set up, how can we be sure that everything is running smoothly? How can we make use of the collected data?

First, we will demonstrate how to collect data from Home Assistant using its plugin. Then, we will explore our data with the help of the OpenSearch Dashboard. Finally, we will show how to make our smart home devices' data useful through anomaly detection and alerting.

In a comprehensive demonstration based on a real-life scenario, we will thoroughly explore all facets of OpenSearch monitoring.

Since the inception of OpenSearch, the indexing capabilities of OpenSearch have evolved to become faster at parsing documents, handling inter node traffic, replicating documents, and creating more efficient data structures. This has mode OpenSearch faster at indexing. The talk will cover how OpenSearch is able to improve indexing one use-case at a time, what is the methodology followed, how are the improvements measured, and how they translated to user success stories.

Also, with too many optimizations, comes the responsibility of understanding those, and enabling those for your use-cases.
In OpenSearch, ABC templates are predefined system templates designed to simplify the process of configuring indexes for various use cases by providing predefined settings and configurations out of the box. These templates encapsulate various optimized settings, mappings, and configurations tailored for different use cases, eliminating the need to manually handle each setting individually. The talk will also add details on how the idea came along, and how it provide a seamless out of box experience for new and seasoned users of OpenSearch.

With NiFi version 2.x, the ability to interface with OpenSearch using pre-built Python components (processors) was introduced.

These processors address various needs related to AI solutions, specifically vector embedding using tools like ChatGPT or HuggingFace. The selection and implementation of these components were partly driven by the increasing interest in these topics in recent times.

In the presentation, we will showcase an ETL flow for populating an OpenSearch index with vector fields. The flow is built using NiFi for both data acquisition and vector embedding generation, as well as querying via the dedicated OpenSearch components.

A growing trend in recent times is the demand for AI systems that use local engines, independent of external access.

As an example of possible application, we will present a solution based on customizing PutOpenSearchVector QueryOpenSearchVector processors to perform vector embedding and search using local LLMs based on Ollama.

In this session, we'll delve into the critical optimizations and fine-tuning strategies that propelled HDI Germany P&C and Life Insurance, a Talanx AG subsidiary to develop a high-performance, productionized Retrieval-Augmented Generation (RAG) based GenAI solution. Leveraging OpenSearch with hybrid search, our system now empowers 300+ employees. Our journey from concept to large-scale deployment uncovered numerous minor yet powerful tweaks that significantly enhanced both, speed and accuracy of our Opensearch indexing and retrieval mechanisms. We’ll dissect the key parameters crucial for optimal RAG performance, presenting benchmarks and metrics to quantify these improvements achieved through the optimization process.

Attendees will gain practical insights into maximizing OpenSearch's potential for RAG applications, learning how to leverage these often-overlooked adjustments to achieve substantial performance gains.
Join us for an in-depth exploration of the nuances that can make a significant difference in your OpenSearch-powered RAG solutions, and leave with actionable tips to implement in your own projects.

With the rapid rise of Large Vision Language Models, radically new approaches to multimodal information retrieval have been proposed. Methods like ColPali and Document Screenshot Embeddings bypass the classical document preprocessing pipeline by representing document pages only in terms of vision based image patches. This allows unprecedented accuracy for Retrieval Augmented Generation (RAG) in answering questions about figures, tables, charts and visually complex documents like Powerpoint slides, without the hassle of complex customized extraction pipelines. And unlike earlier multimodal approaches based on CLIP (Contrastive Language-Image Pre-Training) the new approach also enables retrieval of detailed text passages from normal text documents. In this talk we will present how these methods can be implemented using OpenSearch, how this compares to other implementations, and evaluate how well the resulting multimodal RAG system performs in terms of search effectiveness and efficiency.

Machine learning can feel complex, but what if it didn’t have to be? OpenSearch takes the power of ML and puts it right at your fingertips—no extra headaches required. From detecting anomalies in real time to delivering hyper-relevant search results with vector data, OpenSearch is the ultimate partner for turning your data into actionable insights.

In this session, we’ll explore how OpenSearch effortlessly integrates machine learning into your workflows, whether through k-NN search, anomaly detection, or real-time predictions. We’ll also showcase innovative use cases like fraud detection, personalized recommendations, and predictive analytics—all powered by OpenSearch.

If you’re looking to make machine learning accessible, scalable, and downright useful, this talk will inspire you to rethink how you approach data and decision-making with OpenSearch.

Many OS users, especially those new to Lucene-based search engines, struggle with sharding best practices. This often results in inefficient shard sizes, leading to performance issues and resource waste. Our talk introduces a novel approach: Dynamic Sharding Strategy.

We'll explore how this strategy adaptively determines the optimal primary shard count for indices, particularly beneficial for time-series data with fluctuating ingestion rates. By analyzing past ingestion rates, cluster resources, and workload patterns, our method predicts and applies suitable shard counts for each index rollover.

We'll discuss
* Common sharding pitfalls and their impact
* The idea behind our dynamic sharding approach
* Implementation as a Python script integrated with OpenSearch
* Real-world case studies and performance improvements
* Potential for native integration into OpenSearch's Index State Management

This talk aims to empower OpenSearch users with a practical solution for maintaining optimal cluster performance, reducing storage skewness, and minimizing manual interventions in shard management. Join us to learn how dynamic sharding can revolutionize your OpenSearch cluster efficiency

This talk will explore how OpenSearch can be leveraged to build a centralized and multi-tenant log analysis platform that provides logging-as-a-service to several Kubernetes enterprise environments.

We’ll address solutions for critical topics like cluster and tenant separation strategies, Kubernetes integration, automation, capacity management and monitoring best practices. Additionally, we’ll present our approach to non-technical challenges such as handling diverse tenant requirements, effective customer support and different areas of tenant collaboration.

Throughout the talk, there will be a strong emphasis on standardization as the primary solution component to achieve heterogeneity within the different configuration domains (retention policies, RBAC, etc.), while simultaneously maintaining the flexibility to be able adapt to customers' needs.

Drawing experience from a recent real-world implementation, this session will provide attendees with actionable insights and our lessons learned to overcome technical and operational challenges in building multi-tenant logging platforms using OpenSearch.