OpenSearchCon Europe 2025

Abstract:

Contributing to an open-source project like OpenSearch can seem daunting, especially without prior knowledge of the product. In this presentation, I will share my personal journey of starting as a complete beginner and becoming an active contributor to OpenSearch.

Description:

Open-source contributions are a journey of growth, collaboration and innovation. I started contributing to OpenSearch about 8 months back. I will discuss the strategies and resources that helped me to overcome the initial learning curve, including understanding the codebase, finding beginner-friendly issues, and engaging with the community. I contributed to 3 different projects so far (OpenSearch core, Security and ml-commons). I will go through some of these PRs and explain how and why I chose these tasks. I will talk about different meetups like triage meetings, community meetings, code walkthrough sessions and conferences and what we can learn from these.

Key Takeaways:

Practical steps to begin contributing to OpenSearch.
Overcoming the learning curve and finding beginner-friendly issues.
Leveraging community support and resources for successful contributions.

As a Maintainer of the OpenSearch Specification Repository, I’d like to share how we addressed a major challenge in maintaining OpenSearch clients (Ruby, Python, Java, Node.js).

Every OpenSearch update required manual replication of changes across all clients, consuming countless hours since version 1.0. By version 2.18, it became clear we needed a scalable solution. Our team built a Specification Repository that centralizes all possible request and response schemas, enabling automated client code generation.

We use OpenAPI specifications and a custom testing framework to ensure consistency, minimize human error, and accelerate development. This approach eliminates repetitive work, improves reliability, and scales better with OpenSearch's growth.

I will demonstrate how the repository works, share code generation examples, and discuss its impact on reducing maintenance overhead while improving client reliability. Additional resources showcasing real-world applications will be provided.

This solution not only streamlines client updates but also sets a foundation for innovation in OpenSearch's ecosystem.

Join us to explore how user research drives technical innovation in OpenSearch. Discover how our UX team rebuilt OpenSearch Dashboards' information architecture by starting with comprehensive user personas of Security and SRE teams and optimized for the workflows of real-time operational analytics.

We'll share our research insights, persona frameworks, and the resulting information architecture that transformed our UI approach.

Key topics include:

• How security and observability personas shaped our core design decisions
• Creating an information hierarchy that supports real-time analysis workflows
• Mapping common patterns between security analytics and observability use cases
• Design patterns that emerged from our persona-driven approach
• Future extensibility opportunities for the community

Whether you're contributing to OpenSearch or building analytics tools, you'll gain practical insights into designing interfaces that scale across diverse use cases. We'll share our learnings from the recent redesign of the information architecture and invite community feedback on our proposed framework.

Unlock the insights hidden within unstructured PDF documents! Many PDFs contain multimodal elements like text, tables, and images, and relying solely on text-based processing risks overlooking critical information. This session explores two powerful approaches to address this challenge: building specialized pipelines that integrate OCR and ML models for handling diverse modalities, and leveraging cutting-edge Vision-Language Models like ColPali to represent multimodal data in a unified format. Join us to discover how both methods can be applied to build an intelligent conversational search application using open-source technology, OpenSearch by leveraging it's powerful search and ingest pipelines. The session includes a live demonstration to showcase practical implementations, empowering you to choose the approach that best fits your needs!

In modern big data ecosystems like Databricks, logs from different jobs, stages, and components can be highly diverse, making it difficult to gain comprehensive insights into system health, performance, and operational flow. This presentation will explore how to consolidate these disparate logs into a unified OpenSearch platform, enabling advanced log analysis and real-time visibility. By leveraging large language models (LLMs) and contextual dashboards, we will demonstrate how to create a "single pane of glass" for monitoring entire operational workflows. Attendees will learn how this approach improves visibility, traceability, and transparency, enabling quicker actions and decisions based on knowledge-driven insights. Key topics include log mapping, lineage tracking, and using OpenSearch to facilitate both broad and granular insights for operational teams. By the end of this session, you'll understand how to streamline complex log data from systems like Databricks, empowering your teams with smarter, faster decision-making and more efficient troubleshooting.

OpenSearch provides several tools for search and information retrieval, important in many stages of prompt processing. However, to be used in a varied range of industries and regulations such as healthcare, telco or finance, the solution must be proven to be secure, robust and highly available.

In this talk we will discuss how to build a secure conversational chat infra from the ground up to answer your prompts, using OpenSearch. We will explore how we can use existing features of OpenSearch to build the basis of this infra, e.g. vector search and RAG. We will discuss the latest technologies such as confidential computing and hardware acceleration to get the most out of the environment. We will also explore an OpenSearch VM operator that helps with the lifecycle of the cluster. It provides the features needed to scale, upgrade, specialize nodes in different roles according to the needs and manage security such as certificate rotation and user management.

Versioning OpenSearch indexes is critical for maintaining data integrity, ensuring seamless updates, and enabling rollback capabilities in modern CICD pipelines. This presentation explores how Terraform and Python can be combined to manage the versioning of OpenSearch indexes effectively. By leveraging Terraform’s infrastructure-as-code capabilities and Python’s scripting flexibility, teams can automate the lifecycle management of indexes, ensuring consistent reintegration and streamlined operations. Key benefits include enhanced automation, improved traceability, and reduced manual overhead.

In this technical session, I'll demonstrate practical implementations of OpenSearch Security Analytics powered by community-driven detection rules. Through hands-on examples, you'll learn:

1. Configuring Security Analytics dashboards for threat detection
2. Building and sharing custom detection rules that benefit the community
3. Testing and validating community-contributed security rules
4. Contributing your validated rules back to OpenSearch

The demonstrations will cover security rule syntax, detection logic, and dashboard configuration. You'll learn both the technical implementation and the collaborative workflow for sharing security rules with the community. We'll explore how shared security patterns strengthen the entire OpenSearch ecosystem, turning individual security insights into community-wide protection.

Target Audience: Security practitioners and OpenSearch operators with basic security monitoring knowledge.

Join us for an in-depth exploration of tiered caching in OpenSearch, a powerful search and analytics engine. Caching plays a crucial role in optimizing performance for performance-intensive applications like OpenSearch. This session delves into the limitations of on-heap caching and introduces tiered caching, a multi-layered caching approach that balances performance and capacity. Learn how tiered caching leverages both on-heap and disk-based caches to store larger datasets efficiently without consuming excessive heap memory. Discover the workings of tiered caching, its benefits, and when to implement it. Uncover performance test results comparing tiered caching with the default on-heap cache, and get insights into future enhancements in tiered caching technology. Don't miss this opportunity to enhance your understanding of caching strategies in OpenSearch for improved query response times and application performance.

Building a communication platform that handles diverse data types—emails, chats, attachments, and files—can be complex. Storing and organizing this data in a way that makes it searchable, optimizable, and queryable is key to providing valuable insights. Using OpenSearch as the backbone for data storage and search, this presentation explores how to structure multi-modal data and integrate it with Retrieval-Augmented Generation (RAG) applications powered by Large Language Models (LLMs).

The goal is to help users query and generate contextual replies based on a wide variety of input formats. We will share a use case showcasing how OpenSearch can be leveraged to create a seamless experience for managing and retrieving text-based and multimodal data efficiently. Tips and insights from our experience building such an application will be shared, covering best practices for managing data at scale, optimizing search performance, and implementing LLM-based feedback mechanisms to assist users in formulating contextually relevant responses.