Name: A Tool for Eyeballing: Better Than NDCG? - Stavros Macrakis, OpenSearch @ AWS & Eric Pugh, OpenSource Connections
Start: 2025-04-30T14:30:00+0200
End: 2025-04-30T15:10:00+0200

In-person
30 April - 1 May 2025
Learn More and Register to Attend

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for OpenSearchCon Europe 2025 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

Please note: This schedule is automatically displayed in Central European Summer Time (UTC+02:00). To see the schedule in your preferred timezone, please select from the drop-down located at the bottom of the menu to the right.

IMPORTANT NOTE: Timing of sessions and room locations are subject to change.

Wednesday April 30, 2025 14:30 - 15:10 CEST

Zurich I

There are lots of statistics we can calculate for search relevance: NDCG, precision, MAP, …. OpenSearch 3.0 calculates and displays them for you.

But metrics don’t tell the whole story. In particular, NDCG can’t give you the qualitative analysis that one algorithm is great for author names, and a different one is good for shapes and sizes. For that, you need eyeballing, that is, human intuition and judgment.

That’s why we’ve built tools to help you dig deeper into your search results. The Search Comparison tool in OpenSearch 2.11 is just the starting point – in 3.0, it lets you define search configurations and test sets of multiple queries and determine which queries’ results have changed the most… and then zoom in to the individual cases. You can integrate data on judgments and query frequency from User Behavior Insights for deeper analysis.

In this talk, we’ll discuss how traditional statistical metrics are complemented by eyeballing, how the new tools in OpenSearch 3.0 support disciplined eyeballing, and how you can use them to improve the quality of your search results.

Speakers

Stavros Macrakis

Product Manager, Search, OpenSearch @ AWS

Stavros is passionate about search relevance. At AWS, Google, GLG, FAST, and Lycos, he has worked with a wide variety of organizations and applications, and is frequently astounded at how even large, sophisticated organizations fail to collect the data needed to evaluate and tune... Read More →

Eric Pugh

Co-Founder, OpenSource Connections

Eric Pugh is the co-founder of OpenSource Connections. Today he helps OSC’s clients, especially those in the ecommerce space, build their own search teams and improve their search maturity. Fascinated by the craft of software development, Eric Pugh has been involved in the open... Read More →

Wednesday April 30, 2025 14:30 - 15:10 CEST
Zurich I

OpenSearchCon Europe 2025

Stavros Macrakis

Eric Pugh

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!