Measuring What Matters: Evaluation as a Driver of Generative AI
Deep generative models are advancing the state of the art in artificial intelligence by effectively capturing and recreating complex patterns in high-dimensional, real-world data. Recent breakthroughs in generating text, images, and video have led to the development of new ecosystems and are already making a tangible impact on the world. Applications in other modalities are expected to soon experience similar growth.
However, the evaluation of generative AI systems presents greater challenges compared to traditional machine-learning evaluation, as there is rarely a single correct output. Multimodal generative models introduce additional complexities, requiring assessment of consistency, complementarity, and coherence across different modalities.
This forum will bring together global experts in deep generative modeling – including both problem owners and model developers from industry, academia, and other sectors – across various domains. The goal is to bring about new, effective evaluation methods that will drive future progress in multimodal generative models, alongside practical tools to promote their adoption within the research community.
This event is a follow-up on the successful Wallenberg Advanced Scientific Forum 2025 under a new name and managed by WARA Media & Language.
Date and time
The workshop starts with lunch on November 3rd and ends after lunch on November 6th, all at Skåvsjöholm located close to Stockholm.
Forum structure
The forum is designed to be highly interactive, consisting nearly exclusively of structured and directed discussion in smaller focus groups, rather than panels, talks, or plenaries. To make the most out of the event, we will use the so-called double-diamond framework for structured problem solving. This organises problem solving into two stages, first selecting the right problem(s) to tackle and then finding solutions for these. Each stage has two phases: first divergence (exploring possibilities), then convergence (narrowing down choices). The in-person forum will also be preceded by one or two short online workshops to kickstart the discussion before we meet up on site.
Programme
Tuesday 3 November
| 12:15–13:15 | Lunch |
|---|---|
| 13:15–13:45 | Introduction from the organisersMain room |
| 13:45–14:00 | Lightning talks pt. 1Main room |
| 14:00–15:00 | Icebreaker and poster mingle |
| 15:00–15:15 | Lightning talks pt. 2Main room |
| 15:15–16:00 | Coffee and check-inLobby |
| 16:00–17:30 | Introduction to group workMain room |
| 19:00–21:00 | Dinner |
Wednesday 4 November
| 09:00–09:05 | Group composition announcedMain room |
|---|---|
| 09:05–12:15 | Groups finalise the benchmark card standard pt. 1All roomsIncluding coffee 10:30–11:00 (lobby) |
| 12:15–13:15 | Lunch |
| 13:15–14:15 | Mid-time reports from groups (5 min report + 5 min discussion each)Main room |
| 14:15–17:30 | Groups finalise the benchmark card standard pt. 2All roomsIncluding coffee 15:30–16:00 (lobby) |
| 19:00–21:00 | Dinner |
Thursday 5 November
| 09:00–09:05 | Attendees choose their sub-workshopMain room |
|---|---|
| 09:05–12:15 | Parallel sub-workshops• SE.LLMA: Evaluating linguistic style and cultural aspects in LLMs• Future evaluation of speaking agentsMain room and secondary roomBoth including joint coffee 10:30–11:00 (lobby) |
| 12:15–13:15 | Lunch |
| 13:15–13:45 | Sub-workshops report to each other |
| 13:45–15:00 | Paper writing introductionMain room |
| 15:00–17:30 | Paper writing pt. 1All roomsIncluding coffee 15:30–16:00 (lobby) |
| 19:00–21:00 | Dinner |
Friday 6 November
| 09:00–10:30 | Paper writing pt. 2All rooms |
|---|---|
| 10:30–11:00 | CoffeeLobby |
| 11:00–12:15 | Conclusion and next stepsMain room |
| 12:15–13:15 | Lunch and departure |
Programme committee
Jonas Beskow, Professor, Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden
Desmond Elliott, Associate Professor, Department of Computing Science, University of Copenhagen, Denmark
Johanna Björklund, Docent and Professor, Department of Computing Science, Umeå University, Sweden
Gustav Eje Henter, Docent and Associate Professor, Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden
Jonas Unger, Professor, Department of Science and Technology, Linköping University, Sweden
Lonni Besançon, Assistant Professor, Department of Science and Technology, Linköping University, Sweden
Questions about event logistics can be sent to [email protected]