ALLIES Evaluation for Autonomous Speaker Diarization Systems

 

The ALLIES project aims at catalysing the development of autonomous lifelong intelligent systems by providing the community with scenarios, evaluation plans and metrics to evaluate those systems.

ALLIES focuses on two tasks: speaker segmentation, and machine translation. The speaker segmentation evaluation relies on a new corpus of audio-visual documents (news, debates, talk-shows…) from the French channel LCP. The evaluation is coordinated by the LNE (Laboratoire national de métrologie et d’essais, France), LIUM (Le Mans Université, France), IDIAP (Switzerland) and UPC (Spain).

The ALLIES evaluation focuses on freeing speaker diarization systems from the need of machine learning expert interventions upon three axes:

  • Lifelong learning: automatic systems use the stream of incoming data to update their knowledge and adapt to new data across time in order to sustain performance across time;

  • Interactive learning with user initiative: given the current knowledge of an automatic system and a set of documents to process, a human domain expert provides corrections on the automatic systems outputs until the system produces a good enough output.

  • Active learning with system initiative: the system itself asks the domain expert corrections of its diarization hypothesis.

 

Schedule:

  • 14th of December: Announcement of the ALLIES / ALBAYZIN evaluation
  • 1st of January until 31st of March: Registration for participants is open
  • 1st of March: Release of Development data
  • 1st of March: Beat platform is open for development
  • 1st of June: Evaluation data is released
  • 30th of June: Final submission
  • End of July: Paper submission deadline
  • November: Iberspeech conference in Valladolid