NIST Evaluations in Speaker Diarization

The National Institute of Standards and Technology (National Institute for Standards and Technology, 2006) (NIST) is an agency of the U.S. Commerce Department's Technology Administration that was created to provide standards and measurements for the U.S. industry. Within NIST, the speech group's mission is to contribute to the advancement of spoken language processing technologies (both speech recognition and understanding) so that spoken language can reliably serve as an alternative modality for the human-computer interface of the future. This is done by defining performance measurement methods, providing reference materials, performing benchmark tests within the research and development community and building prototype systems to show the latest speech technology advances and future applications.

In the last decade NIST has been performing a series of evaluations in the topic of speaker diarization in order to empower research institutions to evaluate their research systems using a common framework, including data and specifications to follow. Each evaluation in speaker diarization has been included within a more general framework including other research areas. From 2000 to 2002 speaker diarization was run within the speaker recognition evaluation (SRE, see NIST Speech Recognition Evaluation (2006)) and from then on it has been included in the Rich Transcription Evaluation (RT eval, see NIST Rich Transcription evaluations, website: http://www.nist.gov/speech/tests/rt (2006)).

ICSI has been participated regularly to the speaker diarization evaluations and also to the speaker recognition and speech to text evaluations. During my stay at ICSI I have been a part of the team entering the latest broadcast news speaker diarization evaluation and on the latest two evaluations of speaker diarization for meetings. In this chapter an overview is first given of the evaluation campaigns in speaker diarization for meetings for the last two years and then the systems that ICSI participated with are explained, as well as which performance they achieved. Finally some personal insight is offered about my opinion of pros and cons of these evaluations.