African Next Voices is a multilingual speech dataset targeting over 3,000 hours of scripted and unscripted audio across Dholuo, Kikuyu, Somali, Kalenjin and Maasai, collected under the Kenya pilot led by the KenCorpus Consortium and funded by the Gates Foundation. Recordings span eleven domains including agriculture, healthcare, financial transactions and digital government services, collected through ethical, community-led processes with speaker-disjoint train/dev/test splits.
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
CommonLID is a community-created language identification benchmark of web text manually annotated for language, covering 109 languages (78 with at least 100 lines of data), totalling over 350,000 annotated lines. It was built as a shared task at the Workshop on Multilingual Data Quality Signals and released under the Common Crawl terms of use for evaluation-only use.
Size
350,000+ annotated lines across 109 languages
Format
Text annotations
License
CC BY
More datasets
View allPolitiKweli is the first Swahili-English code-switched dataset for political misinformation classification in Kenya. It contains 6,345 code-switched texts alongside 22,954 English texts and 211 Swahili texts, sourced from Twitter (now X) and labelled as fake, fact or neutral against a fact-checked reference set built as part of the same study.
Global PIQA is a participatory commonsense reasoning benchmark covering over 100 languages, built by hand by more than 350 researchers from over 65 countries. The non-parallel split covers 136 language varieties across five continents, 18 language families and 24 writing systems, with 100 examples per language and over 50% referencing local foods, customs or other culturally specific elements.