AI models need data to learn about the world. In the past decade, scaling up model parameters and data together has driven tremendous progress in AI. This is the motivation behind building large scale datasets at MLCommons® such as the People's Speech and the Multilingual Spoken Words Corpus. However, datasets are not just about quantity. Research shows that data diversity and quality are also critically important for good performance. We need “bigger” data, but we also need “better” data.
Dynamic Adversarial Data Collection (DADC) is an approach to collecting data that relies on collaboration between humans and AI models, e.g. using the Dynabench platform, to identify and improve accuracy. It is Adversarial because humans collaborate by finding examples of input data that ‘trick’ the model into misclassification errors. It is Dynamic because the model-fooling process is iterative, building a progressively stronger model as more 'tricky' data is discovered and incorporated into training.
DADC has already demonstrated its effectiveness on a number of tasks such as question answering, sentiment analysis, natural language inference and hate speech. While it leads to data that is more creative and diverse, the DADC-collected data often represents unusual corner cases and should be combined with real-world examples to be truly effective.
In the next decade, we envision a shift in focus from quantity alone towards a more balanced approach that incorporates quality, creativity, diversity and collaboration. DADC operates at the next frontier of model capabilities, providing a data- and human-centric tool to find and begin to bridge the gaps between machine and human intelligence.
In a large scale collaboration to address these challenges, researchers from University College London, the University of Oxford, The Alan Turing Institute, Stanford University, the University of North Carolina at Chapel Hill, Carnegie Mellon University, the University of Sheffield, Simon Fraser University, Meta / FAIR, Hugging Face and MLCommons are rethinking model benchmarking through Dynabench: a platform to facilitate DADC research. Dynabench brings models and humans together in an easy, collaborative and community-driven way to push the boundaries of current data collection methods.
To encourage even more exciting collaboration, we will be holding the First Workshop on Dynamic Adversarial Data Collection co-located with NAACL ‘22 in Seattle on the 14th of July. We have an exciting day planned with amazing keynote speakers, a diverse and engaging panel, and presentations from our Shared Task participants. We encourage you to get involved! To stay up to date with all DADC-related developments please follow @DADCWorkshop, @DynabenchAI & @MLCommons on Twitter. Help us keep growing this community effort, and don’t hesitate to get in touch if you would like to be involved.