[on behalf of Andrew Lan] A NIPS 2014 Workshop /Saturday, December 13, 2014 Montreal, Canada / In typical applications of machine learning (ML), humans typically enter the process at an early stage, in determining an initial representation of the problem and in preparing the data, and at a late stage, in interpreting and making decisions based on the results. Consequently, the bulk of the ML literature deals with such situations. Much less research has been devoted to ML involving "humans-in-the-loop," where humans play a more intrinsic role in the process, interacting with the ML system to iterate towards a solution to which both humans and machines have contributed. In these situations, the goal is to optimize some quantity that can be obtained only by evaluating human responses and judgments. Examples of this hybrid, "human-in-the-loop" ML approach include: * ML-based education, where a scheduling system acquires information about learners with the goal of selecting and recommending optimal lessons; * Adaptive testing in psychological surveys, educational assessments, and recommender systems, where the system acquires testees' responses and selects the next item in an adaptive and automated manner; * Interactive topic modeling, where human interpretations of the topics are used to iteratively refine an estimated model; * Image classification, where human judgments can be leveraged to improve the quality and information content of image features or classifiers. The key difference between typical ML problems and problems involving "humans-in-the-loop" and is that in the latter case we aim to fit a model of human behavior as we collect data from subjects and adapt the experiments we conduct based on our model fit. This difference demands flexible and robust algorithms and systems, since the resulting adaptive experimental design depends on potentially unreliable human feedback (e.g., humans might game the system, make mistakes, or act lazily or adversarially). Moreover, the "humans-in-the-loop" paradigm requires a statistical model for human interactions with the environment, which controls how the experimental design adapts to human feedback; such designs are, in general, difficult to construct due to the complex nature of human behavior. Suitable algorithms also need to be very accurate and reliable, since humans prefer a minimal amount of interaction with ML systems; this aspect also prevents the use of computationally intensive parameter selection methods (e.g., a simple grid search over the parameter space). These requirements and real-world constraints render "humans-in-the-loop" ML problems much more challenging than more standard ML problems. In this workshop, we will focus on the emerging new theories, algorithms, and applications of human-in-the-loop ML algorithms. Creating and estimating statistical models of human behavior and developing computationally efficient and accurate methods will be a focal point of the workshop. This human-behavior aspect of ML has not been well studied in other fields that rely on human inputs such as active learning and experimental design. We will also explore other potential interesting applications involving humans in the loop in different fields, including, for example, education, crowdsourcing, mobile health, pain management, security, defense, psychology, game theory, and economics. The goal of this workshop is to bring together experts from different fields of ML, cognitive and behavioral sciences, and human-computer interaction (HCI) to explore the interdisciplinary nature of research on this topic. In particular, we aim to elicit new connections among these diverse fields, identify novel tools and models that can be transferred from one to the other, and explore novel ML applications that will benefit from the human-in-the-loop of ML algorithms paradigm. We believe that a successful workshop will lead to new research directions in a variety of areas and will also inspire the development of novel theories and tools. FORMAT This workshop will consist of invited talks and paper presentations/posters. We invite research from both academia and industry to attend the workshop. As this is a workshop, there will be no formal proceedings. ORGANIZERS * Richard G. Baraniuk, Rice University * Michael C. Mozer, University of Colorado Boulder * Divyanshu Vats, Rice University * Christoph Studer, Cornell University * Andrew E. Waters, Rice University and Openstax College * Andrew S. Lan, Rice University CALL FOR PAPERS The NIPS 2014 workshop on Human Propelled Machine Learning will held in conjunction with NIPS 2014 on Saturday, December 14, 2014, at the Palais des Congrès de Montréal Convention and Exhibition Center in Montreal, Quebec, Canada. We invite the submission of papers on all topics related to machine learning problems involving one or more "humans-in-the-loop", including but not limited to: * Machine learning-based education * Adaptive testing in psychological surveys, educational assessments, and recommender systems * Interactive topic modeling * Computer vision with human collaboration * Knowledge graph construction * Other novel machine learning applications involving humans-in-the-loop Submissions should follow the regular NIPS paper format. Papers submitted for review do not need to be anonymized. Accepted papers will be made available on the workshop website, since there will be no official proceedings. Accepted papers will be presented either as a talk or as a poster. We welcome submissions with either novel results that have not been published previously or extensions of the authors' previous work that has been recently published or is under review in another conference or journal. In the interest of spurring the discussion, we also encourage authors to submit extended abstracts and work-in-progress papers with only preliminary results. For additional information, please visit the workshop website: http://dsp.rice.edu/HumanPropelledML_NIPS2014 PAPER SUBMISSION Submissions will be judged on their novelty and potential impact in the emerging field of human propelled machine learning. Please send your submissions via email to hpml.nips2014@xxxxxxxxx <mailto:hpml.nips2014@xxxxxxxxx> Questions about the workshop can be sent to the same e-mail address. IMPORTANT DATES * Paper Submission: *EXTENDED to October 17, 2014 * * Author notification: October 24, 2014 * Camera ready versions of accepted submissions: October 27, 2014 * Final workshop schedule: October 31, 2014 * Workshop: December 13, 2014 For more information about the NIPS conference, please visit http://nips.cc/Conferences/2014 CONFIRMED SPEAKERS * Emma Brunskill, Carnegie Mellon University * Lawrence Carin, Duke University * Todd Coleman, University of California, San Diego * Jordan Boyd-Graber, University of Colorado Boulder * Pedro Domingos, University of Washington * Robert Nowak, University of Wisconsin Madison * Devi Parikh, Virginia Tech * Jacob Whitehill, HarvardX * Beverly Park Woolf, University of Massachusetts Amherst * Yisong Yue, California Institute of Technology -- -------------------------- yours, Andrew