[on behalf of Steve Ritter]
Call for Papers/Participation: First Workshop on Educational A/B Testing at
Scale (at Learning @ Scale 2020)
August 12, 2020, virtual workshop, 12.30PM - 3.30PM EDT (USA)
http://learningatscale.acm.org/las2020/workshops/educational-a-b-testing-at-scale/
"There is no simple path that will take us immediately from the
contemporary amateurism of the college to the professional design of
learning environments and learning experiences. The most important step is
to find a place on campus for a team of individuals who are professionals
in the design of learning environments — learning engineers, if you will."
[2]
The emerging discipline of Learning Engineering is focused on putting into
place tools and processes that use the science of learning as a basis for
improving educational outcomes [3]. A/B testing can be an important part of
this approach [1], but educational software tends to lag other fields in
the use of A/B testing, particularly at scale. This workshop will explore
ways in which A/B testing in educational contexts differs from its use in
other domains and discuss proposals to overcome these challenges so that
A/B testing can become a more useful tool in the learning engineer’s
toolbox.
We invite papers (up to 4 pages in CHI Proceedings format) addressing
issues with conducting A/B tests at scale, including:
• managing unit of assignment issues
• measurement, including both short and long-term outcomes
• practical considerations related to experimenting in school settings,
MOOCs, & other contexts
• ethical and privacy issues
• relating experimental results to learning-science principles
• understanding use cases (core, supplemental, in-school, out-of-school,
etc.)
• accounting for aptitude-treatment interactions
• A/B testing within adaptive software
• adaptive experimentation
• attrition and dropout
• stopping criteria
• User experience issues
• Educator involvement and public perceptions of experimentation
• Balancing practical improvements with generalizable science
Submission Details
----
Submission Deadline: July 31, 2020
Acceptance Notification: by August 7, 2020
CHI2020 Proceedings Format:
https://chi2020.acm.org/authors/chi-proceedings-format/
EasyChair Submission Site:
https://easychair.org/conferences/?conf=eduabtestatscale2020
Papers will be reviewed by the workshop organizers and a small program
committee.
Please direct questions about the workshop to Steve Ritter (
sritter@xxxxxxxxxxxxxxxxxxxx).
Organizers
----
• Steve Ritter, Carnegie Learning
• Neil Heffernan, WPI
• Joseph Jay Williams, University of Toronto
• Burr Settles, Duolingo
• Phil Grimaldi, Kahn Academy
• Derek Lomas, Delft University of Technology
References
----
[1] Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu & Nils
Pohlmann. 2013. Online controlled experiments at large scale. In
Proceedings of the 19th ACM SIGKDD international conference on Knowledge
discovery and data mining (pp. 1168-1176).
[2] Herbert A. Simon. 1967. The job of a college president. Educational
Record, 48, 68-78.
[3] Melina R. Uncapher. 2018. From the science of learning (and
development) to learning engineering, Applied Developmental Science, DOI:
10.1080/10888691.2017.1421437