[on behalf of Neil Heffernan]
Hello EDM community,
I hope everyone is staying healthy and sane in these very confusing times
making sense of our reaction to the COVID-19 virus.
I wanted to share some good news with the EDM Community; the result of the
2nd Annual WPI-UMASS-UPENN EDM Data Mining Challenge. Those of you that
came to EDM2019 in Chicago might recall my announcement of this
competition.
First a quick recap of the winners:
-
1st place is a early doctoral student at Columbia University
-
2nd place is a team, inspired by a class taught in India!
-
3rd place is Nigel Bosch a member of the EDM community known for his
best student papers in both 2015 and 2017.
I am pleased that there were both competitors new to the EDM community and
those who are a part of the EDM family. NSF funded UMASS-WPI-UPENN to get
new folks involved in these exciting data sciences applications to a useful
problem.
Now I would like to express my gratitude:
-
I would like to give a shout of thanks to my PhD student March Thanaporn
who engineered this competition site. Well done March!
-
This was a team effort with Ryan Baker at UPENN and Bev Woolf at
UMASS(and their respective team members). Thanks Ryan and Bev for being
such great partners!
-
Since ETS was involved in getting us the NAEP data, a big shout out to
Irv Katz at ETS and his team.
-
Thanks to Andrew Olney, Editor of JEDM
<https://jedm.educationaldatamining.org/index.php/JEDM> for accepting
our request to host a special issue of EDM. (While there is no monetary
prize, we are inviting those that did well to contribute a paper to a
special issue of JEDM. [Wouldn't the world be a great place if making
better detectors was something that got monetarily awarded!)
I personally plan to make sure this is an annual event so look for an
announcement of the Third Annual competition soon where we plan to release
a data set of experiments where competitions will be challenged to figure
out how a system should personalize to students characteristics! Maybe a
great thing for folks to work on while social distancing themselves due to
COVID19.
Finally, If you are interested in Data mining and NAEP data and there is a new
job available
<https://www.edsurge.com/jobs/learning-engineering-fellowship-national-center-for-education-statistics-job-at-the-learning-agency-lab>!
Schmidt Futures is one of the supporters of this work--and it’s a great
opportunity to create some powerful competitions!
The full press release can be found on the competition website
<https://sites.google.com/view/dataminingcompetition2019/winners> and and
it is copied below
Press Release
2019 NAEP Educational Data Mining Competition Results Announced
Researchers at Worcester Polytechnic Institute (WPI), University of
Pennsylvania, University of Massachusetts Amherst and ETS have announced
the results of an educational data mining competition, co-sponsored by Big
Data for Education Spoke of the NSF Northeast Big Data Innovation Hub
<https://sites.google.com/view/bigdataforeducation/home> and ETS.
The Competition:
The goal of the competition was to engage leading researchers and promising
doctoral students to push the field of educational data mining forward,
develop metrics for measuring students’ test-taking activities, and help
develop and test evaluation methods for educational analysis.
Competition participants were invited to use data produced by students on
the first half of the National Assessment of Educational Progress (NAEP)
test to predict which student would demonstrate ineffective test-taking
behaviors.
The competition used a NAEP dataset, with permission from the National
Center for Education Statistics (NCES), provided by ETS. Since 1969, the
NAEP is the only assessment that measures U.S. student knowledge nationwide
across academic subjects, in urban, suburban and rural areas.
The goal of this competition was to understand effective and ineffective
test-taking behaviors, and to determine how quickly these behaviors can be
detected. Specifically, participants in the competition were given data
from students’ early performance in the test and asked to predict which
students would later be flagged for potentially not being as motivated in
the second half of the test, identified by whether students spent enough
time on problems and whether they completed all the problems on the test.
The full details of this competition can be found on the competition website
<https://sites.google.com/view/dataminingcompetition2019/home>.
What makes this competition different?
This competition was designed to improve the scientific understanding of
student test-taking strategies. The results of this competition show that
as early as two minutes into the test, the best of these algorithms could
predict with 65% accuracy whether or not the data was from a student
student who was not as motivated in the second half of the test.
Professor Neil Heffernan, director of the PhD program in Learning Sciences
and Technology at WPI and one of the organizers, "The Nation's Report
Card’s mission is to show the trend line of our nation's progress in
developing student knowledge. This competition is one step in helping to
improve our understanding of the NAEP, as there is a concern that students
might not be taking the NAEP test as seriously as they used to. For
instance, we could use this data to identify a student who is potentially
not motivated throughout the test, and between sections, invite the
student's teacher to offer encouragement. It's too early to know how NAEP
should use these algorithms, but this competition could be an important
step in developing appropriate interventions"
Details on the data being measured are available at the competition’s
website <https://sites.google.com/view/dataminingcompetition2019/dataset>.
Participation:
The Nation's Report Card 2019 Data Mining Competition had 89 individual and
team participants in the competition, totaling 723 submissions. Researchers
and students from 11 countries and 24 U.S. states participated in the
competition. Some of the research teams were made up entirely of
undergraduates. The organizers are pleased that this competition inspired
undergraduates to care about educational data and become interested in its
use in research.
The Results and the Winners:
Winners were judged based on the final score of their submission using the
evaluation criteria specified in our competition website
<https://sites.google.com/view/dataminingcompetition2019/submission>.
The first place winner was Nathan Levin from Teachers College, Columbia
University in <https://sites.google.com/view/levinnathan/home> New York
City. He constructed and refined features based on student click data and
the time students spent working on problems. He then applied XGBoost
Regressor on the final feature set.
The second place winners were Nirmal Patel
<https://www.linkedin.com/in/nirmalpatel21/?originalSubdomain=in>, Aditya
Sharma, and Tirth Shah from Playpower Labs. They constructed a large number
of features using the results of their previous research, many of which
were inspired by Process Mining and Curriculum Pacing. They then applied
Genetic Algorithm-based feature selection and modeling. The predictions
from multiple models were then assembled together to create a single final
prediction.
The third place winner was Assistant Professor Nigel Bosch
<https://ischool.illinois.edu/people/nigel-bosch> from the iSchool at the
University of Illinois Urbana-Champaign. He constructed a large number of
features (> 4,000) using both domain knowledge and automatic feature
engineering methods, specifically TSFRESH and FeatureTools.
Participants of the top submissions will receive an invitation to submit
their work and findings to a special issue of the Journal of Educational
Data Mining. This should help to further improve the field’s understanding
of this important work.
Honorary Mentions:
Among all of the participating teams, two additional teams showed
outstanding efforts and achieved impressive results in both the
leaderboard and the final test set: KLETech B Division from KLE
Technological University (Huballi, India) and LTWZ from the Columbia
University (New York City) and the University of Arizona (Tucson).
KLETech B Division treated the hidden dataset as three different tasks and
developed a model for each task based on the different amounts of
information provided (e.g., only the first 10 minutes, only the first 20
minutes, and all 30 minutes of log data). LTWZ developed their model using
features based on student test-taking behaviors, such as the frequency of
how often each student checks the test timer.
The Annual Competition:
This was the second competition in what is anticipated to be an annual
learning analytics competition. The first competition involved 11-year
longitudinal predictions of student success. The third competition will be
announced in a few months, and will be designed to challenge researchers to
determine how to personalize learning.
Competition Organizers:
Technical Directors: Thanaporn “March” Patikorn
<https://sites.google.com/site/tpatikorn>,1 Neil Heffernan
<https://scholar.google.com/citations?user=dmCZvnsAAAAJ&hl=en>1
Organizers: Ryan Baker
<https://www.gse.upenn.edu/academics/faculty-directory/baker>,2 Beverly
Woolf <https://www.cics.umass.edu/faculty/directory/woolf_beverly>,3 Irvin
Katz <https://scholar.google.com/citations?user=qLI_bp4AAAAJ&hl=en>,4 Carol
Forsyth <https://scholar.google.com/citations?user=gTrGB0QAAAAJ&hl=en>4
and Jaclyn
Ocumpaugh <https://scholar.google.com/citations?user=uCl6NE4AAAAJ&hl=en>2
1 Worcester Polytechnic Institute, 2 University of Pennsylvania
3 University of Massachusetts-Amherst, 4 Educational Testing Service
Acknowledgements:
The running of this competition was funded by National Science Foundation
grants to UMASS, WPI and UPENN (1636782
<https://www.nsf.gov/awardsearch/showAward?AWD_ID=1636782&HistoricalAwards=false>
, 1636847
<https://nsf.gov/awardsearch/showAward?AWD_ID=1636847&HistoricalAwards=false>,
and 1661987
<https://www.nsf.gov/awardsearch/showAward?AWD_ID=1661987&HistoricalAwards=false>
) but the opinions expressed here are not those of NSF. We kindly thank
NCES for helping to make the NAEP data available. We thank ETS for helping
coordinate the use of this NAEP data to run this competition.
--
Neil Heffernan <http://www.cs.wpi.edu/~nth> nth@xxxxxxx neilheffernan.net
<https://www.neilheffernan.net/>
The William Smith Dean's Professor of Computer Science
Director of the PhD Program in Learning Science and Technologies
<https://www.wpi.edu/academics/departments/learning-sciences-technologies>
Department of Computer Science- Fuller Labs Room 237
100 Institute Road
Worcester Polytechnic Institute, Worcester, MA 01609
Office (508) 831-5569 ; Cell (774) 239-6826
Founder and Board President of The ASSISTments Foundation
<http://www.assistments.org/>