OLI and Feedback Loops
ON THIS PAGE:
One of the most powerful features of technology-enhanced learning environments is that they enable us to embed ongoing formative assessment and feedback into instructional activities.
By “feedback” we mean information derived from student activities that is used to influence or modify further performance.
In the case of feedback to students, we refer to corrections, suggestions and cues that are tailored to the individual’s current performance and that encourage revision and refinement. Many learning studies have shown that students’ learning improves and their understanding deepens when they are given timely and targeted feedback on their work (Butler & Winne, 1995; NRC, 2004.) Regarding the timing and frequency of feedback, the best learning outcomes occur when feedback comes immediately after the students’ response but not before the student is ready to revise his or her understanding (Corbett & Anderson, 2001).
All OLI courses include frequent opportunities for students to assess their own learning and receive immediate context specific feedback. Fortunately, we benefit from inheriting some of the best work done in the area of online tutoring from Carnegie Mellon and University of Pittsburgh faculty. Many OLI courses feature Cognitive Tutors and “mini-tutors” that give students feedback in the problem solving context. A Cognitive Tutor is a computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutor - i.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well. This approach differs from traditional computer aided instruction in that traditional instruction gives didactic feedback to students on their final answers whereas the Cognitive Tutors and “mini-tutors” provide context specific assistance during the problem solving process.
On the next few pages, you will see examples from some OLI courses of the mini-tutors and other activities that give feedback to students. On this page we discuss three other critical feedback loops: feedback to course designers; feedback to science of learning researchers; and, our new area of study, feedback to instructors.
Feedback to Course Designers
During the design process and during use, we continuously evaluate the courses by studying data from student use and learning.
With the students' permission, we digitally record interaction level detail of student learning activities in all OLI courses and labs. The student learning data is stored in a standard SQL database and can be mined using standard query tools. We analyze the student activity data to learn how students are using the material and the impact of their use patterns on learning outcomes. In each of the course description pages that follow, we will share some of the interesting results we have gathered from analyzing the log data from the interactive activies described:
We also use student log data to evaluate and iteratively refine parts of the course. For example, by examining the data from students working through our Causal and Statistical Reasoning course, we observed that students were engaging in all of the learning activities we had designed but still failed on a target skill of “building causal response structures.” We constructed six additional learning activities and mini-tutors designed to support students to understand and practice this target skill. The follwing semester we analyzed the student data again to confirm that students were using the new activities and that using the activities resulted in the students’ learning the target skill.
Feedback to Learning Science Researchers
Some OLI courses also serve as part of the research environment for the Pittsburgh Science of Learning Center (PSLC). Learning researchers affiliated with the PSLC can embed experimental manipulations in OLI courses to test specific learning theories. The researchers then analyze the data collected by the OLI logging service using the PSLC datashop tools.
The PSLC datashop has created a number of tools specifically designed to generate meaningful displays of student learning data.
The graph shown above was created using the PSLC datashop learning curve analysis tool. The data was generated as a group of students worked through a stoichiometry tutor in the chemistry course. The curve tracks students' decreasing need for tutorial assistance as they practice relevant knowledge components in the domain, like "Select-Solution-Concentration-Reason". The Y-axis is the Assistance Score, calculated as a function of the number of hints the student requests and the number of incorrect student attempts for the selected knowledge component. The X-axis is the Opportunity Number, the number of times the student had the opportunity to demonstrate this knowledge component.
The curve is trending downward which means that the student needed fewer hints and gave fewer incorrect answers before giving a correct answer as the student progressed through the material. A downward sloping curve shows that learning is occurring. The graph shown is for all students in the group and for all knowledge components in the stoichiometry tutor. The datashop analysis tool also allows researchers to look at learning curve graphs for each individual knowledge component and to identify individual knowledge components which may show a learning curve that is not trending so neatly downward. This kind of analysis is useful to researchers when comparing results from alternate experimental conditions and may also be useful to course designers when evaluating a need for revision of the instructional approach.
Feedback to Instructors
We capture the student data in real time; however, the analysis that drives the feedback loops described above is retrospective (not real time) and is generated and used by the researchers and course designers (not by the instructors).
In traditional face-to-face classrooms, the feedback loop between student learning activities and instructor activities is always cycling unmediated by technology. The traditional instructor-student feedback loop works well in classrooms that have an expert teacher and a not unreasonably heterogeneous class. For example, the instructor recognizes a need for additional explanation or practice by listening students' questions or by seeing confused looks on students' faces. In an online environment, this dynamic feedback loop is broken. The richness of the data we are collecting about student use and learning provides an unprecedented opportunity for keeping instructors in tune with the many aspects of students’ learning.
Ideally, OLI courses can assist instructors in addressing the challenges they confront as a result of the increasing variability in their students’ background knowledge, relevant skills and future goals. Creating an effective feedback loop to instructors using the OLI courses is our current area of research. Some of the research questions we are exploring in this area are:What measures best capture when students “get it” ? How do we construct an accurate representation of students’ knowledge? What is the most effective way to present this information to instructors? How can we best support instructors in making use of such information?
First published on Jul 06, 2007.
