Candace Thille - Carnegie Mellon University
http://edcommunity.apple.com/ali/item.php?itemID=11365Using intelligent tutoring systems, virtual laboratories, simulations, and frequent opportunities for assessment and feedback, OLI builds courses that are intended to enact instruction - or, more precisely, to enact the kind of dynamic, flexible, and responsive instruction that fosters learning.
As we deliver the instruction, we use technology to collect real-time interaction level data of all student use. We use this data to create four positive feedback loops:
In this article, we describe how OLI uses the web to deliver online instruction that instantiates course designs based on research from the learning sciences.
The article includes interactive examples of tutors, simulations and virtual labs from three OLI courses - engineering statics, chemistry and biology. The reader can try the tutors and simulations in the article and link directly into the OLI courses for further exploration.OLI is an open educational resources project that began in 2002 with a grant from The William and Flora Hewlett Foundation. Like many open educational resources projects, ours makes its courses openly and freely available. However, our courses are not mere collections of material created by individual faculty to support traditional instruction. While our courses are often used by instructors to support classroom instruction, our goal is to create courses that enact instruction that is, to create complete online courses from which learners can learn even if they do not have benefit of an instructor or a class. Our courses are developed by teams of learning scientists, faculty content experts, human computer interaction experts and software engineers. They are the product of a community-based research activity.
The Academic Intersections Journal is about the intersection of technology, scholarly and creative works, and the scholarship of teaching and learning in higher education. This article is about that same intersection and describes how OLI uses technology to deliver theoretically informed instruction that instantiates teaching and learning goals and simutaneously uses technology to collect data on student learning that informs the next iteration of the course and the underlying learning theory.
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies. Each of the course descriptions on the following pages includes a discussion of some of the domain specifcic teaching and learning challenges the OLI course is designed to address
In all OLI courses, we begin our design by articulating a set of student-centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
Following the discussion of the centrality of feedback loops to our work, you will find descriptions and examples of our use of learning theory and feedback loops in the design of the OLI engineering statics and OLI chemistry courses.
The integration of theory and practice that is at the heart of OLI depends on the participation of a diverse community of developers, teachers and learners. As you read through this article, think about the role you might play in improving postsecondary education as part of a community-based research activity.
ON THIS PAGE:
One of the most powerful features of technology-enhanced learning environments is that they enable us to embed ongoing formative assessment and feedback into instructional activities.
By “feedback” we mean information derived from student activities that is used to influence or modify further performance.
In the case of feedback to students, we refer to corrections, suggestions and cues that are tailored to the individual’s current performance and that encourage revision and refinement. Many learning studies have shown that students’ learning improves and their understanding deepens when they are given timely and targeted feedback on their work (Butler & Winne, 1995; NRC, 2004.) Regarding the timing and frequency of feedback, the best learning outcomes occur when feedback comes immediately after the students’ response but not before the student is ready to revise his or her understanding (Corbett & Anderson, 2001).
All OLI courses include frequent opportunities for students to assess their own learning and receive immediate context specific feedback. Fortunately, we benefit from inheriting some of the best work done in the area of online tutoring from Carnegie Mellon and University of Pittsburgh faculty. Many OLI courses feature Cognitive Tutors and “mini-tutors” that give students feedback in the problem solving context. A Cognitive Tutor is a computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutor - i.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well. This approach differs from traditional computer aided instruction in that traditional instruction gives didactic feedback to students on their final answers whereas the Cognitive Tutors and “mini-tutors” provide context specific assistance during the problem solving process.
On the next few pages, you will see examples from some OLI courses of the mini-tutors and other activities that give feedback to students. On this page we discuss three other critical feedback loops: feedback to course designers; feedback to science of learning researchers; and, our new area of study, feedback to instructors.
During the design process and during use, we continuously evaluate the courses by studying data from student use and learning.
With the students' permission, we digitally record interaction level detail of student learning activities in all OLI courses and labs. The student learning data is stored in a standard SQL database and can be mined using standard query tools. We analyze the student activity data to learn how students are using the material and the impact of their use patterns on learning outcomes. In each of the course description pages that follow, we will share some of the interesting results we have gathered from analyzing the log data from the interactive activies described:
We also use student log data to evaluate and iteratively refine parts of the course. For example, by examining the data from students working through our Causal and Statistical Reasoning course, we observed that students were engaging in all of the learning activities we had designed but still failed on a target skill of “building causal response structures.” We constructed six additional learning activities and mini-tutors designed to support students to understand and practice this target skill. The follwing semester we analyzed the student data again to confirm that students were using the new activities and that using the activities resulted in the students’ learning the target skill.
Some OLI courses also serve as part of the research environment for the Pittsburgh Science of Learning Center (PSLC). Learning researchers affiliated with the PSLC can embed experimental manipulations in OLI courses to test specific learning theories. The researchers then analyze the data collected by the OLI logging service using the PSLC datashop tools.
The PSLC datashop has created a number of tools specifically designed to generate meaningful displays of student learning data.
The graph shown above was created using the PSLC datashop learning curve analysis tool. The data was generated as a group of students worked through a stoichiometry tutor in the chemistry course. The curve tracks students' decreasing need for tutorial assistance as they practice relevant knowledge components in the domain, like "Select-Solution-Concentration-Reason". The Y-axis is the Assistance Score, calculated as a function of the number of hints the student requests and the number of incorrect student attempts for the selected knowledge component. The X-axis is the Opportunity Number, the number of times the student had the opportunity to demonstrate this knowledge component.
The curve is trending downward which means that the student needed fewer hints and gave fewer incorrect answers before giving a correct answer as the student progressed through the material. A downward sloping curve shows that learning is occurring. The graph shown is for all students in the group and for all knowledge components in the stoichiometry tutor. The datashop analysis tool also allows researchers to look at learning curve graphs for each individual knowledge component and to identify individual knowledge components which may show a learning curve that is not trending so neatly downward. This kind of analysis is useful to researchers when comparing results from alternate experimental conditions and may also be useful to course designers when evaluating a need for revision of the instructional approach.
We capture the student data in real time; however, the analysis that drives the feedback loops described above is retrospective (not real time) and is generated and used by the researchers and course designers (not by the instructors).
In traditional face-to-face classrooms, the feedback loop between student learning activities and instructor activities is always cycling unmediated by technology. The traditional instructor-student feedback loop works well in classrooms that have an expert teacher and a not unreasonably heterogeneous class. For example, the instructor recognizes a need for additional explanation or practice by listening students' questions or by seeing confused looks on students' faces. In an online environment, this dynamic feedback loop is broken. The richness of the data we are collecting about student use and learning provides an unprecedented opportunity for keeping instructors in tune with the many aspects of students’ learning.
Ideally, OLI courses can assist instructors in addressing the challenges they confront as a result of the increasing variability in their students’ background knowledge, relevant skills and future goals. Creating an effective feedback loop to instructors using the OLI courses is our current area of research. Some of the research questions we are exploring in this area are:What measures best capture when students “get it” ? How do we construct an accurate representation of students’ knowledge? What is the most effective way to present this information to instructors? How can we best support instructors in making use of such information?
ON THIS PAGE:
Statics is a sophomore level engineering course and lays the foundation for design of mechanical systems. In Statics we study methods of quantifying the forces between parts of mechanical, structural, and biological systems. For example, we use statics to design bridges that will safely withstand all the possible forces on it, including its own weight, traffic, wind, changes in temperature, earthquakes.
Prior to developing the OLI statics course, the OLI course authors conducted extensive research to identify shortcomings in traditional statics instruction, and devised classroom methods that take advantage of known approaches that can facilitate learning (Steif and Dollár 2005, Dollár and Steif 2006). At the same time, the authors, along with others, have been involved in efforts to identify the conceptual difficulties in learning statics and developing a testing instrument, the Statics Concept Inventory, to measure a student’s ability to use key statics concepts in isolation (Steif, 2004; Steif, Dollár, Dantzler, 2005; Steif and Hansen, 2007). Throughout the development and use of the course, we use the Statics Concept Inventory as one instrument to measure the effectiveness of the course.
Learning is an active process. In all OLI courses we promote engagement in ways that help the learner select, integrate and retrieve new knowledge.
In the Statics course, we use short economical sections of expository text interspersed with graphical representations and interactive guided simulations. We give students frequent opportunities to assess their learning and receive targeted feedback as they work through problems. A typical example of a learning sequence from the course is three screens that cover the topic of combining concurrent forces.
Click here to enter the OLI Engineering Statics course and view the 3-page sequence.
Within this short section of the course, the student:
The digital envrinmont allows us to make forces and their effects visible to students in ways that are not possible in the traditional classroom. In teaching Statics, simulations of motion are critical to conveying the various effects of forces, and the conditions for equilibrium. In a traditional classroom, neither a traditional textbook, nor an instructor, can offer dynamic simulations with parameters which are controlled by the learner seeking to explore relevant phenomena. In the OLI statics course, learners can experiment with the parameters and see the effects of their experimentation in Interactive Guided Simulations such as the ones shown below. We often introduce the guided simulations with a question for the student to answer and follow it with a succinct description of an observation the student should have made.
Students learn when engaging in problem-solving activities in which they can both succeed and fail, but will get meaningful feedback in either case. In learning-by-doing activities, there is a risk that students will strengthen incorrect knowledge, acquire invalid procedures or strengthen inappropriate connections. Learners need sufficient support so that they do not discover, practice and encode incorrect knowledge. Studies have shown that immediate feedback leads to significant reductions in time taken by students to achieve a desired level of performance (Anderson, Conrad, Corbett; 1989). Learning-by-doing approaches work best when a student has access to support and feedback at exactly the moment that it is needed. The mini tutor below combines simulation with hints and feedback.
Try the mini tutor below.
If you are not sure how to proceed click the hint button. select your answers from the pull down menus and do not hesitate to ask for hints. There are multiple levels of hints for each question, you may continue to ask for hints by clicking the 'get next hint' link at the bottom of the hint window until you reach the final hint that gives you the answer for that step and allows you to continue working on the problem.
Click the image to the left to see a demonstration of the tutor.
The tutor demonstrated to the left and available to use below appears in an advanced section of the OLI Engineering Statics Course on Summing Force Vectors. The tutor is an opportunity for students to do a "self-check" to make sure they understand the concept.
The student is presented with a graphical representation of the problem and asked for the answer. If the student is unsure of the procedure for solving the problem, the first hint provides a link which, when clicked, expands the tutor into the various steps needed to solve the problem. The tutor provides scaffolding to support the student to learn the steps of the procedure when needed. The hints and feedback given by the tutor change depending on which part of the exercise the student is attempting. The tutor recognizes when a student has used the scaffolding and hints. When the student gives the correct answer after having used the scaffolding and hints, the tutor suggests the student to try another problem. The problem statement, hints, feedback and answers are dynamically-generated. The student can work through the tutor multiple times, receiving a different problem each time, until the student is confident that he or she understands the concept and has developed fluency with the procedure. This provides the student with virtually unlimited opportunities for supported practice.
Try the mini tutor below.
If you are not sure how to proceed click the hint button. You may need to click the link in the first hint that expands the tutor into the multiple steps that are required to solve this problem. Type your answers into each box and do not hesitate to ask for hints for each step as you work through the problem. There are multiple levels of hints for each step, you may continue to ask for hints by clicking the 'get next hint' link at the bottom of the hint window until you reach the final hint that gives you the answer for that step and allows you to continue working on the problem.
ON THIS PAGE:
Prior to beginning work on the OLI chemistry course, the chemistry group conducted an analysis of chemistry as practiced versus chemistry as taught. For evidence of the domain of chemistry as practiced, they chose for the Nobel prizes awarded for past fifty years along with all chemistry related articles for one year of the New York Times Science Times and Scientific American News Bites. This led to a conceptual map of the domain based on three main activities: explaining phenomena, analyzing matter, and synthesizing new substances. These activities are supported by a toolbox of basic notational and quantitative schemes. A similar analysis of chemistry as taught, based on the California state content standards and two best selling textbooks, showed significant misalignment, with traditional introductory courses focussing almost exclusively on the toolbox and the explain activity. (Evans, Karabinos, Leinhardt, Yaron; 2006) This misalignment is evidence that traditional courses do not meet one of the basic goals of scientific literacy, that of showing what chemists do. This misalignment also hides from students the very things that are most interesting about the field.
Much of college level chemistry is often taught out of context as a set of abstract mathematical skills. Students employ learning strategies to solve typical text book problems and perform well on traditional chemistry exams but often fail to see either the relationship between the mathematical procedures and the chemical phenomena those procedures represent or the relationship between the chemical phenomena and the real world. The OLI chemistry course is designed to address both of these educational challenges.
We address the challenge of connecting the mathematical procedure to use in chemistry by replacing traditional textbook problems with problems to be constructed and solved in the virtual chemistry lab. We use the virtual chemistry lab to create learning environments with ill-structured, ambiguous problems that require flexible application of procedural knowledge.
We address the second challenge of connecting the procedures of chemistry to the real world by employing scenario based learning. The OLI introductory Chemistry course situates the learning of chemistry in an authentic investigation that addresses questions that are significant to the domain of chemistry and to real world problems.
For example, The OLI Chemistry course unit on stoichiometry is situated in a real world problem of arsenic contamination of the water supply in Bangladesh.
The video above appears early in the course and introduces the student to the problem they will use the tools of chemistry to solve.
The video gives the student an overview of the real world problem, the distribution of arsenic in Bangladesh, the health effects of arsenic poisoning, the difficulty in discouraging people from drinking the contaminated well water because the well water is clear, and the arsenic is an odorless, colorless, and tasteless poison, the need for an inexpensive and easy way to test the level of contamination in each well and the need for an inexpensive and easy way to remove a sufficient amount of arsenic from the water to make it safe to drink.
Following the video, the student engages in a process to solve the problem as a chemist would. At each step of exploring a solution to the arsenic contamination problem, the student is introduced to and practices one of the target stoichiometric concepts or skills. In the very first step, determining the level of arsenic contamination in a sample of well water, the student uses the Chemistry Virtual Lab to analyze a well water sample and compare the level of arsenic found in the sample to the acceptable levels set by the World Health Organization. The challenge the student confronts is that virtual lab experiment gives them the concentration of AsO2 in units of moles/liter. The WHO gives its safety standard as 10 micrograms as As per liter. The student must be able to convert the results from the lab to evaluate the concentration of elemental arsenic in units of micrograms per liter. In order to evaluate the safety of the water, the student must either understand the concept of the âmoleâ? and apply dimensional analysis, composition stoichiometry and solution stoichiometry.
At one point in the course, the student explores a potential solution to the arsenic contamination problem developed by a Bangladeshi chemist who uses powders made from local bricks to make filters that remove arsenic form the water. In the course, the student uses the virtual lab to reenact the work of the Bangladeshi chemist.
In the video below you see a screen capture of a student working through the virtual lab problem as the student reenacts the work of the Bangladeshi chemist. We give the student samples of various powders in the virtual lab and ask students to characterize the powders' ability to absorb arsenic. Students must determine the amount of arsenic that can be absorbed by 100g of absorbent, to an accuracy of 3 significant figures.
As you view the video of the student working through the problem, notice that students may ask for hints as they design the experiment and get immediate feedback on the results. If a student enters the incorrect answer 3 times, the lab gives the correct answer and prompts the student to try again. The software then generates a new problem with different values.
The virtual lab in the course provides opportunities for students to interact with the environment by exploring and manipulating objects, wrestling with questions and designing experiments. Students find such experimental design problems considerably more challenging than text book problems but struggling with these types of problems supports better learning outcomes.
Click here to enter the OLI Chemistry course at the beginning of the 3 page that covers the solution of the Bangladeshi chemist. You can try the tutors and the virtual lab activities. The last page of the sequence has the virutal lab activity that you just viewed above.
The OLI Modern Biology course is built around six key concepts that provide unifying explanations for how and why structures are formed and processes occur throughout the study of biology. In biology, as in many other complex disciplines, although the conceptual structure of knowledge in a domain is clear to experts, it is not to novices. The array of new ideas and unfamiliar terminology in introductory courses tends to overwhelm students into memorizing a set of isolated facts without understanding the underlying common principles (Chi, 2005; diSessa, 1993). One of the primary goals of the OLI biology course is for students to learn the many definitions of the concepts and also to recognize when they are operating in the process being studied. We introduce concepts in basic form and scaffold the extension of the concepts to other contexts giving students the opportunity to explicitly connect their knowledge and generalize their understanding.
The biological processes are complex and dynamic and not easily represented via text and static pictures. The biology team has developed a general-purpose simulation environment that links an underlying mathematical model to a computer animation so that the output of the mathematical model drives the details of the animation. Using this simulation environment, the observable properties of almost any biological process can be calculated in real time and then realized through the computer animation. With scientifically accurate models specified by the biologists on our team, the instructional activities we build within this simulation environment are scientifically authentic. The simulations allow us to present high-fidelity depictions of complex biological processes flexibly to students at different levels minimizing the likelihood of engendering student misconceptions.
Click the image to the left to run the simulation
This simulation is presented early in the course and is intended for introductory level students. Many objects can be displayed in the simulation window at a given point in time. If all objects (animation, equations and graphs) were presented at once students may not see how all these objects relate to each other, to the process of protein-ligand binding, or even what they represent. We start with the animation and direct the student to identify the various molecules depicted in the animation and we direct the student’s attention to the key aspects of the biological process (e.g., the bound versus free oxygen molecules). Research has shown that such techniques for focusing students’ attention are critical to helping students learn from animations and simulations because, although it is obvious to experts, students often do not know what to look for in a dynamic visualization. As we introduce each additional representation (the changing values in the equation and graphs), we pause the animation and direct the student’s attention to each of the representations and explicitly draw the connections between the representations and the biological process.
As the activity progresses, we gradually direct the student’s attention to more complex concepts and relationships. For more advanced students, these simple instructions are de-emphasized in favor of those that involve recognizing, applying, and synthesizing concepts in new situations.
While the behavior of the molecules in this animation is scientifically accurate, we intentionally chose a representation of a limited set of molecules even though in a normal biological system a large number of molecules would be involved. At this stage in a student’s learning, more detail and realism would not lead to more learning but rather distracts from the key features to which the student should be attending. Research has shown that instruction based on animations and pictures is more effective for novices when the graphics are simplified and provide explicit pointers to the important features (Biederman & Shiffrar, 1987; Clark & Mayer, 2003). We employ this principle in our animations and structure the materials so that students’ gradual learning is supported.
At the point in the simulation that we increase the number of ligand molecules, we encourage students to reflect on what is happening and predict how the system will react if we increase the number of ligand molecules. In the current version of the simulator, the student writes their prediction prior to running the simulation in the altered state. We log the students answer and the student receives feedback on their prediction through observing the simulation and reading the explanation.
Currently, we are incorporating mini-tutors (demonstrated in the engineering course page) into the simulation environment so that as students run the simulation, we can directly assess the knowledge they are supposed to be learning and provide more targeted, context-specific feedback to help students refine their understanding .
Click the image to the left to run the funcitonal groups activity
In the course, the students use the functional groups activity to explore the significant properties of the functional groups and to view text and 3-D representations of multiple example molecules within each group. Following this exploration, students engage in an interactive game that gives them the opportunity to apply what they have just investigated about functional groups.
An analysis of the data logs from an OLI biology study revealed that the more time students spent interacting with the functional groups activity, the better they performed on the corresponding quiz but not on other topics’ quizzes. The students’ time spent in OLI biology course was not significantly correlated with their other exam score, i.e., score they received on material they had learned without OLI-Biology materials. This suggests that the positive relationship we found in the former case was not simply a byproduct of better students showing both greater time on task and higher exam scores overall but rather that students’ time on task with the functional groups activity directly maps onto their learning of functional groups (Lovett 2006)
Using intelligent tutoring systems, virtual laboratories, simulations, and frequent opportunities for assessment and feedback, we build courses that are intended to enact instruction - or, more precisely, to enact the kind of dynamic, flexible, and responsive instruction that fosters deep learning.
As we deliver the instruction, we use technology to collect real time interaction level data of all student use. We use this data to create four positive feedback loops: feedback to students, feedback to course designers, feedback to learning science researchers and feedback to instructors.
Our current area of research is investigating how best to present this information to instructors in real time so that they can use the feedback to dynamically adapt their instruction.
At the beginning of the article, we asked you to think about how you might participate - some ways include:
The OLI project receives intellectual and financial support from:
The William and Flora Hewlett Foundation.Carnegie Mellon University through the Office of Technology for Education and the Eberly Center for Teaching Excellence
The author would also like to thank Judy Brooks for the graphic design work on this article; Joel Smith, her co-PI on the Open Learning Initiative, and the many faculty and staff who worked on the OLI course and evaluation projects described in this article:
Diana Bajzek (Biology), Aaron Bauer (HCI), Judy Brooks (Graphic Design), William Brown (Biology), Joe Bunick (Chemistry & Software Development), Jim Burnette (Biology), Jordi Cuadros (Chemistry), Anna Dollar (Engineering Statics), Mathew Easterday (Causal Reasoning), Renee Fisher (Engineering Statics & Course Design), Bill Jerome (Software Development), Michael Karabinos (Chemistry), Kenneth Koedinger (HCI, Cog Sci), Gaea Leinhardt (Evaluation), Marsha Lovett (Cog Sci, Evaluation), Ornella Pagliano (Biology simulations) John Rinderle (Software Development), Gordon Rule (Biology), Ross Strader (Engineering Statics & Software Development), Richard Scheines (Causal Reasoning), Paul Steif (Engineering Statics), David Yaron (Chemistry)
Anderson, J.R., Conrad, F.G. & Corbett, A.T. (1989), Skill acquisition and the lisp tutor. Cognitive Science, 13, 467-505
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245-281
Corbett, A. T. & Anderson, J. R. (2005). Proceedings of ACM CHI-2001: Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes.Dollár, A. & Steif, P.S. (2006). Learning modules for statics. International Journal of Engineering Education , Vol. 22, pp.381-392
Evans, K., Karabinos, M., Leinhardt, G., Yaron, D. (2006). Chemistry in the field and chemistry in the classroom: A cognitive disconnect? Journal of Chemical Education, 83/(4), 655
Evans, K., Leinhardt, G. (2006). Learning stoichiometry: Contrasting online and paper-pencil instruction. Technical Report to The Open Learning Initiative.
Lovett, M. (2006). Technical report on the fall 2005 deployment & assessment of the OLI-biology course.
National Research Council. (2004). How people learn: Brain, mind, experience, and school. Expanded Edition. Washington, DC: National Academy Press.
Steif, P.S. (2004). Proceedings from the 34th ASEE/IEEE Frontiers in Education Conference: An articulation of the concepts and skills which underlie engineering statics . Savannah, GA., October 20 – 23, 2004.
Steif, P.S. & Dollár, A. (2005). Reinventing the teaching of statics. International Journal of Engineering Education , Vol. 21, pp.723-729
Steif, P.S. Dollár, A. & Dantzler, J.A. (2005) Proceedings from the 35th ASEE/IEEE Frontiers in Education Conference: Results from a statics concept inventory and their relationship to other measures of performance in statics. Indianapolis, IN., October 19 – 21, 2005.
Steif, P.S. & Hansen, M. A. (2007) New practices for administering and analyzing the results of concept inventories. Journal of Engineering Education , Vol. 96, pp. 205-212.
