The Business Building at Penn State
A team at Penn State is taking a deep dive into some of its graduate courses by studying the data that show how students learn and behave. They hope to use their findings to redesign courses in ways that will help students be more successful online learners.
The team of instructional designers, faculty and teaching support specialists has been analyzing these learning analytics from two years’ worth of courses in the Penn State Online MBA program, which is offered through Penn State World Campus and led by the Smeal College of Business.
The data consist of an expansive variety of activities, from the links students click to the depth and quality of the posts they write on discussion boards. The data will be used to guide an innovative approach to course design, aligning the program’s main learning goals to the ways the team wants students to act and engage in the courses.
The work in the online master of business administration (MBA) courses is the first of its kind at Penn State and among the first in online graduate higher education in the country.
“The data represent a study of student behaviors based on the unique graduate level teaching and learning experience and tell us a story about what is happening in our courses,” said Janet Duck, the online MBA faculty chair who has championed the project.
“We can use this story, through deliberate course design, to inspire student behavior that solidifies course competencies and provides an engaged experience for our professional students.”
This screenshot from the MBA learning analytics shows an analysis of the data.
A never-before-seen look into courses
With almost 1,000 students, the Penn State Online MBA is the highest-enrolling graduate program offered through Penn State World Campus.
The program is led by the Smeal College and includes a consortium of faculty from Penn State Behrend, Penn State Harrisburg and Penn State Great Valley School of Graduate Professional Studies. It has consistently ranked among the top-10 online MBA programs in the country by U.S. News and World Report.
For the past two years, the team has been collecting data from the 13 core online MBA courses in Canvas, the University’s learning management system, and Zoom. The data are being visualized on a dashboard using charts, tables and graphs.
Jessica Resig, the lead instructional designer for business programs offered through Penn State World Campus, said this approach lets the team see the students’ behaviors in a new way.
For instance, the data may show trends, such as if students are struggling with a concept as evidenced by low grades on the same assessment each semester. Or they could see whether students are choosing to watch videos that are included in their weekly lessons but are not required viewing.
“Those kinds of things can give us insights about where students are spending their time or where they're engaging with our material, to help us make decisions about if the content is meeting their needs,” Resig said. “This is a comprehensive snapshot that’s presented in way that we didn’t have before.”
Resig said students’ privacy is paramount in the data collection.
“Our goal is to understand the effectiveness of each course and the design decisions we're making, not to focus on individual course participants,” Resig said. “With that in mind, we aggregate all data used as part of the course analysis process and have implemented a robust set of ethical and procedural standards to ensure that student privacy and data security are strictly maintained.”
This screenshot shows data about how students engage with various pieces of content in the online MBA courses.
Re-envisioning course improvement
Designers hope the learning analytics will mean they can make ongoing improvements to the online MBA courses as needs arise instead of waiting until courses are scheduled for updates every few years.
Duck said that even though the team has just started to analyze the data, the learning analytics are already pointing to improvements and a more holistic understanding of teaching and learning.
For instance, the study of the depth of replies on discussion boards shows that when students write long discussion board posts, those posts tend to attract fewer replies. The data show which content pages students are visiting the most and places where student engagement tends to drop, Resig said.
“Are there ways that we can redirect them to that course content?” Resig said. “Can we re-envision the way the content is being presented to make it more valuable for them?”
Duck said data are being used to drive behaviors in other aspects of everyday life, and she advocates for applying the approach in higher education.
“We know that our professional graduate students want to be engaged in their learning experience,” Duck said. “We know that they want to be able to immediately apply what they learned, and they want to understand the whole picture, both in terms of themselves and business and industry. We want engaged, applied and integrated learning, and data can help us get there.”
This screenshot shows the data about discussion posts by students in the online MBA program.
Spreading the word outside of Penn State
Resig and Duck are sharing their knowledge with their peers in online learning across the country, and Duck said she receives inquiries about these efforts.
Resig and Chris Millet, the senior director of learning design for Penn State World Campus, led a virtual presentation in September on learning analytics during the Online Learning Consortium’s OLC Accelerate conference, one of the largest meetings of online education practitioners. Resig and Millet have authored two book chapters in scholarly publications about this use of learning analytics.
Duck and Resig will lead a virtual presentation on the topic in November during the Quality Matters Connect LX conference.
Resig said another group of online MBA courses will be revised next fall, and the last group in the program will be revised in the fall of 2023. Afterward, the team expects to use the dashboard to spot trends or warnings in the learning analytics and make continuous improvements in the course content.
That is why the data-backed approach to learning design has the potential to be so transformational.
“It lets us be smarter and more intentional about our design decisions moving forward,” Resig said.
Visit the Penn State World Campus website for more information about learning online.
