McMaster University is spending thousands of dollars on critical reflection

Technology has changed education. From online resources to course management, post-secondary education has been reshaped by the internet and computers. Among the many affected areas is critical reflection. McMaster University is spending hundreds of thousands of dollars to try to address this.

In 2013, McMaster University created the Learning Portfolio. According to a 2013 Learning Portfolio Working Group report, the project aimed to “enhance the experience of undergraduate and graduate students at McMaster” by creating an online resource where students could track both academic and extracurricular experiences. Students would create online portfolios for themselves, documenting learning goals and reflecting on their experiences.

What has ensued in the four years since its creation are a series of changes and, more recently, a struggle between the university and the McMaster Students Union. The MSU has asked for the LP project to be discontinued, but the university continues to make changes to try to push the LP as an important tool for students.

How the LP came to be 

The LP idea is a “major… initiative arising from Forward With Integrity” which is a 2011 letter from Patrick Deane that outlines priorities for the university moving forward. The idea comes from a group called the Student Experience Task Force. A SETF document dated July 12, 2012 introduces the project as a way to capture the entire student learning experience, while encouraging mentorship and personal critical reflection. The LP is part of the Teaching and Learning branch of Forward With Integrity, along with the MacPherson Institute (formerly known as McMaster Institute for Innovation and Excellence in Teaching and Learning).

“For a lot of us, it means connecting all the dots in the learning process, taking learning as a holistic entity; what’s happening in the class, in courses, across courses, co-curricular activities and professional volunteerism,” said Zafar Syed, associate director, educational technology at the MacPherson Institute.

“It’s a way of capturing and connecting all the various inputs that you’re getting and making meaning out of that.”

The McMaster LP is a part of the larger higher education trend towards e-portfolios, an umbrella term referring to digital portfolios. E-portfolios have been a part of higher education academia since the early 2000s and education academics are optimistic about the tool’s ability to facilitate reflection and authentic learning experiences; ideas that are at the centre of McMaster’s goals for their own LP.

The LP launched in Sept. 2013. According to the press release, “[r]oughly 3,000 students, primarily in first year, [would] create portfolios as part of their course work.” The purpose of this was to introduce students to the tool, not to mandate its use, as research has shown that e-portfolios have to be student-driven to be successful. Syed says that many programs have introductory courses that use the LP.

Mixed Results

In the past four years, the project has used a number of different tactics to encourage student engagement with the tool. After two years of the LP being hosted by Desire2Learn, the same company that manages Avenue to Learn, the university adopted a different e-portfolio software: PebblePad.

“A number of people who started using [the Desire2Learn tool] had found that challenging. It wasn’t doing the kinds of things they were wanting to do with this process,” said Syed.

PebblePad is not without its own challenges. The Student Success Centre provides links to four sample learning portfolios. Within these samples, there are pictures that are cut off and pixelated, text that overlaps and garish backgrounds colours. The samples look like blogs rather than an academic tool.

This sample portfolio was made in 2014, using the old software. It is still on the Student Success Centre's website as an example.

When asked about what measurables MacPherson uses to determine the success or effectiveness of the project, Syed said the nature of the LP makes that difficult.

They can track the number of users and number of uploads, but they do not know how much of the content is related to course requirements. There is no way for the university to track if students are using the LP after an introductory course.

“Some of that information is probably difficult to determine because the portfolio belongs to the individual. We can’t go in there and ask them ‘oh you created this thing, what it is for?’” said Syed.

Despite some hiccups, Syed is positive about the LP so far.

“If you’re going to measure it as have we achieved the goals that we put in front of us, I think, in some cases I would say yes. If the goal was to get more people exposed to and using this process, I think there’s been quite a bit of success.”

Good idea, poor execution 

MSU president Justin Monaco-Barnes paints a much grimmer picture.

Each year, the MSU makes recommendations to the university on funding. Monaco-Barnes says the MSU is recommending that the university cease to operate the LP, citing consultation with students and professors about the effectiveness of the program.

“There are profs who have told us that not only has it not improved the experience but it’s made the experience worse for students in the classes. It’s hindering some students’ learning experiences. These are signs to me that something needs to be addressed,” Monaco-Barnes said.

Conversations about the LP have been taking place throughout the past few years. A committee meets to discuss the project and how to move forward and improve the project. Monaco-Barnes and vice-president (Education) Blake Oliver sit on this committee, representing students, along with other university staff. Their experience has not been positive.

“Some of the feedback we provide doesn’t seem to fully translate into next steps, which can be frustrating at times for us, considering this program is meant for the students that we directly represent,” he said.

The president says the MSU agrees with the importance of the different issues that the LP tries to address, like critical reflection and mentorship. He believes that the LP lacks specific direction.

“There needs to be a target. Is it leadership, is it mentorship, is it reflection? The way it is now, it’s so convoluted that everyone has a different perception of what it is, it’s hard to make forward progression because everyone has this unique thing in their mind that we can’t agree upon,” said Monaco-Barnes.

At the heart of the issue for Monaco-Barnes is cost, and he has not received a clear answer on what the university is spending.

Evaluating the costs and effectiveness

Through a Freedom of Information and Protection and Privacy Act request, the Silhouette was able to obtain a variety of costs since the beginning of the project. To date, the university has spent $710,789.93 on the project. They are also currently paying for software they do not use, as they signed a contract with Desire2Learn and abandoned it in favour of PebblePad.

Determining a true cost of the learning portfolio project is difficult. The LP is housed under the MacPherson Institute, and according to the university, employees are not specifically tasked with working with the e-portfolio, therefore the labour costs are not available.

The university has developed pilot courses in an attempt to engage students with the LP. Two of these courses were Social Science and Humanities 2LP3, run in 2013 and 2015 respectively. The costs to develop and run these courses are unknown, and both courses ran for only one term.

McMaster has spent $135,000 over three years to fund “learning portfolio fellows”; professors who “developed research proposals that study the utility and effectiveness of the Learning Portfolio” according to a press release on Forward With Integrity’s website.

It is equally difficult to evaluate the project’s effectiveness because of the lack of both statistics available and goals for the LP. One original target has changed.

“There was talk initially that every student should have this and that it would be across the whole system. That’s a very high-level goal that requires resources and set up. I don’t think we’re there,” Syed said.

MacPherson and the MSU both rely on anecdotal evidence from students and professors to inform their stance on the project. It is unclear what the next steps for the project are, as one side says it can be effective because some students and professors have had positive experiences, while the other side claims the opposite.

A time for critical reflection

The current Board of Directors’s term is up, but the LP struggle will not end with their tenure. Monaco-Barnes says he will be discussing the LP with the incoming president, as addressing the project is a priority.

Syed says MacPherson will continue to work through “user-case examples” to find out what the next steps are.

But is ending the project a possibility? Given that the university’s belief in the Learning Portfolio stems from the university president’s vision letter, this seems unlikely. Abandoning the project without an alternative means the university would leave behind a stated goal of their president.

The university will continue to foot the bill for a project whose legitimacy has failed to be proven in four years of existence.