Adapting to Agile EdTech Development: A Case Study of University of Pretoria’s e-Education Team

The impact of agile development, particularly prevalent in the realm of cloud-based EdTech systems, requires instructional designers (IDs) to exhibit a higher level of commitment and adaptability. These professionals, often regarded as agents of change within university settings (Anderson et al, 2019; Pollard & Kumar, 2022), are entrusted with the implementation and management of Educational Technologies (EdTech). In the past, conventional change management strategies guided EdTech system implementations, adhering to structured timelines, with users eventually reaching a productivity plateau.

Agile development necesitates IDs to navigate the gap between their institution’s slower adaptation and the rapidly evolving EdTech landscape. This demands a high level of adaptability as they integrate fast-evolving teaching tools into the institution’s unhurried educational ecosystem. In addition to staying abreast of routine system upgrades, they must also translate these changes within the unique context of their respective universities and effectively communicate these changes to the system’s users in a constructive manner.

In this article, the author delves into the strategies adopted by the e-education team at the University of Pretoria to effectively manage the dynamic landscape of agile EdTech system development. Additionally, the IDs perspectives on the regular updates is explored within this context.

Background

Integrating technology into education has been a cornerstone of the University of Pretoria’s (UP) strategic vision for equipping students with the skills they need for the future, since the 1990s. Situated in a developing country, UP, as a prominent residential and research-focused institution, made a deliberate choice to deploy “commercial off-the-shelf” Educational Technology solutions. UP’s investment centered on two core EdTech systems, to support their hybrid teaching approach: a robust Learning Management System (LMS) and a dedicated e-assessment platform. Initially, on-premise, these systems received regular technical updates, with occasional functional adjustments.

A team of experienced Instructional Designers (IDs) within the Department for Education Innovation is responsible for assessing and implementing UP’s EdTech systems. Two subteams in this group serve as internal change agents, empowering lecturers to effectively utilize the LMS and e-assessment systems for educational purposes.

The team who supports the e-assessment system, is known as the computer-based testing (CBT) team. They have successfully overseen four implementations, with their continuous support leading to a peak usage of 138,983 assessments in 2019. The LMS team, which has managed three implementations since 1998, transitioned to a managed hosting environment in 2015 to ensure system stability and scalability. Thanks to the stable LMS, ongoing training, and dedicated support, 95% of all modules were accessible through the system in 2020.

After a decade of stability within both systems, the status quo among the IDs was disrupted by two significant announcements. Firstly, the university declared its intention to transition to the next version of the LMS in 2019. Shortly thereafter, the vendor of the e-assessment system announced its “end-of-life” in 2020, which required a different product be implemented as the next version did not comply with the requirements of UP.

The new platforms that were chosen steered the institution toward cloud-based environments offered by companies employing agile development methodologies for their platforms. Each team was thus confronted with the introduction of not just one but two foreign elements. Firstly, they had to acquaint themselves with the new system, oversee its implementation, and assist lecturers in navigating the transition. Secondly, both teams had to grapple with the ongoing changes stemming from the agile development of these systems.

The two teams exhibited varying responses to the changes introduced. To gain insights into these responses, the author conducted two surveys, one of which aimed to uncover the prevalence of observed emotions and reactions that might suggest resistance to the changes. The other survey sought to understand the teams’ perspectives and emotions concerning the impact of the agile developments on the systems. Notably, all seven members of the LMS team participated in both surveys, while feedback from six CBT team members was gathered in response to the resistance survey, and five members responding to the survey focused on the updates.

Main resistance factors influencing system implementation

The CBT Team

The chosen platform addressed 90% of the UP requirements and the team continued with the implementation despite the COVID-19 lockdowns to complete the deployment by the end of 2021.

The factor that caused the most resistance within the team, was the greater autonomy lecturers would have over the full process of setting assessments within the system. Team members were stressed and some expressed fear because they were concerned that lecturers might incorrectly configure their assessments. Lecturer autonomy also had the potential to reduce the team’s workload, sparking varied emotions within the group. Some felt relieved, while others worried about their future roles, including job security.

Although the team indicated that they experienced passive resistance on an individual level they recognized the urgency to implement the new system to ensure continued service delivery of assessments. Details regarding this system implementation are described in this case study.

The LMS Team

The initial evaluations of the new LMS exposed a significant mismatch between its low maturity level and the institution’s actual requirements. This discovery triggered a notable conflict and a surge of overt resistance among the LMS team. The group openly and collectively resisted the implementation of the new LMS, resulting in a delay in the rollout. As the COVID-19 pandemic unfolded, lecturers’ extensive utilization of the original LMS led to a substantial growth in their proficiency within the system which further intensified the team’s concerns regarding the impending new product implementation.

To effectively overcome this resistance and promote the adoption of the new LMS, a series of initiatives were implemented:

Vendor meetings were held to discuss the unique context of the University of Pretoria and address identified product limitations.
Dedicated work sessions were conducted to enhance the IDs’ system knowledge. Drawing on their understanding of UP’s needs, these sessions facilitated collaborative exploration and documentation of specific workflows within the product. The resulting documents later served as quick reference guides for developing the help site and integrating activities into lecturers’ training courses. These sessions not only highlighted the “wins” in the new LMS, altering the team’s perceptions, but also revealed system and help site limitations.
These limitations revealed during the process exploration sessions were documented as ideas for improvement in the vendor’s idea exchange platform, fostering a dynamic partnership.
Collaborative development of lecturer training courses, including a quality assurance process for each course, further deepened the team’s understanding of the new system and its application within the teaching context.

Impact of Agile Development

The agile development approach of the new systems requires new processes to help both teams to internalize changes and effectively communicate them to end-users. To stay updated, the system administrators and project managers actively monitor product roadmaps and align their planning with product release timelines. Monthly system release notes are used to assess the significance of changes to UP and their potential impact on IDs’ and lecturers’ existing skills. When a change is considered complex, the team conducts testing in the staging environments after its release to evaluate the need to update training material and help sites and communicate changes to existing users.

Changes are communicated to users through three channels, namely help sites, emails and training. The IDs responsible for the two helpsites update them to allow users 24/7 access to the information. The help sites also support the other teams in their communication efforts. The (newly established) communication team craft the necessary messages and share them through monthly emails and LMS announcements. The training teams assess the changes and integrate them into the relevant courses to train new users.

Results from the survey provide valuable insights into how the IDs perceive the regular system updates. Firstly, a significant majority view updates as a constructive means to enhance the system’s performance and overall quality. Notably, 100% of respondents perceive these updates as an exciting opportunity to address customer requirements and further improve the technical aspects of the systems. Moreover, nearly 85% of respondents recognize these updates as a demonstration of the company’s commitment to fulfilling customer needs. However, it is important to acknowledge that 42% of the respondents find these updates to be somewhat challenging in their work lives.

Members of both teams disclosed a predominantly positive sentiment towards the routine updates of the two systems. It is noteworthy that most team members consistently express curiosity when confronted with a new release, accompanied by a prevailing sense of enthusiasm stemming from the anticipation of fresh functions and possibilities. This optimism aligns with their belief that the vendor is dedicated to enhancing the system, evoking feelings of gratitude for the product’s continuous improvement. Furthermore, the team underscores the importance of feeling respected by the vendor in their interactions

On the flip side, both teams also recorded uncomfortable emotions. Half of the respondents experience frustration or feel rushed by the swift pace of new changes, often leading to stress. Two IDs recorded that they almost always feel skepticism about the touted improvements. Two members of the CBT-team occasionally experience bouts of discouragement, while another occasionally feels disrespected.

The following quotes summarise the mixed emotions, and opinions about the updates of the systems:

“Good to know they are improving the system but not always well tested before the release.” (CBT S2, 1)

“…While this is somewhat easy for us as IDs to adapt, it is never the same with some academics, they sometimes feel they can’t catch up or have no time to figure out what has changed but we are always available to offer support. If updates can take place twice or thrice in a year, that might be a relief !!!” (CBT S2,3)

“We appreciate the transparency and timely communication from the system administrators, which allows for better preparedness when updates are scheduled. Additionally, the smooth transition and minimal disruptions experienced following these updates have garnered satisfaction among the user base. Overall, I believe the updates highlights the effectiveness of the communication and the successful implementation of updates in the assessment system” (CBT S2,5)

“Updates and improvements key to the uptake, quality of use of the system. Without it, we might see people abandoning the system.” (LMS S2,1)

“Our approach of testing and working around issues brings many ideas for updates.” (LMS S2,3)

“It is a pain to redo my work but I understand why it is important and has to happen.” (LMS S2,2)

“I always tell myself to think about the updates in terms of who they are meant for and to always remember the bigger picture and that I am here to be an agent of change for educational advancement and these updates are for the better.” (LMS S2,2)

Conclusion

The strategies employed to address resistance within the LMS team during the implementation of the new system yielded several noteworthy benefits. They provided UP with an opportunity to collaborate with the vendor, resulting in system improvements tailored to the requirements of institutions dealing with large class sizes. Internally, these efforts not only enhanced the team’s understanding of the system but also cemented practices that continue to prove valuable during the assessment of subsequent updates.

While the incorporation of this additional agile change management process has undeniably increased the workload for the e-education team, it has also provided a precious opportunity. This approach enables us to actively engage with EdTech companies, providing continuous feedback that shapes these products into platforms tailored to our vision for the future of teaching and learning at the University of Pretoria, and ultimately the rest of the world.

Resources

Anderson, M. C., Love, L. M., & Haggar, F. L. (2019). Looking Beyond the Physician Educator: The Evolving Roles of Instructional Designers in Medical Education. Medical Science Educator, 29(2), 507–513. https://doi.org/10.1007/s40670-019-00720-6

Pollard, R., & Kumar, S. (2022). Instructional Designers in Higher Education: Roles, Challenges, and Supports. The Journal of Applied Instructional Design, 11(1). https://doi.org/10.51869/111/rp

Written for OEB Global 2023 by Detken Scheepers.