Agile Methodology in a University Setting: Thoughts from a Chief Data Officer
By Dr Nick Creagh, Chief Data Officer, University of East London.
How do you go about implementing an approach born in the West Coast of America in 2000 in some of the oldest institutions in the Western World?
British Universities are old. Very old. The oldest (Oxford) is at least nine hundred years old, whilst many have several centuries under their belt. Having only a single century of history marks an institution as a newcomer in this sector. Indeed, a recent independent review of a sector-wide data program noted that:
Institutions are autonomous bodies, each with their own processes, systems and structures.
Universities are renowned for their rigorous academic standards and gated processes. This is essential for ensuring trustworthy research and graduates, which is integral to academic rigour and independence. And that’s a good thing.
At the University of East London (UEL), we’re proud to be at the forefront of innovation in higher education, leveraging technology to serve our communities and create real-world impact. This commitment fosters a culture that embraces change, making us an ideal environment for exploring agile methodologies.
The Agile 'Fail Fast' Approach
Starting life in a ski resort in Utah, Agile methodology is a term for an approach or set of approaches to developing software. Emphasising flexibility, rapid iteration, and continuous feedback, it allows teams to quickly adapt to changes, make incremental improvements, and learn from failures. The 'fail fast' approach is particularly advantageous in developing data products—such as dashboards and predictive models—where incremental gains and close relationships with users are vital, where the ability to quickly test hypotheses, integrate new data sources, and refine algorithms is crucial.
Agile encourages a collaborative environment where cross-functional teams work together to deliver small, functional pieces of a project in short cycles. This iterative process enables teams to identify and address issues early, reducing the risk of major failures and ensuring that the final product is more aligned with user needs and expectations.
Clashing Cultures: Gated vs. Agile
The fundamental differences between the gated approach and Agile methodology can create significant challenges in a university setting:
Speed vs. Rigour
The agile approach prioritizes speed and adaptability, often at the expense of thorough documentation and validation. In contrast, the academic gated approach values meticulous planning, extensive documentation, and rigorous peer review. This clash can lead to resistance from academic staff who are accustomed to the traditional methods of project management.
Flexibility vs. Structure
Agile methodology thrives on flexibility and the ability to pivot quickly based on feedback and new information. However, the structured nature of the gated approach can make it difficult to implement such changes without significant delays and additional approvals.
Iterative Development vs. Linear Progression
Agile's iterative development process involves frequent testing and refinement, which can disrupt the linear progression and comprehensive validation typically found in academic environments. This difference in approach can hinder collaboration and create friction between teams.
Overcoming the Challenges
Successfully integrating Agile methodology in a university setting requires a careful balance between maintaining academic rigour and embracing the flexibility of agile. Here are some strategies to help navigate this transition:
Change the Language: SCRUMs, Sprints, Minimum Viable Products. All of these terms cause confusion. I was asked by one customer representative if I was a rugby player, because I kept talking about scrums. Ok, so I was, many years ago, but I took the point. So now we talk about regular check-ins, show-and-tells and incremental gains. That keeps everyone with us.
Employ Hybrid Approaches: We’ve run some projects using a hybrid approach that combines elements of both agile and gated methodologies. We kept the stand-ups but also provided board meetings and progress reports. This can help maintain the necessary rigour while allowing for more flexibility and rapid iteration.
Implement Incrementally: Start small. Basically, use agile methods to roll out agile practices by implementing agile practices in smaller, less critical projects to demonstrate their value and build confidence among stakeholders.
Foster Collaboration: Encourage collaboration between agile teams and academic staff to build trust and create a shared understanding of project goals and methodologies.
Continuous Improvement: Regularly review and refine the implementation of agile practices to ensure they are meeting the needs of the university environment and delivering the desired outcomes.
Conclusion
While integrating Agile methodology in a university setting presents challenges, it also offers significant opportunities for innovation and efficiency in developing data lakes and data products. At the University of East London, we recognize that adaptability and forward-thinking are essential in today's fast-paced academic landscape. By finding a balance between academic rigour and agile flexibility, we can harness the strengths of both approaches to enhance our research capabilities and project outcomes.
In essence, our adaptable approach channels the innovative spirit of the forward-thinking individuals within our university. By treating data product developments as we would traditional research projects, we can leverage our storied history while embracing modern methodologies. Through collaboration and continuous improvement, UEL remains committed to setting a benchmark in higher education innovation.
