Systemic Evaluation Methodology for Technology Supported Learning

Developments in information and communication technologies have opened up new possibilities for organising and supporting learning activity. Appropriate evaluation methodology is required to help practitioners understand how and why their implemented learning designs work, so that they can improve their practice. It is argued that progress in learning about good practice in this domain has been slow because of inadequate methodology for inquiry into the complexity of organised learning activity. Methodology is needed to develop a better understanding of the social factors that influence learning and its outcomes, and the role of technology in this process.
This research has investigated the use of systems thinking in evaluation methodology. It is argued that using systems thinking to underpin evaluation practice provides a theoretically robust evaluation methodology. The research principally draws from critical systems theory to rationalise an approach to integrating ideas from different systems models, frameworks and methodologies, in particular those from social systems theory (Churchman 1971), Soft Systems Methodology (Checkland 1981), cybernetic systems from a human-centred perspective (Beer 1972, 1979, 1982, 1985) and Critical Systems Heuristics (Ulrich 1983). The central contribution is evaluation methodology that has been found to work in practice and has been rigorously tested through an action research approach across multiple cases. It is argued that the methodology helps to further clarify the complexity associated with evaluating organised human activity, for example associated with the roles, relationships and activity of multiple stakeholders with an interest in the activity being evaluated and the evaluation activity, and the relationship between strategies at different levels of organisation such as projects, programs, and the wider society in which they are situated. This leads to improvement in the relevancy of information generated by evaluation for answering different questions of interest to these multiple stakeholders.