A sociocultural approach to data visualisation evaluation: understanding the role of a large-scale tool-based approach to evaluation within the commercial production process.

This study investigated the potential of undertaking sociocultural evaluation of data visualisation within the context of commercially driven data visualisation production. This study takes the stance that current approaches to evaluating data visualisation do not take into consideration the context within which they are consumed, as well as the external factors that can impact upon this consumption. As such this study drew upon the work of Seeing Data, in particular the work of Kennedy et al., (2016) in utilising the approach of Stuart Hall (1973) in understanding the encoding and subsequent decoding process of consuming data visualisations to develop a sociocultural approach to evaluating data visualisations.
In doing so, this study utilised an interdisciplinary mixed methods approach, utilising both qualitative and quantitative data collection. In order to ascertain the existing forms of evaluation, and under which conditions they took place, a participant observation oriented ethnographic study of a commercial design studio took place over two visits. Drawing on the data collected and analysed from these visits, as well as the existing literature on the evaluation of data visualisation, a digital tool was developed through an iterative design process to capture sociocultural data from the consumers of data visualisation.
This tool was developed using a participatory design approach, involving user testing and the input of data visualisation practitioners through a co-design session. The tool was subsequently evaluated using the approach of tool criticism, utilising a critical tool-walkthrough method to interrogate the suitability of the tool to capture sociocultural data as well as its place within the production process within the commercial setting.
Findings from the commercial design studio revealed the complexities of undertaking sociocultural evaluation at scale within the constraints imposed by the resources available. These constraints resulted from the reliance on the existing conventions of data visualisation practice, and the availability of resources such as time, finances, as well as access to the end-consumers and data. Often these constraints arose out of the complex relationship with the client that had to be negotiated by the design studio and its practitioners. The tool received praise from the practitioners and proved successful in gathering data valuable to the production process. As such this revealed the potential for developing tools, and therein tool kits and novel approaches, that can be imbedded within the production process to navigate the constraints found within the commercial setting of data visualisation production.