Our approach to measuring impact of digital inclusion

Defining short term outcomes:

To understand the difference that digital inclusion support makes in the short-term to people facing digital exclusion, we focus on three core aspects:

• Being digitally equal: defined as having access to appropriate digital connectivity (mobile data and / or broadband), and to an appropriate digital device, indicating equity of access to the internet.
• Being digitally able: defined as having skills and confidence to get online and do what you want or need to do online.
• Being digitally safe: defined as feeling able to keep yourself safe online when doing what you want or need to do online.

We take ‘short term’ to mean the weeks and months directly following digital inclusion support, typically a period of around one to three months after support.

Defining longer term outcomes:

We assume that digital inclusion support will also lead to several longer term outcomes, likely experienced between from about six months after receiving support. To measure these outcomes, we focus on three core aspects:

• Feeling happier: defined in terms of social connection, and feeling better able to connect with friends and / or family as a result of receiving support.  This assumes that better digital access and capability will mean people are better able to use online tools to keep in contact with others, resulting in feeling more socially connected and happier.
• Feeling healthier: defined in terms of feeling better able to use digital tools to access information about health and health services. This assumes that better digital access and capability will mean people have more opportunities to manage their health and seek appropriate healthcare, resulting in improved wellbeing.
• Feeling better off: defined in terms of i) feeling better able to use online tools for managing finances, paying bills and / or shopping, and ii) having better opportunities for employment, training and / or education. This assumes that better digital access and capability will mean people have greater opportunities to save money through using online services, and will have greater opportunities to increase their income as a result of increased skills and training.

To understand the difference that digital inclusion support makes to people’s lives, we collect both quantitative and qualitative data, via a range of approaches:

Survey of people who have received support: 

We have developed a short (10 question) questionnaire, designed to be simple and accessible for people who have received support to complete. The questionnaire asks:

• Basic demographic information (year of birth, ethnicity, first part of postcode)
• Type of support received (data, device, skills, other)
• Level of agreement (agree, disagree, not sure) with outcome statements (able, safe, happier, healthier, better off)
• Any other feedback (open text box).

We ask only three questions about demographic characteristics to minimise the burden on the person completing the questionnaire, and because we are aware of some people’s reluctance to share personal information. 

The questionnaire is distributed in two ways to people who have received support:

1. Via hubs: hubs give out paper versions of the questionnaire and / or share the link to the online version of the survey to the people they have supported. The data from the paper questionnaires are shared with Good Things via email or inputted directly into the online survey platform. 
2. Direct to people supported: a link to an online questionnaire is emailed to people who have received one or more of our services (Databank, Device Bank, Learn My Way) and who have given permission to us to contact them.

The questionnaire is shared with people typically between two and six weeks after receiving support, though this timing is flexible for hubs that are distributing the paper version in-house.

For measuring the short term outcome of ‘equal’, we use receipt of mobile data through the National Databank and / or of a device through the National Device Bank as a proxy indicator that people are digitally equal, and therefore we do not ask about this outcome through the survey.

Qualitative research:

To help us to understand in depth the barriers people have faced to being online, their experience of receiving digital inclusion support, and how the support has impacted on their lives, we regularly engage with people through interviews and group discussions. This work includes: 

• Regular research visits to hubs, to observe digital inclusion support delivery, conduct group discussions and / or interviews with people who have received support, and interviews with staff / volunteers (approximately one hub visit per month)
• Virtual / phone interviews with hub staff members (approximately two per month).

The table below summarises the information described above about how we measure impact, the data we collect for each outcome and the method we use to collect the data.

We have an accessibility description available for this table. 

Our approach for measuring the impact of digital inclusion support reflects our learning over the years about what does and does not work to engage the communities supported with digital inclusion in research and evaluation work. This learning includes:

• Simple questionnaire with few demographic questions: We know that many people who struggle to get online face difficulties with literacy and English as a second or other language. We also know that some people who receive digital inclusion support are reluctant to share too much personal information. To reflect this, we have evolved our questionnaire to be as short and simple as possible, asking for only minimal demographic information and no personal contact information.
• Short-term follow-up only: Response rates to surveys among people who receive digital inclusion support can be very low, and we know that some people may change contact information frequently. Also, given that we do not ask for personal contact information to assure anonymity, it is not possible for us to track who has completed a questionnaire. Therefore, to maximise impact data collection, we only use a post-hoc approach, rather than a baseline and follow-up approach, which is distributed between two and six weeks after receipt of support. We are exploring ways of measuring longer term impact through qualitative approaches.

• Multiple data collection methods: While online surveys are a cost-effective way to reach recipients of support directly, they are likely to be off-putting or challenging for many people who face difficulties being online. We identified that engaging hubs in the collection of data, through paper questionnaires, or supporting people to complete an online survey, is an effective, alternative method. We are also exploring ways to collect impact data through text messages, to increase engagement in evaluation. 
• Basic core questions that can be used flexibly: The core questions asked through the survey remain constant, to enable comparison of data over time. However, we are able to add in a few extra questions when needed to tailor the data to the needs of specific programmes, whilst retaining a consistent set of core impact data. 
• Evolving qualitative approaches: We have developed a range of approaches for engaging people to talk about their experiences of digital inclusion support, recognising that some people may not be confident speaking English and / or may be reluctant to speak to strangers in a research setting. We work with hubs to facilitate group discussions and interviews in familiar spaces, with interpretation when needed. We have trialled visual tools and mapping techniques to help people talk about their experiences around digital, and the impact of receiving support. We are also piloting a longitudinal approach, following up with a cohort of people on a regular basis over 10 months, to understand the longer term impact of receiving support.