Evaluation frequently relies on the collection and analysis of data in number form. Approaches which involve this are termed ‘quantitative’.  They range in complexity from simple monitoring to the randomised control trial. Most arts and health project evaluations will benefit from using a mix of simple quantitative and qualitative methods.

Even the smallest scale project evaluation will involve some kind of monitoring of attendance or number of activities delivered.  Quantitative data are also often collected at the end of a project or activity using surveys or questionnaires. Reporting on these can be a good way to describe impacts on participants. However, they cannot tell us much about whether a project has actually had a measurable effect. To find out whether a change might have taken place, you will need to measure at both the beginning and end of a project.

Use of a measure developed by researchers and ‘validated’ or tested on similar participants in a similar context, for example, the Warwick-Edinburgh Mental Well-being Scale, can give more reliable data.  Validated scales also allows what you find for participants in your project to be compared with those of similar projects elsewhere.

It is always good for those delivering arts and health projects to clearly understand what the project costs and how this might compare to the cost of delivering similar work.  However, as well as demonstrating outcomes, project providers sometimes feel they need to show that they can deliver specific economic benefits for commissioners and other stakeholders. If this is the case for your project, it will be important to develop a collaborative relationship with the commissioner to ensure that you both understand what is required.

The ‘value’ of an arts and health project may be measured in various ways.  Formal approaches such as ‘cost benefit analysis’ value outcomes of a project in monetary terms but may not easily capture many important more subjective aspects of an arts project.  Economic evaluation that seems better suited to the context might seek to assess the benefits of an intervention in terms of Quality of Life Years or QALYs.  Another commonly-used tool in arts and health is ‘Social Return on Investment’ (SROI). SROI seeks to establish what the cost and impact to society would have been if an intervention had not taken place.

Economic evaluation of any kind requires significant skills and resources to conduct. Results will only be reliable if a project has first established a clear understanding of why the work it does has the effect it does – a theory of change. Further information about a variety of approaches to measuring value can be found by downloading Economic Evaluation.

Qualitative evaluation approaches focus on exploring the experiences, perspectives and opinions of those involved. They can help us understand what an arts activity or process ‘means’ to participants. They can be illuminating about other subjective elements such as project delivery. These approaches frequently rely on interviews and focus groups to collect data which is then analysed to identify themes and patterns within participants’ words.

It is important to make a distinction between ‘anecdote’ and qualitative evaluation. Personal testimonial or case studies are often used in advocacy by arts and health organisations but these ‘anecdotes’ are often chosen because they tell a good story, usually positive. This process lacks credibility, and is unlikely to be helpful in developing a project in the longer term. In contrast, balanced reporting of qualitative data, carefully collected from a sample of participants, can produce rich, detailed evidence and stories. This can inform advocacy and give really meaningful information to support improvement of a project.

Arts and health evaluations may benefit from using a mix of qualitative and quantitative methods to both identify specific outcomes for participants and explore their subjective experience.

 

An evaluator demonstrates ‘creativity’ every time they view an evaluation problem from a fresh perspective, or devises an evaluation approach tailored to a particular context.   While creativity is very definitely not just the province of artists or the arts, there is a growing interest in the use of evaluation methods that use film and visual arts, poetry and creative writing, music, drama and performing arts.

Arts-based methods can be particularly powerful in uncovering hidden perspectives and in empowering participants.  They may also be less intrusive than more clinically-based evaluation tools as they can be inspired by and modeled on the intervention itself. They also involve a number of challenges. For example, the results you get from them (pictures, performances or poems for example) are by nature difficult to interpret and you may require technical skills that are not a part of standard evaluation.

It is particularly important to make sure that any creative evaluation approach you use ‘fits’ the project and the people who are taking part in the evaluation.

There is power in a good story.  You may have used case study stories for many reasons, including when advocating for a project.  Personal testimonial from participants who have had a positive experience can be very persuasive. However, this is not what is meant by ‘case study’ in evaluation. For an evaluator, the quality of a case study hinges on the methods used to collect and analyse the data and the writing or presentation of the story.  The evaluator will select case studies that are relevant to the outcomes or issues the evaluation seeks to explore. You can have a case study of a single individual, groups of individuals, an object, even a place or a situation.  A range of different kinds of data may be pieced together in a case study to tell a story relevant to the questions that inform the evaluation.

It is important to remember that a case study story is particular to an individual case; because one participant experienced something during a project, this does not necessarily mean that all participants did, or that any future participants will.  This is why it is vital to select your cases carefully, so that they can genuinely help to illuminate the particular issues you are investigating.

When assessing the impacts of arts and health projects on those who take part, some conventional evaluation approaches may be lacking if they do not attempt to understand the meaning of arts participation for those involved. Participatory Action Research (PAR) is an approach that places the participant at the centre, as the person who knows best what has happened and what this has meant in terms of their own life.  It can very effective in empowering and engaging participants in an evaluation.  It is undertaken in a reflective cycle that can be deeply embedded within the process of delivering a project itself.