Summarized highlights from Surfacing Values Through Rubric Development | Julian King

“Rubrics, introduced into the field of evaluation by E. Jane Davidson, are one way of specifying criteria and standards, with descriptors of what the evidence would look like at the intersection of each criterion and standard.”

“Criteria and standards help people to place value on things, by describing what aspects of value matter to people, and what good value looks like.”

“According to the General Logic of Evaluation, evaluative judgements flow from: • Evidence (independent observations of performance) • Criteria (defined aspects of value) and • Standards (defined levels of value).”

“Interpreting evidence through the prism of criteria and standards is called evaluative reasoning. It isn’t a method or tool – it’s a logic that underpins evaluation.”

“As evaluators, part of our job is to facilitate a process to surface the values of those who impact or are impacted by a program, such as its architects, funders, providers, and communities. Co-creating rubrics brings stakeholders to the table to define what ‘good’ looks like for a specific intervention and context.”

“Collaboratively developing rubrics surfaces stakeholders’ values, helps different stakeholders to experience and understand the diversity of perspectives involved, and guards against the evaluation only reflecting the values and assumptions of commissioners and evaluators.”

“Asking directly about ‘what aspects of performance should we focus on?’ can work OK with stakeholders who are used to thinking analytically (like policy analysts and researchers). But it’s usually helpful to be prepared with some conversation starters like ‘what would success look like to you?’, ‘what ways of working are important to help the program be successful?’ and so on.”

“In professional evaluation, involving value judgements by, with, or for people affected by policies and programs, conclusions are strengthened by showing our working (being transparent about how we got from the evidence to a judgment). Rubrics help to provide the needed transparency.”

“Rubrics pay back in clarity throughout the evaluation process by guiding decisions about evidence needs, method selection, tool design, organising evidence, making evaluative judgements, and structuring a report. At each step, this clarity enhances the efficiency and quality of the evaluation.”

“Co-developing rubrics supports utilisation-focused evaluation by helping stakeholders to gain a deeper understanding and experience of evaluation. This makes it more likely that the evaluation processes and products will be understood, endorsed and used.”

“Rubrics define criteria and standards – not indicators. While indicators generally specify measurable attributes, rubrics describe broader aspects of performance, success, quality and value, in line with their intended purpose of helping people to make evaluative judgements.”

“It’s important to set aside measurement decisions during rubric development. Once the rubric is defined, you can focus on identifying the right mix of evidence to support the evaluation.”