Salzburg Global: What's Measured, Matters

Published date

Feb 18, 2022

Written by

Peace & Justice Update

How capturing better data and measuring social trust could help build better criminal justice systems—for the people within it and their communities

Published date

Feb 18, 2022

Written by

Martin Silva Rey

Men wearing yellow vests sitting together

READI is one of the largest anti-violence programs in the US (Photo credit, READI Chicago Heartland Alliance)

How capturing better data and measuring social trust could help build better criminal justice systems—for the people within it and their communities

Eddie Bocanegra was 18 when he was sent to jail. Born and raised in Chicago’s Little Village—a community heavily impacted by gun violence—the 45-year-old social worker has experienced violence from both sides.

“I myself have witnessed a number of losses, people being shot in front of me, friends being killed, and then unfortunately being part of the problem myself,” Bocanegra reflects.

At the age of 14, Bocanegra joined a local street gang. He was convicted four years later and finaly got out when he was 32, after serving 14 years and 3 months.

In the 12 years since, Bocanegra has completed various degrees in social work and now “I leverage my own lived experience to create solutions to these issues of gun violence in criminal justice reform” by running one of the largest anti-violence programs in the United States: READI Chicago.

In this role, he oversees the management and implementation of the evidence-based and trauma-informed program to reduce gun violence and promote safety and opportunity in the city.

“Living where I grew up, being in a gang, going to prison, coupled with my education and career, is what also gives me a very unique perspective to the field,” says Bocanegra, also a proud father of seven.

“I wanted people who look like me to be able to speak on these issues... and I will tell you that too often, we haven’t really looked at those who really drive gun violence really carefully because it’s very difficult for nonprofits in many cases to have access to them.”

Some of the people the program targets do not even want to be found, he explains.

“We use police data, hospital data, and a number of data points to help us identify individuals who are at the highest risk of gun involvement,” which according to Bocanegra, leads to “99% accurate” conclusions.

“We are targeting the right population, and that’s what makes us very different.”

“We use police data, hospital data, and a number of data points to help us identify individuals who are at the highest risk of gun involvement... We are targeting the right population, and that’s what makes us very different.”

– Eddie Bocanegra

Launched in 2017 in the wake of a spike in Chicago’s violent crime rates, READI connects its participants with mental health support, paid transitional jobs, and professional development, working in partnership with community-based organizations and local jails.

Early analyses show “all positive results,” Bocanegra stresses.

“We have 30% less of our men being shot or killed… and we are also seeing 80% less of our men arrested for murder compared to the control group. That’s a huge number!”

He points out the value of listening to the affected communities themselves, but “people with lived experience are overlooked.”

Peter Dixon, a leading expert and lecturer in coexistence and conflict resolution, agrees with Bocanegra.

“The communities and people impacted by reforms or by policies do not see the world in the same way that those making the policies see them. That’s not necessarily a bad thing, but to truly understand the success or failure of a policy, you really have to engage in a conscious effort to step into the shoes of those who are actually impacted by the reform,” he explains.

“Convincing policymakers that they should listen to the local everyday lived experiences of communities is quite a challenge. The key here is starting with this rich-detailed locally relevant qualitative information, but translate it into a kind of technocratic language of quantification and measurability—and that’s the language that policymakers tend to react to.”

“Convincing policymakers that they should listen to the local everyday lived experiences of communities is quite a challenge. The key here is starting with this rich-detailed locally relevant qualitative information, but translate it into a kind of technocratic language of quantification and measurability—and that’s the language that policymakers tend to react to.

– Peter Dixon

“Over-researched and underserved”

A sociologist by training, Dixon has worked in peacebuilding strategies and transitional justice. He started at the International Criminal Court in The Hague, Netherlands, and then moved on to post-conflict reconstruction in East Africa and Latin America, mostly in Colombia.

Now, he is working with the team at Everyday Peace Indicators, a network of researchers based mostly in the United States, to assess how communities across the city of Oakland, California, define safety and wellness in their everyday lives. Their goal is to use those indicators to assess policies related to criminal justice reform instead of others created by people external to the community.

“Once these policies are put in place and actually implemented in communities on the ground, what difference will they actually make in the everyday lived experience of safety and wellness as defined by these communities themselves?” Dixon asks himself.

“How do you determine what to measure?” also asks Edward Mulvey, a professor of psychiatry emeritus and former chair of the United States Department of Justice Office of Justice Programs (OJP) Science Advisory Board.

“Because what are potentially favorable outcomes for some people might be unfavorable to others or less important, or a lower priority.”

Mulvey has directed numerous research studies on the link between mental illness and violence, the development of juveniles in the justice system, and the impact of sanctions and interventions for young people who have committed serious crimes.

“There’s a whole sort of critical initial stage of understanding the phenomenon from the perception of the people who are affected by it and the perception of the people who are involved in the system administrating it,” he explains.

To achieve that, Dixon says, the first step is defining the community. That can be done according to various criteria, ranging from identity, or race or ethnicity to socio-economic status and geography, depending on the context. The second step then is bringing people together in a way that they can talk about their experiences to create indicators that are representative of their everyday lives.

“That’s the easy part: bringing folks together for a discussion that lasts a couple of hours where the question is, ‘What does safety mean in this community, what does safety look like, what does the lack of safety look like when you think about everyday life?’

“And that’s when you start getting things like hearing shots or being able to walk to the corner store at night, being able to sleep with my windows open, knowing the names of my neighbors, seeing children play in the park.”

For Dixon, who has worked with several international organizations engaged in peace, conflict and justice, “connections between seeing children playing in the park and police reform might not be so apparent to a policymaker” from a traditional perspective, but “for folks living in those communities, that connection is direct and very important.”

The hardest part comes next.

“There’s a ton of skepticism from community members when another round of surveys comes around, another round of focus groups is called… There’s a lot of mistrust... They don’t actually feel it’s listened to.”

– Peter Dixon

“Researchers but also policymakers sort of think of the measurement stage just like a technical process… but often the perspective of community members themselves in these communities where research on criminal justice reform is happening is feeling over researched and underserved,” Dixon remarks.

“There’s a ton of skepticism from community members when another round of surveys comes around, another round of focus groups is called… There’s a lot of mistrust about whether it's worth people’s time to keep on coming out and talking about their lived experience or talking about their perspectives when they don’t actually feel it’s listened to.”

That is why Dixon and his team invest so much time at the community level—to build relationships with those involved and gain their trust.

“And once the data is collected, use that data in a way so that it’s not simply extracted, but also productive for the community members themselves.”

Not the whole picture

Dixon and Bocanegra, who like all the people featured in this article are part of Salzburg Global Seminar’s Global Innovations on Youth Violence, Safety and Justice initiative, are not the only experts challenging the traditional ways in which crime, violence, and the performance of the justice system are measured.

Research on criminal justice and justice reform brought Israeli and American academics, Hadar Dancig-Rosenberg and Joshua Kleinfeld together. Despite coming from different backgrounds—Kleinfeld is admittedly more “tough on crime” to Dancig Rosenberg’s “soft on crime” approach—they found a common standpoint to work together on innovative ways to measure how the justice system is doing its job.

“When you look at criminal justice systems today and how policymakers structure the system and run it, you see that usually people use the same bunch of metrics. The most popular are crime rates, arrest rates, conviction rates, recidivism rates, cost, and while each one of those of course has its own rationale, and it does a partial job, none of them is dominant or widely used,” says Dancig-Rosenberg, a criminal law professor also involved in non-punitive, rehabilitative, and community-oriented criminal justice programs.

“There is a cacophony of metrics. These existing metrics are intensely flawed, and although each one of them has some value… they are still incomplete or problematic because while measuring something, they ignore other things that we think we should care about in the criminal justice system: respectful treatment of victims, humane treatment of offenders, strengthening and rehabilitating communities, repairing the harms,” she adds.

But that does not mean the traditional indicators should be dismissed, the researchers point out.

“When I talk about bottom-up or community-constructed indicators, it’s not to say that the so-called top-down indicators that are traditionally used are not worthwhile,” Dixon says.

“They are just very powerful indicators that are only telling a very small part of the story and they can really capture a lot of attention…

“They touch on people’s sense of physical security and they are very visceral, but they are also used strategically by politicians in a way that they also occupy a lot of public discourse, whereas those arguing for more city parks, for more mental health support, etc., don’t often have the same voice or the same power behind their voice.”

For Mulvey, “there are many researchers who just want to look at the numbers or just want to look at the things people tell them that are most important to look at, let’s say policymakers. That’s important but it isn’t the whole picture.”

“There is a cacophony of metrics... They ignore other things that we think we should care about in the criminal justice system: respectful treatment of victims, humane treatment of offenders, strengthening and rehabilitating communities, repairing the harms.”

– Hadar Dancig-Rosenberg

A community’s perception of success

“We need the community to champion programs that really work. We need the participant who is engaged in those services to speak up on what they have learned and how useful a program was or was not,” Bocanegra assures.

If their perceptions are not measured, the causes behind those successful outcomes could go unseen.

“While we can measure the jail population—we know if it’s going up or down—the issue then is what’s driving it, and that requires more of an understanding of all the processes that go behind any of those outcome measures,” Mulvey remarks.

And individual characteristics, which shape the work done by Bocanegra’s READI in Chicago, are often ignored by the system.

“We unfortunately operate the system by retributory categories rather than individual characteristics, and it’s been shown time and time again that the charge coming in is a lousy predictor of what happens to that person when they are out, and yet we organize the system largely according to charges,” Mulvey laments.

“We pass laws that say, ‘Anybody charged with this, this, this, and that offence goes to adult court instead of juvenile court.’ That’s probably one of the stupidest things you can do! Why don’t you just do it by shoe size?

“It has to move towards an individual assessment of characteristics or activities that are actually relevant to how well that person’s life will turn out.”

After years of research on the “gold ring” indicator and its positive results, Dancig-Rosenberg and Kleinfeld want to take their initiative to the next level and make social trust a widespread metric that drives policy.

They are likely to face resistance from the justice system on their way, and the quest for funds will not be an easy enterprise either.

But they are determined.

“We don’t think that we need to throw to the garbage all the existing metrics,” Dancig-Rosenberg concludes.

“We just think they are not enough—they are flawed. We can’t use them and only them. We need to add something more to the picture.”

And that something more is social trust.

“While we can measure the jail population—we know if it’s going up or down—the issue then is what’s driving it, and that requires more of an understanding of all the processes that go behind any of those outcome measures.”

– Edward Mulvey

Topic

Peace & Justice

Program

Global Innovations on Youth Violence, Safety and Justice