Denver's STAR Program and Disorder Crime Reductions

Note: This is one of a series of posts I created for the American Society of Evidence Based Policing, called The Criminal Justician. With the migration to the new site, these posts were not migrated, so putting them back here on the Crime De-Coder site for exposure. Encourage everyone to join ASEBP!

Hi everyone,

Some of y'all may know me, but for those who don’t, my name is Andy Wheeler. I am a recovering criminologist, and I periodically write opinion articles on contemporary crime science topics.

It is really hard to read academic journal articles and understand them – it takes not only understanding of empirical methods, but also understanding of the nature of the intervention, as well as often understanding of the nature of crime data. That is where I come in – I will take recent articles or strands of research and attempt to break them down into more manageable chunks to understand the results. Hence the new post series here on ASEBP, The Criminal Justician.

The article I will give a rundown today is in reference to analyzing a program that swapped out mental health professionals and paramedics to respond to low risk 911 calls that had a person in mental distress (in place of police officers) – the STAR program in Denver. Dee & Pyne (2022), DP for short from here on, analyzed crime incidents in Denver in conjunction with the program, and estimated that STAR resulted in an over 30% decrease in low level disorder crime incidents in the targeted areas (based on public open source crime data Denver disseminates).

Long story short, I don’t believe the estimated crime reductions DP find are reasonable given the scope of the program. And DP appear to have some confusing logic stemming from swapping calls for service vs recorded incidents/arrests. It takes some work, but below I will walk through my reasoning.

Recapping The Police Data Funnel

First, before we go into the nitty gritty on DP, lets do a recap on what the funnel of police data looks like. It can admittedly be confusing for outsiders and boring for CJ professionals, but is important to be really clear about each of these stages. Understanding the stages will guide our intuition for what a program like STAR could reasonably achieve.

We typically have a path from a 911 call that looks something like:

Call 911 -> Officer Responds -> Maybe Incident Report -> Maybe Arrest

So the STAR program swaps out mental health professionals and a paramedic instead of a police officer. So we have instead the path:

Call 911 -> Mental Health Prof. + Medic Responds -> Maybe Treatment

So the typical means via which individuals argue for alternative agencies to respond to 911 calls is harm reduction. There are potential harms associated with a police going to the scene (such as an arrest or even more severe uses of force against the person with mental health issues).

STAR responded to 748 calls for service over a pilot period of 6 months. Some were self initiated, but you can consider mostly just swapping out the STAR staff instead of police for 748 calls for service (to keep it simple I am treating 911 calls and calls for service the same, the latter also include police initiated events). Although DP call these STAR "incidents" – to be clear these 748 STAR calls are more equivalent to police calls for service – so I am going to refer to these as 748 STAR calls.

The thing that is maybe counter-intuitive to many folks, most of those calls for service, even when police respond, result in a nothing burger. It isn’t like if Denver PD took those 748 calls 100% would have resulted in an arrest. In fact the proportion of calls that ultimately lead to an arrest (or even an offense report) is quite tiny. If police get called to someone being disorderly, I would guess a big chunk are resolved before police even get to the scene (someone leaves when they know an officer is coming), and then another big chunk the police just tell a person to chill and leave (and no report for the maybe misdemeanor is ever filed).

So the correct estimate of the counterfactual harm reduced via a program such as STAR is to say “for these 748 calls for service, how many would have resulted in an arrest if STAR did not exist” (or some other type of negative CJ event, such as a citation).

While Denver does not release open source data on calls for service, I somewhat randomly choose Cincinnati and looked at calls for service with categories of Mental Health Disorders, Disorderly Conduct, and Drug Activity (so pretty close to the selection criteria for Denver). For those calls for service, counting up offense reports (which could later lead to an arrest), citations, and on-scene arrests, it ends up being less than 5% of those calls in Cincinnati lead to any criminal justice system touch. (Of those 5%, the vast majority are arrests, very few reports/citations. I would even guess most of the arrests are not for the on-scene behavior directly, but for people having prior warrants.)

So if we go back to the 748 STAR calls, if police instead of STAR responded, maybe only 40 would have resulted in later on harm directly related to the CJ system. Extrapolating from Cincinnati to Denver that would be my guess. (If an analyst from Denver wants to chime in let me know!)

Back to DP and their estimated crime reductions

So now that I have gone through all that. I think for a program like STAR, diverting a small number of arrests is a reasonable expectation (and may totally justify the program!)

One common thing you need to watch out for in social science research is there being some slippage between what we think may be going on, and the empirical data we use to see if a program is working. DP don’t look at arrests, but estimate crime reductions following the introduction of STAR.

There is a plausible (although I think probably unlikely) mechanism via which a program like STAR can reduce crime. It isn’t via the local 748 calls STAR responded to – those have already happened, STAR can’t undo those events. But, if the individuals STAR interacts with would go onto commit multiple future crime incidents, STAR could feasibly prevent future incidents.

DP’s analysis is really only relevant to those potential future crime incidents. Over a 6 month period, DP estimates that STAR prevented a total of almost 1400 low level disorder crime incidents.

Going back to the distinction between calls-for-service and crime incidents, DP intepret this 1400 as clear evidence of extra crime deterrence/prevention (1400 reduced incidents vs 748 calls for service). But again as I said above, those 748 calls result in subsequent crime incidents/arrests in very few cases. So the vast majority of reductions in crime incidents can only be reasonably attributable to reductions in future offending, not swapping out those 748 calls-for-service. (DP I think confuse themselves by referring to the 748 STAR “incidents”, whereas they are equivalent to 748 calls for service in the typical police funnel as I described earlier.)

This crime reduction estimate I find unlikely, but I cannot entirely rule out. When I worked as a crime analyst at Troy, NY, a single individual broke into something like 50 cars in a few months time frame. In other work I’ve done in Syracuse, the most prolific offender had 46 arrests over a 6 year period (Wheeler, 2012). (And that person had many arrests that were targeted by STAR, such as indecent exposure, but also a few violent incidents.)

Although STAR responded to 748 calls, they only ended up providing services to 243 individuals. Even if the program was wildly successful in reducing future crime, preventing almost 6 future low level crimes per person over a horizon of less than 6 months I don’t think is very realistic. Even if STAR specifically targeted chronic offenders (like the person with 46 arrests in Syracuse), they wouldn’t have a 100% success rate.

Note as well that STAR isn’t itself a treatment regime, it is just a mental health officer/paramedic who are suppossed to refer the individual to appropriate treatment (which can also occur via more typical police involvement). So DP are saying simply the on-scene diversion results in all those subsequent future crimes prevented.

Even though the empirical strategy DP use I have no direct fault with (and have used myself, difference-in-differences with comparison areas), it may be some Covid weirdness going on is throwing off the results. (The STAR pilot period started in June 2020.)

Am I against alt-police responses?

In principle, I think having alternative resources to police responders can make sense. Just as we don’t need to send out firetrucks to every emergency incident, we don’t need to send the police calvary to every call of a homeless person trespassing.

It happens though that other programs trying to divert police resources, such as the 311 system (Mazerolle et al., 2002), resulted in net widening, not an overall reduction in total calls. I suspect that is also what will happen long term with STAR or similar programs. Matching people to potential services is a good thing, even if it doesn’t reduce overall crime (which I think is an unlikely short term outcome from these programs).

So I am pretty skeptical STAR is as effective as DP estimate in reducing crimes, even though I don’t deny it may be a good program overall. I think a simpler analysis in terms of estimating future crime reductions will need to look at people actually served by the program, not just neighborhood crime rates. That is someone with access to the records looks at the 248 people STAR served, create a matched comparison control sample, and look at follow up rates of future incidents for those STAR vs control individuals. This I would bet is possible with data that Denver agencies already collect, but wouldn’t be possible just via the public data Denver disseminates.

References