Family Surveillance by Algorithm

Janice Howe's grandchild Derrin Yellow Robe, 3, stands in his great-grandparents' back yard on the Crow Creek Reservation in South Dakota. Along with his twin sister and two older sisters, he was taken off the reservation by South Dakota's Department of Social Services in July of 2009. READ

It took over a year and a half for Erin Yellow Robe, a member of the Crow Creek Sioux Tribe, to be reunited with her children. Based on an unsubstantiated rumor that Erin was misusing prescription pills, authorities took custody of her children and placed them with white foster parents — despite the federal Indian Child Welfare Act’s requirements and the willingness of relatives and tribal members to care for the children.

For white families, these scenarios typically do not lead to child welfare involvement. For Black and Indigenous families, they often lead to years — potentially a lifetime — of ensnarement in the child welfare system or, as some are now more appropriately calling it, the family regulation system.

Child Welfare as Disparate Policing

Our country’s latest reckoning with structural racism has involved critical reflection on the role of the criminal justice system, education policy, and housing practices in perpetuating racial inequity. The family regulation system needs to be added to this list, along with the algorithms working behind the scenes. That’s why the ACLU has conducted a nationwide survey to learn more about these tools.

Women and children who are Indigenous, Black, or experiencing poverty are disproportionately placed under child welfare’s scrutiny. Once there, Indigenous and Black families fare worse than their white counterparts at nearly every critical step. These disparities are partly the legacy of past social practices and government policies that sought to tear apart Indigenous and Black families. But the disparities are also the result of the continued policing of women in recent years through child welfare practices, public benefits laws, the failed war on drugs, and other criminal justice policies that punish women who fail to conform to particular conceptions of “fit mothers.”

Turning to Predictive Analytics for Solutions

Many child welfare agencies have begun turning to risk assessment tools for reasons ranging from wanting the ability to predict which children are at higher risk for maltreatment to improving agency operations. Allegheny County, Pennsylvania has been using the Allegheny Family Screening Tool (AFST) since 2016. The AFST generates a risk score for complaints received through the county’s child maltreatment hotline by looking at whether certain characteristics of the agency’s past cases are also present in the complaint allegations. Key among these characteristics are family member demographics and prior involvement with the county’s child welfare, jail, juvenile probation, and behavioral health systems. Intake staff then use this risk score as an aide in deciding whether or not to follow up on a complaint with a home study or a formal investigation, or to dismiss it outright.

Like their criminal justice analogues, however, child welfare risk assessment tools do not predict the future. For instance, a recidivism risk assessment tool measures the odds that a person will be arrested in the future, not the odds that they will actually commit a crime. Just as being under arrest doesn’t necessarily mean you did something illegal, a child’s removal from the home, often the target of a prediction model, doesn’t necessarily mean a child was in fact maltreated.

We examined how many jurisdictions across the 50 states, D.C., and U.S. territories are using one category of predictive analytics tools: models that systematically use data collected by jurisdictions’ public agencies to attempt to predict the likelihood that a child in a given situation or location will be maltreated. Here’s what we found:

• Local or state child welfare agencies in at least 26 states plus D.C. have considered using such predictive tools. Of these, jurisdictions in at least 11 states are currently using them.
• Large jurisdictions like New York City, Oregon, and Allegheny County have been using predictive analytics for several years now.
• Some tools currently in use, such as the AFST, are used when deciding whether to refer a complaint for further agency action, while others are used to flag open cases for closer review because the tool deems them to be higher-risk scenarios.

The Flaws of Predictive Analytics

Despite the growing popularity of these tools, few families or advocates have heard about them, much less provided meaningful input into their development and use. Yet countless policy choices and value judgments are made in the course of creating and using the tool, any or all of which can impact whether the tool promotes “fairness” or reduces racial disproportionality in agency action.

Moreover, like the tools we have seen in the criminal legal system, any tool built from a jurisdiction’s historical data runs the risk of continuing and increasing existing bias. Historically over-regulated and over-separated communities may get caught in a feedback loop that quickly magnifies the biases in these systems. Who decides what “high risk” means? When a caseworker sees a “high” risk score for a Black person, do they respond in the same way as they would for a white person?

Ultimately, we must ask whether these tools are the best way to spend hundreds of thousands, if not millions of dollars, when such funds are urgently needed to help families avoid the crises that lead to abuse and neglect allegations.

What the ACLU is Doing

It’s critical that we interrogate these tools before they become entrenched, as they have in the criminal justice system. Information about the data used to create a predictive algorithm, the policy choices embedded in the tool, and the tool’s impact both system-wide and in individual cases are some of the things that should be disclosed to the public before a tool is adopted and throughout its use. In addition to such transparency, jurisdictions need to make available opportunities to question and contest a tool’s implementation or application in a specific instance if our policymakers and elected officials are to be held accountable for the rules and penalties enforced through such tools.

In this vein, the ACLU has requested data from Allegheny County and other jurisdictions to independently evaluate the design and impact of their predictive analytics tools and any measures they may be taking to address fairness, due process, and civil liberty concerns.

It’s time that all of us ask our local policymakers to end the unnecessary and harmful policing of families through the family regulation system.

Read the full white paper:

https://www.aclu.org/fact-sheet/family-surveillance-algorithm

We need you with us to keep fighting

News Provided By

ACLU