Europe’s Decision Machines

They do so under the promise that these data-driven tools help public officials to make objective and fair decisions. But the secrecy surrounding these machines has so far prevented the public from learning how they actually work and the impact they have on people's lives.

Just after Europe passed its new Artificial Intelligence regulation, this research sheds light on several risk assessment tools used by law enforcement and judicial authorities in Spain, the Netherlands and Italy. Through unpublished technical materials and human stories, this project connects the dots between three specific automated tools and the lives of thousands of individuals in Europe that are in prison, detained or simply under suspicion.

In Catalonia, Spain, prison authorities have been using for more than a decade RisCanvi, a tool that tries to predict which inmates will reoffend. Technical details never disclosed before have allowed us to recreate this algorithm and explain its inner workings to a non-specialist reader. At the same time, the testimonies collected demonstrate how RisCanvi’s predictions play a decisive role in the decisions of judges and, consequently, in the social rehabilitation of inmates.

In the Netherlands, several government projects assess which youth are at risk of becoming criminals. We have had access to project plans of multiple municipalities and two ‘risk taxation forms’. Considering that these projects focus on specific neighbourhoods in municipalities, we have also obtained insight into data analyses that determine which neighbourhoods are singled out. Also, we have found families where the government intervened because they are considered high risk, with severe consequences even for children who had never been in contact with law enforcement.

In Italy we have analysed a predictive policing software developed by KeyCrime, a company now in liquidation due to the AI Act, which for years have been applied in the city of Milan. And the system XLAW, which underwent a trial period at the Police Headquarters of Naples, Prato, Venice, Modena, and Parma until March 2021. This system, however, has never been formally validated by the Central Anti-Crime Directorate of the Italian State Police, contrary to what the company has claimed.

Photo by El Confidencial