Dutch childcare benefit scandal is an urgent wake-up call to ban racist algorithms

[ad_1]

The Dutch government risks exacerbating racial discrimination by continuing to use unregulated algorithms in the public sector, Amnesty International said in a damning new analysis of the childcare allowance scandal in the country.

The Xenophobic Machines report outlines how racial profiling was incorporated into the design of the algorithmic system used to determine whether claims for child care allowances were flagged as incorrect and potentially fraudulent. Tens of thousands of parents and guardians, mostly from low-income families, have been falsely accused of fraud by the Dutch tax authorities, with people from ethnic minorities disproportionately affected. While the scandal brought down the Dutch government in January, sufficient lessons have not been learned despite multiple investigations.

Governments around the world are rushing to automate the delivery of public services, but it is the most marginalized in society who are paying the highest price.

Merel Koning, Senior Technology and Human Rights Advisor

“Thousands of lives were ruined by a shameful process that included a xenophobic algorithm based on racial profiling. The Dutch authorities risk repeating these catastrophic mistakes as human rights protections are still lacking in the use of algorithmic systems, ”said Merel Koning, Senior Technology and Human Rights Advisor at Amnesty International.

“Alarmingly, the Dutch are not alone. Governments around the world are rushing to automate the delivery of public services, but it is the most marginalized in society who are paying the highest price. “

Amnesty International calls on all governments to implement an immediate ban on the use of nationality and ethnicity data in law enforcement risk assessment in the search for potential crime suspects or of fraud.

Thousands of lives were ruined by a shameful process that included a xenophobic algorithm based on racial profiling.

Merel Koning Discriminating loop

From the outset, racial and ethnic discrimination has been central to the design of the algorithmic system introduced in 2013 by the Dutch tax authorities to detect incorrect claims for family allowances and potentially fraud. The tax authorities used information indicating whether an applicant had Dutch nationality as a risk factor and non-Dutch were given higher risk scores.

Parents and guardians selected by the system have had their benefits suspended and been subjected to hostile investigations, characterized by tough rules and policies, rigid interpretations of laws and ruthless benefit collection policies. This resulted in devastating financial problems for affected families, ranging from debt and unemployment to forced evictions because people were unable to pay their rent or pay off their mortgages. Others have found themselves with mental health issues and stress in their personal relationships, leading to divorces and broken homes.

The design of the algorithm reinforced the existing institutional bias of a link between race and ethnicity and criminality, as well as the generalization of the behavior to an entire race or ethnic group.

These discriminatory design flaws were reproduced by a self-learning mechanism which meant that the algorithm adapted over time based on experience, without any significant human oversight. The result was a discriminatory loop with non-Dutch nationals reported as potentially fraudulent more frequently than those of Dutch nationality.

Lack of responsibility

When someone was flagged as being at risk of fraud, an official had to perform a manual review, but had no information on why the system generated a higher risk score. Such opaque “black box” systems, in which system inputs and calculations are not visible, have led to a lack of accountability and control.

“The black box system has resulted in a black hole of accountability, with Dutch tax authorities trusting an algorithm to help decision making without proper oversight,” said Merel Koning.

A perverse incentive existed for the tax authorities to seize as much money as possible regardless of the veracity of the fraud accusations, because the effectiveness of the algorithmic decision-making system had to be proven. For years, parents and guardians identified by tax authorities as fraudsters received no answers to questions about what they had done wrong.

The findings of xenophobic machines will be presented at a side event at the United Nations General Assembly on algorithmic discrimination on October 26. This year, Amnesty International is launching an Algorithmic Accountability Lab, a multidisciplinary team to investigate and campaign on the human rights risks of automated decision-making systems in the public sector. Amnesty International calls on governments to:

Prevent human rights violations related to the use of algorithmic decision-making systems, including by implementing a mandatory and binding human rights impact assessment prior to the use of such systems. Establish effective mechanisms for monitoring and controlling algorithmic systems in the public sector. accountable for violations, and provide an effective remedy to individuals and groups whose rights have been violated.

[ad_2]