AI and Government Reform: Towards Humanless Bureaucracies?

Text originally published in French in November 2021 on AOC media.

In recent years, with extensive use of vocabulary borrowed from the “start-up” world, we have witnessed the consolidation of a new iteration of neoliberal state reform. Through concepts such as “platform state” or “state start-up”¹, new reformers have been counting on “corporate hackers” and disruptive innovation to transform public bureaucracies from within, unleash creativity, restore transparency, deploy “agile” methods, and adapt to a constantly changing environment—all at lower cost. The “digital state”—a concept praised in June 2017 by Emmanuel Macron during his famous ode to the “startup nation”—is also and above all a state undergoing automation: to accompany the objective of not replacing nearly 50,000 civil servants by 2022, the Public Action 2022 plan launched in October 2018 relied on “technologies such as artificial intelligence and RPA (’robotic process automation’),” in order to “automate repetitive and low value-added tasks”².

The context is indeed conducive to the automation of numerous administrative functions. Beyond the conversion of part of the political and administrative elites to the concept of “data-driven governance,” the technological offerings of private providers have also become structured, thanks in particular to the rapid development of so-called “artificial intelligence” (AI) techniques. On the side of the “administered,” this project encounters greater acculturation of the French population to digital technology³, with a high rate of equipment and numerous programs to “train” them in digital tools, whether initiated by public authorities or by companies like Google (a company welcomed with open arms by several French cities to install its “digital workshops”).

These various factors are driving a multiplication of “algorithmic assemblages” in administrations⁴. Whether it be the digitization of numerous administrative procedures, the partial automation of post-baccalaureate placement with Parcoursup, the automation of job seeker support with the “Mon Assistant Personnel” project tested by Pôle Emploi, the automation of social fraud prevention by family allowance offices or tax fraud prevention by the tax authority, the automation of identity checks via facial recognition at PARAFE terminals installed in several airports, or the automation of suspicious communications or behaviors within police or intelligence services, many administrative practices are now increasingly articulated with semi-autonomous algorithms.

Faced with this process, concerns are being expressed in certain segments of society. Notably mentioned are risks to privacy, the advent of a “surveillance society” supported by information technology, the opacity of algorithmic assemblages, the aggravated forms of dehumanization induced by automation, or the accentuation of structural inequalities. Even within the bureaucracies affected by these transformations, fear of the displacement of humans by machines, increased procedures, the risk of technical bugs, or the creeping phenomenon of privatization also generate their share of resistance among agents.

At the same time, advances in the field of AI have led to a proliferation of work dedicated to the social, ethical, and legal issues of these technologies increasingly integrated into the functioning of large organizations. The objective is to establish minimal rules guaranteeing the “social acceptability” of this new stage in the computerization process. Through a certain number of procedures, the aim is for example to formalize ethical choices made in the development of algorithms, to guarantee their transparency, their “explainability” or their “auditability,” or to reduce as much as possible the risks to fundamental rights through approaches called “privacy-by-design” and other “ethical charters.”

The Failure of the Procedural Approach

These reflections have led to a proliferation of reports, white papers, or soft-law texts emanating from the private sector, data protection organizations like CNIL, institutions like the European Parliament, but also international organizations like the Council of Europe or the United Nations Human Rights Council. At the international level, between 2016 and 2019, more than seventy ethical charters dedicated to artificial intelligence were published by various organizations, often adopting a probabilistic and quantifiable approach to the risks associated with AI⁵. On the public administration side, this work has sometimes resulted in ad hoc legislative or regulatory provisions, such as the “algorithmic impact assessment” now required of Canadian federal administrations⁶, but also the French law which, since 2016, guarantees a principle of transparency for public algorithms used to make individual administrative decisions⁷.

Despite the generally dominant impression of facing entirely unprecedented challenges, it is striking to observe the similarity of current controversies with those that marked the first cycle of computerization of society in the 1970s. Yet, whether it was the first laws on personal data protection against computerized filing or progress in administrative transparency to compensate for the increased opacity induced by state computerization, these procedural arrangements have considerably failed to achieve the objectives assigned to them.

This failure occurred in a rather favorable political and technological context: a period when, in response to the protests of the time, states were adopting major liberal legislation supposed to extend rights and freedoms, and when the marketing of microcomputers and the advent of decentralized telecommunications networks led many commentators to predict a historic advance in the democratization process. Forty years later, at a time when Big Data or artificial intelligence is accompanied by a phenomenal recentralization of computing capacities, at a time when the accumulation of crises and emergency legislation leads to the hypertrophy of executive power and the consolidation of increasingly polycentric police control, such a decoupling between power and information technology seems less likely than ever.

Despite this unpromising context, contemporary controversies continue to rely on the same remedies, made of safeguards for rights and freedoms and transparency obligations. Recently, one of the rapporteurs in Parliament for the so-called “global security” bill well summarized the situation by evoking his conviction that despite their dangers, the use of surveillance drones by the police could be framed by appropriate guarantees: “I’m trying to find,” he explained, “reasonable accommodations, that is to say ‘yes, but’.” This is how at each parliamentary debate on bureaucratic computerization, we choose to “take the risk,” trying to counter it by the eternal return of “reasonable accommodations.” To such an extent that we would like to oppose the logic of “yes, but” with the old Latin saying: “Errare humanum est, perseverare diabolicum” (“to err is human, to persevere [in one’s error] is diabolical”).

History is not, however, the only discipline rich in lessons about the inadequacy of these “accommodations” for the defense of democratic values. Certain work in the sociology of science and technology has also shown the limited practical scope of these procedural approaches, the rules thus posed often being inapplicable, circumvented, or deprived of their effects. On the legal level finally, experience also shows that the “safeguards” adopted by the legislator or judges prove very fragile. Given the bureaucratic logics of optimization (functional, budgetary, etc.) of administrative tasks, technological innovation (in identity checks, placement of high school graduates, etc.) systematically tends to create a need, which sooner or later results in reforms aimed at legalizing uses that previously seemed technically unthinkable or socially unacceptable.

Dispensable Civil Servants

By reducing the issues to procedural questions—thus to often technical questions about how to regulate the use of a device—and by focusing on technological artifacts while neglecting their institutional environment, current controversies also tend to evade certain political problems raised by bureaucratic automation. For example, there is a fundamental question that is rarely debated: what would be the political significance of an almost entirely automated bureaucracy? The critique of the anti-democratic tendencies of bureaucratic power has occupied an important place in political theory since the 19th century. But without minimizing the forms of violence associated with past and present bureaucracies, the fact remains that in practice, their dystopian potential can be partly contained by the women and men within them.

This is what numerous works in political sociology highlight. For Michael Mann, for example, Max Weber’s critique of bureaucracies exaggerates the extent of the power they can exert on society: according to him, since the state must proceed with the massive recruitment of civil servants to fill the ranks of ever more sprawling bureaucracies, the “infrastructural power” of the state over “civil society” develops at the cost of a penetration of its own administrative structures by this same civil society, which through these increasingly numerous civil servants is able to influence the state⁸. In an approach that has the merit of avoiding too binary an opposition between state and “civil society,” Timothy Mitchell also evokes forms of internal resistance nourished by “political subjects […] formed within the organizational sphere called the state”⁹.

More empirically, the sociology of public action has largely emphasized the role played by “street-level bureaucrats"—that is, agents placed in direct contact with the administered—in the work of interpreting and adapting rules to the realities of publics and fields¹⁰. Within the discipline, a debate has been ongoing for twenty years to elucidate the effects of the introduction of information technologies on this work of interpretation¹¹. While these effects appear ambivalent, the fact remains that in a certain number of contexts, technology already makes such hermeneutics almost impossible. And even though more field studies would be necessary to assess it precisely, we can expect that the increasing automation of certain administrative tasks linked to the latest technological innovations will further erode the margins of maneuver left to “street-level bureaucrats.”

This could moreover be one of the tacit drivers of the bureaucratic automation movement. If, as anthropologist David Graeber suggests, bureaucracy is “made, first and foremost, of aggressions against those who persist in defending other schemas or different interpretations”¹², it is hardly surprising that it apprehends the subjectivity of its agents as a dispensable hazard, approaching the civil servant and their moral dilemmas as a bug to be corrected¹³. In an automated bureaucracy, no disobedience is possible in the application of the rule inscribed in the device, no more whistleblower whose conscience could be so shocked by bureaucratic violence that they decide to bring the matter to public knowledge. By eliminating the human element in the application of rational and impersonal rules, bureaucratic automation has the potential to bring about the totally dehumanized government that Hannah Arendt described at the turn of the 1960s: “in a fully developed bureaucracy, there is no one left with whom one can discuss, to whom one can present grievances (…)”¹⁴.

This analysis leads to another question: doesn’t the fact of accepting to be administered by machines lead to breaking with the political horizon of humanism, by accepting to conceive ourselves—administered persons—as machines, that is to say as objects of government rather than political subjects? In a recent symposium in which I participated and where some were worried about the impossibility of granting legal responsibility to an autonomous device (for example an “intelligent” car), a CNRS research director, specialist in decision support, felt that such concerns were unfounded. According to him, we cannot either—at least not always—explain the motivations, reasoning, and decisions of an individual. However, this does not prevent, in the current state of law, considering these same individuals as subjects of law. From his point of view, there would therefore be no particular problem in apprehending a computer as a legally responsible person.

Governed Like Machines

As innocent as it may seem, this kind of assertion actually assumes a break with the humanist principles at the foundation of liberal law¹⁵. The latter considers indeed that, even if the motivations and mental states of others cannot always be probed, they are in fact equivalent to ours. According to this conception, others are presumed endowed with the same faculties as us, also animated by will and desires. This is what makes them a peer who should enjoy rights and dignity equal to ours. Even if the fact of being administered by bureaucratic organizations does constitute a form of dispossession and violence, at least it is partly made more tolerable (because more “negotiable”) by the existence of human interlocutors (hoping that these people will be able to make good use of a certain margin of interpretation).

As soon as this “human element” disappears, as soon as only the cold and impenetrable functioning of an algorithm based on abstract simplifications and a quantitative and probabilistic approach to reality remains (even if parameterized by an engineer), this capacity to negotiate with a peer disappears. As Arendt noted, it then becomes impossible to assert our specific situations, our subjectivities and our affects by counting on the empathy of others. By building ever more automated bureaucracies, we therefore risk settling this “humanist anthropology” at the foundation of liberal law to substitute for it a warrior and cybernetic anthropology within which, as philosopher of science Peter Galison emphasizes, each is considered as a “black box,” with “inputs and outputs and no access to the inner life of others¹⁶.” Which amounts, ultimately, to accepting to consider each other as machines.

The procedural focus and the primacy given to socio-technical devices in the evaluation of “risks” also sets aside other issues raised by the logic of bureaucratic optimization. For example, one of the most common justifications for the computerization of administrations relates to accounting and budgetary objectives: by achieving a qualitative leap in automation, AI would allow “scaling up” this or that bureaucratic process while achieving economies of scale. Thus, in a letter to CNIL, the PACA region recently defended the experimentation of facial recognition around high schools by asserting that this project constituted “a response to the growing gap observed between the requirements for securing entrances to establishments and the human resources available in high schools, within the framework of successive plans to reduce staff in the civil service”¹⁷. Similarly, so-called “backtracking” applications popularized during the COVID-19 pandemic consist of automating contact-tracing strategies traditionally conducted by health professionals or volunteers to identify contamination chains—approaches that are financially costly to scale up. As for the automatic detection of social and tax fraud based on the analysis of publications on social networks currently being tested by the French tax authority, it occurs in a context where more than a quarter of the approximately 12,000 positions dedicated to tax auditing have been eliminated¹⁸.

These are not isolated cases: Romain Boucher, a “data scientist” who took part as a consultant in numerous “transformation of public action” projects, summarizes his experience in these terms:

“After a while you realize that the types of missions that come back are surveillance missions and staff rationalization (…) We were asked to make beautiful presentations to show Public Transformation how much we were saving. The logic of improving public service no longer existed”¹⁹.

Intimately linked to austerity policies, automation thus continues the neoliberal movement of undermining public service but also of precarization and deskilling of part of the civil service, victim of increasing atomization and the dismantling of its collective action capacities.

Among the other issues frequently evaded by the procedural focus, we could also mention the ecological cost generated by the automation process (one of the arguments opposed by Enedis subscribers to the automation of their electricity consumption readings via connected meters), the privatization of expertise and the weakening of public agents’ skills since algorithms are most often designed by external providers (which police unions opposed to partnerships with companies specializing in surveillance technologies have complained about²⁰), the frequently observed inability of major digitization and automation projects to achieve the objective of “simplification” and their very real effects on the aggravation of inequalities²¹, or the legitimacy of the demands of those who refuse to respond to multiple technological injunctions and wish to continue to benefit from public services through physical counters, populated by humans in the flesh²².

If bureaucratic automation aggravates the totalitarian tendencies of bureaucratic domination, and if the multiplication of procedures to contain its deleterious effects has proven too ineffective, then it may be time to draw the consequences. Without necessarily giving up on overcoming the antagonism between bureaucracy and democracy²³, it would then be a matter of adhering to a precautionary principle by joining our voices to those who, in different places in the social world, loudly proclaim their refusal of a “digital state” and the technocratic project from which it proceeds.

---