Jason Jia-Xi Wu*
Download a PDF version of this article here.
The rampant growth of artificial intelligence (AI) has reshaped the landscape of credit underwriting and distribution in consumer financial markets. Despite expanding consumers’ access to credit, the unbridled use of AI by creditors has widened credit inequality along racial, gender, and class dimensions. Existing regulatory paradigms of consumer financial protection fail to meaningfully protect consumers against potential AI discrimination and exploitation. At its core, the failure of the existing legal regime lies in its fetishization of free markets and consumer autonomy—the two ideological pillars of neoliberalism. Judges and lawmakers who subscribe to neoliberal ideals have consistently attributed credit market defects to individual choices, rather than systemic and inherited social inequalities. Today, this neoliberal ethos continues to inform mainstream legal responses to the threats posed by AI.
This article proposes an alternative. It argues that thinking of AI governance in purely individualist, dignitarian terms obscures the real source of algorithmic harm. Contrary to neoliberal assumptions, AI-inflicted harms in credit markets—e.g., algorithmic discrimination and exploitation—are not the result of irresponsible creditor conduct or opaque markets. Rather, they are caused by unjust relations of data production, circulation, and retainment that reflect and reproduce systemic social inequalities. Understanding algorithmic harm as both individually and socially constituted can help lawmakers move away from the outdated neoliberal paradigms that idolize individual responsibility. It also opens new avenues for legal reform. To reshape unjust data relations, this article proposes a propertarian approach to AI governance that involves: (1) reimagining the nature of data ownership, (2) creating a collective intellectual property right in data, and (3) building a collective data governance infrastructure anchored in the open digital commons.
Introduction
For decades, our legal system has embraced neoliberalism as the dominant regulatory ethos for consumer financial protection.1 Its twin ideals—free markets and consumer autonomy—serve as the guiding principles governing the supply and underwriting of credit.2 For markets to be free, constraints on informational flow must be removed, price distortions must be controlled,3 and governments should not regulate absent market failure.4 For consumers to be autonomous, markets must be transparent enough to enable unhindered consumer decision-making.5 Viewed holistically, these two pillars of neoliberalism undergird the prevailing ideology of consumer protection: the freer the markets, the more autonomous the consumers.
The ideal of free markets finds legal expression in consumer credit reporting and disclosure laws. Such laws aim to facilitate the efficient and transparent flow of market information. The Truth in Lending Act (TILA)6 and the Fair Credit Reporting Act (FCRA)7 require creditors to disclose lending terms, as well as material risks and consequences therefrom. With the enactment of these laws, Congress endorses the view that disclosure reveals the true cost of lending, which can level the playing field for creditors, and enable consumers to compare similar or substitutable products.9 Born out of the 1970s civil rights movement, the Equal Credit Opportunity Act (ECOA)10 and Fair Housing Act (FHA)11 have applied colorblind principles12 of non-discrimination and race-and-gender-neutrality to the underwriting of consumer credit.13 These statutes reflect the congressional view that disparate treatment14 in credit undermines consumers’ exercise of individual free choice and agency.15
Together, these consumer financial protection laws, which embody the twin ideals of free markets and consumer autonomy, reinforce the neoliberal ideology of individual responsibility.16 Rather than treating credit inequality as a socially-constructed systemic problem, our consumer financial protection laws deem inequality as outcomes of individual choice.17 Absent from the regulatory toolkit is the language to describe systemic injustices, redress collective harm, or install broad social infrastructures. Over the past five decades, this ideal of individual responsibility has coalesced into a neoliberal consensus that crowded out alternative visions for our consumer financial protection regime.
However, this neoliberal consensus is now disrupted by the rise of artificial intelligence (AI) in consumer finance.18 Increasingly, credit unions, banks, and lenders use AI to underwrite consumer credit.19 Because AI does not need transparent market information or human actions in making credit decisions, it renders the current disclosure-based consumer protection regime20 ineffective. Advanced machine learning21 techniques such as deep learning (DL) can now scrape and process unimaginable volumes of data in the blink of an eye.22 These algorithms can continually adapt and tune their parameters to reflect new informational intake with minimal or no human supervision.23 Due to the algorithms’ black-box properties, even original programmers cannot understand some of AI’s predictions.24 Moreover, AI generates predictions about consumer creditworthiness even without credit history or formalized financial data. Instead, AI analyzes “fringe data”25—e.g., online subscriptions, club memberships, browser history, location, and social media—information that may be irrelevant to determinations of creditworthiness.26 This process can be entirely unsupervised and incomprehensible, undermining the fairness of credit provision.27
Normative and Legal Implications
This article examines how AI disrupts the normative and legal underpinnings of neoliberalism embedded in our consumer financial protection regime.
From a normative perspective, AI problematizes neoliberal ideals of free markets and consumer autonomy. With regards to the free market ideal, AI challenges the notion that prices can ever be transparent or neutral. In digital environments where AI could use scraped data to manipulate consumer behavior and tailor-recommend products at inflated prices,28 prices do not reflect the objective market value that consumers (as market agents) ascribe to their preferences.29 With regard to the consumer autonomy ideal, AI defies the prevailing understanding that more information is always better for consumers. Through manipulating personal data and inundating consumers with information, AI can easily distract consumers from their true product preferences.30 Under the psychological mechanism of confirmation bias,31 overwhelmed consumers can easily agree to terms against their best interests.32 Thus, widespread, unrestrained adoption of AI solutions in the consumer financial market can undermine both free choice and market transparency.
From a legal perspective, AI exposes the blind spot of individualist consumer protection regimes; its commitment to formal equality conceals systemic inequalities. Existing disclosure and fair lending laws embrace the assumptions of market neutrality and formal equality of economic opportunities without recognizing the substantive, systemic inequalities in credit provisions.33 Consequently, they adopt individual-based solutions to credit inequality, which are inherently ill-fit for systemic problems. Both the ECOA34 and TILA35 look exclusively to creditors’ individualized conduct when the laws should instead look to the parties’ market relations.
Essentially, neoliberalism’s emphasis on formal equality and individualism obscures the source of algorithmic harm: unjust market relations. AI aggregates data of specific consumers in unaccountable ways and derives knowledge about general consumer groups from this aggregated data (i.e., knowledge discovery); this affects both consumers within direct transactional relations with creditors and nonparties.36 Whether intentional or not, creditors’ widespread use of AI for credit underwriting may reinforce unjust market relations between creditors and all consumers. This occurs because creditors, as owners and users of AI systems, control the channels of consumer data production, circulation, and retainment.
Key Concepts and Definitions
Before delving into the details, it is necessary to first clarify some key concepts being invoked throughout this article:
(i) Artificial Intelligence: When this article uses the term AI, it focuses on a subset of machine learning,37 or deep learning (DL), that is currently being deployed by FinTech lenders to assess and underwrite consumer credit.38 DL uses a layered decision-making structure called artificial neural networks, which simulates the neural networks of a biological brain.39 Like other machine learning techniques, DL algorithms operate by harvesting training data, extracting features from datasets, learning from these features, and “apply[ing] what they learned to larger datasets to determine or predict something about reality.”40 The key difference is that, while earlier iterations of machine learning required human instructions to extract features from data inputs, DL recognizes patterns automatically.41 What this means is that a DL algorithm can engage in its own feature extraction, continuously learn from past mistakes, and self-adjust future interactions with consumer data inputs each time it makes a prediction.42 After a few iterations, the DL model matures its decision logic by eliminating noise data that is contradictory or irrelevant.43 Although FinTech lenders and creditors also use other AI technologies for credit underwriting, their use of DL models currently raises regulatory concern due to their opaqueness and self-learning capabilities.44 Regulators’ primary concern is that DL models often use concepts that produce unpredictable outcomes.45
(ii) Algorithmic Harm: This article identifies two sources of algorithmic harm: (1) algorithmic decisional harm, which refers to the harm that consumers incur when algorithms exploit consumers (through price discrimination)46 by taking advantage of their market-induced insecurities or cognitive flaws through the use of biased information that the algorithm has garnered about individual consumers or consumer groups,47 and (2) algorithmic informational harm, which refers to the harm that consumers suffer due to how information about them (whether consumer-owned or within their reasonable expectations of privacy) is collected, processed, and engineered to construct archetypes of consumer preferences for market usage.48 Whereas the former category describes harms associated with problematic outputs, the latter describes harms associated with problematic inputs.
(iii) Knowledge Discovery: This refers to the process by which data (e.g., digital footprint, market information, online records) regarding any consumer group or individual is discovered—that is, through scraping, mining, and aggregating.49 Data discovered via this process is then tuned and optimized to generate behavioral insights (i.e., knowledge) about consumers who are subjects of algorithmic decision-making. Machine learning is a technique to conduct knowledge discovery. By way of illustration, machine learning generates predictions through the following repeating steps: (1) data gathering and cleansing; (2) splitting the data into a training and a testing dataset; (3) training the predictive model with training dataset based on the algorithm’s instructions; (4) validating the model with the testing dataset.50
(iv) Consent Manufacturing: This refers to processes of information control that manipulate consumer desire and influence consumers to make market decisions against their interests. In AI-mediated credit markets, consent-manufacturing takes two forms: (1) creation of personalized information silos that control expectations of consumers who engage in a credit transaction with an AI-informed creditor; and (2) production of generalized knowledge about group consumption behaviors designed to manipulate prospective consumers and nonparties to the credit transaction.51
(v) Credit Underwriting: This refers to the practice of underwriting consumer credit through risk-based assessment of consumer creditworthiness.52 Typically, creditors base their decisions to extend or deny credit to a consumer on the following considerations: (1) the probability of default or delinquency (i.e., consumer credit risk); (2) the opportunity cost of underwriting (i.e., expected return); (3) the possibility of loan recovery for the type of financial product offered, factoring in the creditor’s asset portfolio (i.e., risk adjustment).53 If the creditor accepts the consumer’s application for a loan, then the creditor calculates an estimated price range for the risk-return tradeoff that would render the credit extension profitable.
Traditionally, creditors rely on the credit reports issued by credit bureaus (e.g., Equifax, Experian, and TransUnion) to conduct risk-based lending.54 Over the past three decades, credit scores and automated scoring systems have become the dominant method for underwriting consumer credit.55 Regulators have criticized credit reports and credit scores as systematically disadvantageous to consumers with thin credit histories or lack of prior engagement with the banking system.56 In the last five years, creditors have increasingly shifted to AI to assess and underwrite consumer credit.57 The rise of AI credit underwriting coincided with the emerging practice of using alternative “fringe data” to assess consumer creditworthiness, which does not require formalized credit information used by conventional credit reporting and scoring.58 Bankers and FinTech lenders tout the use of AI as the panacea to enhance credit access for the “unbanked” and the “underbanked” consumers.59 Its usage is most concentrated in the underwriting of unsecured personal loans and credit cards.60 From 2015 to 2019, FinTech lenders nearly “doubled their share” in the unsecured personal loan market and “now account for 49% of originated loans.”61 Auto-lending62 and small business lending63 are also areas where machine learning algorithms are used for credit underwriting.
Analytical Roadmap
The remainder of this article proceeds as follows. Part I investigates two questions that lie at the heart of this article: How are AI technologies being introduced in ways that intensify systemic credit inequalities? To the extent that AI is being used to exploit consumers through the extraction and commodification of consumer data, where is the locus of algorithmic harm in these spaces?64 To answer these questions, Part I articulates a theory of price engineering and consent manufacturing to explain why and how AI technologies have been used to perpetuate unjust market conditions for credit access.
Part II explains why the contemporary consumer financial protection regime, informed by the neoliberal ideals of free markets and consumer autonomy, fails to address the risks of algorithmic harm. The principal reason for failure is that the existing fair lending and disclosure laws overly fixate on protecting individual market freedom with minimal regard to systemic and relational inequalities. As this Part aims to demonstrate, the neoliberal idolization of consumer free choice in the credit industry traces its roots to federal credit legislation that began in the 1970s.
Part III criticizes two dominant legal proposals on the table: algorithmic input scrutiny and regulatory technology. Despite correctly identifying the source of algorithmic harm, such proposals do not interrogate the flawed assumptions of free markets and consumer autonomy. Their solutions tend not to venture beyond the classic neoliberal arguments for data transparency and consumer education.65 The incompleteness of these proposals often leads to wrongheaded solutions that ultimately reinforce unjust market relations.
Part IV proposes alternative pathways to build AI accountability. It lays out steps to reshape the presently unjust market relations of data production, circulation, and retainment through (1) reimagining the nature of data ownership, (2) creating a collective intellectual property right in data, and (3) building a collective data governance infrastructure anchored in the open digital commons.
I. Current Landscape of Algorithmic Exploitation
AI is transforming the field of consumer credit. Since the mid-2010s, AI has become exponentially more accessible, sophisticated, and commercializable.66 A 2018 Fannie Mae report found that 27% of mortgage originators used machine learning and artificial intelligence in their origination processes while 58% of mortgage originators expected to adopt the technology within two years.67 In a 2020 lender survey, approximately 88% of U.S. lenders reported that they planned to invest in AI applications for credit risk assessment.68 In the U.K., 72% of financial services firms use machine learning or some variation of AI in their businesses.69 With the release of advanced DL technologies in 2023—including Generative AI and large language models that utilize artificial neural networks—AI has become more deeply integrated into the consumer underwriting industry.70 Within this decade, it is exceedingly likely that AI credit underwriting will become the new market imperative.
The rapid adoption of AI in the credit market has spawned a range of responses. On one end of the spectrum, FinTech and banks have painted a rosy image. They argue that AI can help creditors revitalize so-called credit deserts by reaching the unbanked and underbanked.71 For them, AI’s ability to amass fringe data and gain insights about consumers’ market behavior presents a valuable business opportunity: creditors will be able to lend to consumers who were previously denied credit due to the lack of formalized credit information.72 In the meantime, markets will work on their own without government regulation. On the opposite end of the spectrum, regulators and consumer advocates have expressed concern that the unbridled use of AI can encroach data privacy and erode due process.73 As creditors delegate credit decisions to AI, the credit-underwriting process can become less transparent, which will make consumer litigation under the fair lending laws more difficult.74
The reality, however, is that both responses evade the root problem. FinTech and banks are wrong to assume that free markets will eliminate credit inequalities. Regulators and consumer advocates are right to worry about AI, but they have misdiagnosed the problem as the erosion of consumer autonomy and free choice. As this Part seeks to illustrate, the true source of algorithmic harm of AI credit-underwriting is unjust relations of data production, circulation, and control that dictate the outcome of AI’s knowledge discovery processes. It is harmful, not because it is more discriminatory or intrusive than credit decisions made by human loan officers, but because AI can direct creditors’ market power towards more exploitative domains of credit consumption through engineering price-signals and manufacturing consumer consent.75
How Does AI-Based Credit Underwriting Harm Consumers?
Algorithmic Decisional Harm
How does a lender’s use of advanced credit-underwriting algorithms generate risks of consumer exploitation? To thoroughly understand the current state of algorithmic exploitation, consider three scenarios:
Scenario A1: Suppose a creditor is seeking to expand its business into a new community. The creditor purchases from data brokers a right to access a private database containing vast volumes of alternative data regarding what people in the target community consume, purchase, desire, and browse online. This private database sources its data from a wide range of intermediaries that collect personal data from mobile apps, websites, tracking devices, and social media—and it happens to include data about me collected from my daily iPhone usage. To make sense of the information gathered from this private database, the creditor uses an advanced DL algorithm to summarize its patterns and generate predictions. With this data, the algorithm reveals that my family currently suffers from a short-term liquidity crisis because I have lost my manufacturing job. It also learns, from reading my search history, that I need quick cash to pay medical expenses for my uninsured family member. Based on this information, the algorithm can micro-target me with predatory advertisements and recommend a loan that could allow me to defer interest payments for the first month (but I will have to pay a higher compounding interest after the first month according to the terms of agreement). I accept the terms because I do not have alternatives.
Scenario A2: Suppose that, after one month, I am lucky enough to find a new job and my financial situation has improved. I am no longer in need of short-term loans, but I do not yet have enough cash to pay off the entire principal and interest accrued from my previous debt. Again, with the aid of a DL algorithm, the creditor can recommend a new package that allows me to further defer the interest, but under the condition that I borrow more. I end up accepting a combined loan package that is much more costly than others who are similarly situated.
Scenario A3: Now, suppose further that another individual from my community who has similar income levels, family obligations, savings, and consumption levels is looking for new sources of credit. Like me, she has low credit scores and struggled to obtain loans from large banks. Using the information harvested from me, the AI-informed lender can engage in the same pattern of microtargeting against her and trap her into a cycle of indebtedness.
What distinguishes these three scenarios? Scenario A1 exemplifies what economists identify as first-degree price discrimination (FDPD). FDPD occurs when businesses charge the maximum possible price for each unit of goods or services consumed by the consumer.76 Scenario A2 exemplifies what economists call second-degree price discrimination (SDPD). SDPD occurs when businesses charge different prices for different quantities consumed.77 Finally, Scenario A3 exemplifies third-degree price discrimination (TDPD). TDPD occurs when businesses charge different prices to different consumer groups.78 These three forms of price discrimination differ from each other in terms of the relationship and direction of exploitation between sellers and buyers in a market transaction.
Conventionally, FDPD, SDPD, and TDPD occur on separate domains. Standard economics textbooks generally characterize price discrimination as symptoms of market failure, caused by either the lack of competition or lack of informational transparency.79 FDPD (also known as perfect price discrimination) occurs due to informational asymmetries between creditors and consumers on a direct and discrete basis, which commonly manifests in the form of “take-it-or-leave-it” situations.80 SDPD (also known as nonlinear price discrimination) occurs due to the absence of consumer bargaining power and the inability to exit an exploitative business relationship with the creditor.81 TDPD (also known as market-wide price discrimination) occurs due to monopolies over a coveted resource or informational failures across similarly situated consumers who would have shared the same market preferences absent the monopoly.82 By definition, the three domains of price discrimination must remain separate because each domain correlates with a failure in a different market relationship.
However, in the age of AI, the three domains of price discrimination are no longer separate. Rather, these domains build on each other and intensify their exploitative effects. The AI-informed creditor’s microtargeting in Scenario A1 paved the foundations for further exploitation that occurred in Scenario A2. Using the same information extracted from the consumer, the creditor in Scenario A3 can now subject another consumer that is not within the privity of contract with the initial consumer to exploitative lending terms. The creditor’s use of AI for credit-underwriting allows each form of price discrimination to overlap; advanced AI models can use data garnered from one consumer to make predictions about other members of the consumer group based on classifications from the knowledge discovery process. Moreover, with the assistance of AI, creditors can more accurately target vulnerable consumers through scraping, processing, and analyzing mass volumes of consumer data obtained from data aggregators. AI drastically lowers the cost for creditors to engage in these three levels of price discrimination.
Algorithmic Informational Harm
In addition to causing decisional harms through price discrimination, AI-based credit underwriting can cause informational harms depending on how the AI model intakes data. Typically, consumers suffer two types of informational harm—(1) individual informational harm, which refers to “harm[s] that a data subject may incur from how information about [individuals] is collected, processed, or used,”83 and (2) social informational harm, which refers to the “harms that third-party individuals may incur when information about a data subject is collected, processed, or used.”84 To understand the two forms of informational harm, consider two scenarios:
Scenario B1: Suppose a FinTech lender uses an advanced DL algorithm to underwrite consumer credit and evaluate creditworthiness. The target borrower whom the lender seeks to evaluate does not have a FICO credit score. She also lacks any other formal credit history that is indicative of creditworthiness. In fact, the borrower belongs to an underrepresented minority group whose members historically had limited prior engagement with the formal banking system (i.e., credit invisible consumers). Undeterred by the lack of available credit information, the lender purchases a right to access a nonpublic database that sources data from people’s mobile apps, online subscriptions, browser history, social media, and other “fringe data.” The database includes the borrower’s sensitive personal medical information and records of hospital visits. The lender then instructs its DL algorithm to scrape data from the nonpublic database and trains the algorithm to make predictions about the borrower’s likelihood of default. Since the frequency of medical visits and the borrower’s condition is positively correlated with indebtedness, the algorithm gave the borrower a low hypothetical credit score and computed a rate of lending return based on that information. Without knowing this data, the FinTech lender used the algorithm’s results and offered the borrower a costly short-term loan with unfavorable rates based on the assumption that she is at a high risk of default. Here, the borrower suffered individual informational harm because her sensitive medical data was being used for a different, unrelated purpose that resulted in her getting a low hypothetical credit score.
Scenario B2: Suppose the same facts as above, except that the algorithm also scraped data from other people who are similarly situated as the initial borrower. After analyzing the profiles of 1,000 individuals, the algorithm finds out that a particular minority group disproportionately suffers from the same medical conditions as the initial borrower. In fact, people from the same cultural heritage who share the same dieting habits are 50% more likely to develop the medical condition than the population average. Defining this pattern as relevant information, the algorithm factors that disparity into its learning process. When the next borrower comes to the same lender and applies for a loan, the algorithm automatically computes a hypothetical credit score that takes the medical condition into consideration. Even though the algorithm did not make a prediction based on race, ethnicity, or religious classifications, the result has a disproportionate adverse impact on borrowers from the same group. Here, the new borrower suffered social informational harm because data harvested from a different individual was repackaged into new datapoints that were used against her.
While both harms can be caused by AI information-processing systems, the two differ in terms of the directionality of informational control which generate the harms. Individual harm is caused by situating consumers within highly monitored and engineered informational systems where owners/users of AI (creditors) exert vertical control over the circulation of data and the social relations of data production.85 Social harm is produced when owners/users of AI export individual harm to similarly-situated consumers outside the vertical information flow, thereby “amplify[ing] social processes of oppression along horizontal data relations.”86
Existing data privacy laws address some aspects of individual informational harm. Generally, individual informational harm is accounted for in laws governing: (1) consent-less data collection,87 (2) denial of informational access,88 (3) consent-less disclosure of personal data (i.e., data breaches),89 and (4) use of inaccurate information in credit reporting.90 But, under existing law, individual informational harm is redressable only if such harm constitutes a violation of some aspect of individual autonomy or dignity91––e.g., right to access, right to identification, right to be informed, right to withdraw consent, right to accurate information, and right to be forgotten.92 Under existing statutory and doctrinal frameworks, individual informational harms outside the domain of intrusions raise no cause of action.
For social information harms, redresses in existing legal regimes are entirely absent from the legal lexicon. No law in the U.S. has accepted a theory of data governance beyond the protection of individual autonomy or dignity. Even the European Union’s General Data Protection Regulation (GDPR)—supposedly the “strongest data privacy and security law in the world”93—fails to account for social informational harms resulting from unjust effects of data production, circulation, and retainment.94 In strengthening consumers’ control over the terms of data extraction and use, dignitarian data-governance regimes such as the GDPR seek to rebalance the power disparities between data-collectors (owners/users of AI) and data-subjects (consumers) within the vertical relations of informational control.95 But these regimes ultimately “fail to apprehend the structural conditions driving the behavior they aim to address.”96 As demonstrated in this section, even the most progressive dignitarian data governance systems to date are incomplete in their attempts to redress social informational harm.
How Is AI Changing the Credit Market for the Worse?
The Nature and Impact of Price/Consent Defects
This section examines the nature and impact of price engineering and consent manufacturing on consumers. It explains how consumers respond to price/consent defects from a socio-behavioral perspective and how this article’s characterization of consumer behavior departs from the neoliberal presumptions.
Within the classical neoliberal imaginary, consumer preferences are exogenous to market mechanisms.97 When prices are rigged—usually because of excessive social or governmental meddling (i.e., central planning)—consumers will refuse to transact on the market because the underlying goods and services do not match their range of price preferences.98 In the same vein, neoliberals imagine consent defects to be the result of consumers’ knowledge deficiency or inability to adequately communicate their (exogenous) preferences—i.e., inability to exercise their best interests—given the resources they own.99
From the neoliberal perspective, the problems of price-engineering and consent-manufacturing are results of imperfect markets and irrational market agents. Their solution, of course, is to restore perfect markets and rational agents.100 These problems fall squarely within the remedial zones of disclosure and fair lending. Once these institutions are in place, consumers will be able to vindicate their rights through private litigation.
But this characterization of consumer behavior is inaccurate. Consumer preferences are not exogenous to the market; they are shaped by market power and reflective of socialized choices.101 What consumers choose to purchase are reflections of how they would like to perceive themselves, how they would like to situate themselves in communities and social networks where they have standing, and what markets tell them about how consumption would help them achieve their goals.102 Broadly speaking, consumer preferences involve the values and tastes that shape people’s market activities—i.e., aspects of economic decision-making that the neoliberal assumptions of exogeneity and rational choice fail to explain.
What this means is that consumer preferences are not concrete, itemized, and preexisting desires that consumers carry to the market. Instead, consumer preferences are fluid, broad, and formed within the market’s allocative processes through consumers’ constant shopping activities or engagement with other market actors.103 Thus, neoliberals misunderstand the implications of price-engineering and consent-manufacturing.104 While neoliberals strive to minimize price-engineering and consent-manufacturing because they corrupt the neoliberal ideals of free markets and consumer autonomy (and therefore make deregulation more difficult to achieve), this article argues that price-engineering and consent-manufacturing justify a shift away from individualist solutions towards greater public regulation of the private markets.
Once we understand that consumer choices are socialized and embedded, it is not hard to see why the current system—built on the discourse of individual rights and the legal infrastructure of private litigation—fails to fulfill its promises of economic justice.105 No matter how exploited the consumers are or how widespread the exploitative practice, consumers whose preferences are formed by price/consent defects will not file a case to begin with. From a critical perspective, the legal and technical protocols originally designed to protect consumers are in fact hurdles obstructing consumers from achieving meaningful credit equality. The following paragraphs explore how the business applications of AI in credit underwriting are conducive to price-engineering and consent-manufacturing.
Price Engineering in AI-Mediated Credit Markets
There are several common misconceptions about what AI does to price-signals in credit markets. The first—and perhaps most popular—misconception relates to the nature of AI decision-making. According to the mainstream argument advanced by the first generation of algorithmic enthusiasts (and endorsed by FinTech and banks), AI improves the accuracy of credit risk predictions because it (1) is better at absorbing, processing, and analyzing large volumes of information than human decision-makers; and (2) acts upon such information without human biases. This translates into more accurate pricing of consumer credit risks and more optimal allocation of financial resources. The advantage of AI, the argument goes, is that it substitutes for biased human judgment.106 It concludes that AI’s “suppression of some aspect of the self, the countering of subjectivity” leads to more desirable market outcomes.107
But the mainstream argument suffers from a critical flaw: unlike what enthusiasts depict, AI makes decisions by replicating, rather than displacing, human bias. Recall that AI decisions are made through (1) scraping available individual/market-level information about their subjects, (2) repackaging scattered data into behavioral archetypes, (3) generating predictions about human behavior based on these constructed archetypes, and (4) adjusting predictions to reflect new informational intake.108 This process inevitably recycles past human prejudice and erroneous judgements into AI’s present and future predictions.109 For instance, data about consumers’ education level, incarceration history, and court records—i.e., outcomes of past societal disparities resulting from racial-class subjugation—are typically picked up by AI in the scraping process and repackaged into behavioral archetypes about the consumer’s behavior.110 Even pure economic data—e.g., consumer income, household indebtedness, and credit history—may reflect racial-class disparities, since minorities are more frequently targeted by predatory creditors.111 When these specific individual-level data are absent, AI fills in the blank using behavioral archetypes of other consumers from the same constructed group.112 Thus, credit pricing by AI is anything but value-neutral.
The second common misconception is that AI lowers the cost of lending and increases credit access. Advocates for de-regulating AI argue that the market adoption of AI has made the underwriting process more equitable and inclusive.113 They attempted to marshal empirical support, for example, from a National Bureau of Economic Research report indicating that “FinTech algorithms discriminate 40% less than face-to-face lenders”114 when it comes to mortgage prices.115 Another study, conducted by the Consumer Financial Protection Bureau (CFPB), indicates that creditors using AI approve 23–29% more loan applicants than creditors who purely rely on human judgment for their credit decisions.116 The same study also shows that AI lending lowers the annual average interest rates by 15–17% for approved loans.117
However, if we pay attention to other metrics, it becomes unclear whether the current uses of AI in lending meaningfully improve consumers’ access to equal credit. Using administrative data of 10 million U.S. mortgages originated between 2009 and 2016, Fuster et al. found that, while AI has indeed increased aggregate credit access and average loan acceptance rates, it also widened cross-group disparity: “[W]hile a large fraction of borrowers who belong to the majority group … experience lower estimated default propensities under the machine learning technology … these benefits do not accrue to some minority race and ethnic groups … to the same degree.”118 Even within racial minority groups, disparities in lending are discovered. Those who benefit from AI are disproportionately White-Hispanic and Asian. Amongst those who lose are non-White Hispanics.119
Thus, focusing on loan acceptance rates as the measurement for credit access obscures more than it illuminates. While AI does approve more loans than human loan officers, the data does not tell us about the quality and substance of the loans being approved. A more plausible explanation for the positive correlation between AI adoption and credit access is that AI helps creditors identify previously invisible profit-making opportunities. Since AI allows creditors to assess credit risks of consumers without the use of formalized credit information, it also enables them to reach the unbanked and underbanked communities.120 But, to compensate for the high risks of lending, in these “credit deserts,” creditors need to adjust the prices to match the risks if they hope to make a profit.121 To do this, creditors typically reduce the upfront prices of lending (to make them more accessible by the low-income) but increase prices on the backend—through deferred interest payments, buy-now-pay-later schemes,122 balloon payments,123 or negatively-amortizing interest rates.124 With the use of more sophisticated AI credit models, such as continuously-learning DL algorithms, creditors can more easily reap profits from low-income borrowers and extract rents by obscuring the actual costs of consumer financial products. Increasing credit access in this way will only widen the wealth gap and systemic credit inequalities. What the mainstream proposition omits, therefore, is the flipside of credit cheapness: low quality.
The third common misconception is that more data leads to more accurate algorithmic predictions. This claim builds on the techno-chauvinist assumption that greater informational intake necessarily produces more rational decisions.125 By implication, if an AI ever makes an “irrational” decision, such as discriminating against minority consumers in the credit underwriting process, then the problem must be inadequate or insufficient data inputs.126
But the reality is that more data can reinforce algorithmic biases. Even though AI’s information-retaining capacity and computing power are vastly superior to humans’, AI makes decisions by replicating the human decision-making structure. Contrary to the public imagination, AI doesn’t make use of every piece of data gathered.127 When AI receives new data in raw, scattered form, its first task is categorizing them into existing archetypes.128 Since AI is trained using data from the observable human environment, archetypes constructed by AI inevitably reflect the same biases that exist in the human environment.129
Contrary to the techno-chauvinist assumption, AI decisions tend to emulate pre-existing staple decisions—i.e., norms that can be summarized into statistical patterns.130 These staple decisions then form the basis of AI’s self-learning process—e.g., how it tunes its parameters to reflect new information, what weight it gives to each factor, and which data it determines to be distractive or noisy.131 By design, AI marginalizes any “splinter data” that cannot be mapped onto a pre-existing norm.132 This means that AI, like humans, can exhibit confirmation biases when fed too much information.
Nevertheless, the fallacy of “more-data-means-better-outcomes” runs deep in the credit industry. The idolization of informational quantity has largely fueled the movement within the credit industry to expand the use of alternative “fringe” data. This wave began with FinTech’s push for “big data” analytics in the personal loan and small-business credit underwriting space. In 2012, a Los Angeles-headquartered start-up, ZestFinance (now “Zest AI”), became the first company to combine “machine learning style techniques and data analysis with traditional credit scoring.”133 ZestFinance’s marketing strategy emphasized AI as a solution to the persistent problem of credit invisibility in low-income communities.134 It framed its approach as using “all data as credit data.”135 By 2022, alternative data usage had become widespread.136
Piercing through the rosy image painted by ZestFinance, the reality is that proxy discrimination is ingrained in each step of AI’s analysis.137 ZestFinance’s AI model takes into consideration data that “appear to have little connection with creditworthiness.”138 For example, the AI model measures “how responsible a loan applicant is” by analyzing the speed she “scrolls through an online terms-and-conditions disclosure.”139 The number of social media connections a person has, the frequency that she deactivates an account, and the number of connections she unfriends are also used as proxies to measure risk-taking tendencies.140 The model also considers spending habits in the context of the loan applicant’s geographic location.141 For example, “paying half of one’s income [on rent] in an expensive city like San Francisco might be a sign of conventional spending, while paying the same amount in cheaper Fresno could indicate profligacy.”142 These proxies were not inserted by their human programmers—they were generated automatically via algorithmic knowledge discovery processes that merely seek to model and replicate human decision-making.143
In a nutshell, all three common misconceptions stem from a misunderstanding of how AI works in credit markets. These misconceptions are rooted in the belief that AI is fundamentally different from human intelligence and exogenous to the human environment. Yet, as the foregoing paragraphs demonstrate, these assertions cannot be further from the truth. In making predictions about human behavior and acting upon them, AI embeds, repackages, and reifies the very inequalities found in the human world. But AI also goes one step further: AI amplifies these biases by building on each other’s biases.144 Once an AI model computes a result and wraps it in the form of packaged data, such data then enters the stream of market data that is constantly being scraped and analyzed by other AI models.145 In this digital ecosystem where data is incessantly rinsed and remade, price-signals reflect the aggregate biases of the market rather than the inherent value of goods and services being transacted.
Consent Manufacturing as Information Control
Consent manufacturing is not new. It is part and parcel of the market’s disciplinary power to manipulate consumers into buying what they do not need. It is also integral to the state’s propaganda power to mobilize citizens into acting against their self-interests and serving the elite consensus.146 Its origins and manifestations are well documented in Edward Herman and Noam Chomsky’s seminal work, Manufacturing Consent. Since its coinage, the term consent-manufacturing has been amply applied to studies of social media, the internet of things, and other engineered information environments.147
Like mass communications technologies, AI ushered in an era of unprecedented suppression of the self via creating a chronic “reliance on market forces, internalized assumptions, and self-censorship, and without overt coercion.”148 This interweaving web of suppressive forces is reinforced by both the culture of neoliberal individualism149 and the material conditions of market dependency.150 It exists in all informational systems operating under capitalist logic, whether undergirded by old or new technologies.151 Here, what distinguishes AI’s suppression from that of mass communications is the form of control and the impact it has on the lives of those subject to the suppression.
In the credit market, AI manufactures consumer consent through two distinctive yet mutually-reinforcing pathways: (1) creation of personalized information silos designed to control and reset expectations of consumers within the immediate zone of the credit transaction; and (2) production of generalized knowledge about group consumption behaviors designed to manipulate prospective consumers and those who are nonparties to the credit transaction.152 Whereas the first pathway concerns the control over vertical data flows between consumers and creditors, the second concerns the control of horizontal data flows between consumer peers by creditors.153
In the first pathway, AI creates a system of self-hallucination through harvesting consumer data to learn about the consumers’ behavioral proclivities while simultaneously reshaping consumer expectations by pressing their cognitive weak spots. Within this system, consumers are ceaselessly inundated with information nudging them to choose credit products that are more exploitative and profitable for the creditor. The classic example is data aggregation in payday lending. Payday loans notoriously attract low-income, low-savings, and socially desperate consumers because they do not require credit scores or other formal credit history from the loan applicant.154 Such loans tend to have high backend costs (albeit with low entry prices) that can trap borrowers into persistent indebtedness.155 With the use of AI, payday lenders can more accurately seek out situationally precarious consumers and those who have tendencies to reborrow at high costs with very little information about any individual consumer.156 In the process of learning about the consumers’ needs, inclinations, and predispositions, the AI mixes and matches price terms in ways that consumers will most likely accept. AI can also design the optimal payday loan structure that attracts consumers who do not need or would not have otherwise applied for the loan.157 Here, the role of AI is to augment the power of creditors over consumers—via giving creditors the control over vertical flows of data between the creditor and the consumer.
In the second pathway, AI creates an ecosystem of peer-hallucination via aggregation of data from a particular consumer group and using it to shape the expectations of prospective consumers who are not a party to the credit transaction. This ecosystem undercuts consumer power on two parallel dimensions.
First, as between consumers, AI creates a horizontal system of norm-convergence whereby consumers in the same affiliated groups and their proximate social networks are exposed to the same expectations. For instance, when consumer A0 applies for a loan underwritten by AI, those within the same group—consumers A1 and A2—will be exposed to similar expectations as A0 when they apply for a loan.158 If A0’s consumer expectations are skewed by processes of self-hallucination, A1 and A2 will most likely experience the same effect. This is because the nature of AI—and especially for DL algorithms—is that it “can be used to know things about [A1] that [A1] does not know [about herself], by referring back to [A1] from [A0].”159 And, to the extent that certain aspects of group An intersect with group Bn, “data from An can be used to train models that ‘know’ things about Bn, a population that may not be in any vertical relation with the system’s owner.”160
Second, as between creditors, AI generates data flows between users of AI engaged in the same underwriting practice. It creates a two-tiered digital environment: on the one hand, creditors can share information they collect about the consumers in a networked environment constructed by AI. On the other hand, consumers who are subjects of data scraping are isolated and kept mostly in the dark about what information they generate. Like in the payday lending industry, the “data of those who have applied for a loan can be shared among lenders for retargeting.”161 Payday lenders can use horizontal behavioral insights about the consumer to target entire communities and trap repeat borrowers into unending cycles of indebtedness. Here, the role of AI is to sever direct horizontal ties between consumers while granting creditors visibility and control over the horizontal flow of consumer data.
Through the interplay of self/peer-hallucinating forms of consent-manufacturing, AI creates a digital environment where consumers are turned into data-producing machines—churning out new data each time they participate in the digital economy. Within this constructed environment, consumers are incessantly generating new marketable data through their routine engagement with the credit system. Data extracted from consumers’ everyday life are split apart, atomized, and reassembled into market price-signals; the price-signals are then re-consumed by consumers and turned into new data—a cycle of digital cannibalization.162 In this system, consumers become part of the products that they ultimately consume.
II. Neoliberal Roots of Consumer Financial Protection
This Part unearths the history of how the neoliberal ideals of free markets and consumer autonomy became entangled with the current normative paradigm of consumer financial protection. In doing so, this Part shows that neoliberal ideals are not timeless tenets of economic justice. Rather, they are products of congressional politics that served one particular historical purpose—to legitimate the federal government’s divestiture from public welfare and incorporate minorities into the free-market capitalist status quo. As such, this Part delegitimizes the dominant normative justification for delegating public solutions to credit inequality to the private markets.
Since the late-1960s, Congress has enacted a series of consumer financial protection laws163—e.g., FHA, ECOA, TILA, FCRA—to bolster consumer autonomy and facilitate competitive, transparent, and equitable markets for credit provision.164 Enacted at the height of the civil rights movement, these laws used credit access as a means to solve race-based economic inequality and placate social unrest.165 Yet, as the federal government gradually aligned itself with neoliberalism beginning in the mid-to-late 1970s, the civil rights notion of equal credit access merged with the individualist, laissez-faire ideology that saw market freedom as a panacea to poverty.166 This merger became a bipartisan consensus that guided almost all significant federal regulatory responses to credit inequality, giving rise to the belief that credit inequality can largely be resolved by maintaining efficient markets and race-and-gender-neutrality.167
As the following sections aim to demonstrate, our existing consumer financial protection regime, informed by neoliberal individualism, is ill-equipped to address the novel threats of algorithmic harm because it overly fixates on the protection of private rights. Despite Congress’s intention to eradicate systemic credit inequality, these laws have had limited impact in protecting consumers. The failures of the contemporary consumer financial protection regime trace their origins to historical path-dependencies set in the 1970s.
How Neoliberalism Became Entrenched in Credit Regulation
The Pre-Neoliberal History of Congressional Credit Legislation
Before the late-1960s, credit was in congressionally uncharted waters, and instead governed by a fractured regime of state laws, industry norms, and banking customs.168 State law only regulated loan size and usury limits,169 but left “the decision as to whom credit should be granted” to creditors.170 The dominant practice among creditors in the 1960s was to consider the “three C’s of credit”: the character, capacity, and capital of the credit applicant.171 A popular credit underwriting manual in 1961 instructed creditors to label divorcees, indigenous peoples, and those living in “untidy homes” or “rundown neighborhood[s]” as having high credit risks.172 The Federal Trade Commission (FTC)’s 1970 study of major lending companies found collecting racial information a standard practice.173 In essence, credit underwriting in this era was done informally as a “relationship business” anchored in social networks, which enabled animus and bias to escape government detection.174
When Congress initially contemplated federal credit reporting and fair lending legislation in 1968, it confronted a vibrant yet unequal landscape of credit provision. For the white American working class, credit had become cheap and abundant. On the demand side, the stagnation of wages and inflation in the 1970s drove up the cost of living, turning debt-based consumption into a market imperative;175 credit became necessary for anyone hoping to purchase essential goods and services.176 Consequently, banks had to increase their credit supply. By the mid-decade, credit had “ceased to be a luxury item.”177 These institutional changes in credit provision made borrowing an essential component of the everyday consumer experience in white working-class America.
But this expansion of credit was also unequal: the 1970s marked the emergence of a credit apartheid that segregated the American consumer population. The rise of banking made borrowing easy for the suburban white middle class, but not for African Americans who made up a large portion of the urban poor.178 For them, credit was scarce and unavailable.179 Congress found the unequal access to credit to be among the leading causes for social unrest amongst the urban poor.180 In a hearing before the Senate Committee on Banking and Currency, the FTC testified that credit unavailability was the cause of economic desperation of the urban poor.181 By the mid-70s, credit inequality had become an urgent issue of social stability that Congress could not afford to ignore.
Responding to gaping credit inequality and unrest, Congress enacted the first comprehensive fair lending law: the ECOA.182 The ECOA saw the use of any racial or gender information in credit underwriting as an infringement on the individual’s exercise of free choice and economic opportunity.183 Race-and-gender neutrality and individualism were the bedrocks of fair lending protection. The House Committee on Banking, Currency, and Housing, quoting the U.S. Commission on Civil Rights, explained:
It would be difficult to exaggerate the role of credit in our society. Credit is involved in [an] endless variety of transactions reaching from the medical delivery of the newborn to the rituals associated with the burial of the dead. The availability of credit often determines an individual’s effective range of social choice and influences such basic life matters as selection of occupation and housing. Indeed, the availability of credit has a profound impact on an individual’s ability to exercise the substantive civil rights guaranteed by the Constitution.184
This notion—that unrestrained credit access undergirds consumer autonomy—embodied the consensus that Congress reached after a decade-long ordeal to grapple with entrenched credit inequality.185
Despite Congress’s good intentions, the passage of ECOA produced unintended consequences. Specifically, Congress’s reimagining of credit as a vehicle for individual social choice legitimized the federal government’s later divestiture from social welfare, which began with the government’s delegation of poverty reduction to private credit-underwriting institutions in the early 70s.186 Credit was reframed as the “private-sector alternative to the welfare state.”187 Moreover, recasting credit access as a precondition for the meaningful exercise of civil rights redirected the focus of credit access from redressing systemic racial-gender inequalities to incorporating minorities into the free-market status quo.188 As the next section will illustrate in further detail, these congressional endeavors paved the groundwork for the modern neoliberal consumer protection regime.
Displacement of Public Regulation by Private Enforcement
The rise of individualism and neutrality had profoundly impacted legislative responses to credit inequality since the mid-70s—they directed the focus of credit legislation to expanding the scope of creditor liability and access to banking services. For instance, subsequent amendments to ECOA almost exclusively revolved around adding new categories to the list of protected characteristics, bolstering consumers’ procedural rights, and adjusting the creditors’ disclosure obligations. The 1976 amendment added race, age, color, religion, national origin, and the collection of public assistance income to the original categories of sex and marital status as criteria prohibited from consideration in the credit underwriting process.189 The 1988 amendment imposed additional disclosure obligations on creditors to (1) give formal written notice to applicants of business credit about reasons of credit denial and (2) retain records for business credit applications for at least a year.190 The 1991 amendment heightened creditors’ disclosure obligations regarding residential mortgage lending.191 The 2003 revision to Regulation B, which implements ECOA, imposed an “adverse action” notice192 requirement on creditors to deliver written explanations to consumers when they make any credit decisions adversely affecting consumers’ rights under ECOA.193 Similarly, amendments to FHA in 1974, 1988, and 1996 mostly centered on heightening creditors’ disclosure obligations and consumers’ procedural rights—changes that largely mirrored amendments to ECOA.194
One reason for the growing legislative emphasis on disclosure and formal equality is that Congress increasingly pushed for private litigation as the principal means to vindicate consumers’ rights under the fair lending laws.195 When ECOA was originally legislated in 1974, Congress employed a dual enforcement model—allocating rulemaking power to the Federal Reserve Board (FRB) while delegating the power to bring enforcement actions to the FTC.196 But, beginning with the 1976 amendment, Congress gradually replaced the dual enforcement model with one that was centered on civil lawsuits.197 Subsequent amendments raised the punitive damage ceiling but further constrained the agencies’ substantive rulemaking power. While agencies were granted discretion to implement procedural safeguards protecting consumers’ right to know and creditors’ duty to inform, their authority to craft rules identifying and prohibiting new harmful lending practices shrunk dramatically from 1976 to the 2000s.198 Together, these legislative changes were designed to elevate private enforcement and relegate public enforcement to a secondary role.
However, despite the dominance of the individual rights model, empirics on private enforcement show that consumer welfare did not meaningfully improve in the decades that followed the ECOA’s enactment. Although Congress intended for private lawsuits to be the cornerstone of enforcement, the fair lending laws spawned surprisingly little litigation. For a statute promising to eradicate credit discrimination, the ECOA invited fewer than 50 cases in the decade after its enactment199-—fewer than the number of cases brought under the TILA per month during a similar period200––and far fewer than the number of employment discrimination cases filed per week under Title VII.201 This individualist regime had exacerbated credit inequality since it also amputated agencies’ substantive rulemaking power.
Ironically, an individual rights model centering on private enforcement ended up hurting individual consumers. The most critical failures of this regime are twofold.
First, the legislative emphasis on disclosure and formal equality marginalized questions about bargaining power disparity—i.e., the most central causes of transactional inequality in credit markets. This problem permeates most federal consumer financial protection laws. Under the TILA, for instance, a creditor’s good faith compliance with proper underwriting procedures and standardized disclosure forms immunizes her from liability.202 Under the ECOA, a creditor is deemed compliant with her notice obligations as long as she clearly explains reasons for denying the consumer’s credit application and demonstrates that race or gender play no part in the creditor’s decision-making.203 Under the existing individual rights regime, a consumer’s consent—even constructive consent upon sufficient disclosure—to a loan makes her responsible for the underlying consequences (including wage garnishment and collateral-repossession following an event of default).204 It matters not that she is desperate, materially deprived, lacks a viable alternative, or fell prey to exploitative terms.205
Second, a private-enforcement regime shifts the cost of compliance from creditors and regulators to consumers. Whoever contests the fairness of a transaction bears the legal costs and evidentiary/pleading burdens. Additionally, unsuccessful credit applicants are reluctant to assert their rights against creditors, large or small, out of fear of the institutions, of reprisal, and of the risks associated with alienating creditors.206 Therefore, the irony of private enforcement is that the poorest and most precarious consumers—e.g., minorities, women, immigrants, and other status-subordinated people who are most in need of protection—are typically the ones who are barred from asserting their interests in the current legal regime.207
Contemporary Neoliberal Legal Response to Credit Inequality
At its core, the contemporary neoliberal legal paradigm can be characterized as a series of commitments to the individual rights model, implemented by statutes protecting the autonomy of markets and delegating public functions to private enforcement.208 Today, these commitments have coalesced into a consistent regulatory methodology, consisting of two components: (1) elevating cost-benefit analysis above other modes of policy inquiry;209 and (2) conditioning substantive regulation upon a finding of “market failure.”210 No matter what type of credit is being regulated, how it injures consumers, or where the locus of harm lies, regulators would follow these two methods drawn straight out of the neoliberal handbook. The following paragraphs explain the logic of each method and their legal manifestations.
Elevating Cost-Benefit Analysis Above Other Inquiries
Cost-benefit analysis concerns how regulators should exercise their discretion in crafting rules to address social and economic harms in markets.211
Neoliberals prefer cost-benefit analysis to other modes of regulatory inquiry because they see it as value-neutral and derived from the unbiased analysis of market data—i.e., data produced by optimal and self-correcting market processes that are dis-embedded from extrinsic social or governmental influences.212 While the proliferation of cost-benefit analysis in policy-making and judicial review has no doubt revolutionized the administrative process by eliminating arbitrary agency actions, it has also substantially restrained the federal bureaucracy’s power to enforce established congressional public policies.213
What is critical about the neoliberal transformation is that it elevated cost-benefit analysis to the exclusion of other modes of policy inquiry—by promising to be dis-embedded, value-neutral, and untainted by political influence.214 Policies premised on the radical redistribution of wealth and reconfiguration of market power are dismissed as advancing a subversive ideological agenda.215 The elevation of cost-benefit analysis also made the presumption of free and neutral markets uncontestable in the lawmaking and policymaking forums.216
But, despite its façade of neutrality, cost-benefit analysis is value-laden and ideologically-driven. For one, numbers and statistics are highly susceptible to manipulation.217 What goes into the baseline, denominators, and benchmarks of empirical comparison are conscious political choices about who can and cannot be counted as subjects of policy inquiry. Yet, framing these conscious choices as neutral reflections of market conditions obscures the power relations that dictate what goes into the analysis.218
In the field of consumer credit, the hegemony of cost-benefit analysis is most saliently manifested in two legal standards codified in the core consumer financial protection statutes: (1) legal thresholds of recovery conditioned upon the balancing of interests between consumers and creditors that are inherently conflictual in the credit-underwriting process; and (2) judicial tests requiring agencies to show that the benefits of regulatory intervention outweigh the costs of disrupting the private ordering in markets.
The first—the balancing of consumer and creditor interests—is embedded in the very definition of discrimination in the credit inequality statutes.219 Under the classic definition of discrimination as disparate treatment, consumers seeking recovery are required to show that creditors undertook adverse credit actions against the consumers because of their protected characteristics (e.g., race, gender).220 Even under the more progressive definition of discrimination as disparate impact, plaintiffs cannot raise a cause of action if the creditors can demonstrate that the challenged practice is (1) “necessary to achieve one or more of the substantive, legitimate, nondiscriminatory goals” of the creditor; and (2) “those [legitimate] interests could not be served by another practice that has a less discriminatory effect.”221
The second—the balancing of regulatory benefits and market costs—finds legal expression in statutory provisions governing the scope of federal agencies’ substantive rulemaking power. The Dodd-Frank Act restrains the CFPB’s enforcement power to identify and prohibit “unfair” credit practices by conditioning regulatory action upon a finding of (1) substantial consumer injury; (2) such injury is not reasonably avoidable by consumers; and (3) the regulatory benefits are not outweighed by the costs to the market.222 Similarly, the FTC’s “unfairness” power to govern credit provisions is also constrained by a three-prong countervailing benefits test that requires the Commission to balance any regulatory gains from agency action against the potential business losses of creditors.223
Like any legal tests anchored in cost-benefit analysis, these statutorily mandated countervailing benefits tests are not value-neutral. By tying the hands of federal agencies through the cost-benefit inquiry, Congress opened a narrow legal forum for organized business interests to impede or push back against progressive agency actions. In the fields of payday lending224 and mortgage lending,225 creditors have successfully defeated several of the agencies’ proposed rules to regulate “unfair” credit practices by exaggerating the market costs and diminishing the regulatory gains via manipulating the parameters of comparison. In judicial review of agency action, the banking industry has persuaded federal courts to overrule newly promulgated rules on the grounds that such agency actions exceeded their statutory authority by failing the cost-benefit analysis test.226 From the lens of neoliberal politics, thus, the elevation of cost-benefit analysis over other modes of policy inquiry created a route for organized business interests to propel deregulatory agendas and impede consumer protection programs. It also led to the “judicialization” of policymaking—i.e., the removal of important policy decisions on distributive trade-offs from domains “subject to open deliberation to arenas insulated from such deliberation through legal protocols and layers of protective rules about who may access the knowledge.”227
Conditioning Intervention Upon a Finding of Market Failure
Whereas cost-benefit analysis relates to the exercise of regulatory discretion, theories of market intervention concern the goal of consumer financial protection.
Over the past five decades, neoliberalism has transformed the goal of consumer protection from directly preventing consumer harm to removing constraints on consumers’ free choice to satisfy their preferences through markets.228 For neoliberals, the regulator’s job is simple: (1) to help consumers communicate their preferences in the market through the production of neutral price-signals, and (2) to ensure markets fulfill their intended functions of satisfying consumer preferences. If companies mess with the market’s price-signals, the argument goes, there will be a chain of harmful externalities that ripple through the dynamic and complex ecosystem of market agents who respond to the signal (e.g., creating arbitrage, inefficiencies, or deadweight losses).229 Thus, regulators should only intervene where market failures prevent markets from fulfilling their natural mandate. In doing so, regulators should only intervene to the degree necessary to rectify these failures.230 Under the market failure test, agencies that pursue aims beyond these two goals are not only abusing their discretion but also doing their jobs incorrectly.
Although the market failure test purports to constrain arbitrary and paternalistic agency actions, it ends up fetishizing an idealized notion of consumer choice. This ideology is most visible in two sets of rules which dictate when a federal agency can intervene to remediate harmful practices in consumer financial markets: (1) interpretative rules confining the agencies’ rulemaking power to merely correcting market failures; and (2) judicial doctrines invalidating agency actions that “misidentified” market failures.
One of the clearest examples of such fetishization is the FTC’s 1980 Policy Statement on Unfairness (hereafter the “Policy Statement”).231 A response to congressional worries of FTC’s “overregulation,” the Policy Statement established a three-prong standard232 to limit the FTC’s exercise of rulemaking power to prohibit “unfair” market practices under section 5 of the Federal Trade Commission Act (FTCA).233 In explaining the rationale for issuing the Policy Statement, the FTC stated:
Normally, we expect the marketplace to be self-correcting, and we rely on consumer choice—the ability of individual consumers to make their own private purchasing decisions without regulatory intervention—to govern the market. We anticipate that consumers will survey the available alternatives, choose those that are most desirable, and avoid those that are inadequate or unsatisfactory. However, it has long been recognized that certain types of sales techniques may prevent consumers from effectively making their own decisions, and that corrective action may then become necessary. Most of the Commission’s unfairness matters are brought under these circumstances. They are brought, not to second-guess the wisdom of particular consumer decisions, but rather to halt some form of seller behavior that unreasonably creates or takes advantage of an obstacle to the free exercise of consumer decision-making.234
Adopted amidst the height of a neoliberal takeover of Congress and the courts, the Policy Statement reflected a deep suspicion towards regulatory paternalism and an idolization of consumer free choice.235 These sentiments were also amply echoed by the prevalent legal scholarship of the time. For instance, the then-FTC Director of Policy Planning and later-U.S. Secretary of Labor, Robert Reich, wrote that a paternalistic approach to consumer protection is “fundamentally incompatible with the liberal assumption that each person is the best judge of his or her own needs.”236 “A consumer-protection rationale focusing on the likelihood that consumers within particular markets will misestimate physical or economic risks attendant upon their purchases,” Reich explained, “can provide a strong basis for government intervention, untainted by paternalism.”237 This growing suspicion towards regulatory paternalism, both in and outside of the administrative state, converged with the prevailing neoliberal paradigm of free-market fundamentalism that was advocated by the Chicago School of law and economics.238
In the early 2000s, the FTC’s modern theory of “market failure” emerged. In the 2003 annual Marketing and Public Policy Conference, the then-Director of the FTC’s Bureau of Consumer Protection J. Howard Beales delivered a public speech, stating that “[t]he primary purpose of the Commission’s modern unfairness authority continues to be to protect consumer sovereignty by attacking practices that impede consumers’ ability to make informed choices.”239 Central to the FTC’s new unfairness standard is the notion that free markets operate in the consumer’s best interests, making regulatory intervention appropriate only when there is a clearly identifiable “substantial consumer injury caused by [a] market failure.”240 Beales’ understanding reflects the neoliberal consensus that became widely shared by both academics and regulators by the 2000s: i.e., that the government should not disrupt the market’s private ordering absent the occurrence of a market failure. Throughout the FTC’s exercise of “unfairness” rulemaking powers, business associations and financial institutions frequently invoked the “market failure” notion to challenge the validity of FTC rules in court.241
Crucially, courts do not possess the full knowledge and expertise to determine questions of economic policy. But, by enabling courts to act as regulators and overturn agencies’ decision-making, the “market failure” test transferred vital questions of economic trade-offs in consumer protection from fields of open democratic deliberation to enclosed legal institutions—a domain gate-kept by a class of legal professionals and allied business elites.242 As such, questions of market failure evolved into resource contests over who can hire the most sophisticated expert witness. Oftentimes, litigation over the evidential sufficiency of market failure became legal battles between the agencies and the organized business interests. The voices of consumers and their advocates were either watered-down or absent.
In sum, neoliberalism has reshaped both the goal and the substance of consumer financial protection. Whatever consumer financial protection used to be, it is now principally concerned with the protection of free markets and consumer autonomy. In this neoliberal transformation, each branch of the federal government played complementary roles: Congress laid down the legal foundations by creating an individual rights model of credit regulation; the agencies tied their own hands by adopting the cost-benefit analysis and market failure test; the courts disciplined the agencies for venturing beyond the unspoken neoliberal norm via judicial review. Collectively, this system created a neoliberal consensus whereby all problems arising from the credit markets—whether results of individual conduct or social processes—were approached as if they were outcomes of individual choice. This system represents the institutional equilibrium that our lawmakers, judges, and regulators have found to entrench and stabilize business interests amidst the changing credit distribution landscape from the 1970s to the 2000s.
III. Beyond Neoliberalism: Critique of Current Proposals
This section focuses on the ways in which some of the most prevalent proposals for legal reform of credit underwriting on the table have ignored the relational aspects of algorithmic harm. With some variations, most proposals advocate for: (1) enabling regulatory inspection of algorithmic inputs used in AI credit models by means of mandatory disclosure, or (2) delegating regulatory burden to private markets through fostering technological entrepreneurship investing in the development of “RegTech” solutions.243
What these proposals have in common is treating algorithmic harm as outcomes of discrete individual acts, or practices of individual creditors, divorced from the context and social relations through which such harms are produced. While each proposal addresses a particular dimension of algorithmic injustice, none of them challenge the flawed assumptions of individual responsibility—a model of credit governance that has been deeply entrenched in the current regulatory consciousness since the 1970s. Existing proposals are, by and large, progenies of the neoliberal consensus. Most proposals continue to draw extensively from the neoliberal rulebook—that is, to restore perfect markets and rational market agents through disclosure and removal of choice constraints. These proposals see public regulation only as a compliment, rather than a supplement, to the market’s private ordering. But, as the following paragraphs will show, such efforts tend to miss the target because they fail to recognize that a significant portion of algorithmic harm is generated by unjust relations between creditors and consumers in AI-mediated markets.
The Futility of Algorithmic Input Scrutiny
The dominant approach to AI governance in consumer credit is to enhance regulatory visibility of how algorithmic inputs—i.e., raw consumer data—are processed by AI models in the credit underwriting processes. To implement this approach, proponents of input scrutiny argue that regulators should demand creditors and data aggregators disclose AI training data, computational formulas, and software source codes to federal agencies by means of regulatory fiat.244 Data transparency would help regulators better identify discriminatory practices, patterns, and hold creditors accountable under existing fair lending laws. In this regard, input scrutiny shares the same goals of most existing disclosure mandates: (1) enhancing price transparency;245 (2) facilitating informed consumer choice by creating the infrastructure for fair market competition and cost comparison;246 and (3) nudging consumer choice towards welfare-optimizing financial products.247 From the proponents’ point of view, the AI-mediated credit market is sufficiently opaque and unfair that even the most devout neoliberals should find the present conditions to be a “market failure,” justifying regulatory intervention.
The algorithmic input scrutiny proposal presents two obvious advantages. First, this approach can easily fit into the existing notice-and-consent frameworks of fair lending. For instance, under Regulation B (implementing the ECOA), creditors taking an adverse action against a loan applicant are required to deliver to the applicant a notification in writing containing “a statement of specific reasons” for the adverse action “within 30 days” after taking such action.248 If this notice requirement is not followed, the creditor is deemed to have violated ECOA (a strict liability regime). If implemented, the input scrutiny mandate may phase out the use of “black-box” AI models in lending decision-making.249 Creditors seeking to comply with ECOA’s adverse action notice requirements will be incentivized to adopt “white-box”250 AI models to underwrite consumer credit.251
Second, enhancing algorithmic input aligns with the current regulatory agenda to push for more individualist, dignitarian data privacy reforms. In March 2023, the CFPB promulgated a final rule252 to compel creditors to share with consumers any data they have collected about them.253 Any potential input scrutiny rulemaking can build on the existing legal infrastructure of financial data sharing.
Despite its alignment with existing regulatory agendas, the input scrutiny approach fails to meaningfully account for either informational or decisional harms stemming from unjust data relations. Its push for dignitarian reform distracts us from the real source of algorithmic harm, which lies in creditors’ informational control over horizontal and vertical data flows. If the material underpinnings of unjust data relations remain unchanged, it is questionable whether more data transparency could lead to meaningful consumer choice and autonomy.
The input scrutiny approach also fails to address the problem of AI proxy discrimination. Without race or gender inputs, the AI model can still engage in price discrimination because it draws indirect and unsupervised inferences based on engineered data and sources that reflect preexisting socioeconomic inequalities, which are embedded in the data used to train the algorithm.254 This occurs because AI makes decisions by replicating and reinforcing human bias.255 The AppleCard, for instance, recently drew intense criticism when a male applicant complained that he received a line of credit 20 times higher than that offered to his spouse, even though the two filed joint tax returns, lived in the same community, and owned the same property.256 Goldman Sachs, the issuer of AppleCard, responded to the complaint by stating that it could not discriminate against her because its algorithm “doesn’t even use gender as an input”257 Goldman’s response belies the reality that gender-blind algorithms can still be biased against women if they draw statistical inference from inputs that happen to correlate with gender, such as purchase history and credit utilization.258 Even though the New York State Department of Financial Services subsequently investigated Goldman’s credit card practices, it concluded that Goldman did not violate its fair lending obligations under ECOA because it “did not consider prohibited characteristics.”259 The AppleCard case challenges the notion that removing suspect algorithmic inputs indicating consumers’ protected characteristics can eliminate AI bias. More importantly, the failure of algorithmic input scrutiny to eliminate AI bias calls into question the effectiveness of the colorblind approach of the ECOA and FHA to equal credit access protection.260
The Illusory Promises of “RegTech”
The emergence of “RegTech”261—i.e., information technologies used by financial institutions to address the challenges posed by FinTech and ensure regulatory compliance—presents an alternative to the top-down regulatory initiatives discussed earlier. In general, RegTech encompasses a wide range of technological solutions, including those used to detect and prevent financial fraud, safeguard consumer data protection, optimize asset-liability management, monitor anti-money laundering, and automate tax/financial reporting.262
At its core, RegTech promises to safeguard equal credit access protection by tapping the strength of competitive financial markets to self-correct, adapt, and innovate.263 Proponents of RegTech argue that, by investing in informational technologies regulating AI, the market can solve its own problems through entrepreneurship and innovation—i.e., “pure” market processes untainted by regulatory paternalism. Proponents also envision RegTech to be the perfect solution to balance free markets against market-generated injustices, a pathway for financial institutions to redeem themselves. In the era of congressional gridlock and legislative inaction, RegTech presents an attractive “third way” that echoes with the existing cries for corporate social responsibility.264 Essentially, the RegTech proposal seeks to reinvent the neoliberal consensus through technology: financial institutions, by adopting RegTech to keep AI in check, can help the credit market cleanse its own imperfections through the private ordering.
But the promise of RegTech is illusory because, without changing the material conditions of exploitation that currently undergird unjust data relations, it is doubtful whether RegTech can meaningfully empower consumers against creditors. In fact, the opposite is more likely to be true. Currently, we are witnessing a wave of RegTech and FinTech acquisitions by some of the largest financial intermediaries. In June 2020, payments giant Mastercard acquired Finicity, one of the leading data aggregators in the U.S.265 Mastercard’s competitor, Visa, acquired Plaid, another leading data aggregator.266 Similarly, banks have also tried to control and internalize the process of data aggregation by pushing data aggregators to sign bilateral agreements governing their collection and transmission of consumer data from the banks’ platforms.267 As of September 2020, Wells Fargo signed seventeen such agreements with data aggregators, governing “ninety-nine percent of the information being collected from its platforms for use by other financial institutions.”268 What this means is that RegTech, like FinTech, will further empower creditors against consumers. With RegTech incorporated into creditors’ business model, creditors will effectively gain control of the entire data production process—including data aggregation, processing, distribution, and explanation.
RegTech therefore embodies a common symptom found in most neoliberal responses to social problems: subscription to the belief that the market is disconnected from social relations, and that technological problems in the market can be self-contained and resolved by technology alone. Proponents of RegTech have articulated a flawed vision of market internalism,269 that all problems stemming from the markets can be solved by the markets themselves. On a technical level, the RegTech movement has also embraced a similarly flawed vision on technology—that all problems stemming from technology are self-containable through the development of new technologies.270 But the RegTech movement has failed to realize that neither markets nor technologies can be dis-embedded from the social relations that constitute them. In ignoring the unjust social conditions giving rise to the problems that technologies were employed to solve, the RegTech and XAI movements have reframed the problem as outcomes of deviant individual conduct. As a result, the only viable solution they see is using technologies to discipline recalcitrant creditors, facilitate compliance, and then delegating the enforcement to the private markets. In this regard, RegTech has distracted us from the real sources of algorithmic harm—that is, unjust market relations of data production that enabled AI technologies to be used for commodification and exploitation.
IV. Towards Propertarian Reform: Alternative Pathways
So far, my analysis has largely centered on the dimensions of algorithmic exploitation in AI-mediated credit markets and how current proposals informed by neoliberal ideals fail to meaningfully address the risks of algorithmic exploitation. A lingering question is how to move forward.
As the last five decades of poverty intensification and systemic credit inequality have shown, neoliberalism has failed its promise of delivering meaningful equal credit access protection. The failures of neoliberalism are becoming even more salient today in the age of informational capitalism as AI has exposed the limits of free markets and consumer autonomy presumptions of regulation. To remediate these flaws, this Part explores possibilities of legal reform through (1) reimagining the nature of data ownership, (2) creating a collective intellectual property right in data, and (3) building a collective data governance infrastructure anchored in the open digital commons.
Why Collective Propertarian Data Governance?
By “propertarian reform,” I do not mean to limit the discussion to private property rights. Instead, I refer to a panoply of property-related reforms that vests legal entitlement in the ownership of things rather than of self. This includes variations of common property, such as common pool governance, collective property, and joint ownership. As Salomé Viljoen has pointed out, thinking of data governance only in narrow dichotomous terms—“propertarian” versus “dignitarian”—constrains our imagination of what is possible.271 The move to understand data in relational terms rejects the notion that individualist solutions are the only possibility for meaningful reform.
This article imagines collective data ownership as an alternative pathway to data governance. While individual data ownership helps rearrange unjust social relations of data production, circulation, and retainment within vertical systems of informational control, collective data ownership addresses horizontal relations.272 Collective data ownership also rebalances the power disparities between the owners/users of AI (creditors) and the subjects of AI (consumers) on both vertical and horizontal dimensions. Since data is the most valuable and vital input for AI systems, changing the legal foundations of data ownership will impact the occurrence of algorithmic informational and decisional harms.
In the context of consumer credit, granting consumers some form of property entitlement to the data can radically reshape existing relations of data aggregation and reorient the direction of power along the chains of data supply. For instance, if consumers are granted full property ownership over the data generated through their online activities—including the rights to possess, control, manage, use, enjoy, dispose, and sell273—then the data aggregators and brokers will need to purchase from consumers a right to access consumer data to conduct their business. Admittedly, full data ownership may have chilling effects on the speed and efficiency of data circulation since it breaks down existing economies of scale already formed between data aggregators and creditors, but full data ownership can also redirect power from creditors to consumers by incentivizing the market to invest in consumer-empowering FinTech and push data aggregators to disentangle with creditors. Even from a dignitarian standpoint, granting consumers a right to exclude others from accessing the data—anchored in the notion of personal dominion and sovereignty over things—can prevent the erosion of privacy and autonomy.274 A propertarian data governance reform that entirely transforms the material underpinnings of data production can protect consumer autonomy better than any neoliberal regulation.
Alternatively, formalizing a partial property ownership of data can also reshape data relations, albeit with less radical restructuring effects on the credit market. For example, conceptualizing data ownership as an asset or an entitlement to income can reduce consumers’ chronic dependence on unjust data relations to access the means of basic economic subsistence. Under an income-entitlement regime, data aggregators may not need explicit consumer consent to harvest data and sell them to creditors. But consumers will be entitled to a “data dividend” for the wealth generated from data usage.275 While this approach to propertarian data governance might not break up existing bonds between data aggregators and creditors, it can certainly provide a wealth cushion that helps alleviate the burdens of the low-income and reduce credit inequality.276
In contrast to an individualist or dignitarian approach, a propertarian approach to data governance reform can remediate unjust relations of data production and circulation—the root causes of algorithmic harm. Whether in full or partial form, formalizing a property right to data can provide consumers a means to regain control over the processes and fruits of AI’s atomization of consumer selfhood. However, to say that we should embrace a propertarian reform does not suggest that dignitarian interests in data are unimportant, or that individual rights do not matter. Individual autonomy, dignity, and integrity do matter—and, as the Introduction and Part I of this article have illustrated, they are embedded in the purpose of equal credit access protection. But a propertarian approach can protect these interests as well. A propertarian reform can also address systemic inequalities that have been ignored by the dignitarian approach for far too long.
Recommendations for Reshaping Unjust Data Relations
Of course, no legal reform is ever perfect—not even a radical restructuring of the market through consumer data ownership. While a propertarian framework for data governance can help us directly address the root causes of algorithmic harm in ways that no individualist or dignitarian regime can, it is important to recognize that there is no silver bullet to our present problems.277 Ultimately, whether or not we should opt for full or partial data ownership (and, in the event we opt for partial ownership, which sticks within the bundle of rights to prioritize) is a trade-off between thoroughness and administrability of legal reform that should be considered in light of the current social priorities. That trade-off should be a subject of democratic, public, and open deliberation—a policy choice that lies beyond the scope of this article. Nevertheless, there are concrete steps we can take to remove distractions obstructing our clear view of what is possible. The following paragraphs illuminate what a thorough propertarian reform to reshape unjust market relationships will likely require.
Reimagining the Nature of Data Ownership
Any propertarian reform must first address a threshold question: what does it mean to say someone owns data?278 Currently, several analogies are being deployed to make sense of data ownership: data as oil, as personhood, as salvage, and as labor.279 Each time a “data-as” analogy is proposed, the proponent is suggesting that data should be regulated the same way the other thing is currently governed. The logic of each “data-as” analogy is as follows: First, it makes an analytical claim about what makes data valuable. Second, by identifying what makes it valuable, the analogy makes a normative judgment about who should own the data. Third, to implement the normative ideal, the analogy makes a legal claim about what rights, duties, and powers should be established to buttress its particular vision of data ownership.280
(i) Data Is Not Oil: The most common legal analogy is that data is just like oil, or any depletable natural resource. This concept is popularized by British mathematician Clive Humby, who declared in 2006 that “data is the new oil.”281 What Humby meant is that data, like oil, is valueless and useless in its raw state; to generate value, data needs to be refined, processed, and turned into something else—the value of data lies in its potential.282 But “data-as-oil” fails as a legal analogy. Unlike oil, data can be infinitely supplied by its producers. It is continually updated by the consumer’s daily engagement with the credit system, whether directly (e.g., applying for loans) or indirectly (e.g., supplying credit information). In that sense, data is not like oil—oil is relatively scarce, fungible, and rivalrous in consumption; whereas data is abundant, non-fungible, and non-rivalrous.283 This challenges a central claim that many businesses have articulated in their legal battles to claim ownership of consumer data: i.e., that unprocessed data is merely raw material floating freely in the natural domain readily available for economic appropriation.
(ii) Data Is Not Personhood: A competing analogy, anchored in dignitarian concepts of personal sovereignty, sees data as imprints of human expression in cyberspace.284 Whereas “data-as-oil” views data as extracted from the natural domain, “data-as-personhood” views data as emanated from human subjectivity.285 Under this theory, data is an extension of the self, an aspect of individual integrity and autonomy that is immune from appropriation (or expropriation). This analogy encourages us to think of data as not being owned at all. It urges legislators and policymakers to completely de-commodify access to data and make it unavailable for all market actors. But this legal analogy is flawed for two reasons. First, the analogy conflates the purpose and outcome of individual expression. While it’s true that people express their personal desires, anxieties, thoughts, and lived experiences through communications in the digital medium, data is merely a byproduct of that expression. People do not engage with cyberspace for the purpose of producing data. Second, the analogy fails to recognize that people have more than a dignitarian interest in data. However uncomfortable it may be, data does have commercial value. If given the opportunity, many would trade their dignitarian interests for material benefit. Thus, the more sensible approach is to accommodate both dignitarian and propertarian interests by having consumers retain a portion of the wealth that is created through the commercialization of data.
(iii) Data Is Not Salvage: “Salvage” is defined as “a rescue of endangered property.”286 In maritime law, “salvage award” is a compensation for people who have rescued property that is lost at sea.287 In finance, “salvage value” describes the remaining value that someone should receive after disposing of an asset that has exhausted its useful life.288 What is common in both is the idea that whoever rescues an imperiled property from waste should be entitled to the value of the labor they have invested to save a property that would have perished but for the labor. In data governance, the analogy of “data-as-salvage” echoes with the sentiment that data miners and processors should be compensated for turning data into marketable outputs.289 However, this analogy is also flawed because it fails to recognize that data is collectively generated. There’s no doubt that data miners and processors have “mix[ed] their labor” in generating marketable data.290 But to say that data miners “saved” data from an “imperiled state” and turned them into something useful is to grossly overstate their contribution to data production. Let us not forget that each cog in the chain of data production—consumers, data aggregators, miners, distributors, and financial intermediaries—have materially contributed to the process. Remove any single actor from the chain, and data would not be marketable.
(iv) Data Is Not Labor: Among the pantheon of analogies, the “data-as-labor” analogy is the most promising. At its core, this analogy aims to distribute the fruits of data production according to the proportion of labor invested by each actor on the data production chain.291 Under this framework, consumers, data miners, and aggregators will each be entitled to compensation for the “wage labor” they invested in producing the data. This analogy strikes a balance between protecting both dignitarian and propertarian interests in data. It recognizes that, while people do express personhood value in the production of data, they will readily trade it for material benefit when given the opportunity. The “data-as-labor” analogy has also garnered much academic support. Glen Weyl and Eric Posner have introduced a proposal called Radical Markets, which “seeks to introduce a labor market for data.”292 In doing so, they aim to uproot the unjust foundations of data production, upon which the uncompensated fruits of “data laborers” are “distributed to a small number of wealthy savants rather than to the masses.”293 But there are still reasons to be skeptical of this analogy. First, if wage labor is equivalent to the value that each actor has invested in the production of data, then the distribution of wealth will be inherently unequal. Producers located on the lower-end of the data value chain (i.e., consumers responsible for data provision) will get minimally compensated, while producers located on the higher-end of the chain (i.e., data processors responsible for data repackaging and refinement) will retain most of the economic surplus. Second, “data-as-labor” does not account for market externalities. Crucially, markets and market prices are not neutral conduits for inherent value. While the market may be able to account for individualized value within the vertical relations of data production, it cannot account for the aggregate costs imposed on horizontal flows of data. The analogy’s key omission is assuming that markets are dis-embedded.
Creating a Collective Intellectual Property Right in Data
(i) Data as Collectively Generated Patterns: If data is not oil, personhood, salvage, or labor, then what is it? Mattias Risse conceptualizes data as collectively generated patterns.294 The idea is that the value of data “does not consist in individual items but in the emerging patterns.”295 Data is valuable not only for those who provide data within the vertical relations of data production, but also for people situated in horizontal relations of data flow, circulation, and distribution.296
The proposal that data consists of collectively generated patterns differs from other “data-as” proposals in that it is not an ontological claim about what data is or ought to be.297 It is a purely descriptive and pragmatic claim about how data currently fits into the existing “human practices of assigning commercial value to entities.”298 From a descriptive lens, data is a microcosm of vast social networks that are continually adapted, updated, and reflected by those who generate, use, and consume data for economic means.299 Thinking of data in relational rather than ontological terms helps us detect the blind spots of each aforementioned analogy.
From a legal standpoint, understanding data as collectively generated patterns opens new possibilities for restructuring the currently unjust data relations. If we accept the fluidity and amorphousness of data, then we can design a legal system that directly protects the data subjects’ (consumers and platform users) access and engagement with other sources of data production. Thinking of data in fluid terms thus enables us to formulate a collective property right in data deriving from the management of social relations. For instance, we can imagine a membership-based joint tenancy or co-ownership of data that places the onus of data management on the community. Another possibility is to grant consumers a right to access, control, and withdraw personal data from the digital commons, without granting a right to exclude. These propertarian reforms do not require analogizing data to already-existing things. Instead, it allows us to accept data as it is—that data is sui generis.
Here, it is important to note that the concept of collective property rights in data does not repudiate the notion that individuals have important dignitarian interests in data. But it does repudiate the idea that individual dignitarian interests in data are the only interests that matter to data governance law. It also rejects the notion that any interest in data is reducible to individual dignitarian interests. The fetishization of individualism, autonomy, and dignity is part and parcel of neoliberalism’s effect of reducing complex social problems into outcomes of individual choice, as well as neoliberalism’s legitimization of a systematic program of governmental divestment from public goods. By liberating ourselves from the intellectual constraints of neoliberalism, we can see new propertarian reforms for data governance and directly address the root causes of algorithmic harm.
(ii) Where Data Meets Intellectual Property: Once we recognize that the value of data lies in its circulation and compilation as collectively generated patterns, the next step is to conceptualize alternative forms of legal ownership to capture that value for the benefit of consumers. This is where data governance intersects with intellectual property (IP). Although conventional legal scholarship often associates IP with individualist propertarian solutions,300 this subsection investigates ways in which new developments in intellectual property rights (IPR) protection outside the U.S. can offer powerful insights for collectivist propertarian reform.301 Currently, copyright law protects data only in the narrow context of individual original authorship. In the U.S., copyright protection applies to data produced in connection with a creative activity or embedded in a creative expression.302 Raw data is uncopyrightable because courts consider them to be mere facts that are “discovered,” rather than “created,” under the existing copyright regime.303 Processed data, such as compilations of data in algorithmic or automatic databases, may be copyrightable as “literary works” under section 102 of the Copyright Act.304 Such data are copyrightable only if their arrangement or compilation is sufficiently creative that it amounts to original authorship.305
However, the problem with traditional IPR solutions is that they tend to reinforce, rather than redistribute, existing power inequalities in the value chain. Consumers have little control over the production and trade of consumer-generated data, despite being the ones who are subject to the information systems.
Fortunately, the U.S. can draw lessons from the legal experiments for database protection in other jurisdictions. For instance, the EU has created a sui generis legal protection for databases that are not covered by copyright.306 Protection under the EU sui generis database right is not contingent on originality, creativity, novelty, or even commercial value.307 Instead, any “maker” who takes the initiative to obtain, verify, or present the contents of the database and assumes its underlying risks is afforded property protection.308 Anyone who takes a “substantial investment” in the above can also become a rightsholder of the database.309 This broad definition of “maker” enables any collective or organization to claim direct or derivative rights in the database.
While the EU’s sui generis database right is certainly not perfect, the U.S. can learn from the EU’s successes and avoid its mistakes. To avoid the risks of over-protection impeding the free flow of data,310 the U.S. should create a two-tiered database protection system that distinguishes between original and derivative data compilations.311 For example, original databases could continue to be protected under the copyright regime, while derivative databases could be protected under the sui generis right. To ensure that the database does not devolve into a tragedy of the anti-commons, the sui generis database right should accompany legal mechanisms to ensure the free flow of information—such as restricting the sui generis database owner’s right to exclude while retaining their rights to enjoyment. Additionally, the legislation could set up sub-hierarchies of database rights within the sui generis legal conception by distinguishing between the “makers” of derivative data compilations and the rightsholders who merely “take substantial investment” in the preparation of derivative databases. These proposals are by no means exhaustive, but they can expand our imaginations of possible legal reform.
Building the Infrastructure for Open Digital Commons
This subsection considers what information infrastructures can be built to make the collective property right in data meaningfully enforceable.312 In line with existing legal scholarship on the digital public domain, this subsection considers the creation of a digital commons as the foundation for any meaningful exercise of non-exclusive right to access, use, and withdraw data.313 To implement this concept, this subsection illustrates steps to ensure that the digital commons remain open and common—meaning that it will neither regress into “tragedies of the commons”314 or devolve into “tragedies of the anti-commons.”315
(i) The Public Data Trust Option: To preserve the openness and commonality of the digital economy, it is necessary for us to resist and reverse the privatization of consumer data by creditors. One possibility is to develop an open database like the Human Genome Project.316 Another is to establish a national data trust for the public good, under the supervision of an independent public-data management authority.317 We can also draw inspiration from other countries. The UK and Canada explored national data trusts as a means to govern citizen data and regulate their access by businesses corporations.318 A public data trust would allow individuals, communities, and organizations to grant the rights of control and access their data to entrusted entities to manage their data for their benefit.319 This would turn data intermediaries into data fiduciaries—meaning that they would be subject to the heightened duties of data stewardship.
(ii) The Public Utilities Option: An alternative solution is to build on existing informational infrastructures of credit data collection and distribution. Three of the largest National Credit Reporting Agencies (NCRAs)—Equifax, TransUnion, and Experian—have already amassed vast volumes of consumer data for credit reporting.320 NCRAs have also developed extensive networks of data supply through business partnerships with FinTech companies and data aggregators.321 One possibility to create a collective propertarian data infrastructure is to regulate NCRAs as public utilities—the same way that natural gas, electric power, cable, telecommunications, and water companies are governed.322 In the common law tradition, courts have developed the public utility doctrine to ensure that industries providing goods and services essential to the public offer them “under rates and practices that [are] just, reasonable, and non-discriminatory.”323 Industries that qualify as public utilities typically meet two conditions: they are considered “natural monopolies”324 and are “affected with a public interest.”325 Today, NCRAs and other credit data platforms have already satisfied the two conditions that historically triggered a public utility recognition. As public utilities, they will have affirmative obligations to the public to provide open data access, non-discrimination, and universal service. This “ensure[s] collective, social control over vital private industries that provide[] foundational goods and services on which the rest of the society depends.”326
(iii) Collective Social Governance of Data: Whether we select the public trust or the public utilities option, governing data as open commons invites an additional challenge: how do we ensure data is made as openly accessible as possible, while still limiting access to data with the potential to do harm? Admittedly, not all data is appropriate for open public access.327 Restriction is warranted for data that contain sensitive personal information or otherwise carry potential for intentional or accidental misuse.328 Leakage of certain data can also pose security risks.
Establishing a legal infrastructure for the collective social governance of data can remediate unjust data relations without compromising people’s privacy and security interests in data. One way to achieve this is to simultaneously vest the power of data management in the hands of consumer communities, while granting data access to an independent, entrusted entity acting under public interest.329 Currently, the EU has considered a similar proposal that would allow public authorities to access data where doing so is “in the ‘general interest’ and would considerably improve the functioning of the public sector.”330 This proposal follows the logic of the 2016 French Digital Republic Act.331 In the U.S., statistical agencies, census bureaus, and the Library of Congress have also established professional expertise in managing data for the public good while adhering to strict public-purpose limitations and high confidentiality standards.332 These existing forms of public data management systems can serve as a model for collective social data governance.
Conclusion
Over the past half century, neoliberalism has entrenched a regulatory paradigm that saw social problems as outcomes of individual choice. This paradigm saw free markets and consumer autonomy as the panacea to market injustices. The twin ideals of neoliberalism find ubiquitous presence in our laws governing the supply and distribution of credit. Instead of providing meaningful credit access and equality, they have distracted us from the root problems: unjust market relations stemming from systemic social inequalities.
If the failures of neoliberal ideals of free markets and consumer autonomy were once hidden, then the ascendancy of AI made them apparent. AI situates the vast majority of consumers within systems of informational control where market price-signals are engineered and consent is manufactured. Within these digital environments, consumer data is ceaselessly harvested, extracted, refined, and repackaged into marketable products. This also causes the exploitation of consumers through microtargeting and price discrimination. Yet, existing proposals for AI governance, informed by neoliberalism, have continued to cast these problems as outcomes of imperfect markets and individual choice. They obscure the true source of algorithmic harm—unjust market relations of data production, circulation, and control that entrench and reproduce systemic inequalities.
Moving beyond neoliberalism, recognizing algorithmic harm as both individually and socially constituted can help us imagine new possibilities to address the root causes of systemic credit inequality. A purely dignitarian reform of data governance which addresses only individual harm is bound to be incomplete. To fundamentally reshape the unjust social relations that currently underpin AI exploitation and build a just credit market, we need to push for a collective propertarian reform. To strive for this possibility, we must reimagine the nature of data ownership as collectively generated and relational, conceptualize a collective IPR in data, and construct an alternative information infrastructure to govern data as open commons.