Bias in Computer Systems

Bias in Computer Systems

BATYA FRIEDMAN

Colby College and The Mina Institute

and

HELEN NISSENBAUM

Princeton University

From an analysis of actual cases, three categories of bias in computer systems have been

developed: preexisting, technical, and emergent. Preexisting bias has its roots in social

institutions, practices, and attitudes. Technical bias arises from technical constraints or

considerations. Emergent bias arises in a context of use. Although others have pointed to bias

in particular computer systems and have noted the general problem, we know of no comparable

work that examines this phenomenon comprehensively and which offers a framework

for understanding and remedying it. We conclude by suggesting that freedom from bias should

be counted among the select set of criteria—including reliability, accuracy, and efficiency—

according to which the quality of systems in use in society should be judged.

Categories and Subject Descriptors: D.2.0 [Software]: Software Engineering; H.1.2 [Information

Systems]: User/Machine Systems; K.4.0 [Computers and Society]: General

General Terms: Design, Human Factors

Additional Key Words and Phrases: Bias, computer ethics, computers and society, design

methods, ethics, human values, standards, social computing, social impact, system design,

universal design, values

INTRODUCTION

To introduce what bias in computer systems might look like, consider the

case of computerized airline reservation systems, which are used widely by

travel agents to identify and reserve airline flights for their customers.

These reservation systems seem straightforward. When a travel agent

types in a customer’s travel requirements, the reservation system searches

This research was funded in part by the Clare Boothe Luce Foundation.

Earlier aspects of this work were presented at the 4S/EASST Conference, Goteborg, Sweden,

August 1992, and at InterCHI ’93, Amsterdam, April 1993. An earlier version of this article

appeared as Tech. Rep. CSLI-94-188, CSLI, Stanford University.

Authors’ addresses: B. Friedman, Department of Mathematics and Computer Science, Colby College,

Waterville, ME 04901; email: b_friedm@colby.edu; H. Nissenbaum, University Center for Human

Values, Marx Hall, Princeton University, Princeton, NJ 08544; email: helen@phoenix.princeton.edu.

Permission to make digital /hard copy of part or all of this work for personal or classroom use

is granted without fee provided that the copies are not made or distributed for profit or

commercial advantage, the copyright notice, the title of the publication, and its date appear,

and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to

republish, to post on servers, or to redistribute to lists, requires prior specific permission

and/ or a fee.

© 1996 ACM 1046-8188/96/0700–0330 $03.50

ACM Transactions on Information Systems, Vol. 14, No. 3, July 1996, Pages 330–347.

a database of flights and retrieves all reasonable flight options that meet or

come close to the customer’s requirements. These options then are ranked

according to various criteria, giving priority to nonstop flights, more direct

routes, and minimal total travel time. The ranked flight options are

displayed for the travel agent. In the 1980s, however, most of the airlines

brought before the Antitrust Division of the United States Justice Department

allegations of anticompetitive practices by American and United

Airlines whose reservation systems—Sabre and Apollo, respectively—dominated

the field. It was claimed, among other things, that the two reservations

systems are biased [Schrifin 1985].

One source of this alleged bias lies in Sabre’s and Apollo’s algorithms for

controlling search and display functions. In the algorithms, preference is

given to “on-line” flights, that is, flights with all segments on a single

carrier. Imagine, then, a traveler who originates in Phoenix and flies the

first segment of a round-trip overseas journey to London on American

Airlines, changing planes in New York. All other things being equal, the

British Airlines’ flight from New York to London would be ranked lower

than the American Airlines’ flight from New York to London even though in

both cases a traveler is similarly inconvenienced by changing planes and

checking through customs. Thus, the computer systems systematically

downgrade and, hence, are biased against international carriers who fly

few, if any, internal U.S. flights, and against internal carriers who do not

fly international flights [Fotos 1988; Ott 1988].

Critics also have been concerned with two other problems.