A vulnerability analysis: Theorising the impact of artificial intelligence decision-making processes on individuals, society and human diversity from a social justice perspective

(1)

Tilburg University

A vulnerability analysis

Krupiy, Tetyana

Published in:

Computer Law and Security Review

DOI:

10.1016/j.clsr.2020.105429

Publication date:

2020

Document Version

Publisher's PDF, also known as Version of record

Link to publication in Tilburg University Research Portal

Citation for published version (APA):

Krupiy, T. (2020). A vulnerability analysis: Theorising the impact of artificial intelligence decision-making

processes on individuals, society and human diversity from a social justice perspective. Computer Law and

Security Review, 38(September), 1-25. [105429]. https://doi.org/10.1016/j.clsr.2020.105429

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal

Take down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

(2)

Availableonlineatwww.sciencedirect.com

journalhomepage:www.elsevier.com/locate/CLSR

A

vulnerability

analysis:

Theorising

the

impact

of

artificial

intelligence

decision-making

processes

on

individuals,

society

and

human

diversity

from

a

social

justice

perspective

Tetyana

(Tanya)

Krupiy

TilburgUniversity,MontesquieuBuilding,Room808,Prof.Cobbenhagenlaan221,Tilburg,NorthBrabant,5037DE, theNetherlands

a

r

t

i

c

l

e

i

n

f

o

Keywords: Artificialintelligence Datascience Decision-makingprocess Socialjustice Humandiversity Vulnerabilitytheory Feminism

Queerlegaltheory Criticaldisabilitytheory

a

b

s

t

r

a

c

t

Thearticleexaminesanumberofwaysinwhichtheuseofartificialintelligence technolo-giestopredicttheperformanceofindividualsandtoreachdecisionsconcerningthe enti-tlementofindividualstopositivedecisionsimpactsindividualsandsociety.Itanalysesthe effectsusingasocialjusticelens.Particularattentionispaidtotheexperiencesof individ-ualswhohavehistoricallyexperienceddisadvantageanddiscrimination.Thearticleuses theuniversityadmissionsprocesswheretheuniversityutilisesafullyautomated decision-makingprocesstoevaluatethecapabilityorsuitabilityofthecandidateasacasestudy. Thearticlepositsthattheartificialintelligencedecision-makingprocessshouldbeviewed asaninstitutionthatreconfigurestherelationshipsbetweenindividuals,andbetween indi-vidualsandinstitutions.Artificialintelligencedecision-makingprocesseshaveinstitutional elementsembeddedwithinthemthatresultintheiroperationdisadvantaginggroupswho havehistoricallyexperienceddiscrimination.Dependingonthemannerinwhichan artifi-cialintelligencedecision-makingprocessisdesigned,itcanproducesolidarityor segrega-tionbetweengroupsinsociety.Thereisapotentialfortheoperationofartificialintelligence decision-makingprocessestofailtoreflectthelivedexperiencesofindividualsandasa re-sulttounderminetheprotectionofhumandiversity.Someoftheseeffectsarelinkedtothe creationofanableistcultureandtotheresurrectionofeugenics-typediscourses.Itis con-cludedthatoneofthecontextsinwhichhumanbeingsshouldreachdecisionsiswherethe decisioninvolvesrepresentingandevaluatingthecapabilitiesofanindividual.The legisla-tureshouldrespondaccordinglybyidentifyingcontextsinwhichitismandatorytoemploy humandecision-makersandbyenactingtherelevantlegislation.

E-mailaddress:t.krupiy@uvt.nl https://doi.org/10.1016/j.clsr.2020.105429

(3)

EricaCurtis,aformeradmissionsevaluatoratBrown Uni-versityintheUnitedStates,hasnotedthatsheevaluatedeach student’sapplicationconsisting ofstandardisedtestscores, thetranscript,thepersonalstatement,andmultiple supple-mentalessayswithinatwelve-minutetimeframe.1_Arguably,

thisisaveryshortperiodoftimewithinwhichanadmissions officercanevaluatetheapplicant’spersonalityandacademic qualitiesholistically.2_The_time_constraints_create_a_possibility

thattheadmissionsofficermayfailtodetecttheapplicants’ capabilitiesorhowsocietalbarriersdiminishedtheirabilityto realisetheirpotential.Anotherconcernwithhuman decision-making isthatthe decision-makerofficermay act arbitrar-ily inthecourseofexercising discretion3 _by_putting

differ-entweightoncomparableattributesthatcannotbemeasured. What ismore,anadmissions officercould treatapplicants on an unequal basisdue tobeing influenced byconscious or unconscious biases.4 _Advances _in _artificial _intelligence

(hereinafterAI)technologygiverisetoadiscussionwhether organisationsshoulduseAIsystemstoselectapplicantsfor admission touniversity.5 _Technology _companies_market _AI

systemswithacapabilitytopredict thecandidates’ perfor-manceandtofollowadecision-makingprocedureas possess-ing thecapacitytoeliminatebiasandtoimprove decision-making.6 _The_computer_science_community_is_now_working

onembeddingvalues,suchasfairness,intotheAI

decision-Acknowledgments:IwouldliketothankProfessorCorienPrins forherfeedbackonthedraftversionofthisarticle.Iamgratefulto AtienoSamandari,StuMarvel,ProfessorMarthaAlbertson Fine-man,ProfessorNicoleMorrisandProfessorPaulMyersfortheir feedbackonapresentationwhichformedthefoundationforthis article.Additionally,Iwishtothankscholarswhoasked stimulat-ingquestionsduringtheEthicsofDataScience:Addressingthe FutureUseandMisuseofOurDataConference,theBIASin Artifi-cialIntelligenceandNeuroscienceTransdisciplinaryConference, andtheMedia&Space:TheRegulationofDigitalPlatforms,New Media&TechnologiesSymposiumwhereIpresentedmyongoing work.

1_Joel _Butterly, _‘7 _Admissions _Officers _Share _the _Things

They Never Tell Applicants’ (Insider Inc., 2018) <https: //www.businessinsider.com/7-things-college- admissions-officers-wishevery-applicant-knew-2018-2?international= true&r=US&IR=T>accessed26June2019

2_Ibid

3_Mark _Bovens _and _Stavros _Zouridis, _‘From _Street-Level _to

System-LevelBureaucracies:HowInformationand Communica-tionTechnologyis TransformingAdministrative Discretionand Constitutional Control’ (2002) 62 Public Administration Review 174,181

4_Josh _Wood, _‘“The _Wolf _of _Racial _Bias": _the _Admissions

Lawsuit Rocking Harvard’ The Guardian (London 18 October 2018) <https://www.theguardian.com/education/2018/oct/18/ harvard-affirmative-action-trial-asian-american-students> accessed10March2019

5_Moritz_Hardt,_How_Big_Data_is_Unfair:_{Understanding}_Unintended

SourcesofUnfairnessinDataDrivenDecision-making(Medium Cor-poration2014)

6_Ekta_Dokania,_‘Can_AI_Help_Humans_Overcome_Bias?’_The

Seat-tle Globalist(Seattle 22May2019) <https://www.seattleglobalist. com/2019/05/22/can-ai-help-humans-overcome-bias/83957> ac-cessed3March2019

makingprocedure.7 _Daniel _Greene _and_colleagues _view _the

focusonachievingfairnessbyincorporatingvaluesintothe designofthesystemasshort-sighted.8_The_attention_on_how

toembedfairnessinto thedecision-making procedureofa technicalsystemside-linesthediscussionhowthe employ-mentofAIdecision-makingprocessesimpactsonachieving socialgoals,suchassocialjusticeand‘equitablehuman flour-ishing.’9_Virginia_Eubank’s_work_underscores_the_importance

ofinvestigatinghowtheuseofAIdecision-makingprocesses impactsindividualsandsociety.Herinterviewswithaffected individualswhoappliedtoaccessstatebenefitsinthestateof IndianaintheUnitedStates10_demonstrate_that_the

employ-mentofAIdecision-makingprocessescanleadtothe deepen-ingofinequality,11_to_social_sorting12_and_to_social_division.13

Theenquiryisparticularlypertinentgiventhefactthatnotall sourcesreportadverseoutcomes.TheBritishUniversitiesand CollegesAdmissionsServiceassertsthatinitspilotprojectan algorithmicprocessselectedthesamepoolofapplicantstobe admittedtouniversitiesasadmissionsofficers;the organisa-tiondidnotrevealthealgorithm’sdesignandoperation pro-cedure.14

Thepresent paper exploressome ofthe hitherto unre-solved longstanding societalproblems and newissues the employmentofAIdecision-makingprocessesraises.It con-tributestoexistingliteraturebyproposingthatanAI decision-makingprocessshouldbeunderstoodasaninstitution.TheAI decision-makingprocessreconfiguresrelationshipsbetween individualsaswellasbetweenindividuals andinstitutions. Thepaperexaminessomeofthevaluesandtypesof institu-tionalarrangementstheemploymentofAIdecision-making processesembedsintosociety.Thisissueissignificant.The CouncilofEuropeCommitteeofMinistersstatedthatwhen data-driven technologies operate ‘at scale’ their operation prioritisescertainvaluesoverothers.15 _The_assertion_of_the

CouncilofEuropeCommitteeofMinistersthatdata-driven technologiesreconfiguretheenvironmentinwhich individ-ualsprocessinformation16_should_be_extended_to_encompass

7 _Aditya_Krishna_Menon_and_Robert_C_Williamson,_‘The_Cost_of

FairnessinBinaryClassification’(2018)81ProceedingsofMachine LearningResearch1,10

8 _Daniel_Greene,_Anna_Lauren_Hoffman_and_Luke _Stark,_Better,

Nicer,Clearer,Fairer:ACriticalAssessmentoftheMovementfor Eth-icalArtificialIntelligenceandMachineLearning(TheProceedingsof the52ndHawaiiInternational ConferenceonSystem Sciences, Hawaii,2019)2122

9 _Ibid

10_Virginia_Eubanks,_Automating_Inequality:_How_High-tech_Tools

Pro-file,Police,andPunishthePoor(StMartin’sPress2018)10

11_Ibid₂₀₄ 12_Ibid₁₂₂ 13_Ibid_196-97

14_Ben_Jordan,_Minimising_the_Risks_of_Unconscious_Bias_in_University

Admissions:2017UpdateonProgress(UniversitiesandColleges Ad-missionsService2017)11

15_Council_of_Europe_Committee_of_Ministers,_{‘Declaration}_by_the

CommitteeofMinistersontheManipulativeCapabilitiesof Algo-rithmicProcessesDecl(13/02/2019)1’(1337thmeetingofthe Minis-ters’Deputies,CouncilofEurope2019)<https://search.coe.int/cm/ pages/result_details.aspx?objectid=090000168092dd4b>15 Febru-ary2019

(4)

therelationshipsindividualshavewitheachotherandwith theinstitutions.Thearticleexaminessomeofthetypesof so-cialtransformationsthattheuseofAIdecision-making pro-cesses acrossdomainswillaccentuate.Whilethedesignof AIdecision-making processeswillshape whethertheir op-erationgivesrisetosolidarityorsegregation,thereisa po-tentialforthesesystemstoadverselyaffectindividualswho have historically experienceddiscrimination, disadvantage, disempowermentandmarginalisation.Theprovisionsin in-ternationalhumanrightstreatiesprohibitingdiscrimination provideanon-exhaustivelistofindividualswhoexperience discrimination,exclusion,oppression,disempowermentand disadvantage.17_The_{characteristics}_such_individuals_possess

includesex,genderidentity,sexualorientation,age, ethnic-ity,race,colour,descent,language,religion,politicalorother opinion,nationalorsocialorigin,property,birthanddisability amongstothers.18

Theuniversityadmissionsprocessservesasacasestudy forcontextualisingthediscussioninthepresentpaper.One ofthereasonsforusingacasestudyforfocusingthe discus-sionisthatanevaluationofanytechnologyneedstobe con-textspecific.JaneBaileyandValerieSteevesobservethat tech-nologyisneithergoodnorbad.19_Everything_depends_on_how

developersdesignatechnology,howthelawregulatesitand whatvaluesthedevelopersembedintothetechnology.20_One

mayaddtothisobservationthathowindividualsusethe tech-nologyandforwhatpurposematterstoo.Clearly,itis possi-bletouseAItechnologytoadvancesocietalobjectives.Bruce DHaynesandSebastianBenthallproposethatcomputer sci-entistsshoulddevelopAIsystemsthatdetectracial segrega-tioninsociety.21_This_information_can_then_be_used_to_detect

similartreatmentofindividuals.22_Since_individuals_have

dis-parateopportunitiesasaresultoflivinginsegregatedareas withinthesamecity,23_the_use_of_AI_technologies_to_remedy

segregationwouldcontributetotheattainmentofsocial jus-tice.

Thisarticlefocusesonuncoveringanumberofadverse im-pactstheuseofanAIdecision-makingsystemislikelytohave bothonindividualsandsocietyfromtheperspectiveof

ad-17_Convention_for_the_Protection_of_Human_Rights _and

Funda-mentalFreedomsart14;AfricanCharteronHumanandPeoples’ Rightsart2;AmericanConventionofHumanRightsart1; Interna-tionalCovenantonCivilandPoliticalRights(adopted16 Decem-ber1966,enteredintoforce23March1976)999UNTS171art26; InternationalConventiononEconomic,SocialandCulturalRights (adopted16December1966,enteredintoforce3January1976)993 UNTS3art2.2

18_Ibid; _Convention _on _the _Rights _of _Persons_with _Disabilities

(adopted 13 December2006,entryinto force 3May 2008)2515 UNTS3art5(2);IdentobaandOthersvGeorgiaAppNo73235/12 (EC-tHR,12May2015),para96.

19_Jane_Bailey_and_Valerie_Steeves,_{‘Introduction:}_{Cyber-Utopia?}

GettingBeyondtheBinaryNotionofTechnologyasGoodorBad forGirls’inJaneBaileyandValerieSteeves(eds),eGirls,eCitizens: PuttingTechnology,TheoryandPolicyintoDialoguewithGirls’andYoung Women’sVoices(UniversityofOttawaPress2015)5

20_Ibid

21_Sebastian_Benthall_and_Bruce_D_Haynes,_Racial_Categories_in

Ma-chineLearning(AssociationforComputingMachinery2019)9

22_Ibid₈ 23_Ibid₇

vancingsocialjustice.Itisconfinedtoscrutinisingthe con-textwhereeducationalinstitutionsautomatetheprocessof theselection ofstudentsbyemployingAI decision-making processes.Suchcriteriacouldincludeperformanceon exami-nations,extra-curricularactivities,personalstatements, sam-plesofstudentworkandsoon.Whilethearticleuses exam-plesfromanumberofcountries,thefindingscanbeextended toalluniversitiesthatuseavarietyofcriteriatojudge the meritofindividuals.Theanalysisdoesnotincludewithinits scopeAIsystemsthatallocatestudentstouniversitiesbased onthestudents’preferencesforastudyprogrammewithout referencetothemeritcriteria.Anexampleoftheuniversity admissionsprocessesbeyondthescopeofthispaperisthatof theFrenchstateuniversitiesotherthangrandesécoles.24_The

algorithmallocatesplacesatFrenchstateuniversitiesto stu-dentsaccordingtothestudent’shighestpreferencefora pro-grammeandaccordingtowhetherastudentliveswithinthe districtwheretheuniversityislocated;arandomprocedureis usedtobreaktheties.25_For_reasons_of_space_it_is_beyond_the

scopeofthepresentenquirytoconsiderthebeneficialusesto whichavarietyofAItechnologiesmaybeput.

Section1maintainsthatitismoremeaningfultotalkof anAIdecision-making process ratherthan an AI decision-making system. It defines the elements comprising an AI decision-makingprocessforthepurposeofsituatingthe dis-cussion.Section2introducesMarthaAlbertsonFineman’s vul-nerabilitytheory26_as_a_theoretical_framework_for_examining

someofthewaysinwhichtheuseoftheAIdecision-making processeswillaffectindividualsandsocietyfromthe perspec-tiveofsocialjustice.Section3investigatessomeofthetypes ofvaluesthattheoperationofAIdecision-makingprocesses givesriseto.Thediscussiondrawsonthevulnerability the-orytoillustratesomeofthewaysinwhichtheseprocesses reconfiguresocialandinstitutionalrelationshipsinwhich in-dividualsareembedded.27_It_scrutinises_how_the_employment

ofAIdecision-makingprocessesimpactsonhowsociety un-derstandslivedhumanexperienceandhumandiversity.Itis concludedthatoneofthecontexts inwhichit isdesirable topreservehumandecision-makingprocessesiswherethe decisionconcernsevaluatingthecapabilityoftheindividuals forthepurposeofdeterminingtheirentitlementtoresources. Automateddecision-makingshouldbeavoidedwherethe de-cisioninvolvesrepresentingindividualsingeometricspace. Theuniversityadmissionsprocessisanexamplewherethe decision-makerevaluatesthecapabilitiesofindividualsby as-sessingtheirskillsandpersonalqualities.Thepresentwork isdesignedtobeastartingpointforfurtherscholarly explo-rationforhowthe useofAIdecision-makingprocesses re-configuressocietalarrangementsandproducessociety-wide

24_Lucien _Frys _and _Christian _Staat, _{‘University} _Admission

Practices-France’ (Matching in Practice, 2016) <http://www. matching-in-practice.eu/university-admission-practices-france> accessed1August2019

25_Ibid

26_Martha_Albertson_Fineman,_‘Equality_and_{Difference–the}

Re-strainedState’(2015)66AlabamaLawReview609,614

27_Martha_Albertson_Fineman,_‘Equality,_Autonomy_and_the

(5)

effects.Greaterscholarlyattentionisneededtoaddressthe questioninwhatcontextsthelegislatureshouldrequire hu-mandecision-making.

1. A

definition

of

an

artificial

intelligence

decision-making

process

AnevaluationofAI-baseddecision-makingprocesses neces-sitatesunderstandingwhatAIis,howitfunctionsandwhat elementscomprisethedecision-makingprocess.Thedecision toframethe discussioninterms ofanAIdecision-making processasopposedtoanAIdecision-makingsystemis inten-tional.OneofthereasonsforthischoiceisthatAItechnology isevolving.Forthisreason,itismoremeaningfultofocuson thetypesofproceduresandprocessesthatunderliepresentAI technologiesratherthanonhowcomputerscientistsdesign suchsystems.TheevolvingnatureofAIsystemsisillustrated bythefactthatmultipledefinitionsofartificial intelligence existandthedefinitionshavebeenevolvingovertime.28_One

ofthereasonswhyitisdifficulttodefinethetermAIstems fromthefactthatitisunclearwhatsocietymeansbytheterm intelligent.29_According_to_John_McCarthy,_‘the_problem_is_that

wecannotcharacteriseingeneralwhatkindofcomputational procedureswewanttocallintelligent.’30_Given_that_AI_as_a

disciplineisasocialphenomenonshapedbyindividuals,Bao ShengLoeandcolleaguesrecommendthatthedefinitionofAI becontinuouslyupdated.31_A_present_common_{understanding}

ofanAIsystemisthatitautonomouslylearnsfrombeing ex-posedtoitsenvironmentandmakeschangestoitsmodelof theexternalenvironmentbasedonthesensedchangesinthe environment.32

Itismorefruitfultounderstandtheterm AIintermsof howaparticularsystemisdesignedandoperatesratherthan byreferencetothetermintelligence.IgSnellenarguesthat intelligenceisametaphorinthecontextoftechnicalsystems becausehumanbeingsdothethinkinginthecourseof cre-atingthe system’s architecture.33 _Similarly,_the _Dutch_Raad

28_Bao _Sheng_Loe _and_others,_The _Facets_of_Artificial_{Intelligence:}

AFrameworktoTracktheEvolutionofAI(InternationalJoint Con-ferencesonArtificialIntelligenceOrganization,Stockholm,2018) 5180

29_Max _Vetzo, _Janneke_Gerards _and _Remco _Nehmelman,

Algo-ritmesenGrondrechten(BoomJuridisch2018)41

30_John _McCarthy, _‘What _is _Artificial _{Iintelligence?} _Basic

Questions’ (Stanford University, 2007) <http://jmc.stanford.edu/ artificial-intelligence/what-is-ai/index.html> accessed 13 May 2019

31_Loe_and_others,_The_Facets_of_Artificial_{Intelligence:}_A_Framework_to

TracktheEvolutionofAI5186

32_Vetzo,_Gerards_and_Nehmelman,_Algoritmes_en_Grondrechten₄₁ 33_Ignatius _Theodorus _Maria _Snellen, _‘Het _{Automatiseren} _van

BeschikkingenBestuurskundigBeschouwd’inHansFrankenand others (eds), Beschikken en Automatiseren, Preadviezen Voor de Vereniging voorAdministratiefRecht (Samsom HDTjeenkWillink 1993) 55,quotedinBeppieMargreetAlizevanEck, ‘Geautoma-tiseerde Ketenbesluiten & Rechtsbescherming: Een Onderzoek NaardePraktijkvanGeautomatiseerdeKetenbesluitenOvereen FinancieelBelanginRelatieTotRechtsbescherming’(PhDthesis, TilburgUniversity2018)193

vanState(theCouncilofState)34_maintains_that_it_is

mislead-ingtocallAIdecision-makingsystemsself-learningbecause theydonotunderstandreality.35_The_processes_underlying_the

constructionand operationofAIsystemswillbeexamined toshowwhythetermintelligenceshouldbeunderstoodas havingaspecialistmeaninginthecontextofanAIsystem. Thediscussionwilldemonstratethatitismorefruitfultotalk ofanAIdecision-makingprocessratherthananAI decision-makingsystem.

Computer scientists draw on data science techniques whencreatingAIsystems.36_When_one_understands_the

de-signandoperationofAIsystemsitemergesthatthese sys-temsare notintelligent inthe senseinwhich societies at-tributethetermintelligencetohumanbeings.Computer sci-entistsbeginthedevelopmentofanAIsystembyformulating aproblemforwhichtheyaimtogenerateusefulknowledge.37

Computerscientiststhenpreparedatabyconvertingitintoa formatanAIsystemcanprocess.38_The_end_result_the

com-puterscientistsstrivetoachievewillinfluencehowthey ma-nipulateandlabelthedata.39_The_next_step_is_to_use_the_data

tocreateamodeloftheexternalenvironmentthatcaptures theobjectofinterest,40_such_as_a_student’s_predicted

examina-tiongrades.Themodellocatespatternsinthedataby detect-ingcorrelationsbetweenpiecesofdata.41_The_model_identifies

whatpiecesofinformationarerelatedtoeachother.42_In_the

unsupervisedlearningprocesscomputerscientistsletthe sys-temsearchforpatterns;thesystemallocatesindividualsinto groupsbasedonsharedcharacteristics.43 _In_the_supervised

learningprocessthecomputerscientistsformulateacriterion andtheAIsystemsortsindividualsintogroupsbasedontheir likelihoodoffulfillingthatcriterion.44_The_model_the_system

generatesallowstheusertopredictthatanindividualbelongs toaparticulargroupofpeoplewithsharedcharacteristics.45

TheAIsystempredictsanindividual’sperformancebasedon theperformanceofindividualswhomittreatsashaving sim-ilarcharacteristicstotheindividualinquestion.46_It_applies_a

34_The_Council_of_State_Advises_the_Government_and_Parliament

onLegislationandGovernance.RaadvanState,‘TheCouncilof State’(RaadvanState2019) <https://www.raadvanstate.nl/talen/ artikel>accessed25July2019

35_Raad_van_State,_Advies_{W04.18.0230/I:}_Ongevraagd_Advies_Over

deEffectenvandeDigitaliseringVoordeRechtsstatelijkeVerhoudingen (RaadvanState2018)9

36_Rosaria_Silipo,_‘What’s_in_a_Name?_Artificial_Intelligence_or_Data

Science?’(BetaNewsInc,2019)<https://betanews.com/2019/02/05/ artificial-intelligence-or-data-science>accessed14May2019

37_Foster _Provost _and _Tom _Fawcett, _Data _Science _for _Business

(O’ReillyMediaInc2013)19

(6)

decision-makingprocedurefordeterminingwhetheran indi-vidualisentitledtoapositivedecision.47

Presently,AIsystemslackhumanintelligence.Theydonot havethecapacitytounderstandwhatthecorrelationbetween thepiecesofdatameans,whetherthecorrelationhas signif-icanceandhowthiscorrelationcorrespondstophenomena intheworld.WhenanAIsystemfindsacorrelationbetween twopiecesofdata,thisdoesnotsignifythatAcausesB.48_In

fact,thecorrelationstheAIsystemdetectscanbespuriousor accidental.49_For_instance,_there_is_a_high_correlation_but_not

causationbetweenice-creamconsumptionandsharkattacks; bothtendtooccurduringwarmerseasons.50_Another_reason

whyAIsystemsarenotintelligentstemsfromthefactthat theycannotindependentlyreflectonwhattheirpredictions signifyandwhetherthedecision-makingprocedureproduces societallydesirableoutcomes.

David Preiss and Robert Sternberg view human beings and technologyashavingareciprocalinfluence.51

Technol-ogytransformshumancognitiveskills52_as_well_as_the

under-standingofwhatishumanintelligence.53_Meanwhile,_cultural

contextinfluencestechnologies.54_This_observation_is

corrob-oratedbythefactthatasAIgainsnewcapabilitiestosolve problems,societyredefinesthe termhumanintelligencein order todifferentiatebetweenAIandhumanbeings.55 _The

betterapproachistoacknowledgethatanydefinitionof hu-manandmachineintelligenceistentative.Thereneedstobe anawarenessofwhatdefinitionofintelligenceonechooses, whyandwithwhatconsequences.Giventhatsomecomputer scientistsseektoreplicatehumanintelligenceinAI,56_there

maycomeatimewhenthedividinglinebetween‘artificial’ and‘human’intelligenceislessclear.Societyshouldreflect onthesocialrolethetermintelligencehasasitcontinuesto refinethemeaningofthisterm.

Currently, different definitions of algorithmic or auto-mateddecision systemsexist.Definitions framingthe sub-ject matterbroadly andbyreference toanartificial intelli-gencedecision-makingprocessarepreferable.TheAustralian Human Rights Commission defines ‘AI-informed decision-making’as‘decision-making whichrelieswhollyorinpart

47_Sorelle_A_Friedler,_Carlos_Scheidegger_and_Suresh

Venkatasub-ramanian,‘Onthe(Im)possibilityofFairness’(2016)1609.07236v1 arXiv1,10

48_Eubanks,_Automating_Inequality:_How_High-tech_Tools_Profile,_Police,

andPunishthePoor144

49_Ibid_144-45 50_Ibid₁₄₅

51_David_Preiss_and_Robert_Sternberg,_{‘Technologies}_for_Working

Intelligence’inDavidPreissandRobertSternberg(eds),Intelligence andTechnology:theImpactofToolsontheNatureandDevelopmentof HumanAbilities(Routledge2005)199

52_Ibid 53_Ibid_184-85 54_Ibid₁₉₉

55_Chris_Smith,_{‘Introduction’,}_The_History_of _Artificial_Intelligence

(UniversityofWashington2006)4

56_Ben_Goertzel_and_Pei_Wang,_{‘Introduction:}_Aspects_of_Artificial

GeneralIntelligence’inBenGoertzelandPeiWang(eds),Advances inArtificialGeneralIntelligence:Concepts,ArchitecturesandAlgorithms ProceedingsoftheAGIWorkshop2006(IOSPress2007)1

onartificialintelligence.’57_By_hingeing_the_definition_on_the

termartificialintelligenceandthenotionofdecision-making, the AustralianHumanRightsCommission conceivesof AI-based decision-making in terms of the computer science techniquesunderpinningthe decision-makingprocess. Pro-videdonegivesthetermartificialintelligenceaholistic inter-pretation,thedefinitionoftheAI-informeddecision-making can be interpreted as covering all the stages involved in the decision-making processes. What is significant is that theAustralianHumanRightsCommissionusestheterm ‘AI-informed decision-making’ rather than the term decision-makingsystem.

Incontrast,theDirectiveonAutomatedDecision-Making oftheGovernmentofCanadacontainsanarrowerdefinition becausesitemploysthetermanautomateddecision-making system.Itdefinesanautomateddecisionsystemas

[A]nytechnologythateitherassistsorreplacesthe judge-ment of human decision-makers. These systems draw fromfieldslikestatistics,linguistics,andcomputerscience, andusetechniquessuchasrules-basedsystems, regres-sion,predictiveanalytics,machinelearning,deeplearning, andneuralnets.58

WhatiscommontothedefinitionsoftheAustralian Hu-man Rights Commission and the Directive on Automated Decision-Makingisthattheydiscussthe respectiverolesof artificial intelligence technology and humanbeings inthe decision-makingprocess.Whatismore,thedefinitions cen-treonthetypesofcomputersciencetechniquesinvolved.One ofthereasonswhythetermautomateddecision-making sys-tem isnarrowerthan the term decision-making process is becauseitexcludesstagesthatbearontheoutcomeofthe decision-makingprocessbutthattakeplacepriortothe ac-tualconstructionofthe system.Inparticular,the decision-makingprocessbeginswhenthecomputerscientist formu-latesaproblemtobesolvedusingtheAI-drivenprocedure be-causethisstagebearsontheoutcomeofthedecision-making process.SolonBarocasandAndrewDSelbstpositthat com-puterscientistsexercisesubjectivitywhentheyformulatea problemthemachineshouldsolve.59_They_do_this_by_defining

thevariableforwhichthemachinemakesaprediction.60_How

computerscientistsdefinethisvariable,suchasagood em-ployee,shapeswhatrelationshipsbetweendatathemachine findsandthereforeitspredictionsaboutthesuitabilityofthe applicantfortheposition.61_According_to_Reuben_Binns,_if_the

computerscientistusesabiasedvariableasabenchmarkfor thebasisonwhichtheAIdecision-makingprocesspredicts

57_Australian _Human _Rights _Commission, _‘Decision

Mak-ing and Artificial Intelligence’ (Australian Human Rights Commission, 2019) < https://tech.humanrights.gov.au/ decision-making-and-artificial-intelligence>accessed22January 2020

58_Government _of _Canada,_‘Directive _on _Automated

Decision-Making’ (Government of Canada 1 April 2019) <https://www. tbs-sct.gc.ca/pol/doc-eng.aspx?id=32592> accessed 23 January 2019

59_Solon_Barocas_and_Andrew_D_Selbst,_‘Big_Data’s_Disparate

Im-pact’(2016)104CaliforniaLawReview671

(7)

futureperformancethentheoutcome willbebiased.62 _The

termdecision-makingprocessispreferablebecauseitcanbe definedtoincludethestagewherethecomputerscientist for-malisestheproblemtobesolvedusinganAI-driven proce-dure.

TheCouncilofEuropeCommitteeofExpertsusestheterm anAIdecision-makingsystemandinthisrespectmirrorsthe approachoftheDirectiveonAutomatedDecision-Making.The CouncilofEuropeCommitteeofExpertsliststhetypesoftasks theAIdecision-makingsystemcarriesoutandtheprocesses thatcomprisetheoperationofsuchsystems.63_In_particular,

itdefinesalgorithmicsystemsasapplicationsthat‘perform one or moretaskssuch as gathering,combining,cleaning, sorting andclassifyingdata,aswellasselection, prioritisa-tion,recommendationanddecision-making.’64_The_definition

oftheCouncilofEuropeCommitteeofExpertsispreferableto thedefinitiontheCanadianDirectiveonAutomated Decision-Makingoffers.Itfocusesonthestepsinvolvedinconstructing amodelthattheAIdecision-makingsystemusesformaking predictionsaboutfutureperformanceandtheprocessof pro-ducingadecisioninrespectofanindividual.Thus,thereisan emphasisontheprocessleadingtothedecisionratherthan onthetypeoftechniquesthecomputerscientistsusein or-dertoprogramAIsystems.ThisaspectmakesthenatureofAI decision-makingsystemsexplicitbylistingthestepsentailed inproducingadecision.Thisdefinitionmakesiteasierforthe lawmakersandenduserstodebatethesocialconsequences ofusingAIdecision-makingsystems.

Anotheradvantageofdefiningthetermintermsofwhat elements comprise the AI decision-making process is that it providesanunderstanding ofwhatthesystem doesand how it achieves its objective. On the other hand,the term AI decision-makingsystem isopaque.Little understanding may begleanedfrom thisterm.Societyuses theterm sys-tem to refertointerdependent and interactingelements.65

ThefactthatAItechnologyutilisesacombinationof differ-entelementsrevealslittleaboutthenatureofthetasksit per-formsandhowitperformsthem.Thisstemsfromthefactthat thetermsystemdrawsattentiontothephysicalarchitecture ofthesystemand whatcomponents orelementscomprise thesystem.Whatiscoreforunderstandingdecision-making istheprocessthroughwhichonearrivesatadecisionrather thanthefactthatvariousinterdependentstagesareinvolved inthedecision-makingprocess.

Arelevantconsiderationisthatsincesocietal understand-ingofAIisevolving66_it_could_be_that_in_fifty_years_time_the

62_Reuben_Binns,_‘Imagining_Data,_Between_Laplace’s_Demon_and

theRuleofSuccession’inIrinaBaraliucandothers(eds),Being Pro-filed:CogitasErgoSum(AmsterdamUniversityPress2018)

63_Committee_of_Experts_on_Human_Rights_Dimensions_of

Auto-mated DataProcessing andDifferentFormsofArtificial Intelli-gence,DraftRecommendationoftheCommitteeofMinisterstoMember StatesontheHumanRightsImpactsofAlgorithmicSystems(Councilof Europe2019)par3

64_Ibid

65_{Merriam-Webster}_{Incorporated,}_‘System’₍_{Merriam-Webster}

In-corporated,2019)<https://www.merriam-webster.com/dictionary/ system>accessed22January2020

66_Loe_and_others,_The_Facets_of_Artificial_{Intelligence:}_A_Framework_to

TracktheEvolutionofAI5180

definitionofwhatAIisandhowitoperateswillbevery dif-ferent.Forinstance,thearchitectureofAIcouldhaveahighly distributedformwhereitisunclearexactlywhatitselements areandhowitsdifferentelementsinterplay.Giventhat com-puterscientistsusetheknowledgeaboutthehumanbrainas inspirationtocreatenewAItechniques,67_AI_could_resemble

thefunctioningofahumanbodyveryclosely.Such develop-mentscouldmakeitdifficulttospeakaboutthemachineasa system.

Ofsignificanceisthatthereisaparallelbetweenthe el-ementscomprisinghumanandtheAIdecision-making pro-cesses. The term AI decision-making system fails to cap-ture this important element.This is because society does not conceive of humanbeings and their deliberation as a system.However,onecantalk aboutthe similaritiesinthe decision-making process humanbeings engage inand the AIsystemscarryoutbecausehumanbeingsdeveloptheAI decision-makingprocess.Giventhathumanbeingsexercise theirjudgementindevelopingAIdecision-makingprocesses, itisnotsurprisingthattherecanbeadegreeofsimilarity be-tweenhumanandAIdecision-makingprocesses.

Thehumandecision-makingprocessbeginswiththe fram-ingofthegoalthatthedecision-makingprocedureisdesigned toachieve andwith theidentification ofthe criteria corre-spondingtoentitlementtoapositivedecision.Forinstance, whenauniversitysetsupanadmissionsprocess,it formu-latesa set ofgoals.Thegoals could betoattract students who possessaparticularskillset,who haveparticular esti-matedacademicperformanceor who are representativeof thepopulationasawhole.Theuniversitycouldaimto miti-gatetheexistenceofsocietallyembeddedinequalitiesby plac-ing emphasison attracting candidates who experience so-cialbarriers.Thegoaltheuniversitysetswilldeterminewhat criteria it choosesasa basison which the officialsshould selectthe students.Theadmissionscriteria willdetermine whetherthehumandecision-makerconsidersonlythe stu-dent’sgradesoradditionalcriteria,suchasextracurricular ac-tivities,workexperienceandtheapplicant’spersonal circum-stances.Thedecision-makingcriteriadeterminewhat quali-tiesthedecision-makertakesintoaccountorignores.

When computer scientistsdecide how to formulate the problemandwhattheAIprocesspredicts,theyselectthegoal forthe decision-makingprocess and the criteria thatform thebasisofthedecision-makingprocess.68 _When

formulat-ingtheproblemtobesolvedthecomputerscientsitsarein asimilarpositiontohumandecision-makerswhoaretasked withdevelopingandapplyingadecision-makingprocedure. Inbothcasesthegoaltobeachieveddetermineswhatcriteria thedecision-makersadoptforselectingapoolofcandidates. Thedifferenceisthathumandecision-makerscanchoose cri-teriathatcanbeexpressedinbothquantitativeand qualita-tiveterms.Ontheotherhand,computerscientistscanselect onlythosecriteriaasindicatorsofagoodcandidatethatcan beexpressedinquantitativeterms.69 _Examples_of quantita-67_Zhongzhi_Shi,_Advanced _Artificial_Intelligence_(World_Scientific

Publishing2011)10par1

68_Barocas_and_Selbst,_‘Big_Data’s_Disparate_Impact’_678-80 69_Friedler, _Scheidegger _and _{Venkatasubramanian,} _‘On _the

(8)

tiveproxiesaregrades,rankingsandnumberofoutputs. Com-puterscientistsneedtofindquantitativeproxiesiftheywant tocapturequalitativecharacteristics.70_Qualitative

character-isticsrefertomultidimensionalandmultitexturedqualities, suchascreativityandleadershipskills.

When decision-makersuseagradetocapturea qualita-tivecharacteristic,suchasintelligence,theyemployaproxy.71

For instance,the standardised admission test forgraduate programmes GeneralRecord Examinationmeasures analyt-ical,quantitativeandverbalskills.72_The_scores_of_this_test

didnothaveaccurate predictivecapacityforhow lecturers atYaleUniversityratedthestudents’analytical,creativeand practicalskillsonagraduatepsychologyprogramme.73_Thus,

AIdecision-makingprocessesshouldbeviewedasmapping proxiesontothemodelalongsideactualcharacteristics.Itis difficulttoexpressqualitativecharacteristics,suchas inter-personalandteamworkskills,numericallybecausethey re-latetohowindividualsinteractwitheachother.While class-matescouldbeaskedtorankeachotheronthemetricofa teamworkingskill,suchresponseswouldbeunreliable. Indi-vidualscanbeinfluencedbytheirpersonalattitudes,bya de-siretocompeteorbyunconsciousbiases.Theymaylackthe distancenecessarytoreflectonhowallteammembers inter-acted.Qualitativedescriptionsofhowanindividualactedin particularcircumstancesprovidemoreinformationaboutan individual’steamworkskills.TheAIdecision-makingprocess shouldbeunderstoodasamorelimitedprocedurethana hu-mandecision-makingprocessbyvirtueofitslimitedcapacity tocapturequalitativedataandthecontextbehindthisdata.

Thereanother importantdifference betweenthe human and the AI decision-making processes. Human decision-makersdeterminewhat facetsoftheperson theyconsider throughchoosingthedecision-makingcriteria.Incontrast,AI decision-makingprocessescreatewhatLukeStarkcallsthe ‘scalablesubject.’74_The_system_purports_to_model_the

indi-vidualbutinfactreflectscorrelationsbetweencharacteristics presentwithinagroupthatmaynotapplytotheindividual inquestion.75 _Annamaria_Carusi_elaborates _that_the _model

representstheindividual inareductiveway76_and _that_the

approachtorepresentationcontainsparticularvalues.77_The

lackofgranularinformationcanresultinthemodelmaking

70_Provost_and_Fawcett,_Data_Science_for_Business₃₃₉

71_Friedler, _Scheidegger _and _{Venkatasubramanian,} _‘On _the

(Im)possibilityofFairness’3

72_Educational_Testing_Service,_‘About_the_GRE_General_Test’₍

Ed-ucational Testing Service,2019) <https://www.ets.org/gre/revised_ general/about>accessed22July2019

73_Robert_J_Sternberg_and_Wendy_M_Williams,_Does_the_Graduate

RecordExaminationPredictMeaningfulSuccessinPsychology(Yale Uni-versity1994),quotedinRobertJSternberg,‘Myths,Countermyths, andTruthsaboutIntelligence’(1996)25EducationalResearcher11, 14

74_Luke_Stark,_{‘Algorithmic}_{Psychometrics}_and_the_Scalable

Sub-ject’(2018)48SocialStudiesofScience204,207

75_Ibid₂₁₃

76_Annamaria_Carusi,_‘Beyond_Anonymity:_Data_as

Representa-tioninE-researchEthics’(2008)37InternationalJournalofInternet ResearchEthics37,61

77_Ibid₄₂

generalisationsthatare unfair totheindividual.78 _To

illus-trate,sincethemodelgroupsstudentsbasedonpast exam-inationsperformanceforthepurposeofmakingaprediction aboutfutureresults,itwouldgroupstudentswhoperformed poorlyirrespectiveofthereasonforthelowresults.Thismay resultintheAIsystemfalselypredictingalowgradefora stu-dentwhoseperformanceinthepasthadbeeninfluencedby anillnessbutwhorecoveredlater.

PatrickAllocomments thatthemodeldepictingthe pat-terns in the data represents a proxy for what one is try-ingtopredictratherthantheactualstateoftheworld.79 _A

modelthat predicted the student’s performanceon an ex-aminationthattestedhowwellthestudentmemorisedthe materialis not a reliable proxy for the student’s aptitude. Whilethemodelprovidesindirectinformationaboutan in-dividual’smemorycapacity,ittellslittleaboutthestudent’s aptitudes,such as problem-solvingcapacity and creativity. ThedifferencesbetweenhumanandAIdecision-making pro-cessesdonotprecludeconsideringtheautomationof deci-sionsintermsofanAIdecision-makingprocess.Infact,the termprocesshighlightsthefactthathumanbeingsconstruct thedecision-makingprocedurewithinthemachine.Whatis more,thistermmakesitpossibletodemarcateatwhatpoint thedecision-makingcommencesandends.Crucially,the defi-nitionthatfocusesontheprocessratherthanthesystem pre-ventsmisrepresentation.Thetermartificialintelligence sys-temcancreateamisleadingimpressionthatthesystemhas capabilitiesthatcorrespondtohumanintelligenceorthatthe decisionofcomputerscientistsrelatingtothevariabletobe predictedhadnoimpactontheoutcomefortheapplicants.

ThequestioniswhatelementscomprisetheAI decision-makingprocess.Theproposeddefinitiondoesnotcover situa-tionswhereahumandecision-makerusestheanalyticstheAI systemgeneratesasasoleorpartialbasisforreachinga deci-sion.BycombiningdefinitionsoftheCouncilofEurope Com-mitteeofExpertsandtheAustralianHumanRights Commis-siononecanarriveatasuitabledefinitionofanAI decision-makingprocess.Whatneedstobeincludedinadditionisthe elementofhumandecision-makinginvolvedinformulating aproblemtobesolvedand howtoconstructthesystemto achievethisgoal.TheAIdecision-makingprocessshouldbe understoodasstartingwiththecomputerscientist formulat-ingtheproblemtobesolvedandthegoalstobeachieved.It encompassesthecollection,cleaning,labelling,aggregation, analysis,manipulationand processingofdata.Thesesteps arecarriedoutinordertopredictfutureperformanceandto produceadecision affectingtheright orentitlement ofan individualtoapositivedecision.Thedefinitionincludesthe applicationofatemplatefordeterminingwhetheran indi-vidualshouldbegrantedapositivedecision.Thisdefinition isappropriate eventhoughthe processofcreatingamodel oftheenvironment asabasisformakinga predictionisa separatestagefrom the decision-makingprocedure for de-termining anindividual’sentitlement toan affirmative

de-78_Frederick_Schauer,_Profiles,_{Probabilities}_and_Stereotypes_(Belknap

Press2006)3

79_Patrick_Allo,_{‘Mathematical} _Values _and _the_Epistemology _of

(9)

cision.80 _A _broad_definition_encompassing_all _the _elements

thatbearonhowthealgorithmicprocessmeasuresand pre-dictsfutureperformanceaswellashowitproducesa deci-sionisneeded. Thisapproachensuresadequate protection ofindividuals.Thisapproachleavesscopeforthefactthata computerscientistcouldembedavarietyofdecision-making procedures for arriving at a decision. For instance, an AI decision-makingprocess couldallocatetheresourcesto in-dividualswiththehighestscoreforpredictedperformance.81

Itcouldtakeintoaccountotherconsiderations.Toillustrate, AdityaKrishnaMenonandRobertCWilliamsondesignedan algorithmicdecision-makingprocedureforanAIsystemthat theyargueachievesthebesttrade-offbetweenaccuracyand fairness.82

WhyisanexpansivedefinitionforthetermAI decision-makingprocessnecessary?Theprocessofmappingthedata ontothemodelandofpredictinganindividual’sperformance based onthe modelhas influenceonwhether the individ-ual willreceiveapositivedecision.Theproposeddefinition isdesignedtocapturethefactthatcomputerscientistsmake subjective decisions inthe courseofcreating the architec-turethatenablestheAIdecision-makingprocess tocollect, aggregateandanalysedata.83_The_choices_computer_scientists

makeaffecthowtheAIdecision-makingprocessproduces de-cisions andwhat kind ofdecisionanindividual receives.84

Often,thedecisionsofcomputerscientistsarehiddenand re-flectaparticularunderstandingoftheworld.85_For_example,

computerscientistsmakeassumptionswhendecidinghow torepresentapersoninamodel.86_Since_individuals_are

mul-tidimensionalandcannotbedescribedexhaustively,itisin theorypossibletocreateaninfinitenumberofsnapshotsof theindividualdependingonwhatcombinationof character-isticsoneinputs.Forinstance,acandidatecanbedescribed asafemalewithascoreofeightypercentformathematics andascoreoffiftypercentforEnglishlanguage.Alternatively, thesamecandidatecanbedesignatedasafemalecandidate who isenroledin aschoollocated inanunderfunded dis-trict.Shelearnsinanovercrowdedclassroomduetoa short-age ofEnglishteachers.Dependingon whatcharacteristics one chooses asbeing relevant for the purpose of generat-ingamodel,onecangetadifferentsnapshotoftheperson. What ismore,sinceinequalitiesare structurallyembedded insociety,groupswillberepresentedinadistortedmanner whenmappedontothemodel.87_The_researchers_cite_the_fact

thattheverbalsectionofthestandardisedAmerican univer-sityadmissiontestSATfunctionsdifferentlyforthe African-Americanindividuals.88_It_follows_that_there_is_a_discrepancy

80_Provost_and_Fawcett,_Data_Science_for_Business₂₅

82_Menon_and_Williamson,_‘The_Cost_of_Fairness_in_Binary

Classi-fication’2

84_Ibid 85_Ibid 86_Ibid_6-7 87_Ibid₇

88_Ibid _8; _Maria _Veronica _Santelices_and _Mark _Wilson,_‘Unfair

Treatment?TheCaseofFreedle,theSAT,andtheStandardisation

betweentherealworldandhowtheAIdecision-making pro-cessmapstheworldontoageometricalspaceaspartof gen-eratingamodeloftheworld.89

Computerscientistscansteerthedataanalysisprocessby framingforwhatmetrictheAIdecision-makingprocess for-mulatesthepredictionsandbychoosingaparticularapproach todataanalysis.90_In_the_context_of_AI_and_human

decision-makingprocessesthechoiceofcharacteristicstodenotemerit forthepurposeofselectingstudentsshapeswhether individ-ualshaveanequalopportunitytobeadmittedtouniversity. Someselectioncriteria appearneutral but infact hide the fact thatthe decision-making procedurecreates admission barriersforchildrenfrompoorsocioeconomicbackgrounds. Forinstance,thecomputerscientistcansetagoodcandidate performanceforadmissionstoauniversityinterms of ex-cellingatplayingamusicalinstrument,painting,playing pro-fessionalsportsorwinningadancecontest.Thisapproachto studentadmissionsresembleshowthehighestranked uni-versitiesintheUnitedStatesselectstudents.91_Children_have

unequalaccesstoparticipationinextracurricularactivities. AnnaBull examines how complex factors lead tochildren frommiddle-classandupper-classfamiliesbeingmorelikely toplayamusicalinstrument.92_The_reasons_include_the_cost

ofmusiclessonsandthefactthattheapproachtoteaching musicreflectsthenatureofinteractionsprevalentin middle-classteachingsettings.93_This_example_shows_that_the

crite-riacomputerscientistsembedinto the AIdecision-making processandthemetricsbywhichtheprogramgeneratesthe predictionwillshapewhetherindividualshaveequalaccess touniversityeducation.Accordingly,theAIdecision-making processshouldbedefinedtoincorporateallstagesofsystem developmentandoperationbeginningwithformulationofthe problemtobesolvedandendingwithadecisionoutput.

2. Introducing

the

theoretical

framework:

the

vulnerability

theory

Thevaluesoneholdsamongstotherswilldeterminehowone evaluatestheAIdecision-makingprocess.Forinstance,those whovalueefficiencywillaskquestionssuchaswhetherthe useoftheAIdecision-makingprocesscutscostsorshortens

ApproachtoDifferentialItemFunctioning’(2010)80Harvard Edu-cationalReview106,126

90_Felix _Stalder, _‘From _{Inter-subjectivity} _to _{Multi-subjectivity:}

KnowledgeClaimsandtheDigitalCondition’inIrinaBaraliucand others(eds),Beingprofiled:CogitasErgoSum(AmsterdamUniversity Press2018)136

91_Ilana _Kowarski, _‘How _Colleges _Weigh _{Applicants’}

Ex-tracurricular Activities’ (US News, 2018) <https://www. usnews.com/education/best-colleges/articles/2018-10-25/ how-colleges-weigh-applicants-extracurricular-activities> ac-cessed14May2019

92_Anna _Bull, _{‘Reproducing} _Class? _Classical _Music _Education

and Inequality’ (Discover Society, 2014) <https://discoversociety. org/2014/11/04/reproducing-class-classical-music- education-and-inequality/>accessed14May2019

(10)

thedeliberationtime.94_Economists_define_efficiency_as_‘a

situ-ationwhereeachgoodisproducedattheminimumcostand whereindividualpeopleandfirmsgetthemaximum bene-fitfromtheuseoftheresources.’95_Equity_is_a_different_type

ofvalueincomparisontoefficiency.96_Equity_concentrates_on

whetherthereisfairnessandjustice.97_Individuals_who_value

equitywillaskdifferenttypesofquestionsthaneconomists whenassessingthedesirabilityofusingAIdecision-making processes.HowoneevaluatesAIdecision-makingprocesses willdifferdependingonhowonedefinesfairness.Different people have a different understanding ofwhat constitutes fairness.98 _{Additionally,} _there _is _a _difference _between _how

scholars99 _and _the_general_population_define _fairness.100 _In

discussingfairnessitisimportanttoacknowledgethevalue of pluralism and cultural diversity.Respect for individuals is contingenton a recognitionof theiropinions and value systems. Therepresentation ofaplurality ofviews is con-ducivetoinformeddiscussionsaboutwhatconstitutesagood life.Itwidensthearrayofargumentsandintroducesnew vis-tasfromwhichtoassesspropositions.

Roger Brownsword argues that to gain legitimacy regu-lators shouldadoptinstrumentsthat capturecommon val-uesandconcernswhileleavingroomforlocaldifference.101

The present article uses the vulnerability theory as a lens forevaluatingAIdecision-makingprocessesbecausethe the-ory captureshow citizens conceive ofcore components of fairnessand justice.MarthaAlbertson Finemanformulated ‘vulnerabilitytheory’asan‘analternativetotheoriesof so-cialjusticeandresponsibilitythatfocusonachievingformal equality.’102_The_term_‘social_justice’_focuses_on_the_position

ofmanyindividuals withinasociety.103 _{Traditionally,}

advo-catesofsocialjusticecalledforajustdistributionofresources and ofthe fruitofeconomicproductionamongst the indi-viduals.104_What_{differentiates}_Fineman’s_approach_to

under-standinghowtoadvancesocialjusticeisthatsheexamines theimpactoflegallyconstructedsocialinstitutionsand rela-tionshipsonthelivesofindividuals.105_The_{vulnerability}

the-oryreflectshowcitizensunderstandfairnessbyfocusingon thewayinwhichthestateconstructsrelationshipsbetween individualsandinstitutions.106_A_study_found_that

individu-94_Vishal_Marria,_The_Future_of_Artificial_Intelligence_In_The_Workplace

(ForbesMediaLLC2019)

95_John_Sloman,_Economics₍₆_edn,_Prentice_Hall₂₀₀₆₎₉ 96_Ibid₁₁

97_Ibid 98_Ibid

99_Norman _J_Finkel,_Rom_{Harré and}_José-Luis_Rodriguez_Lopez,

‘CommonsenseMoralityAcrossCultures:NotionsofFairness, Jus-tice,HonorandEquity’(2001)3DiscourseStudies5,5

100_Ibid₂₁

101_Roger_Brownsword,_{‘Regulatory}_{Cosmopolitanism:}_Clubs,

Com-mons,andQuestionsofCoherence’(2010)018/2010TILTLaw& TechnologyWorkingPaper2,4

102_Nina_A _Kohn,_{‘Vulnerability}_Theory_and_the_Role_of

Govern-ment’(2014)26YaleJournalofLaw&Feminism2,6

103_Martha _Albertson _Fineman, _{Vulnerability} _and _Social _Justice

(EmoryUniversity2019)1

104_United_Nations,_Social_Justice_in_an_Open_World:_The_Role_of_the

UnitedNations(UnitedNations2006)7

105_Fineman,_{Vulnerability}_and_Social_Justice₂ 106_Ibid

alsusethetermsfairnessandjusticeinterchangeablytorefer toviolationsofequityandequality.107_Individuals_understand

fairnesstoincludebothhowindividualsarepositionedin re-lationtootherindividualsinrelationshipsaswellashow indi-vidualsarepositionedinrelationtoinstitutionsinsociety.108

Theemploymentofthevulnerabilitytheoryallowsoneto as-sesswhatimpact the use ofAIdecision-making processes hasonindividualsandsociety.Thepresentarticleevaluates anumberofwaysinwhichtheuseoftheAIdecision-making processesimpactsontheindividualsandsocietyfromthe per-spectiveofsocialjustice.Itisbeyondthescopeofthiswork toevaluatetheAIdecision-makingprocessfromthevantage pointofalltheories ofjusticeand fairnessacrosscultures. Neitherisitpossibletoexamineinacomprehensivemanner allthewaysinwhichthecumulativeuseoftheAI decision-makingprocessindifferentdomainswilltransformsociety.

Theuseofthevulnerabilitytheoryapproachavoids draw-inganarbitrarydistinctionbetweentheprivateandthe pub-licdomains.109_What_becomes_relevant_for_the_analysis_is_how

theemploymentoftheAIdecision-makingprocessesaffects the subject of the decision-making procedure and society ratherthanwhethertheinequityarosefromtherelationship withthestateor withotherindividuals.Incontrast, schol-arlywritingsinpoliticalscienceandphilosophydistinguish betweenpublicandprivatedomainstodemarcatewhenthe statecanintervenetoregulate.110_This_distinction_is_arguably

apparentfromhowsomescholarscontrastthetermsjustice andfairness.111_John_Rawls_for_example_defines_justice_as

re-latingtotheinstitutionalarrangementsandpracticesthat de-finerights,dutiesandoffices.112_He_defines_fairness_as_relating

totherightsofpersonsarisinginthecourseoftheir interac-tionwithoneanotheronanequalbasis.113_Individuals_would

agreeonrulestoensurethattheydidnotfeeltheywere be-ingtakenadvantageofininterpersonalrelationships.114_The

drawingofadistinctionbetweentherelationshipsindividuals havewitheachotherandwiththeinstitutionsforthepurpose ofassessingtheimpactofAIdecision-makingprocessesis un-desirable.Fairnessandjusticeareopen-endedtermsthat so-cietyusesasheuristicdevicestoredressinequities.The con-tentofthetermsjusticeandfairnesscanbegivendifferent meanings115_{depending on the}_{context to which individuals} 107 _Finkel,_{Harré and}_Lopez,_{‘Commonsense}_Morality_Across

Cul-tures:NotionsofFairness,Justice,HonorandEquity’21

108 _Ibid

109 _Martha_Albertson_Fineman,_‘Injury_in_the_Unresponsive_State:

WritingtheVulnerableSubjectIntoNeo-LiberalLegalCulture’in AnneBloom,DavidMEngelandMichaelMcCann(eds),Injuryand Injustice:TheCulturalPoliticsofHarmandRedress(Cambridge Uni-versityPress2018)19

110 _See_for_instance_the_work_of_Max_Weber,_Isaiah_Berlin,_Jürgen

Habermas,RichardRorty,Michael Walzerand JohnStuartMill. RaymondGeuss,PublicGoods,PrivateGoods(PrincetonUniversity Press2003)10

111 _Finkel,_{Harré and}_Lopez,_{‘Commonsense}_Morality_Across

Cul-tures:NotionsofFairness,Justice,HonorandEquity’5

112 _John_Rawls,_‘Justice_as_Fairness’₍₁₉₅₈₎₆₇_The_{Philosophical}

Re-view164,164

113 _Ibid₁₇₈ 114 _Ibid

115 _Manuel _Velasquez _and _Claire _Andre, _‘Justice _and _Fairness’

(11)

applytheseterms116_and_depending_on_the_society’s_value

sys-tem.117 _If_one_is_to_address _inequities_{comprehensively,}_one

should focusonall sourcesfrom whichtheinequities may arise.

Thevulnerabilitytheorydemonstrateswhyitisimportant tofocusbothonhowtheoperationoftheAIdecision-making processconstructsrelationshipsbetweenindividuals,and be-tweenindividualsandinstitutionsinanalysingtheimpactof thesesystemsfromtheperspectiveofsocialjustice.According tothevulnerabilitytheory,individualsaresituatedindifferent economic,social,culturaland institutionalrelationships.118

Theserelationshipscannotbeclearlydemarcatedasbeing ei-therprivateorpublic.119_The_position_of_the_individual_within

theserelationshipsdetermineswhethertheinstitutional ar-rangements create opportunities or impediments.120 _These

institutionsformasystemthatdeterminestheresilienceof the individual.121 _The_term_resilience _refers_to _the

individ-ual’sabilitytorecoverfromlife’ssetbacksandtotake advan-tageofopportunities.122_There_are_five_types_of_resources_that

theinstitutionsprovidethatarecrucialforhuman flourish-ing.123_First,_material_goods_determine_the_{individuals’}_quality

oflifeandallowthemtoaccumulateadditionalresources.124

Second,individualsderive supportfromsocialnetworks.125

Third, human assets, such as education and employment, placetheindividualsinapositiontodeveloptheir capabili-ties.126_Fourth,_individuals_benefit_from_having_access_to

exis-tentialandaestheticresources,suchasreligion,cultureand art.127 _Fifth,_individuals _need_ecological _assets,_such_as_the

naturalenvironment,tomaintainphysicalwell-being.128

Ac-cesstointerpersonalresourcesofsupport,suchasfamily129

and socialnetworks,constitute relationshipsthatone typi-callyviewsasprivate.Inpractice,suchprivaterelationships cannotbeseparatedfromrelationshipswiththestate.130_For

instance,lawsprohibitingharassment,bullyingand discrim-inationplayacrucialroleincreatinginclusivespaceswhere individualscanengageininterpersonalrelationships.

Conse-116_Michael_Adler,_‘Fairness_in_Context’₍₂₀₀₆₎₃₃_Journal_of_Law_and

Society615,638

117_Kenneth_A_Rasinski_and_Leslie_A_Scott,_‘Culture,_Values,_and

Be-liefsAboutEconomicJustice’(1990)4SocialJusticeResearch307, 320

118_Fineman,_‘Equality,_Autonomy_and_the_Vulnerable_Subject_in

LawandPolitics’22

119_Fineman,_‘Injury_in_the_Unresponsive_State:_Writing_the

Vul-nerableSubjectintoNeo-LiberalLegalCulture’19

120_Fineman,_‘Equality,_Autonomy_and_the_Vulnerable_Subject_in

LawandPolitics’23

121_Ibid₂₂

122_Fineman,_‘Equality_and_{Difference–the}_Restrained_State’_622-23 123_Fineman,_‘Equality,_Autonomy_and_the_Vulnerable_Subject_in

LawandPolitics’22-23

124_Ibid₂₂ 125_Ibid₂₃ 126_Ibid 127_Ibid 128_Ibid

129_Margaret_Thornton,_‘The_Cartography_of_Public_and_Private’_in

MargaretThornton (ed),PublicandPrivate:FeministLegalDebates (OxfordUniversityPress1995)2

130_Fineman,_‘Injury_in_the_Unresponsive_State:_Writing_the

Vul-nerableSubjectintoNeo-LiberalLegalCulture’19

quently,itisartificialtodistinguishbetweenjusticeand fair-nessbasedonwhethertherelationshipispublicorprivatein nature.

Forthepurposeofthisarticle,itappearsdesirabletouse thevulnerabilitytheoryratherthantheoriesoffairness,which focus onthe treatmentofanindividual forthepurposeof evaluatingtheAIdecision-makingprocesses.Theschool al-locationsysteminNewYorkCityisacasestudythat illus-trateshowafocusontheimpactoftheemploymentoftheAI decision-makingprocessontheindividualcanresultin fail-ingtodetectbothsourcesofsocialinjusticeandunfairness fortheindividual.Currently,theauthoritiesintheNewYork Cityuseanalgorithmtoplacechildrenintohighschools.131

Childrenprovidealistoftwelveschoolchoicestothe author-ities.132_The_algorithm_allocates_children_to_a_school_by

select-ingapoolofcandidateswiththehighestgrade.133_This_means

thatinpracticeeachschoolwillhavestudentswithina par-ticulargraderange.Schoolsthatareinthehighestdemand willhaveapoolofstudentswithtopgrades.Schoolswitha lesserdemandwillhaveapoolofstudentswhohavegrades inthemidorlowrange.Thisapproachtousingan algorith-micprocesstoallocatechildrentoschoolsresultsin segre-gation.Childrenwithhighgradesstudyindifferentbuildings andaregeographicallyseparatedfromthechildrenwithlow grades.Thisfindingshouldbeviewedagainstthebackdrop thatschoolsworldwidehaveracialand socioeconomic seg-regation.134_James_A_Allen_expresses_a_broader_concern_that

theuseofAIdecision-makingprocessesperpetuatesand re-inforcesexistingsegregation.135

The focus on whether the selection procedure the AI decision-makingprocessutilisedisfairforaparticular stu-dentintermsofmeritoccludesthewidersocialjustice con-cerns.When athirteen-year-old Jimmy(not hisreal name) voicedhisoppositiontobeingrejectedbyfiveofhistop pref-erenceschools,hewastoldthathisgradeofeightyfivedidnot qualifyhimforadmission.136_The_cut-off_point_for_admission

tothoseschoolswasagradeofninety.137_A_focus_on_whether

Jimmy performedbetter incomparisonto another student precludesamorein-depthenquiry.Thegradehasan appear-anceofbeinganobjectivemarkerthatmeasuresthestudents’

131_Alvin _Roth, _‘Why _New _York _City’s _High _School

Ad-missions Process Only Works Most of the Time’ ( Chalk-beat, 2015) <https://www.chalkbeat.org/posts/ny/2015/07/02/ why-new-york-citys-high-school-admissions-process-only-works-most-of-the-time>accessed15May2019

132_Ibid 133_Ibid

134_Thomas _Toch, _‘The _Lottery _That’s _{Revolutionizing}

D.C. Schools’ (The Washington Post, 2019) <https://www. washingtonpost.com/news/magazine/wp/2019/03/20/feature/ the-lottery-thats-revolutionizing-ddd-c-schools> accessed 15 May2019

135_James _A _Allen, _‘The _Color _of _Algorithms: _An _Analysis _and

ProposedResearchAgendaforDeterringAlgorithmicRedlining’ (2019)46(2)FordhamUrbanLawJournal219,234

136_Alvin _Roth, _‘Why _New _York _City’s _High _School

Ad-missions Process Only Works Most of the Time’ ( Chalk-beat, 2015) <https://www.chalkbeat.org/posts/ny/2015/07/02/ why-new-york-citys-high-school-admissions-process-only-works-most-of-the-time>accessed15May2019