How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India
Accountability, a requisite for responsible AI, can be facilitated through
transparency mechanisms such as audits and explainability. However, prior work
suggests that the success of these mechanisms may be limited to Global North
contexts; understanding the limitations of current interventions in varied
socio-political conditions is crucial to help policymakers facilitate wider
accountability. To do so, we examined the mediation of accountability in the
existing interactions between vulnerable users and a 'high-risk' AI system in a
Global South setting. We report on a qualitative study with 29
financially-stressed users of instant loan platforms in India. We found that
users experienced intense feelings of indebtedness for the 'boon' of instant
loans, and perceived huge obligations towards loan platforms. Users fulfilled
obligations by accepting harsh terms and conditions, over-sharing sensitive
data, and paying high fees to unknown and unverified lenders. Users
demonstrated a dependence on loan platforms by persisting with such behaviors
despite risks of harms such as abuse, recurring debts, discrimination, privacy
harms, and self-harm to them. Instead of being enraged with loan platforms,
users assumed responsibility for their negative experiences, thus releasing the
high-powered loan platforms from accountability obligations. We argue that
accountability is shaped by platform-user power relations, and urge caution to
policymakers in adopting a purely technical approach to fostering algorithmic
accountability. Instead, we call for situated interventions that enhance agency
of users, enable meaningful transparency, reconfigure designer-user relations,
and prompt a critical reflection in practitioners towards wider accountability.
We conclude with implications for responsibly deploying AI in FinTech
applications in India and beyond.