Unlike students taking a high school math exam, black-box AI systems aren’t required to show their work when answering high-stakes questions. And some decision-makers may prefer it that way—especially when their money and morals are at stake.

In an AI-assisted loan-approval experiment, participants acting as lending officers reviewing real $10,000 loan requests often chose not to know why an AI system flagged one borrower as riskier than another, even though the explanation could have revealed whether race or gender influenced the decision, research by Harvard Business School Assistant Professor Alex Chan shows.

Participants were most likely to skip explanations when their bonuses depended on loan repayment, choosing to follow the algorithm’s recommendations rather than examine whether its decisions were biased. The findings suggest that people often actively avoid seeking additional information because knowing more might complicate their decision or create moral discomfort.

As companies race to embed AI in a variety of decision-making processes, from hiring and credit approval to medical testing and judicial proceedings, pressure is mounting to ensure those systems are fair, transparent, and trustworthy. That scrutiny has fueled growing interest in “explainable AI,” which is focused on showing users the reasoning behind AI-generated responses rather than simply delivering answers.

But Chan’s research challenges a common assumption: that people naturally want more transparency from AI systems. His working paper, “Preference for Explanations: Case of Explainable AI,” updated in February, finds that in some cases, people may not want more information if it makes their decisions harder or more uncomfortable.

“Humans interacting with AI are not perfectly rational Bayesian agents,” Chan says. “They are strategic, motivated, and sometimes willfully ignorant.”

How Participants Evaluated Loan Requests

To test how people interact with AI explanations, Chan recruited 2,512 participants online and asked each to review pairs of $10,000 loan requests to a private U.S. lender from unemployed individuals who needed money for living expenses.

Participants viewed basic demographic information about prospective borrowers, including income, race, gender, and family size, as well as an AI-generated prediction of default risk, then had the option to view an explanation for that prediction.

In each pair of loan requests, the algorithm classified one borrower as low risk, with less than a 10% chance of default, and the other as high-risk, at greater than 90%. For a randomly selected group of participants, earning a higher financial bonus was tied directly to whether the borrower repaid the loan.

Participants acting as loan officers could choose to:

1. Override the algorithm and approve both loans.
2. Follow the algorithm’s recommendation on which loan to fund.
3. Choose whether to view an explanation of the factors AI used to make its prediction before deciding.

Read the full article here.