Is Model Bias a Threat to Equal and Fair Treatment? Possibly, Maybe Not.


Summary: There is a fantastic color and cry about the threat of predisposition in our predictive models when used to high significance events like who gets a loan, insurance, a great school assignment, or bail. It’s not as simple as it appears and here we attempt to take a more nuanced look. The result is not as threatening as lots of headlines make it seem.

Is social bias in our models a hazard to equivalent and fair treatment? Here’s a sample of recent headings:

  • Biased Algorithms Are Everywhere, and No One Seems to Care
  • Researchers Combat Gender and Racial Bias in Artificial Intelligence
  • Bias in Machine Learning and How to Stop It
  • AI and Machine Learning Bias Has Dangerous Implications
  • AI Professor Details Real-World Dangers of Algorithm Bias
  • When Algorithms Discriminate

Holy smokes! The sky is falling. There’s even an entire conference dedicated to the topic: the conference on Fairness, Accountability, and Transparency (FAT *– it’s their acronym, I didn’t make this up) now in its 5th year.

But as you dig into this topic it’s a lot more nuanced. As H.L. Mencken stated, “For every complex issue there is a basic solution. And it’s constantly wrong.”

Exactly What Type of Bias are We Talking About?

There are actually several types of bias. Microsoft recognizes five types: association bias, automation predisposition, interaction bias, confirmation bias, and dataset predisposition. Others utilize various taxonomies.

However, we should not blend up socially hazardous bias that can have a concrete and unfavorable effect on our lives (who gets a loan, insurance coverage, a job, a house, or bail) for types of predisposition based upon our self-selection, like what we select to check out or who we choose to pal. It’s the crucial things we wish to focus on.

How Many Sources of Bias in Modeling Are There?

First, we’re not talking about the academic and well comprehended tradeoff between predisposition and variance. We’re speaking about what causes models to be incorrect, however with a high degree of self-confidence for some subsets of the population.

It’s Not the Type of Model That’s Used

Black box models, which imply ANNs of all stripes and especially deep neural webs (DNNs) are constantly being fingered as a culprit. I’ll approve you that ANNs in general and especially those utilized in AI for image, text, and facial recognition normally do not have enough transparency to explain exactly why a particular individual was scored as accepted or declined by the model.

But the same is real of improved, deep forest, ensembles, or numerous other methods. Today, just CNNs and RNN/LSTMs are being used in AI applications however there are no examples I might find of systems using these designs which have really been widely adopted and are causing damage.

Yes, some DNNs did classify people of color as gorillas but nobody is being rejected an important human or social service based upon that type of system. We recognized the problem prior to it impacted anybody. It’s the excellent old ‘customer choice’ scoring designs that pick winners and losers where we need to look.

The important thing to comprehend is that even using the most transparent of GLM and easy choice tree models as our regulated markets are needed to do, predisposition can still slip in.

Costs and Benefits

Remember that the value of utilizing more complex if less transparent strategies is increased precision. That suggests lower expense, greater efficiency, and even the elimination of some kinds of hidden human predisposition.

We’ve chosen as a society that we want our regulated industries, mainly insurance coverage and lending to use totally explainable and simple designs. We intentionally provided up a few of this performance.

While openness has actually been accepted be of value here, do not forget the truth that this clearly indicates that some individuals are paying more than they have to, or less than they ought to if the benefits and threats were more properly designed.

Since we discussed how designs can eliminate some types of concealed human bias, here’s a brief note on a 2003 research study that showed that when employers were offered identical resumes to review, they picked more candidates with white-sounding names. When evaluating resumes with the names redacted however chosen by the algorithm as potential excellent hires, the bias was eliminated. Maybe as typically as algorithms can introduce predisposition, they can likewise secure against it.

The Problem Is Always in the Data

If there were enough data that equally represented results for each social quality we desire to safeguard (generally race, gender, sex, age, and faith in regulated industries) then modeling might constantly be fair.

Sounds simple however it’s in fact more nuanced than that. It requires that you initially define precisely what kind of fairness you desire. Is it by representation? Should each protected group be represented in equivalent numbers (equivalent parity) or proportionate to their percentage in the population (proportional parity/disparate effect)?

Or are you more concerned about the impact of the false positives and incorrect negatives that can be lessened however never ever eliminated in modeling?

This is particularly important if your design affects an extremely little portion of the population (e.g. bad guys or people with unusual diseases). In which case you need to additional choose if you wish to safeguard from false negatives or incorrect positives, or at least have parity in these events for each safeguarded group.

IBM, Microsoft, and others are in the process of trying to provide tools to spot various types of predisposition, but the Center for Data Science and Public Policy at University of Chicago has currently released an open source toolkit called Aequitas which can examine models for bias. They use this decision tree for deciding which type of bias you desire to concentrate on (it most likely is not possible to resolve for all four types and six variations at the same time).

Too often journalism and even some well-read pundits have recommended extreme options that merely don’t take these facts into factor to consider. For example, the AI Now Institute published this suggestion as very first amongst its 10 ideas.

“Core public agencies, such as those responsible for criminal justice, healthcare, well-being, and education (e.g. “high stakes” domains) should no longer utilize ‘black box’ AI and algorithmic systems.”

Their suggestion: send any proposed design to field evidence screening not unlike that utilized before drugs are allowed to be recommended by the FDA. Postpone benefit maybe, but for just how long and at what expense to carry out these tests. And what about design drift and revitalize?

Before we think about such extreme actions, we need to analyze the utility these systems are supplying, what human biases they are getting rid of, and particularly what kind of predisposition we wish to protect against.

Having said that, this organization in addition to others is onto something when they blame public agencies.

Where Are We Most Likely to be Harmed?

In addition to examining the literature I likewise contacted some pals responsible for modeling in regulated industries, particularly insurance coverage and lending. I’ll talk more about that a little more down but I came away with the very strong impression that where we’ve defined ‘regulated’ markets for modeling purposes, defined specifically what data they can and can not use, and after that made them liable to usually state-level agencies who evaluate these concerns, that predisposition is a very little issue.

We Need to Watch Out for Unregulated Industries– The Most Important of Which are Public Agencies

This is not a pitch to extend information regulation to great deals of other personal sector industries. We’ve currently basically covered that waterfront. Turns out that the sort of “high stakes” domains described above are practically all in the public sector.

Since the public sector isn’t understood for investing greatly in data science talent this leaves us with the double whammy of high effect and modest insight into the problem.

However, using examples called out by these sources does not necessarily show that the designs these agencies utilize are prejudiced. Oftentimes they are just incorrect in their assumptions. Two examples called out by the AI Now Institute in fact date back 3 and 4 years and don’t plainly reveal bias.

Teacher Evaluation: This is a controversial model currently being litigated in court in NY that rates teachers based upon just how much their students have advanced (student development percent). Long story short, an instructor on Long Island frequently appreciated efficient was suddenly demoted to inefficient based on the improvement rate of her associate of trainees but outside of her control. It’s a little complex however it smacks of bad modeling and bad presumptions, not bias in the model.

Student School Matching Algorithms: Good schools have actually ended up being a limited resource looked for after by moms and dads. The nonprofit IIPSC created an allocation design utilized to assign trainees to schools in New York, Boston, Denver, and New Orleans. The core is an algorithm that creates one finest school deal for every single trainee.

The model combines information from 3 sources: The schools households in fact want their children to participate in, listed in order of preference; the variety of readily available seats in each grade at every school in the system; and the set of rules that governs admission to each school.

From the review this sounds more like a professional system than a predictive design. Even more, evidence is that it does not improve the great deal of the most disadvantaged trainees. At best the system stops working openness. At worst the underlying design might be totally flawed.

It also illustrates a risk distinct to the public sector. The system is extensively applauded by school administrators since it drastically reduced the work produced by overlapping deadlines, several applications, and some admissions video game playing. It appears to have benefited the firm however not necessarily the trainees.

COMPAS Recidivism Prediction: There is one example of bias we can all most likely agree on from the public sector which’s COMPAS, a predictive design commonly utilized in the courts to predict who will reoffend. Judges throughout the United States use COMPAS to direct their choices about sentencing and bail. A well-known research study showed that the system was biased versus blacks but not in the method you might expect.

COMPAS was discovered to properly anticipate recidivism for black and white accuseds at approximately the same rate. However the incorrect favorable rate for blacks was practically two times as high for blacks when it comes to whites. That is, when COMPAS was wrong (predicted to reoffend but did not) it did so twice as often for blacks. Remarkably it made an in proportion false unfavorable forecast for whites (predicted not to reoffend however did).

Curiously these were the only three examples used for danger in the public sector, only one of which seems to be legally a case of modeling predisposition. However, offered the expansion of predictive analytics and AI, the general public sector as a “high stakes” arena for our personal freedoms appears like an excellent location for this discussion to begin.

How Regulated Industries Really Work

Space doesn’t permit a deep dive into this topic but let’s begin with these three realities:

  1. Certain types of information are off limits for modeling. This includes the obvious protected categories, typically race, gender, sex, age, and religion.  This extends to data which could be proxies for these variables like geography.  I’m told that these businesses also elect not to use variables that might look like ‘bad PR’.  These include variables such as having children or lifestyle patterns like LGBTQ even though these are probably correlated with different levels of risk.
  2. Modeling techniques are restricted to completely transparent and explainable simple techniques like GLM and simple decision trees. Yes that negatively impacts accuracy.
  3. State agencies like the Department of Insurance can and do question the variables chosen in modeling. In business like insurance in most states they have the authority to approve profit margin ranges for the company’s total portfolio of offerings.  In practice this means that some may be high and others may be loss-leaders but competition levels this out in the medium time frame.

What doesn’t happen is checking for predisposition beyond these managed solutions and restrictions. There’s no particular test for diverse effect or for predisposition in false negatives or false positives. The policy is presumed to suffice backed up by the fact that there are no monopolies here, and competitors quickly weeds out the outliers.

Similarly you might have questioned equal parity or proportionate parity testing. It’s simply not possible. An insurance business for instance will have lots of policy programs each targeting some subset of the population where they think there is competitive benefit.

So long as those targeted subsets don’t trespass on the secured variables they are OKAY. So for example, it’s completely OKAY to have actually a policy targeted at school instructors and another at a city-wide location in a city controlled by blue collar manufacturing. There’s no other way to properly test for parity in these examples because they are developed to be demographically distinct.

How Much Bias is Too Much Bias?

You may be interested to understand that the government has actually currently ruled on this question, and while there are somewhat different guidelines utilized in unique scenario, the response is 80%.

The estimation is simple, if you hired 60% of the applicants in an unprotected class and 40% in a protected class the calculation is 40/60 or 66% and does not satisfy the 80% threshold. However 40% versus 50% would be determined as 80% and would meet the requirement.

This guideline dates back to 1971 when the State of California Fair Employment Practice Commission (FEPC) put together a working group of 32 specialists to make this decision. By 1978 it was codified at the federal level by the EEOC and the DOJ for Title VII enforcement.

So Is Bias in Modeling Really a Problem?

Well yes and no. Our managed markets seem to be doing a respectable job. Not possibly approximately the statistical standards of information science. They don’t safeguard versus all 4 kinds of predisposition identified by the University of Chicago, however from a practical perspective, there are not a lot of grievances.

The public sector as a “high stakes” arena deserves our attention, however of the limited variety of examples put forth to prove bias, only one, COMPAS plainly highlights a statistical bias issue.

Still, provided the quick expansion of analytics and the restricted data science talent in this sector, I elect watching on it. Not nevertheless with the requirement to right away stop utilizing algorithms at all.

To paraphrase a well turned observation, if I examine the danger of modeling predisposition on a scale of 0 to 10, where 0 is the tooth fairy and 10 is Armageddon, I’m going to offer this about a 2 till better proof exists.

Other short articles by Bill Vorhies.

About the author: Bill Vorhies is Editorial Director for Data Science Central and has actually practiced as a data researcher because 2001. He can be reached at:

Please follow and like us: