Originally Posted by sebastian_dangerfield (Post 492534)

I don't see this improving a whole hell of a lot anytime soon. The shift toward hiring/renting/giving credit based on surface information is only increasing. Big data is going to bifurcate us unfairly and arbitrarily more than we were ever able to on our own in the past.

I think lawsuits over discrimination-by-algorithm are going to become more frequent and much larger in the next few decades.

Jumping off a bit, management and governance from the "helicopter view" are going to make a shit show of society, and I don't see any hope of us stopping it. There's a delusion being bought by almost all people with power that huge organizations, markets, and even countries can be fully understood and managed by viewing mere data about how they operate in aggregate.*

This thinking isn't irrational, of course, but we don't have to look far to see its Achilles Heel - the 2008 Crash. The aggregate data never provide a complete picture of what's going on at the street level. God only knows how much damage and unknown risk we'll cause by applying such know-it-all-ism and total reliance on necessarily blunt data in realms beyond finance and insurance.

______
*Wall Street would tell us otherwise, but their broad analyses only appear accurate because its all the same self-reinforcing data passed back and forth between and among the same limited actors, upon which those actors engage in herd investing, the timing of which creates profits. You don't have to have accurate data to win at musical chairs.

Originally Posted by ThurgreedMarshall (Post 492523)

http://www.nytimes.com/2015/01/04/up...?smid=fb-share

TM

Originally Posted by Sidd Finch (Post 492532)

That is really disturbing and upsetting.


Of course, most of those studies happened before we had a black president. it's all better now, right?

Originally Posted by Sidd Finch (Post 492535)

Huh? The whole point of the studies discussed in the article was that there were no differences attributable to "big data" or "algorithm," but only differences in race (based on skin color or names).

Originally Posted by sebastian_dangerfield (Post 492534)

I don't see this improving a whole hell of a lot anytime soon. The shift toward hiring/renting/giving credit based on surface information is only increasing. Big data is going to bifurcate us unfairly and arbitrarily more than we were ever able to on our own in the past.

Originally Posted by sebastian_dangerfield (Post 492534)

I don't see this improving a whole hell of a lot anytime soon. The shift toward hiring/renting/giving credit based on surface information is only increasing. Big data is going to bifurcate us unfairly and arbitrarily more than we were ever able to on our own in the past.

I think lawsuits over discrimination-by-algorithm are going to become more frequent and much larger in the next few decades.

Jumping off a bit, management and governance from the "helicopter view" are going to make a shit show of society, and I don't see any hope of us stopping it. There's a delusion being bought by almost all people with power that huge organizations, markets, and even countries can be fully understood and managed by viewing mere data about how they operate in aggregate.*

This thinking isn't irrational, of course, but we don't have to look far to see its Achilles Heel - the 2008 Crash. The aggregate data never provide a complete picture of what's going on at the street level. God only knows how much damage and unknown risk we'll cause by applying such know-it-all-ism and total reliance on necessarily blunt data in realms beyond finance and insurance.

______
*Wall Street would tell us otherwise, but their broad analyses only appear accurate because its all the same self-reinforcing data passed back and forth between and among the same limited actors, upon which those actors engage in herd investing, the timing of which creates profits. You don't have to have accurate data to win at musical chairs.

Originally Posted by Adder (Post 492539)

Hm. Theoretically big data will actually be able to replace the prejudices of the past with things that are actually predictive.

Originally Posted by Adder (Post 492539)

Hm. Theoretically big data will actually be able to replace the prejudices of the past with things that are actually predictive.

Certainly some things that are predictive will correlate with race (or other historically disadvantaged classes) but others will not, leaving the black people who do not have those characteristics at least arguably better off.

"We won't lend do you because we have a lot of data showing you're a bad credit risk" is a different problem than "we won't lend to you because you're black" and, if the data's really there, substantially less unfair and arbitrary.

Originally Posted by Sidd Finch (Post 492535)

Huh? The whole point of the studies discussed in the article was that there were no differences attributable to "big data" or "algorithm," but only differences in race (based on skin color or names).

Originally Posted by sebastian_dangerfield (Post 492545)

Hence the caveat before the last section, "Jumping off..."

I see a future in which algorithms are used to sort people based on criteria discrimination law was designed to eliminate. And then are used as defenses where found to be doing so. "It wasn't me. It was the computer system." I've seen that defense myself in a discrimination case. It was not successful, but it was a legitimate defense, and can be easily employed given the increasing automation of everything.

Originally Posted by taxwonk (Post 492540)

Of course that thinking is irrational. Pick a stereotype, any stereotype. Now, list the number of people you know who fit it. Compare to the number of people you know that don't.

Originally Posted by taxwonk (Post 492547)

No. It isn't. We all know good and well that any statistical study is only going to reflect the prejudices of the designer. Anything other than a study that it purely quantitative is going to reflect bias that will be both racial and socioeconomic. Unless you're prepared to give a 3-bedroom house on the Main Line exactly the same weighting as a three-bedroom house in the shittiest neighborhood in Philly, your algorithm reflects bias.

Originally Posted by sebastian_dangerfield (Post 492534)

I don't see this improving a whole hell of a lot anytime soon. The shift toward hiring/renting/giving credit based on surface information is only increasing. Big data is going to bifurcate us unfairly and arbitrarily more than we were ever able to on our own in the past.

I think lawsuits over discrimination-by-algorithm are going to become more frequent and much larger in the next few decades.

Jumping off a bit, management and governance from the "helicopter view" are going to make a shit show of society, and I don't see any hope of us stopping it. There's a delusion being bought by almost all people with power that huge organizations, markets, and even countries can be fully understood and managed by viewing mere data about how they operate in aggregate.*

This thinking isn't irrational, of course, but we don't have to look far to see its Achilles Heel - the 2008 Crash. The aggregate data never provide a complete picture of what's going on at the street level. God only knows how much damage and unknown risk we'll cause by applying such know-it-all-ism and total reliance on necessarily blunt data in realms beyond finance and insurance.

______
*Wall Street would tell us otherwise, but their broad analyses only appear accurate because its all the same self-reinforcing data passed back and forth between and among the same limited actors, upon which those actors engage in herd investing, the timing of which creates profits. You don't have to have accurate data to win at musical chairs.

Originally Posted by taxwonk (Post 492541)

Take away the "theoretically" and this may be the dumbest thing ever said. Leave the word in and you're only in the top 10. For what it's worth, I don't really think you believe either one.

Originally Posted by ThurgreedMarshall (Post 492550)

I am having trouble reconciling the fact that you have written English words in order such that they are actual sentences without having the ability to read.

TM

Originally Posted by sebastian_dangerfield (Post 492545)

I see a future in which algorithms are used to sort people based on criteria discrimination law was designed to eliminate.

Originally Posted by Adder (Post 492553)

Why? Because the data crunchers and their bosses just really want to discriminate?

Nah. They want to make money. They will end up using criteria that correlate with race but that they can show actually have meaning for the decision they are making.

Leaving the different and more difficult problem of how to level the playing field.

Originally Posted by sebastian_dangerfield (Post 492549)

It's a legitimate defense in that, a court allowed it to be raised. "Sustainable" is perhaps the better legal term.

ETA: You'd be surprised how many people would disagree with the argument, "there are lies, damned lies, and statistics."

Originally Posted by Adder (Post 492551)

Believe what you like, but I'm a whole lot more comfortable with lending decisions, for example, being made on the basis of a whole lot of data analysis than because the loan officer doesn't like the sound of the name on the form.

Originally Posted by Adder (Post 492553)

Nah. They want to make money. They will end up using criteria that correlate with race but that they can show actually have meaning for the decision they are making.

Originally Posted by sebastian_dangerfield (Post 492552)

Do I have to say I agree with the article? Seems a waste of effort. How couldn't I? I read it twice by the way, having also scanned it in the Times yesterday.

I decided to offer a thought I had after reading it. This seemed the more interesting thing to do.

Originally Posted by sebastian_dangerfield (Post 492554)

Data crunchers can't think fast enough (to borrow the article's term).

Originally Posted by taxwonk (Post 492557)

What criteria?

Originally Posted by Adder (Post 492553)

They want to make money. They will end up using criteria that correlate with race but that they can show actually have meaning for the decision they are making.

Originally Posted by sebastian_dangerfield (Post 492561)

They don't intend to come out anywhere. They merely seek to avoid risk and predict the future for profit. And the actors we're talking about, at least in insurance and finance, will not lose sleep over using prohibited criteria. Particularly where they can blame it on a learning algorithm.

Originally Posted by Adder (Post 492564)

You're still assuming that stereotypes are profit-maximizing. I think that's highly unlikely to be true, especially in a big data world.

Originally Posted by Adder (Post 492559)

It may be true that someone from the wrong neighborhood, the wrong school, the wrong type of job or with the wrong history with the legal system is a worse credit risk, and all of those things may correlate closely with race and lead to results that we think are unfair, but that's a different issue.

Originally Posted by Adder (Post 492560)

Yes, wonk, the data will bear the stamp of our unequal history and thus reliance on it will not bring an even playing field.

As I've said repeatedly, that's a different issue from Sebby's assertion that people will use with the specific intent to discriminate.

They don't need data for that.

Originally Posted by Not Bob (Post 492562)

Pardon my skepticism, but I have long heard arguments about how the Invisible Hand of The Market prevents discrimination because companies that "irrationally" discriminate (i.e., on the basis of race, sex, religion, etc.) would be forced out of business by losing out to companies that "rationally" discriminate (i.e., on the basis of whatever the market is - skills, credit-risk, speed of fastball, etc.).

With the notable exception of professional sports, I don't think that this has occurred.

Originally Posted by sebastian_dangerfield (Post 492563)

No. I am not saying they all intend to discriminate. The overwhelming majority will not. They will intend nothing more than to make money. The discrimination will be largely unintentional. Think of it as algorithmic laziness. Company X finds that, rather than spending time assessing more complex traditional data on hiring, it can shortcut using the number of certain letters in a first name (which just happen to indicate minority background). If the Company finds this is effective and saves time, it will be pressured to use this discriminatory technological advantage.

Originally Posted by taxwonk (Post 492566)

That's a crock and we both know it. The fact that those things correlate with race is unfair and that's NOT a different issue. It's the same issue because those neighborhood lines were drawn that way for a reason, the school sucks because of the place it's located, and the job is directly related to the education, neighborhood, and yes, race or socioeconomic status (something often, but not exclusively, tied to race).

That is the exact issue.

Originally Posted by taxwonk (Post 492567)

They don't need data to have an intent to discriminate. But here you are, yourself, using it as an excuse. That excuse is just that. The intent is to discriminate.

Originally Posted by Greedy,Greedy,Greedy (Post 492571)

Just a point of order, is using the term "heuristic" a corollary to Godwin's Law or the basis for a separate but parallel law?

Originally Posted by taxwonk (Post 492566)

That's a crock and we both know it. The fact that those things correlate with race is unfair and that's NOT a different issue. It's the same issue because those neighborhood lines were drawn that way for a reason, the school sucks because of the place it's located, and the job is directly related to the education, neighborhood, and yes, race or socioeconomic status (something often, but not exclusively, tied to race).

That is the exact issue.

Originally Posted by Adder (Post 492573)

Take it up with Kahnemann and Tversky.

Originally Posted by Greedy,Greedy,Greedy (Post 492575)

They ripped it off from Habermas, anyways. Wherever the word appears, someone is trying to make a small and questionable thought seem big and profound.

Originally Posted by Adder (Post 492570)

It's not the issue we've been discussing, wonk. We've been discussing whether the rise of big data is going to lead to more or less overt discrimination. Sebby contends that it will, while I contend it will not.