318. Metacognitive disabilities

Hi!  I'm back!

Um.  Back.  From being away.  See, I've been --

Well, okay.  *ahem*  Yes, indeed.  I have my notes right here -- just take me a second --

One of the advantages of being on-the-verge-of-a-nervous-breakdown busy and overwhelmed - though not, I hasten to add, quite to the laced-gezpacho verge - is that others step in to do the arduous research necessary to keep this blog going. 

For instance, from one loyal reader - okay, my brother, if you must know, but he still counts - I learned that the validity of post 317 has been scientifically demonstrated.  The eerie thing is that it was demonstrated even before it was written!  In 1999, Justin Kruger and David Dunning published a research paper in the Journal of Personality and Social Psychology entitled: "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments."  Here's the abstract:

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. ...  Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

That, I think, explains a great deal about our legal system, or at least that part of it with which I have professional contact.  To take an example from the media, The New Yorker last spring had a story about the NYPD crime lab's hair and fiber unit (or what the magazine insisted on calling its "hair-and-fibre" unit).  A pair of FBI scientists described a review they performed of 170 hair comparisons, which resulted in 80 "associations" (which, as anybody who's dealt with hair-comparisons knows, is always presented to the jury in terms of "cannot be eliminated" rather than as a "match"):

But subsequent mtDNA tests of the hairs showed that in nine cases—more than ten per cent—the samples could not have come from the same person. The number of errors was concerning, Judge [Harry T.] Edwards said. According to his calculations, he added, the study’s error rate was actually close to thirty-five per cent.

Now, Edwards is one of those people inevitably described as "respected," a term that when applied to a person who has spent many decades exercising near-arbitrary power means little more than that he's never actually been indicted.  Despite all that respect, I'm nonetheless prepared to accept that Judge Edwards is perfectly competent within the sphere of his expertise, that is, the correct standard of review to be applied in an appeal from an agency decision.  (I'm not being snide.) 

But what are the odds that he understands statistics better than a pair of scientists?  Any scientists, it doesn't matter.  Edwards spent five years in the actual practice of law a couple generations ago, then ten years bullying law students, and the rest of his career receiving lawyers bowing and scraping and tugging their forelocks.  He could be a whole lot less ignorant about statistics than the average lawyer and still know less than any recent recipient of a B.S. degree.  By orders of magnitude.

If Edwards had any metacognitive ability to assess his own knowledge of statistics, he wouldn't have spoken so confidently to Jeffrey Toobin about the calculations he scribbled on his hotel-ballroom napkin.   As it is, I'm confident he didn't understand that he was embarrassing himself in a national magazine.  He was too clueless to know he was clueless - and, I suspect, too accustomed to being treated like royalty even to perceive the possibility. 

As it happens, Professor Dunning can explain the second part of that disrespectful evaluation, too, judging from the abstract of a paper in press.  Called "Faulty Self-Assessment: Why Evaluating One’s Own Competence Is an Intrinsically Difficult Task," it makes the point that "although people receive feedback over time that could correct faulty self-assessments, this feedback is often biased, difficult to recognize, or otherwise flawed."  Think for a second about the kind of "feedback" judges get: determined attempts to manipulate by flattery.  And that's just about it.

Arrogance is ignorance backed by power, and American judges are often arrogant.  Imagine, if you can, a world in which judges were required to understand a field of human activity before asserting their authority to regulate it.  For example, what if judges weren't permitted to rule on the reasonableness of a search pursuant to a warrant until they had been the first officer through the door a couple dozen times? 

Comments

[leslie]

I'm glad you're back.

[neilalice]

Me too! I was about ready to start printing out all 317 posts just to get my fix....