Reblogging 2008: Failing to recognize your own incompetence

From the “10 years of keetblog – reblogging: 2008”: On those uncomfortable truths on the difference between knowing what you don’t know and not knowing what you don’t know… (and one of the earlier Ig Nobel prize winners 15 years ago)

Failing to recognize your own incompetence; Aug 25, 2008

 

Somehow, each time when I mention to people the intriguing 2000 Ig Nobel prize winning paper “Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments” [1], they tend to send (non)verbal signals demonstrating a certain discomfort. Then I tone it down a bit, saying that one could argue about the set up of the experiment that led Kruger & Dunning to their conclusion. Now—well, based on material from a few years ago but I found out recently—I cannot honestly say that anymore either. A paper from the same authors, “Why people fail to recognize their own incompetence” [2], reports not only more of their experiments in different settings, but also different experiments by other researchers validating the basic tenet that ignorant and incompetent people do not realize they are incompetent but rather think more favourably of themselves—“tend to hold overinflated views of their skills”—than can be justified based on their performance.

Yeah, the shoe might fit. Or not. In addition to the lower end of the scale overestimating their competencies by a large margin, the converse happens, though to a lesser extent, at the other end of the scale, where top-experts underestimate their actual capabilities. The latter brings it own set of problems and research directions, which I will set aside for the remainder of this blog post. Instead, I will dwell a bit on those people bragging to know this that and the other, but, alas, do not perform properly and, moreover, do not even realize they do not. Facing a person who knows s/he does not have the required skills is one thing and generally s/he’s willing to listen and learn or say to not care about it, but those people who do not realize the knowledge/skills gap they have are, well, a hopeless bunch futile to waste your time on (unless you teach them anyway).

 Let us have a look what those psychologists provided to come to this conclusion. Aside from the experiment about jokes in the ’99 paper, which are at least (sub)culture-dependent, the data about the introductory-level psychology class taken by 141 students is quite telling. Right after the psych exam, the students were asked about their own estimate of performance & mastery of the course material (relative to other students in their class) and to estimate their raw score of the exam. These were the results ([2] p84, Fig.1):


If you think such kind of data is only observed with undergraduates in psychology, well, then check [2]’s references: debate teams, hunters about their firearms, medical residents (over)estimating their patient-interviewing techniques, medical lab technicians overestimating their knowledge of medical terminology—you name it, the same pattern, even if the subjects were held a carrot of monetary incentive in an attempt to assess themselves honestly.

 Imagine you going to a GP or doctor of a regional hospital who has the arrogance to know it all and does not call in a specialist on time. One can debate about the harmfulness or harmlessness about such cases. A very recent incident I observed was where x1 and x2 demanded from y to do nonsensical task z. Task z—exemplifying ignorance and incompetence of x1 and x2—was not carried out by y for it could not be done, but it was nevertheless used by x1 and x2 to “demonstrate” “(inherent) incompetence” of y because y did not do task z, whereas, in fact, it the only thing it shows is that y, unlike x1 and x2, may actually have realized z could not be done, hence, understand z better than x1 and x2 do. One’s incompetence [in this case, of x1 and x2] can have far-reaching effects on others around oneself. Trying to get x1 and x2 to realize their shortcomings has not worked thus far. Dunning et al’s students, however, had exam results for unequivocal feedback and there was an additional test set up with a controlled setting where they had built-in a lecture to teach the incompetent so as to rate their competencies better (which worked to some extent), but in real life those options are not always available. What options are available, if any? A prevalent approach I observed here in Italy (well, in academia at least) is that Italians tend to ignore those xs so as to limit as much as possible the ‘air time’ and attention they have, i.e., an avoidance strategy to leave the incompetent be, whereas, e.g., in the Netherlands people will tend to keep on talking until they have blisters on their tongues (figuratively) to try to get some sense in the xs heads, and yet others attempt to sweep things under the carpet and pray there will not appear any wobbles one could fall over. Research directions, let alone some practical suggestions on “how to let people become aware of their intellectual and social deficiencies”—other than ‘teach them’—were not mentioned in the article, but made it to the list of future works.

 You might wonder: does this hold across cultures? The why of the ‘ignorant and unaware of it’ gives some clues that, in theory, culture may not have anything to do with it.

“In many intellectual and social domains, the skills needed to produce correct responses are virtually identical to those needed to evaluate the accuracy of one’s responses… Thus, if people lack the skills to produce correct answers, they are also cursed with an inability to know when their, or anyone else’s, are right or wrong. They cannot recognize their responses as mistaken, or other people’s responses as superior to their own.” ([2], p. 86—emphasis added)

The principal problem has to do with so-called meta-cognition, which “refers to the ability to evaluate responses as correct or incorrect”, and incompetence then entails that one cannot successfully complete such a task; this is a catch-22, but, as mentioned, ‘outside intervention’ through teaching appeared to work and other means are a topic of further investigation. Clearly, a culture of arrogance can make significant stats more significant, but it does not change the principle of the cause. In this respect, the start of the article aptly quotes Confucius: “Real knowledge is to know the extent of one’s ignorance”. Conversely, according to Whitehead (quoted on p. 86 of [2]), “it is not ignorance, but ignorance of ignorance, that is the death of knowledge”.

References

[1] Kruger, J., Dunning, D. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of personality and Social Psychology, 1999, 77: 1121-1134.

[2] Dunning, D., Johnson, K., Ehrlinger, J., Kruger, J. Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 2003, 12(3): 83-87.

 p.s.: I am aware of the fact that I do not know much about psychology, so my rendering, interpretation, and usage of the content of those papers may well be inaccurate, although I fancy the thought that I understood them.

Yet another software-based clicker system: LetsThink – GoAnswer

Every now and then, I dabble into trying to improve on teaching, which in this case of the post’s topic, is in the form of an implementation project. Yes, one of those with working software as result, thanks to the programmer and CS honours student Yaseen Hamdulay who implemented it. In short: we now have a software-based audience response system that satisfies some pertinent requirements for its use in lectures: math support, figures, and diacritics, and one-click question release and showing the results so as not to hold up the lecture. It will be presented at UCT’s upcoming Teaching & Learning Conference on Oct 22-23.

The context

Peer instruction (PI) enables students to learn from each other. Unlike some other educational interventions, PI has been shown to improve the final grade by up to 34-45%, regardless the discipline it is used (computer science, genetics, zoology, psychology—you name it). On top of that, it generally receives positive feedback from students, because it makes the lectures more engaging, or in any case at least less dull.

From the practical side on how to go about using it in class, there are four principal options: expensive hardware-based ‘clickers’, software-based audience response systems (ARSs) with students’ smartphones or laptops, card and picture-taking (image processing), and none of that through indicating the concept test’s answer with one’s fingers on one’s chest. Regarding resources, the software-based ARS sits somewhere in the middle, which suits our setting well. There are quite a few software-based ARSs (see, e.g., this review). However, upon analysis, they all have issues hampering effective classroom use. For instance, they may not have a single-question release or are for-payment only (e.g., Socrative, Google Forms, Pinnion, and eClicker), or have an impractical results display—they make money from corporate accounts, so the tools are geared for that. The major drawbacks of all these systems are that at least the free versions, if not all, have very limited question setting and answer options, mostly having just plaintext. So, no pictures in the question, answers can be only 100 or 140 characters short, let alone displaying mathematical symbols. Perhaps surprisingly in the international arena, they don’t do diacritics either, even though multiple natural languages have them. For the lectures, this ends up really clumsily and cumbersome, where such questions also must have projected on the screen an ‘offline’ version and then toggling with the screen of the voting progress and results.

The problems

Thus, the main problem is that there is no software-based ARS that contains basic required features for smooth use of PI in the classroom irrespective of one’s discipline, hampering the uptake of this active learning intervention. The current software-based ARSs are prohibitive in uptake of PI in the classroom, yet in our—and many others—medium-level resource setting, extensive use of hardware clickers is not a sustainable option, nor are structural expenses of the user-based payment schemes an option, for they effectively discourage broad participation due to the by-participant payment scheme.

The solution

To address these issues, we developed an ARS that has the following features: support for figures, mathematical formulae, proper HTML text for display of diacritics, one-click question release, showing voting progress, results, question reset, saving of the voting results of a session (among others). This is a web-accessible ARS that can cater for various disciplines within one software system and that is simple and fast to work with.

The project page has several screenshots and the link to the tool. It requires a simple registration (at this stage, we don’t bother with email verification—so don’t lose your password), but is then also immediately usable.

To brighten up this post, here are some screenshots; others are on the project page and some have explanations in a previous post on using PI in a networks course, like the one that has maths and a picture in one.

Screenshot of a question with a picture:

Question with a figure

Question with a figure

Note that the lecturer controls (bottom left, and top-left to go back) are intentionally kept in small font. It’s easily readable from the lecturer’s machine, but it’s irrelevant to read for the audience.

Screenshot of a question with some maths, with first the interface for creating the question, then the one projected, and finally the student’s interface when voting. Note that this is just to give an impression that you can do it, not that it is in any way a sensible concept test. Anything that can be done with the latex plugin for html (MathJax) can be used in the question and in the answer box, but thus not your own macros. (For the curious: the answer to the question is A (read the problem).)

Interface for creating a question

Interface for creating a question

The question rendered in the presentation interface

The question rendered in the presentation interface

Small screenshot of the voting interface

Small screenshot of the voting interface

Screenshot of a question with diacritics:

Some question with diacritics.

Desktop/laptop interface of the question to answer, after having entered the question ID:

Section of the lecturer/admin interface, where you can choose to group questions

Exported results from the voting imported into a spreadsheet:

LetsThink-GoAnswer has been beta-tested with my networks students last semester, has been demo-ed in a CILT seminar on ARSs, and the occasional lecturer already is/will be experimenting with it. The code will be made available on GitHub soon. In the meantime, please don’t hesitate to contact me if you have any questions, and we’d love to hear your feedback!

Fruitful ADBIS’15 in Poitiers

The 19th Conference on Advances in Databases and Information Systems (ADBIS’15) just finished yesterday. It was an enjoyable and well-organised conference in the lovely town of Poitiers, France. Thanks to the general chair, Ladjel Bellatreche, and the participants I had the pleasure to meet up with, listen to, and receive feedback from. The remainder of this post mainly recaps the keynotes and some of the presentations.

 

Keynotes

The conference featured two keynotes, one by Serge Abiteboul and on by Jens Dittrich, both distinguished scientists in databases. Abiteboul presented the multi-year project on Webdamlog that ended up as a ‘personal information management system’, which is a simple term that hides the complexity that happens behind the scenes. (PIMS is informally explained here). It breaks with the paradigm of centralised text (e.g., Facebook) to distributed knowledge. To achieve that, one has to analyse what’s happening and construct the knowledge from that, exchange knowledge, and reason and infer knowledge. This requires distributed reasoning, exchanging facts and rules, and taking care of access control. It is being realised with a datalog-style language but that then also can handle a non-local knowledge base. That is, there’s both solid theory and implementation (going by the presentation; I haven’t had time to check it out).

The main part of the cool keynote talk by Dittrich was on ‘the case for small data management’. From the who-wants-to-be-a-millionaire style popquiz question asking us to guess the typical size of a web database, it appeared to be only in the MBs (which most of us overestimated), and sort of explains why MySQL [that doesn’t scale well] is used rather widely. This results in a mismatch between problem size and tools. Another popquiz question answer: the 100MB RDF can just as well be handled efficiently by python, apparently. Interesting factoids, and one that has/should have as consequence we should be looking perhaps more into ‘small data’. He presented his work on PDbF as an example of that small data management. Very briefly, and based on my scribbles from the talk: its an enhanced pdf where you can access the raw data behind the graphs in the paper as well (it is embedded in it, with OLAP engine for posing the same and other queries), has a html rendering so you can hover over the graphs, and some more visualisation. If there’s software associated with the paper, it can go into the whole thing as well. Overall, that makes the data dynamic, manageable, traceable (from figure back to raw data), and re-analysable. The last part of his talk was on his experiences with the flipped classroom (more here; in German), but that was not nearly as fun as his analysis and criticism of the “big data” hype. I can’t recall exactly his plain English terms for the “four V4”, but the ‘lots of crappy XML data that changes’ remained of it in my memory bank (it was similar to the first 5 minutes of another keynote talk he gave).

 

Sessions

Sure, despite the notes on big data, there were presentations in the sessions that could be categorised under ‘big data’. Among others, Ajantha Dahanayake presented a paper on a proposal for requirements engineering for big data [1]. Big data people tend to assume it is just there already for them to play with. But how did it get there, how to collect good data? The presentation outlined a scenario-based backwards analysis, so that one can reduce unnecessary or garbage data collection. Dahanayake also has a tool for it. Besides the requirements analysis for big data, there’s also querying the data and the desire to optimize it so as to keep having fast responses despite its large size. A solution to that was presented by Reuben Ndindi, whose paper also won the best paper award of the conference [2] (for the Malawians at CS@UCT: yes, the Reuben you know). It was scheduled in the very last session on Friday and my note-taking had grinded to a halt. If my memory serves me well, they make a metric database out of a regular database, compute the distances between the values, and evaluate the query on that, so as to obtain a good approximation of the true answer. There’s both the theoretical foundation and an experimental validation of the approach. In the end, it’s faster.

Data and schema evolution research is alive and well, as were time series and temporal aspects. Due to parallel sessions and my time constraints writing this post, I’ll mention only two on the evolution; one because it was a very good talk, the other because of the results of the experiments. Kai Herrmann presented the CoDEL language for database evolution [3]. A database and the application that uses it change (e.g., adding an attribute, splitting a table), which requires quite lengthy scripts with lots of SQL statements to execute. CoDEL does it with fewer statements, and the language has the good quality of being relationally complete [3]. Lesley Wevers approached the problem from a more practical angle and restricted to online databases. For instance, Wikipedia does make updates to their database schema, but they wouldn’t want to have Wikipedia go offline for that duration. How long does it take for which operation, in which RDBMS, and will it only slow down during the schema update, or block any use of the database entirely? The results obtained with MySQL, PostgreSQL and Oracle are a bit of a mixed bag [4]. It generated a lively debate during the presentation regarding the test set-up, what one would have expected the results to be, and the duration of blocking. There’s some work to do there yet.

The presentation of the paper I co-authored with Pablo Fillottrani [5] (informally described here) was scheduled for that dreaded 9am slot the morning after the social dinner. Notwithstanding, quite a few participants did show up, and they showed interest. The questions and comments had to do with earlier work we used as input (the metamodel), qualifying quality of the conceptual model, and that all too familiar sense of disappointment that so few language features were used widely in publicly available conceptual models (the silver lining of excellent prospects of runtime usage of conceptual models notwithstanding). Why this is so, I don’t know, though I have my guesses.

 

And the other things that make conference useful and fun to go to

In short: Networking, meeting up again with colleagues not seen for a while (ranging from a few months [Robert Wrembel] to some 8 years [Nadeem Iftikhar] and in between [a.o., Martin Rezk, Bernhard Thalheim]), meeting new people, exchanging ideas, and the social events.

2008 was the last time I’d been in France, for EMMSAD’08, where, looking back now, I coincidentally presented a paper also on conceptual modelling languages and logic [6], but one that looked at comprehensive feature coverage and comparing languages rather than unifying. It was good to be back in France, and it was nice to realise my understanding and speaking skills in French aren’t as rusty as I thought they were. The travels from South Africa are rather long, but definitely worthwhile. And it gives me time to write blog posts killing time on the airport.

 

References

(note: most papers don’t show up at Google scholar yet, hence, no links; they are on the Springer website, though)

[1] Noufa Al-Najran and Ajantha Dahanayake. A Requirements Specification Framework for Big Data Collection and Capture. ADBIS’15. Morzy et al. (Eds.). Springer LNCS vol. 9282, .

[2] Boris Cule, Floris Geerts and Reuben Ndindi. Space-bounded query approximation. ADBIS’15. Morzy et al. (Eds.). Springer LNCS vol. 9282, 397-414.

[3] Kai Herrmann, Hannes Voigt, Andreas Behrend and Wolfgang Lehner. CoDEL – A Relationally Complete Language for Database Evolution. ADBIS’15. Morzy et al. (Eds.). Springer LNCS vol. 9282, 63-76.

[4] Lesley Wevers, Matthijs Hofstra, Menno Tammens, Marieke Huisman and Maurice van Keulen. Analysis of the Blocking Behaviour of Schema Transformations in Relational Database Systems. ADBIS’15. Morzy et al. (Eds.). Springer LNCS vol. 9282, 169-183.

[5] Pablo R. Fillottrani and C. Maria Keet. Evidence-based Languages for Conceptual Data Modelling Profiles. ADBIS’15. Morzy et al. (Eds.). Springer LNCS vol. 9282, 215-229.

[6] C. Maria Keet. A formal comparison of conceptual data modeling languages. EMMSAD’08. CEUR-WS Vol-337, 25-39.