INTERCULTURAL INSIGHTS: Caveats

Without taking away anything from the value of these guidelines and the conscientious effort of the people who did the rating, I hasten to add that the "top-20" results do not reflect a scientifically controlled study. Residual guilt from several semesters of research design and psychometric statistics requires me to point out the following:

The inter-rater reliability is very low. A few (but not too many) items received the entire range of ratings from 1 to 5. (Long live diversity!)
The criteria used for the rating scale were not clearly pre-specified. (This was intentional.) Different people used different criteria for their evaluation. (For some samples, see the criteria page.) In one sense, we are comparing mangoes and durians.
Some people dropped out in the middle of their rating cycle. (If you are one of them, we hope you will come back to complete the task.) It is possible that if these folks had submitted their ratings, the numbers would have been different.
Different batches were rated by different numbers of people: All batches were rated by at least five different raters. Some batches were rated by as many as 10 different raters.
Some raters had participated in the earlier 101 game of generating the guidelines while others participated only in the EDITOR game. The former group probably had more familiarity with a wider range of criteria.
Participants from the Diane's Intercultural Insights e-group probably had an entirely different mental model compared to participants from some of my other groups who do not have a similar conceptual framework.

I could keep adding to the list but methinks I doth protest too much.

Use these guidelines wisely.

Back to the list of guidelines

Some criteria people used in making their editorial decisions