Intercoder Reliability

Intercoder Reliability

What is intercoder reliability?

Intercoder reliability is an indicator of measurement consistency. It helps us to determine whether two coders are consistent in evaluating a characteristic of a message.

Why we need to assess and report intercoder reliability?

Although intercoder reliability does not insure validity, it is a crucial component in content analysis. Without a proper establishment of intercoder reliability, the interpretation cannot be valid and likely to be doubted by reviewers. Neuendorf (2002) stated that "given that a goal of content analysis is to identify and record relatively objective (or at least intersubjective) characteristics of messages, reliability is paramount. Without the establishment of reliability, content analysis measures are useless." (p. 141)

How to assess intercoder reliability?

There is no consensus on a single "best" way of assessing the intercoder reliability. It is recommended that researchers select one or more appropriate indices based on the characteristics of their variables, such as:.

1. Percent agreement: Is limited to nominal coding with only two coders with the same number of coded units. Although it is the easiest way to computer inter-coder reliability, there are some drawbacks, and the use of percentage agreement is not usually recommended.

2. Cohen's Kappa: Is a popular and widely cited method for estimating reliability for nominal data

3. Krippendoff's Alpha: Can operate for nominal and ordinal data

Step-by-Step Instructions

Suggested Readings

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological

Measurement, 20(1), 37-46.

Hayes, A. F. & Krippendorff, K. (2007). Answering the call for a standard reliability measure for

coding data. Communication Methods and Measures 1,1:77-89.

Krippendorff, K. (2007). Computing Krippendorff's alpha reliability. Departmental Papers

(ASC), 43.

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.

Riff, D., Lacy, S., & Fico, F. (2014). Analyzing media messages: Using quantitative content

analysis in research. Routledge.

 

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.