Chapter 18 Using the ROCK for Cognitive Interviews
Cognitive interviews are a method to study cognitive validity (see Cognitive Interviews, Chapter 4 of this version of the book, for an introduction). The ROCK supports coding of notes or transcripts from cognitive interviews, and this chapter introduces this functionality.
There are two ROCK patterns specific to cognitive interviews:
The unique item identifier, or
uuidfor short. To specify which item a part of the source describes, unique item identifiers can be added using the pattern[[uuid:item_identifier]], whereitem_identifiershould be replaced with the unique item identifier itself.The cognitive interview code, which is denoted with
ci. For example, to code a problem with comprehension, the pattern[[ci:comprehension]]can be added.
To specify codes in a source, you can use iROCK (https://i.rock.science; see The iROCK interface, Chapter ?? of this version of the book, for an introduction) if you prefer to drag and drop codes, or alternatively, you can use whichever plain text editor you prefer (or you can even use a word processor such as Microsoft Word or LibreOffice Writer, as long as you remember to save the source in plain text format).
Once you have coded one or more sources, you can analyse these using software that implements the ROCK standard, such as the Shiny ROCK or the rock R package.
18.1 Analysing cognitive interviews using the Shiny ROCK
The Shiny ROCK app for analysing cognitive interviews is called Amethyst. You can access it at https://preciousrock.shinyapps.io/shiny-rock-amethyst.
Once the app loaded, first upload your sources. To do this, click the “Sources” tab at the top and on that page, click the “Browse…” button to select one or more .rock or .txt files, or one .zip file.
Once the sources are uploaded, you can parse them to process all codes. To do this, click the “Parse Sources” button. If the sources can be parsed succesfully, two additional tabs will appear at the top: “Qualitative Data Table” and “Cognitive Interviews”. You can ignore the first; you don’t need it for processing cognitive interviewing results. Therefore, click on the last tab, “Cognitive Interviews”.
There, you can produce the Cognitive Interview Heatmap. To do that, simply press the “Generate heatmap” button. The heatmap will appear, and you can download it using the “Download heatmap” button. This will download a .png file to your hard disk.
This heatmap shows your unique item identifiers in its rows and the codes you applied in the columns. This provides a quick overview of the types of problems exhibited by each item, quickly allowing you to spot which items are the most and the least problematic.