Chapter 18 Using the ROCK for Cognitive Interviews
Cognitive interviews are a method to study cognitive validity (see Cognitive interviews, chapter 4 in this version of the book, for an introduction). The ROCK supports coding of notes or transcripts from cognitive interviews, and this chapter introduces this functionality.
There are two ROCK patterns specific to cognitive interviews:
The unique item identifier, or
uuidfor short. To specify which item a part of the source describes, unique item identifiers can be added using the pattern[[uuid:item_identifier]], whereitem_identifiershould be replaced with the unique item identifier itself (for example,item_1oricecream_question).The cognitive interview code, which is denoted with
ci. For example, to code a problem with comprehension, the pattern[[ci:comprehension]]can be added.
Both unique item identifiers and cognitive interview codes can only contain lowercase and uppercase Latin letters (a-z and A-Z), Arabic numerals (0-9), and underscores (_). They cannot contain spaces or other special characters. If you do use those, the identifiers or codes containing them will be ignored.
This is an example of a very brief source that represents notes from a cognitive interview as coded using the ROCK standard:
###-----------------------------------------------------------------------------
How large is your family? [[uiid:familySize_7djdy62d]]
Participant also counts 3 dogs and 7 goldfish [[ci:understanding]]
Participant does not count own brothers and sisters, only their partner and children [[ci:retrieval]]
###-----------------------------------------------------------------------------
How many windows are there in your house? [[uiid:windows_7djdy62d]]
Participant does not live in a house, but in an apartment [[ci:content_adequacy]]
Participant also counts windows in doors inside the house [[ci:understanding]]
###-----------------------------------------------------------------------------To specify codes in a source, you can use iROCK (https://i.rock.science; see The iROCK interface, chapter 14 of this version of the book, for an introduction) if you prefer to drag and drop codes, or alternatively, you can use whichever plain text editor you prefer (or you can even use a word processor such as Microsoft Word or LibreOffice Writer, as long as you remember to save the source in plain text format).
Once you have coded one or more sources, you can analyse these using software that implements the ROCK standard, such as a Shiny ROCK app or the rock R package.
18.1 Analysing cognitive interviews using the Shiny ROCK Beryl
A simple Shiny ROCK app exists to produce a heatmap from a single coded cognitive interview transcript or set of notes. You can access it at https://preciousrock.shinyapps.io/shiny-rock-beryl.
The Beryl app only processes the five codes from the Peterson, Peterson & Powell (2017) coding scheme: understanding, retrieval, judgment, response, and content_adequacy (see Cognitive interviews, chapter 4 in this version of the book). In addition, Beryl requires you to use unique item identifiers (uiids) to mark which part of the source covers which item. This is an example of a valid source that Beryl can process:
As you see, each item text is marked with a uiid (specifically, [[uiid:familySize_7djdy62d]] and [[uiid:windows_7djdy62d]]), and each cognitive interview coder is marked as such, too (specifically, [[ci:understanding]], [[ci:retrieval]], and [[ci:content_adequacy]]).
To use the app, open the second tab (marked “🚀 Heatmap”) and copy-paste the text of your coded source (or the example source shown above) into the large text box. Beryl will then show the heatmap below.
You can then tweak the dimensions of the heatmap (width, height, and font size) until you are happy, and download the result. For a high-quality file for sharing, you may want to download the PDF version; if you still want to make manual edits using Inkscape, you may want to download the SVG version; and if you want to embed the figure into a word processor document (e.g. LibreOffice Writer or Microsoft Word), you may want to download the PNG version.
18.2 Analysing cognitive interviews using the Shiny ROCK Amethyst
A slightly more advanced Shiny ROCK app for analysing cognitive interviews is called Amethyst. You can access it at https://preciousrock.shinyapps.io/shiny-rock-amethyst.
Once the app loaded, first upload your sources. To do this, click the “Sources” tab at the top and on that page, click the “Browse…” button to select one or more .rock or .txt files, or one .zip file.
Once the sources are uploaded, you can parse them to process all codes. To do this, click the “Parse Sources” button. If the sources can be parsed succesfully, two additional tabs will appear at the top: “Qualitative Data Table” and “Cognitive Interviews”. You can ignore the first; you don’t need it for processing cognitive interviewing results. Therefore, click on the last tab, “Cognitive Interviews”.
There, you can produce the Cognitive Interview Heatmap. To do that, simply press the “Generate heatmap” button. The heatmap will appear, and you can download it using the “Download heatmap” button. This will download a .png file to your hard disk.
This heatmap shows your unique item identifiers in its rows and the codes you applied in the columns. This provides a quick overview of the types of problems exhibited by each item, quickly allowing you to spot which items are the most and the least problematic.