Abstract

The Concurrent think-aloud (CTA) is primarily used to understand users’ task based cognitive processes. However, is not without limitations. CTA procedures varies widely among practitioners. Also, it has been known to cause reactivity: an artificial change in task performance. This is problematic because it may alter the accuracy of task performance. Also, research on reactivity within usability testing have shown mixed findings. Thus, conclusions cannot be drawn to attest to
whether reactivity occurs due to varying administration procedures and therefore we must now consider its relationship to other test-based factors. This research will be the first to systematically investigate the impact of task-type on reactivity of the CTA and the first to systematically investigates practitioners working habit in terms of their views on reactivity when using CTA in practice. Three studies were conducted, the first study investigates the Impact of
task-types on the Reactivity of CTA and uses a mixed design. The results suggest that, thinking aloud during usability testing does not cause reactivity, and task type does not impact concurrent think-aloud. However, sensemaking tasks increase mental demand. The second study investigates the
impact of task-type on two different think-aloud protocols and uses a mixed design. The result indicates that, the classic think-aloud method led to more successful task completion and no reactivity, while the explicit instruction produced fewer successful task completions and a higher mental workload. The explicit instruction produced less verbalisation, resulting in fewer relevant explanatory utterances, contradicting expectations. The third study uses an interview method to
explores practitioners' experiences, views on reactivity and challenges when using the think-aloud method within usability testing. These studies demonstrates unequivocally that CTA should not be abandoned in usability studies as it provided valuable think-aloud data and helped identify
usability issues. Additionally, practitioners should not replace the traditional think- aloud approach with explicit instruction, as explicit instruction had a greater influence on participants' behaviour. Ericsson and Simon's recommendations should be used for concurrent data collection, as it ensures data validity and generates the same type of data as explicit instruction while reducing reactivity.