Mention a case vignette to anyone who has gone through the certification process for the CANS, ANSA, CAT/CSPI or FAST and you will likely hear groans, and see eye rolls. Very few people are big fans of testing and vignettes are never as clear or easy as a trainee wishes. So, if test vignettes bring up such emotions, why do we still use them for testing?
The Nature of Vignettes
Vignettes are one page summaries that are based on actual cases, created by clinicians who are pursuing trainer certification. These cases can feel one dimensional, and seem to be “missing” important information that you “need” in order to be able to rate the items accurately. But vignettes do have all the information needed to identify needs and strengths and apply the action levels. Nothing provided on delusions, hallucinations, or bizarre thoughts? Rate a “0” on Psychosis; if it’s not mentioned, it’s not a need. No mention of whether the caregiver has family, friends or community connections to help with caregiving? Rate the caregiver a “0” on Social Resources. No evidence about the child’s outlook on life? Rate the child a “3” on Optimism for no evidence of a strength in this area.
The Purpose of Testing
Communimetric tools like the CANS and others noted above are meant to communicate action levels for needs and strengths. We ask: What are the needs/strengths of the individual/family and what needs to be done about them? In order to be confident that we are assessing risk level accurately and because we depend on the accurate output of the assessment to help us make decisions throughout our work with individuals and families, we need to make sure that we can use this tool reliably.
What is reliability? In this case, it means that when two people get information about an individual, they can agree that it either requires or does not require action. While two people may not agree on severity – one might think we must take immediate action while the second thinks action is needed. We refer to reliability as ratings being in the “ballpark,” because the closer you are to the preferred score (the “gold” score for each item in the test vignette), the more reliable you are. You become less reliable when you see evidence of a problem where no evidence exists (i.e., read into it), or when you miss evidence of a problem when evidence is present.
Testing is required in order to ensure that users of the tools are able to identify needs and strengths and use the action levels appropriately. Testing also allows us to confirm that the trainee has become familiar with the six key principles of Communimetric tools and to ensure that they are being applied when rating items. If you think of a Communimetric tool as a ‘common language,’ the vignette allows us to test on people’s mastery of the vocabulary. Remember, this is NOT a test of clinical skills.
Succeeding in “Vignette World”
Vignettes can be challenging, but if we understand that they help us build Communimetric skills, and learn not to “read in” to the vignette then we can accept them as a way to show reliable use of the tool.
Successful certification means mastering the test vignette both practically and emotionally. Use our test-taking tips document which we developed after finding patterns in testing failure. Embrace the concrete nature of the vignette and only rate what is there. Focus on what action is needed (or what strengths can be used to address needs). Vignettes remind us that the work begins with a conversation with the individual and other involved family members to determine where they are now, where they hope to be, and how we can help them get there.
A full discussion of the reliability of the CANS (and other tools) can be found in “Communimetrics: A Communication Theory of Measurement in Human Service Settings” written by Dr. John Lyons. Additional independent research and reviews have been conducted on the reliability and validity of the CANS and other Communimetric tools in this executive summary.
Currently we are using the FAST tool. I am wondering if anyone is using a form of tracker that highlights outcome and measurements that we can speak to regarding individual families?
Thanks in Advance
This article hits on some really important points. Fundamentally, the whole point of TCOM is to give order and structure to the very complicated work that we do. It’s not a simple process of ensuring the criteria of any individual item is met, it’s about bringing this tool to someone’s complicated life in a way that gives it structure and direction. Vignettes model that process: a real situation is in front of you–can you score the tool? Yes, you have to hunt and peck. Yes, some things are not presented (and are thus not needing action.) And if you had the real person in front of you, you could ask more questions to be sure about your hunches … but this is a model of the experience, not the experience itself. Showing that you can do this is the bottom line of doing the work, and ensures a reliably applied system of transformation.
Where can I find the Test Taking Tips document referred to here? Thanks.