Facilitated Collaborative Inquiry: Using Data to Show Differences in Outcomes Part 2/4
by Stephen Shimshock, PhD
Director of Systems-Data and Reporting, Casey Family Programs
—
This post is part 2 of a 4 part series on Facilitated Collaborative Inquiry (FCI). This post builds on the foundational material presented in the previous post . In this post we will explore the first phase of FCI: Identify the Opportunity/Gap.
In general, child welfare agencies aim to improve outcomes for the youth and families they serve. While it is easy to intuitively agree with that statement, it can be a difficult statement to operationalize and measure. Too often we (the child welfare community) leap right into designing solutions (actions we think will improve outcomes) without properly identifying and measuring the outcomes we plan to improve. The first step in FCI is to identify an opportunity for improvement by finding youth who are having different outcomes than others. This creates the time and space for us to get really clear on what we are trying to improve and how we will know if we made a difference.
There are three foundational axioms in FCI that help orient us to finding and understanding differences in outcomes:
- We believe that every youth and family that walks through our doors is a test of our system.
- We believe that the youth and families who have exited from our care can teach us about the quality and efficiency of our system.
- We believe that the youth currently active in our programs, and those yet to come, represent our opportunity to improve (Figure 1).
This foundation is important because it keeps the focus on our system in relation to youth and family outcomes. For example, if we find that a particular group of youth are not achieving outcomes at the same rate as others, we didn’t find “hard to serve youth” but rather, we found youth for whom our system is struggling to meet their unique needs and build upon their unique strengths. How do we adapt to serve that group more effectively? We examine those who have exited for clues (axiom 2). The ultimate goal is to motivate us into action by taking what we learn and applying it to those we are serving today and those yet to come (axiom 3).
Casey Family Programs’ “system” (as per our Practice Model: https://www.casey.org/practice-model/) aims to ensure that youth are safe and feel safe, that they have legal and relational permanency, and that youth and families experience improved well-being. With this understanding of our system’s intent, we explore our data to learn which youth are achieving legal and relational permanency and which ones are not. We then examine our Child and Adolescent Needs and Strengths (CANS) data to see which youth are experiencing well-being improvements and which youth are not.
There are several methods you can use to find these differences. We have found that using time-to-event curves with entry cohorts are very effective in showing differences. One option is to use a formal Kaplan-Meier[i] curve, which you can generate using a statistical package such as R to visually show the time it takes to a specific event (e.g., attaining legal permanency) for youth in an entry cohort. We display our data in monthly increments. You can think of it as asking what happened to the youth after 0 days in care, after 30 days in care, after 60 days in care, etc.
Figure 2 shows two time-to-event curves (data are censured at 2 years) for our 2014 entry cohort. The cohort is segmented into two different groups of youth. The group on the left had five or more actionable items (2s or 3s) on their baseline CANS in the Behavioral, Life Functioning, and Risk domains. The group on the right had fewer than five actionable items in those same domains. The x-axis represents days in care (with Casey), and the y-axis represents the percentage of the cohort. The blue line signifies youth in care (it descends as youth exit). The green ascending line signifies those youth that exited to legal permanency and the orange line signifies those youth that exited without legal permanency. It is important to note that as a private foster care agency, Casey may have youth that exit the program but remain in foster care (unlike a state system in which youth are in care until they achieve permanency or age out).
From this graph (generated in Tableau) you can see that youth with five or more actionable items at entry were less likely (38%) to achieve permanency after 2 years than those youth that entered with fewer than five actionable items (62%).
This is just one example of a way to visualize differences in outcomes. As you explore your data you can find other patterns. You can even test hypotheses (this can be especially powerful if the hypotheses are coming from front line social workers). For example, you may want to know if a certain age threshold has an impact on outcomes. Visualizing the differences in outcomes and isolating specific issues will help identify youth that would benefit the most from more focused attention. Remember, we are finding youth for whom our system is struggling to meet their unique needs and build on their unique strengths. Interventions will not just be focused on youth and families, but will likely need to focus on systemic changes as well.
Once we have identified a subgroup of youth having different outcomes than others, the next step is to engage front-line staff in unpacking what the patterns mean. We need the “story” behind the data. In the next post, I will talk about how we engage staff in collecting stories about past successes and challenges with identified subgroups of youth. These stories serve as the foundation of their hypothesis or theory of what they think is causing the pattern. This analysis is a critical component of FCI, and it positions front-line staff as the primary analyzers of the data. This step emphasizes the “Collaborative” aspect of Facilitated Collaborative Inquiry.
References
[i]. Rich J. T., Neely J. G., Paniello R. C., Voelker C. C., Nussenbaum B., et al., 2010. A Practical Guide to Understanding Kaplan-Meier Curves. Retrieved July 11, 2017, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3932959/
For more information on FCI please contact Stephen Shimshock, Director of System, Data and Reporting at Casey Family Programs 206-216-4178 (sshimshock@casey.org)