#SystemsofCareInsights: Policymaker Insights – Rethinking Evaluations and Using Multiple Measures (3/3)

Rethinking Evaluations

Navigating services related to health– housing, employment, transportation, etc.– can be extremely difficult. That’s why state leaders in North Carolina created NCCARE360, a network that provides public access to resources and aids organizations in collaborating on referrals. As the first statewide coordinated care network, NCCARE360 has onboarded over 2,500 organizations and helped over 42,000 users. But how can state leaders even get started creating a network like NCCARE360? Medicaid 1115 waivers grant state leaders the authority to design programs to better serve the health of their populations. With Medicaid 1115 waivers, state leaders can create extensive and effective systems of care networks.  
Before state-level leaders decide to create systems of care, they must acknowledge several valuable insights that can help them make a more significant impact with coordinated care networks. One such insight that policymakers should keep in mind is that evaluations of networks should extend beyond just outcomes.
2
When evaluating a network’s operations, policymakers must look beyond network characteristics and individual leadership. The three metrics of interest within the IBM report (accuracy, efficiency, and effectiveness) provide a lens to better understand how referral network processes perform across key indicators and how the services offered influence referral network performance. State-level policymakers looking to create and manage statewide systems of care must realize that comparing systems on aggregate measures can oversimplify data and lead to unfair performance judgments.
Policymakers also need to work with researchers to establish and adopt a standardized method of describing network performance based on complexity. This standard must align with how policymakers and practitioners evaluate networks, allowing network policy research to expand beyond just analyzing case studies. By choosing a standard categorization system of human services, new knowledge can be gained about the types of activities, systems, and interventions that reliably improve network outcomes without encouraging networks to favor easier services. The Gravity Project is one such national public collaborative aiming to design standardized data sharing on social determinants of health. 
Finally, policymakers must include process metrics in data infrastructure to allow them to course-correct ahead of time rather than after the fact. The insights of these process metrics are timely and will allow policymakers to identify an issue and work towards tailoring a solution immediately. The insights gained from process metrics data can, in turn, be used to create more accurate, efficient, and effective referral technologies in the future. The referral technology market is rapidly growing, and better data infrastructures can be precisely the tool that policymakers need to finely tune their referral systems and internal operations.
All in all, systems of care have accomplished what they aim to do. Coordinated care networks effectively solve the problem of reducing barriers to care and increasing accountability for human services organizations. However, they don’t resolve organizational capacity issues. If networks struggle to maintain the space and resources necessary to fulfill referrals and provide care, systems of care aren’t an automatic fix to this capacity issue. Systems of care also don’t increase knowledge sharing amongst organizations about best practices. Transference of knowledge requires thoughtful decision-making between organizations, agreements to pool resources, and collaboration. Systems of care are a tool, but only one tool that state Departments of Human Services and Medicaid administrators need to manage. Shifting the evaluation focus from the entire network to just the targeted areas for improvement allows policymakers to keep systems of care strong and functional.

Using Multiple Measures

Policymakers looking to create and manage systems of care must address more complex services and client populations instead of penalizing them. As mentioned in our latest blog, nuance is key. To address the various nuances within systems of care, networks must be evaluated across multiple measures.
Policy that includes multiple measures of effectiveness can generate more precise evidence of network functioning and effectiveness, giving communities and governments a better understanding of how the network is operating and where, if anywhere, it’s coming up short. Multiple measures of effectiveness also allow networks to respond with more appropriate interventions to achieve expected community outcomes. When networks can put together a more logical plan structured around the insights they gain from multiple measures, they can address issues more precisely.
Using multiple measures of effectiveness has also proved successful in reducing skimming (skimming refers to individuals manipulating or gaming incentive structures to achieve better performance outcomes). Using a single metric to evaluate performance can encourage undesirable behaviors. For example, measuring network performance based only on wait times (shorter wait times equals better performance) may incentivize networks to provide easy and immediate services to increase their performance scores without actually providing better care. When networks are evaluated across multiple measures of effectiveness, there is less of an opportunity to game the system. Policymakers should also establish policy that includes line items, offering higher rates for more complex services.
To further understand why multiple measures are necessary to provide an accurate view of network performance, policymakers must address complexity. Saitgalina and Council (2020) define complex services as those in which there are, “strict eligibility requirements or the nature of them may not yield an immediate result.” Each client requesting care in a network will vary in the complexity of the services necessary to fulfill their request and deliver positive results. For clients with more complex requests, the Coordination Center may require more time to evaluate the appeal and consider the various eligibility requirements. Furthermore, complex services often require more investment and resources from the network. These long wait times can reflect negatively on performance reports if the network is evaluated across only one measure, such as effectiveness. Using one measure to pass judgment on network performance completely overlooks the impact that complexity has on the operations of a network, and so it is vital that policymakers establish measures that will account for all contributing factors to provide a faithful depiction of network performance.
These insights will preserve networks’ time and effort by eliminating the need for more general interventions. Multiple measures allow networks to generate evidence of issues, figure out tailored solutions to these issues, and carry out these interventions to better their communities without wasting resources.
Want to learn more? Check out the full research article here:
Collaborative Networks: The Next Frontier in Data Driven Management | IBM Center for The Business of Government. https://www.businessofgovernment.org/report/collaborative-networks-next-frontier-data-driven-management.