By Matilda Le
Assessing the effectiveness of veteran referral networks has traditionally focused on the volume of services provided, but this approach overlooks the crucial question: Are these services truly enhancing client outcomes in the long run? The Trajectories of Care (ToC) metric, developed by NNSI researchers, offers a solution by providing a more client-centered evaluation of how veterans navigate care networks and how their needs evolve. In this interview, researchers Qiwen Zhang and Marwa Tahboub dive into their current research project, explaining the development, importance, and application of the ToC metric as an innovative way to assess the performance of referral networks and understand their impact on veterans’ well-being.
Could you explain how the ToC metric works and why it is important in evaluating veteran referral networks?
Tahboub: The ToC metric is a cost-effective approach for referral networks to evaluate returning clients’ outcomes. It refers to the dynamic pathways that clients follow while navigating services through the referral network system over time. Traditionally, networks have been evaluated on the demand and supply of services. A network is “good” if it can refer people to service providers that give a wide range of support to people’s needs. Very few care about an individual-level understanding of network effectiveness, that is, clients’ outcomes. Do services improve clients’ general well-being? We try to fill this gap using our metric.
What are some limitations of the current evaluation metrics for referral networks that motivated you to develop the ToC metric?
Tahboub: Our main goal in developing the ToC metric is to address the limitations of the current evaluation metrics, which focus on quantifying service volume or evaluating referral processing metrics. We find that these metrics are insufficient for understanding client outcomes over time. The ToC changes this and gives networks the opportunity to understand how their clients experience the referral process as their needs may change.
Zhang: Some networks (like Combined Arms and AmericaServes, the networks we are examining in this research) have some outcome evaluation metrics. For example, AmericaServes providers record the outcomes of each referral. Combined Arms uses client surveys to self-evaluate their well-being. However, the former requires providers to enter the referral results; the latter is not a mandatory survey, resulting in a low response rate. Compared to these two approaches, the ToC metric is more cost-effective.
What factors influence a client’s trajectory of care?
Zhang: Demographics, services, and the number of services requested at first influence a client’s trajectory of care. Demographically, being a female, served during or after the Vietnam War, being an African-American, and owning a residence have negative impacts on clients’ trajectories. Service-wise, we found that requesting basic-level services (e.g., employment, mental & behavioral health, physical health, etc.) in the beginning positively influences clients’ trajectories. On the other hand, requesting higher-level services (e.g., education, legal assistance, social & spiritual enrichment, sports & recreation, etc.) at first has a negative impact on the trajectories. Clients who requested more types of services at first also tend to have more positive trajectories of care.
What types of data are required to calculate the ToC metric, and how is this data gathered across different networks?
Tahboub: All data in this research is system data gathered by a community resource referral platform. System data includes demographics, referral data, organizations that provide services, service types, timestamps of case creation and closures, and referral outcomes.
After rounds of test ratings by experts in coordinated care (i.e., coordination center employees and research staff members from our partner at Syracuse University), we developed a formula to calculate the ToC metric. The metric is based on four elements:
- First, the metric includes whether the same service is repeatedly requested. If clients repeatedly ask for the same service, it means clients’ needs haven’t been met, and our metric punishes that every time it occurs
- Next, the metric includes the changes in the acuteness of services that are requested. If a client first requests housing and later asks for volunteer programs, their housing problems are probably resolved and their needs become less acute.
- Then, the metric evaluates changes in the volume of services requested over time – if a client requests a bunch of services at first and gradually requests fewer services over time, the client is very likely doing better and the metric awards that, too.
- Finally, among all the services, we found housing and shelter particularly crucial because everything else builds on top of it – clients need stable housing to apply for banking accounts, find a job, and become self-sufficient. So, the metric also reflects that by making a deduction if housing is requested more than once.
How could the ToC metric be used to inform policy changes or funding decisions for coordinated care networks?
Zhang: Since the ToC metric provides a new client-centered approach, veteran care networks like AmericaServes and Combined Arms are better equipped to identify high-priority clients. Rather than focusing on the volume of cases or measuring success by the number of referrals, these networks can holistically understand how clients enter with certain needs and navigate through their systems as their needs change, subsequently affecting their outcomes over time.
There is a growing emphasis on building successful health and social care networks that can also show that they are effective through their evaluations. This paper introduces the ToC as a cost-effective approach for these referral networks to evaluate client outcomes for returning clients. As the demand for more effective and efficient care grows, the ToC metric offers a promising solution for building stronger, more responsive care systems for veterans, ultimately driving positive change in their lives. NNSI director Michelle Shumate and researchers Zhang and Tahobub will present their research at the 2025 SIREN National Research meeting in February. We’ll update the resource directory with that presentation after February 2025.