兰德-SimCoach-Evaluation_-A-Virtual-Human-Intervention-to-Encourage-Service_145页_3mb

SimCoach Evaluation Summary

Core Content

This report presents a comprehensive evaluation of SimCoach, a virtual human intervention designed to encourage service members and veterans to seek help for posttraumatic stress disorder (PTSD) and depression. The evaluation was conducted by the RAND Corporation at the request of the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE). The report includes both formative and summative evaluations, assessing the program's development process and its effectiveness in promoting help-seeking behavior.

Main Objectives

The primary objectives of the evaluation were:

• To document and assess the development procedures of SimCoach.
• To test the efficacy of SimCoach in increasing the intention to seek help for PTSD and depression.
• To provide recommendations for developers and policymakers based on findings.

Key Findings

Formative Evaluation

• SimCoach Beta was developed using a scalable architecture and an iterative approach that incorporated user feedback.
• The program was aligned with best practices in software development and clinical intervention design.
• However, personalized recommendations were less well-developed, with some cases directing users to web pages that did not include treatment or help-seeking options.
• The distress signal phrases that triggered recommendations were limited, potentially missing some expressions of distress.
• The screening tools used in SimCoach were adapted conversational versions of standardized instruments that had not been fully validated.
• Mobile compatibility was a technical limitation during pilot testing, as many users accessed the program from mobile devices that were not supported.

Summative Evaluation

• A randomized controlled trial (RCT) was conducted with 557 eligible participants who were randomized into three arms: SimCoach, content-matched control, and no-treatment control.
• 280 participants completed the full trial and survey.
• The primary outcome—intention to seek help—did not show a significant effect of SimCoach Beta compared to the control groups.
• Secondary outcomes, such as perceived barriers to care, were also assessed, but no significant differences were found between groups.
• User experience was satisfactory, but it did not translate into a measurable increase in help-seeking intentions.

Key Recommendations

For SimCoach Developers

• Implement best practices for the development of help-seeking interventions.
• Consider new marketing strategies to address user reluctance to interact with a virtual human.
• Use validated screening instruments to ensure reliability, sensitivity, and specificity.
• Adopt an outcome-oriented, iterative development process for future improvements.
• Continue to design dialogue and content that effectively reaches service members, veterans, and their families.
• Develop mobile-compatible versions of SimCoach if future iterations prove effective.
• Explore the use of SimCoach in other contexts involving sensitive health information.

For DCoE and Policymakers

• Consider changing funding models to promote best practices in intervention development, such as requiring pilot data before funding larger projects.
• Support pilot evaluations and dissemination strategies in different contexts.
• Invest in strategies to guide the development of technology-based clinical interventions.
• Play an active role in the design and monitoring of outcome-oriented progress metrics for technology development.

Conclusion

• While SimCoach Beta was developed in line with DoD best practices and users had a satisfactory experience, it did not significantly increase help-seeking intentions.
• The evaluation suggests that user experience may be more important than outcome-oriented development in this context.
• A consensus process for developing best practices in technology-driven behavioral interventions is recommended for future programs.

Document Structure

• Preface, Figures and Tables, Executive Summary, and Acknowledgments provide introductory and methodological context.
• Chapter One introduces the background, objectives, and structure of the report.
• Chapter Two presents the formative evaluation, including software and content assessment.
• Chapter Three details the summative evaluation, including methods, results, and limitations.
• Chapter Four summarizes findings and provides recommendations.
• Appendixes include additional information on revisions to recommendations, distress signal phrases, participant comments, and proposal and evaluation details.

Additional Notes

• The RAND Corporation is a nonprofit, nonpartisan research organization focused on public policy solutions.
• The evaluation was part of the "Innovative Practices for Psychological Health and Traumatic Brain Injury" project.
• The report is available for download or access at the provided URLs.
• The copyright and distribution rights are outlined in the document.

Key Terms

• PTSD: Posttraumatic Stress Disorder
• TBI: Traumatic Brain Injury
• RCT: Randomized Controlled Trial
• SBIRT: Screening, Brief Intervention, and Referral to Treatment
• VR: Virtual Reality
• SME: Subject-Matter Expert
• DCoE: Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury
• ICT: Institute for Creative Technologies
• PHQ-9: Patient Health Questionnaire (9-item version)
• PCL: PTSD Checklist

Limitations

• The evaluation was based on one version of SimCoach (Beta).
• Mobile compatibility was a major limitation.
• The intervention was not designed to deliver psychological health services directly.
• No conclusive evidence of efficacy was found.