Skip to main content
SpringerLink
Log in

A Clinical Leadership Lens on Implementing Progress Feedback in Three Countries: Development of a Multidimensional Qualitative Coding Scheme

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Background

Progress feedback, also known as measurement-based care (MBC), is the routine collection of patient-reported measures to monitor treatment progress and inform clinical decision-making. Although a key ingredient to improving mental health care, sustained use of progress feedback is poor. Integration into everyday workflow is challenging, impacted by a complex interrelated set of factors across patient, clinician, organizational, and health system levels. This study describes the development of a qualitative coding scheme for progress feedback implementation that accounts for the dynamic nature of barriers and facilitators across multiple levels of use in mental health settings. Such a coding scheme may help promote a common language for researchers and implementers to better identify barriers that need to be addressed, as well as facilitators that could be supported in different settings and contexts.

Methods

Clinical staff, managers, and leaders from two Dutch, three Norwegian, and four mental health organizations in the USA participated in semi-structured interviews on how intra- and extra-organizational characteristics interact to influence the use of progress feedback in clinical practice, supervision, and program improvement. Interviews were conducted in the local language, then translated to English prior to qualitative coding.

Results

A team-based consensus coding approach was used to refine an a priori expert-informed and literature-based qualitative scheme to incorporate new understandings and constructs as they emerged. First, this hermeneutic approach resulted in a multi-level coding scheme with nine superordinate categories and 30 subcategories. Second-order axial coding established contextually sensitive categories for barriers and facilitators.

Conclusions

The primary outcome is an empirically derived multi-level qualitative coding scheme that can be used in progress feedback implementation research and development. It can be applied across contexts and settings, with expectations for ongoing refinement. Suggestions for future research and application in practice settings are provided. Supplementary materials include the coding scheme and a detailed playbook.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Arbuckle, M. R., Foster, F. P., Talley, R. M., Covell, N. H., & Essock, S. M. (2020). Applying motivational interviewing strategies to enhance organizational readiness and facilitate implementation efforts. Quality Management in Health Care, 29(1), 1–6. https://doi.org/10.1097/QMH.0000000000000234.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barber, J., & Resnick, S. G. (2022). Collect, share, Act: A transtheoretical clinical model for doing measurement-based care in mental health treatment. Psychological Services. https://doi.org/10.1037/ser0000629.

    Article  PubMed  Google Scholar 

  • Barkham, M., De Jong, K., Delgadillo, J., & Lutz, W. (2023). Routine outcome monitoring (ROM) and feedback: Research review and recommendations. Psychotherapy Research, 0(0), 1–15. https://doi.org/10.1080/10503307.2023.2181114.

    Article  Google Scholar 

  • Baumhauer, J. F., & Bozic, K. J. (2016). Value-based healthcare: Patient-reported outcomes in clinical decision making. Clinical Orthopedics and Related Research, 474(6), 1375–1378. https://doi.org/10.1007/s11999-016-4813-4.

    Article  Google Scholar 

  • Bickman, L., Douglas, S., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. https://doi.org/10.1007/s10488-015-0647-8.

    Article  PubMed  Google Scholar 

  • Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2013). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25(1), 6–19. https://doi.org/10.1080/10503307.2013.817696.

    Article  PubMed  Google Scholar 

  • Boswell, J. F., Hepner, K. A., Lysell, K., Rothrock, N. E., Bott, N., Childs, A. W., Douglas, S., Owings-Fonner, N., Wright, C. V., Stephens, K. A., Bard, D. E., Aajmain, S., & Bobbitt, B. L. (2023). The need for a measurement-based care professional practice guideline. Psychotherapy, 60(1), 1–16. https://doi.org/10.1037/pst0000439.

    Article  PubMed  Google Scholar 

  • Bovendeerd, B., De Jong, K., De Groot, E., Moerbeek, M., & De Keijser, J. (2022). Enhancing the effect of psychotherapy through systematic client feedback in outpatient mental healthcare: A cluster randomized trial. Psychotherapy Research, 32(6), 710–722. https://doi.org/10.1080/10503307.2021.2015637.

    Article  PubMed  Google Scholar 

  • Bovendeerd, B., De Jong, K., de Groot, E., Moerbeek, M., Hafkenscheid, A., & de Keijser, J. (2023). The effect of therapist characteristics on the use and outcome of systematic client feedback in outpatient mental healthcare. Clinical Psychology & Psychotherapyn/a(n/a). https://doi.org/10.1002/cpp.2873.

    Article  Google Scholar 

  • Boyd, M. R., Powell, B. J., Endicott, D., & Lewis, C. C. (2018). A method for tracking implementation strategies: An exemplar implementing measurement-based care in community behavioral health clinics. Behavior Therapy, 49(4), 525–537. https://doi.org/10.1016/j.beth.2017.11.012.

    Article  PubMed  Google Scholar 

  • Brattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C., Linaker, O. M., Ryum, T., Wampold, B., Lara-Cabrera, M. L., & Iversen, V. C. (2018). The effects of routine outcome monitoring (ROM) on therapy outcomes in the course of an implementation process: A randomized clinical trial. Journal of Counseling Psychology, 65(5), 641–652. https://doi.org/10.1037/cou0000286.

    Article  PubMed  Google Scholar 

  • Carlier, I. V. E., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104–110. https://doi.org/10.1111/j.1365-2753.2010.01543.x.

    Article  PubMed  Google Scholar 

  • Castonguay, L. G., Barkham, M., Youn, S. J., & Page, A. C. (2021). Practice-based evidence – Findings from routine clinical settings. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change (50th Anniversary Edition) (7th ed., pp. 191–222). John Wiley and Sons.

  • Centers for Medicare & Medicaid Services (2016). CMS Quality Measure Development Plan: Supporting the Transition to the Merit-based Incentive Payment System (MIPS) and Alternative Payment Models (APMs)

  • Coalition for the Advancement and Application of Psychological Science (2018). Evidence-based practice decision-making for mental and behavioral health care.

  • Connors, E. H., Douglas, S., Jensen-Doss, A., Landes, S. J., Lewis, C. C., McLeod, B. D., Stanick, C., & Lyon, A. R. (2021). What gets measured gets done: How mental health agencies can leverage measurement-based care for better patient care, clinician supports, and organizational goals. Administration and Policy in Mental Health and Mental Health Services Research, 48(2), 250–265. https://doi.org/10.1007/s10488-020-01063-w.

    Article  PubMed  Google Scholar 

  • Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), https://doi.org/10.1186/1748-5908-4-50.

  • De Jong, K. (2016). Challenges in the implementation of Measurement Feedback systems. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 467–470. https://doi.org/10.1007/s10488-015-0697-y.

    Article  PubMed  Google Scholar 

  • De Jong, K., Conijn, J. M., Gallagher, R. A. V., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021). Using progress feedback to improve outcomes and reduce drop-out, treatment duration, and deterioration: A multilevel meta-analysis. Clinical Psychology Review, 85, 102002. https://doi.org/10.1016/j.cpr.2021.102002.

    Article  PubMed  Google Scholar 

  • De Jong, K., Douglas, S., Wolpert, M., Delgadillo, J., Aas, B., Bovendeerd, B., Carlier, I. V. E., Compare, A., Edbrooke-Childs, J., Janse, P. D., Lutz, W., Moltu, C., Nordberg, S., Poulsen, S., Rubel, J., Schiepek, G. K., Schilling, V., van Sonsbeek, M. A. M. S., & Barkham, M. (2023). Using progress feedback to enhance treatment outcomes: A review. Clinical Psychology Unit, Institute of Psychology, Leiden University. Manuscript submitted for publication.

  • Dedoose web application for managing, analyzing, and presenting qualitative and mixed method research data (Version 9.0.62). (2022). SocioCultural Research Consultants, LLC. www.dedoose.com.

  • Directorate of Health. (2019). Patient pathways in mental health and addiction treatment. Department of Health. Oslo.

  • Douglas, S., Button, S., & Casey, S. E. (2016). Implementing for sustainability: Promoting use of a measurement feedback system for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 286–291. https://doi.org/10.1007/s10488-014-0607-8.

    Article  PubMed  Google Scholar 

  • Douglas, S., Page, A. C., Moltu, C., Kyron, M., & Satterthwaite, T. (2023). The connections matter: Bi-directional learning in program evaluation and practice-oriented research. Administration and Policy in Mental Health and Mental Health Services Research. Advance online publication. https://doi.org/10.1007/s10488-023-01304-8.

  • Duncan, E. A., & Murray, J. (2012). The barriers and facilitators to routine outcome measurement by allied health professionals in practice: A systematic review. BMC Health Services Research, 12(1), 96. https://doi.org/10.1186/1472-6963-12-96. https://bmchealthservres.biomedcentral.com/articles/.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods, 5(1), 80–92. https://doi.org/10.1177/160940690600500107.

    Article  Google Scholar 

  • Fitzpatrick, J. L. (2012). An introduction to context and its role in evaluation practice. New Directions for Evaluation, 2012(135), 7–24. https://doi.org/10.1002/ev.20024.

    Article  Google Scholar 

  • Forti, A., Nas, C., Geldrop, A. V., Franx, G., & Petrea, I. (2014). Mental Health Analysis Profiles (MhAPs): Netherlands (OECD Health Working Papers No. 73). https://doi.org/10.1787/5jz158z60dzn-en.

  • Fortney, J. C., Sladek, R., Unutzer, J., Kennedy, P., Harbin, H., Emmet, B., Alfred, L., & Carneal, G. (2015). Fixing behavioral health care in America: A national call for measurement-based care in the delivery of behavioral health services. The Kennedy Forum. https://www.thekennedyforum.org/news/measurement-based-care-issue-brief.

  • Fortney, J. C., Unützer, J., Wrenn, G., Pyne, J. M., Smith, G. R., Schoenbaum, M., & Harbin, H. T. (2017). A tipping point for measurement-based care. Psychiatric Services, 68(2), 179–188. https://doi.org/10.1176/appi.ps.201500439.

    Article  PubMed  Google Scholar 

  • Gelkopf, M., Mazor, Y., & Roe, D. (2022). A systematic review of patient-reported outcome measurement (PROM) and provider assessment in mental health: Goals, implementation, setting, measurement characteristics and barriers. International Journal for Quality in Health Care, 34(Supplement_1), ii13–ii27. https://doi.org/10.1093/intqhc/mzz133.

    Article  Google Scholar 

  • Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V., Bickman, L., Douglas, S., & Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 426–440. https://doi.org/10.1007/s10488-015-0642-0.

    Article  PubMed  Google Scholar 

  • Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x/full. http://onlinelibrary.wiley.com/doi/.

    Article  PubMed  PubMed Central  Google Scholar 

  • Greenhalgh, J., Dalkin, S., Gooding, K., Gibbons, E., Wright, J., Meads, D., Black, N., Valderas, J. M., & Pawson, R. (2017). Functionality and feedback: A realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. Health Services and Delivery Research, 5(2), 1–280. https://doi.org/10.3310/hsdr05020.

    Article  Google Scholar 

  • Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010.

    Article  PubMed  Google Scholar 

  • Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O’Neal, L., McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., & Duda, S. N. (2019). The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics, 95, 103208. https://doi.org/10.1016/j.jbi.2019.103208.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hermann, R. C., Chan, J. A., Zazzali, J. L., & Lerner, D. (2006). Aligning measurement-based quality improvement with implementation of evidence-based practices. Administration and Policy in Mental Health and Mental Health Services Research, 33(6), 636–645. https://doi.org/10.1007/s10488-006-0055-1.

    Article  PubMed  Google Scholar 

  • Hovland, R. T., Ytrehus, S., Mellor-Clark, J., & Moltu, C. (2023). How patients and clinicians experience the utility of a personalized clinical feedback system in routine practice. Journal of Clinical Psychology, 79(3), 711–728. https://doi.org/10.1002/jclp.22992.

    Article  PubMed  Google Scholar 

  • Hysong, S. J. (2009). Meta-analysis: Audit & feedback features impact effectiveness on Care Quality. Medical Care, 47(3), 356–363. https://doi.org/10.1097/MLR.0b013e3181893f6b.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., & Hawley, K. M. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0.

    Article  PubMed  Google Scholar 

  • Jensen-Doss, A., Douglas, S., Phillips, D. A., Gencdur, O., Zalman, A., & Gomez, N. E. (2020). Measurement-based care as a practice improvement tool: Clinical and organizational applications in youth mental health. Evidence-Based Practice in Child and Adolescent Mental Health, 5(3), 233–250. https://doi.org/10.1080/23794925.2020.1784062.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kitto, S. C., Chesters, J., & Grbich, C. (2008). Quality in qualitative research. Medical Journal of Australia, 188(4), 243–246. https://doi.org/10.5694/j.1326-5377.2008.tb01595.x.

    Article  PubMed  Google Scholar 

  • Krippendorff, K. (2011). Agreement and information in the reliability of coding. Communication Methods and Measures, 5(2), 93–112. https://doi.org/10.1080/19312458.2011.568376.

    Article  Google Scholar 

  • Låver, J., McAleavey, A., Valaker, I., Castonguay, L. G., & Moltu, C. (2023). Therapists’ and patients’ experiences of using patients’ self-reported data in ongoing psychotherapy processes—A systematic review and meta-analysis of qualitative studies. Psychotherapy Research, 1–18. https://doi.org/10.1080/10503307.2023.2222896.

  • Laverty, S. M. (2003). Hermeneutic phenomenology and phenomenology: A comparison of historical and methodological considerations. International Journal of Qualitative Methods, 2(3), 21–35. https://doi.org/10.1177/160940690300200303.

    Article  Google Scholar 

  • Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., Simon, G., & Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329.

    Article  PubMed  Google Scholar 

  • Lewis, C. C., Hannon, P. A., Klasnja, P., Baldwin, L. M., Hawkes, R., Blackmer, J., Johnson, A., Blasi, P., Buist, D., Cole, A., Dorsey, S., Gray, M. F., Henrikson, N. B., Issaka, R., Jones, S., Knerr, S., Lyon, A. R., Palazzo, L., & Panattoni, L. (2021). Optimizing implementation in Cancer Control (OPTICC): Protocol for an implementation science center. Implementation Science Communications, 2(1), 44. https://doi.org/10.1186/s43058-021-00117-w. … OPTICC Consortium, represented by B. J. W.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lewis, C. C., Boyd, M. R., Marti, C. N., & Albright, K. (2022a). Mediators of measurement-based care implementation in community mental health settings: Results from a mixed-methods evaluation. Implementation Science, 17(1), 71. https://doi.org/10.1186/s13012-022-01244-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lewis, C. C., Klasnja, P., Lyon, A. R., Powell, B. J., Lengnick-Hall, R., Buchanan, G., Meza, R. D., Chan, M. C., Boynton, M. H., & Weiner, B. J. (2022b). The mechanics of implementation strategies and measures: Advancing the study of implementation mechanisms. Implementation Science Communications, 3(1), 114. https://doi.org/10.1186/s43058-022-00358-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Liberati, E. G., Ruggiero, F., Galuppo, L., Gorli, M., González-Lorenzo, M., Maraldi, M., Ruggieri, P., Polo Friz, H., Scaratti, G., Kwag, K. H., Vespignani, R., & Moja, L. (2017). What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implementation Science, 12(1), 113. https://doi.org/10.1186/s13012-017-0644-2.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lutz, W., De Jong, K., Rubel, J., & Delgadillo, J. (2021). Measuring, predicting and tracking change in psychotherapy. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and Behavior Change (7th ed., pp. 89–133). Wiley.

  • Lutz, W., Deisenhofer, A. K., Rubel, J., Bennemann, B., Giesemann, J., Poster, K., & Schwartz, B. (2022a). Prospective evaluation of a clinical decision support system in psychological therapy. Journal of Consulting and Clinical Psychology, 90(1), 90–106. https://doi.org/10.1037/ccp0000642.

    Article  PubMed  Google Scholar 

  • Lutz, W., Schwartz, B., & Delgadillo, J. (2022b). Measurement-based and Data-informed psychological therapy. Annual Review of Clinical Psychology, 18(1), 71–98. https://doi.org/10.1146/annurev-clinpsy-071720-014821.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 441–466. https://doi.org/10.1007/s10488-016-0719-4.

    Article  PubMed  Google Scholar 

  • Malterud, K. (2001). Qualitative research: Standards, challenges, and guidelines. Lancet, 358(9280), 483–488. https://doi.org/10.1016/S0140-6736(01)05627-6.

    Article  CAS  PubMed  Google Scholar 

  • Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444.

    Article  PubMed  Google Scholar 

  • Martin, A. M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: A qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16(3), 407–420. https://doi.org/10.1177/1359104510366284. PubMed.

    Article  PubMed  Google Scholar 

  • Maxwell, J. (2009). Designing a qualitative study. In L. Bickman, & D. Rog (Eds.), The SAGE handbook of Applied Social Research methods (pp. 214–253). SAGE Publications, Inc. https://doi.org/10.4135/9781483348858.n7.

  • May, C., & Finch, T. (2009). Implementing, embedding, and integrating practices: An outline of normalization process theory. Sociology, 43(3), 535–554. https://doi.org/10.1177/0038038509103208.

    Article  Google Scholar 

  • May, C., Finch, T., Ballini, L., MacFarlane, A., Mair, F., Murray, E., Treweek, S., & Rapley, T. (2011). Evaluating complex interventions and health technologies using normalization process theory: Development of a simplified approach and web-enabled toolkit. BMC Health Services Research, 11(1), 245. https://doi.org/10.1186/1472-6963-11-245.

    Article  PubMed  PubMed Central  Google Scholar 

  • May, C., Cummings, A., Girling, M., Bracher, M., Mair, F. S., May, C. M., Murray, E., Myall, M., Rapley, T., & Finch, T. (2018). Using normalization process theory in feasibility studies and process evaluations of complex healthcare interventions: A systematic review. Implementation Science, 13(1), 80. https://doi.org/10.1186/s13012-018-0758-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • May, C., Albers, B., Bracher, M., Finch, T. L., Gilbert, A., Girling, M., Greenwood, K., MacFarlane, A., Mair, F. S., May, C. M., Murray, E., Potthoff, S., & Rapley, T. (2022). Translational framework for implementation evaluation and research: A normalisation process theory coding manual for qualitative research and instrument development. Implementation Science, 17, 19. https://doi.org/10.1186/s13012-022-01191-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • McAleavey, A. A., & Moltu, C. (2021). Understanding routine outcome monitoring and clinical feedback in context: Introduction to the special section. Psychotherapy Research, 31(2), 142–144. https://doi.org/10.1080/10503307.2020.1866786.

    Article  PubMed  Google Scholar 

  • McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/.

    Article  PubMed  PubMed Central  Google Scholar 

  • McLeod, B. D., Jensen-Doss, A., Lyon, A. R., Douglas, S., & Beidas, R. S. (2022). To Utility and Beyond! Specifying and advancing the utility of measurement-based care for Youth. Journal of Clinical Child & Adolescent Psychology, 51(4), 375–388. https://doi.org/10.1080/15374416.2022.2042698.

    Article  Google Scholar 

  • Mellor-Clark, J., Cross, S., Macdonald, J., & Skjulsvik, T. (2016). Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 279–285. https://doi.org/10.1007/s10488-014-0587-8.

    Article  PubMed  Google Scholar 

  • Nurjannah, I., Mills, J., Park, T., & Usher, K. (2014). Conducting a grounded theory study in a language other than English: Procedures for ensuring the integrity of translation. SAGE Open, 4(1), https://doi.org/10.1177/2158244014528920.

  • Patton, M. Q. (2014). Qualitative Research & Evaluation Methods: Integrating Theory and Practice (4th edition). SAGE Publications, Inc.

  • Perry, C. K., Damschroder, L. J., Hemler, J. R., Woodson, T. T., Ono, S. S., & Cohen, D. J. (2019). Specifying and comparing implementation strategies across seven large implementation interventions: A practical application of theory. Implementation Science, 14(1), 32. https://doi.org/10.1186/s13012-019-0876-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pfeffer, J., & Sutton, R. I. (2000). The knowing-doing gap: How smart companies turn knowledge into action. Harvard Business School Press.

  • Porter, M. E., Larsson, S., & Lee, T. H. (2016). Standardizing patient outcomes measurement. New England Journal of Medicine, 374(6), 504–506. https://doi.org/10.1056/NEJMp1511701.

    Article  CAS  PubMed  Google Scholar 

  • Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157. https://doi.org/10.1177/1077558711430690. http://journals.sagepub.com/.

    Article  PubMed  Google Scholar 

  • Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), https://doi.org/10.1186/s13012-015-0209-1.

  • Powell, B. J., Mandell, D. S., Hadley, T. R., Rubin, R. M., Evans, A. C., Hurford, M. O., & Beidas, R. S. (2017). Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study. Implementation Science: IS, 12. https://doi.org/10.1186/s13012-017-0593-9.

  • Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 139. https://doi.org/10.1186/1748-5908-8-139.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rognstad, K., Wentzel-Larsen, T., Neumer, S. P., & Kjøbli, J. (2023). A systematic review and meta-analysis of measurement feedback systems in treatment for common mental health disorders. Administration and Policy in Mental Health and Mental Health Services Research, 50(2), 269–282. https://doi.org/10.1007/s10488-022-01236-9.

    Article  PubMed  Google Scholar 

  • Saldańa, J. (2015). The Coding Manual for qualitative researchers. Sage.

  • Sichel, C. E., & Connors, E. H. (2022). Measurement feedback system implementation in public youth mental health treatment services: A mixed methods analysis. Implementation Science Communications, 3(1), 119. https://doi.org/10.1186/s43058-022-00356-5.

    Article  PubMed  PubMed Central  Google Scholar 

  • Skillman, M., Cross-Barnet, C., Singer, R. F., Rotondo, C., Ruiz, S., & Moiduddin, A. (2019). A framework for rigorous qualitative research as a component of mixed method rapid-cycle evaluation. Qualitative Health Research, 279–289. https://doi.org/10.1177/1049732318795675.

  • Solstad, S. M., Castonguay, L. G., & Moltu, C. (2019). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research, 29(2), 157–170. https://doi.org/10.1080/10503307.2017.1326645.

    Article  PubMed  Google Scholar 

  • Tanana, M., Hallgren, K. A., Imel, Z. E., Atkins, D. C., & Srikumar, V. (2016). A comparison of Natural Language Processing methods for Automated Coding of Motivational Interviewing. Journal of Substance Abuse Treatment, 65, 43–50. https://doi.org/10.1016/j.jsat.2016.01.006.

    Article  PubMed  PubMed Central  Google Scholar 

  • The Joint Commission (2018). Revised outcome measures standard for behavioral health care. R3 Report: Requirement, Rationale, Reference. https://www.jointcommission.org/-/media/tjc/documents/accred-and-cert/bhc/rationale-cts030109-revised.pdf.

  • van Sonsbeek, M. A. M. S., Hutschemaekers, G. J. M., Veerman, J. W., Vermulst, A., Kleinjan, M., & Tiemens, B. G. (2021). Challenges in investigating the effective components of feedback from routine outcome monitoring (ROM) in youth mental health care. Child & Youth Care Forum, 50(2), 307–332. https://doi.org/10.1007/s10566-020-09574-1.

    Article  Google Scholar 

  • Waltz, T. J., Powell, B. J., Matthieu, M. M., Damschroder, L. J., Chinman, M. J., Smith, J. L., Proctor, E. K., & Kirchner, J. E. (2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert recommendations for Implementing Change (ERIC) study. Implementation Science, 10(1), 109. https://doi.org/10.1186/s13012-015-0295-0.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. (2016). A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and Diabetes services. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 309–315. https://doi.org/10.1007/s10488-014-0586-9.

    Article  PubMed  Google Scholar 

  • Youn, S. J., Boswell, J. F., Douglas, S., Harris, B. A., Aajmain, S., Arnold, K. T., Creed, T. A., Gutner, C. A., Orengo-Aguayo, R., Oswald, J. M., & Stirman, S. W. (2023). Implementation science and practice-oriented research: Convergence and complementarity. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-023-01296-5.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This study was initially conceptualized from discussions between feedback researchers during the International Network for Psychotherapy Innovations and Research into Effectiveness (INSPIRE) meetings held at Leiden University in 2017 and 2018. The authors would like to thank Julian Edbrooke Childs, Miranda Wolpert, and Günter Schipek who participated in these discussions. Our appreciation goes to Peabody College at Vanderbilt University for partial support of the project with a Peabody small grant award, and to the Vanderbilt University students who helped along the way, including Ariane Willson and Maria Sheridon. Førde Hospital Trust supported Runar Hovland’s participation in the project with a short-term grant. The Dimence Group supported the Netherland’s authors with funding for transcription services.

Author information

Authors and Affiliations

  1. Department of Leadership, Policy and Organizations, Vanderbilt University, Peabody College, 230 Appleton Place PMB #414, Nashville, TN, 37203-5721, USA

    Susan Douglas, Mya Manns, Xavier Patrick Milling, Ke’Sean Tyler & Tim Satterthwaite

  2. Department of Clinical Psychology and Experimental Psychopathology, Faculty of Behavioural and Social Sciences, University of Groningen, Groningen, the Netherlands

    Bram Bovendeerd

  3. Dimence, Center for Mental Health Care, Deventer, the Netherlands

    Bram Bovendeerd

  4. Pro Persona Research, Renkum, the Netherlands

    Maartje van Sonsbeek

  5. WestEd, Chicago, IL, USA

    Nisha Bala

  6. Faculty of Health and Social Sciences, Western Norway University of Applied Sciences, Førde, Norway

    Runar Tengel Hovland

  7. Department of Research and Innovation, Førde Hospital Trust, Førde, Norway

    Runar Tengel Hovland

  8. Villa Sana – Centre for Work Health, Modum Bad, Norway

    Ingunn Amble

  9. Department of Psychology, Bar-Ilan University, Ramat-Gan, Israel

    Dana Atzil-Slonim

  10. Clinical and Applied Psychology Unit, Department of Psychology, University of Sheffield, Sheffield, UK

    Michael Barkham

  11. Clinical Psychology Unit, Institute of Psychology, Leiden University, Leiden, the Netherlands

    Kim de Jong

  12. Primary Care Research Centre, University of Southampton, Southampton, UK

    Tony Kendrick

  13. Department of Behavioral Health, Reliant Medical Group, Worcester, MA, USA

    Samuel S. Nordberg

  14. Department of Psychology, University of Trier, Trier, Germany

    Wolfgang Lutz

  15. Department of Psychology, Psychotherapy Research Lab, Justus Liebig University Giessen, Giessen, Germany

    Julian A. Rubel

  16. Linköping University, Linköping, Sweden

    Tommy Skjulsvik

  17. District General Hospital of Førde, Førde, Norway

    Christian Moltu

  18. Department of Health and Caring Science, Western Norway University of Applied Science, Førde, Norway

    Christian Moltu

Authors
  1. Susan Douglas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bram Bovendeerd
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maartje van Sonsbeek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mya Manns
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xavier Patrick Milling
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ke’Sean Tyler
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nisha Bala
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Tim Satterthwaite
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Runar Tengel Hovland
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ingunn Amble
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Dana Atzil-Slonim
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Michael Barkham
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Kim de Jong
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Tony Kendrick
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Samuel S. Nordberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Wolfgang Lutz
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Julian A. Rubel
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Tommy Skjulsvik
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Christian Moltu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Susan Douglas managed the research efforts and chaired the discussion meetings during which the project was conceptualized, in which Bram Bovendeerd, Maartje van Sonsbeek, Ingunn Amble, Dana Atzil-Slonim, Michael Barkham, Kim de Jong, Tony Kendrick, Samuel S. Nordberg, Wolfgang Lutz, Julian A. Rubel, and Tommy Skjulsvik participated. Except for the last three, the same group conceptualized the research design and development of the research protocol. Susan Douglas, Bram Bovendeerd, Maartje van Sonsbeek, Christian Moltu, Nisha Bala, and Runar Tengel Hovland recruited participants, conducted interviews, developed, and applied the first order qualitative coding scheme. Except for the last two, the same group plus Mya Manns, Xavier Patrick Milling, and Ke’Sean Tyler developed and applied the second order qualitative coding scheme. Analysis of qualitative coding was conducted by Susan Douglas, Tim Satterthwaite, and Bram Bovendeerd. Susan Douglas drafted a first version of the manuscript to which Bram Bovendeerd, Maartje van Sonsbeek, and Christian Moltu contributed. Christian Moltu provided expert guidance on qualitative methods. All authors contributed to subsequent versions of the manuscript and agree with the content of the paper.

Corresponding author

Correspondence to Susan Douglas.

Ethics declarations

Declarations

Vanderbilt University and Susan Douglas receive compensation related to the Peabody Treatment Progress Battery; and Susan Douglas has a financial relationship with MIRAH, and both are Measurement-Based Care (MBC) tools. The author declares a potential conflict of interest. There is a management plan in place at Vanderbilt University to monitor that this potential conflict does not jeopardize the objectivity of Dr. Douglas’ research. Christian Moltu and Samuel S. Nordberg own equity in a Norwegian private company that markets and sells a clinical feedback technology based on the Norse Feedback methodology. The authors declare a potential conflict of interest. There is a management plan in place at Førde Hospital Trust to monitor that this potential conflict does not jeopardize the objectivity of Dr. Moltu’s research. Michael Barkham was the Principal Investigator in the development of the CORE Outcome Measure (1995-98) which has been used in feedback research. He is a CORE System Trustee but receives no financial benefit from the use of the measure.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Supplementary Material 3

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Douglas, S., Bovendeerd, B., van Sonsbeek, M. et al. A Clinical Leadership Lens on Implementing Progress Feedback in Three Countries: Development of a Multidimensional Qualitative Coding Scheme. Adm Policy Ment Health (2023). https://doi.org/10.1007/s10488-023-01314-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10488-023-01314-6

Keywords

Search

Navigation