Assessing the accuracy of patient report of the 5As (ask, assess,advise, assist,

Susan A. Flocke, Elizabeth Antognoli

1. Department of Family Medicine and Community Health,Case Western Reserve University, Cleveland, OH, USA

Assessing the accuracy of patient report of the 5As (ask, assess,advise, assist, and arrange) for smoking cessation counseling

Susan A. Flocke1, Elizabeth Antognoli1

1. Department of Family Medicine and Community Health,Case Western Reserve University, Cleveland, OH, USA

Objective:The 5As framework (ask,advise,assess,assist,arrange) is a recommended strategy for smoking cessation counseling in primary care. This study compares patient report with direct observation to assess the degree of recall bias for each of the 5As.Methods:Primary care visits by 107 adult smokers and 16 physicians were audio-recorded.Within 48 hours after the visit, patients completed a survey assessing whether or not smoking was discussed and items specific to each of the 5As. The audio recordings were evaluated to assess the presence of eachA. The sensitivity, specificity, positive predictive value, and negative predictive value of patient report versus direct observation were computed.Results:The frequency of the 5As based on evaluation of the audio recording ranged from 13% (arrange) to 98% (ask). The sensitivity and specificity of patient report were 92% and not applicable forask, 90% and 50% forassess, 94% and 33% foradvise, 90% and 50% forassist, and 85% and 67% forarrangefollow-up. Positive predictive values ranged from 28% to 98%; negative predictive values ranged from 0% to 97%.Conclusion:Compared with the gold standard of direct observation, patient report of each of the 5As is reasonably sensitive but not specific. Patients overreport the occurrence of each of the 5As.

5As (ask, assess, advise, assist,arrange); smoking cessation; primary care

Introduction

Tobacco use continues to be one of the leading, preventable causes of premature morbidity and death in the United States [1, 2]. Smoking causes nearly 500,000 deaths per year in the United States, and is the leading cause of cancer [3]. The trend of annually declining prevalence of tobacco use has stalled in recent years [4]. A fourth of all US adults continue to smoke [4—6], with smoking prevalence highest among the most disadvantaged demographic groups [6, 7].

The clinical practice guideline for treating tobacco use and dependence is a highly regarded and frequently cited guide for smoking cessation treatment in clinical settings [8, 9]. The guideline specifically recommends the 5As framework (ask,advise,assess,assist,arrange) as the basis for brief smoking cessation counseling in the primary care context [10]. Evaluation of the actual implementation of the 5As in clinical practice and accurate assessment of their association with patient outcomes are important elements in determining the effectiveness of this approach and the effi-cient deployment of resources. However, assessment of provision of the 5As relies on patient report [11], physician report[12, 13], or medical record documentation [14], each of which is subject to limitations and biases [15].

Our team developed the 5As direct observation coding scheme (5A-DOC) [16]. This measure uses clinician—patient talk to determine the degree to which each of theAs is accomplished. The 5A-DOC is reliable, valid, and evaluates delivery of eachAso that patterns of advice can be classified with regard to completeness and appropriate sequencing of delivery of the elements. This measure is designed to serve as a gold standard to evaluate the delivery of the 5As in the primary care setting. While accurate, this method requires standardized coding of audio recordings. A less expensive method is to use patient report of the content of the smoking cessation discussion to evaluate the degree to which the 5As are delivered.However, the accuracy of patient report of the 5A elements compared with direct observation is not known. The goal of this study was to assess the accuracy of patient report of the 5A elements with use of the 5A-DOC as the gold standard.

Methods

This study is a cross-sectional multimethod study. Data for this study are drawn from a sample of audio-recorded patient visits to primary care physicians that were part of a group-randomized clinician-focused intervention, and the methods have been reported in detail elsewhere [17, 18]. Briefly, this report is focused on a subsample of 107 adults who reported smoking cigarettes or small cigars “some days” or “every day” and reported smoking, on average, at least one cigarette per day or one small cigar per week. Participants completed items reporting the occurrence of smoking discussion and activities specific to each of the 5As. These five items were based on self-report items in a previous study [19], and are dichotomously scored to represent if each counseling activity (ask,advise,assess,assist,arrange) happened. Patients were surveyed within 48 hours after a routine primary care visit. The university hospitals,MetroHealth Medical Center, and Cleveland Clinic institutional review boards approved the study procedures.

Patient visits were audio-recorded by research staff. The audio recordings were evaluated with the 5A-DOC for smoking cessation counseling, a reliable and valid tool to evaluate the 5As using patient—clinician talk. EachAwas scored as having occurred or not occurred on the basis of the audio recording.Two trained coders applied the 5A-DOC to the audio data.Throughout the coding period of the overall study, a random 15% sample of audio recordings were assigned to both coders to assess the coding reliability. Coders were blind to which cases were reliability cases.

Analyses

Inter-rater coding reliability was assessed with kappa and evaluated with kappa greater than 0.80 as almost perfect agreement and 0.61—0.80 indicating substantial agreement [20].The primary analyses for this report include the frequencies of each of the 5As for patient report and the 5A-DOC and computation of the sensitivity, specificity, positive negative predictive value, and negative predictive value with the 5A-DOC as the gold standard.

Results

The characteristics of the sample are shown in Table 1. Overall,60% of patients were female, 44 years old on average, 26%black, and 21% reported having less than a high school degree.On average, patients smoked 11 cigarettes per day.

The reliability of the coding of the audio recording for each visit using the 5A-DOC is reported in Table 2. The inter-rater agreement is 90% or above for each 5A element evaluated with the 5A-DOC. Kappa statistics ranged from 0.78 to 1.0 and were evaluated as excellent for each of the 5A elements.

The frequency for each 5A element is reported in Table 3 for both the 5A-DOC and patient report. From comparison of patient report with the 5A-DOC, the sensitivity of patient report is good. Sensitivities ranged from 0.83 forarrangefollow-up to 0.96 forassessreadiness to quit. The specificity of patient report for the 5As, however, is poor, and ranged from 0.29 foradviseto 0.65 forarrangefollow-up. The positive predictive value forarrangewas particularly low (0.23), indicating that among those cases where patients indicated thatarrangeoccurred, only 29% were positive on the basis of direct observation. The negative predictive values for all butarrangearepoor. For example, among those cases where patients indicated thatadvisedid not occur, only 58% were negative using direct observation thatadvise did not occur.

Table 1. Sample characteristics (n=107)

Discussion

In this study we found that primary care patients asked to report about discussions of smoking cessation within a 48-hours period after a visit tended to overreport the occurrence of the 5As. Explicitly, compared with the gold standard of direct observation, patient report of each of the 5As is reasonably sensitive but not specific. Patient overreporting of discussion of tobacco cessation advice might in part be due to a phenomenon calledtelescoping[21], or the recalling of unpleasant events as having occurred more recently than theyactually did. This type of recall bias can be counteracted by assessment of patient report in close proximity to the event,such as was accomplished in this study, where patient report was assessed within 48 hours of the visit. Another possible explanation is that patient overreporting may be influenced by repeated discussions of the topic of tobacco over time. When the topic of tobacco is addressed during most primary care visits, it may result in a patient reporting tobacco advice activities that happened at prior visits rather than the index visit with which the report is being compared. In addition to conducting the survey within 48 hours of the visit, the survey administrator oriented the patient to the visit for which the questions pertained, thus reducing the likelihood of the patient reporting about a different visit.

Table 2. Inter-rater agreement and reliability

The implication of patient overreporting is that estimates of the frequency of the 5As relying on patient report is lower than might have been previously believed. Further, what patients recall as having been discussed seems to be less exact than our operational definition of eachA, which suggests that efforts to refine the patient report items may be warranted.Research that clarifies patient ability to distinguish between offers of cessation assistance and arranging follow-up, for example, will be important if researchers and health systems need to rely on patient report of the 5As as a data source.Strategies in which patients are asked to listen to their own audio-recorded visits, are prompted to talk about what specifi-cally happened at different points of the visit, and then reflect on ways to phrase survey questions that capture those specific activities could be a valuable approach to refine patient report measures of the 5As.

Conroy et al. [22] examined implementation of the 5As by patient report, clinician report, and medical record review,and found poor agreement between the three methods. Theyspecifically found that the concordance between patient report and documentation in the medical record for specific 5A elements was very poor, with kappa ranging from 0.01 to 0.22,and patients reported 5A elements as occurring more frequently than documented in the medical record. However, this study does not clarify if the lack of concordance is overreporting on the part of the patient or lack of documentation in the medical record.

Table 3. Comparison of direct observation to patient report of 5As tasks (n=107)

The low positive predictive value ofarrangein this study is somewhat puzzling. This implies that when the physician does accomplisharrange, the patient has a low likelihood of reporting that it occurred. This is interesting becausearrangeis the onlyAthat points to a concrete action to be taken in the future. It could be that patients are more attuned to theAs that are discussion based and are selectively ignoring this action-based step.

This study is important because it provides evidence about the direction and magnitude of measurement error of relying on patient report of smoking cessation discussions. The study is unique in its use of audio recording, detailed assessment of the 5As, and patient report of the 5As, and it adds to the few other studies that have examined multiple methods to assess delivery of tobacco advice [22, 23]. However, a few study limitations are worth noting. The main limitation of the study is the sample size of 107, which inhibits our ability to examine patient and visit characteristics that are associated with the accuracy of patient reporting. Replication of this study with a larger sample, coupled with examination of patient and visit characteristics associated with inaccurate reporting, could inform future decisions about when patient report might be an acceptable method.

Further efforts to establish accurate methods to document the 5As or other indicators of the delivery of tobacco assessment and assistance advice are important, both to inform current performance and to evaluate interventions to improve tobacco cessation advice in the primary care setting. Studies have largely relied on patient report [24—29],clinician report [25—28, 30, 31], or medical record review[15, 25, 26, 32, 33]. Underreporting of tobacco cessation advice in the medical record has been identified as a limitation of this method [15, 22, 23]. Recent advances in the methods to comprehensively search the medical record to assess the delivery of the 5As for tobacco cessation were reported by Williams et al. [14]. The approach included assessment of the progress notes, patient education materials printed for the patient, and data linked to the encounter, including vital signs, diagnoses, procedures ordered, and medications[14]. While substantially more comprehensive in capturing the ways in which tobacco advice might be documented by clinicians versus relying on discrete fields to indicate completion of each of the 5As, this approach was time intensive to acquire and code the data. Additional research that compares this comprehensive electronic health record approach with other methods, including audio recording and patient report of the 5As, would greatly inform the accuracy of this medical record audit method and could inform a more streamlined and automated approach.

In conclusion, patient report of the 5As for smoking discussions is acceptably accurate for when the activity happens but is not sufficiently accurate for reporting when a 5A activity does not happen. Investigators should carefully consider the reporting source when measuring the provision of tobacco cessation advice.

Acknowledgments

We wish to thank Brigid Jackson MA and Sue Krejci MBA for assistance in coding some of the audio cases.

Conflicts of interest

The authors declare no conflicts of interest.

Funding

This research was supported by a grant from the National Cancer Institute (PI： Flocke, NCI R03CA176132, R01CA105292), and the Behavioral Measurement Shared Resource of the Case Comprehensive Cancer Center, NCI P30CA043703.

1. U.S. Department of Health and Human Services Agency for Healthcare Research and Quality. The guide to clinical preventive services 2010—2011： recommendations of the U.S. Preventive Services Task Force; 2010.

2. Yoon PW, Bastian B, Anderson RN, Collins JL, Jaffe HW.Potentially preventable deaths from the five leading causes of death—United States, 2008—2010. MMWR Morb Mortal Wkly Rep 2014;63：369—74.

3. U.S. Department of Health and Human Services. Current cigarette smoking among adults — United States, 2005—2012. MMWR Morb Mortal Wkly Rep 2014;63：29—34.

5. Johnson NB, Hayes LD, Brown K, Hoo EC, Ethier KAl. CDC national health report： leading causes of morbidity and mortality and associated behavioral risk and protective factors--United States,2005—2013. MMWR Surveill Summ 2014;63 (Suppl 4)：3—27.

6. Kasza KA, Ambrose BK, Conway KP, Borek N, Taylor K,Goniewicz, ML, et al. Tobacco-product use by adults and youths in the United States in 2013 and 2014. N Engl J Med 2017;376：342—53.

7. Centers for Disease Control and Prevention. Behavioral risk factor surveillance system survey data, 2000—2007. Atlanta： Centers for Disease Control and Prevention; 2008.

8. Fiore M, Bailey W, Cohen S, Dorfman S, Goldstein M, Gritz E, et al. Treating tobacco use and dependence. Clinical practice guideline. Rockville： US Department of Health and Human Services; 2000.

9. Fiore MC, Jaen CR, Baker TB, Bailey WC, Benowitz NL, Curry SJ, et al. Treating tobacco use and dependence： 2008 update.Rockville： US Department of Health and Human Services; 2008.

10. Whitlock EP, Orleans CT, Pender N, Allan JA. Evaluating primary care behavioral counseling interventions： an evidencebased approach. Am J Prev Med 2002;22：267—84.

11. Lebrun-Harris LA, Fiore MC, Tomoyasu N, Ngo-Metzger Q.Cigarette smoking, desire to quit, and tobacco-related counseling among patients at adult health centers. Am J Public Health 2015;105：180—8.

虽然蔬菜的腌制已经有非常悠久的历史，腌制的蔬菜深受人们的喜爱，但是这些腌制的食品中含有亚硝酸盐，亚硝酸盐的含量如果过高，人体在食用之后不仅会出现组织缺氧等现象，严重时还会引发癌变[6]。对食品中的亚硝酸盐进行具体测定时，需要与实际情况进行结合，将所有样品的温度控制在35℃以下，并且保证一定的干燥性。脂对亚硝酸盐含量测定的影响如表3所示。

12. Bartsch A-L, Harter M, Niedrich J, Levke Brutt A, Buchholz A. A systematic literature review of self-reported smoking cessation counseling by primary care physicians. PLoS One 2016;11：e0168482.

13. Kruger J, O’Halloran A, Rosenthal A. Assessment of compliance with U.S. public health service clinical practice guideline for tobacco by primary care physicians. Harm Reduct J 2015;12：7.

14. Williams RJ, Masica AL, McBurnie MA, Solberg LI, Bailey SR,Hazlehurst B. Documentation of the 5 As for smoking cessation by PCPs across distinct health systems. Am J Manag Care 2014;20：e35—42.

15. Glasgow RE, Emont S, Miller DC. Assessing delivery of the five “As” for patient-centered counseling. Health Promot Int 2006;21：245—55.

16. Lawson PJ, Flocke SA, Casucci B. Development of an instrument to document the 5A’s for smoking cessation. Am J Prev Med 2009;37：248—54.

17. Flocke SA, Antognoli E, Step MM, Marsh S, Parran T, Mason MJ. A teachable moment communication process for smoking cessation talk： description of a group randomized clinicianfocused intervention. BMC Health Serv Res 2012;12：109.

18. Flocke SA, Step MM, Antognoli E, Lawson PJ, Smith S, Jackson B, et al. A randomized trial to evaluate primary care clinician training to use the teachable moment communication process for smoking cessation counseling. Prev Med (Baltim)2014;69：267—73.

19. Katz DA, Muehlenbruch DR, Brown RB, Fiore MC, Baker TB.Effectiveness of a clinic-based strategy for implementing the AHRQ Smoking Cessation Guideline in primary care. Prev Med(Baltim) 2002;35：293—301.

20. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33：159.

21. Lavrakas PJ, editor. Encyclopedia of survey research methods.Thousand Oaks： Sage Publications; 2008.

22. Conroy MB, Majchrzak NE, Silverman CB, Chang Y, Regan S,Schneider LI, et al. Measuring provider adherence to tobacco treatment guidelines： a comparison of electronic medical record review, patient survey, and provider survey. Nicotine Tob Res 2005;7 (Suppl 1)：S35—43.

23. Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA,et al. How valid are medical records and patient questionnaires for physician profiling and health services research?： A comparison with direct observation of patient visits. Med Care 1998;36：851.

24. An LC, Foldes SS, Alesci NL, Bluhm JH, Bland PC, Davern ME,et al. The impact of smoking-cessation intervention by multiple health professionals. Am J Prev Med 2008;34：54—60.

25. Conroy MB, Majchrzak NE, Regan S, Silverman, CB, Schneider LI, Rigotti NA. The association between patient-reported receipt of tobacco intervention at a primary care visit and smokers’satisfaction with their health care. Nicotine Tob Res 2005;7(Suppl 1)：S29—34.

26. Dixon LB, Medoff D, Goldberg R, Lucksted A, Kreyenbuhl J,DiClemente C, et al. Is implementation of the 5 A’s of smoking cessation at community mental health centers effective for reduction of smoking by patients with serious mental illness? Am J Addict 2009;18：386—92.

27. Katz DA, Vander Weg MW, Holman J, Nugent A, Baker L,Johnson S, et al. The Emergency Department Action in Smoking Cessation (EDASC) trial： impact on delivery of smoking cessation counseling. Acad Emerg Med 2012;19：409—20.

28. Katz DA, Muehlenbruch DR, Brown RL, Fiore MC, Baker TB. Effectiveness of implementing the agency for healthcare research and quality smoking cessation clinical practice guideline： a randomized, controlled trial. J Natl Cancer Inst 2004;96：594—603.

29. Quinn VP, Stevens VJ, Hollis JF, Rigotti NA, Solberg LI, Gordon N, et al. Tobacco-cessation services and patient satisfaction in nine nonprofit HMOs. Am J Prev Med 2005;29：77—84.

30. Dosh SA, Holtrop JS, Torres T, Arnold AK, Baumann J, White LL. Changing organizational constructs into functional tools： an assessment of the 5 A’s in primary care practices. Ann Fam Med 2005;3 (Suppl 2)：S50—2.

31. Szpunar S, Williams P, Dagroso D, Enberg RN, Chesney JD. An assessment of user acceptance and satisfaction with the tobacco use cessation automated clinical practice guideline. Am J Manag Care 2007;13：313—5.

32. Hazlehurst B, Sittig DF, Stevens VJ, Smith KS, Hollis JF, Vogt TM, et al. Natural language processing in the electronic medical record： assessing clinician adherence to tobacco treatment guidelines. Am J Prev Med 2005;29：434—9.

33. Bentz CJ, Bayley KB, Bonin KE, Fleming L, Hollis JF, Hunt JS, et al. Provider feedback to improve 5A’s tobacco cessation in primary care： a cluster randomized clinical trial. Nicotine Tob Res 2007;9：341—9.

Susan A. Flocke, PhD

Department of Family Medicine and Community Health, Case Western Reserve University,11000 Cedar Ave., Suite 402,Cleveland, OH 44106, USA

E-mail： susan.flocke@case.edu

11 April 2017;

Accepted 31 May 2017