Scholarly article on topic 'Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey'

Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey Academic research paper on "Clinical medicine"

Share paper

Academic research paper on topic "Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey"


Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey

Hardeep Singh,1 Christiane Spitzmueller,2 Nancy J Petersen,1 Mona K Sawhney,1 Michael W Smith,1 Daniel R Murphy,1 Donna Espadas,1 Archana Laxmisan,1 Dean F Sittig3

'Department of Medicine, Houston VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center and Section of Health Services Research, Baylor College of Medicine, Houston, Texas, USA 2Department of Psychology, University of Houston, Houston, Texas, USA 3University of Texas School of Biomedical Informatics and UT-Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas, USA

Correspondence to

Dr Hardeep Singh, Department of Medicine, Houston VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center (152), Section of Health Services Research, Baylor College of Medicine, 2002 Holcombe Blvd, Houston, TX 77030, USA;

Received 13 August 2012 Revised 19 November 2012 Accepted 20 November 2012 Published Online First 25 December 2012


Context Failure to notify patients of test results is common even when electronic health records (EHRs) are used to report results to practitioners. We sought to understand the broad range of social and technical factors that affect test result management in an integrated EHR-based health system. Methods Between June and November 2010, we conducted a cross-sectional, web-based survey of all primary care practitioners (PCPs) within the Department of Veterans Affairs nationwide. Survey development was guided by a socio-technical model describing multiple inter-related dimensions of EHR use. Findings Of 5001 PCPs invited, 2590 (51.8%) responded. 55.5% believed that the EHRs did not have convenient features for notifying patients of test results. Over a third (37.9%) reported having staff support needed for notifying patients of test results. Many relied on the patient's next visit to notify them for normal (46.1%) and abnormal results (20.1%). Only 45.7% reported receiving adequate training on using the EHR notification system and 35.1% reported having an assigned contact for technical assistance with the EHR; most received help from colleagues (60.4%). A majority (85.6%) stayed after hours or came in on weekends to address notifications; less than a third reported receiving protected time (30.1%). PCPs strongly endorsed several new features to improve test result management, including better tracking and visualization of result notifications.

Conclusions Despite an advanced EHR, both social and technical challenges exist in ensuring notification of test results to practitioners and patients. Current EHR technology requires significant improvement in order to avoid similar challenges elsewhere.

To cite: Singh H, Spitzmueller C, Petersen NJ, et al. J Am Med Inform Assoc 2013;20:727-735.


Failure to follow up on abnormal test results ('missed test results') is a global patient safety concern.1-5 Electronic health records (EHRs) are increasingly used to notify practitioners of abnormal test results.6-8 EHR-based test result notification systems can reduce, if not eliminate, many of the communication problems inherent with paper-based records.9-12 However, ensuring that test results receive appropriate follow-up remains challenging even with electronic transmission.5 13 14 Previous studies of practitioner responses to abnormal test result notifications through EHRs found that 7-8% of abnormal test alerts had no evidence

of timely follow-up, even when practitioners had electronically acknowledged result receipt.10 11 14

Although the proposed Stage 2 meaningful use regulations15 to be implemented in 2014 in the USA include structured laboratory result reporting functionality, little is known about how to optimize EHR-based test result management. Currently, several major EHR vendors employ a reporting functionality whereby providers receive notification of results in their inboxes (similar to email). This functionality is being increasingly adopted across EHRs and is likely to have a great impact on future test result management practices.

In complex healthcare settings that use EHRs, test result management encompasses many contextual factors such as clinical workflow, user behaviors, and organizational policies and procedures.16 17 Electronically transmitted test results may be missed or overlooked for a variety of reasons related to EHR design and use.18 19 For instance, while most EHRs deliver abnormal results to practitioners through an inbox, the inbox often also holds less important data ('noise') that could dilute important or urgent information ('signals').19 Other factors contributing to missed test results include policies and procedures related to the use of EHRs, such as unclear responsibility for follow-up of abnormal results when multiple providers are involved in patient care.10 Thus, contextual factors20 that affect test results management derive from both technical and social dimensions of the EHR-enabled health care system.

To better understand contextual factors that affect practitioners' test results management practices within the setting of a comprehensive EHR, we designed and administered a nationwide survey to primary care practitioners (PCPs) practicing in all facilities of the Department of Veterans Affairs (VA). Our objective was to use survey data on PCPs' experiences, practices, and preferences to create a body of knowledge upon which to base future improvements in EHR-based test result reporting systems.


Between June 10, 2010 and November 5, 2010, we conducted a cross-sectional, web-based survey of PCPs in VA settings nationwide. The local institutional review board approved the study. Within the VA system, most patients are assigned to staff PCPs, who serve as a coordinating hub for most care and

thus depend heavily on the clinical information sent to their EHR inboxes. A smaller number of patients are assigned to trainees or subspecialists who see patients on a part-time basis,

Table 1 Eight interactive socio-technical health information

technology (HIT) dimensions addressed for missed test results

Dimension Definition

Hardware and software Equipment and software required to run the

healthcare applications

Clinical content Data, information, and knowledge entered,

displayed, or transmitted in EHRs

Human computer interface Aspects of the EHR system that users interact with

(eg, see, touch, or hear)

People Humans involved in the design, development,

implementation, and use of HIT

Workflow and Work processes needed to ensure that each

communication patient receives the care they need at the time

they need it

Organizational policies and Internal culture, structures, policies, and

procedures procedures that affect all aspects of HIT

management and healthcare

External rules, regulations, External forces that facilitate or place constraints

and pressures on the design, development, implementation, use,

and evaluation of HIT in the clinical setting

System measurement and Measurement of system availability, use,

monitoring effectiveness, and unintended consequences of

system use

EHR, electronic health record.

usually one half-day per week. Thus, these practitioners have a smaller panel (the total number of patients a PCP is responsible for) due to their other competing responsibilities. Our previous experience suggested that staff PCPs' experiences, practices, and preferences related to test result management might be different from those of trainees and subspecialists who serve as PCPs.16 18 Thus, we used a large administrative VA database, the Veterans Health Administration Support Service Center Clinical Care: Primary Care Management Module, to identify all PCPs with a minimum primary care patient panel size of 250 (N=5001). This strategy allowed us to exclude trainees and subspecialists because they are generally expected to have small panel sizes.

Survey development

Survey development was guided by literature review1 6 21-34 and by a conceptual model describing multiple inter-related dimensions of EHR use (see 'Survey content areas' section below; also see table 1 which lists model dimensions).35 A psy-chometrician (CS) guided the survey development process, which included item writing and refinement, soliciting input from subject matter experts in EHR use, and iterative content review. After refining all survey items, we pilot tested the survey with 10 PCPs for readability, clarity, and ease of completion in a web-based format. With the exception of demographic items and 10 open-ended items, items were rated on a 5-point Likert-type scale with response options ranging from 'strongly disagree' to 'strongly agree.' Survey completion time was approximately 20-25 min.

Patient selection

Jatient List

(• Default: Wednesday Clinic

C Providers C £linics

C Xeav/Ftrícfíl 1" Wards

C specialties r £1

Patients [Wednesday GinicJ


No Appointments

Patient, Testl Patient, Test2 Patient. Testa Patient, Test4 Patient. Test5 Palient, Tests Patient, Test? Patient, Testä Patient, Testô Patient. Testl 0 Patient, Testl 1 Patient, Testl 2 Patient, Testl 3 Patient, Test14 Patient, Testl 5

Save Parent List Settings


Info | Palient 1 Location I Urgency ] Alert Dale/Time I Message | Forwarded ByAVhen

Patient, Test! 32 Moderate 01M1/2010@03:14 Scheduled Consult: ORTHOPEDICS

paient, Testl 64 Moderate Q1ÍQ1Í2Q1Q@15:09 Scheduled Consult: NON-INVASIVE ECHO

Palient, Test243 Moderate 01J02M10@02:0S COMPLETED EMERGENCY ROOM NOTE available for ADD'L

Patient. Test32 HIGH 01/02Í2010@05:35 Medications Nearing expiration

Palient, Testl 7 Moderate 01/02/2010@08:D9 Imaging request held: KNEE 3 VIEWS, LEFT

Patient, Test267 SM Moderate 01/03/2010@07:09 Abnormal labs ■ [TROPONIN I]

Palient, Testl 65 Moderate 01/04Í2010@03:23 Abnormal labs - [LIPID PROFILE, COMPREHENSIVE]

Palient, Testl 3S Moderate 01/04/2010@06:46 Abnormal labs ■ [BASIC METABOLIC PANEL)

Palient, Testl 06 Moderate Q1ÍQ5Í2010@Q2:Q6 Labs resulted ■ |micrOalbumin:ufune]

Palient, Test286 Moderate 01W2010@04:33 COMPLETED PHARMACY PATIENT ENCOUNTER available ft

1 Palient, Test362 GM HIGH 01í05í2010@0e:32 Critical CR 2,3 05/27 05:00

Palient, Testl 51 Moderate 01/06/2010@07:09 Imaging Results, Non Critical: SPINE LUMBOSACRAL MIN 2 VI

Palient, Test209 Moderate Oira7/20'Ü@05:DG Discontinued consult DERMATOLOGY

Palient, Testl 22 GM Moderate 01/08f2010@02:Q8 Comment Added to Consult PAIN MANAGEMENT

Palient, Test305 HIGH 01rt»2010@08:41 Abnl Imaging Resit: Needs Attn: CT ABDOMEN W&W/O CO

Palient, Testl 32 Moderate 01i08f2010@09:07 Completed Consul! NURSING CASE MANAGEMENT

Palient, Test57 Moderate 01/08Í2010@16:09 Abnl Imaging Resit: Needs Attn: MAGNETIC IMAGE LUMBAR 5

Palient, Testl 60 Moderate 01,'09l20-iÜ@03;09 UNSIGNED PRIMARY CARE FOLLOW-UP avallable for SIGNA

Process Info

Process All



Show Comments


Figure 1 Alert notification window in the VA Computerized Patient Record System (CPRS).

Figure 2 Distribution of respondents within 21 Veterans Affairs Networks across the USA (Veterans Integrated Service Networks or VISNs).

Survey content areas

Hardware and software, content, and user interface

The VA uses the Computerized Patient Record System (CPRS) as its EHR in all its facilities. CPRS uses the 'View Alert' notification system to communicate test results (as well as other important clinical information) to practitioners through an inbox. The View Alert system also displays notifications as alerts with various priorities. Practitioners see their patients' alerts each time they log in to the system or switch between patient records (figure 1). Alerts remain within the inbox until read by the practitioner, but they may be removed automatically if unopened after a certain time period (eg, 14 or 30 days). Although this functionality is used across the VA, individual facilities have discretion over which types of alerts practitioners must receive (eg, they can allow flexibility for practitioners to turn off certain notifications, such as normal test results). Our survey items assessed PCPs' perceptions of the View Alert notification system, including their views on the content and quantity of alerts received, perceived ease of use, and use of the EHR's features for alert processing and follow-up.


Our survey assessed several characteristics of the respondents including age, gender, race, job classification (ie, academic physician, non-academic physician, nurse practitioner, physician assistant), years employed at the VA, native language (English or other), perceived adequacy of training related to the EHR notification system, and prior use of an EHR other than CPRS.

Workflow and communication

Workflow items assessed perceptions of alert burden, processes related to notifying patients of their test results, and practices used to support alert management.

Table 2 Characteristics of survey respondents (n=2590)

Characteristic n (%)

20-39 338 (13.1)

40-49 685 (26.4)

50-59 961 (37.1)

60 and over 402 (15.5)

Missing 204 (7.9)


Male 1080 (41.7)

Female 1343 (51.9)

Missing 167 (6.4)

White 1630 (62.9)

Black 118 (4.6)

Asian 431 (16.6)

Other 188 (7.3)

Missing 223 (8.6)

Job classification

Physician, academic 438 (16.9)

Physician, non-academic 1228 (47.4)

Nurse practitioner 561 (21.7)

Physician assistant 204 (7.9)

Missing 159 (6.1)

Years at VA

<2 437 (16.9)

2-10 1219 (47.1)

11-20 589 (22.7)

>20 201 (7.8)

Missing 144 (5.6)

Native language

English 1911 (73.8)

Other 498 (19.2)

Missing 181 (7.0)

VA, Department of Veterans Affairs.

Table 3 Technology-related items associated with EHR-based test result management

Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing

Item n (%) n (%) n (%) n (%)

Hardware and software

CPRS has convenient features for notifying patients of test results 628 (24.2) 1437 (55.5) 421 (16.3) 104 (4.0)

Has a convenient way of generating letters in CPRS for patient notification 1091 (42.1) 1042 (40.2) 347 (13.4) 110 (4.2)

Has an automated voice messaging system that can notify patients of test results 38 (1.5) 2337 (90.2) 113 (4.4) 102 (3.9)

Uses My HealtheVet* to notify patients of test results 45 (1.7) 2308 (89.1) 134 (5.2) 103 (4.0)

Clinical content

Acknowledges (ie, clicks on) all high priority alert notifications 2365 (91.3) 80 (3.1) 112 (4.3) 33 (1.3)

Follows up on all high priority alert notifications received 2372 (91.6) 77 (3.0) 97 (3.7) 44 (1.7)

Acknowledges all alert notifications received regardless of priority 2169 (83.7) 228 (8.8) 146 (5.6) 47 (1.8)

Human-computer interface

Finds the alert notification system in CPRS easy to uset 1842(71.1) 326 (12.6) 393 (15.2) 29(1.1)

Follows up on all alert notifications received 1968 (76.0) 311 (12.0) 258 (10.0) 53 (2.0)

Alert notification system in CPRS makes it possible for providers to miss test results 1440 (55.6) 555 (21.4) 506 (19.5) 89 (3.4)

Sorts alert notifications when necessary according to urgency, patient name, location, or alert date/time 1625 (62.7) 562 (21.7) 381 (14.7) 22 (0.8)

Uses the Process All function of the system when necessaryt 981 (37.9) 1219(47.1) 367 (14.2) 23 (0.9)

Italicized items are discussed in the text. *My HealtheVet is a free, online personal health record for VA patients. tCPRS (Computerized Patient Record System) is the Department of Veterans Affairs electronic health record software. ♦The Process All feature allows clinicians to efficiently process alerts one after another without returning to the View Alert window. EHR, electronic health record; VA, Department of Veterans Affairs.

Organizational features

These items assessed institutional cultural norms and expectancies regarding management of alerts, technical support for alert notifications, organizational support to facilitate patient notification, and the amount of protected time (ie, specifically designated and compensated time) clinicians were given by their organizations to manage alerts. Protected time was measured in hours per week; all other response choices and coding were the same as for other item categories on the 5-point Likert-type scale.

External rules and regulations

We assessed whether practitioners were aware of and adhered to a national VA policy36 released in 2009 requiring notification of patients within 14 days for both normal and abnormal test results.

System measurement and monitoring

We assessed perceptions of practices related to measuring and monitoring alert follow-up at the system level.

New features and functions to improve EHR-based notification

We inquired whether PCPs would endorse several potential strategies to improve EHR-based notification. Strategies were selected based on thematic areas of improvement identified in previous work8 10 11 16 18 37 and addressed both technical and social dimensions.

Survey administration

We solicited support from the section chiefs of primary care at 142 VA facilities nationwide by asking them to email all PCPs at their respective sites to orient them to the project. We subsequently invited all participants by sending a personalized email from the principal investigator (HS) that described the study

and provided a link to the web-based survey instrument. To increase response rates, invitation emails and subsequent reminders were followed by telephone attempts to reach non-respondents. In keeping with VA policies, we did not use monetary or other incentives for participation.

Data analysis

Data were downloaded from the internet survey administration service and were analyzed using SPSS statistical software. We generated descriptive statistics to summarize the characteristics of respondents and to aggregate responses to other survey items using descriptive statistics. Likert-type item responses were collapsed into three categories of 'agree or strongly agree,' 'disagree or strongly disagree,' and 'neither agree nor disagree.'


Of 5001 PCPs invited, 2590 (51.8%) responded. Figure 2 shows the geographic distribution of the respondents by Veterans Integrated Service Network (regional divisions of VA). Table 2 shows the characteristics of the respondents. Characteristics of non-respondents were not available for comparison with the respondent group. The vast majority of respondents had considerable experience with the VA EHR, having worked within the VA for 2 or more years. Less than half (45.7%) reported having received sufficient training on the View Alert system, and only a minority (13.7%) reported any refresher training (data not shown in the table). Nevertheless, the majority believed they had the knowledge (74.4%) and proficiency (81.8%) necessary to use the View Alert system (data not shown in the table). Nearly half (46.6%) also had prior experience using a non-VA EHR. Of these, only 19% of providers thought that the non-VA EHRs they used were better than VA's CPRS; 55% indicated that the non-VA EHR they used was overall inferior to CPRS, and 26% perceived it was about the

same. No particular non-VA EHR was consistently identified as better or worse than CPRS.

Table 3 shows the distribution of responses of selected items related to use and perceptions of the EHR technology (hardware/software, clinical content, and human-computer interface). Although most PCPs found the EHR alert notification system easy to use, more than half believed that the EHR software did not have convenient features for notifying patients of test results. Less than two-thirds of practitioners used a basic sorting feature for alerts, and just over a third used an enhanced functionality to process alerts. Both of these features are intended to promote efficiency when managing alerts. In addition, a significant proportion of PCPs had not altered their alert notification filters to help customize the amount and types of alerts they received. Thus, they received only notifications determined important (and set as default) by the facility (43.0%; data not shown).

Table 4 shows the distribution of responses for survey items related to workflow and communication. A majority of practitioners stayed after hours or came in on weekends to address alerts; only about a third had remote access to the EHR to manage alerts after hours. More than half of PCPs electronically assigned a surrogate or covering practitioner to handle their alerts when out of the office, a process we defined as 'electronic hand-off.' Just over a third reported having the help needed for notifying patients of test results. Patient notification varied between normal and abnormal results. Almost half of PCPs reported that they did not immediately notify patients of normal test results and relied on the patient's next visit to notify

them, whereas about one-fifth relied on the next visit to notify patients of abnormal results.

Table 5 summarizes responses to items that assessed people and organizational features affecting alert management. Only about a third of respondents reported having or accessing an assigned technical contact to help with alerts; most admitted receiving help with alert management from colleagues. About a third of respondents reported receiving protected time for alert management; of these, most respondents reported it to be at least 4 hours per week (69.8%).

About half of respondents (54.2%) were aware of the 2009 Veterans Health Administration (VHA) policy (ie, external rules and regulations) regarding notification of patients regarding test results. However, among those aware of the policy only 20% reported that they changed their result notification practices accordingly (data not shown in tables). Almost a third (31.9%) reported that their supervisors monitored how they managed notifications, but fewer (23.9%) received feedback on their follow-up practices (ie, system measurement and monitoring).

Table 6 summarizes PCPs' assessments of potential new features and functions to improve EHR-based notification. PCPs strongly endorsed four functionalities to reduce loss of alert information in the EHR ( preventing automated deletion of alerts, being able to retrieve deleted alerts, having a back button, and being able to access alerts in the inbox for at least 30 days). To improve alert management options, most agreed with the need for a feature to remind them at a later date of the necessity to take a follow-up action. In addition, most PCPs endorsed the need for a separate messaging system within the

Table 4 Workflow and communication items associated with EHR-based test result management

Agree or strongly Disagree or strongly Neither agree nor

agree disagree disagree Missing

Item n (%) n (%) n (%) n (%)

Workflow and communication

Alert burden

The number of alerts exceeds what they can effectively manage 1803 (69.6) 321 (12.4) 432 (16.7) 34 (1.3)

Receives too many alerts to easily focus on most important ones 2078 (80.2) 215 (8.3) 244 (9.4) 53 (2.0)

In the past year, missed test results that led to delayed patient care 772 (29.8) 1212 (46.8) 537 (20.7) 69 (2.7)

Uses remote access after hours or on weekends to manage alerts 893 (34.5) 1327 (51.2) 320 (12.4) 50 (1.9)

Has remote access to CPRS 953 (36.8) 1599 (61.7) - 38 (1.5)

Stays after hours or comes in on weekends to manage alerts 2218 (85.6) 219 (8.5) 105(4.1) 48 (1.9)

Wishes the system provided them with more alerts 151 (5.8) 2190 (84.6) 208 (8.0) 41 (1.6)

Receives too many alert notifications per day 2251 (86.9) 164 (6.3) 166 (6.4) 9 (0.3)

Uses additional paper-based methods to help follow test results 1451 (56.0) 1103 (42.6) - -

Gets too many FYI only alert notifications that require a signature even 2071 (80.0) 250 (9.7) 258 (10.0) 11 (0.4)

though no action on their part is required

Often receives alert notifications where they are unsure as to why they were 1601 (61.8) 539 (20.8) 432 (16.7) 18 (0.7)

sent to them

Patient notification

Consistently notifies patients of abnormal test results 2146 (82.9) 151 (5.8) 185 (7.1) 108 (4.2)

Has the help needed for notifying patients of test results 981 (37.9) 1180 (45.6) 332 (12.8) 97(3.7)

Consistently notifies patients of normal test results 1167 (45.1) 902 (34.8) 415 (16.0) 106 (4.1)

Relies on a patient's next visit to notify them of their abnormal test results 521 (20.1) 1556(60.1) 407(15.7) 106(4.1)

Relies on a patient's next visit to notify them of their normal test results 1193(46.1) 858(33.1) 431 (16.6) 108 (4.2)

Use of surrogates

Alert notifications related to surrogates create new safety concerns 1339 (51.7) 482 (18.6) 730 (28.2) 39 (1.5)

Has support staff to assist with management of test result alert notifications 873 (33.7) 1304 (50.3) 311 (12.0) 102 (3.9)

Assigns a surrogate to take care of alert notifications when out of the office 1525 (58.9) 788 (30.4) 239 (9.2) 38 (1.5)

Italicized items are discussed in the text.

CPRS, Computerized Patient Record System; EHR, electronic health record; FYI, for your information.

Table 5 Organizational and people items associated with EHR-based test result management

Agree or strongly Disagree or strongly Neither agree nor

agree disagree disagree Missing

Item n (%) n (%) n (%) n (%)

Organizational policies and procedures

Technical support

Clinic has an assigned technical contact that can be accessed for help 908(35.1) 947 (36.6) 651 (25.1) 84 (3.2)

with alert notifications

IT help person provides useful resources 621 (24.0) 882 (34.1) 988 (38.1) 99 (3.8)

With questions about or problems with alert notifications, gets the help 887 (34.2) 809 (31.2) 807 (31.2) 87 (3.4)

needed from IT

With questions about or problems with alert notifications, gets the help 1564 (60.4) 348 (13.4) 587 (22.7) 91 (3.5)

needed from colleagues

Protected time

Has protected clinical time slots to manage alert notifications* 779(30.1) 1773 (68.5) - 38 (1.5)


Supportive norms

Colleagues believe the alert notifications in CPRS help them get their 640 (24.7) 849 (32.8) 1041 (40.2) 60 (2.3)

job done effectively

Supervisor believes alert notifications in CPRS are an essential 1403 (54.2) 143 (5.5) 981 (37.9) 63 (2.4)

component of effective primary care

Senior management has emphasized the importance of the use of alert 1359 (52.5) 189 (7.3) 981 (37.9) 61 (2.4)

notifications in high quality care

Performance expectancy

Using alert notifications in CPRS enhances providers' ability to provide 2098 (81.0) 167 (6.4) 275 (10.6) 50 (1.9)

safe patient care

Using alert notifications in CPRS enhances providers' effectiveness on 1803 (69.6) 311 (12.0) 423 (16.3) 53 (2.0)

the job

Using alert notifications in CPRS increases providers' productivity 952 (36.8) 858 (33.1) 723 (27.9) 57 (2.2)

Using alert notifications in CPRS allows providers to meet performance 1312 (50.7) 542 (20.9) 679 (26.2) 57 (2.2)


Italicized items are discussed in the text.

*Of the 779 providers who reported receiving protected time, 224 (28.8%) reported <4 h/week, 544 (69.8%) reported >4 h/week, and 11 (1.4%) did not report the number of hours

of protected time.

CPRS, Computerized Patient Record System; EHR, electronic health record.

EHR to allow providers to communicate, rather than the traditional method of including these human-generated messages along with the EHR-generated notification system that was used for result notification. About two-thirds of respondents endorsed one or more new visualization techniques, including a separate window for high-priority alerts or a method to filter or color-code alerts based on type. Most PCPs supported strategies to improve safety related to hand-offs, including being able to assign responsibility of test result follow-up and to display who is responsible for follow-up. About half were in favor of receiving feedback about their alert management performance.


Test result reporting has substantial patient safety implications and is now being considered in the meaningful use criteria for EHRs.38 However, our data suggest that despite the use of an advanced EHR system, both social and technical challenges exist in ensuring the reliability of test result notification to practitioners and patients within one of the world's largest healthcare systems. Other healthcare systems, most of which have a shorter history of EHR use, are likely to face similar challenges as they begin adopting EHRs. Several technical as well as social ( personal, workflow, and organizational) factors need to be addressed in order for EHR-based test reporting functionality to be successful. PCPs in our sample endorsed several new features and functions to reduce loss of information in the EHR and to improve visualization of alerts that communicate test results.

We found that most providers do not routinely notify patients of normal test results, and a substantial proportion use the next scheduled visit to notify patients of both normal and abnormal results. A 1996 study conducted outside the VA reached similar conclusions, suggesting that this pattern has changed little in the last two decades.30 Recent initiatives within the VA are developing additional guidance to address test result management, which might change some of the patterns we found.39 As healthcare quality improvement efforts increasingly emphasize patient engagement, alternative methods are being explored for patients to receive test results. For instance, results are increasingly accessible to patients through online portals,40 although adoption has been slower than expected.41 42 Furthermore, the Department of Health and Human Services recently proposed a rule43 allowing patients to access test results directly from the laboratory upon patient request (ie, bypassing the provider). However, providers' interpretation of test results within the context of the patient's other clinical conditions remains essen-tial.44 Because most PCPs receive hundreds of test results a week,31 one area for potential improvement is PCPs' resources to facilitate patient notification. Just over a third of the PCPs reported having administrative support for patient notification, presenting an opportunity to leverage the current shift towards team-based models of care (eg, medical home teams45). For example, with appropriate task delegation and clarity of roles, other members of the team could be responsible for tracking results and notifying patients.45 Future EHR development

Table 6 New features and functions to improve EHR-based notification

Agree or Disagree or Neither agree nor

strongly agree strongly disagree disagree Missing

Item n (%) n (%) n (%) n (%)

Hardware and software

Improving communication and alert management options

I would like to be able to set reminders for myself for future actions 2160 (83.4) 64 (2.5) 213 (8.2) 153 (5.9)

I would like to have a messaging system within CPRS that would allow providers to 1826 (70.5) 174 (6.7) 444 (17.1) 146 (5.6)

communicate with one another - this would be outside the View Alert system

Human-computer interface

Improving alert visualization

I would like to receive high priority test result notifications in one window, and all 1611 (62.2) 474 (18.3) 370 (14.3) 135 (5.2)

other alert notifications in another window

I would like an option to display only certain alert notifications at a time (ie, filter to 1722 (66.5) 321 (12.4) 398 (15.4) 149 (5.8)

display only surrogate, inpatient, or high priority alerts)

I would like to have my alert notifications color-coded according to type (eg, 1720 (66.4) 316 (12.2) 414 (16.0) 140 (5.4)

surrogate, inpatient, or high priority alerts)

Better processing and tracking of alerts

I would like to be able to retrieve my deleted alert notifications 2036 (78.6) 193 (7.5) 228 (8.8) 133 (5.1)

High priority alert notifications should not disappear until I actively delete them after 1913 (73.9) 266 (10.3) 273 (10.5) 138 (5.3)

taking follow-up action

There should be a mechanism in the alert notification system in CPRS to display the 1596 (61.6) 292 (11.3) 557 (21.5) 145 (5.6)

name of the person responsible for following up on the test result alert

I would like to have a 'back button' in CPRS to retrieve the prior window 2173 (83.9) 67 (2.6) 212 (8.2) 138 (5.3)

All unacknowledged alert notifications should stay in the alert notification window for 1689 (65.2) 318 (12.3) 445 (17.2) 138 (5.3)

at least 30 days

Workflow and communication

Improving the surrogate process

As a surrogate, I should only receive high priority alert notifications for patients 1659 (64.1) 432 (16.7) 360 (13.9) 139 (5.4)

assigned to the provider who is out of the office

Currently it is not possible for a provider to directly assign, without IT assistance, 1555 (60.0) 264 (10.2) 618 (23.9) 153 (5.9)

more than one surrogate in CPRS. This capability should be introduced

System measurement and monitoring

Improving feedback

I would like to receive feedback about my performance related to follow-up of high 1264 (48.8) 479 (18.5) 708 (27.3) 139 (5.4)

priority alert notifications

CPRS, Computerized Patient Record System; EHR, electronic health record.

should focus on innovative methods to facilitate patient notification of test results.

Our findings underscore the additional time burden induced by EHR-based alert systems, with the majority of respondents staying after hours, coming in on weekends, or using remote access from home to deal with notifications. Unfortunately, there is little or no reimbursement for non-face-to-face time related to documentation, follow-up, and patient notification of test results. Although the interventions proposed in the survey focused mostly on technological features and workflow, several organizational and policy interventions are also needed to improve the outcomes of EHR-based test results reporting.

Based on our data, we recommend several strategies for improving test results reporting though EHRs. First, being able to easily retrieve and readily access critical test result data is an absolute requirement for safe and efficient care, and the ability to do so (eg, to retrieve previously viewed alerts) was endorsed by a large majority of respondents. For instance, providers might accidently delete a result notification or need to return to an alert they processed some time ago, and these workflow requirements should inform EHR design. Second, to effectively address alert-related information overload,19 providers need better methods to display, sort, and visualize test result informa-tion46 according to patient, date, type, priority, and origin of alerts. Usability testing is essential to evaluate EHR support

features for planning and prioritizing among high volumes of alerts. Third, most providers agreed with the need for strategies to improve electronic hand-offs in the EHR, including assignment of responsibility. There is surprisingly little knowledge to guide effective policy setting or practice management in this area. While health information technology can prevent communication breakdowns,47-50 vulnerabilities related to teamwork and care coordination51 52 have received less attention, and novel methods to reduce ambiguities related to these hand-offs are needed. Lastly, EHRs need to be able to support the 'prospective' memory53 of providers (memory of an intent to perform a future task) by facilitating alert tracking and self-reminders for future tasks. Some of these features should be deemed as universal requirements in EHRs.

Most of our results are generalizable outside the VA even though there are differences in EHR features, usage, organizational, workflow, and policy-related factors. This is due to the fact that the VA has been a leader in the successful integration and use of comprehensive EHRs, and many of our findings are valuable for other healthcare institutions that are moving to more integrated, technology-enabled care environments as required to meet the new patient management challenges prescribed by accountable care organizations.54 Although the alert notification system was specific to the VA's EHR, most major EHRs use similar methods to notify providers of important clinical

findings, and others will follow in compliance with meaningful use requirements. Our study had a relatively low response rate. However, this is comparable to or higher than the response rates of many other physician surveys,55-58 which traditionally are not very high.59 60 In fact, the large number of completed surveys represents one of the largest total responses regarding EHR characteristics ever reported.61 62 Studies that report higher response rates for physician surveys often utilize monetary incentives for participants (which may lead to a different bias in the results), a practice not permitted for data collection within the VA. Lastly, we did not have access to any demographic or practice-based characteristics of non-respondent physicians for comparison.

Overall, our survey data suggest that current capabilities for test result management are limited even within a well-established, mature EHR. A comprehensive socio-technical approach is needed to optimize EHR-based test result management. Key components of this socio-technical approach will include: (1) design, development, and testing of EHR features and functions to support physicians' test result management workflows and inclusion of these new features in all EHRs; (2) new policies and procedures at the local institutional and national level regarding appropriate methods and timeliness of patient notification; and (3) commitments from organizational leadership and payers to acknowledge and support the additional, non-face-to-face work required to provide care for patients in an EHR-enabled healthcare system.

Acknowledgements We thank Annie Bradford, PhD for assistance with medical editing.

Funding The research reported here was supported by the Department of Veterans Affairs National Center for Patient Safety and in part by the Houston VA HSR&D Center of Excellence (HFP90-020), the National Library of Medicine (R01 LM006942), and the Office of the National Coordinator for Health Information Technology (ONC #10510592).

Competing interests None.

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or other funding agencies. The corresponding author had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Ethics approval Baylor College of Medicine Institutional Review Board approved this study.

Provenance and peer review Not commissioned; externally peer reviewed.

Open Access: This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: licenses/by-nc/3.0/


1 Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007;8:32.

2 Schiff GD, Hasan O, Kim S, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881-7.

3 Lorincz CY, Drazen E, Sokol PE, et al. Research in Ambulatory Patient Safety 20002010: a 10-year review. Am Med Assoc 2011. Available at: patientsafety. (accessed 3 Dec 2012).

4 Wynia MK, Classen DC. Improving ambulatory patient safety. JAMA 2011;306:22 2504-5.

5 Callen JL, Westbrook JI, Georgiou A, et al. Failure to follow-up test results for ambulatory patients: a systematic review. J Gen Intern Med 2011.

6 Murff HJ, Gandhi TK, Karson AK, et al. Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems. Int J Med Inform 2003;71:137-49.

7 Poon EG, Wang SJ, Gandhi TK, et al. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36:80-91.

8 Singh H, Vij MS. Eight recommendations for policies for communicating abnormal test results. Jt Comm J Qual Patient Saf 2010;36:226-32.

9 Tenner CT, Shapiro NM, Wikler A. Improving health care provider notification in an academic setting: a cascading system of alerts. Arch Intern Med 2010;170:392.

10 Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med 2009;169:1578-86.

11 Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med 2010;123:238-44.

12 Singh H, Arora H, Vij M, et al. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14:459-66.

13 Casalino LP, Dunham D, Chin MH, et al. Frequency of Failure to Inform Patients of Clinically Significant Outpatient Test Results. Arch Intern Med 2009;169:1123-9.

14 Elder NC, McEwen TR, Flach J, et al. The management of test results in primary care: does an electronic medical record make a difference? Fam Med 2010;42:327-33.

15 Health Information Technology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information Technology. Office of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services. Federal Register 2012 (cited 2012 July 25); 77:13845 (accessed 3 Dec 2012)

16 Hysong SJ, Sawhney MK, Wilson L, et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak 2011;11:22.

17 Final Report: Summary review: Evaluation of Veterans Health Administration procedures for communicating abnormal test results. Report number 01-01965-24. 11-25-2002. Ref Type: Report.

18 Hysong SJ, Sawhney MK, Wilson L, etal. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc 2010;17:71-7.

19 Singh H, Spitzmueller C, Petersen NJ, et al. Information overload and missed test results in electronic health record-based settings. Arch Intern Med. In press.

20 Taylor SL, Dy S, Foy R, et al. What context features might be important determinants of the effectiveness of patient safety practice interventions? BMJ Qual Saf 2011;20:611-17.

21 Yano EM, Simon B, Canelo I, et al. 1999 VHA survey of primary care practices. VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. 00-MC12. 2000. VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. Ref Type: Report.

22 Forward momentum: Hospital use of information technology. http://www.aha org/ content/00-10/FINALNonEmbITSurvey105 pdf (2005 (accessed 3 Dec 2012).

23 Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995;7:57-78.

24 Clason DL, Dormody TJ. Analyzing data measured by individual Likert-type items. J Agricultural Educ 2000;35:31-6.

25 Lyons SS, Tripp-Reimer T, Sorofman BA, et al. VA QUERIinformatics paper: information technology for clinical guideline implementation: perceptions of multidisciplinary stakeholders. J Am Med Inform Assoc 2005;12:64-71.

26 Campbell EG, Regan S, Gruen RL, et al. Professionalism in medicine: results of a national survey of physicians. Ann Intern Med 2007;147:795-802.

27 Jha AK, Ferris TG, Donelan K, et al. How common are electronic health records in the United States? A summary of the evidence. Health Aff (Millwood) 2006;25: w496-507.

28 Cutler DM, Feldman NE, Horwitz JR. U.S. adoption of computerized physician order entry systems. Health Aff (Millwood) 2005;24:1654-63.

29 Ash JS, Gorman PN, Seshadri V, et al. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11:95-9.

30 Boohaker E, Ward RE, Uman JE, et al. Patient notification and follow-up of abnormal test results. A physician survey. Arch Intern Med 1996;156:327-31.

31 Poon E, Gandhi T, Sequist T, et al. "Iwish I had seen this test result earlier!": dissatisfaction with test result management systems in primary care. Arch Intern Med 2004;164:2223-8.

32 Hickner J, Graham DG, Elder NC, et al. Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care 2008;17:194-200.

33 Elder NC, Dovey SM. Classification of medical errors and preventable adverse events in primary care: a synthesis of the literature. J Fam Pract 2002;51:927-32.

34 Elder NC, Pallerla H, Regan S. What do family physicians consider an error? A comparison of definitions and physician perception. BMC Fam Pract 2006;7:73.

35 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010;19(Suppl 3):i68-74.

36 VHA Directive 2009-019: Ordering and Reporting Test Results. Veterans Health Administration 2009 pub_ID=1864 (accessed 3 Dec 2012)

37 Singh H, Wilson L, Reis B, et al. Ten strategies to improve management of abnormal test result alerts in the electronic health record. J Patient Saf 2010;6:121-23.

38 Blumenthal D, Tavenner M. The "Meaningful Use" Regulation for Electronic Health Records. N Engl J Med 2010;363:501-4.

39 Sittig DF, Singh H. Improving test result follow-up through electronic health records requires more than just an alert. J Gen Intern Med 2012;27:1235-37.

40 Fraser HS, Allen C, Bailey C, et al. Information systems for patient follow-up and chronic management of HIV and tuberculosis: a life-saving technology in resource-poor areas. J Med Internet Res 2007;9:e29.

41 Archer N, Fevrier-Thomas U, Lokker C, et al. Personal health records: a scoping review. J Am Med Inform Assoc 2011;18:515-22.

42 Sittig DF. Personal health records on the internet: a snapshot of the pioneers at the end of the 20th Century. IntJMed Inform 2002;65:1-6.

43 CLIA Program and HIPAA Privacy Rule. Patients' Access to Test Reports (42 CFR 493, 45 CFR 164). Fed Regist 2011;76.

44 Davis Giardina T, Singh H. Should patients get direct access to their laboratory test results? An answer with many questions. JAMA 2011;306:2502-3.

45 Singh H, Graber M. Reducing diagnostic error through medical home-based primary care reform. JAMA 2010;304:463-4.

46 Woods DD, Patterson E, Roth EM. Can we ever escape from data overload? A cognitive systems diagnosis. Cogn Technol Work 2002;4:22-36.

47 Brantley SD, Brantley RD. Reporting significant unexpected findings: the emergence of information technology solutions. J Am Coll Radiol 2005;2:304-7.

48 Kuperman GJ, Teich JM, Tanasijevic MJ, et al. Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc 1999;6:512-22.

49 Petersen LA, Orav EJ, Teich JM, et al. Using a computerized sign-out program to improve continuity of inpatient care and prevent adverse events. Jt Comm J Qual Improv 1998;24:77-87.

50 Singh H, NaikA, Rao R, et al. Reducing diagnostic errors through effective communication: harnessing the power of information technology. J Gen Int Med 2008;23:489-94.

51 Woolf SH, Kuzel AJ, Dovey SM Jr, et al. A string of mistakes: the importance of cascade analysis in describing, counting, and preventing medical errors. Ann Fam Med 2004;2:317-26.

52 Gandhi TK, Sittig DF, Franklin M, et al. Communication breakdown in the outpatient referral process. J Gen Intern Med 2000;15:626-31.

53 Dismukes K. Remembrance of things future: prospective memory in laboratory, workplace, and everyday settings. Hum Factors Ergon Soc 2010;6:79-122.

54 Fisher ES, McClellan MB, Safran DG. Building the path to accountable care. N Engl J Med 2011;365:2445-7.

55 Kellerman SE, Herold J. Physician response to surveys. A review of the literature. Am J Prev Med 2001;20:61-7.

56 Singh H, Thomas EJ, Wilson L, et al. Errors of diagnosis in pediatric practice: a multisite survey. Pediatrics 2010;126:70-9.

57 Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129-36.

58 Cummings SM, Savitz LA, Konrad TR. Reported response rates to mailed physician questionnaires. Health Serv Res 2001;35:1347-55.

59 Halpern SD, Kohn R, Dornbrand-Lo A, et al. Lottery-based versus fixed incentives to increase clinicians' response to surveys. Health Serv Res 2011;46:1663-74.

60 Nicholls K, Chapman K, Shaw T, etal. Enhancing response rates in physician surveys: the limited utility of electronic options. Health Serv Res 2011;46:1675-82.

61 Poon EG, Wright A, Simon SR, et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care 2010;48:203-9.

62 Simon SR, Soran CS, Kaushal R, et al. Physicians' use of key functions in electronic health records from 2005 to 2007: a statewide survey. J Am Med Inform Assoc 2009;16:465-70.