Studying workflow and workarounds in EHR-supported work to improve health system performance

Clinical workflow represents the instantiation of all clinical activities. The transition from paper to electronic health records (EHRs) over the past decade has been characterized by profound challenges supporting clinical workflow, impeding frontline clinician ability to deliver safe, efficient, and effective care. In response, there has been substantial effort to study clinical workflow as well as workarounds – exceptions to routine workflow – in order to identify opportunities for improvement. In this paper, we describe predominant methods of studying workflow and workarounds as well as provide examples of the applications of these methods along with the resulting insights. We also present challenges to studying workflow and workarounds, along with recommendations for how to approach such studies. While there is not yet a set of standard approaches, our work helps advance workflow research that ultimately serves to inform how to coevolve the design of EHR systems and organizational decisions about processes, roles, and responsibilities in order to support clinical workflow that more consistently delivers on the potential benefits of a digitized healthcare system.

INTRODUCTION

The past decade has witnessed a rapid transition from largely paper-based documentation to largely digital documentation, with the vast majority of hospitals(1) and the majority of ambulatory providers using at least a basic electronic health record (EHR).(2) The transition was motivated by the expectation of substantial quality and efficiency gains as well as the ability to use digital data from EHRs to support clinical research, quality improvement, public health, and more. A foundational assumption was that EHRs would be readily able to integrate into and support clinical work. Reality proved more complicated and EHR implementation has often resulted in profound challenges with supporting clinical workflow. Suboptimal EHR-mediated workflows, in turn, promote clinicians working around the EHR. Workarounds contribute to a digital record that does not accurately capture patient care and also drive clinician dissatisfaction as well as unintended adverse consequences for patients.(3–5) (6–9)

There is therefore a critical need to study workflow and workarounds in the context of EHR use. Such work serves to identify opportunities to improve EHRs and EHR-mediated workflows as well as more broadly inform research and health system improvement in domains of safety, quality, and efficiency. In contrast to traditional clinical research that has been adapted to leverage EHR data, research focused on EHR-mediated workflows is a new domain of inquiry. As a result, there are few research and reporting standards available. To speed maturation, in this paper we provide an overview of commonly-used research methodologies, followed by three examples of insights into safety, quality, and efficiency that can be gleaned from using these methods to study workflow and workarounds. We then discuss the methodological challenges to studying workflow and workarounds, and our recommendations for how to improve the scientific rigor and results reporting consistency of such studies.

OVERVIEW OF KEY CONCEPTS

Clinical workflow

Workflow is “a set of tasks, grouped chronologically into processes, and the set of people or resources needed for those tasks that are necessary to accomplish a given goal.”(10) Clinical workflow is the nexus of all clinical activities. It is essential to the effective and safe delivery of patient care. In most care settings, clinical workflow is complex, reflecting the multifaceted nature of clinical tasks and the interdependencies between them. Clinical workflow is also fragile and can be easily disrupted by changes in the order or methods by which clinical tasks are completed.(5)

In the EHR era, coordination of clinical workflow increasingly relies on the use of computerized systems but evidence suggests that current-generation EHR systems inadequately support clinical workflow and the cognitive tasks of clinicians.(11) Suboptimal workflow is therefore a common phenomenon in EHR implementation projects, and results from a wide range of problems, including poor software usability, complex intersystem dependencies, and the lack of sociotechnical integration of software systems into complex behavioral, organizational, and societal surroundings.

Workarounds

Workarounds are “informal temporary practices for handling exceptions to normal workflow.”(12) In healthcare, workarounds are often characterized as clinicians’ self-created solutions to accomplishing a work goal within a system of dysfunctional work processes that prohibits or impedes accomplishing that goal.(13) (14) Nonetheless, a key reason for adopting computerized systems in healthcare is to intentionally introduce blocking conditions into workflow, such as electronic patient identity verification and computerized medication safety alerts, to prevent unsafe practices (hard stops) or make them more difficult to conduct (soft stops).(15) Some of these workflow blocks may however be perceived by clinicians as unnecessary or inconvenient, provoking the workaround behavior.

On the other hand, workflow blocks enforced through health IT could be introduced improperly or unintentionally, due to the misalignment between the ideal workflow as perceived by software designers and healthcare administrators versus what clinicians experience in their day-to-day practice.(16) Thus, workflow predefined in EHR systems is often idealistic, linear, and unable to accommodate complex or unexpected situations.(17) This issue is further exacerbated by poorly designed software user interfaces, insufficient user training, low specificity of computerized alerts, and adoption of a common EHR across medical specialties.

APPROACHES TO STUDYING WORKFLOW AND WORKAROUNDS

There is increasing recognition of the need for research that examines workflow and workarounds in the context of EHR use. The following scenario describes an IT implementation and key issues in designing a scientifically-rigorous study to investigate the associated clinical workflow:

Hospital Hope recently implemented a computerized prescriber order entry (CPOE) function in its EHR system. The purpose was to streamline medication ordering processes and improve patient safety through adding computerized decision-support capabilities such as drug-drug interaction alerts. Hospital administrators are aware that use of the CPOE system to enter medication orders could produce undesirable workflows that include unwanted workaround behaviors and have detrimental effects on time efficiency and patient safety. They therefore ask a research team to study provider workflow when using the CPOE function and to identify any issues that could contribute to unintended adverse consequences for providers or patients. To conduct the study properly, the researchers need to make many choices on study design, including but not limited to: (1) what are core attributes of clinical workflow related to medication ordering that could impact efficiency and safety outcomes; (2) how to operationalize these attributes so that they can be empirically measured; and (3) how to collect unbiased, empirical data to create these measures in a manner that does not interfere with clinical work.

Identifying the aspects of clinical workflow that are important to study and can be reliably measured is a longstanding challenge to workflow studies. While suboptimal workflow is a commonly discussed subject in health IT research(18), few studies actually measure workflow directly. Instead, most studies report indications that workflow had likely been altered from it’s prior or intended state as evidenced by differences observed in distal measures (e.g., improvement in guideline compliance) or clinical outcomes (e.g., reduction in patient safety events).(19) Even among studies that directly assess changes in workflow, many focus on health IT’s impact on time utilization (e.g., average total time spent in direct patient care activities vs. using the computer), rather than flow of the work. This distinction is important because the spirit of workflow lies in the chronological organization of clinical tasks and the temporal (inter)dependencies among them.(20) More broadly, these studies make an implicit assumption that pre-IT workflow is the appropriate comparator. However, it may be that pre-IT workflow was also suboptimal, such that evidence of “improvement” could result in misguided conclusions about a successful transition. Focusing on pre-to-post changes in workflow may also blind researchers to the unique IT-related issues that can lead to suboptimal workflow, such as alert fatigue. Therefore, the appropriate research question framing is not whether workflow is better or worse than prior to IT adoption, but instead whether there is any evidence that workflow is suboptimal when using IT to support clinical work.

Collecting unbiased empirical data that accurately reflect the desired workflow measures is a distinct but closely related challenge. The most commonly-used approaches in the literature for studying workflow and workarounds are qualitative methods, such as ethnographic observations (including video ethnography), interviews, and focus groups. For example, through user interviews and document analysis, Niazkhani et al. found that workflow interruptions from a CPOE system encouraged clinician workarounds.(21) Qualitative methods are particularly well-suited to generating rich user accounts and capturing user perspectives on workflow. However, they are susceptible to prejudices and biases commonly found in self-reported data due to factors such as reluctance to change, negative emotion, and recall errors.(22)

Quantitative studies on workflow that do not rely on self-reported data typically assess what aspects of the workflow are modified (pre-post) or assess cross-sectional variation. Among these studies, time-motion and log analysis are two methods that have been most often used. In time-motion studies, independent human observers shadow clinician subjects for a continuous period of time to record how they perform their clinical tasks (what, when, where, how, and for how long).(20) This method, originally developed in industrial engineering, is considered the most accurate way to collect workflow data as compared to alternative methods such as work sampling and time efficiency questionnaires.(23, 24) However, conducting time-motion studies is resource intensive, and results are subject to limitations such as small sample size, observer bias, and the Hawthorne effect (the alteration of behavior by the subjects of a study due to their awareness of being observed).(25) A comprehensive review of time-motion studies in healthcare can be found in Lopetegui et al.(26)

Another popular quantitative method for studying clinical workflow is log file analysis. In addition to time-stamped clinical data, all computerized systems in healthcare, mandated by the Health Insurance Portability and Accountability Act and the Meaningful Use criteria, must implement robust security auditing mechanisms for detecting malicious access to, or alteration of, protected health information. Specifically, log files capture user activity while logged in to the EHR, though the level of granularity and comprehensiveness of what user behaviors are logged may differ across EHR systems. Despite limitations (including that data only exist following EHR implementation), such data have been increasingly used by researchers to examine clinician interaction with health IT systems. For example, Zheng et al. studied clinicians’ workflow in an EHR system using automatically recorded log files and found that how clinicians actually navigated in the system deviated dramatically from how designers of the system intended it to be used(27). Tai-Seale et al. used EHR log files to examine physician workflow and Hirsch et al. examined time utilization in primary care practices.(28, 29) A growing number of studies also use a mixed-methods approach that leverages log analysis to supplement qualitative investigation.(30)

In recent years, several new methods have emerged for studying workflow and workarounds that leverage data automatically collected using software tools (e.g., screen capturing software) or through sensor technology such as eye tracking devices, 3D infrared laser projectors (e.g., Microsoft Kinect), and radio-frequency identification (RFID).(31–33) These methods, collectively referred to as “computational ethnography,” have the potential to substantially reduce the resource requirement for conducting large-scale workflow studies while producing more granular data delineating detailed patterns of workflow and the workaround behavior than are typically captured by direct observation or log files.(34)

EXAMPLES OF INSIGHTS FROM STUDYING WORKFLOW AND WORKAROUNDS

To illustrate the value of these methods in generating insights to improve workflow and associated outcomes, this section describes three examples of the application of these methods and the specific resulting insights.

Example 1. Evaluating Computerized Provider Order Entry (CPOE) System Impact on Pediatric ICU Workflow

Zheng et al. conducted a time-motion study in a pediatric ICU to observe resident physicians’ clinical workflow before and after introducing CPOE.(20) The study was also designed to investigate a disagreement in the literature between quantitative studies that consistently found insignificant or marginal impact of CPOE on workflow compared to qualitative investigations that consistently reported that “new work” and “unfavorable workflow” were two prominent adverse consequences.

The investigators used a set of new workflow measures, such as workflow fragmentation (the frequency of task switching and interruptions), and new workflow modeling approaches, such as sequential pattern mining and task transition analysis, to uncover hidden regularities embedded in clinical workflow. The results showed that, consistent with prior quantitative studies, how clinicians distributed their time across different patient care tasks only changed marginally before and after CPOE implementation. In particular, the post-implementation workflow became more fragmented, consisting of shorter duration of tasks and higher frequency of task switching. The authors argued that changes to the temporal dynamics of workflow, rather than redistribution of clinician time, might explain why clinicians perceived higher workload and disrupted workflow after health IT adoption, even though the aggregated total amount of time they spent on performing each type of clinical tasks did not change substantially. The results of the study also provided implications for redesigning health IT, clinical practice routines, and the physical layout of medical facilities to reduce workflow disruption, by minimizing cognitive load associated with fast task switching and switching between tasks of distinct natures, and avoiding extra physical activities (e.g., locating computer workstation or printer) and unnecessary waiting and idling (e.g., waiting to be re-logged into the computer due to rapidly expired sessions).

Example 2. Understanding Physician EHR Workflow Patterns and the Impact on Task Time and Errors.

A study by Ratwani et al. used screen-capture software to record all EHR interface-level interactions, such as mouse clicks and keystrokes, to understand emergency physician workflow and workarounds.(35) The study was conducted at four healthcare systems that use two different EHR vendor products (Epic and Cerner) and sought to measure the variability in workflow and resulting time-on-task and error rates as physicians completed six different clinical scenarios. Screen capture software enabled an understanding of EHR interaction patterns that would not traditionally be captured by log data, such as processes used to search for information or hovering the mouse over different medication options in a non-intuitive display.

The results revealed wide variability in workflow, time-on-task, and error rates, both within and across healthcare systems. Within a single healthcare system, physicians demonstrated very different workflows, which resulted in some physicians completing tasks in half the time as others and without any errors. Across healthcare systems, there was tremendous variability for some tasks, such as ordering a lab that had an eight-fold difference in time and clicks from one site to another. Error rates for a task such as ordering a medication ranged from zero to thirty percent. These results highlight the need to more closely analyze workflow to reduce burdensome processes and reduce the likelihood of error. Time-on-task and error rates can be reduced by increasing physician awareness of different EHR workflows so that those physicians using more cumbersome ones can change their behavior and by simplifying EHR workflow through redesigning EHR systems to eliminate unnecessary keystrokes and clicks.

Example 3. Improving Value in the Emergency Department Setting from Rapid Access to Outside Records.

A study by Everson, Kocher, and Adler-Milstein used log files to capture the workflow in two emergency departments (ED) within a large academic medical center related to requesting and viewing patient records from outside provider organizations.(36) The study sought to measure how often requested records were returned and, when returned, how quickly they were viewed by the requesting provider, using log file measures to compare the usual care workflow (i.e., EHR-based request and viewing of scanned records that were retrieved by phone/fax) to a newly available EHR-based workflow in which records could be queried for and retrieved from EHR-to-EHR (i.e., electronic health information exchange, or HIE). Ultimately, the research sought to assess whether there was a relationship between information request-to-viewing time and clinical outcomes for ED patients.

The research revealed that, while a greater proportion of requests were fulfilled via usual care, when fulfilled, providers were more likely to view those returned via HIE and to view them on average 58.5 minutes earlier. Further, for every hour saved between requesting and viewing outside records, patients spent less time in the ED, they were less likely to receive imaging, and they were less likely to be admitted to the hospital. Beyond revealing the value from designing a workflow that ensures rapid availability of outside records to ED clinicians, the study also revealed the need to address upfront challenges with patient matching as well as the downstream challenge of returned records that were never viewed.

METHODOLOGICAL CHALLENGES TO STUDYING WORKFLOW AND WORKAROUNDS

In this section, we describe the methodological challenges related to proper measurement of workflow and how to collect unbiased empirical data that accurately reflect the desired workflow measures. We then provide our recommendations for how to improve the rigor in conducting workflow studies and the consistency in results reporting. These recommendations are also summarized in Table 1 .

Table 1.

Summary of Research and Reporting Recommendations for Studying Workflow and Workarounds in EHR-supported Work

First, while workflow disruption is frequently discussed in the health IT evaluation literature, few studies measure it directly. Instead, time efficiency, clinician performance, or patient outcomes are commonly used as distal or proxy measures.(19) It is also not uncommon that an investigation into workflow changes is only conducted as an afterthought especially when desirable outcomes of a health IT implementation were not achieved. We therefore recommend future efforts recognize the importance of direct study of workflow and workarounds, and use pertinent workflow measures to directly capture the completion of a set of tasks, and their chronology, that seek to accomplish a given goal. In the context of the scenario, the former is represented by the hospital administrators asking a research team to study provider workflow as part of the CPOE implementation. While resource intensive, routine assessment of workflow as part of implementation efforts is likely to result in early identification of potential issues and avoidance of unintended consequences. The latter could be put into practice by the research team by first mapping out all clinical tasks relevant to medication ordering, identifying which ones are likely affected by the CPOE implementation, and identifying members of the clinical team that are involved and the responsibility that each assumes.

Second, workflow is a highly elastic concept that is often interpreted differently across people, settings, and research goals. The research definition of clinical workflow thus tends to be elusive.(37) Consequently, findings on health IT’s impact on workflow can be inconclusive or even conflicting. For example, workflow studies using the time-motion approach have commonly found that health IT implementation does not have a detrimental impact on time efficiency.(38) However, decreased time efficiency and disrupted clinical workflow are among the most frequently reported unintended adverse consequences by end users.(9) This problem is exacerbated by variability in clinical workflow, even within the same institution. We therefore recommend that future research clearly define clinical workflow in the context of the study, and adopt proper workflow measures in line with the research objectives, as well as report on observed variability in the measures once collected. A clear definition would involve specifying the goal of the workflow as well as the tasks and their chronology that are expected to result in the goal, along with how each was measured. In the context of the scenario, this would extend the workflow mapping effort by developing and reporting workflow measures that could specifically contribute to adverse outcomes for patients or providers (e.g., creation of new tasks, shift of responsibility by clinical roles, and altered chronological order of task execution).

Third, data sources and methods that can be used to study workflow and workarounds vary to a great extent.(19) For example, ethnographic observations would be adequate to study the tasks and their chronology, but not support quantifying pre-to-post changes. We therefore recommend that researchers develop familiarity with the range of available data sources and methods, as well as strengths and limitations of each. Further, even among studies using the same method, a considerable degree of discrepancy exists in how the method is applied and how study results are interpreted. For example, among the time-motion studies published to date, there has been a large degree of methodological inconsistency, such as how inter-observer reliability is calibrated and how multitasking is handled.(25) We therefore recommend that, whenever possible, researchers use established methods for common research activities used in workflow studies. These include adopting common workflow measures, using a standardized taxonomy of clinical tasks, and reducing variation in how workflow data are collected (particularly in a pre-post implementation study). In the context of the scenario, the researchers may choose to use the time-motion method to capture workflow data. If they did, they should consider maintaining the consistency of observer-observee pairs across pre- and post-implementation stages, using a clinical task taxonomy based on standard CPOE task classifications (e.g. the one developed by Overhage et al. (38) and extended by Pizziferri et al.(39)), and adopting an existing, validated time-motion data collection tool such as the Time Capture Tool (TimeCat(40)) or the Work Observation Method by Activity Timing (WOMBAT(41)).

Fourth, there is a large degree of inconsistency in how workflow studies report their study design and results, which makes cross-study synthesis very difficult, diminishing the ability to accumulate knowledge as a field. In response, there have been attempts to develop guidelines to improve methodological and results reporting consistency for certain types of workflow studies. For example, Zheng et al. previously developed a checklist called Suggested Time and Motion Procedures (STAMP) that defines a minimum set of 29 information elements that should be reported in workflow studies utilizing the time-motion method.(25) These 29 data elements are organized into nine key areas that guide researchers to provide detailed accounts of the intervention studied, the empirical setting, the research approach (e.g., randomized controlled trial vs. pre-post comparison), the clinical task taxonomy used, observer training and inter-rater reliability assessments, characteristics of study subjects, data recording methods, data analysis methods, and how interruption and interaction are handled. We recommend future workflow studies use such publication standards for reporting their research methods and results.

CONCLUSION

Given the critical patient safety risks as well as broader concerns about burnout and clinical inefficiencies that result from suboptimal workflow, there is an urgent need to continue to devote effort and attention to studying and improving clinical workflow in the EHR context. Workflow and workarounds are complex phenomena and a robust understanding of the challenges and opportunities for improvement require greater attention to the rigor of the research methods and consistency in results reporting. Since the field is sufficiently new that there are not yet widely-adopted standard approaches, our work describes commonly-used qualitative and quantitative research methodologies along with examples of how these methods have been used to glean insights into safety, quality, and efficiency from studying workflow and workarounds. We seek to advance the field by discussing four methodological challenges to studying workflow and workarounds, and offering recommendations on how to improve the scientific rigor and results reporting consistency of such studies. Doing so will spur efforts to continuously improve how EHRs, and the broader work systems in which they are embedded, support clinical workflow.

Acknowledgments

Financial Support: None

REFERENCES

1. Adler-Milstein J, Holmgren AJ, Kralovec P, Worzala C, Searcy T, Patel V. Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide . Journal of the American Medical Informatics Association . 2017; 24 ( 6 ):1142–8. [PMC free article] [PubMed] [Google Scholar]

2. Jamoom EW, Yang N, Hing E. Adoption of Certified Electronic Health Record Systems and Electronic Information Sharing in Physician Offices: United States, 2013 and 2014 . 2016. [PubMed]

3. Kim MO, Coiera E, Magrabi F. Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review . Journal of the American Medical Informatics Association . 2017; 24 ( 2 ):246–50. [PMC free article] [PubMed] [Google Scholar]

4. Rosenbaum L Transitional chaos or enduring harm? The EHR and the disruption of medicine . New England Journal of Medicine . 2015; 373 ( 17 ):1585–8. [PubMed] [Google Scholar]

5. Zheng K, Ciemins E, Lanham H, Lindberg C, Man D. Examining the Relationship Between Health IT and Ambulatory Care Workflow Redesign . Rockville, MD: Agency for Healthcare Research and Quality. 2015. [Google Scholar]

6. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm . Jama . 2018; 319 ( 12 ):1276–8. [PMC free article] [PubMed] [Google Scholar]

7. Ratwani RM, Savage E, Will A, Fong A, Karavite D, Muthu N, et al. Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings . Health affairs 2018; 37 ( 11 ):1752–9. [PubMed] [Google Scholar]

8. Zheng K, Abraham J, Novak L, Reynolds T, Gettinger A. A survey of the literature on unintended consequences associated with health information technology: 2014–2015 . Yearbook of medical informatics . 2016; 25 ( 01 ):13–29. [PMC free article] [PubMed] [Google Scholar]

9. Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting . Journal of the American Medical Informatics Association . 2011; 18 ( 1 ):82–90. [PMC free article] [PubMed] [Google Scholar]

11. National Research Council. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions . Washington DC; 2009. [PubMed] [Google Scholar]

12. Kobayashi M, Fussell SR, Xiao Y, Seagull FJ. Work coordination, workflow, and workarounds in a medical context. CHI’05 extended abstracts on Human factors in computing systems , 2005. ACM: 1561–4. [Google Scholar]

13. Halbesleben JR, Wakefield DS, Wakefield BJ. Work-arounds in health care settings: literature review and research agenda . Health care management review . 2008; 33 ( 1 ):2–12. [PubMed] [Google Scholar]

14. Ash JS, Gorman PN, Lavelle M, Payne TH, Massaro TA, Frantz GL, et al. A cross-site qualitative study of physician order entry . Journal of the American Medical Informatics Association . 2003; 10 ( 2 ):188–200. [PMC free article] [PubMed] [Google Scholar]

15. Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review . Journal of the American Medical Informatics Association . 2018; 25 ( 11 ):1556–66. [PMC free article] [PubMed] [Google Scholar]

16. Zheng K, Hanauer DA, Padman R, Johnson MP Jr, Hussain AA, Ye W, et al. Handling Anticipated Exceptions in Clinical Care: Investigating the Benefits and Consequences of Providing “Exit Strategies” in Computerized Clinical Information Systems . Journal of the American Medical Informatics Association . 2011; 18 :883–9. [PMC free article] [PubMed] [Google Scholar]

17. Niazkhani Z, Pirnejad H, Berg M, Aarts J. The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review . Journal of the American medical informatics Association . 2009; 16 ( 4 ):539–49. [PMC free article] [PubMed] [Google Scholar]

18. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The Benefits Of Health Information Technology: A Review Of The Recent Literature Shows Predominantly Positive Results . Health Affairs . 2011; 30 ( 3 ):464–71. [PubMed] [Google Scholar]

19. Carayon P, Karsh B, Cartmill R, Hoonakker P, Hundt AS, Krueger D, et al. Incorporating health information technology into workflow redesign-summary report . Rockville, MD: Agency for Healthcare Research and Quality. 2010(10–0098). [Google Scholar]

20. Zheng K, Haftel HM, Hirschl RB, O’reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective . Journal of the American Medical Informatics Association . 2010; 17 ( 4 ):454–61. [PMC free article] [PubMed] [Google Scholar]

21. Niazkhani Z, Pirnejad H, van der Sijs H, Aarts J. Evaluating the medication process in the context of CPOE use: the significance of working around the system . International journal of medical informatics . 2011; 80 ( 7 ):490–506. [PubMed] [Google Scholar]

22. Schwarz N, Oyserman D. Asking questions about behavior: Cognition, communication, and questionnaire construction . The American Journal of Evaluation . 2001; 22 ( 2 ):127–60. [Google Scholar]

23. Finkler SA, Knickman JR, Hendrickson G, Lipkin M Jr, Thompson WG. A comparison of work-sampling and time-and-motion techniques for studies in health services research . Health services research . 1993; 28 ( 5 ):577. [PMC free article] [PubMed] [Google Scholar]

24. Burke TA, McKee JR, Wilson HC, Donahue RM, Batenhorst AS, Pathak DS. A comparison of time-and-motion and self-reporting methods of work measurement . Journal of Nursing Administration . 2000; 30 ( 3 ):118–25. [PubMed] [Google Scholar]

25. Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research . Journal of the american medical informatics Association . 2011; 18 ( 5 ):704–10. [PMC free article] [PubMed] [Google Scholar]

26. Lopetegui M, Yen P-Y, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about? Journal of biomedical informatics . 2014; 49 :292–9. [PMC free article] [PubMed] [Google Scholar]

27. Zheng K, Padman R, Johnson MP, Diamond HS. An interface-driven analysis of user interactions with an electronic health records system . Journal of the American Medical Informatics Association . 2009; 16 ( 2 ):228–37. [PMC free article] [PubMed] [Google Scholar]

28. Tai-Seale M, Olson CW, Li J, Chan AS, Morikawa C, Durbin M, et al. Electronic Health Record Logs Indicate That Physicians Split Time Evenly Between Seeing Patients And Desktop Medicine . Health Affairs . 2017; 36 ( 4 ):655–62. [PMC free article] [PubMed] [Google Scholar]

29. Hirsch AG, Jones JB, Lerch VR, Tang X, Berger A, Clark DN, et al. The electronic health record audit file: the patient is waiting . Journal of the American Medical Informatics Association . 2017; 24 ( e1 ):e28–e34. [PMC free article] [PubMed] [Google Scholar]

30. Koppel R, Wetterneck T, Telles JL, Karsh B-T. Workarounds to Barcode Medication Administration Systems: Their Occurrences, Causes, and Threats to Patient Safety . Journal of the American Medical Informatics Association . 2008; 15 ( 4 ):408–23. [PMC free article] [PubMed] [Google Scholar]

31. Furniss SK, Burton MM, Grando A, Larson DW, Kaufman DR. Integrating Process Mining and Cognitive Analysis to Study EHR Workflow . AMIA Annual Symposium Proceedings, Nov 12–16, 2016. American Medical Informatics Association (Chicago, IL): 580. [PMC free article] [PubMed] [Google Scholar]

32. Calvitti A, Hochheiser H, Ashfaq S, Bell K, Chen Y, El Kareh R, et al. Physician activity during outpatient visits and subjective workload . Journal of Biomedical Informatics . 2017; 69 :135–49. [PubMed] [Google Scholar]

33. Kannampallil T, Li Z, Zhang M, Cohen T, Robinson DJ, Franklin A, et al. Making sense: sensor-based investigation of clinician activities in complex critical care environments . Journal of Biomedical Informatics . 2011; 44 ( 3 ):441–54. [PubMed] [Google Scholar]

34. Zheng K, Hanauer DA, Weibel N, Agha Z. Computational Ethnography: Automated and Unobtrusive Means for Collecting Data In Situ for Human–Computer Interaction Evaluation Studies. Cognitive Informatics for Biomedicine : Springer; 2015:111–40. [Google Scholar]

35. Ratwani RM, Savage E, Will A, Arnold R, Khairat S, Miller K, et al. A usability and safety analysis of electronic health records: a multi-center study . Journal of the American Medical Informatics Association . 2018; 25 ( 9 ):1197–201. [PMC free article] [PubMed] [Google Scholar]

36. Everson J, Kocher KE, Adler-Milstein J. Health information exchange associated with improved emergency department care through faster accessing of patient information from outside organizations . Journal of the American Medical Informatics Association . 2016. [PMC free article] [PubMed] [Google Scholar]

37. Unertl KM, Novak LL, Johnson KB, Lorenzi NM. Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review . Journal of the American Medical Informatics Association . 2010; 17 ( 3 ):265–73. [PMC free article] [PubMed] [Google Scholar]

38. Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled Trial of Direct Physician Order Entry: Effects on Physicians’ Time Utilization in Ambulatory Primary Care Internal Medicine Practices . Journal of the American Medical Informatics Association . 2001; 8 ( 4 ):361–71. [PMC free article] [PubMed] [Google Scholar]

39. Pizziferri L, Kittler AF, Volk LA, Honour MM, Gupta S, Wang S, et al. Primary care physician time utilization before and after implementation of an electronic health record: A time-motion study . Journal of Biomedical Informatics . 2005; 38 ( 3 ):176–88. [PubMed] [Google Scholar]

40. Lopetegui M, Yen P-Y, Lai AM, Embi PJ, Payne PR. Time Capture Tool (TimeCaT): development of a comprehensive application to support data capture for Time Motion Studies . AMIA Annual Symposium Proceedings , 2012. American Medical Informatics Association: 596. [PMC free article] [PubMed] [Google Scholar]

41. Westbrook JI, Ampt A. Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication . international journal of medical informatics . 2009; 78 :S25–S33. [PubMed] [Google Scholar]