0
£0.00
0 items
It could be argued that Florence Nightingale, in 1853 during the Crimean War, was the first proponent of ‘clinical audit’ as we see it. Her careful and accurate accumulation of data in a statistical manner using visual data charts, colour coding and tabular presentation of data, enabled her to irrefutably demonstrate a direct relationship between reduction in mortality rate and improved hygiene and training practices.
Clinical audit is now defined as ‘involving improving the quality of patient care by looking at current practice and modifying it where necessary’ (White Paper, Working for Patients, 1998). Audit for sonographers ensures that they are complying with current best practice in standards of care, as advocated in contemporary guidelines and literature (Moore, 2022). This practice aims to identify short comings which could prevent achievement of these standards, and thus enables changes to be implemented to rectify any problems, in a cyclical nature (Figure 1 - Adatia et al., 2022).
Figure 1 – Visual depiction of the six key stages of the audit cycle. Taken from Adatia 2022
Moore (2022) suggests that, within the sonography field, there are 3 practical types of clinical audit. Firstly, a performance-related, peer reviewed audit. This method suggests that a portion of the sonographer’s work should be scrutinized by a clinical lead, for accuracy of imaging, measurement, documentation and report writing and that they are of a given, acceptable standard. Secondly, an outcome-related audit. This suggests that the sonographer’s findings should be reviewed against further investigations, or arthroscopic findings, to assess their accuracy. The last type of audit is the process audit, where pathways and processes in ultrasound departments should be assessed, to ensure the best possible pathway for all aspects of the patient experience. The scope of this audit covers the initial assessment, imaging, follow-ups, and onward referrals with reporting where appropriate.
Furthermore, the British Medical Ultrasound Society (BMUS) (2021) recently published an audit scoring tool (Figure 2), using the peer review process, advocated for by the Sports Medicine Ultrasound Group (SMUG). This method provides ‘agreement scores’ which are graded to indicate the level of discrepancy identified by the supervisor when reviewing the sonographer’s report.
Figure 2 - Audit grading scores table for use with the peer review process. Taken from BMUS (2021)
For the current study, the performance-related audit method was adopted as this was evaluated to be the most achievable method based on the nature of the clinical practice. Using the above method, it is possible to give a measurable value of perceived accuracy for the student's ultrasound scans and reports when scrutinized by an experienced practitioner.
Ultrasound (US) scans were performed on patients between January and June 2023 at a private MSK clinic, in St Peter Port, Guernsey. The patients were initially clinically assessed, with detailed history taken and physical examination performed by an MSK physiotherapist, or the resident orthopaedic surgeon. They were then referred on for US evaluation if appropriate. Approximately half of the scans taken had the clinical assessment performed by the student themself.
As part of the clinical assessment, the student took the patient’s history, performed a physical examination and made a working diagnosis. The student then performed an US scan, took images and produced a report based on the findings observed. Data, comprising images and reports, from 150 US examinations were included in the clinical audit. The scans were all musculoskeletal in nature, assessing joints, muscle, ligament, nerves and tendons.
The images were reviewed retrospectively by a clinical supervisor, MD, who had over 10 years of sonography experience and ran regular, high volume, US ultrasound clinics. The supervisor was unaware of the student’s report. The reviewed images were then discussed and graded in accordance with the ‘agreement score’ established by the BMUS. Any discrepancies between the reports were documented and further evaluated. Where there were differences in diagnoses, the supervisor’s interpretation of the results was deemed to be correct.
The ultrasound machines used were a Siemens Acuson (with 8 Mhz and 9.4 Mhz linear probe) and an Oesoate MyLabX6 (with a frequency range from 4 and 15 Mhz in either linear or curvilinear probes). The images comprised both stills and cine clips. If they needed to re-scan the patient, the same US machine was used, in order to maximise inter-observer reliability. To reduce error and variance in image capture and analysis, both the student and supervisor used a universally accepted gold standard for US examination. The Society of Radiographers (SoR) and the BMUS jointly produced the 2020 version of ‘Guidelines for Professional Ultrasound Practice’ (revised in 2022) which, in conjunction with the European Society of Musculoskeletal Radiology’s (ESSR) ‘Guidelines for Peripheral Joints’ (2010), provided the benchmark reference for this study.
Riley et al, (2010) suggested that the 94.8% agreement rate between a sonographer and two independent radiologists could be used as the gold-standard rate of agreement for future studies. However, this study was based on the images produced by a sonographer with 8 years of experience, rather than a student with less than one, as in the current study. Although there are no studies to suggest an alternate acceptable agreement rate, a lower percentage was concluded as being more appropriate. On discussion with the supervisor, an agreement level of 80% was deemed to be a satisfactory benchmark for the student with their level of training.
The summary of the anatomical sites examined, and agreement scores awarded by the student are depicted in Table 1. The ‘total’ row indicates the number of each scan allocated to each agreement score as result of comparison against the supervisor’s results.
The student demonstrated grade 1 (complete agreement with the supervisor’s report) in 82% of scans. A small proportion (17%) of scans were grade 2 (minor discrepancies) and only two scans were grade 3 (potentially significant discrepancies). No scans were grade 4 (definite, significant discrepancies). Table 2 provides details of the two cases allocated an agreement score of grade 3.
Another important result depicted in Table 1, is the number of Grade 2 discrepancies at the shoulder and foot/ankle (20% and 21%, respectively). On initial analysis of the data this suggested that the incidence of grade 2 discrepancies at these sites were disproportionally large. However, upon further analysis it was noted that most of these diagnoses occurred within the first 2 months of the study, when the student was less confident in the applications of US and diagnosing pathology from the images. As the study progressed and received continued tuition, the number of grade 2 discrepancies in the shoulder and foot/ankle complex decreased.
Table 1 – Audit Agreement results depicting the anatomical sites examined, the agreement scores awarded, and the total number of examinations performed
Table 2 – Details of the two grade 3 discrepancies identified between the student’s US and the retrospective MRI examination by the experienced supervisor(s)
Koshy et al (2007) suggest that reflective practice is one of the most effective ways to convey theoretical knowledge into practical use. This strategy is used extensively in health and medical education to equip students with lifelong skills and can take the form of portfolio assessment. This can be reflection on positive experiences, a satisfied patient, an accurate diagnosis, or a challenging scan performed well. Equally, reviewing images can lead to reflection on less positive incidences, such as a dissatisfied patient, inaccurate diagnoses, or failures to obtain the desired images of a structure.
Throughout this audit process, reflection was a constant companion to encourage improvement and reward good practice. Moore and Reeve (2022) suggest that 5% of US scans should be audited in order to ensure standards are being maintained. The alternative would be to audit a fixed number of scans, if a percentage was impractical. For most clinics this would be done in house, by the most experienced clinician, although external audits would be a useful, independent way of ensuring no bias. However, in a clinical context where diagnoses are far more subjective, an external audit, purely based on images rather than additional information and context provided by the patient, may lead to limitations in accuracy and effectiveness – on this basis the current study largely utilized an internal audit process. There is, however, always the question of who audits the auditors.
In the current study, the US examinations were largely well performed over a range of anatomical sites and had grade 1 agreement scores when compared to experienced sonographers. US of the foot/ankle complex represented the greatest number of examinations (33%), possibly because a number of anatomical sites were included together, but also because of the high uptake of sport locally and therefore, a high incidence of classical ankle sprains and calf/Achilles injuries were seen. Additionally, many examinations concerned the shoulder joint (28%). The disparity in areas investigated may be because US investigation of the shoulder is well-established as a highly accurate, cost effective and simple method for determined shoulder pathology. As US is comparable to MRI in most diagnostic circumstances (with the exception of labral tears) almost all patients with shoulder pains were recommended US.
These two commonly examined areas had good agreement scores but some instances of grade 2 discrepancies. Some discrepancies may have arisen due to the nature of the anatomical sites. For example, Jacobson et al (2004) suggested there is difficulty in differentiating a rotator cuff partial tear, from a tendinopathy, as both can appear hypoechoic and also can be present in the same tendon concurrently and as such may prove challenging for students to determine the exact nature of the pathology. As part of the audit, it would have been beneficial to determine whether the number of non-grade 1 agreement scores decreased over time as the student improved their technique and gained experience.
By contrast, only two examinations (1%) concerned the hand/wrist complex due to a small number of relevant patients. The nature of the injuries tended to be traumatic which resulted in referrals for CT or MRI scans as the initial method of imaging. However, referrals in these instances may have, in part, been due to the relative lack of experience of the therapists in the
clinic regarding the assessment and treatment of hand injuries, as well as the student feeling less comfortable performing US scans of this region. Chiavaras (2014) suggested that US was an effective tool for the assessment of the wrist and hand, but regard should be had for the potential pitfalls, such as limitations in distinguishing erosions from other forms of cortical irregularity. Therefore, student inexperience was a limitation of this assessment and further training on this anatomical site should be sought in future.
In two of the cases during this study, the discrepancies identified were potentially significant (grade 3). Both cases involved an initial US examination by the student followed by a subsequent independent MRI examination (Table 2).
The first case involved a patient with right (dominant side) medial elbow pain following a fall two years previously. The patient was keen to have information at the time of their appointment and so the student performed an immediate US examination of their medial epicondyle (Image 1a and 1b).
Image 1a and 1b - US Report of the first grade 3 discrepancy case showing medial side of elbow and common flexor origin
When the patient returned from their business trip, they had an independent MRI which showed that they had an ‘undersurface tear of the common flexor tendon (CFT) involving more than 50% of the tendon’. The student had misdiagnosed this as anisotrophy in a tendinopathic CFT. The student and supervisor then reviewed the initial US images, which MD then correctly suggested was a high-grade tear (without prior knowledge of the MRI report). On discussion with the student, both parties agreed that, as the injury was missed, this was a potentially significant error and therefore should be a grade 3 discrepancy. Fortunately, as many studies have shown, non- surgical treatment is the mainstay of managing CFT tears (Alrabaa et al, 2019) and the patient responded very well to conservative management.
The student admitted they felt pressured into the examination by the patient and was not pleased with the missed diagnosis but was confident in the imaging technique performed. Kim and Mansfield (2014) in a paper discussing radiologic error categorization produced a classification guide (Figure 3). Although not directly related to US, the categorization suggests the reasoning for the inaccurate diagnosis in this case, where the abnormality was identified but attributed to the wrong cause, was a ‘faulty reasoning’ classification of radiology error. This suggests the imaging was performed well but that additional training is required for the interpretation of more difficult injuries.
Figure 3 – Categorisation system depicting the classification of errors in diagnostic radiology Taken from Kim and Mansfield (2014)
Konarski et al (2022) suggest that, although MRI remains the gold standard for confirming medial epicondyle pathology, US can be a practical and effective alternative with a sensitivity and specificity of 95% and 92% respectively. Similarly, Park et al (2008) in a small-scale study, showed that US had a specificity, sensitivity and accuracy reported at 95.2%, 92% and 93.5% respectively, for the detection of medial epicondylitis. However, this was noted to be highly dependent on the operator's experience. This is demonstrated by Batcha et al (2017), that only reported figures of 64.25%, 85.19% and 72.72%, for the diagnosis and grading of common extensor tears, which suggests that the detection of tears might be more challenging than a simple tendinopathy diagnosis, especially for a less experienced operator.
Ohrndorf et al (2010), in a study comparing 3 ultrasonographers of different training levels, found that ‘ultrasound investigator[s] with 10 months experience reached good to almost perfect agreement with a 10-year experienced senior sonographer and that a little experienced sonographer substantially improved US competence within a period of 2 months’. This suggests that experience is a primary factor in achieving good agreement scores. However, this study had a rheumatology bias and was primarily focused on the detection of erosions and synovitis. In addition, the inexperienced sonographers were personally taught by the experienced practitioner in question and thus were more likely to interpret images in a similar manner.
The second grade 3 discrepancy case involved a patient who presented with a dull ache in their left mid/proximal anterior thigh after heavy lifting three weeks previously. Their GP diagnosed a quads strain, but the patient sought further clarification from the clinic as their pain was not easing. The student performed an immediate US examination of their thigh using the Siemens Accuson, with the lowest (and therefore deepest penetrating) linear probe, of 8 MHz. (Image 2a and 2b).
Image 2a and 2b - US Report of the second grade 3 discrepancy case showing the distal anterior thigh in transverse view (2a) and proximal anterior thigh in transverse view (2b)
As denoted by the images, the examination was limited by body habitus and the images obtained were poor quality proximally. Therefore, a diagnosis was unable to be based on the US images. The patient was therefore referred for an MRI by the student. The results of the MRI showed that the patient had a 2.7 cm tear at the proximal tendon of his Vastus Lateralis tendon proximally (Table 2).
On review of the US and MRI images, both the student and MD agreed that there were several shortcomings regarding the initial US appointment. Firstly, the initial physical examination completed by the student was influenced by the prior diagnoses of quads pain by a GP, as such the physical examination completed had an element of bias and inaccuracy. Consequently, the
assumption was made that the injury would be present in the patient’s rectus femoris muscle, where the patient indicated and where most quad strains are found to be (Cross et al, 2004), and thus the vastus lateralis component of the quads was not assessed. Graber et al (2005) described cognitive biases as one reason why errors occurred in radiology. One type of this bias is anchoring bias, in which an initial impression is formed (in this case of a rectus femoris strain) which then is not altered even considering new information, for example when no rectus femoris strain was seen on US. Lepainen et al (2022) suggested that, although injuries to the quads were typically seen in the rectus femoris, they can occur in the vastus lateralis and sartorius where they may present as a ‘blurring (of) fibres (and) interstitial and intermuscular oedema’, when viewed via US - a presentation which was not recognized by the student.
Chung et al (2004) also suggested that quads ruptures were more likely in tendons weakened by age, fatty infiltration in the obese, previous steroid therapy and diabetes mellitis (amongst other factors. Therefore, an incomplete history being taken by the student, complicated by the patient providing a poor history, was another failure of the overall examination. The patient’s prior steroid therapy and the diabetes mellitis were only highlighted when they attended with their partner, for the MRI. A more objective approach would have been to consider the new information and scrutinize other structures in the area as a result, such as by imaging the vastus lateralis. In future the student should eliminate pre-conceived bias and complete a full US of all potential structures of the relevant area as well as applying good clinical reasoning to support planning for further investigations and assessments.
Furthermore, for this instance it would have been more beneficial to use the Oesoate US machine which benefits from the greater penetrating depths of the lower frequency curvilinear probe.
This is important when considering patients with reduced muscle mass and larger fat deposits over the relevant area. For example, in their 2017 study, Serafin-Krol noted that with deep layers of muscle and thicker layers of fat, the lower frequency (6 Mhz) curvilinear probe was essential to obtain the best images. As the identified learning points for this examination were all under the student’s control, it should be possible to apply these practices going forward.
Despite an 82% grade 1 agreement score during this audit, as discussed above, it is imperative to ensure that shortcomings leading to grade 2 and 3 discrepancies are addressed and managed. As a result of the audit process the student identified the following ways that the standard of their US examinations can be maintained and improved:
The clinical audit performed showed a good agreement level between the student and the supervisor (82% grade 1). Throughout this process the student has been able to identify practical limitations and shortcomings that can be developed and fine-tuned moving forwards in their clinical career. This will result in improved patient experience and outcomes in addition to efficiencies in the local practice.
At the beginning of the study, in January 2023, emphasis was placed on obtaining the correct images of the structure clinically questioned and the technical challenges of probe settings that this might entail. However, throughout the audit process, it became clear that a critical element, whether in paper, or digital form, was the written report. Going forward, the adoption of a standard report, uniform across the sonography world, would help to facilitate access to information, which in turn would make audit, teaching and research significantly improved.
Adatia P. ‘What is clinical audit?’. BDJ Student 2022;29: 12-14
Alrabaa R, Sonnenfield J, Trofa D, Ahmad C. ‘Elbow Common Flexor Tendon Repair Technique.’
Arthroscopy Association of North America 2019;8(11): E1367-1371.
Batcha A, Rowicki K, Kiesel B, et al. ‘Ultrasonography versus MRI in detecting and grading
Common extensor tendon tear in chronic lateral epicondylitis’. PLoS One 2017;12: e0181828.
Chiavaras M, Jacobson J, Yablon C, Brigadoo M, Girish G. ‘Pitfalls in wrist and hand ultrasound’
American Journal of Roentgenology 2014;203: 531-540.
Cross T, Gibbs N, Houang M, Cameron M. ‘Acute quadriceps strains: MRI features and
Prognosis’. Am J Sports Med 2004;32(3): 709-10.
Chung K, Wong T, Yuen M, Kam C. ‘Sonography of quadriceps tendon ruptures’. Hong Kong Journal of Emergency Medicine 2004;11(1): 169-172
Clevert D, Nyhsen C, Ricci P, European Society of Radiology. ‘Position statement and best practice recommendations on the imaging use of ultrasound from the European Society Of Radiology ultrasound sub committee’. Insights Imaging 2020; 11: 115.
Dalili D, Carne A, MacKay J, O’Connor P, Silver D, Robinson P, Mansour R. ‘Musculoskeletal ultrasound imaging standards in the UK: British Society of Skeletal Radiologists (BSSR) position Statement’. Br J Radiol. 2021;94 (1121)
Edwards H, Smith J, Weston M. ‘What makes a good ultrasound report?’. Ultrasound 2014 Feb; 22(1): 57-60.
Graber M, Franklin N, Gordon R. ‘Diagnostic Error in internal medicine’. Arch Intern Med 2005;165: 1493-1499.
Gibbs V. ‘The Role of Ultrasound Simulators in education: an investigation into sonography
Student experiences and clinical mentor perceptions’. Ultrasound 2015;23(4): 204-11.
Jacobsen J, Lancaster S, Prasad A, van Holsbeeck M, Craig J, Kolowich P. ‘Full thickness and partial-thickness supraspinatus tears: value of ultrasound signs in diagnosis’. Radiology 2004; 230(1); 234-242.
Kim Y, Mansfield L. ‘Fool me twice: delayed diagnosis in radiology with emphasis on
Perpetuated errors’. Am Journal of Roentgenology 2014;202(3): 465-70.
Kornarski W, Pobozy T, Kotela A, Hordowicz M, Pobozy K. ‘Ultrasound in th differential Diagnosis of medial epicondylagia and medial elbow pain: Imaging findings and narrative Literature review’. Healthcare (Basel) 2022;10(8): 1529
Koshy K, Limb C, Gundogan B, Whitehurst K, Jafree D. ‘Reflective practice in healthcare and
how to reflect effectively’. Int J Surg Oncl. (NY) 2017 July;2(6): e20
Kumar K, Rawal J, Nakano N, Khanduja V. ‘Pathogenesis and contemporary diagnoses for lateral hip pain: a scoping view’. Knee Surgery, Sports Traumatology, Arthroscopy 2021; 29: 2408-2416.
Lee C, Nagy P, Weaver S, Newman-Toker D. ‘Cognitive and system factors contributing to
Diagnostic Errors in Radiology’. 2013;201: 611-617.
Lempainen L, Mecho S, Valle X, Mazzoni S, Villanon J, Freschi M, Stefanini L, Garcia-Romero- Perez A, Burova M, Pleshkov P, Pruna R, Pasta G, Kosola J. ‘The management of anterior Thigh injuries in soccer players: practical guide’. BMC Sports Sci Med Rehabil. 2022;14: 41.
Moore S, Reeve R. Audit in Ultrasound: An introduction and overview’. Sonography. 2022;9: 123-28.
Necas M. ‘The clinical ultrasound report: Guideline for sonographers’. Australian Journal of Ultrasound in Medicine 2017;21(1): 9-23.
Ohrndorf S, Naumann L, Grundey J, Scheel T, Scheel A, Werner C, Backhaus M. ‘Is Muscolo- Skeletal Ultrasonography an Operator-Dependent Method or a Fast and Reliable Teachable Diagnostic Tool? Inter reader Agreements of Three Ultrasonographers with Different
Training Levels’. International Journal of Rheumatology 2010; ID 164518.
Park G-Y, Lee S-M, Lee M. ‘Diagnostic value of ultrasonography for clinical medial Epicondylitis'. Arch Phys Med Rehabil 2008;89(4): 738-42.
Riley S, Groves C, Chandramohan M. ‘Musculoskeletal ultrasound: Audit of sonographer reporting'. Ultrasound 2010;18(1): 36-40.
Secretary of State for Health. ‘Our healthier nation: a contract for health’. London: Stationery Office; 1998.
Serafin-Krol M, Maliborski A. ‘Diagnostic errors in musculoskeletal ultrasound and how to
Avoid them’. J Ultrason 2017;17: 188-196.