Criteria abstracted from The Users' Guide to Medical Literature, from the Health Information Research Unit and Clinical Epidemiology and Biostatistics, McMaster University

Highlighted lines and questions below provide links to the pertinent description of criteria in The EBM User's Guide, now available at the Canadian Centres for Health Evidence

Specific questions in this review are based on Jaeschke RZ, Meade MO, Guyatt GH, Keenan SP, Cook DJ. How to use diagnostic test articles in the intensive care unit: diagnosing weanability using f/Vt. Crit Care Med. 1997;25(9):1514-2. [abstract]

Article Reviewed:

Please visit the new Evidence Based Journal Club Reviews

Sensitivity and Specificity of Bronchoalveolar Lavage and Protected Bronchial Brush in the Diagnosis of Pneumonia in Pediatric Burn Patients.

Barret JP, Ramzy PI, Wolf SE, Herndon DN.

Arch Surg. 1999;134:1243-1247. [abstract]

Reviewed by Amit Vohra MD, Baylor College of Medicine, Texas Children's Hospital, Houston, TX (currently: Dayton Children's Hospital)

Review posted July 6, 2004

I. What is being studied?

Study objective:

To compare the results of bronchoalveolar lavage (BAL) quantitative cultures and protected bronchial brush (PBB) cultures with autopsy findings in pediatric burn patients. The authors wished to determine the sensitivity and specificity of these techniques in diagnosing pneumonia in these patients.

The study design:

A retrospective case series. There was no blinding.

The patients investigated:

The authors studied all pediatric patients with burns who died at Shriners Burns Hospital, Galveston, TX over a period of 10 years. Antemortem BAL and PBB cultures were available for 43 patients (out of a possible 45). This was a severely burned population; TBSA burned was 70% ± 4%.

II. Are the results of the study valid?

Primary questions:

1. Was there an independent, blind comparison with a reference standard?

Yes. The gold standard (reference standard) chosen was a positive postmortem diagnosis of pneumonia (confirmed at autopsy with pathologic examination and by postmortem culture). The authors believe that choosing "autopsy finding of pneumonia" as a standard criterion for true-positive pneumonia strengthens this study compared to previous studies. However, I must question the validity of this assumption, as I am unsure of the clinical significance of incidental findings of pneumonia at autopsy.

In addition, the article does not mention whether the pathologist was blinded to the clinical diagnosis or if the chart reviewer was blinded to the autopsy diagnosis. Also, most importantly, the diagnostic tests (BAL/PBB) were separated in time from the application of the reference standard sometimes by as much as 72 hours. This, in my opinion, could have caused the non-correlation between the antemortem and the postmortem cultures. If there was appropriate interim treatment for the positive BAL and/or PBB culture, it is not surprising that there would be a discrepancy; it could mean the treatment was effective.

2. Did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice?

Perhaps. These are severely burned children cared for in a specialized burn center and as such, the sample may be skewed towards the most severely affected patients that does not represent the full spectrum of potential subjects. It meets the criteria that the test ought to be applied to a population at risk where the diagnosis is a challenge, i.e. burn patients with inhalation injury in whom a diagnosis of pneumonia needs to be made. Inhalation injury was present in 91% of the children, mean age 3.7 ± 0.5 years and 100% of the children were ventilated.

Secondary questions:

3. Did the results of the test being evaluated influence the decision to perform the reference standard?

No. The reference standard was applied to all patients (deaths). The diagnostic techniques were performed antemortem. The results of the antemortem cultures undoubtedly influenced antibiotic therapy and possibly influenced the patient outcome (death or survival). This would affect the population subjected to the reference standard (the spectrum of dead children). Furthermore, the antibiotic choices must have had an impact on the findings of the reference standard, because only 2 BAL cultures and only 4 PBB cultures were identical to the postmortem cultures.

4. Were the methods for performing the test described in sufficient detail to permit replication?

Yes, the technique of the BAL is well described. It would have been nice to know the actual bronchoscope sizes used for these techniques which could be of use to centers that take care of both adult and child burn victims.

III. What are the results?

1. Are the test's sensitivity, specificity, and likelihood ratios presented (or are the data necessary for their calculation provided)?

The authors do not present the likelihood ratios. However, the raw data is presented within the text to enable the appropriate calculations. Constructing the 2 x 2 tables shows:

Postmortem diagnosis of pneumonia

The clinical diagnosis of pneumonia	Disease present	Disease absent	Totals
Test positive	6 (a)	6 (b)	12 (a+b)
Test negative	13 (c)	18 (d)	31 (c+d)
	19 (a+c)	24 (b+d)	43 (a+b+c+d)

Sensitivity = a/(a+c) = 31%, Specificity = d/(b+d) = 75%
(Clinicians picked up only 31% of the "pneumonias". They also overcalled "pneumonia" in 25% (100-75) of the time.
Positive Predictive Value (PPV) = a/(a+b) = 50%, Negative Predictive Value (NPV) = d/(d+c) = 58%
(When clinicians said there was "pneumonia" they were right only half the time and when they said there was no pneumonia they were correct 58% of the time)

Likelihood ratio for a positive test result = LR+ = Sensitivity/(1 - specificity) = 1.27
(Clinicians were 1.27 times more likely to diagnose pneumonia in children who truly had pneumonia than they were to diagnose pneumonia in children who did not have pneumonia)
Likelihood ratio for a negative test result = LR- = Specificity/(1 - sensitivity) = 0.91
(Clinicians were 0.9 times less likely to diagnose no pneumonia in a patient with pneumonia than they were to make such a diagnosis in a patient with no pneumonia)

A Likelihood ratio for a given diagnostic test result gives the odds that that test result comes from a person who has the disease for which the test was ordered. When the likelihood ratio is multiplied by the "pre-test" odds that the patient has the disease, the product is the "post-test" odds that the person has the disease (PRE-TEST ODDS X LIKELIHOOD RATIO = POST-TEST ODDS). Often we need likelihood ratio LR+ >10 or LR- <0.1 to rely on the test to affect our management by changing the posttest odds of the disease quite significantly.

Now let us examine the 2 x 2 table of one of the diagnostic tests, the BAL:

Bronchoalveolar Lavage	Pneumonia present	Pneumonia absent	Totals
Test positive	10 (a)	17 (b)	27 (a+b)
Test negative	9 (c)	7 (d)	16 (c+d)
	19 (a+c)	24 (b+d)	43 (a+b+c+d)

Sensitivity = a/(a+c) = 53%, Specificity = d/(b+d) = 29%
Positive Predictive Value (PPV) = a/(a+b) = 37%, Negative Predictive Value (NPV) = d/(d+c) = 44%

Likelihood ratio for a positive test result = LR+ = Sensitivity/(1 - specificity) = 0.74
Likelihood ratio for a negative test result = LR- = Specificity/(1 - sensitivity) = 1.6

In addition, when looking at cases as true positives only if BAL cultures matched autopsy cultures, the sensitivity drops to a very low 7% (2/27).

Similarly the PBB had a sensitivity of 53% (10/19), specificity of 58% (14/24), PPV (10/20) of 50%, NPV 0.6, LR+ 1.26 and LR- 0.81. However, when only true positives are considered (i.e. when PBB cultures matched autopsy cultures) the sensitivity drops to a very low 20% (4/19), specificity decreases to 33% (8/24) and the PPV drops to 20% (4/20). The table is below.

PBB	Pneumonia present	Pneumonia absent	Totals
Test positive	10	10	20
Test negative	9	14	23
	19	24	43

IV. Will the results help me in caring for my patients?

1. Will the reproducibility of the test result and its interpretation be satifactory in my setting?

The techniques used in this burn center for the BAL and PBB are fairly skilled procedures especially in as young a population as the one studied (mean age 3.7 ± 0.5 years) with small endotracheal tubes. Many, but not all, surgeons in other specialized burn centers could and do perform these techniques well enough for this test to be reproducible. However, if the numbers are as bad as reported in the results section, it begs the question, "why should these tests be done at all?" There is no mention of the complications if any. In addition, the therapeutic decisions based on the interpretations of these cultures (colonization vs. infection) are likely to be variable too.

2. Are the results applicable to my patients?

Yes. However, my ICU is not a specialized burn center and we would be referring such patients. These results might be applicable to burn centers or in patients with >70% TBSA burns.

3. Will the results change my management?

Twelve of 43 children had a clinical diagnosis of pneumonia per CDC criteria. 19 children had autopsy-diagnosed "true" pneumonia of which only 6 had been suspected by clinicians antemortem. The question is: Did clinicians really miss two thirds of the pneumonias, or did the autopsy results cloud the distinction between clinically significant and clinically silent? Whatever the truth is, and I believe it is the autopsy results which are misleading, this article tries to convince me that BAL and PBB are not the answer to improve our diagnostic accuracy. Both have low sensitivity and specificity in the diagnosis of pneumonia in the pediatric burn patient. I intuitively agree with the author's conclusions but am not convinced from an evidence based medicine perspective. The data presented in the study suffers from inconsistency and the numbers are small. The time lag between the performance of the antemortem and autopsy cultures and the interim use of antibiotics directed against the microorganisms cultured by the BAL/PBB colors the results. The results of this study could have been strengthened by 1) complete/ consistent reporting of raw data with no errors in the calculations, 2) blinding, and 3) multicenter prospective design.

4. Will patients be better off as a result of the test? No. The techniques are no better than a random guess in the diagnosis of pneumonia in the burn patient. This is a vexing problem because clinicians purportedly missed two-thirds (69%; 12/19) of the pneumonia patients. Invasive techniques like lung biopsy have more iatrogenic complications and yet do not improve sensitivity as much as hoped (discussed in the article). Finally, even if the yield on the cultures was higher (i.e. a better test) there is no evidence that this would change patient outcomes in a positive manner.

References:

1. Jaeschke R, Guyatt GH, Sackett DL, and the Evidence Based Medicine Working Group. How to Use an Article About a Diagnostic Test. Based on the Users' Guides to Evidence-based Medicine (1994;271(5):389-391) and (1994;271(9):703-707). [full-text]

 

Comments

Back to the EB Journal Club Index