Criteria abstracted from The Users' Guide to Medical Literature, from the Health Information Research Unit and Clinical Epidemiology and Biostatistics, McMaster University

Highlighted lines and questions below provide links to the pertinent description of criteria in The EBM User's Guide, now available at the Canadian Centres for Health Evidence

Article Reviewed:

Please visit the new Evidence Based Journal Club Reviews

A new prognostic scoring system for meningococcal septic shock in children. Comparison with three other scoring systems.

Castellanos-Ortega I, Delgado-Rodríguez M, Llorca J, et al.

Intensive Care Med 2002; 28: 341-351. [abstract]

Reviewed by Amit Vohra MD, Wright State University School of Medicine, Children's Medical Center, Dayton OH

Review posted August 24, 2004

I. What is being studied?

Study objective:

To develop a new quick and sensitive scoring method for identification of children with presumed meningococcal septic shock (PMSS) at risk of death at admission to the pediatric intensive care unit. In other words, the investigators wanted to develop a mortality prediction tool for PMSS and compare its performance with three other prognostic systems: 1) a generic mortality prediction tool, the Pediatric Index of Mortality (PIM), and 2) and 3), validated specific scores, the Glasgow Meningococcal Septicemia Prognostic Score (GMSPS) and Malley score, respectively.

Study design

Multi-center retrospective cohort

II. Are the results of the study valid?

Note: These questions follow from Randolph AG et al. Understanding articles describing clinical prediction tools. Crit Care Med 1998;26:1603-1612. [abstract]

1. Was a representative group of patients completely followed up? Was follow-up sufficiently long and complete?

Yes. The study involved all children aged between 1 month and 14 years with a confirmed or presumed diagnosis of meningococcal septic shock admitted to 14 PICU's of tertiary level hospitals in Spain. The sample of patients used to derive the scoring tool was drawn from 4 PICU's and the validation sample was drawn from 10 different PICU's. All medical records were reviewed and inclusion and exclusion criteria were clearly defined. Septic shock was defined using the ACCP/SCCM Consensus Conference (1992) definition. The search for patients seemed comprehensive and provided a representative group of patients. Although referral bias arising from the inclusion of only tertiary level hospitals is a concern, cases of meningococcal septic shock are most likely transferred to PICU's rather quickly in developed countries and so the spectrum of patients would be similar to such patients.

Follow up was complete in both the development and validation groups for all patients retrospectively included in the study. The characteristics of eligible but excluded patients were described. In the development sample, 37 of 229 eligible children (16%) were excluded. Though the mortality of the excluded sample was high (26 non-survivors; 70%) and might raise concerns about validity, fifteen of these 26 children died within 2 hrs of PICU admission. This finding does not weaken the study since it is not clinically beneficial to use a model to predict death in cases with imminent mortality. In the validation sample, 18 patients (11%) were excluded, of which four died within 2 hours of admission.

2. Were all potential predictors included?

Almost. All variables necessary for calculating the three comparison scoring systems (the PIM, the GMPSS and Malley score) were evaluated in addition to the variables for the new score. The worst value of each variable during the first 2 hours in the PICU was selected for the analysis. 30 prognostic variables were tested: (demographic and clinical) age, sex, interval between the appearance of petechiae and admission to ICU, cyanosis, cold skin, meningeal signs, ecchymosis, temperature, heart rate, respiratory rate, blood pressure, oliguria, modified Glasgow coma scale; (laboratory data) WBC count, Platelet count, pH, pCO2, PaO2/FiO2, bicarbonate, base deficit, potassium, calcium, glucose, BUN, creatinine, PTT and fibrinogen. Also included were 2 therapeutic variables: use of mechanical ventilation and refractory hypotension. The data collectors were not blinded to outcome. Most of these criteria were sufficiently objective to eliminate bias of the collectors in recording the worst value except I have doubts about the objectiveness and accuracy of the following subjective variables - cyanosis, cold skin and refractory hypotension - which were not objectively defined. Did they omit any important prognostic variables? The use of steroids as a predictor was not evaluated.

3. Did the investigators test the independent contribution of each predictor variable?

Yes. After subjecting all 30 prognostic variables to univariate analysis, 25 variables were still associated with mortality (p > 0.15) and these were included in the subsequent multivariate analysis. The logistic regression identified seven independent predictors of death at admission: cyanosis, coma (GCS < 8), refractory hypotension, oliguria, WBC < 4000/mm3, PTT > 150% of control and base excess > -10mmol/l.

4. Were outcome variables clearly and objectively defined?

Yes. The primary outcome measure was hospital mortality, defined as death occurring before hospital discharge. Overall mortality was 31.5% within the development group and 29% within the validation group.

The prediction rule makes clinical sense because 6 of the 7 predictive variables are routinely measured and are related to the severity of shock and coma.

III. What are the results?

1. What is(are) the prediction tool(s)?

Death was reliably predicted by the new scoring system in the validation group. Each of the variables was assigned a score of 1 to 2 after the multivariate analysis. The maximum score was 10 points. The points were assigned based upon the coefficients from the logistic regression model.

The new scoring system defined three risk groups of mortality (categorical approach):

Score	Risk group	Mortality rates	95% Confidence Interval
≤ 3	low	2.7% (2/74)	0.33% to 9.4%
4-5	intermediate	26.3% (10/38)	13.4% to 43.1%
≥ 6	high	73.9% (34/46)	58.9% to 85.7%

2. How well does the model categorize patients into different levels of risk?

A higher score on the model predicted a higher probability of death. This can be stated as "the model showed high discrimination". The discrimination of the system was reported by the "area under the ROC curve" (AUC) since the outcome (death or survival) was a dichotomous variable. For the development and the validation samples the AUC were 0.91 ± 0.02 and 0.88 ± 0.03 respectively. The other three scores yielded lower ROC areas and this was statistically significant for the differences between the new score and the Malley and GMSPS ROC curves.

The trustworthiness of the risk estimates generated by the model can be known by the "calibration" of the system. The score showed good calibration (Hosmer-Lemeshow goodness of fit test: development (p=0.55) and validation (p= 0.47)) in that it correctly classified an important number of patients in both low and high risk groups.

3. How confident are you in the estimates of the risk?

The 95% confidence intervals for the estimates of risk of death are not stated in the article. However, by converting the % mortality estimate in the three groups into proportions and using the raw numbers given in the article it is possible to calculate the 95% CI. Of note, the confidence intervals in the three risk groups do not overlap.

IV. Will the results help me in caring for my patients?

1. Does the tool maintain its prediction power in a new sample of patients?

Yes. When tested in an independent validation sample the test maintained its prediction power: AUC=0.88. The validation sample for the prediction tool was drawn from 10 different PICU's. However, the new ICUs were still within the same geographic locale (Spain) and hence share important population characteristics. The prediction tool may work differently in a different population, e.g., USA, and so it is important to validate a tool in several different clinical settings especially if the a priori probabilities of death are different. Ideally, a validation of the cutoff values of the predictor variables (the prediction model) should be a) prospective and b) in a new population and c) with a different prevalence and spectrum of the underlying disease. The current study does not meet the first criterion as it was retrospective but does meet the second criterion fully. It meets the third criterion partially as the validation sample had an overall different case mix and study period than the development sample. Specifically, there were significantly more cases of serogroup C, female sex and a shorter time to admission to the PICU from the time of appearance of petechiae.

2. Are your patients similar to those patients used in deriving and validating the tool(s)?

Patients here in the United States are similar in that a clinical picture of purpura and septic shock is not an uncommon event for an intensivist to encounter. However what is different is that meningococcus is not the leading cause of sepsis in children. The fluid management in the first 2 hours is likely to be more aggressive and overall basic management may be different. The older adolescent and the college age populations have shown an increased incidence of the disease in the USA and the current study did not evaluate such patients. So it should be used with caution in patients > 14 years. The overall serogroup prevalence is similar in that types B and C together account for most of the cases of meningococcal sepsis in USA.

3. Does the tool improve your clinical decisions?

The new score differs from a generic prediction tool like PRISM and PRISM II in that it is derived from a more homogenous group of patients ("customized probability model") and is designed to identify risk much earlier in the PICU course. The authors state that they wanted the new score to be valuable in early therapeutic decision making or to facilitate risk-based stratification in clinical trials, a laudable goal. At a local level, the prediction tool can be useful to come up with a guide for standardized management of presumed meningococcal septic shock. For example, a group of intensivists with differing thresholds for initiating CVVH in patients with meningococcal shock might decide that anybody with a score ≥ 4 (predicted mortality ≥ 26% [95% CI range 12% to 86%] would be a candidate for early CVVH.

4. Are the results useful for reassuring or counseling patients?

Neither the new scoring system nor the comparison models were able to predict a risk group with 100% mortality. Such models should be used cautiously if at all when used to drive decisions about individual patients. However, information from the model about the child's average risk of dying, may be useful to surrogate decision makers like parents and to treating clinicians in making informed decisions about the risks/benefits of aggressive/expensive therapies like ECMO and in enrolling patients in clinical trials such as the pediatric activated protein C sepsis trial.

References:

1. Guyatt GH, Walter S, Shannon H, Cook D, Jaeschke R, Heddle H. Basic statistics for clinicians: 4. correlation and regression. Can Med Assoc J 1995; 152:497-504.
2. Exact Binomial and Poisson Confidence Intervals: http://members.aol.com/johnp71/confint.html

 

Comments

Back to the EB Journal Club Index