The need for early Kasai portoenterostomy: a Western Pediatric Surgery Research Consortium study.

PURPOSE: The purpose of this study was to investigate factors impacting transplant-free survival among infants with biliary atresia. METHODS: A multi-institutional, retrospective cohort study was performed at nine tertiary-level children's hospitals in the United States. Infants who underwent Kasai portoenterostomy (KP) from January 2009 to May 2017 were identified. Clinical characteristics included age at time of KP, steroid use, surgical approach, liver pathology, and surgeon experience. Likelihood of transplant-free survival (TFS) was evaluated using logistic regression, adjusting for patient and surgeon-level factors. Secondary outcomes at 1 year included readmission, cholangitis, reoperation, mortality, and biliary clearance. RESULTS: Overall, 223 infants underwent KP, and 91 (40.8%) survived with their native liver. Mean age at surgery was 63.9 days (± 24.7 days). At 1 year, 78.5% experienced readmission, 56.9% developed cholangitis, 3.8% had a surgical revision, and 5 died. Biliary clearance at 3 months was achieved in 76.6%. Controlling for patient and surgeon-level factors, each additional day of age toward operation was associated with a 2% decrease in likelihood of TFS (OR 0.98, 95% CI 0.97-0.99). CONCLUSION: Earlier surgical intervention by Kasai portoenterostomy at tertiary-level centers significantly increases likelihood for TFS. Policy-level interventions to facilitate early screening and surgical referral for infants with biliary atresia are warranted to improve outcomes.

Pediatric dog bites: a population-based profile.

BACKGROUND: Previous studies have identified risk factors for dog bites in children, but use data from individual trauma centers, with limited generalizability. This study identifies a population risk profile for pediatric dog bites using the National Trauma Data Bank. We hypothesized that the population at risk was younger boys, that such bites occur at home, are moderately severe, and are on the face or neck. METHODS: For this retrospective cross-sectional study, a sample of 7912 children 17 years old and younger with International Classification of Diseases (ICD)-9 event code E906.0, for dog bites, were identified. Datasets from 2007 to 2014 were used. Data included patient's gender, age, ICD-9 primary and location E-codes, AIS body region and AIS severity. RESULTS: Most children were 6-12 years old and female, but a similar number fell into the narrower range of 0-2 years old. Injuries in the younger group frequently occurred at home, on the face and head, and with minor severity. Age of the child predicts the location of incident (P<0.001), the severity of injury (P<0.001) and the body region of the injury (P<0.001). Body region of the injury predicted its severity (P<0.001). DISCUSSION: Younger children are more likely to receive dog bites, and bites incurred are likely of greater severity. Children this young cannot yet be taught how to properly interact with a dog. CONCLUSIONS: Dog bites are a significant source of morbidity for children. Based on the population risk factors profile generated, this study recommends targeting live dog education towards the parents of young children.

Formative research experiences in pediatric surgeons: a mixed methods study of Pediatric Trauma Society members.

PURPOSE: A career in pediatric surgery has historically required extensive research experience, but the optimal research training is not well defined. The purpose of this study was to explore the formative research experiences among pediatric surgeons. METHODS: A 1-h focus group was held with 14 pediatric surgeons at the 2017 Pediatric Trauma Society annual meeting. A 17-item survey was also administered. Questions were intended to elicit discussion of prior research experiences. A qualitative analysis of the dialogue was performed. RESULTS: Seventy-seven percent of respondents completed a research fellowship. Most (77%) currently conduct clinical research. Participants most frequently desired additional training in study design (50%), NIH funding (43%), and grant preparation (43%). Seven themes were identified from the focus group: (1) Early research exposure is rudimentary; (2) Resume-building was a motivation; (3) Mentorship is important; (4) Institutional resources are vital; (5) Independent learning is necessary; (6) Protected time is limited; and (7) Basic science research is not always practical. CONCLUSIONS: Many pediatric surgeons feel that their research training can be improved upon. Formal mentorship, dedicated research time, and institutional resources were perceived to be important factors. Education in research study design, grant writing, and NIH funding may be beneficial. LEVEL OF EVIDENCE: V, expert opinion.

Management and outcomes of peripancreatic fluid collections and pseudocysts following non-operative management of pancreatic injuries in children.

BACKGROUND: Peripancreatic fluid collection and pseudocyst development is a common sequela following non-operative management (NOM) of pancreatic injuries in children. Our purpose was to review management strategies and assess outcomes. METHODS: A multicenter, retrospective review was conducted of children treated with NOM following blunt pancreatic injury at 22 pediatric trauma centers between the years 2010 and 2015. Organized fluid collections were called "acute peripancreatic fluid collection" (APFC) if identified < 4 weeks and "pseudocyst" if > 4 weeks following injury. Data analysis included descriptive statistics Wilcoxon rank-sum, Kruskal-Wallis and t tests. RESULTS: One hundred patients with blunt pancreatic injury were identified. Median age was 8.5 years (range 1-16). Forty-two percent of patients (42/100) developed organized fluid collections: APFC 64% (27/42) and pseudocysts 36% (15/42). Median time to identification was 12 days (range 7-42). Most collections (64%, 27/42) were observed and 36% (15/42) underwent drainage: 67% (10/15) percutaneous drain, 7% (1/15) needle aspiration, and 27% (4/15) endoscopic transpapillary stent. A definitive procedure (cystogastrostomy/pancreatectomy) was required in 26% (11/42). Patients with larger collections (≥ 7.1 cm) had longer time to resolution. Comparison of outcomes in patients with observation vs drainage revealed no significant differences in TPN use (79% vs 75%, p = 1.00), hospital length of stay (15 vs 25 median days, p = 0.11), time to tolerate regular diet (12 vs 11 median days, p = 0.47), or need for definitive procedure (failure rate 30% vs 20%, p = 0.75). CONCLUSIONS: Following NOM of blunt pancreatic injuries in children, organized fluid collections commonly develop. If discovered early, most can be observed successfully, and drainage does not appear to improve clinical outcomes. Larger size predicts prolonged recovery. LEVEL OF EVIDENCE: III STUDY TYPE: Case series.

Improving Disaster Preparedness of Families With a Parenteral Nutrition-dependent Child.

OBJECTIVES: Disruption in the care of special healthcare needs children may lead to life-threatening situations or preventable secondary conditions. California averages more than 100 earthquakes per week. Subsequent power outages, damage to utility systems, and road damage after an earthquake can have grave consequences for families with parenteral nutrition (PN)-dependent children. The purpose of the study was to demonstrate that we could improve disaster preparedness of families with PN-dependent children utilizing individualized family education and distribution of personalized disaster survival toolkits. METHODS: We administered a baseline survey to assess disaster preparedness of our families with PN-dependent children followed by individualized disaster survival toolkit distribution and education. We followed up with these families with phone call surveys at 2 and 4 months. A generalized estimating equation with both logistic and linear regression was used to analyze data over the follow-up period. RESULTS: We found statistically significant improvements in developing a family emergency plan (P < 0.0001), having a basic emergency supply kit (P < 0.0001), having a completed emergency information form from the child's provider (P < 0.0001), and the confidence level or readiness for a disaster (P < 0.0001). All participants had extra batteries for PN infusion pumps. Having alternative power sources, such as a generator, did not significantly change over time. CONCLUSIONS: Individualized disaster education helped families with PN-dependent children not only to prepare for a possible earthquake, but also to feel more confident in their ability to handle a natural disaster.

Impact of simulation-based training on perceived provider confidence in acute multidisciplinary pediatric trauma resuscitation.

PURPOSE: Simulation-based training has the potential to improve team-based care. We hypothesized that implementation of an in situ multidisciplinary simulation-based training program would improve provider confidence in team-based management of severely injured pediatric trauma patients. METHODS: An in situ multidisciplinary pediatric trauma simulation-based training program with structured debriefing was implemented at a free-standing children's hospital. Trauma providers were anonymously surveyed 1 month before (pre-), 1 month after (post-), and 2 years after implementation. RESULTS: Survey response rate was 49% (n = 93/190) pre-simulation, 22% (n = 42/190) post-simulation, and 79% (n = 150/190) at 2-year follow-up. These providers reported more anxiety (p = 0.01) and less confidence (p = 0.02) 1-month post-simulation. At 2-year follow-up, trained providers reported less anxiety (p = 0.02) and greater confidence (p = 0.01), compared to untrained providers. CONCLUSIONS: Implementation of an in situ multidisciplinary pediatric trauma simulation-based training program may initially lead to increased anxiety, but long-term exposure may lead to greater confidence. LEVEL OF EVIDENCE: II, Prospective cohort.

Evaluation of Highest Level Pediatric Trauma Activation Criteria.

BACKGROUND: Despite the presence of a tiered in-hospital trauma triage system for the past decade, trauma centers still struggle with a definitive list of highest level activation criteria. In 2002, the American College of Surgeons (ACS) mandated 6 criteria for highest level activation. However, it is unknown if pediatric trauma centers follow these criteria. The purpose of this study is to identify and categorize the highest level pediatric trauma criteria used by pediatric trauma centers in the United States. METHODS: In collaboration with the ACS, we reviewed activation criteria for highest level trauma activation for all ACS-verified level I pediatric trauma centers in the United States. Criteria were sorted by 2 reviewers into categories of indicators used for activation: patient demographic, physiologic, anatomic, intervention/resource usage, mechanism, and other. RESULTS: A total of 51 unique criteria for highest level trauma activation were identified from 54 (96%) of 56 level I pediatric trauma centers. Each center used between 1 and 29 criteria. A total of 42.6% of pediatric trauma centers followed all 6 criteria recommended by ACS. The most commonly omitted criterion was emergency physician discretion. The most common criteria not included in the ACS recommendations, but included in the highest level activation criteria, were amputation proximal to wrist or ankle (63%), and spinal cord injury/paralysis (63%). CONCLUSIONS: There is wide variation in the criteria used for highest level trauma activation among pediatric trauma centers. Further research investigating individual or grouped criteria to determine the most sensitive and specific criteria are necessary for appropriate triage and resource usage.

The Role of Adult Perceptions and Supervision Behavior in Preventing Child Injury.

Supervision is an important factor in reducing injury risk. There are multiple factors that can affect the appropriate level of supervision including risk perception, anticipation of injury, and distracted behaviors. This study examined the perceived risks of child injury among parents and child caregivers and their supervision behavior among adults in an urban playground. Participant data from 25 individuals were collected through observations and anonymous self-reported surveys. More than half of the participants indicated practice of appropriate supervisory behavior, including attentiveness to their child's behavior and proximity to their child during play. Caregivers were more likely to report more careful levels of supervision. One-fourth of participants reported a change in the supervisory behavior during periods of distraction, specifically with phone use. Of the variables tested, there was a significant association between the variable 'talking to other adults' during supervision and 'prior injury' (P value = 0.04, 95% CI 0.03-0.91). Parents were more likely to report that they would leave their child unattended if they believed that the playground was a safe environment for play. There was a difference between self-reported behaviors and actual observed behavior, which is likely due to varying perspectives regarding child safety and injury prevention. In regards to injury risk, findings highlight the important role of appropriate supervisory behaviors and risk perceptions in preventing child injuries.

On Shaky Ground: Learner Response and Confidence After Tabletop Earthquake Simulation.

OBJECTIVE: The aim of this study was to evaluate residents' confidence and attitudes related to management of earthquake victims during a tabletop simulation and 6 months after the intervention. METHODS: Pediatric residents from 4 training programs were recruited via e-mail. The tabletop simulation involved 3 pediatric patients (crush injury, head injury, and a nonverbal patient with minor injuries). A facilitated debriefing took place after the simulation. The same simulation was repeated 6 months later. A survey was administered before the simulation, immediately after, and after the 6-month repeat simulation to determine participants' self-rated confidence and willingness to respond in the event of a disaster. A 5-point Likert scale that ranged through novice, advanced beginner, competent, proficient, and expert was used. RESULTS: Ninety-nine participants completed the survey before the initial simulation session. Fifty-one residents completed the immediate postsurvey, and 75 completed the 6-month postsurvey. There was a statistically significant improvement in self-rated confidence identifying and managing victims of earthquake disasters after participating in the simulation, with 3% rating themselves as competent on the presurvey and 33% rating themselves as competent on the postsurvey (P < 0.05). There was a nonstatistically significant improvement in confidence treating suspected traumatic head injury as well as willingness to deploy to both domestic and international disasters. CONCLUSIONS: Tabletop simulation can improve resident comfort level with rare events, such as caring for children in the aftermath of an earthquake. Tabletop can also be easily integrated into resident curriculum and may be an effective way to provide disaster medical response training for trainees.

A School-Hospital Partnership Increases Knowledge of Pedestrian and Motor Vehicle Safety.

Pedestrian and motor vehicle-related injuries are leading causes of morbidity and mortality in children. Trauma centers have specialized resources to conduct interventions that improve the safety of whole communities. In the present study, we evaluated the effectiveness of a school-hospital partnership in increasing knowledge of pedestrian and motor vehicle safety among students and parents in a large, urban community. Staff from a Level I pediatric trauma center conducted educational interventions in an urban public school district. Elementary school students participated in a pedestrian safety program, middle school students completed a community safety program, and high school students learned about the dangers of drunk and distracted driving. Students completed pre- and post-tests. Parents in the neighboring community received child passenger safety education at two child restraint (CR) inspection events. A total of 2203 students participated at a total of nine schools. Post-test scores were significantly higher than pre-test scores for students in all three age groups and within each grade level. At CR inspection events, 67 CRs were inspected, 49 (73 %) of which were replaced with new age- and weight- appropriate CRs. The most common instance of improper CR use was loose CR fit in vehicle seat (33 %). All 120 observed instances of misuse were corrected by a certified Child Passenger Safety Technician. Educational interventions effectively increased knowledge of pedestrian and motor vehicle safety among students and parents. We have demonstrated the utility of a school-hospital partnership for furthering knowledge of safety in an urban community.

Is routine preoperative screening echocardiogram indicated in all children with congenital duodenal obstruction?

INTRODUCTION: Congenital duodenal obstruction (DO) is frequently associated with congenital heart disease (CHD). Operative repair of DO is often postponed until an echocardiogram is completed, which may result in unnecessary delays. We aimed to identify and characterize CHD in children with DO to determine if appropriately selected patients could forego preoperative echocardiogram. METHODS: A two-center retrospective review of all infants with DO undergoing operative repair with completed echocardiograms was included (2003-2011). Demographics, co-morbid conditions, clinical exam findings, radiologic imaging, and need for cardiac surgery were recorded. RESULTS: 67 children were identified. 47 (70.1%) had CHD on echocardiogram of which 19 (40.5%) had significant CHD. Children without clinical findings, abnormalities on physical examination, and/or abnormal chest x-ray were unlikely to have CHD; i.e., no asymptomatic child had significant CHD. Sensitivity and specificity of clinical findings, physical exam, and/or chest x-ray for significant CHD were 100% (95% CI 0.79-1.0) and 37.5% (95% CI 0.24-0.53), respectively, for major CHD and 87.2% (0.74-0.95) and 60% (0.36-0.80) for any CHD. CONCLUSION: Careful clinical assessment, evaluation with pulse oximetry, and chest x-ray may be sufficient to exclude significant CHD in children with DO. Identifying children at low risk for cardiac lesions may prevent unnecessary delays to operative intervention and may limit medical expenses.

Using mixed methods to assess pediatric disaster preparedness in the hospital setting.

INTRODUCTION: Children are particularly vulnerable during disasters and mass-casualty incidents. Coordinated multi-hospital training exercises may help health care facilities prepare for pediatric disaster victims. PROBLEM: The purpose of this study was to use mixed methods to assess the disaster response of three hospitals, focusing on pediatric disaster victims. METHODS: A full-functional disaster exercise involving a simulated 7.8-magnitude earthquake was conducted at three Los Angeles (California USA) hospitals, one of which is a freestanding designated Level I Pediatric Trauma Center. Exercise participants provided quantitative and qualitative feedback regarding their perceptions of pediatric disaster response during the exercise in the form of surveys and interviews. Additionally, trained observers provided qualitative feedback and recommendations regarding aspects of emergency response during the exercise, including communication, equipment and supplies, pediatric safety, security, and training. RESULTS: According to quantitative participant feedback, the disaster exercise enhanced respondents' perceived preparedness to care for the pediatric population during a mass-casualty event. Further, qualitative feedback from exercise participants and observers revealed opportunities to improve multiple aspects of emergency response, such as communication, equipment availability, and physician participation. Additionally, participants and observers reported opportunities to improve safety and security of children, understanding of staff roles and responsibilities, and implementation of disaster triage exercises. CONCLUSION: Consistent with previous investigations of pediatric disaster preparedness, evaluation of the exercise revealed several opportunities for all hospitals to improve their ability to respond to the needs of pediatric victims. Quantitative and qualitative feedback from both participants and observers was useful for comprehensively assessing the exercise's successes and obstacles. The present study has identified several opportunities to improve the current state of all hospitals' pediatric disaster preparedness, through increased training on pediatric disaster triage methods and additional training on the safety and security of children. Regular assessment and evaluation of supplies, equipment, leadership assignments, and inter-hospital communication is also suggested to optimize the effectiveness and efficiency of response to pediatric victims in a disaster.

A mixed-methods pilot study of disaster preparedness and resiliency among faith-based organizations.

INTRODUCTION: Faith-based organizations represent a source of stability and are an established presence in a community. They frequently serve their community following disasters. They are not formally included or identified as a disaster resource; thus, there is an opportunity to increase the effectiveness with which faith-based organizations prepare for and respond to disasters. Problem This pilot study aimed to assess perceptions of the level of disaster preparedness and resiliency among faith-based organizations as a first step in understanding how to improve disaster preparedness and resiliency among these organizations and their communities. METHODS: Survey and semi-structured interviews were conducted with six faith-based organizations, one with a leader and one with a staff member. Frequency distributions of survey questions were obtained. Interviews were transcribed and thematic analysis was supported by analytical software, ATLAS. ti. RESULTS: Results of the survey indicated strong social networks among congregation and community members. However, half of the members indicated that they did not socialize often with other races and other neighborhoods. Additionally, trust of other groups of people was generally low. Themes that emerged from qualitative analysis were: (1) perceived disaster preparedness and resiliency; (2) barriers to community preparedness and resiliency; (3) lessons learned from past disasters; (4) social services and networks; and (5) willingness to be prepared. CONCLUSIONS: The results suggest that there is a need for interventions to improve disaster preparedness and resiliency among faith-based organizations.

Public safety messaging during super bowl LVI: A pilot study and modified framework.

OBJECTIVE: The Super Bowl is one of the most widely attended sporting events and requires proper communication to ensure emergency preparedness for everyone in the city. This pilot study used Super Bowl LVI as a setting to inform future research evaluating the effectiveness of the public health messaging during mass-gathering events. METHODS: This pilot study modifies past theoretical frameworks and research tools to create a novel survey instrument focused on public safety message effectiveness. This survey was sent to all of those that opted-in to the Joint Information Center's notification platform during Super Bowl LVI. RESULTS: The results show that message comprehension, source credibility, and perceived risk might not be associated with proactive behavior for public safety messaging. However, modality preference results showed that individuals might prefer to receive public safety and emergency alerts by text message. CONCLUSIONS: Factors that influence proactive response towards public safety messaging might differ from emergency alerts. This pilot study has yielded results from a major mass-gathering event that can be used to understand errors that arise during public health and emergency preparedness and strengthen future disaster planning and research.

Super Bowl LVI: Coordinating the Joint Information Center.

OBJECTIVE: The Super Bowl is one of the biggest annual sporting events that creates unique challenges for public health preparedness. The objective of this brief is to discuss the successes and challenges of the Joint Information Center (JIC) during Super Bowl LVI. METHODS: This study analyzed the JIC within 4 realms: pre-activation planning, public safety and alert messaging, communication, and credentialing and access. RESULTS: Pre-activation planning was not attended by all agencies, which made it difficult for effective communication during the activation period. Innovative platforms and social media themes were successfully deployed to enhance inter-operational communications and public safety messaging. Lastly, delayed credentialing and access to the Super Bowl Stadium decreased the ability to have context to safety preparedness. CONCLUSIONS: This study recommends that many of the JIC's successes should be translated to future mass-gathering events. Additionally, we urge that all agencies participate during pre-activation and activation periods to enhance safety operations and communications moving forward.

Assessing Trust in Physician and Vaccine Hesitancy Among Hispanic/Latinx Parents.

Introduction: Representing the USA's largest ethnic/racial group, Hispanic/Latinx (HL) experience health challenges of proportional magnitude. This study investigates the prevalence of vaccine hesitancy among HL adults and trust in their child's pediatrician. Methods: HL parents of children who receive medical care at one of the largest Federally Qualified Health Centers in the United States completed a survey examining associations between physician trust, vaccine hesitancy, and demographics. Data were subjected to ANOVA via SAS9 version 9.0 (Cary, NC) and SPSS version 27 (Chicago, IL) software. Results: With a total of 500 surveys completed (51% response rate; 81% completion rate), the prevalence of vaccine hesitancy amounted to 15.4% (n = 77). Parents with university-level education displayed higher levels of trust and perceptions in favor of routine vaccination practices (p < .01). When medical visits were conducted in Spanish, parents exhibited lower levels of trust and were more vaccine-hesitant (p < .01). Conclusions: Vaccine literacy must be prioritized in early education to reach parents who may not achieve college degrees. Ensuring language concordance within patient-physician dyads may maximize the potential for vaccine uptake and physician trust.

Addressing Health-Care Disparities in Pediatric Disaster Planning: A Qualitative Study.

OBJECTIVE: The Western Regional Alliance for Pediatric Emergency Medicine (WRAP-EM) is a multi-state, Administration for Strategic Preparedness and Response (ASPR) funded pediatric disaster center of excellence. WRAP-EM set out to determine the impact of health disparities on its 11 core areas. METHODS: We conducted 11 focus groups during April 2021. Discussions were led by an experienced facilitator, and participants could also include their thoughts on a Padlet throughout the discussion. Data were analyzed to determine overarching themes. RESULTS: Responses focused on health literacy, health disparities, resource opportunities, addressing obstacles, and resilience building. Health literacy data highlighted the need for development of readiness and preparedness plans, community engagement in cultural and language appropriate means, and increasing diversity in training. Obstacles faced included funding; inequitable distribution of research, resources, and supplies; lack of prioritization of pediatric needs; and fear of retribution from the system. Multiple already existing resources and programs were referenced highlighting the importance of best practice sharing and networking. A stronger commitment to mental health-care delivery, empowerment of individuals and communities, use of telemedicine, and ongoing cultural and diverse education were recurring themes. CONCLUSIONS: Results of the focus groups can be used to prioritize efforts to address and improve health disparities in pediatric disaster preparedness.

A qualitative analysis of public health officials' experience in California during COVID-19: priorities and recommendations.

Objectives: The aim of this study was to collect qualitative data regarding the violence faced by public health officials during the COVID-19 pandemic and create a guideline of recommendations to protect this population moving forward. Methods: Two focus groups were conducted virtually from April 2022 to May 2022. All nine participants were public health officials from across California. A grounded theory approach was used to analyze the data from these focus groups. Results: The main recurrent experiences among public health officials were harassment, psychological impact, systemic backlash, and burnout. Several recommendations for supporting public health officials were highlighted, including security and protection, mental health support, public awareness, and political/institutional support. Conclusion: Our study captures the violent experiences that health officials have faced during the COVID-19 pandemic. To maintain the integrity of the public health system, timely changes must be made to support and protect health officials. Our guideline of recommendations provides a multi-faceted approach to the urgent threats that officials continue to face. By implementing these solutions, we can strengthen our public health system and improve our response to future national emergencies.

The Challenge of Mass Casualty Incident Response Simulation Exercise Design and Creation: A Modified Delphi Study.

BACKGROUND: A Mass Casualty Incident response (MCI) full scale exercise (FSEx) assures MCI first responder (FR) competencies. Simulation and serious gaming platforms (Simulation) have been considered to achieve and maintain FR competencies. The translational science (TS) T0 question was asked: how can FRs achieve similar MCI competencies as a FSEx through the use of MCI simulation exercises? METHODS: T1 stage (Scoping Review): PRISMA-ScR was conducted to develop statements for the T2 stage modified Delphi (mD) study. 1320 reference titles and abstracts were reviewed with 215 full articles progressing for full review leading to 97 undergoing data extraction.T2 stage (mD study): Selected experts were presented with 27 statements derived from T1 data with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. Consensus amongst experts was defined as a standard deviation ≤ 1.0. RESULTS: After 3 mD rounds, 19 statements attained consensus and 8 did not attain consensus. CONCLUSIONS: MCI simulation exercises can be developed to achieve similar competencies as FSEx by incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD study (T2), and continuing to T3 implementation, and then T4 evaluation stages.