Reimagining the machine learning life cycle to improve educational outcomes of students.

Machine learning (ML) techniques are increasingly prevalent in education, from their use in predicting student dropout to assisting in university admissions and facilitating the rise of massive open online courses (MOOCs). Given the rapid growth of these novel uses, there is a pressing need to investigate how ML techniques support long-standing education principles and goals. In this work, we shed light on this complex landscape drawing on qualitative insights from interviews with education experts. These interviews comprise in-depth evaluations of ML for education (ML4Ed) papers published in preeminent applied ML conferences over the past decade. Our central research goal is to critically examine how the stated or implied education and societal objectives of these papers are aligned with the ML problems they tackle. That is, to what extent does the technical problem formulation, objectives, approach, and interpretation of results align with the education problem at hand? We find that a cross-disciplinary gap exists and is particularly salient in two parts of the ML life cycle: the formulation of an ML problem from education goals and the translation of predictions to interventions. We use these insights to propose an extended ML life cycle, which may also apply to the use of ML in other domains. Our work joins a growing number of meta-analytical studies across education and ML research as well as critical analyses of the societal impact of ML. Specifically, it fills a gap between the prevailing technical understanding of machine learning and the perspective of education researchers working with students and in policy.

A problem formulation framework for the application of in silico toxicology methods in chemical risk assessment.

The first step in the hazard or risk assessment of chemicals should be to formulate the problem through a systematic and iterative process aimed at identifying and defining factors critical to the assessment. However, no general agreement exists on what components an in silico toxicology problem formulation (PF) should include. The present work aims to develop a PF framework relevant to the application of in silico models for chemical toxicity prediction. We modified and applied a PF framework from the general risk assessment literature to peer reviewed papers describing PFs associated with in silico toxicology models. Important gaps between the general risk assessment literature and the analyzed PF literature associated with in silico toxicology methods were identified. While the former emphasizes the need for PFs to address higher-level conceptual questions, the latter does not. There is also little consistency in the latter regarding the PF components addressed, reinforcing the need for a PF framework that enable users of in silico toxicology models to answer the central conceptual questions aimed at defining components critical to the model application. Using the developed framework, we highlight potential areas of uncertainty manifestation in in silico toxicology PF in instances where particular components are missing or implicitly described. The framework represents the next step in standardizing in silico toxicology PF component. The framework can also be used to improve the understanding of how uncertainty is apparent in an in silico toxicology PF, thus facilitating ways to address uncertainty.

Probabilistic methods for non-cancer health effects.

Noncancer risk assessment methods and harmonization with cancer assessment methods have advanced from the simple divide a No Observed Adverse Effect Level (NOAEL) by a default safety factor or a linear extrapolation to background of the early 1980's. This advance is due in part due to groups such as the American Industrial Health Council, the National Institute of Environmental Health Sciences, the Society for Risk Analysis, the Society of Toxicology, and the U.S. Environmental Protection Agency, the National Academy of Sciences (NAS), the International Programme on Chemical Safety, and to many independent researchers outside of and within a workshop series sponsored by the Alliance for Risk Assessment prompted by the NAS \. Several of the case studies from this workshop series, and earlier work such as Bogdanffy et al., demonstrate that the dose response assessment of non-cancer toxicity and the harmonization of cancer and non-cancer methods are more than just a simple reflection of treating all non-cancer toxicity as if it has a threshold, or all cancer toxicity as if it did not. Moreover, one recommendation of NAS \ was to develop a problem formulation with risk managers prior to conducting any risk assessment. If the development of this problem formulation only necessitates the determination of a safe, or virtually safe dose, then the estimation of a Reference Dose (RfD) or virtually safe dose (VSD) or similar constructs should be encouraged. Not all of our environmental problems need a precise quantitative solution.

Potential use of gene drive modified insects against disease vectors, agricultural pests and invasive species poses new challenges for risk assessment.

Potential future application of engineered gene drives (GDs), which bias their own inheritance and can spread genetic modifications in wild target populations, has sparked both enthusiasm and concern. Engineered GDs in insects could potentially be used to address long-standing challenges in control of disease vectors, agricultural pests and invasive species, or help to rescue endangered species, and thus provide important public benefits. However, there are concerns that the deliberate environmental release of GD modified insects may pose different or new harms to animal and human health and the wider environment, and raise novel challenges for risk assessment. Risk assessors, risk managers, developers, potential applicants and other stakeholders at many levels are currently discussing whether there is a need to develop new or additional risk assessment guidance for the environmental release of GD modified organisms, including insects. Developing new or additional guidance that is useful and practical is a challenge, especially at an international level, as risk assessors, risk managers and many other stakeholders have different, often contrasting, opinions and perspectives toward the environmental release of GD modified organisms, and on the adequacy of current risk assessment frameworks for such organisms. Here, we offer recommendations to overcome some of the challenges associated with the potential future development of new or additional risk assessment guidance for GD modified insects and provide considerations on areas where further risk assessment guidance may be required.

Systematic identification of plausible pathways to potential harm via problem formulation for investigational releases of a population suppression gene drive to control the human malaria vector Anopheles gambiae in West Africa.

BACKGROUND: Population suppression gene drive has been proposed as a strategy for malaria vector control. A CRISPR-Cas9-based transgene homing at the doublesex locus (dsxFCRISPRh) has recently been shown to increase rapidly in frequency in, and suppress, caged laboratory populations of the malaria mosquito vector Anopheles gambiae. Here, problem formulation, an initial step in environmental risk assessment (ERA), was performed for simulated field releases of the dsxFCRISPRh transgene in West Africa. METHODS: Building on consultative workshops in Africa that previously identified relevant environmental and health protection goals for ERA of gene drive in malaria vector control, 8 potentially harmful effects from these simulated releases were identified. These were stratified into 46 plausible pathways describing the causal chain of events that would be required for potential harms to occur. Risk hypotheses to interrogate critical steps in each pathway, and an analysis plan involving experiments, modelling and literature review to test each of those risk hypotheses, were developed. RESULTS: Most potential harms involved increased human (n = 13) or animal (n = 13) disease transmission, emphasizing the importance to subsequent stages of ERA of data on vectorial capacity comparing transgenics to non-transgenics. Although some of the pathways (n = 14) were based on known anatomical alterations in dsxFCRISPRh homozygotes, many could also be applicable to field releases of a range of other transgenic strains of mosquito (n = 18). In addition to population suppression of target organisms being an accepted outcome for existing vector control programmes, these investigations also revealed that the efficacy of population suppression caused by the dsxFCRISPRh transgene should itself directly affect most pathways (n = 35). CONCLUSIONS: Modelling will play an essential role in subsequent stages of ERA by clarifying the dynamics of this relationship between population suppression and reduction in exposure to specific potential harms. This analysis represents a comprehensive identification of plausible pathways to potential harm using problem formulation for a specific gene drive transgene and organism, and a transparent communication tool that could inform future regulatory studies, guide subsequent stages of ERA, and stimulate further, broader engagement on the use of population suppression gene drive to control malaria vectors in West Africa.

Advancing ecological risk assessment on genetically engineered breeding stacks with combined insect-resistance traits.

To inform the ecological risk assessment (ERA) of a transgenic crop with multiple insecticidal traits combined by conventional breeding (breeding stack), a comparative field study is customarily conducted to compare transgenic protein concentrations in a breeding stack to those in corresponding component single events used in the breeding process. This study tests the hypothesis that transgenic protein expression will not significantly increase due to stacking, such that existing margins of exposure erode to unacceptable levels. Corroboration of this hypothesis allows for the use of existing non-target organism (NTO) effects tests results, where doses were based on the estimated environmental concentrations determined for a component single event. Results from over 20 studies comparing expression profiles of insecticidal proteins produced by commercial events in various combinations of conventionally-bred stacks were examined to evaluate applying previously determined no-observed-effect concentrations (NOECs) to stack ERAs. This paper presents a large number of tests corroborating the hypothesis of no significant increase in insecticidal protein expression due to combination by conventional breeding, and much of the variation in protein expression is likely attributed to genetic and environmental factors. All transgenic protein concentrations were well within conservative margins between exposure and corresponding NOEC. This work supports the conclusion that protein expression data generated for single events and the conservative manner for setting NTO effects test concentrations allows for the transportability of existing NOECs to the ERA of conventionally-bred stacks, and that future tests of the stated hypothesis are no longer critically informative for ERA on breeding stacks.

The Tango of Problem Formulation: A Patient's/Researcher's Reflection on an Action Design Research Journey.

This paper reports on the reflection of the lead researcher, a 48-year-old patient with cystic fibrosis (CF), and aims to portray his real-life experience of a 10-month action design research (ADR) project. Playing a dual role, as both a patient and researcher, the lead researcher reflects deeply on his ADR experience with particular emphasis on the problem formulation stage of creating a simple yet impactful checklist to aid memory recall of CF patients or caregivers during a medical appointment. Using Driscoll's model of reflection, a real-life unsanitized ADR experience is carefully imparted via a series of 4 vignettes, including 4 key learnings, which highlight the connection between a meticulous considered approach to problem formulation and truly effective outcomes. By providing this rich account of problem formulation within ADR, it is hoped that this reflection will help researchers to better understand the complexity of problem formulation in design-oriented research; to avoid making assumptions and becoming fixated on solutions; and to move instead to an end point where several possible ways of examining a problem have been considered, explored, and understood-an end point where successful end results are reached through grit and determination. This paper advocates for the inclusion and portrayal of the actual realities or ups and downs of this dynamic and evolving stage of ADR, capturing the often-tacit knowledge of problem formulation and begetting a sense of realism and humanity to ADR serving as knowledge contributions in their own right. The lead researcher is the patient and researcher in this ADR project. This is my story!

Problem formulation for gene drive mosquitoes designed to reduce malaria transmission in Africa: results from four regional consultations 2016-2018.

BACKGROUND: Gene drive mosquitoes have been proposed as a possible means to reduce the transmission of malaria in Africa. Because this technology has no prior use-history at this time, environmental risk assessments for gene drive mosquitoes will benefit from problem formulation-an organized and ordered process to identify protection goals and potential pathways to harm to the environment, or animal or human health. Recognizing this need, the New Partnership for Africa's Development (NEPAD), with support from African and international partners, organized four regional consultative workshops in Africa to initiate this process. METHODS: The workshops were attended by a diverse set of participants and stakeholders, including scientists, ethicists, health professionals, government regulators in the fields of environment health and biosafety as well government policymakers, who met for 4 days to deliberate on protection goals and pathways relevant to the use of gene drive mosquitoes for malaria control. The goal of the workshops was not to produce a comprehensive and detailed environmental risk assessment of gene drive mosquitoes, but rather to introduce problem formulation as a tool to the stakeholder community, and to serve as a starting point for conducting systematic environmental risk assessments in the future, identifying protection goals related to gene drive mosquitoes that are particular to African stakeholders. RESULTS: Participants in the workshops frequently identified human health and biodiversity as being relevant broad protection goals. Results of the deliberations provide insight into the concerns of African participants at an early stage in the development of gene drive organism/products that should be instructive to developers using this technology. CONCLUSIONS: In general, the African participants of the consultations had a precautionary perspective with regard to environmental risk assessment of gene drive technology. As gene drive technology develops, protection goals will become further refined and candidate products will be further defined. These workshops represent only the beginning of a continuing process that will ultimately inform environmental risk assessment for gene drive mosquitoes to control malaria in Africa.

A simple problem formulation framework to create the right solution to the right problem.

A systematic approach to formulate consistent, technically robust and scientifically tractable problems will facilitate achieving innovative and effective solutions in risk evaluation. The fundamentals of problem formulation have been adapted from environmental and human health risk assessments. A structured problem formulation enables focus on describing and evaluating the specifics of the problem to be solved, instead of immediately creating solutions. First the problem should be framed to provide clarity and gain agreement on the problem to be addressed, resulting in a specific problem statement. Second the problem is explored in order to transform it into an operational state through questions to answer, hypotheses to test, and represented by a conceptual model. Finally the approach to testing hypotheses is mapped and the analysis plan is developed to address the problem statement. This simple adaptable framework can be applied to any circumstance to resolve a specific problem and describe a path to resolution.

Improving risk assessment approaches for chemicals with both endogenous and exogenous exposures.

To conduct risk assessments of exogenous chemicals for which there are also endogenous exposures, knowledge of the chemistry and biology of both types of exposures needs to be integrated into problem formulation and carried through to risk characterization. This issue is framed in a risk assessment context, highlighting the importance of quantifying increments of dose from all sources of the same or similar chemicals interacting with biological targets; understanding the influence of endogenous chemical concentrations on disease risk; and assessing total dose to targets in evaluating risk from incremental environmental exposures. Examples of recent assessments illustrate the importance of addressing this issue. Evaluations of data on blood or organ concentrations of ammonia, methanol, formaldehyde, acetaldehyde, and three gaseous signaling molecules (hydrogen sulfide, carbon monoxide, and nitric oxide) provide examples where current data are already informing perspectives on relative exposures at the portal of entry and systemically. To facilitate quality risk assessments of exogenous chemicals with endogenous exposures, a series of specific questions are presented that need to be addressed in systematic review to enhance problem formulation, improve the development of holistic conceptual models, and to facilitate the identification of priority data needs for improving risk assessments.

Evolution of risk assessment strategies for food and feed uses of stacked GM events.

Data requirements are not harmonized globally for the regulation of food and feed derived from stacked genetically modified (GM) events, produced by combining individual GM events through conventional breeding. The data required by some regulatory agencies have increased despite the absence of substantiated adverse effects to animals or humans from the consumption of GM crops. Data from studies conducted over a 15-year period for several stacked GM event maize (Zea mays L.) products (Bt11 ×  GA21, Bt11 ×  MIR604, MIR604 ×  GA21, Bt11 ×  MIR604 ×  GA21, Bt11 ×  MIR162 ×  GA21 and Bt11 ×  MIR604 ×  MIR162 ×  GA21), together with their component single events, are presented. These data provide evidence that no substantial changes in composition, protein expression or insert stability have occurred after combining the single events through conventional breeding. An alternative food and feed risk assessment strategy for stacked GM events is suggested based on a problem formulation approach that utilizes (i) the outcome of the single event risk assessments, and (ii) the potential for interactions in the stack, based on an understanding of the mode of action of the transgenes and their products.

Risk assessment of gene flow from genetically engineered virus resistant cassava to wild relatives in Africa: an expert panel report.

The probability and consequences of gene flow to wild relatives is typically considered in the environmental risk assessment of genetically engineered crops. This is a report from a discussion by a group of experts who used a problem formulation approach to consider existing information for risk assessment of gene flow from cassava (Manihot esculenta) genetically engineered for virus resistance to the 'wild' (naturalized) relative M. glaziovii in East Africa. Two environmental harms were considered in this case: (1) loss of genetic diversity in the germplasm pool, and (2) loss of valued species, ecosystem resources, or crop yield and quality due to weediness or invasiveness of wild relatives. Based on existing information, it was concluded that gene flow will occur, but it is not likely that this will reduce the genetic diversity in the germplasm pool. There is little existing information about the impact of the virus in natural populations that could be used to inform a prediction about whether virus resistance would lead to an increase in reproduction or survival, hence abundance of M. glaziovii. However, an increase in the abundance of M. glaziovii should be manageable, and would not necessarily lead to the identified environmental harms.

Development of a construct-based risk assessment framework for genetic engineered crops.

Experience gained in the risk assessment (RA) of genetically engineered (GE) crops since their first experimental introductions in the early nineties, has increased the level of familiarity with these breeding methodologies and has motivated several agencies and expert groups worldwide to revisit the scientific criteria underlying the RA process. Along these lines, the need to engage in a scientific discussion for the case of GE crops transformed with similar constructs was recently identified in Argentina. In response to this need, the Argentine branch of the International Life Sciences Institute (ILSI Argentina) convened a tripartite working group to discuss a science-based evaluation approach for transformation events developed with genetic constructs which are identical or similar to those used in previously evaluated or approved GE crops. This discussion considered new transformation events within the same or different species and covered both environmental and food safety aspects. A construct similarity concept was defined, considering the biological function of the introduced genes. Factors like environmental and dietary exposure, familiarity with both the crop and the trait as well as the crop biology, were identified as key to inform a construct-based RA process.

Problem formulation for risk assessment of combined exposures to chemicals and other stressors in humans.

When the human health risk assessment/risk management paradigm was developed, it did not explicitly include a "problem formulation" phase. The concept of problem formulation was first introduced in the context of ecological risk assessment (ERA) for the pragmatic reason to constrain and focus ERAs on the key questions. However, this need also exists for human health risk assessment, particularly for cumulative risk assessment (CRA), because of its complexity. CRA encompasses the combined threats to health from exposure via all relevant routes to multiple stressors, including biological, chemical, physical and psychosocial stressors. As part of the HESI Risk Assessment in the 21st Century (RISK21) Project, a framework for CRA was developed in which problem formulation plays a critical role. The focus of this effort is primarily on a chemical CRA (i.e., two or more chemicals) with subsequent consideration of non-chemical stressors, defined as "modulating factors" (ModFs). Problem formulation is a systematic approach that identifies all factors critical to a specific risk assessment and considers the purpose of the assessment, scope and depth of the necessary analysis, analytical approach, available resources and outcomes, and overall risk management goal. There are numerous considerations that are specific to multiple stressors, and proper problem formulation can help to focus a CRA to the key factors in order to optimize resources. As part of the problem formulation, conceptual models for exposures and responses can be developed that address these factors, such as temporal relationships between stressors and consideration of the appropriate ModFs.

Modernizing problem formulation for risk assessment necessitates articulation of mode of action.

The process of scientific hypothesis formulation affects the experimental designs, methods and interpretations applied, but to be testable, the hypotheses posed must conform to the state of scientific knowledge and available technology. An analogous situation exists in risk assessment, where the questions addressed are typically articulated in the problem formulation phase. Decades ago, regulatory agencies couched problem formulation according to the questions answerable by the science of the day. As regulatory requirements for risk assessment became codified, so too did the rudiments of problem formulation. Unfortunately, codifying problem formulation prevented it from evolving to keep pace with scientific advancements. Today's more advanced science is not always being used effectively and efficiently in risk assessment because the risk assessment problem formulation step still typically poses antiquated questions. Problem formulation needs to be improved so that modern science can inform risk considerations. Based on recent developments in the Human Relevance Framework and using well-studied example chemicals - chloroform and carbon tetrachloride - an approach is proposed for focusing problem formulation on human-relevant hypotheses. We contend that modernizing problem formulation in this way will make risk assessment more scientifically accurate, more practical, and more relevant for protecting human health and the environment.

A 21st century roadmap for human health risk assessment.

The Health and Environmental Sciences Institute (HESI)-coordinated Risk Assessment in the 21st Century (RISK21) project was initiated to develop a scientific, transparent, and efficient approach to the evolving world of human health risk assessment, and involved over 120 participants from 12 countries, 15 government institutions, 20 universities, 2 non-governmental organizations, and 12 corporations. This paper provides a brief overview of the tiered RISK21 framework called the roadmap and risk visualization matrix, and articulates the core principles derived by RISK21 participants that guided its development. Subsequent papers describe the roadmap and matrix in greater detail. RISK21 principles include focusing on problem formulation, utilizing existing information, starting with exposure assessment (rather than toxicity), and using a tiered process for data development. Bringing estimates of exposure and toxicity together on a two-dimensional matrix provides a clear rendition of human safety and risk. The value of the roadmap is its capacity to chronicle the stepwise acquisition of scientific information and display it in a clear and concise fashion. Furthermore, the tiered approach and transparent display of information will contribute to greater efficiencies by calling for data only as needed (enough precision to make a decision), thus conserving animals and other resources.

Risk assessment in the 21st century: roadmap and matrix.

Abstract The RISK21 integrated evaluation strategy is a problem formulation-based exposure-driven risk assessment roadmap that takes advantage of existing information to graphically represent the intersection of exposure and toxicity data on a highly visual matrix. This paper describes in detail the process for using the roadmap and matrix. The purpose of this methodology is to optimize the use of prior information and testing resources (animals, time, facilities, and personnel) to efficiently and transparently reach a risk and/or safety determination. Based on the particular problem, exposure and toxicity data should have sufficient precision to make such a decision. Estimates of exposure and toxicity, bounded by variability and/or uncertainty, are plotted on the X- and Y-axes of the RISK21 matrix, respectively. The resulting intersection is a highly visual representation of estimated risk. Decisions can then be made to increase precision in the exposure or toxicity estimates or declare that the available information is sufficient. RISK21 represents a step forward in the goal to introduce new methodologies into 21st century risk assessment. Indeed, because of its transparent and visual process, RISK21 has the potential to widen the scope of risk communication beyond those with technical expertise.

Introduction and scientific justification of data transportability for confined field testing for the ERA of GM plants.

The concept of Data Transportability (DT) of Confined Field Testing (CFT) to support the Environmental Risk Assessment (ERA) of Genetically Modified (GM) plants was first introduced in the literature by Garcia-Alonso et al., in 2014. Since then, DT has been discussed in many countries and regions as a concept to prevent duplication of regulatory studies without compromising quality of the ERA. However, despite its usefulness and scientific justification, DT is not well adopted at this time and many regulatory agencies around the world require additional in-country CFT be conducted before approving GM plants. Based on the current circumstances, the authors organized a parallel session entitled "Introduction and Scientific Justification of DT for CFT for the ERA of GM plants" at 16th ISBR (the International Society for Biosafety Research). This session mainly consisted of the following three parts. The first two speakers, Andrew Roberts and Abigail Simmons provided an overview of DT and examples of conditions for the transportability of field data/conclusions advocated in the peer-reviewed scientific journals. Next, the current status of DT adoption in some countries/regions such as Japan and Africa, and a theoretical case study for Argentina were introduced by Kazuyuki Hiratsuka, Douglas Miano, and Facundo Vesprini, respectively. Lastly, a risk hypothesis-based approach for DT which was developed in advance by the five speakers of this parallel session, was introduced. During the discussion, there was a common understanding that transition to the risk hypothesis-based approach for DT was scientifically appropriate, considering the accumulated evidences that several countries have conducted confirmatory local CFT for more than 20 years but they have not detected any differences related to the ERA assessment endpoints in GM crops. The risk hypothesis-based approach for DT introduced here is expected to play an important role in discussions on the implementation of DT in various parts of the world in the future.

Modernizing and harmonizing regulatory data requirements for genetically modified crops-perspectives from a workshop.

Genetically modified (GM) crops that have been engineered to express transgenes have been in commercial use since 1995 and are annually grown on 200 million hectares globally. These crops have provided documented benefits to food security, rural economies, and the environment, with no substantiated case of food, feed, or environmental harm attributable to cultivation or consumption. Despite this extensive history of advantages and safety, the level of regulatory scrutiny has continually increased, placing undue burdens on regulators, developers, and society, while reinforcing consumer distrust of the technology. CropLife International held a workshop at the 16th International Society of Biosafety Research (ISBR) Symposium to examine the scientific basis for modernizing global regulatory frameworks for GM crops. Participants represented a spectrum of global stakeholders, including academic researchers, GM crop developers, regulatory consultants, and regulators. Concurrently examining the considerations of food and feed safety, along with environmental safety, for GM crops, the workshop presented recommendations for a core set of data that should always be considered, and supplementary (i.e., conditional) data that would be warranted only on a case-by-case basis to address specific plausible hypotheses of harm. Then, using a case-study involving a hypothetical GM maize event expressing two familiar traits (insect protection and herbicide tolerance), participants were asked to consider these recommendations and discuss if any additional data might be warranted to support a science-based risk assessment or for regulatory decision-making. The discussions during the workshop highlighted that the set of data to address the food, feed, and environmental safety of the hypothetical GM maize, in relation to a conventional comparator, could be modernized compared to current global regulatory requirements. If these scientific approaches to modernize data packages for GM crop regulation were adopted globally, GM crops could be commercialized in a more timely manner, thereby enabling development of more diverse GM traits to benefit growers, consumers, and the environment.

Fully automating LI-RADS on MRI with deep learning-guided lesion segmentation, feature characterization, and score inference.

Introduction: Leveraging deep learning in the radiology community has great potential and practical significance. To explore the potential of fitting deep learning methods into the current Liver Imaging Reporting and Data System (LI-RADS) system, this paper provides a complete and fully automatic deep learning solution for the LI-RADS system and investigates its model performance in liver lesion segmentation and classification. Methods: To achieve this, a deep learning study design process is formulated, including clinical problem formulation, corresponding deep learning task identification, data acquisition, data preprocessing, and algorithm validation. On top of segmentation, a UNet++-based segmentation approach with supervised learning was performed by using 33,078 raw images obtained from 111 patients, which are collected from 2010 to 2017. The key innovation is that the proposed framework introduces one more step called feature characterization before LI-RADS score classification in comparison to prior work. In this step, a feature characterization network with multi-task learning and joint training strategy was proposed, followed by an inference module to generate the final LI-RADS score. Results: Both liver segmentation and feature characterization models were evaluated, and comprehensive statistical analysis was conducted with detailed discussions. Median DICE of liver lesion segmentation was able to achieve 0.879. Based on different thresholds, recall changes within a range of 0.7 to 0.9, and precision always stays high greater than 0.9. Segmentation model performance was also evaluated on the patient level and lesion level, and the evaluation results of (precision, recall) on the patient level were much better at approximately (1, 0.9). Lesion classification was evaluated to have an overall accuracy of 76%, and most mis-classification cases happen in the neighboring categories, which is reasonable since it is naturally difficult to distinguish LI-RADS 4 from LI-RADS 5. Discussion: In addition to investigating the performance of the proposed model itself, extensive comparison experiment was also conducted. This study shows that our proposed framework with feature characterization greatly improves the diagnostic performance which also validates the effectiveness of the added feature characterization step. Since this step could output the feature characterization results instead of simply generating a final score, it is able to unbox the black-box for the proposed algorithm thus improves the explainability.