Predicting Out-of-Hospital Cardiac Arrest in the General Population Using Electronic Health Records.

BACKGROUND: The majority of out-of-hospital cardiac arrests (OHCAs) occur among individuals in the general population, for whom there is no established strategy to identify risk. In this study, we assess the use of electronic health record (EHR) data to identify OHCA in the general population and define salient factors contributing to OHCA risk. METHODS: The analytical cohort included 2366 individuals with OHCA and 23 660 age- and sex-matched controls receiving health care at the University of Washington. Comorbidities, electrocardiographic measures, vital signs, and medication prescription were abstracted from the EHR. The primary outcome was OHCA. Secondary outcomes included shockable and nonshockable OHCA. Model performance including area under the receiver operating characteristic curve and positive predictive value were assessed and adjusted for observed rate of OHCA across the health system. RESULTS: There were significant differences in demographic characteristics, vital signs, electrocardiographic measures, comorbidities, and medication distribution between individuals with OHCA and controls. In external validation, discrimination in machine learning models (area under the receiver operating characteristic curve 0.80-0.85) was superior to a baseline model with conventional cardiovascular risk factors (area under the receiver operating characteristic curve 0.66). At a specificity threshold of 99%, correcting for baseline OHCA incidence across the health system, positive predictive value was 2.5% to 3.1% in machine learning models compared with 0.8% for the baseline model. Longer corrected QT interval, substance abuse disorder, fluid and electrolyte disorder, alcohol abuse, and higher heart rate were identified as salient predictors of OHCA risk across all machine learning models. Established cardiovascular risk factors retained predictive importance for shockable OHCA, but demographic characteristics (minority race, single marital status) and noncardiovascular comorbidities (substance abuse disorder) also contributed to risk prediction. For nonshockable OHCA, a range of salient predictors, including comorbidities, habits, vital signs, demographic characteristics, and electrocardiographic measures, were identified. CONCLUSIONS: In a population-based case-control study, machine learning models incorporating readily available EHR data showed reasonable discrimination and risk enrichment for OHCA in the general population. Salient factors associated with OCHA risk were myriad across the cardiovascular and noncardiovascular spectrum. Public health and tailored strategies for OHCA prediction and prevention will require incorporation of this complexity.

Statistical approaches for the integration of external controls in a cystic fibrosis clinical trial: a simulation and an application.

Development of new therapeutics for a rare disease such as cystic fibrosis (CF) is hindered by challenges in accruing enough patients for clinical trials. Using external controls from well-matched historical trials can reduce prospective trial sizes, and this approach has supported regulatory approval of new interventions for other rare diseases. We consider three statistical methods that incorporate external controls into a hypothetical clinical trial of a new treatment to reduce pulmonary exacerbations in CF patients: 1) inverse probability weighting, 2) Bayesian modeling with propensity score-based power priors, and 3) hierarchical Bayesian modeling with commensurate priors. We compare the methods via simulation study and in a real clinical trial data setting. Simulations showed that bias in the treatment effect was <4% using any of the methods, with type 1 error (or in the Bayesian cases, posterior probability of the null hypothesis) usually <5%. Inverse probability weighting was sensitive to similarity in prevalence of the covariates between historical and prospective trial populations. The commensurate prior method performed best with real clinical trial data. Using external controls to reduce trial size in future clinical trials holds promise and can advance the therapeutic pipeline for rare diseases.

An online framework for survival analysis: reframing Cox proportional hazards model for large data sets and neural networks.

In many biomedical applications, outcome is measured as a "time-to-event" (e.g., disease progression or death). To assess the connection between features of a patient and this outcome, it is common to assume a proportional hazards model and fit a proportional hazards regression (or Cox regression). To fit this model, a log-concave objective function known as the "partial likelihood" is maximized. For moderate-sized data sets, an efficient Newton-Raphson algorithm that leverages the structure of the objective function can be employed. However, in large data sets this approach has two issues: (i) The computational tricks that leverage structure can also lead to computational instability; (ii) The objective function does not naturally decouple: Thus, if the data set does not fit in memory, the model can be computationally expensive to fit. This additionally means that the objective is not directly amenable to stochastic gradient-based optimization methods. To overcome these issues, we propose a simple, new framing of proportional hazards regression: This results in an objective function that is amenable to stochastic gradient descent. We show that this simple modification allows us to efficiently fit survival models with very large data sets. This also facilitates training complex, for example, neural-network-based, models with survival data.

Time to Awakening and Self-Fulfilling Prophecies After Cardiac Arrest.

OBJECTIVES: Withdrawal of life-sustaining therapies for perceived poor neurologic prognosis (WLST-N) is common after resuscitation from cardiac arrest and may bias outcome estimates from models trained using observational data. We compared several approaches to outcome prediction with the goal of identifying strategies to quantify and reduce this bias. DESIGN: Retrospective observational cohort study. SETTING: Two academic medical centers ("UPMC" and "University of Alabama Birmingham" [UAB]). PATIENTS: Comatose adults resuscitated from cardiac arrest. INTERVENTION: None. MEASUREMENTS AND MAIN RESULTS: As potential predictors, we considered clinical, laboratory, imaging, and quantitative electroencephalography data available early after hospital arrival. We followed patients until death, discharge, or awakening from coma. We used penalized Cox regression with a least absolute shrinkage and selection operator penalty and five-fold cross-validation to predict time to awakening in UPMC patients and then externally validated the model in UAB patients. This model censored patients after WLST-N, considering subsequent potential for awakening to be unknown. Next, we developed a penalized logistic model predicting awakening, which treated failure to awaken after WLST-N as a true observed outcome, and a separate logistic model predicting WLST-N. We scaled and centered individual patients' Cox and logistic predictions for awakening to allow direct comparison and then explored the difference in predictions across probabilities of WLST-N. Overall, 1,254 patients were included, and 29% awakened. Cox models performed well (mean area under the curve was 0.93 in the UPMC test sets and 0.83 in external validation). Logistic predictions of awakening were systematically more pessimistic than Cox-based predictions for patients at higher risk of WLST-N, suggesting potential for self-fulfilling prophecies to arise when failure to awaken after WLST-N is considered as the ground truth outcome. CONCLUSIONS: Compared with traditional binary outcome prediction, censoring outcomes after WLST-N may reduce potential for bias and self-fulfilling prophecies.

Overcoming barriers in the design and implementation of clinical trials for acute kidney injury: a report from the 2020 Kidney Disease Clinical Trialists meeting.

Acute kidney injury (AKI) is a growing epidemic and is independently associated with increased risk of death, chronic kidney disease (CKD) and cardiovascular events. Randomized-controlled trials (RCTs) in this domain are notoriously challenging and many clinical studies in AKI have yielded inconclusive findings. Underlying this conundrum is the inherent heterogeneity of AKI in its etiology, presentation and course. AKI is best understood as a syndrome and identification of AKI subphenotypes is needed to elucidate the disease's myriad etiologies and to tailor effective prevention and treatment strategies. Conventional RCTs are logistically cumbersome and often feature highly selected patient populations that limit external generalizability and thus alternative trial designs should be considered when appropriate. In this narrative review of recent developments in AKI trials based on the Kidney Disease Clinical Trialists (KDCT) 2020 meeting, we discuss barriers to and strategies for improved design and implementation of clinical trials for AKI patients, including predictive and prognostic enrichment techniques, the use of pragmatic trials and adaptive trials.

Selective prediction-set models with coverage rate guarantees.

The current approach to using machine learning (ML) algorithms in healthcare is to either require clinician oversight for every use case or use their predictions without any human oversight. We explore a middle ground that lets ML algorithms abstain from making a prediction to simultaneously improve their reliability and reduce the burden placed on human experts. To this end, we present a general penalized loss minimization framework for training selective prediction-set (SPS) models, which choose to either output a prediction set or abstain. The resulting models abstain when the outcome is difficult to predict accurately, such as on subjects who are too different from the training data, and achieve higher accuracy on those they do give predictions for. We then introduce a model-agnostic, statistical inference procedure for the coverage rate of an SPS model that ensembles individual models trained using K-fold cross-validation. We find that SPS ensembles attain prediction-set coverage rates closer to the nominal level and have narrower confidence intervals for its marginal coverage rate. We apply our method to train neural networks that abstain more for out-of-sample images on the MNIST digit prediction task and achieve higher predictive accuracy for ICU patients compared to existing approaches.

Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction.

BACKGROUND: There is increasing interest in clinical prediction models for rare outcomes such as suicide, psychiatric hospitalizations, and opioid overdose. Accurate model validation is needed to guide model selection and decisions about whether and how prediction models should be used. Split-sample estimation and validation of clinical prediction models, in which data are divided into training and testing sets, may reduce predictive accuracy and precision of validation. Using all data for estimation and validation increases sample size for both procedures, but validation must account for overfitting, or optimism. Our study compared split-sample and entire-sample methods for estimating and validating a suicide prediction model. METHODS: We compared performance of random forest models estimated in a sample of 9,610,318 mental health visits ("entire-sample") and in a 50% subset ("split-sample") as evaluated in a prospective validation sample of 3,754,137 visits. We assessed optimism of three internal validation approaches: for the split-sample prediction model, validation in the held-out testing set and, for the entire-sample model, cross-validation and bootstrap optimism correction. RESULTS: The split-sample and entire-sample prediction models showed similar prospective performance; the area under the curve, AUC, and 95% confidence interval was 0.81 (0.77-0.85) for both. Performance estimates evaluated in the testing set for the split-sample model (AUC = 0.85 [0.82-0.87]) and via cross-validation for the entire-sample model (AUC = 0.83 [0.81-0.85]) accurately reflected prospective performance. Validation of the entire-sample model with bootstrap optimism correction overestimated prospective performance (AUC = 0.88 [0.86-0.89]). Measures of classification accuracy, including sensitivity and positive predictive value at the 99th, 95th, 90th, and 75th percentiles of the risk score distribution, indicated similar conclusions: bootstrap optimism correction overestimated classification accuracy in the prospective validation set. CONCLUSIONS: While previous literature demonstrated the validity of bootstrap optimism correction for parametric models in small samples, this approach did not accurately validate performance of a rare-event prediction model estimated with random forests in a large clinical dataset. Cross-validation of prediction models estimated with all available data provides accurate independent validation while maximizing sample size.

Considerations for identifying the "right" subgroup in adaptive enrichment trials.

Adaptive Enrichment Trials aim to make efficient use of data in a pivotal trial of a new targeted therapy to both (a) more precisely identify who benefits from that therapy and (b) improve the likelihood of successfully concluding that the drug is effective, while controlling the probability of false positives. There are a number of frameworks for conducting such a trial and decisions that must be made regarding how to identify that target subgroup. Among those decisions, one must choose how aggressively to restrict enrollment criteria based on the accumulating evidence in the trial. In this article, we empirically evaluate the impact of aggressive versus conservative enrollment restrictions on the power of the trial to detect an effect of treatment. We identify that, in some cases, a more aggressive strategy can substantially improve power. This additionally raises an important question regarding label indication: To what degree do we need a formal test of the hypothesis of no treatment effect in the exact population implied by the label indication? We discuss this question and evaluate how our answer for adaptive enrichment trials may relate to the answer implied by current practice for broad eligibility trials.

An Online Projection Estimator for Nonparametric Regression in Reproducing Kernel Hilbert Spaces.

The goal of nonparametric regression is to recover an underlying regression function from noisy observations, under the assumption that the regression function belongs to a prespecified infinite-dimensional function space. In the online setting, in which the observations come in a stream, it is generally computationally infeasible to refit the whole model repeatedly. As yet, there are no methods that are both computationally efficient and statistically rate optimal. In this paper, we propose an estimator for online nonparametric regression. Notably, our estimator is an empirical risk minimizer in a deterministic linear space, which is quite different from existing methods that use random features and a functional stochastic gradient. Our theoretical analysis shows that this estimator obtains a rate-optimal generalization error when the regression function is known to live in a reproducing kernel Hilbert space. We also show, theoretically and empirically, that the computational cost of our estimator is much lower than that of other rate-optimal estimators proposed for this online setting.

Marginal Bayesian Posterior Inference using Recurrent Neural Networks with Application to Sequential Models.

In Bayesian data analysis, it is often important to evaluate quantiles of the posterior distribution of a parameter of interest (e.g., to form posterior intervals). In multi-dimensional problems, when non-conjugate priors are used, this is often difficult generally requiring either an analytic or sampling-based approximation, such as Markov chain Monte-Carlo (MCMC), Approximate Bayesian computation (ABC) or variational inference. We discuss a general approach that reframes this as a multi-task learning problem and uses recurrent deep neural networks (RNNs) to approximately evaluate posterior quantiles. As RNNs carry information along a sequence, this application is particularly useful in time-series. An advantage of this risk-minimization approach is that we do not need to sample from the posterior or calculate the likelihood. We illustrate the proposed approach in several examples.

Multidimensional analysis and detection of informative features in human brain white matter.

The white matter contains long-range connections between different brain regions and the organization of these connections holds important implications for brain function in health and disease. Tractometry uses diffusion-weighted magnetic resonance imaging (dMRI) to quantify tissue properties along the trajectories of these connections. Statistical inference from tractometry usually either averages these quantities along the length of each fiber bundle or computes regression models separately for each point along every one of the bundles. These approaches are limited in their sensitivity, in the former case, or in their statistical power, in the latter. We developed a method based on the sparse group lasso (SGL) that takes into account tissue properties along all of the bundles and selects informative features by enforcing both global and bundle-level sparsity. We demonstrate the performance of the method in two settings: i) in a classification setting, patients with amyotrophic lateral sclerosis (ALS) are accurately distinguished from matched controls. Furthermore, SGL identifies the corticospinal tract as important for this classification, correctly finding the parts of the white matter known to be affected by the disease. ii) In a regression setting, SGL accurately predicts "brain age." In this case, the weights are distributed throughout the white matter indicating that many different regions of the white matter change over the lifespan. Thus, SGL leverages the multivariate relationships between diffusion properties in multiple bundles to make accurate phenotypic predictions while simultaneously discovering the most relevant features of the white matter.

Proteomics and Population Biology in the Cardiovascular Health Study (CHS): design of a study with mentored access and active data sharing.

BACKGROUND: In the last decade, genomic studies have identified and replicated thousands of genetic associations with measures of health and disease and contributed to the understanding of the etiology of a variety of health conditions. Proteins are key biomarkers in clinical medicine and often drug-therapy targets. Like genomics, proteomics can advance our understanding of biology. METHODS AND RESULTS: In the setting of the Cardiovascular Health Study (CHS), a cohort study of older adults, an aptamer-based method that has high sensitivity for low-abundance proteins was used to assay 4979 proteins in frozen, stored plasma from 3188 participants (61% women, mean age 74 years). CHS provides active support, including central analysis, for seven phenotype-specific working groups (WGs). Each CHS WG is led by one or two senior investigators and includes 10 to 20 early or mid-career scientists. In this setting of mentored access, the proteomic data and analytic methods are widely shared with the WGs and investigators so that they may evaluate associations between baseline levels of circulating proteins and the incidence of a variety of health outcomes in prospective cohort analyses. We describe the design of CHS, the CHS Proteomics Study, characteristics of participants, quality control measures, and structural characteristics of the data provided to CHS WGs. We additionally highlight plans for validation and replication of novel proteomic associations. CONCLUSION: The CHS Proteomics Study offers an opportunity for collaborative data sharing to improve our understanding of the etiology of a variety of health conditions in older adults.

A SIEVE STOCHASTIC GRADIENT DESCENT ESTIMATOR FOR ONLINE NONPARAMETRIC REGRESSION IN SOBOLEV ELLIPSOIDS.

The goal of regression is to recover an unknown underlying function that best links a set of predictors to an outcome from noisy observations. in nonparametric regression, one assumes that the regression function belongs to a pre-specified infinite-dimensional function space (the hypothesis space). in the online setting, when the observations come in a stream, it is computationally-preferable to iteratively update an estimate rather than refitting an entire model repeatedly. inspired by nonparametric sieve estimation and stochastic approximation methods, we propose a sieve stochastic gradient descent estimator (Sieve-SGD) when the hypothesis space is a Sobolev ellipsoid. We show that Sieve-SGD has rate-optimal mean squared error (MSE) under a set of simple and direct conditions. The proposed estimator can be constructed with a low computational (time and space) expense: We also formally show that Sieve-SGD requires almost minimal memory usage among all statistically rate-optimal estimators.

Total uncertainty: a systematic review and thematic synthesis of experiences of uncertainty in older people with advanced multimorbidity, their informal carers and health professionals.

OBJECTIVES: uncertainty pervades the complex illness trajectories experienced by older adults with multimorbidity. Uncertainty is experienced by older people, their informal carers and professionals providing care, yet is incompletely understood. We aimed to identify and synthesise systematically the experience of uncertainty in advanced multimorbidity from patient, carer and professional perspectives. DESIGN: systematic literature review of published and grey qualitative literature from 9 databases (Prospero CRD 42021227480). PARTICIPANTS: older people with advanced multimorbidity, and informal carers/professionals providing care to this group. Exclusion criteria: early multimorbidity, insufficient focus on uncertainty. ANALYSIS: weight-of-evidence assessment was used to appraise included articles. We undertook thematic synthesis of multi-perspective experiences and response to uncertainty. RESULTS: from 4,738 unique search results, we included 44 articles relating to 40 studies. 22 focused on patient experiences of uncertainty (n = 460), 15 on carer experiences (n = 197), and 19 on health professional experiences (n = 490), with 10 exploring multiple perspectives. We identified a shared experience of 'Total Uncertainty' across five domains: 'appraising and managing multiple illnesses'; 'fragmented care and communication'; 'feeling overwhelmed'; 'uncertainty of others' and 'continual change'. Participants responded to uncertainty by either active (addressing, avoiding) or passive (accepting) means. CONCLUSIONS: the novel concept of 'Total Uncertainty' represents a step change in our understanding of illness experience in advanced multimorbidity. Patients, carers and health professionals experienced uncertainty in similar domains, suggesting a shared understanding is feasible. The domains of total uncertainty form a useful organising framework for health professionals caring for older adults with multimorbidity.

Uncertainty in multimorbidity: a shared experience we should recognise, acknowledge and communicate.

Multimorbidity is increasingly common and inevitably results in uncertainties about health, care and the future. Such uncertainties may be experienced by patients, carers and health professionals. Given the ubiquitous presence of uncertainty, all professionals should be prepared to approach and address it in clinical practice. Uncertainty in multimorbidity can rarely be eliminated, and so, must be carefully addressed and communicated; however, there is little evidence on how to approach it. Key areas are: recognising the existence of uncertainty, acknowledging it, and communicating to achieve a shared understanding. Evaluation of what has been discussed, and preparedness to repeat such conversations are also important. Future research should explore optimal communication of uncertainty in multimorbidity.

Conflict-related intentional injuries in Baghdad, Iraq, 2003-2014: A modeling study and proposed method for calculating burden of injury in conflict.

BACKGROUND: Previous research has focused on the mortality associated with armed conflict as the primary measure of the population health effects of war. However, mortality only demonstrates part of the burden placed on a population by conflict. Injuries and resultant disabilities also have long-term effects on a population and are not accounted for in estimates that focus solely on mortality. Our aim was to demonstrate a new method to describe the effects of both lives lost, and years of disability generated by a given conflict, with data from the US-led 2003 invasion and subsequent occupation of Iraq. METHODS AND FINDINGS: Our data come from interviews conducted in 2014 in 900 Baghdad households containing 5,148 persons. The average household size was 5.72 persons. The majority of the population (55.8%) were between the ages of 19 and 60. Household composition was evenly divided between males and females. Household sample collection was based on methodology previously designed for surveying households in war zones. Survey questions were answered by the head of household or senior adult present. The questions included year the injury occurred, the mechanism of injury, the body parts injured, whether injury resulted in disability and, if so, the length of disability. We present this modeling study to offer an innovative methodology for measuring "years lived with disability" (YLDs) and "years of life lost" (YLLs) attributable to conflict-related intentional injuries, using the Global Burden of Disease (GBD) approach. YLDs were calculated with disability weights, and YLLs were calculated by comparing the age at death to the GBD standard life table to calculate remaining life expectancy. Calculations were also performed using Iraq-specific life expectancy for comparison. We calculated a burden of injury of 5.6 million disability-adjusted life years (DALYs) lost due to conflict-related injuries in Baghdad from 2003 to 2014. The majority of DALYs lost were attributable to YLLs, rather than YLDs, 4.99 million YLLs lost (95% uncertainty interval (UI) 3.87 million to 6.13 million) versus 616,000 YLDs lost (95% UI 399,000 to 894,000). Cause-based analysis demonstrated that more DALYs were lost to due to gunshot wounds (57%) than any other cause. Our study has several limitations. Recall bias regarding the reporting and attribution of injuries is possible. Second, we have no data past the time of the interview, so we assumed individuals with ongoing disability at the end of data collection would not recover, possibly counting more disability for injuries occurring later. Additionally, incomplete data could have led to misclassification of deaths, resulting in an underestimation of the total burden of injury. CONCLUSIONS: In this study, we propose a methodology to perform burden of disease calculations for conflict-related injuries (expressed in DALYs) in Baghdad from 2003 to 2014. We go beyond previous reports of simple mortality to assess long-term population health effects of conflict-related intentional injuries. Ongoing disability is, in cross section, a relatively small 10% of the total burden. Yet, this small proportion creates years of demands on the health system, persistent limitations in earning capacity, and continuing burdens of care provision on family members.

Approval policies for modifications to machine learning-based software as a medical device: A study of bio-creep.

Successful deployment of machine learning algorithms in healthcare requires careful assessments of their performance and safety. To date, the FDA approves locked algorithms prior to marketing and requires future updates to undergo separate premarket reviews. However, this negates a key feature of machine learning-the ability to learn from a growing dataset and improve over time. This paper frames the design of an approval policy, which we refer to as an automatic algorithmic change protocol (aACP), as an online hypothesis testing problem. As this process has obvious analogy with noninferiority testing of new drugs, we investigate how repeated testing and adoption of modifications might lead to gradual deterioration in prediction accuracy, also known as "biocreep" in the drug development literature. We consider simple policies that one might consider but do not necessarily offer any error-rate guarantees, as well as policies that do provide error-rate control. For the latter, we define two online error-rates appropriate for this context: bad approval count (BAC) and bad approval and benchmark ratios (BABR). We control these rates in the simple setting of a constant population and data source using policies aACP-BAC and aACP-BABR, which combine alpha-investing, group-sequential, and gate-keeping methods. In simulation studies, bio-creep regularly occurred when using policies with no error-rate guarantees, whereas aACP-BAC and aACP-BABR controlled the rate of bio-creep without substantially impacting our ability to approve beneficial modifications.

Rejoinder to Discussions on "Approval policies for modifications to machine learning-based software as a medical device: A study of bio-creep".

We thank the discussants for sharing their unique perspectives on the problem of designing automatic algorithm change protocols (aACPs) for machine learning-based software as a medical device. Both Pennello et al. and Rose highlighted a number of challenges that arise in real-world settings, and we whole-heartedly agree that substantial extensions of our work are needed to understand if and how aACPs can be safely deployed in practice. Our work demonstrated that aACPs that appear to be harmless may allow for biocreep, even when the data distribution is assumed to be representative and stationary over time. While we investigated two solutions that protect against this specific issue, many more statistical and practical challenges remain and we look forward to future research on this topic.

Nonparametric variable importance assessment using machine learning techniques.

In a regression setting, it is often of interest to quantify the importance of various features in predicting the response. Commonly, the variable importance measure used is determined by the regression technique employed. For this reason, practitioners often only resort to one of a few regression techniques for which a variable importance measure is naturally defined. Unfortunately, these regression techniques are often suboptimal for predicting the response. Additionally, because the variable importance measures native to different regression techniques generally have a different interpretation, comparisons across techniques can be difficult. In this work, we study a variable importance measure that can be used with any regression technique, and whose interpretation is agnostic to the technique used. This measure is a property of the true data-generating mechanism. Specifically, we discuss a generalization of the analysis of variance variable importance measure and discuss how it facilitates the use of machine learning techniques to flexibly estimate the variable importance of a single feature or group of features. The importance of each feature or group of features in the data can then be described individually, using this measure. We describe how to construct an efficient estimator of this measure as well as a valid confidence interval. Through simulations, we show that our proposal has good practical operating characteristics, and we illustrate its use with data from a study of risk factors for cardiovascular disease in South Africa.

Patient and public involvement in palliative care research: What works, and why? A qualitative evaluation.

BACKGROUND: Public involvement is increasingly considered a prerequisite for high-quality research. However, involvement in palliative care is impeded by limited evidence on the best approaches for populations affected by life-limiting illness. AIM: To evaluate a strategy for public involvement in palliative care and rehabilitation research, to identify successful approaches and areas for improvement. DESIGN: Co-produced qualitative evaluation using focus groups and interviews. Thematic analysis undertaken by research team comprising public contributors and researchers. SETTING/PARTICIPANTS: Researchers and public members from a palliative care and rehabilitation research institute, UK. RESULTS: Seven public members and 19 researchers participated. Building and maintaining relationships, taking a flexible approach and finding the 'right' people were important for successful public involvement. Relationship building created a safe environment for discussing sensitive topics, although public members felt greater consideration of emotional support was needed. Flexibility supported involvement alongside unpredictable circumstances of chronic and life-limiting illness, and was facilitated by responsive communication, and opportunities for in-person and virtual involvement at a project- and institution-level. However, more opportunities for two-way feedback throughout projects was suggested. Finding the 'right' people was crucial given the diverse population served by palliative care, and participants suggested more care needed to be taken to identify public members with experience relevant to specific projects. CONCLUSION: Within palliative care research, it is important for involvement to focus on building and maintaining relationships, working flexibly, and identifying those with relevant experience. Taking a strategic approach and developing adequate infrastructure and networks can facilitate public involvement within this field.