Explaining the optimistic performance evaluation of newly proposed methods: A cross-design validation experiment.

The constant development of new data analysis methods in many fields of research is accompanied by an increasing awareness that these new methods often perform better in their introductory paper than in subsequent comparison studies conducted by other researchers. We attempt to explain this discrepancy by conducting a systematic experiment that we call "cross-design validation of methods". In the experiment, we select two methods designed for the same data analysis task, reproduce the results shown in each paper, and then reevaluate each method based on the study design (i.e., datasets, competing methods, and evaluation criteria) that was used to show the abilities of the other method. We conduct the experiment for two data analysis tasks, namely cancer subtyping using multiomic data and differential gene expression analysis. Three of the four methods included in the experiment indeed perform worse when they are evaluated on the new study design, which is mainly caused by the different datasets. Apart from illustrating the many degrees of freedom existing in the assessment of a method and their effect on its performance, our experiment suggests that the performance discrepancies between original and subsequent papers may not only be caused by the nonneutrality of the authors proposing the new method but also by differences regarding the level of expertise and field of application. Authors of new methods should thus focus not only on a transparent and extensive evaluation but also on comprehensive method documentation that enables the correct use of their methods in subsequent studies.

Beneficial effects of starting oral cysteamine treatment in the first 2 months of life on glomerular and tubular kidney function in infantile nephropathic cystinosis.

Nephropathic cystinosis is a rare lysosomal storage disease whose basic defect, impaired transport of cystine out of lysosomes, results in intracellular cystine storage. Affected individuals exhibit renal Fanconi Syndrome in infancy, end-stage kidney disease at approximately 10 years of age, and many other systemic complications. Oral cysteamine therapy mitigates the detrimental effects on glomerular function and prevents most of the late complications of the disease but has not shown benefit with respect to the early tubular damage of cystinosis. This is because cystinosis is generally diagnosed in the second year of life, after the damage to kidney tubular function has already occurred. We longitudinally evaluated 6 infants diagnosed and treated with cysteamine from before 2 months of age. The 4 infants with good compliance with cysteamine and consistently low leucocyte cystine levels maintained normal eGFR values, exhibited only minor degrees of renal Fanconi Syndrome, and maintained normal serum levels of potassium, bicarbonate, phosphate, and calcium without electrolyte or mineral supplementation through 2, 4, 10 and 16 years of age. Thus, renal Fanconi syndrome can be attenuated by early administration of cysteamine and renew the call for molecular-based newborn screening for cystinosis.

Relationship between age at initiation of cysteamine treatment, adherence with therapy, and glomerular kidney function in infantile nephropathic cystinosis.

Infantile nephropathic cystinosis, due to impaired transport of cystine out of lysosomes, occurs with an incidence of 1 in 100-200,000 live births. It is characterized by renal Fanconi syndrome in the first year of life and glomerular dysfunction progression to end-stage kidney disease by approximately 10 years of age. Treatment with oral cysteamine therapy helps preserve glomerular function, but affected individuals eventually require kidney replacement therapy. This is because glomerular damage had already occurred by the time a child is diagnosed with cystinosis, typically in the second year of life. We performed a retrospective multicenter study to investigate the impact of initiating cysteamine treatment within the first 2 months of life in some infants and comparing two different levels of adherence in patients diagnosed at the typical age. We collected 3983 data points from 55 patients born between 1997 and 2020; 52 patients with 1592 data points could be further evaluated. These data were first analyzed by dividing the patient cohort into three groups: (i) standard treatment start with good adherence, (ii) standard treatment start with less good adherence, and (iii) early treatment start. At every age, mean estimated glomerular filtration rate (eGFR) was higher in early-treated patients than in later-treated patients. Second, a generalized additive mixed model (GAMM) was applied showing that patients with initiation of treatment before 2 months of age are expected to have a 34 ml/min/1.73 m2 higher eGFR than patients with later treatment start while controlling for adherence and patients' age. These data strongly suggest that oral cysteamine treatment initiated within 2 months of birth preserves kidney function in infantile nephropathic cystinosis and provide evidence of the utility of newborn screening for this disease.