Bioimage analysis workflows: community resources to navigate through a complex ecosystem.

Workflows are the keystone of bioimage analysis, and the NEUBIAS (Network of European BioImage AnalystS) community is trying to gather the actors of this field and organize the information around them.  One of its most recent outputs is the opening of the F1000Research NEUBIAS gateway, whose main objective is to offer a channel of publication for bioimage analysis workflows and associated resources. In this paper we want to express some personal opinions and recommendations related to finding, handling and developing bioimage analysis workflows.  The emergence of "big data" in bioimaging and resource-intensive analysis algorithms make local data storage and computing solutions a limiting factor. At the same time, the need for data sharing with collaborators and a general shift towards remote work, have created new challenges and avenues for the execution and sharing of bioimage analysis workflows. These challenges are to reproducibly run workflows in remote environments, in particular when their components come from different software packages, but also to document them and link their parameters and results by following the FAIR principles (Findable, Accessible, Interoperable, Reusable) to foster open and reproducible science. In this opinion paper, we focus on giving some directions to the reader to tackle these challenges and navigate through this complex ecosystem, in order to find and use workflows, and to compare workflows addressing the same problem. We also discuss tools to run workflows in the cloud and on High Performance Computing resources, and suggest ways to make these workflows FAIR.

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines.

BACKGROUND: The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems simplify the assembly of such tools into pipelines, assist with job automation, and aid reproducibility of analyses. Many contemporary workflow tools are specialized or not designed for highly complex workflows, such as with nested loops, dynamic scheduling, and parametrization, which is common in, e.g., machine learning. FINDINGS: SciPipe is a workflow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics, and other fields. SciPipe helps in particular with workflow constructs common in machine learning, such as extensive branching, parameter sweeps, and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on flow-based programming principles to support agile development of workflows based on a library of self-contained, reusable components. It supports running subsets of workflows for improved iterative development and provides a data-centric audit logging feature that saves a full audit trace for every output file of a workflow, which can be converted to other formats such as HTML, TeX, and PDF on demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline. CONCLUSIONS: SciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine learning, through a flexible application programming interface suitable for scientists used to programming or scripting.

Construção de uma biblioteca de módulos para utilização na composição de workflows para filogenia / Construction of a library of modules for use in the composition of workflows for phylogeny

Com o advento da era pós-genômica, ocorreu uma explosão de informações onde inúmeras descobertas geraram grande quantidade de dados biológicos, que para serem analisados, necessitavam da cooperação de várias áreas de conhecimento. Inicialmente, as atividades de análises destes dados são suportadas por programas que constituem um fluxo de trabalho, baseado em scripts, que normalmente são executados por linha de comando, obrigando os seus usuários a terem domínio de algoritmos e lógica de programação. Tais scripts auxiliam muito na entrada, processamento e resultado final da análise, mas ainda apresentam dificuldades em interferir, coletar e armazenar dados ao longo de sua execução. Além disso, dependendo da especificidade do script, o seu uso pode ser muito complexo, em função da dificuldade da implementação, manutenção e reuso. Também, neste tipo de ambiente, o registro de execução das atividades do fluxo, da origem dos dados utilizados e das transformações aplicadas aos dados, geralmente, não são mantidos. Para tanto, tem havido o crescente uso de workflows científicos na execução e condução de experimentos científicos. Os workflows científicos pressupõem a resolução de problemas científicos através das técnicas de composição do fluxo de atividades, onde os passos normalmente são compostos por programas de bioinformática que recebem, processam e geram um conjunto de dados que podem ser repassados aos demais passos do workflow. Toda a estrutura de desenvolvimento e execução desses workflows é apoiada por sistemas específicos, conhecidos como Sistemas de Gerenciamento de Workflows Científicos (SGWfC), que possuem seus próprios mecanismos de gerência e linguagem. Considerando as vantagens de uso dos SGWfC no cenário da Bioinformática, este trabalho apresenta o workflow científico para reconstrução filogenética denominado PHYLO...

With the advent of post-genomic era, there was an explosion of information where many discoveries have generated large amounts of biological data, which, to be analyzed, needed the cooperation of various fields of knowledge. Initially, the to industrial activities of analysis of these data are supported by programs that constitute the workflow, based on scripts, that normally run from command line, forcing users to algorithms and programming logic. Such scripts help much the input, processing and outcome of the analysis, but still present difficulties for usersto interfere, collect and store data throughout their implementation. Also, according to the specific use of the script, it can be very complex, depending on the difficultyof implementation, maintenance and reuse. Also, in this type of environment, the registration of the execution of the activities of the flow, the source of data use d and the transformations applied to the data are generally not retained. For these, there has been the growing use of scientific workflows for the implementation and execution of scientific experiments. Scientific workflows assume scientificproblems solving through techniques of composition of the flow of activities, where the steps are usually composed of bioinformatics programs that receive, process and generate a data set that can be passed on to other steps of the workflow. The structure of development and implementation of these workflows is supported by specific systems, known as Scientific Workflows Management Systems (SGWfC), which have their own management mechanisms and language. Considering theadvantages of using the scenario SWfMS in the scientific bioinformatics, this work presents the scientific workflow PHYLO for phylogenetic reconstruction...