Intro

sbv IMPROVER stands for Systems Biology Verification combined with Industrial Methodology for Process Verification in Research. This approach aims to provide a measure of quality control of industrial research and development by verifying the methods used.
So far, the scope of sbv IMPROVER mostly revolved around the verification of methods and concepts in systems biology used at PMI R&D. However, such an initiative could be extended to the verification of research processes in other industries such as pharmaceuticals, biotechnology, nutrition, and environmental safety, to name a few.

Wisdom of the crowds
sbv IMPROVER concept
Editorial_board
Verification
sbv IMPROVER events
Research_tools
Participants
sbv_IMPROVER

Leveraging the wisdom of crowds

Crowdsourcing is characterized by:

  • contribution by many participants
  • participants produce independent methods and submit different solutions which tackle different aspects of a complex problem
  • the combination of solutions often outperforms the best performing submissions, a phenomenon often referred to as the “Wisdom of Crowds”

Self-Assessment of Computational Methods often leads to Bias

In many scientific publications reporting new methods, the authors rank their method using self or independent assessment. This may lead to:

  • selective reporting of performance
  • choice of only one, best metric
  • overfitting
  • parameter tinkering

Proposed solutions for an unbiased assessment:

  • conduct scientific challenges
  • make predictions on unseen data
  • apply crowd sourcing

sbv IMPROVER concept

sbv IMPROVER stands for Systems Biology Verification combined with Industrial Methodology for Process Verification in Research. This approach aims to provide a measure of quality control of industrial research and development by verifying the methods used.

The sbv IMPROVER project initially developed by PMI Research and Development and IBM Research between 2011 and 2013 and is since an effort led and funded by PMI.

The approach is different from other scientific crowdsourcing alternatives as it focused on the verification of processes in an industrial context, and not just on basic questions regarding science. The sbv IMPROVER approach allows an organization to benchmark its methods and industrial processes.

The sbv IMPROVER project comes as a complementary methods to the established peer review system.

The sbv IMPROVER project created strong methodological foundations and demonstrated that crowdsourcing is a viable strategy to verify scientific methods and concepts in an industrial context. Although the scope of sbv IMPROVER was until now the verification of methods and concepts in systems biology research at PMI, it could easily be extended to the verification of research processes in other industries such as pharmaceuticals, biotechnology, nutrition, and environmental safety.

sbv IMPROVER complements the peer review system
Research_pipeline_concept

A complex research program is typically built upon research projects (consisting of “building blocks”) that synergistically support each other towards a final goal. A building block is a standalone research process of a complex workflow. It has a defined input that results in a defined output.

Challenges

A challenge is a scientific problem presented to the community. Some of its basic elements are:

  • the need for a "Gold Standard" or a solution to the challenge
  • each prediction is compared to the "Gold Standard"
  • guidelines about the metrics to be used to evaluate predictions are given prior to the receipt of the predictions

The approach to challenge design should follow specific steps consisting in (1) defining the big question, (2) Narrowing the scope, (3) collecting the necessary data and "Gold Standard", (4) posing the challenge to the community with incentives and communicate about the challenge to ensure participation, (5) scoring the received prediction against the "Gold Standard", and finally (6) analyzing and presenting the results to the scientific community.

The challenges opened within the frame of the sbv IMPROVER project are briefly described below from most recent to oldest. For more information, please go to the respective challenge page.

Make-up-of-a-challenge

Computational challenges

2019-20
MEDIC

Metagenomics Diagnosis for Inflammatory Bowel Disease Challenge (MEDIC)

MEDIC aimed to investigate the diagnostic potential of metagenomics data to classify patients with Inflammatory Bowel Disease (IBD) and non-IBD subjects. The participants have attempted to classify Ulcerative Colitis (UC) and Crohn’s Disease (CD) subjects.

Learn more

2018
Microbiomics challenge

Microbiota Composition Prediction challenge

The first phase of the microbiomics challenge named “Microbiota composition prediction” aimed at identifying state-of-the-art computational microbiome analysis pipeline(s) that can be used as off-the-shelf solutions for scientists to best recover the composition and relative abundance of bacterial communities present in a sample.

Learn more

2014-15
Systems toicology challenge

Systems Toxicology (SysTox) Challenge

The SysTox Challenge aimed at verifying that robust and sparse human-specific and species-independent gene signatures of exposure response can be extracted in whole blood gene expression data from human and rodent to predict exposed and non-exposed group labels.

Learn more

2013
Species translation challenge

Species Translation Challenge (STC)

The Species Translation Challenge aimed at verifying that changes in phosphorylation status and gene set activation induced by cellular response to 52 different perturbations in human cells can be predicted to a certain extent given responses generated in rat cells.

Learn more

2012
Diagnostic Signature Challenge

Diagnostic Signature challenge (DSC)

The goal of this Challenge was to assess and verify computational approaches that classify clinical samples based on transcriptomics data.

Learn more

Network verification challenges

2018-19
NVC3 liver xenobiotic metabolism

Network Verification Challenge 3: Liver xenobiotic metabolism networks

NVC3 aimed at verifying three biological network models to ensure their relevance to liver xenobiotic metabolism.

Learn more

2014-15
NVC2 - lung biology

Network Verification Challenge 2: Lung biology

NVC2 aimed at verifying fifty biological network models describing cell fate, cell stress, cell proliferation, inflammation, and tissue repair and angiogenesis to ensure their relevance to lung biology and COPD.

Learn more

2013-14
NVC1 lung biology

Network Verification Challenge 1: Lung biology

NVC1 aimed at verifying fourteen biological network models describing cell fate, cell stress, and inflammation to ensure their relevance to lung biology.

Learn more

Mini-challenges and datathons

2017
Japan datathon

sbv IMPROVER Datathon - Japan

The datathon organized in Japan addressed how to evaluate qualitatively (on which pathways and biological processes) and quantitatively the extent of biological impact of a system to specific exposure.

Learn more

2017
Istrael mini-challenge

sbv IMPROVER Epigenomics Challenge - Israel

The challenge organized in Israel aimed at answering the questions whether a smoke exposure signature can be extracted (i) from DNA methylation levels of DNA cis-regulatory elements (CRE) or (ii) from expression data of genes controlled by differentially methylated DNA CRE.

Learn more

2016
Singapore datathon

sbv IMPROVER Datathon - Singapore

Omics datasets and functional measurements from a 7-month inhalation toxicology study were provided to encourage scientists creating applications to analyze the datasets that could be later on included into Garuda.

Learn more

Science verification

Many companies and agencies perform peer reviews as part of product risk assessment. For example, risk-ranking methods are available for prioritizing food safety risks (van Asselt et al., 2012).

The methods that are generally used may be qualitative, semi-quantitative, or quantitative, but are typically based on the concept of risk being a function of the presence of a hazard and the severity of its impact on human health (National Research Council - Division on Earth and Life Studies - Board on Environmental Studies and Toxicology - Committee on Improving Risk Analysis Approaches Used by the U.S. EPA, 2009).

The SciPinion peer review process (Kirman et al., 2019) is a model for more in-depth insight and review by an independent panel of experts.

To complement the peer review of publications reporting individual studies, SciPinion LLC has been engaged by PMI to conduct an independent and anonymous scientific peer review of data related to the Tobacco Heating System (THS) 2.2, an electrically heated tobacco product, with the aim to verify that the conclusions made on datasets comparing THS 2.2 with cigarettes are appropriate.

The results have been published (Boue et al., 2019) and are summarized in two study pages in INTERVALS, where you also can learn more about the verification approach used:

Verification
References

sbv IMPROVER events

In order to congratulate the best performers and share the learnings of the challenges, different symposia have been organized, alongside larger conferences, or as standalone events.

To learn more about those events, please visit the respective challenge pages.


sbv_events_map
2018
NVC3 liver xenobiotic metabolism

Network Verification Challenge 3: Liver xenobiotic metabolism networks

The sbv IMPROVER NVC3 workshop 2018 was successfully conducted the PMI R&D facilities in Neuchatel, Switzerland on June 27th 2018.

2017
Japan datathon

sbv IMPROVER Datathon - Japan

The datathon was organized in Tokyo on October 13th 2017. It included lectures and discussions aroud the topics of qualitative and quantitative assessment of biological perturbations.

2017
Istrael mini-challenge

sbv IMPROVER Epigenomics Challenge - Israel

Concluding the epigenomics mini-challenge, the sbv IMPROVER Epigenomics Symposium 2017 was held at the Hilton Tel Aviv Hotel on May 4th 2017.

2016
Singapore datathon

sbv IMPROVER Datathon - Singapore

The sbv IMPROVER Datathon 2016 gathered in Singapore on 23rd and 24th of September 2016 computational biologists with the aim to turn data into knowledge and to learn from each other.

2016
Systems toicology challenge

Systems Toxicology (SysTox) Challenge

The sbv IMPROVER Symposium 2016, concluding the STC, was held at the Walt Disney World Swan & Dolphin Resort on July 11th 2016. The event included lectures and presentations by the best performers in the challenge.

2015
NVC2 - lung biology

Network Verification Challenge 2: Lung biology

The sbv IMPROVER Jamboree 2015 was successfully conducted at the Rey Juan Carlos I Hotel, Barcelona, Spain, on 15 - 18 June 2015. The event included lectures, biological network review sessions (jamboree), and social events and networking.

2014
NVC1 lung biology

Network Verification Challenge 1: Lung biology

The sbv IMPROVER Jamboree 2014 was successfully conducted at Le Montreux Palace, Montreux, Switzerland, on 18 - 20 March 2014. The event included lectures, biological network review sessions (jamboree), and social events and networking.

2013
Species translation challenge

Species Translation Challenge (STC)

The Species Translation Challenge: Understanding the Limits of Rodent Models for Human Biology Symposium was successfully conducted at the Grand Resort Lagonissi Athens, Greece, on 29 – 31 October 2013. The event included lectures, presentations by the best performers in the challenge, and social events and networking.

2012
Diagnostic Signature Challenge

Diagnostic Signature challenge (DSC)

The Diagnostic Signature Challenge: Smarter Algorithms for better Disease Detection Symposium was successfully conducted at the Omni Parker House Hotel in Boston, MA, USA, on 2 – 3 October 2012. The event included lectures, presentations by the best performers in the challenge, and social events and networking.

2011
sbv IMPROVER logo

sbv IMPROVER Methodology

In March 2011, we organized the Systems Biology Symposium “Critical Assessment of Systems Biology: Research Verification in the Age of Collaborative-Competition” in Zurich, Switzerland. At this event, the extended team and keynote speakers presented the sbv IMPROVER approach for the first time to more than 50 scientists from academia and industry receiving a very positive reaction.

News and publications

Scientific publications

In prep
MEDIC

Metagenomics Diagnosis for Inflammatory Bowel Disease Challenge (MEDIC)

Outcome publication in preparation.

In prep
Microbiomics challenge

Microbiota Composition Prediction challenge

Outcome publication in preparation.

Submitted
NVC3 liver xenobiotic metabolism

Network Verification Challenge 3: Liver xenobiotic metabolism networks

Outcome publication in preparation submitted to journal for consideration.

2016
NVC2 - lung biology

Network Verification Challenge 2: Lung biology

2013-15
NVC1 lung biology

Network Verification Challenge 1: Lung biology

2013
Diagnostic Signature Challenge

Diagnostic Signature challenge (DSC)

2011-12
sbv IMPROVER logo

sbv IMPROVER Methodology

sbv IMPROVER in the news

The pdf below gives an overview of the news published on each topic/challenge. For more detail on the news specific to challenges, please refer to the respective challenge pages.


Technology NetworksJan 2017Verification of Scientific Methods and ResultsINTERVALS_Icons-wwwINTERVALS_Icons-pdf
BioSpectrumNov 2015SBV IMPROVER methodology could in theory be applied to any disease areaINTERVALS_Icons-wwwINTERVALS_Icons-pdf
Asia BiotechDec 2013Challenged TO ImproveINTERVALS_Icons-wwwINTERVALS_Icons-pdf

sbv IMPROVER participants

The sbv community steadily grew over years. See below the latest map of participants involved in the respective challenges.


sbv participation map

Testimonials

What they say about sbv IMPROVER

Quote_Hornshaw_sbv
Quote_Bhanot_sbv
Quote_Hou_sbv
Quote_Sobral_sbv
Quote_Dopazo_sbv
Quote_Quackenbush_sbv
Quote_oyyuru_sbv