sbv IMPROVER stands for Systems Biology Verification combined with Industrial Methodology for Process Verification in Research. This approach aims to provide a measure of quality control of industrial research and development by verifying the methods used.
So far, the scope of sbv IMPROVER mostly revolved around the verification of methods and concepts in systems biology used at PMI R&D. However, such an initiative could be extended to the verification of research processes in other industries such as pharmaceuticals, biotechnology, nutrition, and environmental safety, to name a few.
Leveraging the wisdom of crowds ↑
Crowdsourcing is characterized by:
- contribution by many participants
- participants produce independent methods and submit different solutions which tackle different aspects of a complex problem
- the combination of solutions often outperforms the best performing submissions, a phenomenon often referred to as the “Wisdom of Crowds”
Self-Assessment of Computational Methods often leads to Bias
In many scientific publications reporting new methods, the authors rank their method using self or independent assessment. This may lead to:
- selective reporting of performance
- choice of only one, best metric
- overfitting
- parameter tinkering
Proposed solutions for an unbiased assessment:
- conduct scientific challenges
- make predictions on unseen data
- apply crowd sourcing
sbv IMPROVER concept ↑
sbv IMPROVER stands for Systems Biology Verification combined with Industrial Methodology for Process Verification in Research. This approach aims to provide a measure of quality control of industrial research and development by verifying the methods used.
The sbv IMPROVER project initially developed by PMI Research and Development and IBM Research between 2011 and 2013 and is since an effort led and funded by PMI.
The approach is different from other scientific crowdsourcing alternatives as it focused on the verification of processes in an industrial context, and not just on basic questions regarding science. The sbv IMPROVER approach allows an organization to benchmark its methods and industrial processes.
The sbv IMPROVER project comes as a complementary methods to the established peer review system.
The sbv IMPROVER project created strong methodological foundations and demonstrated that crowdsourcing is a viable strategy to verify scientific methods and concepts in an industrial context. Although the scope of sbv IMPROVER was until now the verification of methods and concepts in systems biology research at PMI, it could easily be extended to the verification of research processes in other industries such as pharmaceuticals, biotechnology, nutrition, and environmental safety.
A complex research program is typically built upon research projects (consisting of “building blocks”) that synergistically support each other towards a final goal. A building block is a standalone research process of a complex workflow. It has a defined input that results in a defined output.
Challenges ↑
A challenge is a scientific problem presented to the community. Some of its basic elements are:
- the need for a "Gold Standard" or a solution to the challenge
- each prediction is compared to the "Gold Standard"
- guidelines about the metrics to be used to evaluate predictions are given prior to the receipt of the predictions
The approach to challenge design should follow specific steps consisting in (1) defining the big question, (2) Narrowing the scope, (3) collecting the necessary data and "Gold Standard", (4) posing the challenge to the community with incentives and communicate about the challenge to ensure participation, (5) scoring the received prediction against the "Gold Standard", and finally (6) analyzing and presenting the results to the scientific community.
The challenges opened within the frame of the sbv IMPROVER project are briefly described below from most recent to oldest. For more information, please go to the respective challenge page.
Computational challenges
Metagenomics Diagnosis for Inflammatory Bowel Disease Challenge (MEDIC)
MEDIC aimed to investigate the diagnostic potential of metagenomics data to classify patients with Inflammatory Bowel Disease (IBD) and non-IBD subjects. The participants have attempted to classify Ulcerative Colitis (UC) and Crohn’s Disease (CD) subjects.
Microbiota Composition Prediction challenge
The first phase of the microbiomics challenge named “Microbiota composition prediction” aimed at identifying state-of-the-art computational microbiome analysis pipeline(s) that can be used as off-the-shelf solutions for scientists to best recover the composition and relative abundance of bacterial communities present in a sample.
Systems Toxicology (SysTox) Challenge
The SysTox Challenge aimed at verifying that robust and sparse human-specific and species-independent gene signatures of exposure response can be extracted in whole blood gene expression data from human and rodent to predict exposed and non-exposed group labels.
Species Translation Challenge (STC)
The Species Translation Challenge aimed at verifying that changes in phosphorylation status and gene set activation induced by cellular response to 52 different perturbations in human cells can be predicted to a certain extent given responses generated in rat cells.
Diagnostic Signature challenge (DSC)
The goal of this Challenge was to assess and verify computational approaches that classify clinical samples based on transcriptomics data.
Network verification challenges
Network Verification Challenge 3: Liver xenobiotic metabolism networks
NVC3 aimed at verifying three biological network models to ensure their relevance to liver xenobiotic metabolism.
Network Verification Challenge 2: Lung biology
NVC2 aimed at verifying fifty biological network models describing cell fate, cell stress, cell proliferation, inflammation, and tissue repair and angiogenesis to ensure their relevance to lung biology and COPD.
Network Verification Challenge 1: Lung biology
NVC1 aimed at verifying fourteen biological network models describing cell fate, cell stress, and inflammation to ensure their relevance to lung biology.
Mini-challenges and datathons
sbv IMPROVER Datathon - Japan
The datathon organized in Japan addressed how to evaluate qualitatively (on which pathways and biological processes) and quantitatively the extent of biological impact of a system to specific exposure.
sbv IMPROVER Epigenomics Challenge - Israel
The challenge organized in Israel aimed at answering the questions whether a smoke exposure signature can be extracted (i) from DNA methylation levels of DNA cis-regulatory elements (CRE) or (ii) from expression data of genes controlled by differentially methylated DNA CRE.
sbv IMPROVER Datathon - Singapore
Omics datasets and functional measurements from a 7-month inhalation toxicology study were provided to encourage scientists creating applications to analyze the datasets that could be later on included into Garuda.
Science verification ↑
Many companies and agencies perform peer reviews as part of product risk assessment. For example, risk-ranking methods are available for prioritizing food safety risks (van Asselt et al., 2012).
The methods that are generally used may be qualitative, semi-quantitative, or quantitative, but are typically based on the concept of risk being a function of the presence of a hazard and the severity of its impact on human health (National Research Council - Division on Earth and Life Studies - Board on Environmental Studies and Toxicology - Committee on Improving Risk Analysis Approaches Used by the U.S. EPA, 2009).
The SciPinion peer review process (Kirman et al., 2019) is a model for more in-depth insight and review by an independent panel of experts.
To complement the peer review of publications reporting individual studies, SciPinion LLC has been engaged by PMI to conduct an independent and anonymous scientific peer review of data related to the Tobacco Heating System (THS) 2.2, an electrically heated tobacco product, with the aim to verify that the conclusions made on datasets comparing THS 2.2 with cigarettes are appropriate.
The results have been published (Boue et al., 2019) and are summarized in two study pages in INTERVALS, where you also can learn more about the verification approach used:
References
- van Asselt, E.D. et al. Overview of available methods for Risk Based Control within the European Union. Trends in Food Science & Technology 23 (1), 51–58 (2012).
- Boue, S. et al. Toxicological assessment of Tobacco Heating System 2.2: findings from an independent peer review. Regulatory Toxicology and Pharmacology 104, 115-127 (2019).
- Kirman, C.R. et al. Science peer review for the 21st century: Assessing scientific consensus for decision-making while managing conflict of interests, reviewer and process bias. Regulatory toxicology and pharmacology: RTP 103, 73–85 (2019).
- National Research Council - Division on Earth and Life Studies - Board on Environmental Studies and Toxicology - Committee on Improving Risk Analysis Approaches Used by the U.S. EPA. Science and Decisions - Advancing Risk Assessment. (2009).
sbv IMPROVER events ↑
In order to congratulate the best performers and share the learnings of the challenges, different symposia have been organized, alongside larger conferences, or as standalone events.
To learn more about those events, please visit the respective challenge pages.
Network Verification Challenge 3: Liver xenobiotic metabolism networks
The sbv IMPROVER NVC3 workshop 2018 was successfully conducted the PMI R&D facilities in Neuchatel, Switzerland on June 27th 2018.
sbv IMPROVER Datathon - Japan
The datathon was organized in Tokyo on October 13th 2017. It included lectures and discussions aroud the topics of qualitative and quantitative assessment of biological perturbations.
sbv IMPROVER Epigenomics Challenge - Israel
Concluding the epigenomics mini-challenge, the sbv IMPROVER Epigenomics Symposium 2017 was held at the Hilton Tel Aviv Hotel on May 4th 2017.
sbv IMPROVER Datathon - Singapore
The sbv IMPROVER Datathon 2016 gathered in Singapore on 23rd and 24th of September 2016 computational biologists with the aim to turn data into knowledge and to learn from each other.
Systems Toxicology (SysTox) Challenge
The sbv IMPROVER Symposium 2016, concluding the STC, was held at the Walt Disney World Swan & Dolphin Resort on July 11th 2016. The event included lectures and presentations by the best performers in the challenge.
Network Verification Challenge 2: Lung biology
The sbv IMPROVER Jamboree 2015 was successfully conducted at the Rey Juan Carlos I Hotel, Barcelona, Spain, on 15 - 18 June 2015. The event included lectures, biological network review sessions (jamboree), and social events and networking.
Network Verification Challenge 1: Lung biology
The sbv IMPROVER Jamboree 2014 was successfully conducted at Le Montreux Palace, Montreux, Switzerland, on 18 - 20 March 2014. The event included lectures, biological network review sessions (jamboree), and social events and networking.
Species Translation Challenge (STC)
The Species Translation Challenge: Understanding the Limits of Rodent Models for Human Biology Symposium was successfully conducted at the Grand Resort Lagonissi Athens, Greece, on 29 – 31 October 2013. The event included lectures, presentations by the best performers in the challenge, and social events and networking.
Diagnostic Signature challenge (DSC)
The Diagnostic Signature Challenge: Smarter Algorithms for better Disease Detection Symposium was successfully conducted at the Omni Parker House Hotel in Boston, MA, USA, on 2 – 3 October 2012. The event included lectures, presentations by the best performers in the challenge, and social events and networking.
sbv IMPROVER Methodology
In March 2011, we organized the Systems Biology Symposium “Critical Assessment of Systems Biology: Research Verification in the Age of Collaborative-Competition” in Zurich, Switzerland. At this event, the extended team and keynote speakers presented the sbv IMPROVER approach for the first time to more than 50 scientists from academia and industry receiving a very positive reaction.
News and publications ↑
Scientific publications
Metagenomics Diagnosis for Inflammatory Bowel Disease Challenge (MEDIC)
Outcome publication in preparation.
Microbiota Composition Prediction challenge
Outcome publication in preparation.
Network Verification Challenge 3: Liver xenobiotic metabolism networks
Outcome publication in preparation submitted to journal for consideration.
Systems Toxicology (SysTox) Challenge
- Belcastro, V. et al. The sbv IMPROVER Systems Toxicology Computational Challenge: Identification of Human and Species-Independent Blood Response Markers as Predictors of Smoking Exposure and Cessation Status. Comput Toxicol 5, 38–51 (2018).
- Poussin, C. et al. Crowd-Sourced Verification of Computational Methods and Data in Systems Toxicology: A Case Study with a Heat-Not-Burn Candidate Modified Risk Tobacco Product. Chem. Res. Toxicol. 30 (4), 934–945 (2017).
Network Verification Challenge 2: Lung biology
- The sbv IMPROVER project team and challenge best performers Community-Reviewed Biological Network Models for Toxicology and Drug Discovery Applications. Gene regulation and systems biology 10, 51–66 (2016).
Network Verification Challenge 1: Lung biology
- Boue, S. et al. Causal biological network database: a comprehensive platform of causal biological network models focused on the pulmonary and vascular systems. Database: the journal of biological databases and curation 2015, bav030 (2015).
- Boue, S. et al. Enhancement of COPD biological networks using a web-based collaboration interface. F1000Research 4, 32 (2015).
- Ansari, S. et al. On crowd-verification of biological networks. Bioinformatics and biology insights 7, 307–25 (2013).
Species Translation Challenge (STC)
- Rhrissorrakrai, K. et al. Understanding the limits of animal models as predictors of human biology: lessons learned from the sbv IMPROVER Species Translation Challenge. Bioinformatics 31 (4), 471–83 (2015).
- Dayarian, A. et al. Predicting protein phosphorylation from gene expression: top methods from the IMPROVER Species Translation Challenge. Bioinformatics 31 (4), 462–470 (2015).
- Hafemeister, C. et al. Inter-species pathway perturbation prediction via data-driven detection of functional homology. Bioinformatics 31 (4), 501–508 (2015).
- Biehl, M. et al. Inter-species prediction of protein phosphorylation in the sbv IMPROVER species translation challenge. Bioinformatics 31 (4), 453–461 (2015).
- Poussin, C. et al. The species translation challenge-a systems biology perspective on human and rat bronchial epithelial cells. Scientific data 1, 140009 (2014).
Diagnostic Signature challenge (DSC)
- Tarca, A.L. et al. Strengths and limitations of microarray-based phenotype prediction: lessons learned from the IMPROVER Diagnostic Signature Challenge. Bioinformatics 29 (22), 2892–9 (2013).
- Hoeng, J. et al. sbv IMPROVER Diagnostic Signature Challenge: Preface to this special issue. Systems Biomedicine 1 (4), 193–195 (2013).
- Rhrissorrakrai, K. et al. sbv IMPROVER diagnostic signature challenge: design and results. Systems Biomedicine 1 (4), 196–207 (2013).
- Norel, R. et al. sbv IMPROVER Diagnostic Signature Challenge: scoring strategies. Systems Biomedicine 1 (4), 208–216 (2013).
- Tarca, A.L. et al. Methodological approach from the Best Overall Team in the sbv IMPROVER Diagnostic Signature Challenge. Systems Biomedicine 1 (4), 217–227 (2013).
- Lauria, M. Rank-based transcriptional signatures. Systems Biomedicine 1 (4), 228–239 (2013).
- Nandy, P. et al. Learning diagnostic signatures from microarray data using L1-regularized logistic regression. Systems Biomedicine 1 (4), 240–246 (2013).
- Zhao, C. et al. Relapsing-remitting multiple sclerosis classification using elastic net logistic regression on gene expression data. Systems Biomedicine 1 (4), 247–253 (2013).
- Cho, J.-H. et al. Kernel-based method for feature selection and disease diagnosis using transcriptomics data. Systems Biomedicine 1 (4), 254–260 (2013).
- Song, L., Horvath, S. Predicting COPD status with a random generalized linear model. Systems Biomedicine 1 (4), 261–267 (2013).
- Ben-Hamo, R. et al. Classification of lung adenocarcinoma and squamous cell carcinoma samples based on their gene expression profile in the sbv IMPROVER Diagnostic Signature Challenge. Systems Biomedicine 1 (4), 268–277 (2013).
- Tian, S., Suárez-Fariñas, M. Hierarchical-TGDR. Systems Biomedicine 1 (4), 278–287 (2013).
sbv IMPROVER Methodology
- Meyer, P. et al. Verification of systems biology research in the age of collaborative competition. Nature biotechnology 29 (9), 811–5 (2011).
- Meyer, P. et al. Industrial methodology for process verification in research (IMPROVER): toward systems biology verification. Bioinformatics 28 (9), 1193–201 (2012).
sbv IMPROVER in the news
The pdf below gives an overview of the news published on each topic/challenge. For more detail on the news specific to challenges, please refer to the respective challenge pages.
| Technology Networks | Jan 2017 | Verification of Scientific Methods and Results |  |  |
| BioSpectrum | Nov 2015 | SBV IMPROVER methodology could in theory be applied to any disease area | | |
| Asia Biotech | Dec 2013 | Challenged TO Improve | | |
sbv IMPROVER participants ↑
The sbv community steadily grew over years. See below the latest map of participants involved in the respective challenges.
Testimonials ↑
What they say about sbv IMPROVER
