The aim of the sbv IMPROVER challenge “Markers of Exposure Response Identification” was to verify that robust and sparse human-specific or species-independent gene signatures predictive of smoking exposure or cessation status can be extracted from whole blood gene expression data from human, or human and rodent. The participants were asked to:
- develop inductive models, i.e., the signature model can be applied to a single new sample without retraining) rather than transductive models, i.e., training and test set processed together and used to retrain models prior to classification prediction, including identified gene signatures;
- to classify subjects as smoker versus non-current smoker;
- to discriminate non-current smoker as former smoker versus never smoker.
BACKGROUND ↑
Humans are constantly exposed to individual or mixtures of chemicals, e.g., cigarette smoke, pollutants, pesticides, drugs, that may trigger molecular changes, e.g., gene expression, in their cells. The identification of specific markers of response to those chemicals is important to assess the exposure status of a subject. A subset of exogenous chemicals, chemical-derived metabolites, and endogenous molecules produced by exposed organs, e.g., lung, gut, can pass into the blood stream and induce molecular changes in blood cells, which for a subset can constitute a specific exposure response fingerprint or signature discriminating exposed vs. non-exposed subjects, and also formerly exposed vs never exposed subjects. Whole blood is an easily accessible matrix, but is a complex biofluid/tissue to analyze because of the different cell sub-populations that it contains.
Systems toxicology or 21st century toxicology aims to create detailed understanding of the mechanisms by which biological systems respond to toxicants, so that this understanding can be leveraged to assess the risk of chemicals, drugs, and consumer products.
Steps that define the systems toxicology maturity, from biological network models to dynamic adverse outcome pathway models.Systems toxicology aims to extrapolate short-term observations to long-term outcomes and translate the potential risks identified from experimental systems to humans. Adapted from Figure 2 in (Sturla et al., 2014).
Systems toxicology
Systems toxicology enabling technologies and measurementsIntegrating classical toxicology with quantitative analysis of the molecular and functional changes induced by toxicants, systems toxicology relies on the latest technological developments in both experimental and computational sciences.
The U.S. EPA has commissioned the National Research Council to develop a vision for toxicity testing in the 21st century (Tox 21-c) to base the new toxicology primarily on pathways of toxicity (PoT).
The report by the U.S. National Research Council (NRC) envisioned a shift away from traditional toxicity testing and toward a focused effort to explore and understand the signaling pathways perturbed by biologically active substances or their metabolites that have the potential to cause adverse health effects in humans:
- to achieve testing of broad coverage of chemicals, mixtures, outcomes and life stages;
- to significantly increase human relevance;
- to reduce the cost and time required to conduct chemical safety assessments;
- to reduce and potentially eliminate high-dose animal testing.
The identification of these toxicity pathways is imperative in order to understand the mode of action (MOA) of a given stimulus and for grouping together different stimuli based on the toxicity pathways they perturb.
The first component of the vision focuses on pathway identification, which is preferably derived from studies performed in human cells or cell lines using omics assays. The second component of the vision involves targeted testing of the identified pathways in whole animals and clinical samples to further explain toxicity pathway data. This two-component toxicity-testing paradigm, combined with chemical characterization and dose–response extrapolation, delivers a much broader understanding of the potential toxicity associated with a biologically active substance.
Systems biology plays an important role in this paradigm, consolidating large amounts of information that can be probed to reveal key cellular pathways perturbed by various stimuli.
Microarray-based phenotype prediction
Microarray-based technologies have long been the most widely used platforms for measuring whole-genome gene expression levels. They allow to gain biological insights in various experimental conditions and have been used extensively to develop classification models that include the identification of gene signatures predictive of disease phenotypes, tumor subtyping, adverse drug response, and treatment outcome.
In many instances, models are trained on a subset of a dataset while the other subset of the same dataset is used for prediction. This process can lead to model overfitting with poor classification performance when the model is applied on new independent datasets, indicative of limited generalization of the model.
The Diagnostic Signature Challenge (DSC) organized in the sbv IMPROVER framework was designed to assess to what extent models trained on transcriptomics data available in public repositories could predict disease phenotypes of individual subjects using totally new unrelated datasets.
The challenge additionally evaluated whether some computational approaches performed better than others in this task within and across diseases. One interesting aspect was that most of the classification models developed were transductive (e.g., training and test set processed together and used to retrain models prior to classification prediction) rather than inductive (e.g., the signature model can be applied to a single new sample without retraining), which may be problematic when a single sample needs to be classified. The Systox computational challenge proposed a new classification problem that was constrained to this latter aspect.
Process to extract gene signatures
Application to toxicogenomics
The exposure to cigarette smoke is a major risk factor for the development of various diseases, e.g., cardiovascular and lung diseases. Cigarette smoke contains thousands of constituents. Some of those constituents passing into the blood circulation elicit systemic effects. Therefore, changes in gene expression in circulating peripheral blood cells may be associated with several systemic immune and inflammatory-related disorders.
Smoking cessation has been shown to revert some cigarette smoke-induced functional and molecular changes back to normal or to intermediate levels that are dependent on the subject’s smoking history, e.g., smoking duration, consumption, and cessation period. The identification of specific markers of response to smoking or cessation in whole blood cells is important to monitor the exposure status of an individual subject. Therefore, blood samples collected in independent clinical studies that included conventional cigarette smokers, former smokers, and never smokers have been profiled for gene expression and the data are provided to the participants for the challenge as training and test sets.
Heating rather than burning tobacco markedly decreases the amount of harmful constituents in the aerosol (see the study page). Investigations on the biological impact of heat-not-burn technology-based RRPs* have shown significant reduced exposure effects related to lung inflammation and oxidative stress responses as well as vascular functions (e.g., transmigration, chemotaxis) compared with a conventional cigarette.
In addition to conventional cigarette or cessation treatment groups, experimental study designs have included switching to RRP* (subject smoking a conventional cigarette who switched to an RRP*) treatment groups to assess the biological impact following exposure to RRP* in those groups and to determine whether the level of perturbation is close to the level observed after smoking or cessation. Reduced exposure studies are designed to help us understand if the exposure to harmful and potentially harmful constituents (HPHCs) is reduced in adult smokers who use RRPs* as compared to cigarettes. Blood samples collected in clinical studies conducted with a RRP*, Tobacco Heating System (THS) 2.2, have been profiled for gene expression and provided to participants for verification in the context of this sbv IMPROVER computational challenge.
The challenge ↑
Aim
The Markers of Exposure Response Identification challenge aimed to verify that robust and sparse human-specific or species-independent gene signatures predictive of smoking or cessation status can be extracted from whole blood gene expression data from human, or human and rodent.
SC1: Human blood gene signature as exposure response marker
SC1 aimed to verify that robust and sparse (maximum of 40 genes) human-specific gene signatures can be extracted from whole blood gene expression data to predict smoking exposure (smoker vs. non-current smoker) or cessation (former smoker vs. never smoker) status in human.
SC2: Species-translatable blood gene signature as exposure response marker
SC2 aimed to verify that robust and sparse (maximum of 40 genes) species-independent gene signatures and models can be obtained from human and mouse whole blood gene expression data to predict smoking exposure (smoker vs. non-current smoker) or cessation (former smoker vs. never smoker) status in both species.
Challenge overview
Participants developed inductive, i.e., the signature model can be applied to a single new sample without retraining, rather than transductive (e.g., training and test set processed together and used to retrain models prior to classification prediction) signature models to classify subjects as smokers versus non-current smokers, and then former smokers versus never smokers. For these 2-class problem predictions, participants were provided with training datasets, and had the freedom to use other publicly available gene expression dataset(s). Trained signature models were to be applied directly to predict the class of totally independent and unseen blood gene expression samples (test set), also including samples from subjects who have been exposed to Reduced-Risk Products* (RRP) or who switched to RRP* after smoking conventional cigarettes (verification set, see details below).
SC1: Human blood gene signature as exposure response marker
The scientific questions that were addressed in this sub-challenge were:
- Are gene expression changes in blood sufficiently informative to extract gene signatures predictive of smoking exposure or cessation status in human?
- How do the human clinical samples found in the verification set, classify?
Participants were requested to develop inductive rather than transductive signature models to predict the sample class. Human-specific signature models were developed:
- to predict smoking exposure status discriminating smoker vs. never smoker;
- to predict cessation status discriminating former smoker vs. never smoker in human.
The gene signature should have been sparse, with a maximum of 40 genes.
SC2: Species-translatable blood gene signature as exposure response marker
The scientific questions that was addressed in this sub-challenge was:
- Are whole blood gene-expression-changes in human and mouse, sufficiently informative to define unique signature models that can be applied directly without retraining on both human and mouse samples to predict smoking exposure (smoker vs non-current smoker) or cessation (former smoker vs never smoker) status? How do the samples from the verification set classify?
Participants were requested to develop inductive rather than transductive signature models to predict the sample class (for details see the “Background: Microarray-based phenotype prediction” section). Species-independent signature model(s) were to be developed:
- to predict smoking exposure status discriminating smoker vs never smoker;
- to predict cessation status discriminating former smoker vs never smoker.
After training, unique species-independent signature model/classifier(s) were to be applied directly without retraining on both species data (test and verification sets) to predict the sample class. The gene signature must be sparse with a maximum of 40 genes.
The participants were requested to proceed with class predictions stepwise. Participants have had the freedom to use two separate models for 2-class prediction for each step, or directly a 3-class prediction model.
Data
SC1: Human blood gene signature as exposure response marker
Human blood gene expression datasets from two independent clinical studies are provided for training and testing. The test dataset included additional samples used only for verification purposes, and not considered for scoring. The human blood samples were obtained from our clinical studies or a banked repository:
- Human train dataset: The Queen Ann Street Medical Center (QASMC) clinical case–control study was conducted at The Heart and Lung Centre (London, UK), according to Good Clinical Practices.
- Human test dataset: Blood samples were obtained from a banked repository (BioServe Biotechnologies Ltd., Beltsville, MD, USA) based on well-defined inclusion criteria, and are referenced as BLD-SMK-01.
- Human verification dataset: a series of reduced exposure studies comparing our RRP, THS 2.2 (a heat-not-burn technology also called Platform 1) with conventional cigarettes and cessation (for more details visit www.pmiscience.com). Two are five-day confinement studies conducted in Europe (more details about the study description: ZRHR-REXC-03-EU) and Japan (more details about the study description: ZRHR-REXC-04-JP).
In addition to the Informed Consent Form (ICF) for the participation in these studies, subjects were provided with information and asked for their consent to collect blood samples for bio-banking for transcriptomics profiling. The blood sampling for transcriptomics and the data related to these samples were anonymized. Anonymized data and samples were initially single or double coded where the link between the subjects’ identifiers and the unique code(s) was subsequently deleted.
A description of the composition of the datasets in SC1
To learn more about the datasets, please visit the dedicated study page.
SC2: Species-translatable blood gene signature as exposure response marker
Human and mouse blood gene expression datasets from independent studies were provided for training and testing. The test dataset included additional samples used for verification purposes only - these were not considered for scoring.
The blood samples were obtained from our clinical studies or a banked repository (human samples), and from in vivo mouse studies (mouse samples):
- Human training dataset: The Queen Ann Street Medical Center (QASMC) clinical case–control study was conducted at The Heart and Lung Centre (London, UK), according to Good Clinical Practices.
- Human test dataset: Blood samples were obtained from a banked repository (BioServe Biotechnologies Ltd., Beltsville, MD, USA) based on well-defined inclusion criteria, and are referenced as BLD-SMK-01.
- Mouse training dataset: A 7-month cigarette smoke inhalation study was conducted with C57BL/6 mice. The study design included 5 different groups, however, only 3 groups corresponding to 3R4F-exposed mice (3R4F), mice exposed to air after 2 month-3R4F exposure (Cessation), and mice continuously exposed to air (Sham) were provided for training.
- Mouse test and verification dataset: A 8-month cigarette smoke inhalation study was conducted with Apoe-/- mice. The study design includes 5 different groups cooresponding to 3R4F-exposed mice (3R4F), mice exposed to air after 2month-3R4F exposure (Cessation), mice exposed to RRP (THS2.2), mice exposed to RRP (THS2.2) after 2month-3R4F exposure (Switch) and mice continuously exposed to air (Sham) will be provided for testing and verification as shown in the schema below.
A description of the composition of the datasets. Note: the 3R4F (conventional reference cigarette), cessation, and sham groups in mouse studies correspond to smokers, former smokers, and never smokers in human studies, respectively.
To learn more about the datasets, please visit the dedicated study page.
Rules and awards
You can view the rules that applied for the challenge here.
Challenge participants ↑
Scoring and ranking ↑
Scoring
Gold standard
For each sub-challenge, the submissions were scored by comparing the predictions to the “gold standard” that remained unseen by the participants. The gold standard corresponds to the true class/group labels of the samples present in the test and verification sets. The experimental data used to generate the gold standard were processed and normalized the same way as those corresponding to the training and verification sets provided to the participants.
Scoring methodology
To establish a fair and meaningful score that is not biased by any particular performance measure, different metrics were used and aggregated. The scoring methods and metrics were selected by the Scoring Review Panel based on the application of scientific principles before the opening of the challenge. The scoring methods and metrics have only disclosed once the scoring was completed in accordance with the Challenge Rules. Each sub-challenge was self-contained and scored independently.
Tie resolution
If several teams obtained the same score, the Scoring Review Panel performed a scientific review of the available information to distinguish the submissions as described further in the Challenge Rules. In the case of the tie persisting, the incentives were allocated according to the Challenge Rules.
Scorers and Scoring Review Panel
A team of researchers from Philip Morris R&D in Neuchâtel (Switzerland) established a scoring methodology and performed the scoring on the blinded submissions under the review of an independent Scoring Review Panel. composed of the following experts in the field of systems biology:
- Prof. Dr. Leonidas Alexopoulos, National Technical University of Athens
- Prof. Dr. Rudiyanto Gunawan, ETH Zurich
- Dr. Alberto de la Fuente, Leibniz Institute for Farm Animal Biology
Blinded scoring
Submissions were anonymized before scoring, so that both the scorer and the Scoring Review Panel do not have access to the identity of the participating teams or the members of the teams. To help us maintain this, submissions must not include any information regarding the identity or affiliations of the team or the members of the team.
Submissions and significance
For each sub-challenge, a minimum of five submissions were required. One of the submissions had to be statistically significant in at least one metric, at a level of significance given by a P-value of 0.05. If these requirements are not met for a particular sub-challenge, the challenge organizers retain the right not to declare a sub-challenge best performer in accordance with the Challenge Rules.
Ranking
SC1: Human blood gene signature as exposure response marker
| Team | Final rank | Average rank | AUPR score - S vs NCS | MCC score - S vs NCS | AUPR score - FS vs NS | MCC score - FS vs NS |
|---|---|---|---|---|---|---|
264 | 1 | 1.25 | 0.96 | 0.90 | 0.58 | 0.07 |
225 | 2 | 2.5 | 0.97 | 0.77 | 0.50 | 0.02 |
259 | 3 | 3.75 | 0.95 | 0.79 | 0.47 | -0.02 |
221 | 5 | 5 | 0.79 | 0.78 | 0.48 | -0.01 |
247 | 5 | 5 | 0.90 | 0.67 | 0.45 | 0.04 |
269 | 6 | 6.25 | 0.91 | 0.71 | 0.42 | -0.11 |
283 | 7 | 7 | 0.86 | 0.63 | 0.48 | -0.11 |
250 | 8 | 7.25 | 0.85 | 0.67 | 0.43 | -0.07 |
222 | 9 | 8.625 | 0.87 | 0.67 | 0.35 | -0.32 |
290 | 10 | 8.875 | 0.83 | 0.59 | 0.46 | -0.30 |
257 | 12 | 11.25 | 0.23 | -0.05 | 0.29 | -0.80 |
215 | 12 | 11.25 | 0.21 | -0.50 | 0.30 | -0.79 |
SC2: Species-translatable blood gene signature as exposure response marker
| Team | Final rank | Average rank | AUPR score - S vs NCS | MCC score - S vs NCS | AUPR score - FS vs NS | MCC score - FS vs NS |
|---|---|---|---|---|---|---|
219 | 1 | 1 | 0.93 | 0.78 | 0.45 | 0.04 |
250 | 2 | 2.5 | 0.79 | 0.65 | 0.36 | -0.17 |
264 | 2 | 2.5 | 0.79 | 0.59 | 0.41 | -0.01 |
221 | 4 | 4.75 | 0.55 | 0.54 | 0.34 | -0.39 |
225 | 5 | 4.875 | 0.75 | 0.31 | 0.30 | -0.45 |
247 | 6 | 5.375 | 0.63 | 0.20 | 0.29 | -0.52 |
Challenge symposium ↑
The sbv IMPROVER Symposium 2016, concluding the STC, was held at the Walt Disney World Swan & Dolphin Resort on July 11th 2016. The event included lectures and presentations by the best performers in the challenge.
You can view the slides presented to introduce the challenge here and a summary of the results and lessons learned presented at the symposium here.
Media library ↑
Scientific publications
- Belcastro, V. et al. The sbv IMPROVER Systems Toxicology Computational Challenge: Identification of Human and Species-Independent Blood Response Markers as Predictors of Smoking Exposure and Cessation Status. Comput Toxicol 5, 38–51 (2018).
- Poussin, C. et al. Crowd-Sourced Verification of Computational Methods and Data in Systems Toxicology: A Case Study with a Heat-Not-Burn Candidate Modified Risk Tobacco Product. Chem. Res. Toxicol. 30 (4), 934–945 (2017).
Challenge documents
The challenge in the news
| Front Line Genomics | Sep 2016 | Could transcriptomics predict exposure to toxicants? An interview with Carine Poussin PhD, Philip Morris International on the sbv IMPROVER Systems Toxicology Computational Challenge |  | |
| Clinical Omics | Aug 2016 | From Transcriptomics to Predictive Toxicology - The Systems Toxicology Computational Challenge |  | |
| American Laboratory | Jul 2016 | Systems Toxicology Computational Challenge Impacts Diagnostics, Personalized Medicine, Toxicological Risk Assessment | | |
| BioITWorld | Jul 2016 | Systems Toxicology Computational Challenge Results Announced | | |
| GenomeWeb | Dec 2015 | SBV IMPROVER Project Launches New Systems Toxicology Challenge | | |
| BioITWorld | Dec 2015 | sbv IMPROVER Launches Fourth Systems Toxicology Challenge | | |
Tutorials and webinars
Flyers and posters
Click on the image to open the flyer in pdf.
Click onthe image to open the poster in pdf.
Testimonials ↑
What they say about the challenge
*Reduced Risk Products (“RRPs”) is the term we use to refer to products with the potential to reduce individual risk and population harm in comparison to smoking cigarettes. PMI’s RRPs are in various stages of development and commercialization, and we are conducting extensive and rigorous scientific studies to determine whether we can support claims for such products of reduced exposure to harmful and potentially harmful constituents in smoke, and ultimately claims of reduced disease risk, when compared to smoking cigarettes. Before making any such claims, we will rigorously evaluate the full set of data from the relevant scientific studies to determine whether they substantiate reduced exposure or risk. Any such claims may also be subject to government review and authorization, as is the case in the USA today.