Predicting breast cancer metastasis from whole-blood transcriptomic measurements:
In this exploratory work we investigate whether blood gene expression measurements predict breast cancer metastasis. Early detection of increased metastatic risk could potentially be life-saving. Our data come...
Research note
Open Access
Published: 20 May 2020
Predicting breast cancer metastasis from whole-blood transcriptomic measurements
Einar Holsbø, Vittorio Perduca, Lars Ailo Bongo, Eiliv Lund & Etienne Birmelé
BMC Research Notes volume 13, Article number: 248 (2020) Cite this article
Metricsdetails
Abstract
Objective
In this exploratory work we investigate whether blood gene expression measurements predict breast cancer metastasis. Early detection of increased metastatic risk could potentially be life-saving. Our data comes from the Norwegian Women and Cancer epidemiological cohort study. The women who contributed to these data provided a blood sample up to a year before receiving a breast cancer diagnosis. We estimate a penalized maximum likelihood logistic regression. We evaluate this in terms of calibration, concordance probability, and stability, all of which we estimate by the bootstrap.
Results
We identify a set of 108 candidate predictor genes that exhibit a fold change in average metastasized observation where there is none for the average non-metastasized observation.
Introduction
About one in ten women will at some point develop breast cancer (BC). About 25% have an aggressive cancer at the time of diagnosis, with metastatic spread. The absence or presence of metastatic spread largely determines the patient’s survival. Early detection is hence very important in terms of reducing cancer mortality. A blood sample is cheaper and less invasive than the usual node biopsy. Were we able to detect signs of metastasis or metastatic potential by a blood sample, we could conceivably start treatment earlier.
Several recent articles develop this idea of liquid biopsies [1]. A review in Cancer and Metastasis Reviews [2] lists liquid biopsies and large data analysis tools as important challenges in metastatic breast cancer research.
The Norwegian Women and Cancer (NOWAC) postgenome cohort [3] is a prospective population-based cohort that contains blood samples from 50,000 women born between 1943 and 1957. Out of these in total about 1600 BC case–control pairs (3200 blood samples) have at various times been processed to provide transcriptomic measurements in the form of mRNA abundance. These measurements combine with questionnaires, disease status from the Norwegian Cancer Registry, and death status from the Cause of Death Registry from Statistics Norway to provide a high-quality dataset. These data are used for exploration and hypothesis generation.
We examine 88 breast cancer cases from the NOWAC study. The blood samples were provided 6–358 days before BC diagnosis. We fit a penalized likelihood logistic regression with the ElasticNet-type penalty [4]. This approach provides built-in variable selection in the estimation procedure. Our model suggests 108 predictor genes that form a potential direction for further research.
Main text
Material and methods
Data
We analyze 88 cases with breast cancer diagnoses from the NOWAC Post-genome cohort [5]. For each case, we have an age-matched control that we use to normalize the gene expression levels. For our analysis this is mainly done to mitigate batch effects from the lab processing of the blood samples, cases and controls being kept together for the whole pipeline. Only women who received a breast cancer diagnosis at most one year after providing a blood sample were considered as cases. This limits our sample size but it is more biologically plausible to see a signal in more recent blood samples.
Out of the 88 breast cancers, 25% have metastases. The metastic- and non-metastic cancers are fairly similar in terms of usual covariates. Respectively the proportion of smokers is 13% against 25%. The proportion of hormone treatment is 25% against 31%. The median age (with .05 and .95 quantiles) is 56 (51, 61) against 56 (51, 62). The median BMI is 24.5 (19.4, 35.9) against 25.5 (21.1, 32.4). The median parity is 2 (1, 3) against 2 (0, 3).
The data were processed according to [6] and [7]. The pre-processed data is a 88×12404 fold change matrix, X, on the log2 scale. For each gene, g, and each observation, i, we have the measurement log2xig−log2x′ig. Here xig is the g expression level for the ith case, and x′ig is the corresponding control. The response variable, metastasis, indicates the presence of metastatic spread.
Predictive model
We model the probability of metastasis, p(m), given gene expression across all genes, x, by a penalized likelihood logistic regression with an ElasticNet-type penalty [4]. The likelihood of the logistic model
logp(m)1−p(m)=β0+β1x1+⋯+xp
is maximized under the constraint that (1−α)∑|βj|+α∑β2j≤t for some user-specified penalty size t and mixing parameter α.
We choose α=0.5 a priori and find a penalty size t in a data-driven way by optimizing for the modified version of Akaike’s Information Criterion [8, 9],
AIC′=LRχ2−2k,
where LRχ2 is the likelihood ratio χ2 for the model and k is the number of non-zero coefficients. We use this criterion on the recommendation of Harrell [10], who states that maximizing this criterion in terms of penalty often leads to a reasonable choice. We prefer this to tuning by cross-validation since it does not require data splitting. Data splitting procedures tend to induce more variance, which is undesirable with as few observations as we have. A more detailed discussion of these choices can be found in [11].
Metrics
We evaluate models by several criteria. Brier score [12] is the mean squared error,
B¯=n−1∑(y^i−yi)2,
between the probability that was predicted by the model, y^, and the known outcomes, y. It is a one-number summary of the calibration of predicted probabilities.
We also assess calibration by means of a calibration curve. This is an estimate of proportion of true successes as a function of predicted probability, which we calculate by smoothing the true zero/one outcome as a function of predicted probability (LOWESS with a span of 23). If n observations receive a prediction of p^, np^ of them should have the predicted condition for a well-calibrated model.
Concordance probability is the probability of ranking (in terms of predicted p^) a randomly chosen positive higher than a randomly chosen negative. This is equivalent to the area under the receiver operating characteristic curve (AUC), and is proportional to the Mann-Whitney-Wilcoxon U statistic [13].
Stability is the proportion of overlap between predictor genes chosen during different realizations of the modeling procedure. We follow [14] and measure this by the Jaccard index, |S1∩S2||S1∪S2|, where S1 and S2 are two sets of predictor genes.
Brier score and concordance probability are estimated using the optimism-corrected bootstrap approach described in [15], which has the advantage of using all of the data in estimating model performance opposed to data splitting procedures. Stability is estimated from regular bootstrap resampling.
Results
Evaluation metrics
Figure 1 shows the bootstrap distributions for our estimates of Brier score, concordance probability, and stability. The solid lines show point estimates and the dotted lines indicate the middle .8 of each distribution. The Brier score for our model is roughly .1, while that of an intercept-only null model is roughly .18. Since Brier score is the mean square error of predicted probabilities we can take its root to get an average error on the probability scale; .1−−√≈.32, which suggests that the predicted probabilities are not very accurate on average. Figure 2 corroborates this. The figure shows the pointwise calibration of predicted probabilities, ie., for a given predicted metastasis probability, how great a proportion observations turned out to have metastases. For a predicted metastasis probability <.4 the true proportion is ≈.1, while for a predicted metastasis probability >.8 the true proportion is ≈.7. In other words we overestimate low probabilities and underestimate high ones.
Fig. 1
figure1
Bootstrap distribution of optimism-corrected estimates for Brier score, concordance/AUC, and stability for the Elasticnet model. The solid vertical lines show point estimates, and the dotted vertical lines show the middle .8 of each distribution
Full size image
Fig. 2
figure2
Expected calibration of predicted probabilities shown in solid black. The dotted line shows middle .8 of the bootstrap distribution. Ideally, .8 of the observations for which .8 metastasis probability was predicted should turn out to show metastasis. In other words the ideal calibration is a diagonal line (shown in grey). Our model tends to overestimate lower probabilities and underestimate higher ones
Full size image
Returning to Fig. 1, the concordance probability (or AUC) is quite high at roughly .88, with a lower bound for the middle .8 of the distribution at .81. Contrast this with random guess at .5. This suggests that the model consistently selects gene sets that separate metastases from non-metastases in their expression levels in spite of the fact that the predicted probabilities are poorly calibrated. The stability of these chosen gene sets is around .16, which suggests the likely scenario that there are many correlated genes to choose from. With a stability of .16 for 108 genes you might expect a 17-gene overlap when fitting a similar model to similar data.
Selected genes
We list the 108 genes selected by penalized likelihood and describe them in general quantitative terms. We keep track of the selected gene sets under resampling and can hence calculate statistics for how often a given gene is selected and for how often a given gene is co-selected with any other gene. Table 1 shows the 108 selected genes ordered by their individual selection probabilities. Apart from the first few genes, the selection probabilities are not very high. It is quite likely that (i) a larger set of genes correlate with the ones we select and get selected in their place some of the time, and (ii) our selected genes correlate with one another and the selection of one some times makes the selection of another less likely. This is a natural consequence of doing variable selection: “redundant” information may shrink out of the model.
Table 1 Resampling selection probability for the 108 elasticnet-selected genes
Full size table
The selected genes show a clear difference in fold change between metastasized- and non-metastasized BC cases; we refer interested readers to Additional file 1. Further figures and discussion about, as well as pairwise co-selection can be found in [11].
Limitations
The prospective design of NOWAC yields data prior to the cancer diagnosis, thus allowing to test prediction models on original data corresponding to early-stage cancer. However, there will perforce never be more cases where the blood sample was provided close to diagnosis in this particular study. As the data acquisition technology has changed, there little hope to produce new comparable data outside of NOWAC. Since our data set is small (88 pairs of women for 12404 probes), we expect the success of both variable selection and prediction to be limited.
Concerning variable selection, the set of genes kept in the model is highly unstable under perturbation by resampling, and only a few of them are selected in a meaningful fraction resamples.
Concerning prediction, the AUC is high enough that there is reason for suspicion. The same is the case for Brier score, which is suspiciously low. It is quite likely that the bootstrap corrections for optimism are too. Moreover the bootstrap shows high variability in high dimensions. The calibration curve suggests that the predicted probabilities need to be better calibrated for this model to be useful for prediction in a real setting.
In model selection with small data sets it is recommended to use AUCc, which places a stronger penalty on larger numbers of parameters than the formulation we use [16]. At the same time we overestimate the effective number of parameters by taking k as the number of non-zero parameters, which does not take into account the shrinkage on parameter size. This places a larger penalty than necessary on a given model. Since in our case all models lie on the regularization path decided by the penalty size, a stronger/weaker parameter penalty will lead to similar results in terms of selected genes with some additions/omissions as the case may be.
The model we apply does not control for what is considered usual sources of confounding in breast cancer. This is both out of a desire to identify a pre-diagnosic gene signature for metastasis independent of questionnaire data, and from the realization that this would require the estimation of even more coefficients for already-inadequate data. The potential confounding from sources such as smoking and hormone therapy may not be a problem for prediction, but makes interpretation challenging. On the other hand what is considered a source of confounding for breast cancer may or may not be one when comparing breast cancers to one another in terms of metastasis. The explicit way to deal with this would be to derive a causal model to argue from.
This study is exploratory and not validated in external data. It is important that this work be viewed as hypothesis generating.
Availability of data and materials
The datasets generated and/or analysed during the current study are not publicly available due to restrictions under Norwegian regulations for access to confidential data based on patient consent and Research Ethics terms, but are available from the corresponding author on reasonable request.
Abbreviations
AUC:
Area under the (ROC) curve
BC:
Breast cancer
LOWESS:
Locally weighted polynomial regression
NOWAC:
Norwegian Women and Cancer
ROC:
Receiver operating characteristic
References
1.
Chi KR. The tumour trail left in blood. Nature. 2016;532:269–71.
CAS
Article
Google Scholar
2.
Lim B, Hortobagyi GN. Current challenges of metastatic breast cancer. Cancer Metastasis Rev. 2016;. https://doi.org/10.1007/s10555-016-9636-y.
Article
PubMed
Google Scholar
3.
Lund E, Dumeaux V, Braaten T, HjartÃ¥ker A, Engeset D, Skeie G, Kumle M. Cohort profile: the norwegian women and cancer study-nowac-kvinner og kreft. Int J Epidemiol. 2008;37(1):36–41.
Article
Google Scholar
4.
Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc Ser B. 2005;67(2):301–20.
Article
Google Scholar
5.
Dumeaux V, Børresen-Dale A-L, Frantzen J-O, Kumle M, Kristensen VN, Lund E. Gene expression analyses in breast cancer epidemiology: the Norwegian women and cancer postgenome cohort study. Breast Cancer Res. 2008;10(1):13. https://doi.org/10.1186/bcr1859.
CAS
Article
Google Scholar
6.
Bøvelstad HM, Holsbø E, Bongo LA, Lund E. A standard operating procedure for outlier removal in large-sample epidemiological transcriptomics datasets. bioRxiv 144519 (2017). https://doi.org/10.1101/144519. https://www.biorxiv.org/content/early/2017/05/31/144519.full.pdf.
7.
Lund E, Holden L, Bøvelstad H, Plancade S, Mode N, Günther C-C, Nuel G, Thalabard J-C, Holden M. A new statistical method for curve group analysis of longitudinal gene expression data illustrated for breast cancer in the nowac postgenome cohort as a proof of principle. BMC Med Res Methodol. 2016;16(1):28. https://doi.org/10.1186/s12874-016-0129-z.
CAS
Article
PubMed
PubMed Central
Google Scholar
8.
Akaike H. Information theory and an extension of the maximum likelihood principle. In: 2nd international symposium on information theory. Akademiai Kiado; 1973; p. 267–281.
9.
Verweij PJ, Van Houwelingen HC. Penalized likelihood in cox regression. Stat Med. 1994;13(23–24):2427–36.
CAS
Article
Google Scholar
10.
Harrell F. Regression modeling strategies as implemented in R package ‘rms’ version 2013;3(3)
11.
Holsbø E. Small data: practical modeling issues in human-model -omic data. PhD thesis, UiT—the arctic University of Norway (2019). Online: https://hdl.handle.net/10037/14660.
12.
Brier GW. Verification of forecasts expressed in terms of probability. Monthey Weather Rev. 1950;78(1):1–3.
Article
Google Scholar
13.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. https://doi.org/10.1148/radiology.143.1.7063747.
CAS
Article
PubMed
Google Scholar
14.
Haury A-C, Gestraud P, Vert J-P. The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures. PLoS ONE. 2011;6(12):28210. https://doi.org/10.1371/journal.pone.0028210.
CAS
Article
Google Scholar
15.
Efron B, Gong G. A leisurely look at the bootstrap, the jackknife, and cross-validation. Am Stat. 1983;37(1):36–48.
Google Scholar
16.
Burnham KP, Anderson DR. Model selection and multimodel inference: a practical information-theoretic approach. 2nd ed. New York: Springer; 2002.
Google Scholar
Download references
Acknowledgements
The publication charges for this article have been funded by a grant from the publication fund of UiT The Arctic University of Norway.
Funding
This study was supported by a grant from the European Research Council (ERC-AdG 232997 TICE).
Author information
Affiliations
Department of Computer Science, UiT – The Arctic University of Norway, Tromsø, Norway
Einar Holsbø & Lars Ailo Bongo
Laboratoire MAP5 (UMR CNRS 8145), Université Paris Descartes, Université de Paris, Paris, France
Vittorio Perduca & Etienne Birmelé
Cancer Registry of Norway, Oslo, Norway
Eiliv Lund
Department of Community Medicine, UiT – The Arctic University of Norway, Tromsø, Norway
Eiliv Lund
Contributions
EH provided most writing and data analysis. EB, VP, and LAB contributed substantially to design, interpretation and writing. EL conceived the project and provided study design and data acquisition on the NOWAC side. All authors read and approved the final manuscript.
Corresponding author
Correspondence to Einar Holsbø.
Ethics declarations
Ethics approval and consent to participate
The women in this study have given written informed consent for blood sampling. We have received approval from the Regional Committee for Medical Research Ethics for the basic collection and storing of questionnaire information, blood samples and tumour tissue from patients. All women have provided informed consent for later linkages to the Cancer Registry of Norway, the Norwegian Mammographic Screening Program, and the register of death certificates in Statistics Norway. The informed consent formula explicitly mentions that the blood samples can be used for gene–environment analyses. All data are stored and handled according to the permission given from the Norwegian Data Inspectorate.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Additional file 1. Expression levels of selected genes. This figure shows the expression levels of selected genes ordered by difference in medians between metastasized andnon-metastasized observations.
ENT-MD Alexandros G. Sfakianakis,Anapafseos 5 Agios Nikolaos 72100 Crete Greece,00306932607174,00302841026182,alsfakia@gmail.com
Blog Archive
-
▼
2020
(479)
-
▼
May
(68)
-
▼
May 21
(40)
- Medicine by Alexandros G. Sfakianakis,
- Trigeminal trophic syndrome classically presents w...
- Acupuncture as part of iatr
- Rapid Recovery of Cranial Nerve Deficits After Ant...
- Homocysteine as a potential predictor of cardiovas...
- Endoscopic needle biopsy of thalamic tumors: techn...
- A case of spindle cell dominant histiocytic sarcom...
- Acta Histochemica,
- Oral Biology,
- JAMA Otolaryngology–Head & Neck Surgery Curren...
- Therapeutic effect of Nivolumab for advanced / rec...
- Fluctuant Facial Mass
- An Atypical Cause of Difficulty Swallowing
- Persistent Laryngeal Swelling Caused by Primary In...
- Pseudomonas aeruginosa synthesized silver nanopart...
- Predicting breast cancer metastasis from whole-blo...
- miR-6869-5p I
- Clinical Oral Investigations,
- Necrotizing periodontitis or medication-related os...
- Global prevalence and risk factors for mental ...
- Effect of platelet‐rich plasma on colon anastomosi...
- Dysphagia,
- NBI in Oral Cavity Cancer
- Clinical Nuclear Medicine - Published Ahead-of...
- Italian Journal of Anatomy and Embryology,
- Effect of Different 131I Dose Strategies for Treat...
- 68Ga-DOTATATE Uptake in an Endolymphatic Sac Tumor...
- Italian Journal of Anatomy and Embryology,
- Medicine by Alexandros G. Sfakianakis,
-
▼
May 21
(40)
-
▼
May
(68)
- ► 2019 (2381)
About Me
Labels
Search This Blog
Subscribe to:
Post Comments (Atom)
Blog Archive
- Sep 24 (11)
- Sep 23 (70)
- Sep 20 (22)
- Aug 27 (2)
- Aug 25 (1)
- Aug 24 (2)
- Aug 20 (1)
- Aug 19 (1)
- Aug 18 (2)
- Aug 17 (1)
- Aug 16 (1)
- Aug 13 (1)
- Aug 12 (1)
- Aug 11 (1)
- Aug 10 (2)
- Aug 07 (1)
- Aug 06 (1)
- Aug 05 (1)
- Aug 04 (1)
- Aug 03 (1)
- Aug 02 (1)
- Jul 30 (1)
- Jul 29 (1)
- Jul 28 (1)
- Jul 27 (1)
- Jul 26 (1)
- Jul 23 (1)
- Jul 22 (1)
- Jul 21 (1)
- Jul 20 (1)
- Jul 19 (1)
- Jul 16 (1)
- Jul 15 (1)
- Jul 14 (1)
- Jul 13 (1)
- Jul 12 (1)
- Jul 09 (1)
- Jul 08 (1)
- Jul 07 (1)
- Jul 06 (28)
- Jul 05 (1)
- Jul 02 (1)
- Jul 01 (1)
- Jun 30 (1)
- Jun 29 (2)
- Jun 25 (1)
- Jun 24 (41)
- Jun 23 (7)
- Jun 22 (1)
- Jun 21 (1)
- Jun 18 (1)
- Jun 17 (1)
- Jun 16 (18)
- Jun 15 (1)
- Jun 14 (1)
- Jun 11 (1)
- Jun 10 (1)
- Jun 09 (36)
- Jun 08 (1)
- Jun 04 (1)
- Jun 03 (1)
- Jun 02 (1)
- Jun 01 (1)
- May 31 (8)
- May 28 (1)
- May 27 (1)
- May 26 (1)
- May 25 (1)
- May 24 (1)
- May 21 (40)
- May 19 (1)
- May 18 (1)
- May 17 (1)
- May 14 (2)
- May 13 (1)
- May 12 (1)
- May 10 (1)
- May 07 (1)
- May 06 (3)
- May 05 (2)
- May 03 (1)
- Apr 30 (1)
- Apr 28 (1)
- Apr 27 (1)
- Apr 26 (1)
- Apr 24 (1)
- Apr 22 (2)
- Apr 20 (1)
- Apr 16 (1)
- Apr 15 (1)
- Apr 14 (1)
- Apr 13 (1)
- Apr 10 (1)
- Apr 09 (1)
- Apr 08 (1)
- Apr 06 (2)
- Apr 05 (1)
- Apr 03 (1)
- Apr 02 (2)
- Apr 01 (2)
- Mar 30 (1)
- Mar 29 (1)
- Mar 27 (1)
- Mar 26 (1)
- Mar 24 (1)
- Mar 23 (1)
- Mar 20 (1)
- Mar 19 (1)
- Mar 18 (1)
- Mar 17 (1)
- Mar 16 (1)
- Mar 13 (1)
- Mar 11 (2)
- Mar 10 (1)
- Mar 08 (1)
- Mar 05 (3)
- Mar 04 (2)
- Mar 03 (2)
- Feb 27 (1)
- Feb 26 (2)
- Feb 24 (3)
- Feb 21 (2)
- Feb 20 (1)
- Feb 19 (1)
- Feb 16 (2)
- Feb 13 (1)
- Feb 12 (2)
- Feb 10 (3)
- Feb 09 (1)
- Feb 07 (1)
- Feb 05 (2)
- Feb 04 (1)
- Feb 03 (1)
- Feb 02 (4)
- Jan 30 (2)
- Jan 28 (1)
- Jan 27 (3)
- Jan 26 (1)
- Jan 23 (3)
- Jan 22 (1)
- Jan 21 (3)
- Jan 20 (2)
- Jan 19 (1)
- Jan 16 (1)
- Jan 15 (7)
- Jan 14 (6)
- Jan 12 (1)
- Jan 09 (2)
- Jan 07 (2)
- Jan 06 (3)
- Jan 04 (1)
- Jan 03 (1)
- Jan 02 (2)
- Jan 01 (1)
- Dec 31 (1)
- Dec 30 (2)
- Dec 29 (2)
- Dec 28 (1)
- Dec 26 (1)
- Dec 20 (1)
- Dec 17 (2)
- Dec 16 (1)
- Dec 13 (1)
- Dec 12 (1)
- Dec 11 (1)
- Dec 10 (1)
- Dec 09 (1)
- Dec 04 (1)
- Dec 03 (1)
- Dec 01 (1)
- Nov 30 (1)
- Nov 29 (1)
- Nov 27 (3)
- Nov 26 (1)
- Nov 25 (1)
- Nov 24 (4)
- Nov 23 (1)
- Nov 22 (1)
- Nov 21 (1)
- Nov 19 (2)
- Nov 17 (2)
- Nov 16 (1)
- Nov 14 (1)
- Nov 13 (1)
- Nov 12 (1)
- Nov 11 (2)
- Nov 10 (1)
- Nov 09 (1)
- Nov 07 (1)
- Nov 06 (1)
- Nov 05 (2)
- Nov 04 (3)
- Nov 03 (2)
- Nov 02 (1)
- Nov 01 (1)
- Oct 31 (1)
- Oct 30 (1)
- Oct 29 (1)
- Oct 28 (1)
- Oct 27 (1)
- Oct 26 (1)
- Oct 24 (1)
- Oct 23 (1)
- Oct 22 (1)
- Oct 21 (2)
- Oct 20 (1)
- Oct 18 (1)
- Oct 17 (2)
- Oct 15 (2)
- Oct 13 (2)
- Oct 12 (1)
- Oct 10 (2)
- Oct 09 (3)
- Oct 08 (1)
- Oct 07 (2)
- Oct 06 (2)
- Oct 05 (1)
- Oct 04 (1)
- Oct 02 (3)
- Oct 01 (1)
- Sep 30 (4)
- Sep 29 (3)
- Sep 27 (1)
- Sep 26 (2)
- Sep 25 (2)
- Sep 24 (3)
- Sep 23 (4)
- Sep 19 (3)
- Sep 18 (1)
- Sep 17 (4)
- Sep 16 (1)
- Sep 15 (1)
- Sep 12 (1)
- Sep 11 (2)
- Sep 10 (4)
- Sep 09 (1)
- Sep 08 (2)
- Sep 05 (4)
- Sep 04 (1)
- Sep 03 (3)
- Sep 02 (5)
- Sep 01 (2)
- Aug 30 (2)
- Aug 29 (3)
- Aug 28 (2)
- Aug 27 (1)
- Aug 26 (2)
- Aug 23 (1)
- Aug 22 (1)
- Aug 21 (3)
- Aug 19 (2)
- Aug 18 (3)
- Aug 17 (1)
- Aug 16 (1)
- Aug 15 (1)
- Aug 13 (1)
- Aug 12 (3)
- Aug 11 (6)
- Aug 08 (6)
- Aug 07 (9)
- Aug 06 (5)
- Aug 05 (8)
- Aug 04 (1)
- Aug 01 (5)
- Jul 31 (6)
- Jul 30 (7)
- Jul 29 (6)
- Jul 28 (7)
- Jul 27 (1)
- Jul 26 (1)
- Jul 25 (4)
- Jul 24 (7)
- Jul 23 (10)
- Jul 22 (4)
- Jul 21 (10)
- Jul 20 (8)
- Jul 19 (2)
- Jul 18 (3)
- Jul 17 (5)
- Jul 16 (8)
- Jul 15 (19)
- Jul 14 (15)
- Jul 13 (8)
- Jul 11 (13)
- Jul 10 (26)
- Jul 09 (4)
- Jul 08 (26)
- Jul 07 (7)
- Jul 05 (33)
- Jul 04 (10)
- Jul 03 (24)
- Jul 02 (26)
- Jul 01 (26)
- Jun 30 (23)
- Jun 29 (24)
- Jun 28 (14)
- Jun 27 (19)
- Jun 26 (8)
- Jun 25 (78)
- Jun 24 (19)
- Jun 23 (17)
- Jun 22 (25)
- Jun 21 (12)
- Jun 20 (34)
- Jun 19 (4)
- Jun 18 (1)
- Jun 17 (17)
- Jun 16 (23)
- Jun 14 (2)
- Jun 13 (16)
- Jun 12 (27)
- Jun 11 (30)
- Jun 10 (39)
- Jun 09 (3)
- Jun 08 (15)
- Jun 07 (5)
- Jun 06 (14)
- Jun 05 (16)
- Jun 04 (21)
- Jun 03 (14)
- Jun 02 (33)
- May 31 (4)
- May 30 (23)
- May 29 (8)
- May 28 (23)
- May 27 (16)
- May 26 (22)
- May 25 (8)
- May 24 (12)
- May 23 (7)
- May 22 (1)
- May 21 (36)
- May 20 (4)
- May 19 (21)
- May 17 (24)
- May 16 (17)
- May 15 (30)
- May 14 (19)
- May 13 (6)
- May 12 (18)
- May 09 (6)
- May 08 (3)
- May 07 (27)
- May 06 (1)
- May 05 (9)
- May 03 (7)
- May 02 (15)
- May 01 (34)
- Apr 29 (34)
- Apr 27 (18)
- Apr 25 (19)
- Apr 24 (1)
- Apr 23 (9)
- Apr 22 (23)
- Apr 21 (14)
- Apr 19 (10)
- Apr 18 (34)
- Apr 17 (12)
- Apr 16 (19)
- Apr 15 (12)
- Apr 14 (18)
- Apr 12 (5)
- Apr 11 (17)
- Apr 10 (12)
- Apr 09 (20)
- Apr 08 (14)
- Apr 07 (21)
- Apr 05 (1)
- Apr 04 (26)
- Apr 03 (9)
- Apr 02 (20)
- Apr 01 (22)
- Mar 31 (16)
- Mar 29 (7)
- Mar 28 (29)
- Mar 27 (6)
- Mar 26 (20)
- Mar 25 (18)
- Mar 23 (26)
- Mar 22 (3)
- Mar 20 (18)
- Mar 19 (19)
- Mar 18 (5)
- Mar 17 (2)
- Mar 16 (5)
- Mar 15 (7)
- Mar 14 (27)
- Mar 13 (7)
- Mar 12 (15)
- Mar 11 (1)
- Mar 10 (1)
- Mar 08 (1)
- Mar 07 (6)
- Mar 06 (4)
- Mar 04 (6)
- Mar 02 (4)
- Mar 01 (7)
- Feb 27 (3)
- Feb 26 (6)
- Feb 25 (2)
- Feb 24 (4)
- Feb 22 (2)
- Feb 21 (6)
- Feb 20 (9)
- Feb 19 (4)
- Feb 18 (11)
- Feb 16 (1)
- Feb 13 (8)
- Feb 11 (17)
- Feb 10 (4)
- Feb 07 (7)
- Feb 06 (1)
- Feb 01 (5)
- Jan 26 (2)
- Jan 24 (7)
- Jan 23 (1)
- Jan 22 (2)
- Jan 21 (2)
- Jan 20 (1)
- Jan 17 (10)
- Jan 16 (1)
- Jan 15 (1)
- Jan 14 (7)
- Jan 13 (35)
- Jan 10 (29)
- Jan 08 (2)
- Jan 07 (8)
- Jan 06 (2)
- Jan 05 (1)
- Jan 04 (8)
- Jan 03 (13)
- Jan 02 (12)
- Jan 01 (4)
- Dec 31 (7)
- Dec 30 (4)
- Dec 29 (6)
- Dec 28 (25)
- Dec 27 (6)
- Dec 26 (10)
- Dec 25 (1)
- Dec 24 (1)
- Dec 22 (3)
- Dec 21 (55)
- Dec 20 (71)
- Dec 19 (59)
- Dec 18 (89)
- Dec 17 (19)
- Dec 16 (15)
- Dec 15 (42)
- Dec 14 (57)
- Dec 13 (33)
- Dec 12 (51)
- Dec 11 (30)
- Dec 10 (47)
- Dec 09 (11)
- Dec 08 (46)
- Dec 07 (35)
- Dec 06 (54)
- Dec 05 (34)
- Dec 04 (50)
- Dec 03 (11)
- Dec 02 (9)
- Dec 01 (34)
- Nov 30 (43)
- Nov 29 (46)
- Nov 28 (28)
- Nov 27 (47)
- Nov 26 (37)
- Nov 25 (7)
- Nov 24 (37)
- Nov 23 (38)
- Nov 22 (15)
- Nov 21 (34)
- Nov 20 (40)
- Nov 19 (66)
- Nov 18 (10)
- Nov 17 (32)
- Nov 16 (49)
- Nov 15 (51)
- Nov 14 (40)
- Nov 13 (38)
- Nov 12 (25)
- Nov 11 (22)
- Nov 10 (13)
- Nov 09 (30)
- Nov 08 (40)
- Nov 07 (19)
- Nov 06 (62)
- Nov 05 (45)
- Nov 04 (37)
- Nov 03 (49)
- Nov 02 (17)
- Nov 01 (26)
- Apr 10 (380)
- Jan 08 (404)
- Dec 13 (358)
- Dec 12 (24)
- Dec 07 (304)
- Dec 06 (59)
- Nov 20 (419)
- Oct 30 (423)
- Sep 25 (333)
- Sep 24 (57)
- Sep 13 (290)
- Sep 12 (48)
- Aug 17 (389)
- Jul 31 (340)
- Jul 25 (349)
- Jul 20 (1)
- Jul 19 (443)
Labels
Pages
International Journal of Environmental Research and Public Health IJERPH, Vol. 17, Pages 6976: Overcoming Barriers to Agriculture Green T...
-
Calcium oxalate films on works of art: A review Publication date: Available online 14 June 2019 Source: Journal of Cultural Heritage Author...
-
The conceptualization of gangs: Changing the focus Publication date: July–August 2019 Source: Aggression and Violent Behavior, Volume 47 Au...
-
Increased REDD1 facilitates neuronal damage after subarachnoid hemorrhage Publication date: September 2019 Source: Neurochemistry Internati...
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.