One problem with studies that create an outcome and then find data to support it is they don't have real world application. International Agency for Research on Cancer epidemiologists, for example, have been caught creating a desired warning label for chemicals and then hand-picking studies to support that goal - the opposite of what scientists do. Harvard TS Chan School of Public Health is frequently criticized for data dredging, where they take questionnaires with so many foods and outcomes they are sure to be able to link something and claim it has statistical power.
Yet in the real world, weedkillers don't cause cancer, neonicotinoids don't kill bees, and there are no "miracle vegetables."
Lack of real world application is the reason some ecologists are criticizing the Jena experiment, which was created to learn about the ecosystem by creating vegetation types to imitate biodiversity loss. Like bad epidemiology models, the work has little ecological value if the vegetation types used are rare or non-existent in the real world.
Aerial view of ‘BioDIV experiment’ in Minnesota, which was also analyzed as part of the study. Credit: Forest Isbell
Compared to semi-natural grasslands nearby and a large set of scientifically monitored agricultural sites across Germany, the Jena Experiment only had 28 percent of experimental plots that could be considered similar enough to the natural vegetation that they could classified as realistic, the scholars found. Yet numerous papers promoting concern about biodiversity loss have been written based on the results.
If you get the right answer using the wrong method, does it matter? It does to the public. Comparing the outcomes of the Jena experiment and more real-world sites, the outcomes are unchanged; biodiversity loss is still a bad thing; but people will trust the results more if the methods are used to invoke the real world. Biodiversity matters, conservation is important, the public don't need experiments that can only have one result to understand that.
Citation: Jochum, M. et al. (2020): The results of biodiversity-ecosystem functioning experiments are realistic. Nature Ecology&Evolution, doi: 10.1038/s41559-020-1280-9
