Chasing the Next Big Thing: Research, replication, and review

Dr. Jo Røislien is a famous international science communicator who reaches a wide audience by appearing regularly on radio, television, and the printed press. He delivers lectures on the communication of complex topics, knowledge dissemination, and his own research. He is a Norwegian mathematician, biostatistician, and researcher in medicine, and holds a PhD in geostatistics from the Department of Petroleum Engineering and Applied Geophysics at Norwegian University of Science and Technology (NTNU). Currently, he is an Associate Professor at the Department of Health Sciences, University of Stavanger, with additional posts in the Department of Biostatistics and the Center for Addiction Research at the University of Oslo. For his full bio in which titles of his productions are also listed, please see the first part of his interview.

In the first part of a three-part interview, Røislien provided meaningful insight into the nature of data and statistical analysis. Here in his next segment, Røislien discusses his views on the challenges in journal publishing, basic versus applied research, and replication studies.

As an academic author, can you discuss your opinion about current challenges in research or journal publishing?

In the mathematical sciences we tend to have a relatively relaxed relationship with the impact factor. Because we all know it is a vague measure of the impact of the journal. In mathematical sciences, truth does not have an expiry date, so citations tend to cumulate slowly, and for some papers they continue seemingly forever. Some papers have massive amounts of citations, most have few or none.

The impact factor is an expiry date type measure. Fair enough. It won’t reflect every impact. But, honestly, did you really think it would? Finding one measure for “importance” is obviously impossible. But it doesn’t automatically mean that the impact factor is worthless. In the same way as BMI tells you something about your health, but not everything. Your salary tells you something about your economy, but not everything.

The impact factor is sometimes viewed as a problem in itself, but the problem is when this complex, high-dimensional problem is being reduced to a one-dimensional measure.

I find it more annoying though when people criticize without coming up with an alternative solution. I have an engineering background, and engineers focus on coming up with solutions. Obviously the world isn’t perfect. Of course it can be improved upon. We know. Now instead, can you suggest something better?

As an academic, peer review is part of my everyday life. I submit papers for review; I review papers other people have submitted. And I must admit that I am a bit surprised by the poor quality of surprisingly many of the papers I am asked to review. People tend to think that just because you spent a lot of time on something that qualifies as research. And these poor papers I am asked to review never come from high impact factor journals, but from new journals, low impact factor journals, journals who brag about very quick referee process. Things like that. Bad science is bad for science. So some filter is necessary.

To me it’s obvious that we are gravitating towards a world where all research papers are available to everybody. But it is a process. Even online journals cost money. Peer review processes must still be coordinated. Software updated. Editorial meetings. We must allow these things to take time. Not because we do not want change, but because we must allow ourselves to think these things through properly. 

You seem to promote learning by discovery. What are your views on basic research versus applied research?

When working on my doctorate, in geostatistics and stochastic modelling, my supervisor always encouraged me to think for myself. We would discuss some possible strategies and approaches for solving a particular problem, which would generally imply throwing me into deep mental waters without a life vest, to which I would often mumble “I’ll go have a look in the library,” and my supervisor would go “No-no-no! Try for yourself.” And so I did. The result was that coming up with a possible solution to the problem at hand was sometimes slow, but I learned a lot. No, wrong choice of word. I understood a lot. Big difference.

I tend to take the same approach with my own students. And my kids. Even in the public domain. When we made the hugely successful math TV series “Siffer” (“Digits”) on Norway’s largest national broadcaster NRK1 (the series rated 40% of TV viewers and was nominated for two Norwegian Emmys) we took the same approach. Rather than focusing on theory and hypothetical situations we went out into the real world exploring it through the eyes of mathematicians and statisticians. We explored mathematical graph theory by going to a high school and constructing a make-out-map: Who has made out with whom? And what does that tell us about the spread of infectious diseases? Things like that.

Some of my peers think that my approach is populist, but I find that attitude overly passive. To me it is all about inviting people into the world of knowledge by siding with the natural way for humans to learn. Playing. Being curious. Discovering. How does the world work?

You cannot learn football by watching football on TV. You cannot learn to play an instrument by listening to music. You cannot learn statistical analysis without actually analyzing. You have to do.

And this goes for both basic and applied research. I love basic research, and respect it a lot, and I honestly thought that that would be my career. My life has just brought me elsewhere.

In your field, how important are replication studies and is there a need for more?

There is always a need for replication studies. One study is an anecdote. Give me two, ten, a hundred studies that all point in the same direction. Now we’re talking.

We tend to forget that the scientific theories that are the foundation of our society, Newton’s laws of motion, or Evolution, are not one person’s view, one single awesome study. They are piles and piles of studies giving the same result. Newton’s laws of motion are still developed from the ground up and tested and retested over and over again. Every engineering student in the world has deduced Newton’s law of universal gravity and performed experiments to quantify the gravitational pull of the earth. Evolution has been confirmed in literally millions of studies across the world, but that hasn’t stopped scientists from relentlessly gathering more and more evidence. 

A lot of resources are being put into medical research now. And medical research results are all over the news. The problem (if that is the right word) with medical research being a relatively young research field compared to STEM (where I have my background) is that there is so much new ground to cover, that in medicine ‘exploratory’ tends to be promoted much more than ‘confirmatory.’ So researchers in the field all chase the Next Big Thing. The new result, the new hypothesis, the new treatment. This chase for ‘new’ leaves us in some sort of a vacuum, where the frontier seems to be moving forward in a rapid pace, but not enough studies are being conducted to confirm, or refute, new findings, to establish whether the frontier is indeed moving forward at all, or in the right direction.

I wish I could live a hundred years from now, and look back on today. How will it look? Which results will stand? Which ones will not?

Read the next and final segment of this interview with Dr. Jo Røislien.