What is peer review?

[Sherlock Holmes]

Often when debating atheism or questioning the evolution doctrine, the supporters of evolution will reject arguments against it made by scientists because they insist that only "peer reviewed" publications are to be trusted (else it must be pseudo science).

So I want to ask how does one decide whether a journal is or is not peer reviewed? what definition do people use to help them make this decision?

[brunumb]

Of course I can't answer that question! you are asking me what someone else does, how someone else works, how someone else thinks; I cannot answer on behalf of another person, if you want an answer reach out, I'm sure he'd be happy to respond, most academics are easy to reach and often happy to answer questions.

So you can't really argue that creation science is a genuine thing. If it was you would be able to give us at least an idea of how so-called creation scientists employ the scientific method in their investigation of creation. It is clearly just religion and has no place in scientific journals. Why don't you go off and consult some of those academics and come back when you are better equipped to carry on a meaningful discussion.

[JoeyKnothead]

It's not surprising at all that towns existed in ancient times. The problem comes with trying to show reports regarding personal activities within such towns are reliable.

Is Gone with the Wind a documentary?

[Sherlock Holmes]

It's not surprising at all that towns existed in ancient times. The problem comes with trying to show reports regarding personal activities within such towns are reliable.

Is Gone with the Wind a documentary?

You mean has there ever been a documentary titled "Gone with the Wind"? I don't know.

[Purple Knight]

What is peer review?

Lately? Rubbish. People have exposed it to be so.

https://www.vox.com/2015/12/7/9865086/p ... e-problems

https://www.theatlantic.com/ideas/archi ... ax/572212/

The problem is that there's too much homogeneity of ideology, and even if it's the correct ideology, it makes science not work. Even unpopular or wrong ways of thinking need advocates. When the world was morally heterogeneous, when we had Nazis and eugenicists along with the quite correct people who said such things were abominable, we made progress. Now we're just going to be stuck thinking what we think forever because there's no mechanism for error correction any longer.

It's like evolution. Take this into consideration even if you don't trust that this process occurs.

Assume for a moment that something without a mind built a wing. This beautiful, functional, nigh-perfect thing can come into being without anyone ever knowing how to build it.

How?

Lots and lots and lots of mistakes.

Undirected errors. Simply allowing them. Nobody trying to screen out what we "know" to be wrong. Heck, I'm a cat breeder and if cats never had tongue brushes before, I would have screened out one with fingernails growing out of its tongue. But give that a chance, and wrongness can generate correctitude.

We have to invite errors back in. Seems counterproductive, but we do.

[Difflugia]

What is peer review?

Lately? Rubbish. People have exposed it to be so.

The problem is that there's too much homogeneity of ideology, and even if it's the correct ideology, it makes science not work.

That's not what either of those experiments claims to demonstrate. The result of the first is not that peer review is rubbish, but that papers receive less of it than they should. Every response detailing ways that have produced better results described methods of increasing the amount of peer review. The second experiment shows that the lack of rigorous methodological expectations is what renders peer review ineffective.

[Purple Knight]

That's not what either of those experiments claims to demonstrate. The result of the first is not that peer review is rubbish, but that papers receive less of it than they should. Every response detailing ways that have produced better results described methods of increasing the amount of peer review. The second experiment shows that the lack of rigorous methodological expectations is what renders peer review ineffective.

But you can't make it rigorous. People have biases and robbing the reviewers of their autonomy or placing regulations on how they do things can only hurt the process.

You can, however, balance ideology so that all views have advocates and reviewers won't be able to just agree that this paper's conclusion is good so it must be good.

It's just like nature. Rigorous selection won't happen without selective pressure and at least some heterogeneity. If everyone is doing the same wrong thing, no pressure will stop them from continuing to do it; they'll just all die. You can even get ruts where doing things differently should be an advantage, but due to some form of reinforcement where what is plentiful is rewarded and what is rare is punished simply for being rare, the "better" adaptation dies and the dunces continue to occupy all the niches and thereby kill their species.

[Jose Fly]

But you can't make it rigorous. People have biases and robbing the reviewers of their autonomy or placing regulations on how they do things can only hurt the process.

You address that by increasing the number and diversity of reviewers. IOW, the solution is more peer review, not less.

Of course you don't go so far as bringing in flat earthers and the like either. The reviewers still need to be experts in the subject matter, rather than just random people off the street.

You can, however, balance ideology so that all views have advocates and reviewers won't be able to just agree that this paper's conclusion is good so it must be good.

As someone who's been involved in both sides of the process (author and reviewer), all I can say is that I've never seen anything like that occur.

[Purple Knight]

You address that by increasing the number and diversity of reviewers. IOW, the solution is more peer review, not less.

Of course you don't go so far as bringing in flat earthers and the like either. The reviewers still need to be experts in the subject matter, rather than just random people off the street.

I agree, the solution is more diversity of reviewers, and perhaps more of them. And I think it would be incredibly drastic by modern standards to bring in random people or flat-earthers. Perhaps try that after everything else has failed and science is basically dead because nobody trusts it.

[Difflugia]

But you can't make it rigorous. People have biases and robbing the reviewers of their autonomy or placing regulations on how they do things can only hurt the process.

Why? I'm sure that you think that's true, but I don't know what you think is being hurt or why.

You can, however, balance ideology so that all views have advocates and reviewers won't be able to just agree that this paper's conclusion is good so it must be good.

What ideology? The purpose of reviewers is to ensure that methodology is sound, potentially conflicting results and conclusions of other researchers are addressed, and the scope of the conclusions fits the data. "Good" in this instance is a minimum standard of rigor, not one of correctness.

It's just like nature. Rigorous selection won't happen without selective pressure and at least some heterogeneity. If everyone is doing the same wrong thing, no pressure will stop them from continuing to do it; they'll just all die. You can even get ruts where doing things differently should be an advantage, but due to some form of reinforcement where what is plentiful is rewarded and what is rare is punished simply for being rare, the "better" adaptation dies and the dunces continue to occupy all the niches and thereby kill their species.

I'm not sure what kind of gatekeeping you think peer review either should be responsible for or what is in practice. The present goal of peer review is that each person reading a paper doesn't have to analyze things like whether proper controls were followed or if other researchers have published conflicting results. The first of your linked article offers evidence that the goal may not being met, but reinforces the idea that the goal itself is necessary. The overall idea in that article is that forms of crowdsourcing have been more effective at peer review than the current model of having a journal editor assign specific peer reviewers.

[Jose Fly]

I agree, the solution is more diversity of reviewers, and perhaps more of them.

Glad we agree. The only thing is, some fields are so specialized that it can be difficult to find a real diverse group of reviewers.

And I think it would be incredibly drastic by modern standards to bring in random people or flat-earthers. Perhaps try that after everything else has failed and science is basically dead because nobody trusts it.

Well, when we get to that point we'll certainly have to adjust!