Misplaced Pages

I've added this test {{#ifeq:{{{1|}}}|section|section|article}} to {{medref}}. If, and only if, the first parameter is exactly the word "section", then the template will display "This section ..." instead of "This article ...". This change does not affect cases where the first parameter is a reason. I've used this in an article (Paris polyphylla#Medicinal uses, as of now) and it works fine. Peter coxhead (talk) 08:28, 2 May 2012 (UTC)

Actually I had to make another change to stop the word "section" appearing as a reason as well as forcing "This section ..." when |reason= was absent. I've tested all the cases and it should be ok now. Peter coxhead (talk) 09:18, 2 May 2012 (UTC)

I've also added information about {{Medref}} to the documentation for {{Refimprove}}, which is where I went in the first instance and failed to find the more specific template. Peter coxhead (talk) 09:35, 2 May 2012 (UTC)

Add to Multiple issues?

I think it might be worth adding it to {{Multiple issues}} in line with other standalone templates. Wording (examples) needs to be agreed on.

Something like:

It needs additional medical references for verification. or

It relies on references which may not be reliable sources for medical or health content. or

It includes attribution to sources which may not be reliable for medical content.

Thoughts? --92.6.211.228 (talk) 18:45, 1 May 2012 (UTC)

Yeah, I was planning on getting it set up along with expert-subject, which is also missing, since I've already added medref to the multiple issues template on a couple articles. EDIT: Actually expert-subject is in there, just called "expert". Anyway, I'll go add an edit request for medref. — Bility (talk) 20:52, 1 May 2012 (UTC)

Just an FYI, the request to add medref to multiple issues was denied. I removed it from the three instances of multiple issues that I was aware of. Cheers, — Bility (talk) 16:55, 29 May 2012 (UTC)

The guideline should be changed

"Basic advice", subsection "Assess evidence quality", paragraph 4 includes this statement: " "Assessing evidence quality" means that editors should determine the quality of the type of study. Editors should not perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. "

Recently, this study was proposed as a reference for the article "Chemotherapy". Several editors (including, latterly, myself) were critical of the study and objected to the use of the study as a reference. See full details here.

WhatamIdoing quoted the above text from WP:MEDASSESS as justification for inclusion of Morgan's paper as a reference. In my opinion, the text fails to take into account those rare secondary sources that are severely biased or flawed, such as Morgan's paper.

I see no reason why editors shouldn't perform detailed peer reviews if they so choose. Moreover, I believe that such reviews can be helpful in determining suitability as references for Misplaced Pages. Axl ¤ 19:41, 1 May 2012 (UTC)

Are there other reviews that are more recent that contradict this paper from 2004? We do say evidence from the last 3-5 years. --Doc James (talk · contribs · email) 20:28, 1 May 2012 (UTC)

I don't believe that there are any other reviews of this nature – because the whole premise of the review is flawed. Axl ¤ 20:31, 1 May 2012 (UTC)

For information: the disputed text was added on 27 March 2010 by WhatAmIDoing. A talk page note at the time of the edit did not bring about any further discussion. However, the note refers to this earlier discussion and the whole section was added by Eubulides along with this discussion. -- Colin° 20:36, 1 May 2012 (UTC)

I think we should allow editors to exercise judgement in deciding whether a secondary source is actually suitable. There are loads of secondary sources that are of insufficient quality. In that sense, the "peer review" should assess the quality of the source. Disputes as to the suitability should be resolved by consensus. In this case, there was consensus that the source was not suitable. JFW | T@lk 21:46, 1 May 2012 (UTC)

Yes agree that consensus should be followed over the guideline. If the majority of editors do not see this ref as suitable than it should not be used. Does not necessitate a change in WP:MEDRS though. Many people still fell that WP:MEDRS says using primary sources is fine when high quality secondary sources are available which IMO is a more pressing issue.Doc James (talk · contribs · email) 21:54, 1 May 2012 (UTC)

As per User:JFW and User:Doc James above. Of course secondary sources are vulnerable to bias (and original studies can contain high-quality 'secondary source' material). Metanalyses and other systematic reviews are genuine 'studies' in their own right. Since WP:MEDRS needs to be widely comprehensible and has to safeguard against real-world dangers such as POV pushing and poorly informed editing, it can only provide rather broad guidance. So well-informed talk page discussions can be key. —MistyMorn (talk) 11:24, 2 May 2012 (UTC)

I share some of WhatAmIDoing's concerns (expressed elsewhere) that we have to come up with rules that work for the POV pusher and the undereducated. I'm one of the undereducated. My brain is much smaller than that of MastCell and Axl and WhatAmIDoing; I have no health training beyond a first aid badge when I was in the Scouts; I'm just someone's dad. Some of WP's rules are here purely because of the special situation with our authors: we don't really know who they are; we don't trust who they say they are; most of them are bonkers; some of them aren't geniuses. An editor who is an expert in the subject is likely to find some of these rules frustrating at times.

The problem with the Chemotherapy paper raises two issues. The first is that academic papers, whether primary research or reviews, are written for a certain learned audience, and not for general consumption. That PubMed has made them (and particularly their abstracts) so accessible, is wonderful but at the same time very dangerous. As the article talk page shows, the abstract is over-simplified to the point of being misleading. And it takes someone quite knowledgeable to discover what the analysis actually looked at and is capable of saying. Combine this with the possibility that the paper is flawed/biased and we have a problem. Fortunately, it is not a common problem, and is less of a problem than editors trying to pick/review primary research papers.

The second issue is the level of source we are using here. There's too much emphasis on review papers among WP:MED folk in my opinion. Yes, it is wonderful to be able to read all this material on your browser or PDF on the computer, and search it and collate it on your hard disk. I do so myself. But what about books, guys, books! Chemotherapy is a big subject. I just searched Amazon on it and turned up several suitable texts including ISBN 160831782X: "Handbook of Cancer Chemotherapy", edited by Roland T. Skeel, Samir Khleif, published 2011 (an earlier edition is actually used by the article for one point, but sadly the page number isn't given). This is 832 pages of wonderful readable and up-to-date-enough material. Approximately 300,000 words by a rough estimate. Now our Chemotherapy article is about 4,000 words. Which means you could condense that book 75x to achieve the current text. That's about 5 words from every page.

The chemotherapy article is crap. There are many medical articles on WP I'd recommend to a friend, but that one isn't. Not by a long way. Why are we wasting days discussing the merits or otherwise of some 8-year-old review paper. Visit your university library bookshelves, or order a book (second-hand) from Amazon/etc -- you can always sell it later.

WP:WEIGHT is underused in sourcing disputes. Rather than wikilawering over PSTS or disecting the journal paper or whatever, just find out "What do experts generally say, in published reliable sources, when writing about this subject". We are used to using WEIGHT to judge research findings: we read and use reviews/etc to find that. Similarly, we should use WEIGHT to judge analysis reports/reviews, especially if editors dispute them. What do our serious weighty oncology textbooks have to say on the subject? Rather than fall out with each other over such things, find some expert person/body who has already thought hard about it, published their work, and use that. WP:WEIGHT specifically warns against editors forming their own consensus about what issues/facts to include in an article, so I disagree with JFW/Doc James on that point -- however I suspect if you follow my advice, the review and its conclusions would be rejected anyway. Colin° 22:09, 1 May 2012 (UTC)

I feel that, in this instance, the issue could be resolved by proper usage of existing guidelines. Colin suggested above several alternatives to giving infinite weight to a single source that is seemingly flawed. I briefly proposed another on Talk:Chemotherapy that involves exercising judgment not on the quality of the source, but whether the wiki text correctly reflect the conclusions being drawn by Morgan et al. To wit, is the source saying what the article is saying? In my mind, the source was misused because it was made to support a statement that is too broad. Unless a more systemic problem with WP:MEDASSESS is identified, I do not support making drastic changes to guidelines serving a useful purpose. Wafflephile (talk) 23:30, 1 May 2012 (UTC)

I didn't intend to re-hash the debate about the (lack of) merit of Morgan's paper here. Rather I wish to consider the value of this statement from WP:MEDRS: " Editors should not perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. "

Why shouldn't editors perform a detailed academic peer review, if they so wish?

The second sentence is potentially misleading. With careful reading, it implies that only the type of study needs to be high-quality. As long as the source is a review article, it doesn't matter how poor its quality. It is inevitable that some secondary sources are better quality than others. The second sentence implies that we should give them equal weight, regardless of their quality. Axl ¤ 12:16, 2 May 2012 (UTC)

Most editors aren't qualified, or indeed competent, to perform a detailed academic peer review, and we've no mechanism in place to identify those who are competent and give their views more weight. Consequently, we've no way of allowing experts to reject papers that are truly dire without also allowing POV pushers and fringe theorists to reject papers that they dislike, on the grounds of nothing but their own (often ill-informed or irrational) opinion. It's a similar situation to allowing editors to directly insert OR into articles: an appealing option in some respects, but dangerous in others. Jakew (talk) 12:28, 2 May 2012 (UTC)

Does the fact that "most" editors aren't "qualified" constitute a justification for allowing lunacy to be perpetuated? If I see some kind of glaring flaw in a study or review, am I supposed to act like nothing's wrong? I always thought of Wiki articles as de facto reviews in some cases--balanced, dispassionate, factual surveys of whatever knowledge exists in a given area. Does the fact that my objection to a reviewer's statement that black is white has not yet resulted in a "comment" in a journal or an entry in a blog mean that, because I cannot quote an objection, that we have to pretend that black is white until we can document an alternative position? Here's an example: I discovered in http://www.ncbi.nlm.nih.gov/pubmed/3569020 that Burzynski, in 1987, theorized that one of his panacea molecules, #A10, interacts with DNA (which it does) to compete with carcinogens and thereby keep tumors from growing. The problem is that once tumors are formed, the carcinogen has nothing to do with it, and his molecules competing with carcinogens would have no conceivable bearing on the treatment of tumors (although, if correct, it is conceivable that they might be chemopreventive.) I have seen all kinds of criticism of Dr. B, but I have not seen anyone make that particular observation (maybe I missed it). If I were to be doing some kind of Wiki editing of his stuff (that is probably just as joyless as commenting on chemotherapy itself; I'm not even going to look), would I have to refrain from commenting on that? To me, it indicates an intellectual error that undermines the entire basis of what he thought he was doing. How could it be against Misplaced Pages policy not to point out the obvious? I don't think of Misplaced Pages as having to play the "emperor's new clothes" game.

Even the most prestigious institutions sometimes publish material that cannot be allowed to be unquestioned, and certainly, we all know that possessing even a Nobel Prize does not immunize the recipient to error or to lunacy. (Dare I suggest that it makes it more probable?) "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions"--does that not strike you as insane? I wouldn't not mention it, but it is totally unconscionable to allow lunacy to be perpetuated. Despite what I just said, all other things being equal, I am going to give more weight to something out of a top-tier institution, but I'm going to examine each item on its own merit. Nobody gets a pass. Let me give another example of something I could not allow to pass, although it comes from a highly reputable institution: http://www.ncbi.nlm.nih.gov/pubmed/20043074, from MD Anderson. (Fortunately, that "study" did result in a fairly instant quotable counter-argument, such as at http://scepticsbook.com/2010/02/14/a-giant-leap-in-logic-from-a-piece-of-bad-science/, which also has the pdf of the original paper.) If there were no quotable counter-arguments, would Misplaced Pages have to go into the business of letting people believe that a solution containing no molecules of active ingredients is somehow capable of killing breast cancer cells?

I think we also need better definitions than "reputable" medical journals (what is disreputable, and to whom?), and "widely recognised" standard textbooks (widely recognized by whom), "disreputable" journals or "disreputable" fields or "respected" publisher. Maybe there is an explanation for those terms somewhere; my own feeling is like Justice Potter Stewart: "I know it when I see it." We do have to maintain the attitude the consensus might be wrong on something, as well as what I have found, that even the woo usually has some grain of truth to it.

GeorgeButel (talk) 05:16, 3 May 2012 (UTC)GeorgeButel

Because it isn't necessary. Because it is the same kind of argument that a POV pusher will use and you will fail to convince him just like he will fail to convince you. If you disagree with the conclusions of a source, then so will other people. If lots of people agree with your view, then it shouldn't be hard to prove that using a source-based argument rather trying to explain the merits or deficiencies of the source/study. If you can't find other experts writing about this issue in a way that agrees with you, then either the article should contain the alternative viewpoint, or perhaps the whole issue doesn't deserve the weight editors think it does.

Selecting sources, in general, is only the start. Then you need to work out what the article is going to say. Sourcing debates, without context, are somewhat pointless. The discussion didn't focus on the real questions facing article writing: What exact facts or other point would someone wish to draw from that source, and how would we write it? Is that the best source we can find for those facts (we already know it is 8 years old, which counts against it)? What do other high-quality sources have to say about that fact? Even if the fact isn't disputed, what weight is given to it in high-quality sources that discuss the general topic surrounding the fact, or the article itself? As a quick-and-dirty exercise, I searched the textbook I linked above, using Amazon, to see if it cites that paper or its authors. It doesn't. If a 300,000-word treatise on the article topic fails to use that source, why should we in our 4,000-word piece? Colin° 12:46, 2 May 2012 (UTC)

(edit conflict)

The thing is not that expert editors shouldn't conduct such peer review. Indeed they are free to do so as a reviewer in the real academic publishing world, not on Misplaced Pages. Of course having done so, they should be cautious of a new COI on Misplaced Pages. That paper they reviewed off-wiki doesn't become any more (or less) reliable or relevant on-wiki. Leave its selection up to other editors. So perhaps the sentence should be: " Misplaced Pages is not the place to perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. " LeadSongDog come howl! 13:06, 2 May 2012 (UTC)

That's a definite improvement. We should never try to dictate what editors can do outside of WP. Jakew (talk) 15:29, 2 May 2012 (UTC)

In general I find that incidents like this are best dealt with by finding sources that criticize the original source and either juxtaposing the two in text, or on the basis of the validity of the criticism, not opting to include the study (for instance, specific to Morgan, Ward & Burton, there is a post by Orac on scienceblogs as well as a letter to the editor). In this case, if I cited MW&B at all I'd phrase it as a minority point of view ("In 2004 WM&B said...") with an accompanying reference to the LTTE ("Mileshkin, Rischin, Prince & Zalcberg said in a LTTE that...") It's pretty rare that an article genuinely bucking the mainstream POV is unanswered. In my experience, situations like these that are unanswerable are the minority - actual experts are not stupid, they usually rebut the lunatic fringe stuff.

Regards this section of MEDRS, I wouldn't want to throw the baby out with the bathwater - I've run into far more issues where I've referred to that line/idea as a way of keeping in mainstream views ("You don't get to say that review article can't be included because some of the funding came from Pfizer"). It might be worth refering to WP:FRINGE or WP:UNDUE, since that's the nub of the issue in this case. The idea that chemotherapy is worthless is a fringe idea, most doctors clearly think there is merit to it. WLU (t) (c) Misplaced Pages's rules:/_complex 17:35, 2 May 2012 (UTC)

I don't understand WhatamIdoing's point (if I'm following correctly); sourcing is always a matter for consensus and discussion, and in any realm of editing (not just medical), editors can decide via consensus that a source is not appropriate. This guideline can't be excluding something that is general; we can always discuss, review, and come to consensus to not use a bad source. I must be missing something, because this discussion makes no sense to me. There are plenty of sources about Tourette's, for example, that even though high quality, are just wrong and shouldn't be used. For example, the New England Journal of Medicine published a review about 10 years ago that had the wrong basic definition of TS; editors can review that, see that, and decide not to use a source that is just wrong. Why are we saying editors can't come to consensus to not use a bad source on medical articles, when we can do that anywhere else? SandyGeorgia (Talk) 18:22, 2 May 2012 (UTC)

The guideline refers to "study". I think "sourcing", "study" and "facts" are getting mixed up here. Discussing sourcing without context can only deal with generalities like the type of source or type of publication. Some of our best writers are experts in the subject. Some of our worst POV pushers are experts in the subject. Both can produce lengthy talk page discussions that don't actually focus on proposed article text. What facts/points do we want to say? Are current books and reviews on the topic mentioning it? If not, then let's move on. Colin° 20:09, 2 May 2012 (UTC)

Sandy, the first objection to the source on the talk page was that if patients knew the facts about (cytotoxic) chemotherapy's limited contribution to cancer treatment (in most, but not all, solid tumors, surgery is far more important), then some of them would refuse treatment. Do you think that making patients make the "right" choice in the real world is a legitimate reason for entirely removing all information about efficacy from the article? I don't, but that's what's happened.

The TS comparison is inapt, because in that case, you had dozens of sources that gave a different definition. You could compare it to other sources and choose the many rather than the one. Here, we have just one source (that anyone knows of) that provides the overall statistic. (The point behind the paper was to calculate the overall benefit of all cytotoxic chemotherapy regimens for all patients in all stages of all invasive solid tumors.) It's not like this paper says 2% of survival the five-year mark, and another paper says 5%; this is the only paper that seems to have tried this ambitious approach.

There are good reasons to be careful about how it is presented: it applies only to invasive solid tumors, which means no leukemias (and so understates the benefit) and very little skin cancer (and so overstates the benefit). It considers only cytotoxic drugs, which excludes most of the newer stuff (hormones, monoclonal antibodies, radioactive drugs, etc.). It is looking for a population-wide number, so it includes even people who didn't need or didn't take chemotherapy (this is the difference between "pounds of beef eaten per American" and "pounds of beef eaten per beef-eating American").

These qualifications seem to form the basis of the technical criticisms: they chose to exclude hematological malignancies and non-invasive cancers (both very typical exclusions, by the way), and I would have made a different choice (as a point of fact, I personally would have made a different choice), so it's an unreliable source. They chose to consider only cytotoxic drugs, and I would have made a different choice (actually, I wouldn't have), so it's an unreliable source. They chose to consider all patients rather than some patients, and I would have made a different choice (I would have, too), so it's an unreliable source. NB that every complaint here is not "they got these facts wrong", but "they made different choices than I would have made".

Given that patients often believe chemotherapy is absolutely necessary, sometimes to the point of seeking it out despite multiple oncologists telling them that it is not appropriate, I think it sad that we now have an article that doesn't even mention undisputed facts about efficacy, such as that chemotherapy is hugely important to some cancers (e.g., testicular cancer), but provides very little benefit for others (e.g., lung cancer), even though not one of the people who dislikes this particular source believes that chemotherapy provides identical benefits for all forms of cancer. WhatamIdoing (talk) 22:16, 2 May 2012 (UTC)

The problem with "editors can decide via consensus that a source is not appropriate" is that sooner or later, you'll find yourself on the Abortion talk page where a group of editors have decided by consensus that a secondary, well-conducted review by the Guttmacher Institute is not appropriate "because the Institute is biased towards pro-choice". Or if not that page, then another where a group of SPAs have established their walled garden. I'm sorry, but we need MEDRS as a bulwark against POV-pushers, who would love the opportunity to disallow any reliable secondary source that didn't align with their own views. --RexxS (talk) 23:16, 2 May 2012 (UTC)

(edit conflict) @WhatamIdoing: I think we solve that problem by ensuring that our articles on testicular cancer, non-small-cell lung cancer, etc. each accurately reflect the role and efficacy of chemotherapy in those diseases. The parent article on chemotherapy should make clear that the efficacy of chemotherapy varies substantially by diagnosis and disease stage, and the details should be handled on a disease-by-disease basis. MastCell  23:19, 2 May 2012 (UTC)

That would be fine with me—I have repeatedly said that the 2% statistic is unimportant—but you actually removed every single sentence in the article that "mae clear that the efficacy of chemotherapy varies substantially by diagnosis and disease". WhatamIdoing (talk) 16:33, 4 May 2012 (UTC)

RexxS, WP:CONSENSUS is a policy. In general, it trumps WP:MEDRS. While there may be a cabal of SPAs who are trying to subvert an article, WP:MEDRS cannot be used to overrule them. Such a cabal can only be suppressed by drawing on a wider consensus with more good-faith editors. Axl ¤ 23:45, 2 May 2012 (UTC)

We already have a safeguard against that. Established consensus, in the form of policies and guidelines, trumps local consensus. Jakew (talk) 07:17, 3 May 2012 (UTC)

RexxS, welcome to the slippery slope that is all of Misplaced Pages (POV pushers gang up on good faith editors on talk pages-- try editing Venezuelan articles for a year or two). If you want to use MEDRS to trump CONSENSUS, pretty soon a carefully crafted and well applied guideline will fall into disfavor because it's used as a bludgeon and held to a standard higher than the policies it guides. AKA, shooting ourselves in the feet. MEDRS is a guideline, it enjoys consensus as such, it doesn't trump policy, and if folks start using it inappropriately, they're going to ruin a good thing. I guess I'm not understanding this argument, because whether one review is better, more recent, whatever is something that we can always discuss on talk and those discussions guide our decisions about which reviews to use. SandyGeorgia (Talk) 03:02, 3 May 2012 (UTC)

I'm sorry you thought my contribution was "really stupid", per your edit summary, Sandy. I can assure you that I have the scars from defending NPOV on numerous articles over several years, and we don't need to engage in a pissing contest over who does the most defending – but thank you for the belated welcome anyway. Standing up for NPOV is not a card game, and there are no trumps, but you must recognise that the project-wide consensus enjoyed by MEDRS is not to be ignored lightly by a small group arguing a WP:LOCALCONSENSUS: "Consensus among a limited group of editors, at one place and time, cannot override community consensus on a wider scale" (and that's policy). If every good-faith editor decides it's better not to argue the merits of MEDRS for fear of it falling into disfavour, then we might as well hand over the encyclopaedia to the POV-pushers right now. The practices documented at MEDRS are exactly what we have to convince other editors to use when building a consensus for the neutral point of view, because they are inimical to the methods used to push POV. The problem is you want it both ways: you want to be able to use your expertise to discard a poor secondary source, while expecting others to defend good secondary sources that have been labelled "poor" by a self-proclaimed expert. Have the lessons of Essjay still not been learned here? --RexxS (talk) 12:40, 3 May 2012 (UTC)

I'm disappointed by WhatamIdoing's response as she's falling into the same trap: using her expert knowledge to try to convince other editors, on a talk page, to include/exclude certain facts from the article. And to focus on the good/bad points in a study (or present counter-arguments against other editor's points) rather than discuss what we want to actually say and then present sources that support it. This is something Eubilides was great at: sourced-based discussion of actual or proposed article text.

WP:WEIGHT is policy. It says

An article should not give undue weight to any aspects of the subject but should strive to treat each aspect with a weight appropriate to its significance to the subject....Keep in mind that, in determining proper weight, we consider a viewpoint's prevalence in reliable sources, not its prevalence among Misplaced Pages editors or the general public... Also, if you are able to prove something that few or none currently believe, Misplaced Pages is not the place to present such a proof. Once it has been presented and discussed in reliable sources, it may be appropriately included.

I agree that it is very important how the facts in this case are presented, and that the disputed paper's abstract does an appalling job of that. The exclusions (particularly of modern drugs) are problematic and it will be hard to work out what should be said that is actually relevant to a modern reader in a general article on chemotherapy. But hey, I've got an idea. What do modern texts on chemotherapy say on the subject? It keeps coming back to this. Why on earth are our editors arguing over the merits of an 8-year-old study as a source for an article on chemotherapy in 2012. And since the benefits of chemotherapy seem to depend completely on what cancer is involved, the approach taken by the book I linked (where each type of cancer gets its own chapter) seems to be ideal for writing targeted text on each indication or otherwise for chemotherapy. Colin° 08:01, 3 May 2012 (UTC)

Agree if there are better more recent secondary sources we should be using them. Our guideline recommends against sources older than 5 years when more recent sources are available which I am sure is the case for chemo.--Doc James (talk · contribs · email) 15:05, 3 May 2012 (UTC)

(edit conflict) Agree that Colin's book suggestion could be a good starting point in this particular case (though as a general bibliographic point, perhaps it's worth pointing out that the quality of scientific medical books can vary from standard reference texts, like this, to pot boilers with little or no peer review). Ideally, one would like to have a widely respected reference textbook which is periodically updated. Even so, working up a single, high-quality page on such an extensive topic as chemotherapy remains a major task. This is a genuine issue, imo, because in practice generic pages such as Chemotherapy can be widely consulted and regard matters of considerable real-world interest. 2c only, —MistyMorn (talk) 11:32, 4 May 2012 (UTC)

So what exactly should be changed and how?

1. Editors should not perform a detailed academic peer review.
2. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

Sentence (1) can be left out, it has no practical consequences. Sentence (2) - if a study has inclusion criteria, methodology or factual errors leading to generally implausible results than apply WP:FRINGE.

Sometimes the results are not even generally faulty, but study design or data (in)availability make them useless for encylcopedic purposes. As an example, endometriosis has a Cochrane Review of TCM as a reference. This kind of source is in my opinion fairly useless - even if TCM may somehow help endometriosis the design of the study is not good enough for anything except "a call for further research" which does not belong there. Richiez (talk) 11:22, 4 May 2012 (UTC)

I'm not sure that removing the first sentence would be a good idea. As I interpret it, the current text states that editors shouldn't perform a detailed peer review and then lists as examples some personal objections that are invalid. If the first sentence were removed, then the resulting text would read as though these four types of objections were an exhaustive list of the only invalid grounds for exclusion. But they aren't: for example, editors shouldn't exclude papers on grounds that they disagree with methodology (except the general type of study), assumptions, statistical analysis, ethical problems, etc. This follows naturally from WP:NOR, so it isn't strictly necessary to spell it out, but it's certainly useful to make it clear. Jakew (talk) 11:46, 4 May 2012 (UTC)

Oppose Richiez change suggestion. We don't want editors cherry-picking studies, which means we don't want them filling talk pages with their analysis of a study's failures. Full stop. It doesn't matter if the study is primary research or secondary analsysis of existing research. We build our articles on the work of expert writing, not by choosing which studies we think are good/relevant. You cannot apply WP:FRINGE based on your own view of what is fringe or not or by deciding that since you think the study is flawed, then it must be a fringe conclusion. The "Assess evidence quality" section is more background information than a primary guide to what to include in an encyclopaedia article. Have a look at a NICE guidleline like this one. The difference between the "NICE guideline" and the "Full guideline" is that the latter contains all the evidence for their guidance and how they went about measuring and collecting this evidence. It is a job for experts. It is a difficult job. It is not our job. Colin° 12:41, 4 May 2012 (UTC)

If we are disallowed to judge WP:FRINGE than we can either stop altogether or have a very short list of allowed sources such es Encyclopedia Britanica - and of course we can expand this to all of wikipedia not just medical articles. Richiez (talk) 12:54, 4 May 2012 (UTC)

We identify fringe theories/conclusions by the fact that (a) most reliable sources tell us they are wrong/fringe or (b) most reliable sources ignore the theory/conclusion -- in other words it has no weight to warrant mention in our articles. We don't identify them based on our own understanding. I dare say parts of the US think evolution is a fringe idea... Colin° 18:20, 4 May 2012 (UTC)

I'm with Colin. We don't write a guideline around primary studies because we typically shouldn't be using them anyway-- we use secondary sources and reviews, and we do come to consensus about which reviews are the highest quality. All of this discussion seems to have resulted because folks want to use and do original research with primary sources. SandyGeorgia (Talk) 14:08, 4 May 2012 (UTC)

I did not follow the chemotherapy discussion closely but it was my impression that it was a secondary source in question with results that happened to fall into WP:FRINGE category. Richiez (talk) 16:07, 4 May 2012 (UTC)

It's not really FRINGE. It's just a broader focus than other papers have taken, sort of like "how much has society benefitted frim vaccines" rather than the far more common "how much has society benefited from the polio vaccine". WhatamIdoing (talk) 16:33, 4 May 2012 (UTC)

It appears to be very much against mainstream in some respects. In breast cancer we try to differentiate by age, receptor status, markers or more recently gene activation patterns, response to neoadjuvant therapy and I think everyone is aware that the benefit of chemotherapy is likely to be dramatically different comparing a 40 year old or 85 year old women. It was my impression that the mentioned paper did the exact opposite of this approach by using unusually broad undistinguished patient population.

Now hypothetically assume there is a review that meets all formal criteria of MEDRS and is so fringe that "mainstream" does not really notice, let alone review it. With many thousands (somewhere I recall 33K academic journals?) this might happen from time to time. Applying ] seems obvious? We have to use common sense. -- Richiez (talk) 21:10, 4 May 2012 (UTC)

Whether you regard it as fringe or something of no weight doesn't matter much. The issue for this guideline is that this must be judged with reference to our sources, not on the basis of our own opinions. This is the issue I had with "if a study has inclusion criteria, methodology or factual errors leading to generally implausible results than apply WP:FRINGE". Colin° 21:20, 4 May 2012 (UTC)

Why exactly should WP:MEDRS be exempt from the fringe rules? The essence of fringe is that someone, preferably by consensus regards some material as such. Richiez (talk) 21:39, 4 May 2012 (UTC)

You're not getting the point, Richiez. WP:FRINGE is a valid guideline but if you read it you'll discover that determining what is fringe or not is done with reference to published work, not by WP editors using their own brains to discredit some theory/conclusion. My second point is that this particular point on Chemotherapy doesn't have to be as extremely wrong as a fringe idea in order for us to give it no space. If none of our contemporary sources choose to comment on chemotherapy in the way that 8-year-old paper did, then neither should we. No matter what you or WhatAmIDoing personally think of the study's merits or flaws. Colin° 09:29, 5 May 2012 (UTC)

I support Richiez's recommendation.

" We don't write a guideline around primary studies because we typically shouldn't be using them anyway. "

— SandyGeorgia

That's a self-referential statement. We shouldn't use primary studies because our guideline doesn't recommend them? Anyway, this discussion is not about primary sources.

" All of this discussion seems to have resulted because folks want to use and do original research with primary sources. "

— SandyGeorgia

That is entirely false. No-one has recommended the use of primary sources, nor is anyone recommending original material in an article.

WP:FRINGE includes the statement "We use the term fringe theory in a very broad sense to describe ideas that depart significantly from the prevailing or mainstream view in its particular field." Morgan's paper does indeed depart from the mainstream view, thus it fits this definition of "fringe". Axl ¤ 22:12, 4 May 2012 (UTC)

Axl: see my comment above. If his paper "does indeed depart from the mainstream view" then this should be evidence in the literature (either explicitly or because it is ignored) and therefore it is unnecessary for WP editors to attempt to prove this on a talk page with their own skill. When editors try to push other editors into accepting a viewpoint, without reference to contemporary high-quality sources, then that is POV pushing. Just because it is mainstream POV pushing, doesn't stop it from being something to discourage. Colin° 09:29, 5 May 2012 (UTC)

If you accept "beeing ingored" as evidence in literature than this would appear as reasonable - in theory. When you try to define this properly I expect it will turn out more difficult than a simple apply common sense rule. Eg how long after publication do you want to wait as evidence that something is ignored? We would be forced to publish something and wait 3-5 years for evidence it is ignored (and hope there is not a citation cartel at work)? Richiez (talk) 12:20, 5 May 2012 (UTC)

I've seen this problem before. The disputants will not agree on what is "mainstream" - one side says it's is; the other side insists it's not. If we let editors define mainstream, we never reach consensus. Where the editor's expertise comes in is to point others to how prevalent a view is in the quality literature (which is how we should be defining "mainstream"). If you're lucky, you find one review published in an in-house magazine and a blog, and the contradictory review published in the Lancet and the BMJ. Unfortunately it's often not so clear-cut. --RexxS (talk) 02:22, 5 May 2012 (UTC)

I've found taking a 'vote count' from a sample of reviews on PubMed to be a reasonable solution to this problem. It has its own problems, of course, but to get a rough measure of mainstream thinking it works quite well, and is obviously less susceptible to bias than relying on an editor's expertise. Jakew (talk) 07:21, 5 May 2012 (UTC)

sure that does happen, but I think there are many cases left where there is no doubt about what is not mainstream. Richiez (talk) 12:24, 5 May 2012 (UTC)

Observational studies (and CAM)

Although this is written specifically in the context of CAM (not about it...), I suspect we could use this principle in this guideline:

"Although observational studies cannot provide definitive evidence of safety, efficacy, or effectiveness, they can: 1) provide information on “real world” use and practice; 2) detect signals about the benefits and risks of complementary therapies use in the general population; 3) help formulate hypotheses to be tested in subsequent experiments; 4) provide part of the community-level data needed to design more informative pragmatic clinical trials; and 5) inform clinical practice."

Source: Observational Studies and Secondary Data Analyses To Assess Outcomes in Complementary and Integrative Health Care. June 25, 2012, Richard Nahin, Ph.D., M.P.H., Senior Advisor for Scientific Coordination and Outreach, National Center for Complementary and Alternative Medicine.

This is a RS from the NIH.

It clearly states that we wouldn't be able to use observational studies as RS to make statements of fact about "safety, efficacy, or effectiveness". This actually applies to far more than just CAM, and is very relevant for this guideline. How can this be incorporated here? -- Brangifer (talk) 01:14, 1 July 2012 (UTC)

Any individual study is a primary source, including observational studies, and this guideline strongly urges the use of secondary sources, such as research reviews and meta-analyses. So in a way it's already covered. But we could add it to this sentence: "Roughly in descending order of quality, lower-quality evidence in medical research comes from individual RCTs; other controlled studies; quasi-experimental studies; non-experimental studies such as comparative, correlation, and case control studies; and non-evidence-based expert opinion or clinical experience. " Perhaps after "other controlled studies." TimidGuy (talk) 19:49, 1 July 2012 (UTC)