Misplaced Pages talk:Identifying reliable sources (medicine): Difference between revisions

Browse history interactively ← Previous edit Next edit →Content deleted Content addedVisual WikitextInline

Revision as of 15:12, 1 July 2012 editNenpog (talk \| contribs)453 edits →MEDRS - too complex?← Previous edit		Revision as of 19:49, 1 July 2012 edit undoTimidGuy (talk \| contribs)Extended confirmed users, Pending changes reviewers11,258 edits →Observational studies (and CAM): suggestion how to incorporate into guidelineNext edit →
Line 218:		Line 218:

	It clearly states that we wouldn't be able to use ] as RS to make statements of fact about "safety, efficacy, or effectiveness". This actually applies to far more than just CAM, and is very relevant for this guideline. How can this be incorporated here? -- ] (]) 01:14, 1 July 2012 (UTC)		It clearly states that we wouldn't be able to use ] as RS to make statements of fact about "safety, efficacy, or effectiveness". This actually applies to far more than just CAM, and is very relevant for this guideline. How can this be incorporated here? -- ] (]) 01:14, 1 July 2012 (UTC)

			:Any individual study is a primary source, including observational studies, and this guideline strongly urges the use of secondary sources, such as research reviews and meta-analyses. So in a way it's already covered. But we could add it to this sentence: "Roughly in descending order of quality, lower-quality evidence in medical research comes from individual RCTs; other controlled studies; quasi-experimental studies; non-experimental studies such as comparative, correlation, and case control studies; and non-evidence-based expert opinion or clinical experience. " Perhaps after "other controlled studies." ] (]) 19:49, 1 July 2012 (UTC)

Revision as of 19:49, 1 July 2012

This is the talk page for discussing improvements to the Identifying reliable sources (medicine) page.

Put new text under old text. Click here to start a new topic.
New to Misplaced Pages? Welcome! Learn to edit; get help.

Shortcut

WT:MEDRS

To discuss reliability of specific sources, please go to
Misplaced Pages:Reliable sources/Noticeboard
or the Wikiproject talk pages of WT:MED or WT:PHARM.

This project page does not require a rating on Misplaced Pages's content assessment scale.
It is of interest to the following WikiProjects:

Medicine

This page is within the scope of WikiProject Medicine. Please visit the project page for details or ask questions at Misplaced Pages talk:WikiProject Medicine.MedicineWikipedia:WikiProject MedicineTemplate:WikiProject Medicinemedicine

Pharmacology

This page is within the scope of WikiProject Pharmacology, a collaborative effort to improve the coverage of Pharmacology on Misplaced Pages. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.PharmacologyWikipedia:WikiProject PharmacologyTemplate:WikiProject Pharmacologypharmacology

Archives

Archive 1	Archive 2	Archive 3
Archive 4	Archive 5	Archive 6
Archive 7	Archive 8	Archive 9
Archive 10	Archive 11	Archive 12
Archive 13	Archive 14	Archive 15
Archive 16	Archive 17	Archive 18
Archive 19	Archive 20	Archive 21
Archive 22	Archive 23	Archive 24
Archive 25	Archive 26	Archive 27
Archive 28	Archive 29	Archive 30
Archive 31	Archive 32

This page has archives. Sections older than 60 days may be automatically archived by Lowercase sigmabot III.

"Unless the primary source itself directly makes such a claim"

The sentence "Individual primary sources should not be cited or juxtaposed so as to "debunk" or contradict the conclusions of reliable secondary sources" is wonderfully clear, and I've been wondering whether to incorporate it in WP:PSTS (on grounds that it's a logical consequence of existing policy, and hence applies to all articles). However, the final part of the sentence ("unless the primary source itself directly makes such a claim") doesn't make sense to me. If a source debunks, contradicts, or otherwise comments on the conclusions of a secondary source then it isn't a primary source. It's a secondary (or tertiary) source. What is it intended to mean, and how can it be rephrased to better express that meaning? Jakew (talk) 09:58, 24 April 2012 (UTC)

Agree we should get rid of it. --Doc James (talk · contribs · email) 10:37, 24 April 2012 (UTC)

(outdent) It means, if a primary source directly criticises a secondary source, for example, a primary source, may point out undeclared conflicts of interests or perceived flawed methodology used by reviewers or confounding variables etc. The primary source would basically be saying the review by such and such group of researchers is flawed because,,,,. The primary source would have to specifically mention a secondary source by name. It is NOT the same as using a primary source such as an individual study to debunk a review of studies. I don't think the line should be got rid of but perhaps it is possible to better word it?--Literaturegeek | T@1k? 23:55, 24 April 2012 (UTC)

But many of these comments are not WP:DUE. For example we have this report by the AHRQ which was criticised by one of the leaders of the religious movement in question --Doc James (talk · contribs · email) 06:10, 25 April 2012 (UTC)

Ah. As I see it, whether a source is primary or secondary depends on how it is used. If we cite a source for its criticism of another source, then we aren't using it as a primary source. We're using it as a secondary source. That's why I find it really confusing to see it described as a "primary" source. I think we've got used to thinking "primary source = anything in a primary research study, secondary source = anything in a review", which is often convenient, but it's an oversimplification. As Doc James points out, though, there are frequently undue weight issues associated with citing primary research studies.

How about rephrasing: Individual primary sources should not be cited or juxtaposed so as to "debunk" or contradict the conclusions of reliable secondary sources. Explicit criticism of secondary sources, however, may be included if appropriate, but be sure to assess due weight. Jakew (talk) 07:32, 25 April 2012 (UTC)

I don't agree with the suggested "Explicit criticism of secondary sources, however, may be included if appropriate". We really should try to mostly not discuss the sources in article text at all. Sometimes it is useful to explicitly mention research studies and meta analyses and even literature reviews, in text, but generally we should be in the business of describing facts, not the means by which folk discovered or write about those facts. So I don't want to encourage editors to argue/prove a point in front of the readers.

Note that the guideline text does not say the primary source debunks the text of the secondary source, or the methodology or even the moral character of the authors (that would, indeed, make it a secondary source on the review it was debunking. It says it explicitly debunks or contradicts the conclusions in the secondary source. I think what we're trying to avoid is something like this garbage: "A Cochrane review in 2006 concluded that drinking XXX fruit juice had no protective benefits on YYY cancer. However, recent research shows that XXX fruit juice contains aaa compound , which is protective against cancer . Neither primary research paper directly or explicitly contradicts the conclusions of the review. However, if somone did a long-term study of people who regularly drink XXX fruit juice and those who didn't, and found and stated that it did indeed appear to have protective benefits on YYY cancer.... That recent research could potentially be used to debunk the old review. It would be explicitly debunking/contradicting the conclusions, but doesn't even need to mention previous reviews, so doesn't become a secondary source. Colin° 12:04, 25 April 2012 (UTC)

When we talk about directly making the claim that it de-bunks older works, I think we are looking for a paper that says something like, "The received wisdom in the field (as seen in every textbook for the last 100 years) says that patients should not be permitted to eat anything for 24 hours after major surgery. However, as far as we can make out, this popular old idea is based on zero evidence, and we've actually done a proper randomized, controlled trial, which we outline here, and the data produced says that the old surgeons' ideas about low diet is a bunch of bunk."

Our approach to using them (the textbooks and the RCT paper) would then look something like "Keeping the patient on a low diet after major surgery has been widely recommended, but newer research suggests that it may be unnecessary."

If, on the other hand, we had the same study, but the authors made no direct mention of how their study supersedes the older works, then we would probably ignore it (for now): "Keeping the patient on a low diet after major surgery has been widely recommended." WhatamIdoing (talk) 13:51, 25 April 2012 (UTC)

I disagree. I don't think there is any requirement for the new source text to explicitly name the debunked sources or to imply them with a remark like "every textbook for the last 100 years". The "de-bunks older works" comment falls into the same trap of confusing source text/work with the facts it presents. Criticism of older "works" might happen in a field like history, say. But research in medicine would tend to overturn previous "facts" rather than "works". Also, there's nothing the research authors can say or not say that makes any difference to the WP:WEIGHT we give to their findings. So the argument that, depending on what they say, we could ignore them or not, doesn't work. The guideline we are discussing is preventing original research based on primary sources. It is a separate issue to consider whether that primary research paper's conclusions have enough weight to warrant mention. Colin° 15:22, 25 April 2012 (UTC)

I have to say, I'm not comfortable with citing primary sources in either situation.

In the case that the authors assert that their study is more significant than the sum of previous work, their assessment is unlikely to be wholly objective. They may well be correct, but it wouldn't be surprising if their view of the importance of their work was a little out of proportion. We're all prone to thinking our work is more important than it really is. I would generally think it best to wait until a secondary source cites their paper and judges the significance of their work from a less biased perspective.

In the case that they make no particular assertion, I'd be very concerned if editors were to make that assessment themselves. True, much of the time the decision will be rational and in accordance with principles of evidence-based medicine, but many controversial topics are subject to widespread POV pushing, and it would be best to avoid long-winded debates full of original research about the relative importance of various studies. Again, why not just wait until a secondary source becomes available? It's unlikely to be that urgent, after all... Jakew (talk) 09:35, 27 April 2012 (UTC)

I agree with those above who think that the phrase "Unless the primary source itself directly makes such a claim" is more problematical than helpful. I think there is a consensus for its removal, so I'm going to test that by removing it. I understand that it is possible that a better worded qualification might be preferable, so feel free to revert me and continue discussion here if I've misjudged the opinions expressed so far. --RexxS (talk) 15:32, 2 May 2012 (UTC)

MEDRS - too complex?

I think the guidelines has become too complicated for its own good, specifically about primary sources. For example this vital article http://news.harvard.edu/gazette/2006/12.07/11-dairy.html would be valid in most places but not here, this really concerns me and shows the flaws of Misplaced Pages and the urgent need for revision of the guidelines in general of any field. Helios solaris 16:11, 26 April 2012 (UTC)

It is complex because the problem is complex. The article you link is a newspaper/magazine type of article based on a talk someone gave in their lunch hour. Why do you think a serious encyclopaedia should base its articles on that kind of "I've got a radical new idea and have cherry-picked a bunch of studies that support it" kind of journalism. -- Colin° 16:36, 26 April 2012 (UTC)

Because if not it will fall behind. Helios solaris 17:33, 26 April 2012 (UTC)

I don't mind "falling behind" speculation and cherry-picking. The researcher in your source says that her hypothesis is unproven and not ready for prime time.

That said, this source is usable, for certain limited purposes. You could use it, for example, to support a claim that research is being conducted on the relationship between dairy consumption and hormone-sensitive cancers. You can't use it for a claim that any such link has been proven, or that readers should change their diets. WhatamIdoing (talk) 19:04, 26 April 2012 (UTC)

There is insufficient evidence of WP:DUE. We have much better sources discussing what research is ongoing. --Doc James (talk · contribs · email) 10:19, 27 April 2012 (UTC)

When you realise how hard it is to find secondary sources in the way that pleases Misplaced Pages and add it, even then it's suppressed, it becomes obvious that they are suspiciously superfluous. They exist only to subdue true evidence and strenghten the corporate establishment, just remember the tobacco industry do I need to say more. Critical thinking is clearly not welcomed here. --Helios solaris 16:25, 27 April 2012 (UTC)

There's nothing wrong with critical thinking, we just don't let it affect our edits. WP:NOR is long established policy for one very good reason: when "anyone can edit", some anonymous/pseudonymous editors will be incapable of critical thinking. If we allow it for you, we have to allow it for everyone, including those who can't think at all. Instead, we insist on wp:verifiability and secondary reliable sources. It works, though we have to keep explaining why we do it. LeadSongDog come howl! 19:34, 27 April 2012 (UTC)

I agree with Helios solaris. I witnessed in here how the rules were used in order to serve commercial interests. This is in particular against public interest, when it comes to adverse effects to the commercial products.

Important side effects of the absorbed radiation, of a dose that CT impose, were not accepted, because of bad interpretation of the rules. The side effects included the extent of the caused DNA damage, cataracts, circulatory problems, and cognitive impairments. --Nenpog (talk) 15:12, 1 July 2012 (UTC)

Section tagging

Various templates (such as {{MEDCN}}, {{MEDRS}}) are available to tag reliable source problems inline. Is there one to flag up a whole section, i.e. some "medical source" version of {{Refimprove}}? If not, I think there should be. Peter coxhead (talk) 10:34, 1 May 2012 (UTC)

Sounds like a good idea. --Doc James (talk · contribs · email) 10:36, 1 May 2012 (UTC)

Well, if there isn't one already, how about something like my draft at User:Peter coxhead/Test/T1? As with {{refimprove}}, on which it is based, using the 1st parameter |section will replace "article" by "section". Peter coxhead (talk) 11:12, 1 May 2012 (UTC)

{{refimproveMED}}, though I don't know whether it has a section parameter. Nikkimaria (talk) 13:30, 1 May 2012 (UTC)

{{Medref}} (where did that refimprove come from and which do we prefer), but no section parameter. SandyGeorgia (Talk) 13:32, 1 May 2012 (UTC)

Neither of these have section parameters, although they could be added. {{RefimproveMED}} doesn't do what I want, because it says that the article is a health or medicine one, whereas I want a template to be added to e.g. a plant article that has a section on its supposed health benefits. {{Medref}}, too, suggests more to me that the whole article is a medical one; I'm not sure that WikiProject Medicine would really want to get involved in sorting out a few odd herbal medicine claims added to what is mainly a plant article.

But I don't want to create yet another template unnecessarily, so what do others think? Could {{Medref}} be worded so as to be a bit more appropriate to a non-medical article? I do think it's worth including this quotation "Biomedical information in articles must be based on reliable, third-party, published sources and accurately reflect current medical knowledge" especially in a warning meant for a non-medical article. Editors who add this kind of information to plant articles simply don't know about this requirement. Peter coxhead (talk) 14:23, 1 May 2012 (UTC)

Medref now has a section parameter: {{medref|date=May 2012|small=1}}. Cheers, — Bility (talk) 16:27, 1 May 2012 (UTC)

Medref should now be appropriate for use on non-medical articles, too. The wording settled on ("This article needs more medical references for verification.") and template doc I wrote ("This template is intended to be placed at the top of articles with medical or health content") should cover health sciences related content in an otherwise non medical article. Further tweaks can be made if necessary. It now has a reason parameter |reason=foo is bar for giving specific details. Thanks, --92.6.211.228 (talk) 18:15, 1 May 2012 (UTC)

Actually {{Medref}} doesn't have a section parameter. This line:

| text = This article '''needs more [[Misplaced Pages:Identifying reliable sources

needs article replaced by {{#if:{{{1|}}}|{{{1}}}|article}} exactly as in {{refimprove}}, {{no footnotes}}, {{more footnotes}}, {{how}}, etc. In all of these, if you use {{TEMPLATE-NAME|section}} the message becomes "This section needs more ..." instead of "This article needs more ..." Peter coxhead (talk) 21:57, 1 May 2012 (UTC) It does. Sort of. If you look at the testcases (down the bottom), you can see it's different to those templates in that the small/section version doesn't say "This article" or "This section". --92.6.211.228 (talk) 22:10, 1 May 2012 (UTC) (edit conflict) It's not a parameter called section that I meant, but a parameter you can use for the small version in sections, which is what I thought you guys were talking about. Right now the template uses this first unnamed parameter as the rationale, so you don't have to explicitly use |reason=. We can change this, although if you want to use the small version in a section, it's a moot point since that line won't be displayed. This would also mean any existing templates using the first unnamed parameter for their rationales would need to be updated to use |reason=. — Bility (talk) 22:36, 1 May 2012 (UTC) Added it. Hmm. It replaces the word article with the reason value when all params are used. That's fixable of course. Did you want it to replace |small? Allow both? --92.6.211.228 (talk) 22:27, 1 May 2012 (UTC)

I've added this test {{#ifeq:{{{1|}}}|section|section|article}} to {{medref}}. If, and only if, the first parameter is exactly the word "section", then the template will display "This section ..." instead of "This article ...". This change does not affect cases where the first parameter is a reason. I've used this in an article (Paris polyphylla#Medicinal uses, as of now) and it works fine. Peter coxhead (talk) 08:28, 2 May 2012 (UTC) Actually I had to make another change to stop the word "section" appearing as a reason as well as forcing "This section ..." when |reason= was absent. I've tested all the cases and it should be ok now. Peter coxhead (talk) 09:18, 2 May 2012 (UTC) I've also added information about {{Medref}} to the documentation for {{Refimprove}}, which is where I went in the first instance and failed to find the more specific template. Peter coxhead (talk) 09:35, 2 May 2012 (UTC)Add to Multiple issues?I think it might be worth adding it to {{Multiple issues}} in line with other standalone templates. Wording (examples) needs to be agreed on. Something like:It needs additional medical references for verification. or It relies on references which may not be reliable sources for medical or health content. or It includes attribution to sources which may not be reliable for medical content.Thoughts? --92.6.211.228 (talk) 18:45, 1 May 2012 (UTC)Yeah, I was planning on getting it set up along with expert-subject, which is also missing, since I've already added medref to the multiple issues template on a couple articles. EDIT: Actually expert-subject is in there, just called "expert". Anyway, I'll go add an edit request for medref. — Bility (talk) 20:52, 1 May 2012 (UTC)Just an FYI, the request to add medref to multiple issues was denied. I removed it from the three instances of multiple issues that I was aware of. Cheers, — Bility (talk) 16:55, 29 May 2012 (UTC)The guideline should be changed"Basic advice", subsection "Assess evidence quality", paragraph 4 includes this statement: " "Assessing evidence quality" means that editors should determine the quality of the type of study. Editors should not perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. " Recently, this study was proposed as a reference for the article "Chemotherapy". Several editors (including, latterly, myself) were critical of the study and objected to the use of the study as a reference. See full details here. WhatamIdoing quoted the above text from WP:MEDASSESS as justification for inclusion of Morgan's paper as a reference. In my opinion, the text fails to take into account those rare secondary sources that are severely biased or flawed, such as Morgan's paper. I see no reason why editors shouldn't perform detailed peer reviews if they so choose. Moreover, I believe that such reviews can be helpful in determining suitability as references for Misplaced Pages. Axl ¤ 19:41, 1 May 2012 (UTC)Are there other reviews that are more recent that contradict this paper from 2004? We do say evidence from the last 3-5 years. --Doc James (talk · contribs · email) 20:28, 1 May 2012 (UTC) I don't believe that there are any other reviews of this nature – because the whole premise of the review is flawed. Axl ¤ 20:31, 1 May 2012 (UTC)For information: the disputed text was added on 27 March 2010 by WhatAmIDoing. A talk page note at the time of the edit did not bring about any further discussion. However, the note refers to this earlier discussion and the whole section was added by Eubulides along with this discussion. -- Colin° 20:36, 1 May 2012 (UTC)I think we should allow editors to exercise judgement in deciding whether a secondary source is actually suitable. There are loads of secondary sources that are of insufficient quality. In that sense, the "peer review" should assess the quality of the source. Disputes as to the suitability should be resolved by consensus. In this case, there was consensus that the source was not suitable. JFW | T@lk 21:46, 1 May 2012 (UTC) Yes agree that consensus should be followed over the guideline. If the majority of editors do not see this ref as suitable than it should not be used. Does not necessitate a change in WP:MEDRS though. Many people still fell that WP:MEDRS says using primary sources is fine when high quality secondary sources are available which IMO is a more pressing issue.Doc James (talk · contribs · email) 21:54, 1 May 2012 (UTC) As per User:JFW and User:Doc James above. Of course secondary sources are vulnerable to bias (and original studies can contain high-quality 'secondary source' material). Metanalyses and other systematic reviews are genuine 'studies' in their own right. Since WP:MEDRS needs to be widely comprehensible and has to safeguard against real-world dangers such as POV pushing and poorly informed editing, it can only provide rather broad guidance. So well-informed talk page discussions can be key. —MistyMorn (talk) 11:24, 2 May 2012 (UTC)I share some of WhatAmIDoing's concerns (expressed elsewhere) that we have to come up with rules that work for the POV pusher and the undereducated. I'm one of the undereducated. My brain is much smaller than that of MastCell and Axl and WhatAmIDoing; I have no health training beyond a first aid badge when I was in the Scouts; I'm just someone's dad. Some of WP's rules are here purely because of the special situation with our authors: we don't really know who they are; we don't trust who they say they are; most of them are bonkers; some of them aren't geniuses. An editor who is an expert in the subject is likely to find some of these rules frustrating at times.The problem with the Chemotherapy paper raises two issues. The first is that academic papers, whether primary research or reviews, are written for a certain learned audience, and not for general consumption. That PubMed has made them (and particularly their abstracts) so accessible, is wonderful but at the same time very dangerous. As the article talk page shows, the abstract is over-simplified to the point of being misleading. And it takes someone quite knowledgeable to discover what the analysis actually looked at and is capable of saying. Combine this with the possibility that the paper is flawed/biased and we have a problem. Fortunately, it is not a common problem, and is less of a problem than editors trying to pick/review primary research papers.The second issue is the level of source we are using here. There's too much emphasis on review papers among WP:MED folk in my opinion. Yes, it is wonderful to be able to read all this material on your browser or PDF on the computer, and search it and collate it on your hard disk. I do so myself. But what about books, guys, books! Chemotherapy is a big subject. I just searched Amazon on it and turned up several suitable texts including ISBN 160831782X: "Handbook of Cancer Chemotherapy", edited by Roland T. Skeel, Samir Khleif, published 2011 (an earlier edition is actually used by the article for one point, but sadly the page number isn't given). This is 832 pages of wonderful readable and up-to-date-enough material. Approximately 300,000 words by a rough estimate. Now our Chemotherapy article is about 4,000 words. Which means you could condense that book 75x to achieve the current text. That's about 5 words from every page.The chemotherapy article is crap. There are many medical articles on WP I'd recommend to a friend, but that one isn't. Not by a long way. Why are we wasting days discussing the merits or otherwise of some 8-year-old review paper. Visit your university library bookshelves, or order a book (second-hand) from Amazon/etc -- you can always sell it later.WP:WEIGHT is underused in sourcing disputes. Rather than wikilawering over PSTS or disecting the journal paper or whatever, just find out "What do experts generally say, in published reliable sources, when writing about this subject". We are used to using WEIGHT to judge research findings: we read and use reviews/etc to find that. Similarly, we should use WEIGHT to judge analysis reports/reviews, especially if editors dispute them. What do our serious weighty oncology textbooks have to say on the subject? Rather than fall out with each other over such things, find some expert person/body who has already thought hard about it, published their work, and use that. WP:WEIGHT specifically warns against editors forming their own consensus about what issues/facts to include in an article, so I disagree with JFW/Doc James on that point -- however I suspect if you follow my advice, the review and its conclusions would be rejected anyway. Colin° 22:09, 1 May 2012 (UTC)I feel that, in this instance, the issue could be resolved by proper usage of existing guidelines. Colin suggested above several alternatives to giving infinite weight to a single source that is seemingly flawed. I briefly proposed another on Talk:Chemotherapy that involves exercising judgment not on the quality of the source, but whether the wiki text correctly reflect the conclusions being drawn by Morgan et al. To wit, is the source saying what the article is saying? In my mind, the source was misused because it was made to support a statement that is too broad. Unless a more systemic problem with WP:MEDASSESS is identified, I do not support making drastic changes to guidelines serving a useful purpose. Wafflephile (talk) 23:30, 1 May 2012 (UTC)I didn't intend to re-hash the debate about the (lack of) merit of Morgan's paper here. Rather I wish to consider the value of this statement from WP:MEDRS: " Editors should not perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. " Why shouldn't editors perform a detailed academic peer review, if they so wish? The second sentence is potentially misleading. With careful reading, it implies that only the type of study needs to be high-quality. As long as the source is a review article, it doesn't matter how poor its quality. It is inevitable that some secondary sources are better quality than others. The second sentence implies that we should give them equal weight, regardless of their quality. Axl ¤ 12:16, 2 May 2012 (UTC)Most editors aren't qualified, or indeed competent, to perform a detailed academic peer review, and we've no mechanism in place to identify those who are competent and give their views more weight. Consequently, we've no way of allowing experts to reject papers that are truly dire without also allowing POV pushers and fringe theorists to reject papers that they dislike, on the grounds of nothing but their own (often ill-informed or irrational) opinion. It's a similar situation to allowing editors to directly insert OR into articles: an appealing option in some respects, but dangerous in others. Jakew (talk) 12:28, 2 May 2012 (UTC)Does the fact that "most" editors aren't "qualified" constitute a justification for allowing lunacy to be perpetuated? If I see some kind of glaring flaw in a study or review, am I supposed to act like nothing's wrong? I always thought of Wiki articles as de facto reviews in some cases--balanced, dispassionate, factual surveys of whatever knowledge exists in a given area. Does the fact that my objection to a reviewer's statement that black is white has not yet resulted in a "comment" in a journal or an entry in a blog mean that, because I cannot quote an objection, that we have to pretend that black is white until we can document an alternative position? Here's an example: I discovered in http://www.ncbi.nlm.nih.gov/pubmed/3569020 that Burzynski, in 1987, theorized that one of his panacea molecules, #A10, interacts with DNA (which it does) to compete with carcinogens and thereby keep tumors from growing. The problem is that once tumors are formed, the carcinogen has nothing to do with it, and his molecules competing with carcinogens would have no conceivable bearing on the treatment of tumors (although, if correct, it is conceivable that they might be chemopreventive.) I have seen all kinds of criticism of Dr. B, but I have not seen anyone make that particular observation (maybe I missed it). If I were to be doing some kind of Wiki editing of his stuff (that is probably just as joyless as commenting on chemotherapy itself; I'm not even going to look), would I have to refrain from commenting on that? To me, it indicates an intellectual error that undermines the entire basis of what he thought he was doing. How could it be against Misplaced Pages policy not to point out the obvious? I don't think of Misplaced Pages as having to play the "emperor's new clothes" game.Even the most prestigious institutions sometimes publish material that cannot be allowed to be unquestioned, and certainly, we all know that possessing even a Nobel Prize does not immunize the recipient to error or to lunacy. (Dare I suggest that it makes it more probable?) "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions"--does that not strike you as insane? I wouldn't not mention it, but it is totally unconscionable to allow lunacy to be perpetuated. Despite what I just said, all other things being equal, I am going to give more weight to something out of a top-tier institution, but I'm going to examine each item on its own merit. Nobody gets a pass. Let me give another example of something I could not allow to pass, although it comes from a highly reputable institution: http://www.ncbi.nlm.nih.gov/pubmed/20043074, from MD Anderson. (Fortunately, that "study" did result in a fairly instant quotable counter-argument, such as at http://scepticsbook.com/2010/02/14/a-giant-leap-in-logic-from-a-piece-of-bad-science/, which also has the pdf of the original paper.) If there were no quotable counter-arguments, would Misplaced Pages have to go into the business of letting people believe that a solution containing no molecules of active ingredients is somehow capable of killing breast cancer cells?I think we also need better definitions than "reputable" medical journals (what is disreputable, and to whom?), and "widely recognised" standard textbooks (widely recognized by whom), "disreputable" journals or "disreputable" fields or "respected" publisher. Maybe there is an explanation for those terms somewhere; my own feeling is like Justice Potter Stewart: "I know it when I see it." We do have to maintain the attitude the consensus might be wrong on something, as well as what I have found, that even the woo usually has some grain of truth to it.GeorgeButel (talk) 05:16, 3 May 2012 (UTC)GeorgeButelBecause it isn't necessary. Because it is the same kind of argument that a POV pusher will use and you will fail to convince him just like he will fail to convince you. If you disagree with the conclusions of a source, then so will other people. If lots of people agree with your view, then it shouldn't be hard to prove that using a source-based argument rather trying to explain the merits or deficiencies of the source/study. If you can't find other experts writing about this issue in a way that agrees with you, then either the article should contain the alternative viewpoint, or perhaps the whole issue doesn't deserve the weight editors think it does.Selecting sources, in general, is only the start. Then you need to work out what the article is going to say. Sourcing debates, without context, are somewhat pointless. The discussion didn't focus on the real questions facing article writing: What exact facts or other point would someone wish to draw from that source, and how would we write it? Is that the best source we can find for those facts (we already know it is 8 years old, which counts against it)? What do other high-quality sources have to say about that fact? Even if the fact isn't disputed, what weight is given to it in high-quality sources that discuss the general topic surrounding the fact, or the article itself? As a quick-and-dirty exercise, I searched the textbook I linked above, using Amazon, to see if it cites that paper or its authors. It doesn't. If a 300,000-word treatise on the article topic fails to use that source, why should we in our 4,000-word piece? Colin° 12:46, 2 May 2012 (UTC)(edit conflict)The thing is not that expert editors shouldn't conduct such peer review. Indeed they are free to do so as a reviewer in the real academic publishing world, not on Misplaced Pages. Of course having done so, they should be cautious of a new COI on Misplaced Pages. That paper they reviewed off-wiki doesn't become any more (or less) reliable or relevant on-wiki. Leave its selection up to other editors. So perhaps the sentence should be: " Misplaced Pages is not the place to perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. " LeadSongDog come howl! 13:06, 2 May 2012 (UTC) That's a definite improvement. We should never try to dictate what editors can do outside of WP. Jakew (talk) 15:29, 2 May 2012 (UTC) In general I find that incidents like this are best dealt with by finding sources that criticize the original source and either juxtaposing the two in text, or on the basis of the validity of the criticism, not opting to include the study (for instance, specific to Morgan, Ward & Burton, there is a post by Orac on scienceblogs as well as a letter to the editor). In this case, if I cited MW&B at all I'd phrase it as a minority point of view ("In 2004 WM&B said...") with an accompanying reference to the LTTE ("Mileshkin, Rischin, Prince & Zalcberg said in a LTTE that...") It's pretty rare that an article genuinely bucking the mainstream POV is unanswered. In my experience, situations like these that are unanswerable are the minority - actual experts are not stupid, they usually rebut the lunatic fringe stuff. Regards this section of MEDRS, I wouldn't want to throw the baby out with the bathwater - I've run into far more issues where I've referred to that line/idea as a way of keeping in mainstream views ("You don't get to say that review article can't be included because some of the funding came from Pfizer"). It might be worth refering to WP:FRINGE or WP:UNDUE, since that's the nub of the issue in this case. The idea that chemotherapy is worthless is a fringe idea, most doctors clearly think there is merit to it. WLU (t) (c) Misplaced Pages's rules:/_complex 17:35, 2 May 2012 (UTC)I don't understand WhatamIdoing's point (if I'm following correctly); sourcing is always a matter for consensus and discussion, and in any realm of editing (not just medical), editors can decide via consensus that a source is not appropriate. This guideline can't be excluding something that is general; we can always discuss, review, and come to consensus to not use a bad source. I must be missing something, because this discussion makes no sense to me. There are plenty of sources about Tourette's, for example, that even though high quality, are just wrong and shouldn't be used. For example, the New England Journal of Medicine published a review about 10 years ago that had the wrong basic definition of TS; editors can review that, see that, and decide not to use a source that is just wrong. Why are we saying editors can't come to consensus to not use a bad source on medical articles, when we can do that anywhere else? SandyGeorgia (Talk) 18:22, 2 May 2012 (UTC)The guideline refers to "study". I think "sourcing", "study" and "facts" are getting mixed up here. Discussing sourcing without context can only deal with generalities like the type of source or type of publication. Some of our best writers are experts in the subject. Some of our worst POV pushers are experts in the subject. Both can produce lengthy talk page discussions that don't actually focus on proposed article text. What facts/points do we want to say? Are current books and reviews on the topic mentioning it? If not, then let's move on. Colin° 20:09, 2 May 2012 (UTC)Sandy, the first objection to the source on the talk page was that if patients knew the facts about (cytotoxic) chemotherapy's limited contribution to cancer treatment (in most, but not all, solid tumors, surgery is far more important), then some of them would refuse treatment. Do you think that making patients make the "right" choice in the real world is a legitimate reason for entirely removing all information about efficacy from the article? I don't, but that's what's happened. The TS comparison is inapt, because in that case, you had dozens of sources that gave a different definition. You could compare it to other sources and choose the many rather than the one. Here, we have just one source (that anyone knows of) that provides the overall statistic. (The point behind the paper was to calculate the overall benefit of all cytotoxic chemotherapy regimens for all patients in all stages of all invasive solid tumors.) It's not like this paper says 2% of survival the five-year mark, and another paper says 5%; this is the only paper that seems to have tried this ambitious approach. There are good reasons to be careful about how it is presented: it applies only to invasive solid tumors, which means no leukemias (and so understates the benefit) and very little skin cancer (and so overstates the benefit). It considers only cytotoxic drugs, which excludes most of the newer stuff (hormones, monoclonal antibodies, radioactive drugs, etc.). It is looking for a population-wide number, so it includes even people who didn't need or didn't take chemotherapy (this is the difference between "pounds of beef eaten per American" and "pounds of beef eaten per beef-eating American"). These qualifications seem to form the basis of the technical criticisms: they chose to exclude hematological malignancies and non-invasive cancers (both very typical exclusions, by the way), and I would have made a different choice (as a point of fact, I personally would have made a different choice), so it's an unreliable source. They chose to consider only cytotoxic drugs, and I would have made a different choice (actually, I wouldn't have), so it's an unreliable source. They chose to consider all patients rather than some patients, and I would have made a different choice (I would have, too), so it's an unreliable source. NB that every complaint here is not "they got these facts wrong", but "they made different choices than I would have made". Given that patients often believe chemotherapy is absolutely necessary, sometimes to the point of seeking it out despite multiple oncologists telling them that it is not appropriate, I think it sad that we now have an article that doesn't even mention undisputed facts about efficacy, such as that chemotherapy is hugely important to some cancers (e.g., testicular cancer), but provides very little benefit for others (e.g., lung cancer), even though not one of the people who dislikes this particular source believes that chemotherapy provides identical benefits for all forms of cancer. WhatamIdoing (talk) 22:16, 2 May 2012 (UTC)The problem with "editors can decide via consensus that a source is not appropriate" is that sooner or later, you'll find yourself on the Abortion talk page where a group of editors have decided by consensus that a secondary, well-conducted review by the Guttmacher Institute is not appropriate "because the Institute is biased towards pro-choice". Or if not that page, then another where a group of SPAs have established their walled garden. I'm sorry, but we need MEDRS as a bulwark against POV-pushers, who would love the opportunity to disallow any reliable secondary source that didn't align with their own views. --RexxS (talk) 23:16, 2 May 2012 (UTC)(edit conflict) @WhatamIdoing: I think we solve that problem by ensuring that our articles on testicular cancer, non-small-cell lung cancer, etc. each accurately reflect the role and efficacy of chemotherapy in those diseases. The parent article on chemotherapy should make clear that the efficacy of chemotherapy varies substantially by diagnosis and disease stage, and the details should be handled on a disease-by-disease basis. MastCell 23:19, 2 May 2012 (UTC) That would be fine with me—I have repeatedly said that the 2% statistic is unimportant—but you actually removed every single sentence in the article that "mae clear that the efficacy of chemotherapy varies substantially by diagnosis and disease". WhatamIdoing (talk) 16:33, 4 May 2012 (UTC)RexxS, WP:CONSENSUS is a policy. In general, it trumps WP:MEDRS. While there may be a cabal of SPAs who are trying to subvert an article, WP:MEDRS cannot be used to overrule them. Such a cabal can only be suppressed by drawing on a wider consensus with more good-faith editors. Axl ¤ 23:45, 2 May 2012 (UTC) We already have a safeguard against that. Established consensus, in the form of policies and guidelines, trumps local consensus. Jakew (talk) 07:17, 3 May 2012 (UTC)RexxS, welcome to the slippery slope that is all of Misplaced Pages (POV pushers gang up on good faith editors on talk pages-- try editing Venezuelan articles for a year or two). If you want to use MEDRS to trump CONSENSUS, pretty soon a carefully crafted and well applied guideline will fall into disfavor because it's used as a bludgeon and held to a standard higher than the policies it guides. AKA, shooting ourselves in the feet. MEDRS is a guideline, it enjoys consensus as such, it doesn't trump policy, and if folks start using it inappropriately, they're going to ruin a good thing. I guess I'm not understanding this argument, because whether one review is better, more recent, whatever is something that we can always discuss on talk and those discussions guide our decisions about which reviews to use. SandyGeorgia (Talk) 03:02, 3 May 2012 (UTC) I'm sorry you thought my contribution was "really stupid", per your edit summary, Sandy. I can assure you that I have the scars from defending NPOV on numerous articles over several years, and we don't need to engage in a pissing contest over who does the most defending – but thank you for the belated welcome anyway. Standing up for NPOV is not a card game, and there are no trumps, but you must recognise that the project-wide consensus enjoyed by MEDRS is not to be ignored lightly by a small group arguing a WP:LOCALCONSENSUS: "Consensus among a limited group of editors, at one place and time, cannot override community consensus on a wider scale" (and that's policy). If every good-faith editor decides it's better not to argue the merits of MEDRS for fear of it falling into disfavour, then we might as well hand over the encyclopaedia to the POV-pushers right now. The practices documented at MEDRS are exactly what we have to convince other editors to use when building a consensus for the neutral point of view, because they are inimical to the methods used to push POV. The problem is you want it both ways: you want to be able to use your expertise to discard a poor secondary source, while expecting others to defend good secondary sources that have been labelled "poor" by a self-proclaimed expert. Have the lessons of Essjay still not been learned here? --RexxS (talk) 12:40, 3 May 2012 (UTC)I'm disappointed by WhatamIdoing's response as she's falling into the same trap: using her expert knowledge to try to convince other editors, on a talk page, to include/exclude certain facts from the article. And to focus on the good/bad points in a study (or present counter-arguments against other editor's points) rather than discuss what we want to actually say and then present sources that support it. This is something Eubilides was great at: sourced-based discussion of actual or proposed article text.WP:WEIGHT is policy. It says An article should not give undue weight to any aspects of the subject but should strive to treat each aspect with a weight appropriate to its significance to the subject....Keep in mind that, in determining proper weight, we consider a viewpoint's prevalence in reliable sources, not its prevalence among Misplaced Pages editors or the general public... Also, if you are able to prove something that few or none currently believe, Misplaced Pages is not the place to present such a proof. Once it has been presented and discussed in reliable sources, it may be appropriately included. I agree that it is very important how the facts in this case are presented, and that the disputed paper's abstract does an appalling job of that. The exclusions (particularly of modern drugs) are problematic and it will be hard to work out what should be said that is actually relevant to a modern reader in a general article on chemotherapy. But hey, I've got an idea. What do modern texts on chemotherapy say on the subject? It keeps coming back to this. Why on earth are our editors arguing over the merits of an 8-year-old study as a source for an article on chemotherapy in 2012. And since the benefits of chemotherapy seem to depend completely on what cancer is involved, the approach taken by the book I linked (where each type of cancer gets its own chapter) seems to be ideal for writing targeted text on each indication or otherwise for chemotherapy. Colin° 08:01, 3 May 2012 (UTC) Agree if there are better more recent secondary sources we should be using them. Our guideline recommends against sources older than 5 years when more recent sources are available which I am sure is the case for chemo.--Doc James (talk · contribs · email) 15:05, 3 May 2012 (UTC) (edit conflict) Agree that Colin's book suggestion could be a good starting point in this particular case (though as a general bibliographic point, perhaps it's worth pointing out that the quality of scientific medical books can vary from standard reference texts, like this, to pot boilers with little or no peer review). Ideally, one would like to have a widely respected reference textbook which is periodically updated. Even so, working up a single, high-quality page on such an extensive topic as chemotherapy remains a major task. This is a genuine issue, imo, because in practice generic pages such as Chemotherapy can be widely consulted and regard matters of considerable real-world interest. 2c only, —MistyMorn (talk) 11:32, 4 May 2012 (UTC)So what exactly should be changed and how? Editors should not perform a detailed academic peer review. Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions. Sentence (1) can be left out, it has no practical consequences. Sentence (2) - if a study has inclusion criteria, methodology or factual errors leading to generally implausible results than apply WP:FRINGE.Sometimes the results are not even generally faulty, but study design or data (in)availability make them useless for encylcopedic purposes. As an example, endometriosis has a Cochrane Review of TCM as a reference. This kind of source is in my opinion fairly useless - even if TCM may somehow help endometriosis the design of the study is not good enough for anything except "a call for further research" which does not belong there. Richiez (talk) 11:22, 4 May 2012 (UTC) I'm not sure that removing the first sentence would be a good idea. As I interpret it, the current text states that editors shouldn't perform a detailed peer review and then lists as examples some personal objections that are invalid. If the first sentence were removed, then the resulting text would read as though these four types of objections were an exhaustive list of the only invalid grounds for exclusion. But they aren't: for example, editors shouldn't exclude papers on grounds that they disagree with methodology (except the general type of study), assumptions, statistical analysis, ethical problems, etc. This follows naturally from WP:NOR, so it isn't strictly necessary to spell it out, but it's certainly useful to make it clear. Jakew (talk) 11:46, 4 May 2012 (UTC) Oppose Richiez change suggestion. We don't want editors cherry-picking studies, which means we don't want them filling talk pages with their analysis of a study's failures. Full stop. It doesn't matter if the study is primary research or secondary analsysis of existing research. We build our articles on the work of expert writing, not by choosing which studies we think are good/relevant. You cannot apply WP:FRINGE based on your own view of what is fringe or not or by deciding that since you think the study is flawed, then it must be a fringe conclusion. The "Assess evidence quality" section is more background information than a primary guide to what to include in an encyclopaedia article. Have a look at a NICE guidleline like this one. The difference between the "NICE guideline" and the "Full guideline" is that the latter contains all the evidence for their guidance and how they went about measuring and collecting this evidence. It is a job for experts. It is a difficult job. It is not our job. Colin° 12:41, 4 May 2012 (UTC) If we are disallowed to judge WP:FRINGE than we can either stop altogether or have a very short list of allowed sources such es Encyclopedia Britanica - and of course we can expand this to all of wikipedia not just medical articles. Richiez (talk) 12:54, 4 May 2012 (UTC) We identify fringe theories/conclusions by the fact that (a) most reliable sources tell us they are wrong/fringe or (b) most reliable sources ignore the theory/conclusion -- in other words it has no weight to warrant mention in our articles. We don't identify them based on our own understanding. I dare say parts of the US think evolution is a fringe idea... Colin° 18:20, 4 May 2012 (UTC)I'm with Colin. We don't write a guideline around primary studies because we typically shouldn't be using them anyway-- we use secondary sources and reviews, and we do come to consensus about which reviews are the highest quality. All of this discussion seems to have resulted because folks want to use and do original research with primary sources. SandyGeorgia (Talk) 14:08, 4 May 2012 (UTC)I did not follow the chemotherapy discussion closely but it was my impression that it was a secondary source in question with results that happened to fall into WP:FRINGE category. Richiez (talk) 16:07, 4 May 2012 (UTC) It's not really FRINGE. It's just a broader focus than other papers have taken, sort of like "how much has society benefitted frim vaccines" rather than the far more common "how much has society benefited from the polio vaccine". WhatamIdoing (talk) 16:33, 4 May 2012 (UTC) It appears to be very much against mainstream in some respects. In breast cancer we try to differentiate by age, receptor status, markers or more recently gene activation patterns, response to neoadjuvant therapy and I think everyone is aware that the benefit of chemotherapy is likely to be dramatically different comparing a 40 year old or 85 year old women. It was my impression that the mentioned paper did the exact opposite of this approach by using unusually broad undistinguished patient population. Now hypothetically assume there is a review that meets all formal criteria of MEDRS and is so fringe that "mainstream" does not really notice, let alone review it. With many thousands (somewhere I recall 33K academic journals?) this might happen from time to time. Applying ] seems obvious? We have to use common sense. -- Richiez (talk) 21:10, 4 May 2012 (UTC) Whether you regard it as fringe or something of no weight doesn't matter much. The issue for this guideline is that this must be judged with reference to our sources, not on the basis of our own opinions. This is the issue I had with "if a study has inclusion criteria, methodology or factual errors leading to generally implausible results than apply WP:FRINGE". Colin° 21:20, 4 May 2012 (UTC) Why exactly should WP:MEDRS be exempt from the fringe rules? The essence of fringe is that someone, preferably by consensus regards some material as such. Richiez (talk) 21:39, 4 May 2012 (UTC) You're not getting the point, Richiez. WP:FRINGE is a valid guideline but if you read it you'll discover that determining what is fringe or not is done with reference to published work, not by WP editors using their own brains to discredit some theory/conclusion. My second point is that this particular point on Chemotherapy doesn't have to be as extremely wrong as a fringe idea in order for us to give it no space. If none of our contemporary sources choose to comment on chemotherapy in the way that 8-year-old paper did, then neither should we. No matter what you or WhatAmIDoing personally think of the study's merits or flaws. Colin° 09:29, 5 May 2012 (UTC)I support Richiez's recommendation.

" We don't write a guideline around primary studies because we typically shouldn't be using them anyway. " — SandyGeorgiaThat's a self-referential statement. We shouldn't use primary studies because our guideline doesn't recommend them? Anyway, this discussion is not about primary sources.

" All of this discussion seems to have resulted because folks want to use and do original research with primary sources. " — SandyGeorgiaThat is entirely false. No-one has recommended the use of primary sources, nor is anyone recommending original material in an article. WP:FRINGE includes the statement "We use the term fringe theory in a very broad sense to describe ideas that depart significantly from the prevailing or mainstream view in its particular field." Morgan's paper does indeed depart from the mainstream view, thus it fits this definition of "fringe". Axl ¤ 22:12, 4 May 2012 (UTC)Axl: see my comment above. If his paper "does indeed depart from the mainstream view" then this should be evidence in the literature (either explicitly or because it is ignored) and therefore it is unnecessary for WP editors to attempt to prove this on a talk page with their own skill. When editors try to push other editors into accepting a viewpoint, without reference to contemporary high-quality sources, then that is POV pushing. Just because it is mainstream POV pushing, doesn't stop it from being something to discourage. Colin° 09:29, 5 May 2012 (UTC) If you accept "beeing ingored" as evidence in literature than this would appear as reasonable - in theory. When you try to define this properly I expect it will turn out more difficult than a simple apply common sense rule. Eg how long after publication do you want to wait as evidence that something is ignored? We would be forced to publish something and wait 3-5 years for evidence it is ignored (and hope there is not a citation cartel at work)? Richiez (talk) 12:20, 5 May 2012 (UTC) I've seen this problem before. The disputants will not agree on what is "mainstream" - one side says it's is; the other side insists it's not. If we let editors define mainstream, we never reach consensus. Where the editor's expertise comes in is to point others to how prevalent a view is in the quality literature (which is how we should be defining "mainstream"). If you're lucky, you find one review published in an in-house magazine and a blog, and the contradictory review published in the Lancet and the BMJ. Unfortunately it's often not so clear-cut. --RexxS (talk) 02:22, 5 May 2012 (UTC) I've found taking a 'vote count' from a sample of reviews on PubMed to be a reasonable solution to this problem. It has its own problems, of course, but to get a rough measure of mainstream thinking it works quite well, and is obviously less susceptible to bias than relying on an editor's expertise. Jakew (talk) 07:21, 5 May 2012 (UTC) sure that does happen, but I think there are many cases left where there is no doubt about what is not mainstream. Richiez (talk) 12:24, 5 May 2012 (UTC)Observational studies (and CAM)Although this is written specifically in the context of CAM (not about it...), I suspect we could use this principle in this guideline:"Although observational studies cannot provide definitive evidence of safety, efficacy, or effectiveness, they can: 1) provide information on “real world” use and practice; 2) detect signals about the benefits and risks of complementary therapies use in the general population; 3) help formulate hypotheses to be tested in subsequent experiments; 4) provide part of the community-level data needed to design more informative pragmatic clinical trials; and 5) inform clinical practice."Source: Observational Studies and Secondary Data Analyses To Assess Outcomes in Complementary and Integrative Health Care. June 25, 2012, Richard Nahin, Ph.D., M.P.H., Senior Advisor for Scientific Coordination and Outreach, National Center for Complementary and Alternative Medicine.This is a RS from the NIH. It clearly states that we wouldn't be able to use observational studies as RS to make statements of fact about "safety, efficacy, or effectiveness". This actually applies to far more than just CAM, and is very relevant for this guideline. How can this be incorporated here? -- Brangifer (talk) 01:14, 1 July 2012 (UTC)Any individual study is a primary source, including observational studies, and this guideline strongly urges the use of secondary sources, such as research reviews and meta-analyses. So in a way it's already covered. But we could add it to this sentence: "Roughly in descending order of quality, lower-quality evidence in medical research comes from individual RCTs; other controlled studies; quasi-experimental studies; non-experimental studies such as comparative, correlation, and case control studies; and non-evidence-based expert opinion or clinical experience. " Perhaps after "other controlled studies." TimidGuy (talk) 19:49, 1 July 2012 (UTC) Categories: