Article on Evaluating All Those Med/Science Papers

I was very taken by this article! It's an easy-to-read (though a bit
long) summary of what to look for in assessing the worth of the
various research papers we read. Although Wilshire discusses this
relative to age management studies, the criteria he cites can just as
well be applied to all the pro and con low carb studies that have been
posted here so often lately.

One good take-away from the article: "My personal reviews of the
literature, and much helpful mentoring from some groundbreaking
thought leaders, convinced me that much of the current accepted dogma
was actually built on a virtual house of cards. Not only had these
inaccuracies poorly affected my personal health and that of my
patients, but I came to realize that these issues had profound public
health ramifications as well."

===============================================

The Evidence-Based Nature of Age Management Medicine

By Gil Wilshire, M.D., FACOG

This marks the inaugural edition of the e-Journal of Age Management
Medicine. I am honored to be asked to contribute this introductory
editorial.

Every field of medicine has been undergoing radical revisions over the
past decade. The field of Age Management Medicine is no exception. The
cause of this radical sea change has been the rapid worldwide adoption
of Evidence-Based Medicine. Many are still surprised to learn that
evidence-based standards of research and publishing have only started
to become widespread since 1993. Archie Cochrane, a UK-based
epidemiologist, recognized the lack of rigorous evidence in all fields
of medicine, and began publishing his organization´s now famous
reviews and meta-analyses concerning numerous medical topics. The
ramifications of the Cochrane Collaboration continue to be felt.

In retrospect, with the perspective of 20/20 hindsight, it is not
surprising that erroneous clinical recommendations riddle the
literature and act to the detriment of current standards of practice.
A quick perusal of journals from even ten years ago will demonstrate
widespread defects in investigative protocols and statistical
analyses. Studies that are under-powered or filled with uncontrolled
confounding variables abound. Investigations that are swayed by
numerous sources of bias are also commonplace. Conclusions are
frequently proffered without mention of either alpha or beta error.

My interest in these issues was piqued in the late 1990’s when I came
to realize that widespread standards of practice in my personal fields
of expertise (endocrinology and metabolism) lacked the basic Level 1
evidence one would demand before making sweeping clinical
recommendations. My personal reviews of the literature, and much
helpful mentoring from some groundbreaking thought leaders, convinced
me that much of the current accepted dogma was actually built on a
virtual house of cards. Not only had these inaccuracies poorly
affected my personal health and that of my patients, but I came to
realize that these issues had profound public health ramifications as
well. Since the time of that epiphany, I have gone on to found a
non-profit organization whose mission is to promote the use of
evidence-based data in the creation of nutritional policies. I have
also been active as a reviewer and supporter of some of the numerous
evidence-based journals and organizations that have sprung up to meet
the unmet needs in my personal fields of study. I am trying to do my
part build a new solid foundation of knowledge upon which our
successors can build—and upon which our patients can depend.

The field of Anti-Aging Medicine, as perhaps no other, has been
fraught with wild claims and unsupported assertions. In preparing to
write this editorial, I randomly read through a sampling of various
popular websites. As you can confirm for yourself, the “literature” is
full of case studies (“N’s of one”), cross-species inferences,
improper use of surrogate markers, and lack of proper control groups.
I do not mean to disparage this entire field. By definition, the study
of the medical issues of aging requires a long time to study! The
testing of some of the most important currently contested hypotheses
may potentially take generations to be done properly and effectively.
The field of Age Management Medicine will present enormous challenges
to the courageous investigators who choose to work in this field. For
these reasons, the advent of this new, evidence-based journal
represents a tremendous advancement for this specialty and for the
advancement of human health in general.

I would like to take the remainder of this piece to discuss some
concepts that I have found to be crucial for the critical evaluation
of scientific papers and writing. I will discuss some very basic yet
important “rules of thumb” and “red flag” words that should enrich
your reading and assimilation experiences, and empower you to be the
best and most up-to-date clinician or scientist you can be.

Developing Critical Evaluation Skills

All evidence is not created equal. The “gold standard” that defines
Level 1 evidence comes from prospective studies that properly
randomize subjects into control and intervention groups and then
follow these subjects into the future. The reason these types of
studies are so valuable is that biases are effectively removed from
the equations. These protocols also reduce the number of variables to
a bare minimum so that effects can be properly attributed to the
interventions employed. Because of the very nature of aging, long-term
studies of this type are difficult to do. The most valuable clinical
endpoints, death or significant morbidity, may take a long time to
manifest. In addition, for very long trials it is exceedingly
difficult to ensure that subjects will actually be able to faithfully
comply with the conditions of the study! To reduce the time
requirements for these types of research, the use of surrogate markers
(intermediate endpoints) are often employed. This creates another set
of problems (i.e. are the surrogate markers truly valid predictors of
the outcome of interest?). Readers should be very conscious of the use
of surrogate markers and should be immediately wary of them. One must
not assume that they are valid, despite the location of the research
institution or the reputation of the investigator.

On the whole, Level 2 evidence is more common in the literature. Level
2 studies may be prospective or retrospective in nature, but suffer
more from selection, investigator, or observation biases. The
information gleaned from these Level 2 studies may be very valuable.
They also are more likely to include many more subjects and they have
the ability to capture data over longer time intervals (by way of
chart reviews or population studies); hence they may be the only
practical way to glean particular information in a reasonable time
period. To properly evaluate a Level 2 study, one must read the
Materials and Methods section very carefully. Look for bias in the
creation of the study groups or in the way endpoint data were
collected. Did the subjects have to recall symptoms or self-report
food or drug intakes? Did the investigators relay their opinions or
expectations to the subjects? Did the investigators measure all
relevant endpoints, or only the facts that they felt would confirm
their hypothesis? Do not assume the scientists were neutral with
regard to their findings.

Perhaps the most contentious area of scientific discourse is the
subject of Level 3 evidence. Level 3 is generally defined as an Expert
Opinion, one generally born out of consensus panels of “big name”
personalities from “top notch” institutions. Ideally, this type of
evidence is promulgated with regard to questions that suffer from lack
of Level 1 or 2 data. In the case of rare diseases or topics with
conflicting information, Level 3 evidence may be the best we can
obtain, and must suffice to guide clinical recommendations.

Level 3 evidence, however, is fraught with potential problems. It is
inextricable from bias, outside influences, personalities, egos,
politics, and (even) well-meaning intentions. Those readers who have
ever served on a board or committee, or have seen one in action, can
attest to these observations. Readers of papers that are rated at
Level 3 strength of evidence should be very wary of the study’s
conclusions. It is my personal feeling, in fact, that Level 3 evidence
is not evidence at all: it is opinion. From a historical viewpoint, if
we had relied solely on committee opinions in the past, the world
would still be flat and the Sun would still revolve around the Earth.
I am aware that my stand on this issue is controversial, and I offer
it only as one man’s position. I strongly believe, however, that in
issues that concern large numbers of people and have public health
ramifications, Level 3 evidence should not be used to dictate policy.
Without high level evidence, I believe the maxim primum non nocere
should guide decision making. By way of brief example, I offer the
current state of the “official” dietary recommendations that have been
promulgated to the American public for the past 30 years. These public
health policies completely lack any high-level evidence to support
them. The current, ongoing public health issues in this country that
result from them are very familiar to every reader of this e-journal.
Had policies been restrained until Level 1 evidence had been
collected, then the current state of confusion could have been
avoided.

Common Errors in Research Studies

Although there are myriad ways to undermine a research study, readers
of this e-journal should be on the lookout for issues that frequent
the Longevity and Aging literature. A common issue is the
extrapolation of animal (or even yeast or bacteria!) longevity data to
humans. Beware studies that avoid mentioning the study species in
their abstract or conclusions. They may want you to over-estimate the
import of their work. As was mentioned earlier, beware of surrogate
markers or intermediate endpoints. Have the investigators “cherry
picked” only the markers that prove their bias? For example, the use
of selected blood lipid fractions is a notorious fault of much
nutrition work. Have the investigators examined one clinical endpoint
in isolation, or have they looked at all relevant outcomes (e.g.
all-cause mortality, cancer rates, as well as cardiovascular events
etc.)? Have the scientists thoroughly considered confounding variables
in their experimental model? By way of example, common errors in the
nutritional literature include failure to control for the presence of
harmful fats in total fat consumption measurements. A common error in
the cardiology literature is the failure to consider cholesterol lipid
fractions in the context of ongoing inflammation. These two examples
demonstrate how widespread erroneous associations and conclusions can
come about due to simple failures to control for important confounding
variables.

Rules of Thumb

If a longevity claim is made for a particular treatment or
intervention, is the benefit described as a relative risk (RR) with
confidence intervals? Beware results that are expressed as any
percentage greater than 100. Be mindful that most statisticians and
epidemiologists do not get excited about results until they start to
see RRs greater than 2.5 (that is a two and a half fold increase in
risk).

Look for adequate numbers of study subjects. In general one needs over
20 individuals per study arm to approach adequate approximation of a
general population. Was a power study done before the investigation
commenced to ensure likely success? This will help you recognize a
study that was done with the confidence of obtaining valuable
findings; not one that was lucky enough to generate data that the
investigators found useful to their position.

Insist on being able to understand and grasp a paper on the first
reading. If you find yourself having to read and re-read a study,
beware of “dumbfounding.” If the authors cannot make the protocol and
findings understandable, then they are either confused themselves, or
they are trying to hide irreconcilable defects. Move on from these
papers.

Red Flag Words

The beauty of statistics is that we can all generally agree on what
is “significant.” Confidence intervals are given their specific name
for a reason. Beware studies that include in their conclusions the
words: “may,” “could,” “possibly,” “might,” “imply,” “trend,”
“rational,” or “suggest.” Statistics permit the use of decisive,
declaratory statements such as: “prove,” “demonstrate,” “show,”
“link,” or “cause.” We all know that real-world conclusions cannot be
made with 100 percent certainty all the time. In a well-designed,
well-conducted study, an alpha (a.k.a. “p” value) less than five
percent (or perhaps less than one percent in the context of multiple
comparisons) is where an investigator should be comfortable putting a
stake in the ground. If a study soft-peddles its findings with the
“red flag” words described above, and only concludes that “more
studies are needed,” turn the page; it was probably not well designed
in the first place.

Conflicts of Interest

Always insist on knowing the author(s’) affiliations and potential
conflicts of interest. Although the reasons for wanting to know this
information are self-evident, these competing interests are not always
delineated. Scientific journals continue to improve in disclosing this
type of information. Articles in the lay press and television
interviews are notorious for keeping this information under the radar,
however.

Conclusion

The advancement of scientific knowledge and clinical practice
requires effective research by investigators and critical reading by
those who might apply the new knowledge. The advent of this new
evidence-based journal—the e-Journal of Age Management
Medicine—fulfills the first requirement for continued advancement in
this field. I sincerely hope that this editorial will assist the
critical reading skills of those who might be reading this material
and actually treating patients.

I wish the editors of his fledgling project tremendous success in the
upcoming years!

Potential competing interests: Gil Wilshire, M.D., FACOG is a
Reproductive Endocrinologist who receives diverse sources of funding.
He performs consulting work for numerous U.S. and international
pharmaceutical companies. He is the President of the Carbohydrate
Awareness Council, a non-profit 501(c) 6 corporation in Northern
Virginia . He is a principal member of Axximar, Inc. and is the
Medical Director of ERBLAN Surgical, Inc. He is also on the staff of
Sibley Memorial Hospital in Washington , D.C.

On Thu, 09 Mar 2006 07:59:30 -0500, Hannah Gruen
wrote:

The Evidence-Based Nature of Age Management Medicine

By Gil Wilshire, M.D., FACOG

I forgot to add the URL: http://www.agemed.org/p/4,4.html

HG

Hannah Gruen wrote:
On Thu, 09 Mar 2006 07:59:30 -0500, Hannah Gruen
wrote:

The Evidence-Based Nature of Age Management Medicine


Interesting piece, thanks for posting it. Of course it serves to
confirm what a lot of people who are serious about dieting and execise
already know - that there's an awful lot of nonsense out there, an
increasing amount of which seems to be finding its way into this ng!

Hannah Gruen wrote:
:: On Thu, 09 Mar 2006 07:59:30 -0500, Hannah Gruen
:: wrote:
::
::: The Evidence-Based Nature of Age Management Medicine
:::
::: By Gil Wilshire, M.D., FACOG
::
:: I forgot to add the URL: http://www.agemed.org/p/4,4.html
::

Thanks, I was just about to post asking for this. Good find, BTW.