Methodology

This page outlines a few main ideas garnered from the Society for Acupuncture Research Conference 2013, specifically from talks given by Claudia Witt, MD and two representatives of the National Center for Complementary and Alternative Medicine: Qian Li, PhD, and Kristen Huntley, PhD. The bulk of the information identifies weaknesses or challenges of acupuncture research in the past in order to stimulate productive discussion and implementation of improved study methodology. [Any mistakes are my own- these are based on my notes from their presentations. I am by no means a qualified research methodologist, but I will summarize here some considerations.]

For an in-depth discussion of methodological challenges, please refer to the Society for Acupuncture Research’s White Paper, Paradoxes in Acupuncture Research: Strategies for Moving Forward:

The emergent potential paradoxes that frame this paper are:

(1)A large number of well-designed clinical trials have reported that true acupuncture is superior to usual care, but does not significantly outperform sham acupuncture, findings apparently at odds with traditional theories regarding acupuncture point specificity and needling technique.

(2)While many studies in animal and human experimental models have reported physiological effects that vary as a function of needling parameters (e.g., needle insertion depth, mode of stimulation), the extent to which these parameters influence therapeutic outcomes in clinical trials is unclear.

The charts below outline key recommendations of the white paper:

Figure 2: Illustration of bidirectional translational acupuncture research schema.

Table 1: Summary of combined recommendations for future research based on the paradoxes identified in this paper.


(a) Recommendations based on paradox 1 Expected benefits for future clinical trials

Understanding the physiological effects of acupuncture needling using appropriate controls Prevention of needle-specific physiological effects in sham treatment

Identifying key physiological and psychological nonneedle components of acupuncture treatment Prevention of specific nonneedle effects in sham treatment

Recognizing the extent to which acupuncture provided in the context of a clinical trial reflects acupuncture in real world clinical practice Increased ecological validity

(b) Recommendations based on paradox 2 Expected benefits for future clinical trials

Translation of physiological effects of acupuncture observed in acute animal models and healthy humans to clinical treatment of complex chronic conditions Improved correlations between needling parameters, biomarker changes and clinical outcomes

Translation of basic science findings with electro-acupuncture parameters using few needles to more traditional manual needling using multiple needles Improved understanding of complex manual needling protocols as well as electrostimulation

Cross-correlations between Traditional Chinese Medicine (TCM) and biomedicine syndromes Use of biomarkers to TCM entry criteria as objective outcome measures

So those recommendations were from 2008. Now we are jumping to the SAR Conference 2013, to focus on the details of how these recommendations need to be parced in practice.

Dr. Huntley identified the top four priorities at NCCAM/NIH for funding acupuncture studies right now:

  1. Scientific validation of acupuncture techniques used
  2. Amenability to rigorous scientific inquiry
  3. Potential to impact health care policy
  4. Potential to impact clinical use, clinical practice, and importance for public in general

In order to address these priorities effectively, study design must be very carefully considered. Dr. Qian Li posed the question, what is the first priority of acupuncture research? Mechanism of action or clinical impact?

Dr. Li argues that clinical impact research is more important currently due to the ability to prove the benefit for real people in context of current health care environment in order to help increase funding for both types of research. Although both mechanism/efficacy studies are just as important as clinical impact and comparative effectiveness studies in an absolute sense, comparative effectiveness research (CER) trials are cheaper and present the possibility to nest an efficacy trial within a subset of a large clinical trial. This way we can explore mechanism and comparative effectiveness in the same trial. This can only be done with large sample sizes, but the more well-designed CER trials are created, the more reason there will be for funding larger trials.

Although many research formats may be appropriate for acupuncture as a medical discipline, we are not only in a profession that needs to teach others how our therapies function in the ‘real world’ but also in an era in which comparative effectiveness research has become a national priority. Doctors are overloaded with information on specific medical interventions while often lacking the clinical evidence to inform their decision to use one therapy over another. This issue has been defined and summed up in the publication Initial National Priorities for Comparative Effectiveness Research, a consensus report of the Institute of Medicine of the National Academies. This report expands upon the 2003 alarm call of the Journal of the American Medical Association, Practical Clinical Trials: Increasing the Value of Clinical Research for Decision Making in Clinical and Health Policy (cited over 1000 times according to Pubmed).

Acupuncture can step into this gap and prove its relevance with CER. With the following 2012 systematic review, Dr. Witt has already shown that this trend is happening and must continue to be refined along with expanded CONSORT guidelines for acupuncture studies. How Well Do Randomized Trials Inform Decision Making: Systematic Review Using Comparative Effectiveness Research Measures on Acupuncture for Back Pain by Claudia M. Witt et. al. They concluded that “[the RCTs] with more flexible acupuncture and no placebo control scored closer to effectiveness.”

Figure 2. PRECIS scoring for the 10 included trials comparing different methodological aspects (second rating after consensus procedure), a larger rounder figure would correlate with a higher score on PRECIS representing more the effectiveness side. show more doi:10.1371/journal.pone.0032399.g002

Figure 2. PRECIS scoring for the 10 included trials comparing different methodological aspects (second rating after consensus procedure), a larger rounder figure would correlate with a higher score on PRECIS representing more the effectiveness side.doi:10.1371/journal.pone.0032399.g002 

So what is the hold-up?

According to Dr. Li, so far acupuncture research has been stumbling at six points throughout the design process:

  1. Blinding
  2. Point selection
  3. Roles/training of practitioners
  4. Frequency/ Dosage of treatment
  5. Choice of controls
  6. Small sample size

1) Blinding

Because of acupuncture’s nature, patient blinding is nearly impossible. “Sham acupuncture” has proven to be more of a complicating factor than a control, and we have yet to find an adequate way to blind patients to the acupuncture process. Using non-acupuncture points has also served to confuse study results. The best way to achieve patient blinding is an ongoing debate, and one with possibly no answer. CER trials do not require patient blinding.

During the process of giving/receiving acupuncture, the acupuncturist and patient naturally exchange pleasantries and discuss the procedure. However, in a clinical trial setting, the patient/practitioner interaction must be as uniform as possible, and this kind of limitation must be instituted and recorded as accurately as possible. Current qualitative research into the acupuncture encounter may point towards ways that this can be accomplished in a reproducible way.

Of course the investigators, statisticians, and anyone involved in data collection must be blinded and this must be monitored until the database has been locked.

Dr. Li suggested that in a similar way to psychiatric studies, acupuncture studies could blind the study objectives and all available treatments to the participants. This would reduce non-compliance, reduce cross-contamination of participants seeking treatment that was not assigned to them, and reduce any bias preference toward treatment selection.

And finally, Dr. Li mentioned a few statistical analysis pointers. The NCCAM prefers variable sized block randomization, and suggests the avoidance of minimization randomization. Dr. Li also cautions against over-using stratification factors.

2) Acupuncture Point Selection

Currently, one of the more difficult choices to be made in study design is whether to use a fixed, standardized treatment protocol involving fixed acupuncture points based on the disease pattern, or to use a more complex, personalized intervention that may reflect clinical reality more accurately yet be less replicable.

If using a fixed set of acupuncture points, there are advantages and disadvantages to consider. An advantage of the fixed acupuncture is replicability (as long as the diagnostic parameters are very narrowly defined for that set of points), and for this reason it could be used in an efficacy or effectiveness study (rather than just effectiveness). Another advantage is that the practitioner doesn’t have to add individualized points, allowing for ease of application by less experienced acupuncturists. It fits into the ‘disease’ model of Western medicine, and hence may be more easily accepted or understood by non-acupuncturists.

However, some major disadvantages of the fixed acupuncture points include the disparity of detailed symptom and constitutional composition of the patients in the study – the fixed acupuncture points may not be the best “fit” for all the patients with an overarching similar ‘disease.’ Furthermore, any observed benefit of a fixed point protocol may be diluted by including all patients with a similar disease, ignoring their underlying differences in constitution and other factors normally taken into account with personalized acupuncture point selection. One way to mitigate this is to provide detailed ‘pattern differentiation’ analyses that of patterns that may present themselves frequently in clinic in association with particular diseases. When the more detailed pattern can be accurately identified, using specific symptomatic and objective indicators, the point selection can arguably be slightly personalized while maintaining a formulaic aspect. But the pattern differentiation and diagnosis would then require a higher burden of decision-making on the part of the practitioner, and would be potentially subject to debate since we haven’t reached a universal diagnostic language yet as a profession. And ultimately, this style of treatment is less reflective of acupuncture’s true clinical application, so it follows that the study may not reflect acupuncture’s true clinical effect.

The other option, using a variable or personalized set of acupuncture points, presents its own set of challenges. An advantage is the personalized treatment provided by a patient centered, pragmatic trial design, and this reflects the flexibility of acupuncture as a medicine in responding to a changing symptom pattern within the course of a disease. Acupuncture is unique in that the treatment adapts to changes in the disease, and this can be fully implemented and studied by using a varying acupoint intervention. However, this flexibility requires a high standard of training for the acupuncturists participating in the study, and it also potentially requires the acupuncturists to have a similar enough training to elicit a similar point selection adaptation to a similarly changing disease pattern. This would be difficult to test and more difficult to implement.

Finally, reproducing a study that uses the variable acupoint model is impossible by definition. Some studies use a base point selection with the addition of a number of individualized points on top of that base point selection set. The problem here is that the added points could potentially alter the treatment enough that the original set of points is rendered obsolete and therefore effectively unreplicable. In demonstrating comparative effectiveness, we still need to study the comparative effectiveness from practitioner to practitioner and try to understand why some practitioners obtain better or worse results consistently, or to ‘vet’ the acupuncturists’ clinical effectiveness using a preliminary study, in order to more accurately analyze their results.

3) Roles and training of practitioners

While we are on the topic of variable acupoint treatment methods, we may bring up the importance of the detailed explanation of each participating practitioner’s training in acupuncture. For the variable acupoint trials, both the practitioner’s history and experience with treating the underlying disease or condition must be fully explained as well as the practitioner’s perspective on meridian theory and any other theoretical underpinning of their clinical practice as applied to the variable acupoints in the study.

Beyond this specific information, there are many general considerations regarding practitioner information and disclosure that can be overlooked in study design. Dr. Li suggests the following recommendations. Clarify the number of practitioners in the trial, and describe the training and credentials of each practitioner. Avoid single-practitioner studies in order to help avoid treatment bias or individual influence. Give clear instructions about patient-practitioner interaction to reduce variability and bias during the treatment sessions. And finally, principle investigators must not play a role in delivering the intervention itself.

4) Treatment Dosage

How often do people receive acupuncture, and how often should they receive acupuncture? We currently have limited data regarding treatment frequency guidelines and patterns of treatment frequency in clinical practice. Some acupuncturists treat patients every day for a period of time, and this high frequency of treatment is generally not feasible during a study. What might be the difference in treatment effectiveness between treating every day for five days and once a week for five weeks? More studies investigating frequency of treatment are needed in order to help define our guidelines and inform our studies.

Some qualitative effects can also be initiated with different frequencies of treatment. If a patient comes in every day for acupuncture treatment, the increased and concentrated attention and interaction within those five days may provide a confounding factor due to frequency of treatment. The increased attention within that short period of time may add another variable. To help control for this, more treatment arms with varying treatment frequencies or a higher dose of control attention may be necessary.

Another factor regarding frequency of treatment is brought up by the studies that use variable-acupoint methods. Not only does the symptom picture and state of the meridian system define the changing acupuncture points used from one treatment to another, but it also determines the frequency of treatment. The practitioner must note what factors induced them to reach the frequency and number of treatments in a logical manner, based on the objective and subjective signs of the patients’ symptoms and meridian system.

5) Choice of Controls

The choice of controls depends on the objectives of the study. For an efficacy study investigating the mechanism of acupuncture, of course some type of placebo or sham control is necessary. The types of placebo and sham controls are discussed in a separate page within this website since this is an area of contention and ongoing investigation.

For an effectiveness study, usual care in the comparative effectiveness research model provides the most clear control. Usual care is an ethical control, for example a pharmaceutical that treats the same condition and ideally affects the same biomarker as the acupuncture treatment for that condition. Waitlists also serve as simple controls, although the control effect is weak. Waitlist controls should be used carefully, since it’s difficult to ascertain a specific vs a non-specific effect of the acupuncture treatment, in the same way that ‘continuing care’ is used only weakly to determine whether acupuncture has a statistically significant effect.

6) Small sample size

Because acupuncture research doesn’t garner a huge influx of funding, most studies are currently small scale. Dr. Li gives some tips for increasing the efficiency of these small scale studies. First, and most importantly, find a small homogenous patient population. The condition should be as distinctly defined and limited as possible, for example instead of ‘patients with low back pain,’ one should study ‘patients with low back pain due to osteoarthritic degenerative changes of the lumbar vertebrae.’ Furthermore, once you have obtained members of this select population, make sure to reduce non-compliance with thorough management of patient expectations and careful follow-up.

An enrichment study design that uses a placebo or responder run-in can be helpful to remove placebo responders in a pilot trial and therefore make an even more data-rich patient population. Plus this may be useful for helping us understand the mechanism of acupuncture in the long term, if we keep close track of the placebo responders throughout each study. Finding out which patients are likely to respond to treatment is also important, so we can find predictive markers that can be used to select patients

Bonus:

7) Biomarker Classification

Dr. Li also discussed the basics of defining whether you need a surrogate or mediator biomarker for the clinical assessment.

Surrogate biomarkers replace clinical outcomes, so no causal relationship needs to be proven.

Mediator biomarkers have the advantage of helping us to understand the mechanism of acupuncture, but there needs to be a causal relationship. A difference would need to be shown in the surrogate and mediator marker between the treatment groups, as well as a difference in clinical outcomes between treatment groups. In the superiority type design, the difference in clinical outcomes should be able to be explained by changes in the mediator biomarker(s).

Predictive biomarkers can be useful as well; they help separate different populations with respect to disease outcome in response to a particular targeted treatment. In this case, the study needs a baseline biomarker assessment, then an intervention and control treatment arm, and a specifically positive or negative clinical outcome.

*******                                      *******                                       *******

So now that you have digested all of these considerations, you are interested in whipping up a study and making a grant proposal right?

Luckily Dr. Huntley also spoke of design considerations when first submitting a proposal to the NIH (this advice is for less experienced study designers):

  1. Think of the topic you’d like to study
  2. Find an excellent journal in the biomedical area of the study
  3. Focus on biological mechanisms consistent with Western medicine in the discussion of acupuncture in the grant proposal/application
  4. Acknowledge current issues and controversies in acupuncture research and the biomedical field of study
  5. Include ways to assess the adequacy of blinding techniques and measure of treatment expectation
  6. Use the tried and true format:
    1. Purpose
    2. Problem/Background
    3. Significance of Study
    4. Question and Hypothesis
    5. Design/Analysis
    6. Disclosure of the research team
  7. Remember that there is a turnaround time of about 9 months for NIH proposal processing
    1. Application itself:
      1. R01 application is only 12 pages, difficult to include detail necessary for protocol/methods. To assist, use a template to condense info (see NCCAM research toolbox).
      2. Read the Clinical Terms of Award very carefully
      3. To check for currently funded projects to assure no overlap, you can check the NIH Reporter but may also
      4. Contact staff at NIH- call or email to schedule a call – give them a one-page concept with specific aims, and an SRO will help discuss ways to target the research for funding and confirm that no overlap is occurring.
      5. Scientific Review Officers (SRO)- Panel rosters are published 3 days before review meetings, so any concerns about the content of the roster (who is actually reviewing your study for scientific validity/plausibility) can be discussed with your SRO.
      6. To keep up on funding ideas: Search by mechanism for funding success rates and call NCCAM/ NIH to keep up with any changes that may be implemented in the near future.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: