|
Steering Committee Meeting Notes: June 10, 1999
COSP Steering Committee Teleconference Notes
Special Call on Selection Criteria & Randomization
June 10, 1999
Present:
Sally Rogers, moderator
Jean Campbell
Patrick Corrigan
Dianne Cote
Sita Diehl
Zahira DuVall
Susan Essock
Janine Elkanich
Stephen Fischer
Matt Johnsen
David Lambert
Sarah Lickey
Brian McCorkle
Betsy McDonel
Jo-Anne O’Connor
Ruth Ralph
Magda Randolph
Mark Salzer
Steve Segal
Carol Silverman
Joe Sonnefeld
Phyllis Solomon
Greg Teague
Daphne Walker-Thoth
Brian Yates
Background
Jean Campbell explained that the paper circulated entitled "Selection Criteria
and Randomization" represented a round robin collaborative involving her, Matt
Johnsen, Joe Sonnefeld, Betsy McDonel, and Crystal Blyler. The purpose of the paper was to
present the best case for positions that can be taken related to the selection and
randomization of study participants. Jean stated that the Coordinating Center and federal
representatives wanted to move beyond each site doing its own thing and attempt to reach
some common ground, recognizing that three different models and the diverse populations
involved in this multi-site study would make this difficult.
Matt said the idea for the paper came after a Logistics Subcommittee Group decision to
let the individual study sites do what they wanted in terms of study participant selection
and randomization. As the Coordinating Center and federal representatives thought about
it, they felt the rationale for further standardization hadn’t been articulated. They
wanted to be sure study results have as much interpretability as possible across sites.
Identification of Commonalities
After an extensive discussion, the following was discovered:
5 out of 8 study sites might be able to adopt and use common language during the
orientation and informed consent sessions (the idea of a multi-site informational flier
was mentioned as a means of ensuring that all study participants receive the same
information about the study).
5 out of 8 sites will randomize study participants after the administration of the
baseline protocol. Maine can randomize before or after baseline. Florida and California,
which both are drop-in centers, will randomize before baseline.
The control groups in some sites will be made up of people on the waiting list for the
COSP under study.
Some sites are recruiting study participants by asking the traditional mental health
services providers who among their clients is eligible for the COSP and selecting from
this group, whereas other study sites are having prospective study participants referred
to them by agencies in the community.
Decisions
It was decided that the discussion of additional ideas on selection commonalities will
take place on the list serv. Areas of possibility in addition to the orientation include:
(1) when to randomize, (2) who to select, and (3) from where to select.
Jean will seek consultation on this issue from other multi-site studies such as the
aging and employment. Betsy mentioned Jim Ware’s name as a candidate.
Sally asked that overarching issues such as selection criteria and randomization be
handled by the Steering Committee because they are too important to be handled by
subcommittees.
The Coordinating Center was asked to keep a running list of issues on the list serv and
plan how much discussion needs to take place on issues. In other words, someone needs to
keep things organized and summarized as to where the group is at a particular point and
how much work is needed before a decision can be made. Ruth asked that the Coordinating
Center and federal representatives are clear about what suggestions or changes they want
to improve the cross-site study. The sites also need to know in which areas they need to
be more flexible and in which areas they can be more firm.
Key Points Made Prior to Identification of Commonalties and Decisions
Sally Rogers: The variety of ways people are coming into the COSP is the reason the
Logistics Subcommittee Group thought there couldn’t be uniformity. There didn’t
seem to be a reasonable common place, given all of that variety. Some of the variety was
related to how systems operate and it was context dependent.
Betsy McDonel: There is variety and diversity, but there is a dilemma in conducting a
multi-site study when the recruitment efforts of study participants are so different. If
we don’t move toward uniformity, the study will have less impact and we are not
likely to get the results published in a major journal. Whatever we decide, we need to be
clear about the tradeoffs. We would like both internal and external validity, but if we
can’t have both, we need to make some choices.
At the request of Dianne Cote, Sally defined internal validity as the ability for the
researcher to conclude that the any differences seen between the control and experimental
groups are the result of the intervention. There are things that threaten internal
validity such as people dropping out or crossing over.
Matt defined external validity or generalized validity as the extent one can take the
results of a study and compare them with other groups that are different. An intervention
might not work with different kinds of people in different situations. Very often
researchers can’t get both strong external and internal validity.
A difference in the case mix was described as those differences in characteristics of
people in the study from site to site. Statistical adjustments based on the difference in
the case mix will be used.
Jean Campbell: From a cross-site perspective, we need as much standardization as
possible. The ultimate goal in a multi-site would be to get every site to follow the same
process of selection and to get every site to randomize from the people receiving
traditional mental health services. In reality, we may not be able to get to this ultimate
goal.
Susan Essock: In Connecticut, people who come to the COSP are interested in advocacy.
In the past three years, the COSP only refused to enroll one person. Three people
didn’t enroll in the program. Jean said these consumers are pre-selected for an
interest in advocacy. Susan pointed out that COSP is a volunteer program so people are
self-selecting throughout.
Steve Segal: In California, 71% of people complete their referral to the
COSP. After
the first visit, there is a 29% attrition rate. It is hard to explain to people what they
are getting into. They have to go and see for themselves. Jean said there are certain
things that can be done at the program to help ensure that people stay.
Sally Rogers: In Iowa, intensive orientations were held so that people would fully
understand what they were getting into. At the end of the orientation, the COSP had a pool
of people who were probably motivated. Matt said the concern is that the group of people
who do not opt to enroll is different from site to site because they are presented with
different stimuli. It was suggested that the study sites might have a standardized
introduction to COS that presented something about the kinds of services that are offered
across the country. Sally said an issue is where that standardized introduction might take
place – the person might be sitting in the COSP center at some sites and at other
sites without center-based programs, he/she might be sitting someplace else. Steve said
California would have a problem with this type of introduction – it would ruin
internal validity for the site’s study and result in cross over.
Carol Silverman: For some study participants who are very poor, the money provided for
participation in the study might entice them into the study even if they aren’t
interested in the program. Sometimes you can distinguish if they are saying
"yes" to the study or "yes" to the money.
Jean Campbell: Jean suggested that randomization take place prior to informed consent.
Ruth Ralph pointed out that you don’t have a pool of participants until you have
informed consent.
It was suggested that the demonstrated interest of participants could be one uniform
way or recruitment. Steve pointed out that by doing this, we would be creaming a very
select group of people. Mark Salzer mentioned that the creaming would be the same for the
control and experimental groups so the results would have to be attributable to the
intervention.
Sita Diehl: For small towns, it would be detrimental to promote the COSP too much
before people get into the study. Ruth said in Maine, other agencies inform people about
the self-help groups.
Jean Campbell: Jean suggested that we prioritize cross-site internal validity over
external validity and pointed out that there is a difference between internal validity for
the individual site studies and internal validity for the cross-site study.
Steve and Ruth indicated that they felt the study sites are willing to make adjustments
within the realm of the individual studies to accommodate the multi-site study.
[ Up ] [ Next ](June 23, 1999) | 
5400
Arsenal Street