Archives Calendar Coordinating Center Multisite Activities
Project Description COSP References Study Sites Search Our Site
Table of Contents Upcoming Meeting Agendas______________________________________________
|
|
Steering Committee Meeting
Notes: 1:15 p.m. – 4:35 p.m. Embassy Suites Washington, D.C. Welcome
Tom Summerfelt, Consumer-Operated Service Program (COSP) Steering Committee Chair, opened the meeting with introductions and announcements. Neal Brown Neal Brown, Chief, Community Support Program of the Center for Mental Health Services (CMHS), welcomed the Steering Committee to Washington and to Year 02 of the multi-site study. While acknowledging the difficulty of making such a project work, he congratulated members for a job well done and thanked them for their time, effort, hard work, and understanding. He indicated that the project was “on time.” Neal pointed out that CMHS is into its third generation of multi-site studies and that it is learning as it goes along. Although the multi-site studies are being done better, there still is room for improvement, he said. He asked the group to bear with CMHS. He explained the three ways in which CMHS works on projects—through grants, cooperative agreements, and contracts. CMHS decided that the cooperative agreement was the best route for the COSP multi-site research initiative. Unlike grants, the cooperative agreement allows CMHS to take what it knows and provide the agreement participants with a coordinating center and federal assistance. With a grant, the awardee does his/her own thing, checks in periodically with the project officer, and submits a final report. With a cooperative agreement, there is actual involvement from the project officers; therefore roles are a little different. With this cooperative agreement, the federal representatives are setting wide parameters and trying not to intervene, however there might be times when Betsy McDonel, Crystal Blyler, Mike English, or Neal Brown have to establish boundaries. He said because CMHS is learning as it goes along, it would like to get input from the COSP Steering Committee about what should be done differently in future multi-site studies. Quorum and
Notification of E-mail Votes
Tom reviewed the ground rules for the meeting and indicated that the committee needed to determine what constituted a quorum. Yvette Sangster moved that a quorum for face-to-face and teleconference meetings of the Steering Committee and e-mail votes would consist of 2/3s of the voting members (12 of 18). Proxy votes will be acceptable. Dianne Cote seconded the motion. Zahira DuVall suggested as a friendly amendment that within the quorum there be a minimum number of consumer representatives. Dianne mentioned that although her vote counted as a principal investigator and not a consumer, she is still a consumer as are Ruth Ralph and Jean Campbell. Yvette did not accept the friendly amendment. The motion to
establish 2/3 as the quorum passed unanimously. Sita Diehl moved that a minimum of two business days notice be required for any e-mail vote and that at least two modes of communications be employed for votes with no previous notice. Votes will close at 5 p.m. Pacific Time, 7 p.m. Central Time, 8 p.m. Eastern Time. Any motions must be made by voting Steering Committee members and cleared through the Steering Committee chair or Coordinating Center principal investigator. Betsy seconded the motion. The motion to establish a minimum of two business days notice for e-mail votes passed with 12 votes in favor and 6 votes against. Baseline Common Protocol Tom expressed the desire to put the baseline common protocol “to bed.” Zahira moved to re-open consideration of the introduction to the common protocol. The motion was seconded by David Lambert. The motion passed. Sita then moved to table discussion of a revised introduction submitted by the Consumer Advisory Panel (CAP) until members had the opportunity to review it. The motion was seconded by Greg Teague and passed with 16 votes in favor and 2 votes against. On a separate issue
related to the introduction, Carol Mussey said she didn’t recall having a
discussion about providing respondents with a list of crisis services.
Sally Rogers said Iowa could not provide respondents with a list of
services because they will all be linked already to a mental health provider.
Jean Campbell indicated that this is required by the Coordinating
Center’s IRB for all studies, but that study sites can decide locally what is
on the list. It is possible to have
a list that consists only of one provider.
Sally said Iowa would reconnect respondents with their traditional mental
health services provider. Sally Rogers moved to
re-open movement of the trans-sexual item from the sexual orientation to a
category under gender. Ruth
seconded the motion. Sally asked if
the item is under gender will gender still be marked by the interviewer as
observed. She said she didn’t
think it would yield reliable information if it were just done by observation.
Roy Crew pointed out that transgender is a gender issue and not a
sexuality issue. Betsy indicated
that what Roy described was her understanding of the literature.
The motion to re-open the movement of the trans-sexual item passed with 13 votes in favor, 4 votes against, and 1 abstention. Sarah Diwan moved to
leave gender as observed with only a male and female category and to add a
trans-sexual question prior to the question about sexual orientation.
The motion was seconded by Miriam Righter.
Jo-Anne O’Connor suggested as a friendly amendment that the transgender
question be used as a site-specific question.
The amendment was accepted. The motion to leave gender as observed with only a male and female category and to make the transgender question site specific passed with 17 votes in favor and 1 abstention. Sally Rogers moved to
re-open voting on whether or not to ask for gross or net income in the common
protocol. Since there had been
agreement on the listserv to ask for gross, she withdrew the motion. There was consensus that gross income would be asked for instead of net. Update
on Multi-site Activities IRB Issues – The
National Environment Jean Campbell reported
that the Coordinating Center’s IRB application was submitted by the December 1
deadline with the most updated version of the common protocol that was
available. Word of approval should
come by December 15, but if the application is expedited, the Coordinating
Center might hear sooner. Jean explained why the
Coordinating Center was required to submit an IRB application for the multi-site
study. She said the University of
Missouri—Columbia has a new compliance system and officer and that it is
responding to the national environment on protect of human subjects.
The University is taking precautions to make sure human subjects are
treated as they are supposed to be, that there is no research fraud, and that
recipients of funds for studies are doing what they said they would do.
There are 400 surveillance agents in the field monitoring research. Fines are $10,000 per violation.
If a person blows the whistle, he/she gets half of the money paid for
fines. Jean said the IRB won’t be
looking at each individual study site. The
Missouri Institute of Mental Health (MIMH) has received an IRB exemption for the
focus groups that are being conducted in conjunction with the site visits.
She cautioned that the exempt status doesn’t supercede local site
requirements. Jean said a very serious
environment is emerging, and the university is requiring a higher level of
accountability. Betsy added that
the monitoring agency is the Office for Protection of Research Subjects and it
is housed under the Department of Health and Human Services.
She referred to a handout with information about its website. (Daphne
Walker-Thoth will fax copies to the study sites.)
Betsy said COSP should be careful and do the right thing.
The Coordinating Center’s involvement with the common protocol does not
consist of a passive handling of secondary data since it has been involved in
the actual creation of the data collection tool. Jean said the cost study
was not included in this IRB application, but will be submitted separately
because to combine the two would have been too confusing.
The cost study application will be submitted after the face-to-face
meeting. Question-by-Question
Document Jean explained that part
of adopting the common protocol is having the Q-by-Q. For validated instruments, we can not add or define anything
that has not already been defined in the author’s Q-by-Q. We are collecting Q-by-Qs on the instruments in the common
protocol. Jean asked that all COSP
members who have an instrument in the common protocol forward the Q-by-Qs to her
immediately so that the COSP Q-by-Q can be distributed the first week in
January. Carol Silverman said she
didn’t have a Q-by-Q, but that she could write documentation. Rich Beaulaurier said at
the interviewer training a suggestion was made to print the Q-by-Q on the
reverse side of the common protocol. Jean
asked him to submit the suggestion to the CP Formatting Work Group for
consideration. Brian McCorkle asked
that all suggestions be sent directly to the formatting work group. Sally Clay pointed out
that it will cost Florida $6,000 to produce 2,100 copies of the common protocol.
Dianne Cote said Florida underbudgeted for the printing. Telecommunications Daphne Walker-Thoth
indicated that there are some issues with the telecommunication system that are
being worked on by the Coordinating Center.
Issues include: (1) some consumers at some study sites still don’t have
access to the listservs and website, (2) some members still don’t have the
software needed to open PDF files, (3) there have been problems with the
teleconference provider—Working Assets, (4) the Coordinating Center’s
efforts to make the most effective use of teleconferences and keep costs down
have hindered some subcommittee work, (5) there have been reports that it is not
easy to find things on the website. To
address the access problems, the Coordinating Center would like for each site to
identify a telecommunications liaison to work with Vicki Wieselthier.
We are trying MCI as a teleconference provider, but this may not be the
provider that we ultimately select. Because
there is a limited budget for teleconferences, the Coordinating Center will work
with each subcommittee and work group to establish individual teleconference
budgets. Daphne asked for members
to forward to her any suggestions for making items easier to find on the
website. She said that Vicki has
been available at the MIMH office every Tuesday from 2 to 5 p.m. Central Time to
provide technical assistance, but the study sites have not contacted her very
much. Tom suggested that there
be a secure section of the website that would serve as a repository of documents
so that members would not have to hunt for items. Interviewer Training Daphne reported that 14
participants attended the interviewer train-the-trainer session in St. Louis
November 15-17, 1999. The
Coordinating Center is continuing to provide support to the interviewer
trainers. Kim Einspahr has been
designated as the multi-site interviewer coordinator at the Coordinating Center.
A listserv has been set up for the group. A training kit has been
developed that includes a manual and a video—the video is in the process of
being completed. Overheads for use
by trainers, a pre and post-test, and sample forms and fliers all have been
distributed. A study guide and
monitoring plan will be forthcoming from the Coordinating Center.
In addition, the Coordinating Center will create and maintain on the
website a question and answer document that carries questions that come up from
the field along with the standardized response.
The evaluation results of the training session were distributed in the
face-to-face packet. Some of the
study sites mentioned that if interviews were going to have to be taped for
quality assurance reasons, this might be an IRB issue.
Jean said another choice is to have interviewers interview colleagues and
tape those instead of taping actual interviews. Jo-Anne O’Connor
commented that the interviewer training was extremely well done. Cultural Competency Jean announced that
Jaime Delgado and Denise Jeremiah will serve as co-chairs of the Cultural
Competency & Diversity Task Force. Jaime
will focus on cultural competency and service delivery, whereas Denise will
concentrate on cultural competency and research. They will develop a plan for this year. The Coordinating Center would like to have a liaison at each
of the study sites to serve on the task force.
The cultural competency plan will be shared at the next face-to-face
meeting. In the meantime, Jaime and
Denise will begin producing a manual about cultural competency, service
delivery, and research. Introduction
to Development of Follow-up Common Protocol Matt Johnsen said
R.O.W.’s assumption is that the same follow-up common protocol will be used at
4 and 8 months with changes to the face pages.
Time varying data are going to be measured at one point.
Matt distributed a mock up of the follow-up common protocol that
eliminated the demographic and lifetime service sections.
A measure will be included related to the satisfaction with use of
consumer-operated services (COS) in addition to some possible questions about
program activities within the COS. Some
additional questions might emerge from the subcommittees and work groups. Matt recommended that
the timeframe used in the follow-up protocol be the past four months.
In sections were instruments are on multiple pages instead of one page,
the time stem needs to be repeated by the interviewer. Matt suggested that a
vote be taken on the follow-up protocol before the end of the face-to-face
meeting and a framework put into place whereby changes could be identified and
incorporated. Brian McCorkle and
Jean pointed out that the follow-up protocol needs to be finished in February in
order to give the Coordinating Center time to submit it to the IRB so that it
will be ready by April when Tennessee conducts its 4-month data collection. Cross-Site
Analysis Discussion Matt indicated that a
discussion needed to get underway about the cross-site analysis that would
continue throughout the face-to-face meeting and upcoming meetings.
Rise Goldstein will be the principal one responsible for the analysis
with support from Brian Yates, Joe Sonnefeld, and Matt Johnsen. Rise walked members
through where R.O.W. thinks the process might be heading and invited
participation. In developing the
plan for the analysis, the Coordinating Center looked at the GFA guidelines, the
study site proposals, the Coordinating Center proposal, and previous multi-site
analysis plans. Decision-making has
also been shaped by efforts of the Steering Committee and Logistics Subcommittee
Group. Some of the analysis
strategy will depend on the common protocol. Work groups will be formed to handle specific tasks. Rise then described
starting points. A first step is
taking the study site and Coordinating Center logic models and determining how
well data fit into the logic models. Some
logic models might need to be modified based on what has happened during the
development stage of the study and site visit information.
Another step is to determine how much power we have to do some of the
analyses. Joe and Rise have been
calling the study sites about power issues.
Now that the common
protocol is almost finalized, we need to check the definitions and
operationalization of some of the outcome variables because in the GFA and
site-specific proposals, outcomes were put in that may now be different.
We need to arrange methodological analysis and describe the population in
the study, incorporate data from fidelity and implementation aspects, look at
ranges of outcomes (monetary and non-monetary), and work on cost-related
analyses. Joe distributed a
generic progression of analysis that depicts the steps, but has yet to include
the specific details that will be worked out.
Describing the population is the first step. The primary focus of the baseline analysis would look at
demographics (total sample across all eight sites, and by site, and by
randomization assignment to compare the control and experimental groups).
If randomization works, we would see that the two groups in each site are
equivalent. Another piece of
descriptive analysis would be to identify variables in monitoring study outcomes
to remove effects that would model effects of the outcome.
Basic statistics will be
done—mean, median, standard deviation, etc.—and we would proceed from there.
We want to characterize distribution of the baseline variables.
We will also review the pilot analysis data which might or might not tell
us anything because it might not be representative.
A question is what do we do about missing data.
We have to find out what is causing it.
If there is missing data, for some kind of analyses, we have to throw out
the data. Imputation was described
as trying to find a way to assign a value if an item of data is missing based on
the characteristics of other people like the respondent or based on the
respondent’s own other responses. Susan Essock said
imputation should not be required nor the throwing out of data.
She expressed dislike for imputation.
She said if you impute, then you do sensitivity analysis.
These are unfashionable ways to pursue things these days. Betsy said the decision needs to be made as a group and
information presented at a level that everyone understands. Betsy asked that decisions such as these need to be put into
a research plan ahead of time so that no one can accuse us of not anticipating
these kinds of analyses. Jean pointed out that
the discussion was introductory in nature so that issues could be put on the
table. She suggested that
members read through the articles distributed in the face-to-face packets to get
some idea of the various approaches. It
was pointed out by members that the articles might be more understandable to
some members than others and we need to make sure that everyone understands.
Sally Rogers offered as
a potential step that some demographic analysis be done early enough to correct
for problems with randomization. Brian Yates indicated
that some outcomes are related to the cost study. These outcomes can be monetarized—inpatient
hospitalization, crisis intervention, emergency department visits, etc. In addition to finding
the average cost per consumer, we will be looking at cost for the program at
large. Between programs within the
COS group we can do some differentiating. We
can compare the costs of the traditional mental health services (TMHS) only
group with the costs of the TMHS plus COSP group.
Bonnie Schell from California has an example of the beginning of a
within-program analysis. Adding costs does add a
new level of interest to outcomes. We
need to get the composite index of effectiveness. A question we ask ourselves is of all the things we do, do
some of the things help more than others. Then
we ask how much these things cost and how to maximize them.
We are adding cost to the basic question – Which procedures produce
which changes inside us that produce behavioral changes?
There are different levels of analyses. Jean announced the
publication of Brian’s book entitled “Measuring and Improving Cost,
Cost-Effectiveness, and Cost-Benefit for Substance Abuse Treatment Programs”
by the National Institute on Drug Abuse. One
copy of the book was distributed to each study site. Joe said some work
groups will likely be formed for the cross-site analysis. There has already been some discussion about validity issues.
He said statistical modeling will be employed.
He said differences in outcomes must be understood.
There will be opportunities for publications and papers. Matt and Jean said most of the discussion will take place as
part of the Research Subcommittee and through the research listserv. Goals
of Face-to-Face Meeting Tom identified the
following goals for the December 5-7 face-to-face meeting: 1.
Complete the baseline common protocol 2.
Get the follow-up common protocol well underway 3.
Work on completing the common ingredients/fidelity/implementation efforts 4.
Devise a timeline for Year 2 with analytic strategies |
Missouri
Institute of Mental Health |
5400
Arsenal Street