COSP Multi-Site Research Overview
Section 5: Multisite Research Design and Study Issues
Section 5: Section 5.pdf
*
-
*If
you do not have the reader you can download the program at no cost from: 
http://www.adobe.com/supportservice/custsupport/download.html
-
Scroll
down to the ACROBAT READER line
-
Choose
WINDOWS
or MAC.
-
We can send you the Acrobat reader on
disk if you cannot download it successfully. ADOBE provides help for
downloading too. Click here
to read.
This section summarizes a variety of logistical and
implementation decisions addressed by the Logistics Subcommittee and adopted by
the Steering Committee of the COSP Multi-site Research Initiative.
These issues have been organized in four categories:
identification of target population; data collection; recruitment
procedures; and miscellaneous issues.
5.1
Identification of Target Population
In this section, we lay out details of how the target
population is identified, including inclusion and exclusion criteria and
procedures for screening individuals under study.
5.1.1
Exclusion and Inclusion Criteria
Decisions related to inclusion and exclusion criteria were
guided by the GFA, which laid out the basic parameters for participation within
the study. We have further
specified and operationalized these criteria as follows.
5.1.1.1
COS Involvement
More than minimal involvement in the past 6 months in a COS
similar to the ones under study (i.e., drop-in centers, education and advocacy
programs and peer support programs) results in exclusion from the study. “Minimal
involvement” is defined as more than three visits/meetings at COSP like those
under study. In order to qualify as
a COS, the program has to meet the definition of COSP using the first three
common ingredients structural criteria.
5.1.1.2 TMHS
Involvement
In order to be included in the study, the GFA states a
potential participant must be a recipient of traditional mental health services
for at least one year. The
individual does not need to be a recipient of services with the same TMHS
provider, but must have been on a TMHS roster during that period. The traditional mental health services involvement must meet
three core criteria: recency, longevity, and intensity.
1)
RECENCY OF INVOLVEMENT (an individual coming into the study would have to
have had received services within the past 4 months prior to the date of
“application” to the study);
2)
LONGEVITY (an individual coming into the study would have had to be
involved with a TMHS provider for a full 12 months prior to the date of
application to the study);
3)
FREQUENCY (an individual coming into the study would have to have
received services at least 4 times in the past year, for example, 4 visits with
a case manager, 4 visits for medication checks, etc.).
5.1.1.3 TMHS Diagnosis
Participants in this study must meet the diagnostic
criteria in the GFA which states that the target population for this grant is
individuals with a serious mental illness.
Therefore, applicants to the study must have a DSM-IV Axis I diagnosis of
serious mental illness. Co-occurring
substance abuse disorders are acceptable for entrance as long as the person has
a mental health diagnosis in addition to their substance abuse diagnosis.
Severe mental illness is usually defined as meeting three criteria:
diagnosis, duration, and disability. Certain DSM-IV diagnoses are usually
associated with severe disability: schizophrenia, schizoaffective disorders, any
diagnosis with psychotic features, major depression, and some personality
disorders. In addition, duration is
usually a factor in determining disability, with two years being the commonly
accepted cutoff. Finally,
functional impairment (disability) as a result of the diagnosis is needed.
Criteria two and three (duration and disability) are difficult to
ascertain when simply screening for entry.
Therefore, we must rely primarily on diagnosis for inclusion.
Sites may impose additional inclusion and exclusion
criteria as long as they do not conflict with these basic criteria.
In their recruitment, sites should target the programs and
people within the TMH system that serve people with major mental illness.
Baseline data on diagnosis and functional impairment (e.g., employment status)
should be examined by individual sites and by the multi-site study after 3
months of data collection, and periodically thereafter to insure that we are
reaching our intended population for the study in conformance with the GFA.
5.1.1.3.2
Recording Diagnosis for the Data Repository
The collection of an individual’s mental health diagnosis
will not be made during the face-to-face interviews with the participants in the
study. Each study site must make
arrangements to collect diagnosis from the TMH provider.
Each site should use the data entry structure created by the Data
Repository committee to enter diagnositic information. This will allow merging
of the diagnostic information with the data from the CP.
Every attempt should be made to obtain the most complete and up-to-date
diagnosis possible. All Axis I and
Axis II diagnoses should be recorded. Three fields for Axis I and three fields
for recording Axis II diagnoses using the numbering nomenclature found in the
DSM-IV (e.g., 395.80) will be available.
5.1.2
Screening for Exclusion and Inclusion Criteria
Appendix C contains a copy of a Screening Questionnaire
that can be used by sites to insure that they are screening in and screening out
people who might not be eligible for the study based on the criteria in the GFA.
Use of this Screening Questionnaire is not mandatory, however, each site
must have a way of tracking the screening criteria on the last page of the
Questionnaire for reporting to the CC. All
sites should send a copy of their final, complete inclusion/exclusion criteria
to the CC.
5.2
Data Collection
A series of decisions have been made about the manner in
which data will be collected within the COSP Multisite Research Initiative.
5.2.1
Mode of Data Collection
All interviews are conducted in a face-to-face mode rather
than self-administered because of concerns about literacy.
The paper-and-pencil CP instruments provided by the CC can be used, or
computer-assisted interviewing (CAI) can be performed.
The TN site has developed an Access program which can be used for
computer-assisted interviewing. Using
computer-assisted interviewing conserves resources for data entry, but special
steps need to be taken to preserve and back up the data and to verify the data
entry. Data entry verification will
not be necessary with CAI.
5.2.2 Data Collection Schedule
The baseline version of the Common Protocol (CP) is
administered to all participants who sign an Informed Consent and agree to be in
the study. The date of the baseline
interview becomes the index date from which all subsequent interview
appointments should be calculated. The
target dates for data collection should not be altered if the intervention does
not start in a timely way, as it may not in at least one site. The Follow-up CP
should be re-administered at 4 months, 8 and 12 months post the baseline date of
administration. The 12-month
follow-up is optional but highly recommended.
5.2.2.1 Data
Collection Window
A one-month window around data collection points for follow
up is acceptable. Sites can conduct
the follow-up interviews anywhere between one month before the collection period
to one month after the collection period. For
example, for the 4-month data collection point, sites can collect data from
month 3 to month 5. If interviews
take place after the one-month window, the site should still collect the data.
Calculations can then be made from the date of the baseline to the date
of any particular follow-up to determine what percentage are outside of the
acceptable window.
5.2.3
Randomization Procedures
Randomization to the experimental (COS + TMHS) and control
(TMHS only) will occur after the baseline CP is administered or co-incident with
it. Co-incident means randomization
will occur within one working day.
A multi-site
randomization process was not agreed upon: however, all sites will block on
gender and will generate their own randomization schedules based on gender.
Sites are free to add blocking variables to the randomization process as
long as they document their randomization procedures for the CC. Sites should
guard against any attempts to influence the random assignment of participants by
having the PI closely monitor the process, by taking the random assignment out
of the hands of the interviewers, and by developing procedures that are
“game-proof.”
Appendix D contains a
summary of the randomization procedures across sites and an example of a
randomization schedule that can be generated by Proc Plan using SAS software.
5.3
Issues Around Recruitment and Induction into the Study
5.3.1
Recruitment Materials and Procedures and Demand Characteristics
All
recruitment materials and procedures (brochures, flyers, information provided in
group or individual orientation sessions, common induction materials and IRB or
consent materials) are developed with caution so as not to alert potential
participants to the hypotheses under study.
Potential participants are fully informed about the study, its risks and
benefits and the intent of the data collection. But it is important not to alert them to the specific
hypotheses under study in any written materials, group presentations, screening
meetings, or subsequent contact. Doing
so could create demand characteristics that subtly suggest to the participants
that they answer the questions in the CP in a particular way. It is also very important that there not be an imbalance
between the experimental condition (COS + TMHS) and the control condition (TMHS
only) in terms of this information or these demand characteristics.
For example, we would not want one group to be singled out and have high
expectations for the gain communicated to them.
5.3.2
Guardianship
We recognize that the ability to give informed consent and
to successfully participate in this project is not necessarily limited by
guardianship status (that is, some individuals without guardians may not be able
to give full and knowing consent while some individuals with guardians may).
Unless an individual site’s IRB has required that they exclude
individuals with guardian’s, every attempt should be made to include those
individuals in their recruitment process. At the time of writing, two sites
planned to exclude individuals with guardians (FL and ME) but they will revisit
their procedures. The tracking database allows us to track the number of
individuals with guardians who apply for the study.
5.3.3
Common Induction and Sampling Procedures
In the sampling procedure adopted, we sample/recruit
consumers who are receiving services at a TMH provider who express sufficient
interest in motivation for project and who go through a common induction
procedure.
The common induction process fully informs prospective
study participants about randomization and what is expected of them. We agreed
that all sites would have a common induction process as voted on in the 11/23/99
and 11/29/99 SC calls. The essential elements of a common induction process are
the following: a) that potential participants be informed that this study
involves 7 sites and is funded by the CMHS, b) that they have a 50-50 chance of
being randomized into each of the two conditions; c) that these conditions be
described briefly and neutrally; d) that language discouraging crossover be
included in the common induction in a positive way (e.g., "If you are
randomized to the TMHS condition, the COS will not be available to you for a
period of one year"). This language will need to be tailored to fit
site-specific situations. Whether this language appears in the Informed Consent
or in a separate stand-alone document will be determined by each site.
Sites may overlay their own induction process upon this common induction
process.
As a method of record keeping, all sites should submit a
final version of their induction procedures and materials (e.g., handouts,
fliers, etc) to the CC. This will
allow for follow-up/fidelity checks, and will facilitate future summaries for
articles regarding methods.
5.4
Miscellaneous Issues
A number of additional issues were adressed by the
Logistics Committee.
5.4.1
Intention-to-Serve Analysis
The overall study approach agreed upon was an
“intention-to-serve” (a.k.a., Intention to Treat) design where all
individuals enrolled in the study are tracked regardless of the amount of each
condition they receive. This
approach has power implications. That
is, the more individuals who are randomized to the COS condition who don’t
participate in it, the less likely we will see a difference between the COS +
TMHS and the TMHS group alone.
Therefore, it is important that each site emphasize
strategies to maximize the retention of study participants in the respective
conditions. Secondly, whatever each
site can do within ethical constraints to keep crossover to a minimum will help
preserve the power of the study. If individuals refuse to participate in a
scheduled follow-up interview, the interviewer must assess whether the
individual simply does not want to participate on that day, or is refusing
participation for the duration of the study.
The interviewer should make several attempts (being sensitive to the
individual’s wishes and rights as a research participant) to complete a
follow-up interview, unless the individual emphatically refuses.
“Administrative withdrawals” are individuals who cannot
participate in the study condition to which they are assigned (e.g., they are
arrested shortly after random assignment and would not be able to receive either
COS or the TMH service). For these
individuals we can do early replacement of them in the study, meaning we can
treat them as though they were never in the study and randomly assign someone
else to take their place. We cannot do any other kind of replacement of
participants since we are using an “intention to serve” approach.
Some sites may over-sample to insure they have sufficient power, but they
will still track every person entered into the study.
If individuals want to re-enter the study after an
administrative withdrawal they will not be allowed to.
If an individual withdraws from the service component, they will still be
a research participant. If a participant wants to withdraw from the research and
the services, he or she can do that and request that their data be destroyed.
In that case, locally held data should be destroyed and Matthew Hile
should be instructed to delete data from the repository.
Any withdrawal or re-entry scenario that does not fit the
above description should be posted to the Logistics Subcommittee listserv for
comment and final decision making so that we can keep a log and a protocol for
these unusual situations.
5.4.2
Reviewing Informed Consents after Baseline
Some sites are required by their local IRB to review the
original informed Consent document, or an abbreviated one at each data
collection point. Other sites are
not required to do so. Whether the
Informed Consent is reviewed at each assessment point is left to the discretion
of each individual site and the requirements of their local IRB’s (at present,
Missouri, Connecticut, Tennessee, and Pennsylvania do not review the Informed
Consent after the initial signing; Maine and Illinois review annually and
Florida orally reviews consent at
each follow-up). Sites may wish to
bring the original signed informed consent to the follow-up interviews in case
the participant wishes to review it.
5.4.3
Position of Site Specific Measures in the Common Protocol
Sites may add site specific measures to the Common Protocol
with the following guidelines: a small number of site-specific items may be
added at the end of any one section.
The maximum number of site-specific questions to added by any one site
can be 20.
If an entire scale is to be added by a particular site, it must be added
at the end of the Common Protocol.
5.4.4
Tracking of Participants
The tracking of study applicants needs to occur to provide
the information that will be requested by the CC each quarter. The TN site has
developed an Access database and screen program that will track these data
elements and generate the needed reports. This
tracking software can also be used as a tickler system to help plan and manage
the interviews. Managing the
interviews and interviewer resources is critical for having complete datasets.
Each site is strongly encouraged to use this tracking
software. Sites that adapt the software or develop their own should insure that
they can generate reports that provide the information required in the quarterly
report and/or any other tracking needs that the CC may have.
We will not to track any demographic items on potential
participants because they will not have given their permission at that point.
It was generally agreed that people who refuse to participate in the
study not be asked “why” they refuse. They
might be asked if they are willing to share why they refused, and then if the
information is volunteered, it can be used to help improve future recruitment. Sites can keep track of that information if they choose, but
it is not mandatory.
5.4.5
Confidentiality Certificate
Every site has been urged to apply for a Confidentiality
Certificate from SAMSHA. The
Certificate provides some protection for participants in research studies to
help keep the information they provide confidential and prevent it from being
summoned by a court of law if the research participant is involved in a criminal
justice proceeding. Each site must
apply for the Confidentiality Certificate when their local IRB’s have signed
off on the project and on the Informed Consent. Please note that there
is specific language that is required in an Informed Consent to obtain a
Confidentiality Certificate.
5.4.6 Pledge of
Confidentiality
Sites should instruct every interviewer about the
importance of maintaining information obtained during the interview in the
strictest confidence. Interviewers
are verbally instructed about the importance of confidentiality and also sign a
pledge of confidentiality.
5.4.7
Interviewer Observation Rating at End of Each Interview
At the end of each baseline and follow-up interview, the
person conducting the assessment will complete a brief questionnaire designed to
record any difficulties with the validity or the reliability of the
participant’s answers to the questions in the Common Protocol.
The comments/notes taken by the interviewer throughout the CP interview
should be perused prior to completing the observation scale.
Any interview judged unreliable should be reviewed by the interviewer’s
supervisor.
[ Back ] [ Up ] [ Next ]
|
5400
Arsenal Street