COMMON INGREDIENTS OF
CONSUMER OPERATED SERVICES
Prepared by the Common Ingredients
Subcommittee of
Consumer Operated Service Program Multisite
Research Program
March 15, 2001
INTRODUCTION
This document provides an introduction
and update to the work o f the Common
Ingredients Subcommittee of the Consumer-operated Services Program (COSP)
Multisite Research Program. The COSP Multisite Research Program is
sponsored by the U.S. Center for Mental Health Services, and is designed to test
(using random assignment) the value of adding participation within consumer
operated services to traditional services in eight sites within the U.S. This
document is intended to provide background on the work the Subcommittee has
carried out in recognizing, defining, and measuring the common ingredients of
consumer operated services. We begin with some background of the Common
Ingredients Subcommittee, its work, and the List of Common Ingredients.
At the inception of this grant program, the Federal Government Project Officer
(GPO) emphasized, in an orientation meeting with the Coordinating Center (CC)
staff, the importance of describing the interventions and documenting common
components across sites for analytic purposes. The Coordinating Center’s
proposal, as well as the proposals of several of the sites, acknowledged the
importance of careful implementation evaluation, both about what the
consumer-operated
programs were doing, as well as how well they were implementing
these efforts.
As early as the first Steering Committee (SC) meeting of the COSP Multisite
Program, there was discussion about the need to examine carefully the nature of
the participating consumer-operated services, and to identify the ingredients
that
these services may have in common. An early attempt to characterize such
ingredients was presented at the second meeting of the SC by members of the
CC from ROW Sciences, but this attempt was criticized by some SC participants
as leaving out critical elements around the structure and organization of the
consumer-operated services, as well as the principles underlying them.
With this as a background, in June 1999, a Common Ingredients Subcommittee
was formed to identify similarities and differences among consumer-operated
services, including any common ingredients that extended to all or a significant
proportion of participating consumer-operated research sites. Representation for
this Common Ingredients Subcommittee included consumers, researchers,
representatives of the Coordinating Center, and federal representatives. During
initial listserv discussions in July 1999, representatives from several
participating
consumer-operated services nominated particular program characteristics as
common ingredients, and the COSP CC Principal Investigator, Jean Campbell,
shared an extensive list of ingredients that she and colleagues from other
consumer-operated services had developed.
Common Ingredients of Consumer-operated
Services
One product of this effort was a
consolidated list of Common Ingredients of
consumer-operated services developed during July and August of 1999 through
a collaboration between Eileen Zeller of ROW Sciences and Sally Clay, co-chair
of the Common Ingredients Subcommittee. The initial CC list contained over
160 entries. The final collaborative effort was refined to about two dozen
common characteristics of consumer-operated service programs, and these were
subdivided into categories based on the work done by Carol Mowbray. These
common ingredients were discussed and revised by the Common Ingredients
Subcommittee, and the list was further developed in detail and refined by the
Consumer Advisory Panel (CAP) of the COSP Multisite Research Initiative.
Participants in the Common Ingredients Subcommittee also shared the evolving
list with and received feedback from their local Consumer Advisory Panels.
The Common Ingredients Subcommittee as a whole acknowledged that the CAP
was a particularly appropriate body to bring the document to final form, as the
definitions of common ingredients were based on common ingredients found
within and among consumer-operated services, and the CAP possessed within it
a wealth of direct and personal expertise about these consumer-operated
services. It was also acknowledged at the outset that, while we thought that
these ingredients might share much in common with consumer-operated services
not participating in the multisite research initiative, our effort was bounded
by the
COSP sites, and thus our goal would be to develop common ingredients
applicable to these sites rather than to the universe of possible
consumer-operated
services.
At its August 1999 Steering Committee meeting, the CAP approved the working
definitions of the common ingredients. Zahira Duvall, CAP chair, reported that
"through a marathon dialogue session, the CAP revised the common
ingredients
document. CAP was proud of the level of consensus and amount of participation
in the process of refining the document" (SC minutes, August 1999). The
latest
version of this common ingredients document, with the definitions approved by
the CAP, is given in Appendix 1.
Matt Johnsen, of ROW Sciences and co-chair of the CI Subcommittee, explained
to the Steering Committee that "the Common Ingredients [Subcommittee]….
would help implement the fidelity evaluation required by the Guidance for
Applicants (GFA)." The Common Ingredients Subcommittee, through a series of
working groups, then developed measures to assess implementation of these
components across all sites (SC minutes, August 1999).
Throughout the fall of 1999, the Common Ingredients were assigned by category
to subgroups of the Common Ingredients Subcommittee, and these subgroups
further discussed and operationalized each ingredient to develop the program
implementation, or program fidelity instrument, called "FACIT," for
use within the
COSP Multisite Research Study.
Work on the common ingredients has evolved into three
distinct areas:
1) the Book,
2) the FACIT,
3) Site Visits/Implementation.
Common Ingredients Book
In December, Tom Summerfelt, then SC chairperson, indicated that
"the common
ingredients would be a great place to begin working on the first publication to
come out of the COSP. He suggested that a book be developed in which
consumers at each study site could describe their program and sum it all up with
the common ingredients" (SC minutes, December 1999).
A vote was taken, and approved by consensus, to "form a group making sure
each study site and the federal representatives are included to pursue
development of a book on common ingredients with Sally Clay, Ed Kramer, and
Matt Johnsen serving as editors" (SC minutes, December 1999.) The editors
selected by the Steering Committee met by teleconference a few weeks following
the December 1999 Steering Committee meeting, and came up with a proposed
structure for the book. Subsequently, Ed Kramer indicated that he would not be
able to continue in this role, and several additional individuals volunteered to
take
on leadership roles within the process.
The proposed structure was developed by the editors as "Chapter Guidelines
for
CAP Members Draft 3/10/00". From these guidelines evolved a formal
outline/table of contents that was discussed at the June SC meeting of the CAP,
and approved by the Common Ingredients Subcommittee.
The Fidelity Assessment Common Ingredients
Tool: The FACIT
The fidelity instrument that came to be
known as the FACIT (Fidelity
Assessment/Common Ingredients Tool) emerged as a later product of the
Common Ingredients Subcommittee. As indicated, the FACIT was developed
through an extensive series of teleconferences of several subcommittees of the
Common Ingredients Subcommittee. The composition of this committee included
consumers, researchers, representatives of the Coordinating Center, and federal
representatives, with especially heavy involvement from consumers involved in
the COSP Multisite Program. The process used as a starting point for
deliberations the revised List of Common Ingredients that resulted from the
extensive discussions of the CAP. The first complete draft of the fidelity
instrument was available in draft form for the December 1999 Steering
Committee Meeting. Following this meeting, the instrument was further revised,
and then piloted during the first round of site visits for the COSP Multisite
Research Initiative, which occurred from October 1999 through June 2000. As
indicated by its name, the FACIT instrument began with the common ingredients
as identified during the first phase of the committee’s work, and then
developed
one or more ratings for each of these ingredients. The attempt was to develop a
series of anchored ratings that were measurable, observable, and for which there
was inter-rater reliability.
At the June 2000 meeting of the Steering Committee, a report of the pilot was
presented. The pilot suggested that most, but not all, of the items appeared to
be
reasonable, that some of the items needed further refinement, and that there was
a fair degree of inter-rater reliability. Most recently, the instrument has been refined
further based on pilot results,
and further input and guidance will be required from
the current Common Ingredients Subcommittee, with review by the CAP, in order to
complete revisions for use during the next round of site visits.
Site Visits
Some of the work of site visitors
during the first round of site visits revolved
around common ingredients. During three-day site visits, the visitors toured
each
participating COSP (with the exception of California and Missouri); conducted
interviews with COSP directors, staff, and participants; and conducted focus
groups with COSP participants. In addition, they met with personnel and
participants in traditional mental health services in these settings. Each site
visit
team consisted of 2-3 site visitors selected by the COSP Coordinating Center.
Site visitors received common training, and all utilized site visit guides that
provided a series of questions to collect common information across all COSPs
and traditional mental health services. They later used this information to
independently rate the consumer-operated services on each of the elements of
the FACIT.
During the second round of site visits, it is anticipated that the visit may
focus in
on an even more fine -grained set of questions to describe the interventions.
This
will include again administering the most recent version of the FACIT. The
concerted input of the Common Ingredients Subcommittee will guide the site visit
teams in collecting the information needed to understand similarities and
differences between participating COSPs and traditional mental health settings.
5400
Arsenal Street