Archives   Calendar    Coordinating Center    Multisite Activities    
Project Description  COSP References  Study Sites    Search Our Site   
Table of Contents    Upcoming Meeting Agendas

______________________________________________

COSP Multi-Site Research Design Overview
Chapter 7: Assessing Consumer Operated Services

7.1               Introduction
7.2               Background
7.3               Fidelity and Implementation Assessments
7.4               Basic research Questions
7.5               Development Process
7.6               Proposed Draft Instrument 

7.1 Introduction

This section was developed initially as a background paper for a Steering Committee Meeting of the Consumer Operated Services Program (COSP) Initiative. It has four purposes:

It provides background about the evaluation of program fidelity and program implementation within the COSP Multisite Research Initiative, including a rationale for why it is important to measure these things in the context of a multi-site study initiative.

It describes the process undertaken within the COSP Initiative to develop a draft program fidelity/implementation instrument.

It provides a draft fidelity/implementation instrument, the COSP Fidelity Implementation Tool for review.

It presents a proposed process to: Pilot the proposed fidelity/implementation instrument; refine the instrument; and collect fidelity/implementation data across COSP sites.

Because much of the important work of developing a fidelity/implementation instrument took place in the Common Ingredients Subcommittee, this section seeks to provide a sense of the process and thought that went into the development to the broader group that constitutes the Steering Committee.

7.2. Background

Why look at common ingredients?

One important task undertaken during the first year of the COSP Initiative was to look at similarities and differences among consumer operated service programs included within the COSP Initiative. At first glance, it would appear that there are at least three types of programs included within the COSP Initiative: consumer-run drop in centers, education and advocacy programs, and peer support programs. In thinking about analysis strategies, including the possibility of pooling across all sites in the multisite analysis, we were confronted with several fundamental questions:

Are there enough similarities among the COSPs included within this initiative so that pooling of data across sites makes sense?

Are the internal processes that are set in motion for COSP participants similar across all COSPs, or would we expect that fundamentally different processes are being triggered?

Alternatively, do the apparent differences in program model mask even more basic similarities in underlying structure, program philosophy, peer-to-peer interactions and relationships, which could, arguably, represent the active ingredients involved in any changes?

Are there important differences among programs that follow the same "model"?

These questions flow not only from our own reflection, but parallel Trainor et al. (1997) who in describing 37 Canadian COSPs noted that 80% or more provided mutual support, cultural activities, advocacy, knowledge development, and skill training. Because the answers to these questions are important in developing an analysis plan, rather than attempt to address these questions in an unsystematic way, it was important for the COSP Initiative, as a whole, to explore these questions in a thoughtful, deliberative manner which, ultimately, would be susceptible to empirical validation.

In June 1999, we formed the Common Ingredients Subcommittee. The Subcommittees’ tasks are threefold:

1. Devise a way to identify the elements that compose the COSP at each site. These would include not only descriptions of what services and supports are provided, but also qualitative aspects of the delivery of that care;

2. Discuss and determine which of these elements were the common ingredients of COS programs. Discuss and determine the common ingredients of COS program clusters;

3. Provide advice to the Coordinating Center on the development of strategies and procedures to measure both elements that the COSPs share in common and those that are not shared across all program types, but that may be important ingredients of some.

As byproducts of this activity, we developed a deeper understanding both of the ingredients that are common to COSPs and those elements that distinguish COSPs.

Common ingredients came to be defined to include ingredients of several sorts: some ingredients reflected ways of structuring the organization and others the physical environment of the organization; some ingredients reflected the philosophy and belief systems of the organizations while others reflected the nature of peer support within the group; still others reflected activities and other ways that the organization carried out its work of advocacy and education. Our hope was that we would be able to use the instrument developed with both COSP and mental health programs to assess similarities and differences between the two experimental conditions of the COSP Initiative. This intention will be empirically tested in pilot work to come.

We had already begun some of these discussions in different forums, but felt a need to subsume these efforts together within a single Subcommittee: (1) Some effort had already gone into identifying common processes as a part of the development of the logic model; (2) In the second month of the demonstration, a survey of COSPs was conducted to generate an understanding of the kinds of services that are being provided in participating COSPs; (3) In addition, this effort was closely related to development of a fidelity measure or implementation measure.

We sought representatives from each site as members of the Common Ingredients Subcommittee, and particularly encouraged individuals who participated in the Research Theory Subcommittee discussions, discussions about fidelity and implementation, as well as participants from the Consumer Advisory Panel (CAP) to join this Subcommittee.

The basic questions that we sought to address were: (1) Are there common ingredients of all COSPs?; (2) What distinguishes COSPs from more traditional mental health programs? The initial work of the Subcommittee was somewhat theoretical, testing various formulations of common ingredients against the experiences of the COSPs participating in the COSP Initiative.

The Common Ingredients Subcommittee worked in close coordination with the CAP. Work products from the Common Ingredients Subcommittee were carefully considered by the CAP, and the Common Ingredients Subcommittee then used these revised work products for their ongoing activities. During the course of the initiative, we will empirically test the extent to which these common ingredients are found in COSPs, and alternatively, the extent to which they are or are not found in more traditionally-oriented mental health services.

7.3. Fidelity and Implementation Assessments

Research on the effectiveness of specific models of mental health services has often been limited by lack of documentation of program implementation. In Brekke’s (1988) review of program descriptions of 33 controlled studies, only one satisfied his criteria for complete program description. Implementation assessment is a process evaluation: It describes the program’s implementation, and assesses whether the intended services are being delivered to the targeted participants (Orwin & Goldman, 1995). A crucial first step in documenting program implementation is the specification of critical components for a given model (Bickman, 1987). A second important step is the development of operational definitions for the critical program ingredients (Bond, 1991).

Sechrest et al. (1979) and Yeaton and Seachrest (1981) define treatment strength as the a priori probability that a treatment could have the effect intended assuming that the treatment can be linked theoretically to the effect. They defined treatment fidelity (alternatively referred to as treatment integrity) as the adherence of the actual treatment delivery to the protocol originally developed: Was the program implemented as planned? Fidelity assessment asks, "To what degree was the program implemented as planned?" Implementation should optimally include both treatment delivery and treatment receipt.

Within the mental health field, considerable efforts have gone into developing quantitative measures of program implementation. Much of this was spurred by a desire to operationally define and empirically measure the critical dimensions of the assertive community treatment (ACT) model (McGrew, Bond, Dietzen, & Salyers, 1994; Teague, et al. 1999) that was initially developed in the 1970s (Stein and Test, 1980). McGrew and colleagues (1994) surveyed experts on the model to identify its critical ingredients and assessed 18 program to validate the Index of Fidelity of Assertive Community Treatment (IFACT). They used fourteen items, and scored each from 0 to 1: programs that met or exceeded the expert criterion received a score of 1, and programs that did not meet the standard received scores proportional to the estimated percent of the standard obtained.

Teague et al. (1995) drew from the literature to identify 13 key dimensions of program implementation in the study of an ACT-like intervention for dually diagnosed individuals. Dimensions reflected features of the general ACT model, as well as a particular model of integrated treatment for individuals with dual diagnoses. The Dartmouth ACT Scale (DACTS) extended the original approach, deriving 28 program criteria for fidelity to ACT from multiple sources including observation, interview, and organizational data systems (Teague et al, 1998). For each criterion, a five-point anchored rating was used, with 1 representing the lowest degree of fidelity with the model and 5 representing the greatest degree of fidelity or consistency with the model. The evaluation of the CMHS-funded ACCESS demonstration adopted this fidelity instrument to assess the implementation of case management across the 18 sites (Johnsen, Samberg, Calsyn, Blasinsky, Landow, & Goldman, 1999).

Other models of fidelity assessment include:

The Social Model Philosophy Scale which classifies the extent to which a given treatment program follows a social model approach to treatment (Kaskutas et al. 1988).

The Psychiatric Rehabilitation Environment Scale which has been developed to measure certain aspects of psychosocial rehabilitation programs (Bond, et al., unpublished).

A System Integration Implementation Measure was also used within the ACCESS Demonstration to assess (1) which of 13 system integration strategies had been adopted by the program and (2) the extent to which each of these strategies was implemented (Cocozza, et al., forthcoming).

PASS 3 (Wolfensberger and Glenn 1978) classifies the extent to which a program incorporates normalization related practices and procedures within its operations.

In searching for fidelity instruments appropriate for consumer-operated services, an extensive literature search revealed no instruments that were developed to measure program fidelity or implementation of consumer-operated service programs. While there have been some efforts to understand the range of services provided within the consumer-operated services (Van Tosh & Del Vecchio, forthcoming; Trainor et al., 1997), conceptual work to develop a national survey of self-help organizations (Goldstrom, unpublished, conversation), and some preliminary efforts to identify "essential ingredients of consumer-operated drop-in centers" (Mowbray, unpublished), we found no fidelity instrument that could be adopted for immediate use within the COSP Initiative. Thus, we determined that it would be necessary to develop a fidelity/implementation tool within the COSP initiative in order to address issues of implementation.

7.4. Basic research questions

The basic research questions of this aspect of the multisite evaluation include several questions of an exploratory nature, as well as several of a confirmatory nature. By exploratory questions, we mean questions for which we have no explicit hypotheses, but which we are interested in addressing by an unconstrained look at the available data. By confirmatory questions, we mean questions for which our conceptual work has led to preliminary hypotheses which we can empirically test with the data collected.

Exploratory questions that may be addressed include the following:

How is each COSP organized to deliver its services to program participants?

How is each COSP delivering its services to program participants?

How do the structure and services of COSPs vary from one to another?

How do consumer operated services differ from traditional services at each program site?

How do consumer operated services differ from traditional services across all program sites? Are there sites where these differences are particularly large? Are there sites where these differences are particularly small?

Confirmatory questions that might be addressed include the following:

Are the Common Ingredients identified by CI and CAP actually common to all COSPs participating in the COSP Initiative?

To what extent do the COSPs within the COSP Initiative actually exhibit (or implement) these common ingredients?

To what extent do program participants indicate that they have received these common ingredients?

The exploratory questions are consistent with implementation assessment approach, which, without assuming a common model, attempts to describe, as fully as possible, what is actually happening within the program. The confirmatory questions are more consistent with fidelity assessment, which posits an ideal model and measures conformity with that ideal.

Several limitations and challenges have confronted us as we have strived to create, adapt or adopt an instrument which could assess program implementation or fidelity for the COSP Initiative. It is important to acknowledge these limitations from the outset, as they constrain the approach taken, as well as the speed with which we are able to proceed.

1. Different program models: The COSPs included within the COSP Initiative represent at least three distinct program models (consumer run drop in centers, peer support programs, and education/advocacy programs). While all share certain structural similarities (i.e. board composition, budgetary control, and consumers as staff), they are organized to do quite different things and carry out quite different activities. While they share many common components, there are important differences among programs as well in terms of mission, philosophical orientation, program participants, etc.

2. No clear definition of COSP: While Bickman (1978) suggests that a crucial first step in documenting program implementation is the specification of critical components for a given model, prior to the COSP Initiative, there has been limited information about the critical components of COSPs or even the three types reported here. The previously cited work of Mowbray (unpublished), Van Tosh and Del Vecchio (unpublished), and the preliminary efforts of Summerfelt presented the starting point for these activities. While Van Tosh and Del Vecchio have, for example, provided a useful compilation of program features across a range of COSPs, they do not attempt to provide a list of critical components that define COSPs.

3. Programmatic inconsistencies: Because of the different ways that programs are organized (site-based vs. in vivo), some criteria may be appropriate for most, but not all, programs. For example, rating issues of program environment would be problematic if all program related interactions occur at the participant’s home.

7.5. Development Process

The work of developing a COSP fidelity/implementation instrument may be conceptualized as falling into five phases: identification of critical ingredients; operationalization of concepts; piloting and refinement of the instruments; implementation of fidelity/implementation assessment across all sites; and broader dissemination of the instrument. At the present moment, the Common Ingredients Subcommittee has moved through the first two phases. Phases III and IV will follow completion of Phases I and II. The wider dissemination of this tool (Phase V) will be highly dependent on the demonstrated utility of the instrument.

Phase I: Common Ingredients: Identification/Definition

During Phase I, the primary task of the Common Ingredients Subcommittee was to identify, organize, and define a set of common ingredients which distinguished consumer operated services from other programs. To accomplish this, the Subcommittee met frequently via telephone conference call, and relied extensively on work carried out between meetings by Subcommittee members and staff of the coordinating center.

A variety of background materials were distributed to members of the Subcommittee at several points in time to ensure that Subcommittee members had the benefit of relevant prior work in the area. These included articles which provided examples of studies of program fidelity, implementation assessments, and fidelity instruments themselves. In addition, several articles provided discussions about how particular fidelity instruments had been developed.

Early on, Subcommittee members recognized the important conceptual work undertaken by Carol Mowbray (unpublished) in preliminary work aimed at developing a fidelity instrument appropriate for consumer drop-in centers. While there was not universal endorsement of all aspects of this preliminary work, her work provided an important starting point, and an example of how we might structure our own work. Mowbray grouped her criteria into two overarching domains for conceptual clarity. "Structure" reflected the "relatively stable characteristics of providers of care, of the tools and resources they have at their disposal, and of the physical and organizational settings in which they work" (Donabedian, 1980, p.81). "Process" refers to specific and observable activities in services or in the method of delivering those services (Mowbray, unpublished).

We structured our own discussion of common ingredients in what came to be six domains: structure; environment; belief systems, peer support, education, and advocacy. Like Mowbray’s criteria, these could be grouped into two overarching domains: structure (which would include structure and environment); and process (which would include belief systems, peer support, education, and advocacy).

Ultimately we developed a list of common ingredients and definitions which we shared with the Consumer Advisory Panel at the August 9-11 meeting of the COSP Steering Committee in Washington, DC. The CAP considered this list of common ingredients during extensive discussions, and reached agreement both about (1) the inclusion of particular common ingredients within the list; and (2) definitions of these particular common ingredients. The work of the CAP at this stage was particularly crucial in three respects: (1) CAP members included individuals intimately involved with the day-to-day operations of consumer operated services, and input from these individuals provided an important initial test of the relevance of the ingredients identified; (2) CAP members represented each of the eight study sites, ensuring wider generalizability than one program model alone; and (3) the CAP was able to wrestle with questions about whether common ingredients were, in fact, common across all sites. These modifications were then fed back to the Common Ingredients Subcommittee, which used the work products of the CAP as it entered the measurement phase of the Subcommittee’s work.

The common ingredients approved by the CAP and adopted by the Common Ingredients Subcommittee were presented in Section 3 of this document.

Phase II: Measurement

At the beginning of the measurement phase, in September 1999, the work was divided across three workgroups. Deciding that the work may go more quickly with smaller groups we asked for volunteers to sign up to work in three smaller groups: Structure/Environment, Education/ Advocacy, and Belief Systems/Peer Support. Each workgroup was responsible for two of the domains within the common ingredients list.

These groups were charged with moving from the domain and definition stage into the indicator/measurement phase. Each workgroup had both researcher and consumer input.

We expected that the life of each work group would be relatively short, in that it will be devoted to a particular (and concrete) set of tasks, which should be accomplished in a relatively short period of time. The responsibilities of these three work groups were similar, differing only in the area that they will be directing their efforts toward:

(1) Identify recommended data collection strategies for each common ingredient (i.e. collecting it as part of the individual common protocol, site visit, other data collection activity, etc.) This also involved developing questions that can be used in gathering this information.

(2) Develop anchored ratings for each common ingredient within the particular domains included in the Common Ingredients list.

(3) Return to the Common Ingredients/Research Subcommittee meeting prepared to present these anchored ratings for consideration.

To ensure that each group had some representation of both researchers and consumers, as both sets of insights would be crucial to effectively carrying out these tasks, we proposed membership of these workgroups where people had not volunteered. In practice, the work groups did not select chair persons, but did select individuals to report out the work group activities to the Common Ingredients Subcommittee, and coordinating center staff recorded all decisions of these groups. The work groups did, however, fulfil the goal of ensuring that work groups consisted of strong working partnerships of consumers and researchers.

The work groups were allowed to develop more than one anchored rating for some ingredients, where they thought that this was necessary to have a more reliable measurement. Thus, the "consumer operated" domain may, for example, have several different indicators. In addition, early on, there was some discussion about whether the focus of the information gathering associated with fidelity/implementation assessment should be at the program level or the participant level: For example, should questions about the program be addressed to program directors or program staff, who would answer for the program as a whole, or should questions instead be addressed to program participants who would provide their individual view of these activities? The answer is not an easy one, and while some fidelity/implementation data collection efforts focus exclusively on collecting information from program staff, others employ information from a wide array of sources including administrators, staff, and program participants (i.e. CMHS Housing Fidelity Study). While we reached no decision that would fit across all items, we agreed generally: (1) to address most questions at the program level; (2) to plan on incorporating questions of participants into focus groups rather than adding large numbers of questions to the follow-up common protocol; and (3) to utilize multiple methods where appropriate so that there could be some triangulation in order to validate observations.

We also discussed how to deal with traditional mental health services. We came to agreement among all present that it would be the wisest course of action to ask the same basic questions of the COSP and Traditional MH Services so that we can understand both and document differences between the Traditional and COSP + Traditional conditions. Using the same instrument and asking the same questions across both COSP and Traditional will allow us to make defensible comparisons between the two. This led to a discussion about the need for site visitors (discussed in Section 7) to be keenly aware of the differences between what is stated and practice, and ways of determining these differences.

From September through November, the three work groups met regularly to complete their assigned tasks. In addition, regular, but slightly less frequent meetings of the Common Ingredients Subcommittee continued, providing a forum to provide feedback to workgroups involving all Subcommittee members, and providing a vehicle to ensure accountability of the workgroups to the overall goal of the Common Ingredients Subcommittee.

The COSP Fidelity Assessment/Common Ingredients Tool (COSP/FACIT) represents the combined work of the Common Ingredients Subcommittee and its three workgroups. In its current form, this is a draft, open to comment, critique and suggestion from participants throughout the COSP Initiative. All domains have been operationalized, though it should be stated that the work groups had particular difficulty developing satisfactory approaches to some common ingredients, particularly those ingredients reflecting complex ideas.

Phase III: Pilot/Refinement

In the pilot/refinement phase, we are attempting to use this instrument to assess consumer operated service programs as a part of the scheduled site visits. For each visit, two to three site visitors participate in site visits 2-4 days long. Prior to the site visit, they receive documentary information about the COSP and traditional services associated with the study site. Site visits will consist of tours of programs (where appropriate), interviews with program directors, supervisory staff, program staff, and focus groups with program participants. In addition, site visitors meet with members of local CAPs and members of the research team. In the course of these activities, they gather information that can be used to assess the program with respect to each of the items.

For the remaining site visits, raters will be asked to rate both the COSP and the traditional program at each site on these criteria. Each site visitor will be asked to rate these criteria independently, before developing a consensus rating. This process will allow for assessment of interrater reliability. Using information from site visits, we will collect information from site visitors about ratings that are particular problematic for site visitors, areas in which there is inadequate precision in the proposed ratings, or items for which there is a lack of clarity in their discussion. In addition, we will attempt, using this initial data, to discern particular response patterns among program models, to develop some tentative hypotheses about similarities and differences across sites.

Working with the Common Ingredients Subcommittee, this information will be incorporated into a revised version of the instrument which will be used during the second round of site visits.

Phase IV: Implementation

The fidelity/implementation assessment will be fully implemented during the next round of site visits, which begin in Year 3 (2000-2001) of the COSP Initiative. During this round, the implementation of each COSP will be assessed using the revised instrument. For each site visit, a team of at least two site visitors will rate each item, and internal reliability of the revised instrument will again be assessed. All information will be collected and stored within a project-level database, for use in cross-site data analysis.

7.6. Proposed Draft Instrument

In COSP/FACIT, each project is rated on 47 major characteristics, called ‘ratings.’ These ratings are grouped into clusters, which are successively combined into larger clusters. While some of the ratings or clusters may interrelate with each other, an effort has been made to specify them as independently as possible.

Each of the ratings is scored along a continuum of 3-5 levels. In each case, the lowest level of a rating stands for a level which would indicate that the common ingredient is not present, while the highest level receives the maximum score which the project can obtain for that specific rating. To enable objective decisions, a narrative accompanies each rating, explaining its rationale and criteria, and sometimes providing examples.

Implicit in the development of the COSP Fidelity/Implementation Tool were a number of assumptions first articulated by Wolfensberger and Glenn (1978, p.7) :

1. Desirable as well as undesirable program components can be specified.

2. Program principles can be so specified as to be applicable to virtually any type of program.

3. Specified program components can be graded along a continuum of specifiable increments.

4. In many cases, the criteria for assigning a program component to a level of such a continuum can be objectified and concretized.

There was considerable discussion about the wisdom of grading all program elements across such a continuum. The principal arguments for such a grading include the following: (1) Grading on a common scale makes it possible to easily compare multiple programs in a way that would be much more difficult (or even impossible) using only qualitative data, (2) Use of such scales is more amenable for use within a quantitative data analysis approach, and (3) The proposed approach follows the example of a number of other fidelity or implementation assessment tools (DACTS, PRES, PASS, etc.) The principal arguments against such a strategy included reservations that such a data reduction strategy would not allow for sensitive nuanced analysis, and that we would lose data if we moved from continuous variable to ordinal level variables, there was agreement to attempt this with the assurance that: (1) raw data (such as actual percentages) would be retained for possible use in modeling as needed, and (2) qualitative data would be retained (i.e. site visit notes, etc) and consulted to provide qualitative insights.