This Coordinating Center (CC) proposal is a collaborative effort between two research and evaluation centers: The University of Missouri—Columbia School of Medicine’s Missouri Institute of Mental Health (MIMH); and R. O. W. Sciences, Inc. in Rockville, Maryland. Staff from these institutions in collaboration with the National Empowerment Center (NEC), the National Mental Health Consumers’ Self-Help Clearinghouse, and numerous consumer and professional consultants have designed a plan to provide services to the eight study sites that will be funded under the Cooperative Agreements (CA) to Evaluate Consumer-Operated Service Programs (COS) and to provide technical and logistical support to all stakeholders, task forces (TF), committees and federal staff associated with this effort. The purpose of the CA is to determine selected consumer outcomes (empowerment, housing, employment, social inclusion, and satisfaction with services) and cost effectiveness of COS, to promote partnerships among consumers and mental health professionals, and to disseminate the knowledge gained through the project. The CC will also report on how systems can be tailored to meet the needs of consumers. The guiding mission of the CC is to generate new knowledge by assisting the study sites in accomplishing their evaluation projects, as well as minimizing data gathering burden experienced as a result of participating in the common protocol (CP) effort. The CC will facilitate the construction of the CP, provide training to sites in data gathering and assessment, establish a communication system with sites through web-based technology and teleconferencing, establish a data repository, and conduct data analysis.

Background. In the past twenty years self-help groups have become an important way of helping people cope with various life crises (Gartner & Riessman, 1982). While the range of self-help groups is broad, Durman (1976) found that most self-help groups originated in response to the need for human interaction; quick availability in crisis, at all hours, for potentially long periods of time; and for a focus that did not seek to make basic changes in outlook or personality, but to sustain the ability of members to cope with difficult situations. Historically, the mental health system developed with little or no input from people who received the services (Tanzman, 1990). Consumer-operated services (COS) emerged in the 1990s as an alternative to traditional mental health services. While many argued that traditional programs met the needs of their service recipients, others contended that the revolving-door pattern of care was a sign of system distress (Geller, 1992). The mental health system’s ability to meet the housing and social support needs of consumers was questioned (Carling, 1990; Zinman, 1987). Ridgway’s (1988) review of research on attitudes of consumers and professionals on service delivery pointed to a disparity between professional and consumer perceptions and suggested that these divergent views compromised the ability of professionals to meet consumer needs. On the other hand, Berman and Norton (1985) found that non-professionals working with a broad array of people, including people with mental illness, were as effective as professionals. Tan, Mowbray and Foster (1990) described the goals of mental health consumer-run programs as: (1) providing a safe, supportive and normalizing community environment, (2) providing an atmosphere of acceptance, (3) helping consumers feel needed and helping to promote their self-worth, dignity and respect, and (4) increasing knowledge about the community by learning from one another. However, there are few outcome studies which assess the efficacy of COS.

Program Goals and Key Questions. Mental health services are strained by more diverse demands for services at a time of limited fiscal resources. If self-help services are found to be effective, inexpensive, highly responsive, and accessible to a culturally diverse population of consumers, then the use of COS would be a promising a way to convert problems or needs into resources. In other words, COS can expand the mental health system quantitatively (by reaching more people), and qualitatively (by helping people become both more independent and interdependent) (Gartner & Reissman, 1982). To survive in an era of managed care, consumer-run programs need to examine the cost, effectiveness, quality, utilization and appropriateness of services they provide. They must also develop infrastructure to collect and process data without destroying the fundamental strengths of self-help services. But a survey of data needs of COS found that almost 40% of programs surveyed indicate that data collection from members would discourage people from using services, and 43% felt that data collection would burden the organization (Campbell, 1997). These concerns raise many questions: Is it possible to conduct services research on programs that embrace principles of voluntariness and respect for the desire for anonymity of some participants? Can the self-help experience be parsed into units of service to be measured scientifically? Are there differences between consumers using consumer-run services and those using traditional mental health services? What patterns of help-seeking and utilization of consumer-run programs are found among people with mental illness?

The SAMHSA CA (SM 98-004) to evaluate COS programs and conduct cross-site analysis on outcomes and costs for persons receiving traditional services and consumer-run services is an important step towards finding out "what works, for whom, at what cost, and with what outcomes?" (Leginski et al., 1989; Campbell & Frey, 1993). It supports the partnership of all stakeholders in the mental health system in a consensus building process of knowledge development and dissemination. Drawing upon the knowledge of consumer providers, the behavioral health care system can take advantage of a singular opportunity to rethink the current array of services available to people with serious mental illness, to be more effective and humane, and to support consumer goals of recovery, empowerment, and quality of life.

Challenges for Achieving Consensus. There are many challenges, however, to such a collaborative endeavor. In a review of consumer research, Scott (1993) warns the field of a "turf war over controlling human beings in a landscape that includes an entire array of service options and widely divergent goals and definitions of mental health and quality of life." When defining outcomes, providers tend to look at the way they affect the system, while consumers ask "what does it do for our lives?" What may appear as positive from one perspective may be negatively viewed from the other. Rather than using the language of recovery and self-determination, most professionals talk in clinical terms about symptom reduction, recidivism, and treatment compliance. There is also a concern in the self-help movement about the nature and extent of professionals’ involvement in the functioning of consumer-run groups. Independence from professional intervention has been part of the consumer self-help philosophy from its beginnings (Zinman, 1987). Still, controversy over professional involvement should not obscure the fact that professionals have often been involved with self-help activities. Conversely, many consumers do not hold the view that involvement with the traditional mental health system corrupts the self-help experience. In fact, many professionals and consumer providers agree that there can be a reciprocal relationship between self-help and professional practice. Adopting the slogan "nothing about me, without me," mental health consumers have moved rapidly to be involved in the design and implementation of quality management in psychiatric systems of care. The growth and acceptance of such partnerships shows the potential for progress when consumers and professionals work together in an atmosphere of mutuality and respect (Campbell et al., 1993; Ralph, 1994; Campbell & Johnson, 1995; Campbell, 1997; Ralph, 1997).

C. Wright Mills (1959) recommended that we use sociological imagination to "grasp what is going on in the world, and to understand what is happening in [ourselves] as minute points of the intersections of biography and history within society"(p 7). Now is a rare moment--a clearing horizon of historic opportunity for mental health consumers, traditional and self-help service providers, and researchers to enter into dialogue about the values, goals, and outcomes of COS. The SAMHSA initiative to evaluate COS over the next four years offers those people involved to look within, and to "re-search"--in a literal sense--the terrain of a priori assumptions in order to produce valuable new knowledge and practice. The introduction of multi-stakeholder partnerships in this endeavor offers much promise. It will often be necessary to pause, to support and encourage critical discourse, and to incubate new relationships and ideas. There must also be rigor in the scientific methods and practices, perhaps more attention to the demands of scientific objectivity because the weight of disbelief will demand that evaluators push harder for clarity in the evaluation designs and data quality. This initiative will test the assertion that data can enhance consumer choice, power and knowledge.

Review of the Literature. The Rise of Community-Based Peer Support: Approximately 7.5 million Americans participate in as many as one-half million self-help groups, addressing a range of illnesses, addictions, disabilities, and conditions (Lieberman & Snowden, 1994). Self-help is based on the principle that people with a shared condition can come together to help each other to cope, and the two-way interaction of giving and receiving help is therapeutic in itself. Former U.S. Surgeon General C. Everett Koop stated, "the benefits of mutual aid are experienced by millions of people who turn to others with similar problems to attempt to deal with their isolation, powerlessness, and alienation" (Katz, et al., 1992). A review of research studies that examined the effectiveness of self-help groups found that "numerous research studies show that participation in self-help groups can help people improve the quality of their lives significantly. Studies also show that support groups can reduce the need for medical care and hospitalization" (Medvene, 1986, p 6).

Mental health consumer self-help developed within the general self-help concept over twenty years ago during the period of deinstiutionalization of patients committed to psychiatric hospitals (Chamberlin, 1978; Susko, 1991). Lack of understanding, respect, and trust between consumers and professionals prompted people with psychiatric disabilities to seek the company of their peers for validation and support (Campbell & Schraiber, 1989; Campbell, 1992). The early consumer literature (Chamberlin, 1978; Zinman, 1987) defined other essential characteristics of the self-help program including self-definition of needs, equal power of members, mutual respect, voluntary participation by members, autonomy, and responsivity to other "special populations."

The self-help approach has long been an accepted part of treatment for medical conditions, but until recently, self-help as part of psychiatric treatment received little legitimization (Estroff, 1986). Further, most existing research captures no more than a glimpse of the meaning of self-help to recipients, since such support happens in places, times and ways neither public nor otherwise apparent (Unzicker, 1989). Yet, appreciation for the potential value of peer support, as a principal form of social support stimulated the Community Support Program (CSP) of NIMH to fund local COS Demonstration Projects from 1988-1991. A variety of COS were developed, staffed and evaluated (Furlong-Norman, 1988; Galanter, 1988; Kaufmann, Ward-Colasante, & Farmer, 1993; Heine et al, 1993; Lieberman, Gowdy, & Knutson, 1991; Mowbray & Tan, 1992; Nikkel, Smith & Edwards, 1992). These include drop-in centers, outreach programs, businesses, employment and housing programs, crisis services, among others (Long & Van Tosh, 1988; Penny, 1992). The consumer-generated research of the Well-Being Project (1986-1989) found that self-help groups for people with psychiatric disabilities are, in part, a search for community and mutual support in a world of actual and potential devaluation and hostility (Campbell & Schraiber, 1989). The reassurance that self-help groups provided helped reduce anxiety and restore hope, reinforcing the sense of belonging in a social setting (Breier & Strauss, 1984).

Consumers have gradually become recognized as valuable contributors to the mental health workforce (Furlong-Norman, 1991; NASMHPD, 1993; Stroul, 1986; Specht, 1988). Outside of self-help groups, peer support has also been provided by consumers working within traditional mental health programs in roles such as case-managers (Sherman & Porter, 1991) or peer counselors (McGill & Patterson, 1990). The demonstrated efficacy of peer case management and peer counseling in these programs supports the feasibility of a partnership between self-help groups and mental health professionals (Kaufmann, Freund & Wilson, 1989; Kopolow, 1981). Proponents of self-help groups suggest that such organizations also provide a valuable source of support for people with psychiatric disabilities (Salem, Seidman, & Rappaport, 1988).

A recent study of self-help programs has defined their characteristics and membership (Chamberlin et al., 1996). This survey suggests the feasibility of the proposed evaluation, and results will be helpful in the design, validation and analysis of the CP. The survey indicates that self-help serves a population that is largely white with over a third African American, middle-aged, single or divorced, and male with a significant minority of women. Most have a high school diploma, are unemployed, have monthly incomes of less than $600, and live in private homes or apartments. Members of self-help groups appear to make notable use of the mental health system, suggesting that people use self-help in concert with traditional services rather than as an alternative. Members use about 7 mental health services in the past year and about ½ are currently taking psychiatric medications. They participate in self-help an average of over 15 hours a week, and have been involved in self-help almost 5 years. In terms of outcomes, most appear satisfied with their housing, social situation, and physical well-being, but dissatisfied with their work and finances. They report positive effects of self-help on their general life satisfaction, and over 90% feel more positive about themselves as a result of self-help involvement.

The growing tide of health consumerism in the 1990s is rooted in malpractice litigation and consumer rights protection, on the one hand, and total quality management with its focus on customer satisfaction, on the other hand (Campbell & Johnson, 1995). Based on the assumption that principles of good healthcare must reflect consumer understandings, values, and desires, consumerism has compelled the healthcare field to grapple with the need to be customer-driven as opposed to the traditional view the "the expert knows best." In the field of mental health, consumerism also holds that peer providers will translate consumer perspectives into more effective service systems (Blanch, 1993). Considering the rising cost of mental health care and shrinking social resources, the outcomes of community-based peer support programs and the relationship between help-seeking, coping patterns, service utilization and recovery are much needed areas of investigation. Within the current radical restructuring of mental health services delivery systems and the rise of managed care, it is critical that the role COS support systems play in the rehabilitation of persons with psychiatric disorders be rigorously defined and the effectiveness of peer support models of psychosocial rehabilitation be established. Yet, without the development and use of valid, reliable outcomes assessment tools and protocols that are both appropriate and flexible for evaluation of the wide variety of peer support programs, it is unlikely that these programs will be included in managed care networks. Further, the lack of evaluation information on the outcomes and cost effectiveness of COS disadvantages them in gaining support through competitive resource applications. The availability of an outcomes protocol that programs could use to evaluate their services would enable such programs to systematically improve the quality, access, cost, and programmatic elements of their services.

Consumers as Research Partners. Advocates of peer support services believe that consumers have an insider’s understanding of the expectations that recipients have of services and the ways in which traditional providers meet or fail to meet these expectations (Campbell, 1996). For these reasons the last decade has witnessed the blossoming of a vibrant consumer research agenda and the growing belief that consumer involvement in evaluation holds great promise for both system reform and continuous quality improvement (Campbell, Ralph & Glover, 1993; Campbell, 1997). Interest and support has grown for investigating service priorities and consumers’ desired outcomes of services (Trochim, Dumont & Campbell, 1993) and support for development of consumer research protocols to study peer support programs (Campbell, Ralph & Glover, 1993; Scott, 1993; Campbell, 1997; Leff & Campbell et al., 1997). Using collaborative models of research, progress is being made developing outcome measures that reflect the perspective of recipients of both traditional and peer support services (The Accreditation Council on Services for People with Disabilities, 1993; CMHS releases consumer report card for field testing, 1996).

Outcomes Research on Community Based Peer Support. Since the beginnings of the self-help movement, ex-patients and consumers have asserted that help is best received when the help receiver has direct control over the help and there is reciprocity between help givers and receivers (Chamberlin, 1978; Zinman, 1982). In a series of focus groups supported by CMHS, the Consumer/Survivor Mental Health Research and Policy Work Group began a systematic articulation and exploration of consumer values and desired outcomes of mental health services and supports. Recovery, personhood, well-being and liberty were identified as valued outcomes that are seldom measured or operationalized in traditional mental health research or program evaluation (Consumer/Survivor MH Research & Policy Work Group Task Force, 1992). The Well-Being Project (Campbell & Schraiber, 1989; Campbell, 1992) substantially contributed to understanding quality of life from the perspective of consumers. The study relied entirely on consumers from its inception to dissemination of its findings. The Well-Being Project found that essential to well-being were: validation of personhood, recognition of common humanity, and tolerance for individual differences. The study showed that self-help programs fostered this validation. One the other hand, if consumers were denied voice, validation, information, or were treated with disrespect by people providing mental health services it impacted negatively their perception of well-being. The role of recovery and empowerment in self-help are other important outcomes that is currently being defined and measured by mental health consumer researchers (Campbell, 1997; Chamberlin, 1997; Dumont, 1995; Ralph & Campbell, 1995). Until recently, severe mental illness was considered to be permanent, requiring life-long treatment. Contradicting this assumption, recovery was first introduced in the writings of consumers (Deegan, 1996; Fisher, 1994; Leete, 1988). Previously, researchers documented the importance of social relationships and self-control over psychiatric symptoms (Breier & Strauss, 1984) and demonstrated the effectiveness of personal coping strategies (Cohen & Berk, 1985). A series of dialogues created a forum for psychiatrists and consumers to exchange perspectives, develop a shared vision of recovery and consider ways in which treatment relationships could be more collaborative. Participants indicated that recovery is an active, ongoing, and individual process related not only to the experience of symptoms, but to secondary assaults of stigma and abuse. Hope was seen as a fundamental factor in recovery; a sense of control or free will was also critical (Blanch, et al., 1993). Some research attention has focused on the outcomes associated with self-help groups involving consumers/survivors. Carpinello’s 1990 review of self-help groups reports that Galanter (1988), found a decline in both symptoms and concomitant psychiatric treatment as a result of group membership. Other research helps explain how positive changes in perceptions of self, social functioning, decision making, and symptomatology related to participation in primary mental health self-help groups (Carpinello, Knight, & Jatulis, 1992; Carppinello & Knight, 1993). Recovery is discussed further in the instrumention section of this application.

We will utilize the model proposed adapted from Leff, Campbell, et al. (1997) to drive our understanding of critical issues in consumer operated services and to study the impact of COS on social inclusion, empowerment, housing, employment, and satisfaction (See figure below.). Before turning to our plan for developing a CC to evaluate COS, we must acknowledge some of the difficulties and opportunities in measuring costs, conducting multi-site studies, and effectively addressing issues of cultural competency.

Direct and Indirect Costs. One project goal involves determining the cost of using COS programs. Computing the cost of these services from a societal perspective is more complicated than simply computing the cost of the program itself. If the use of consumer-operated service programs affects other service use, then changes in the cost of these services must also be included. Therefore, the objective of the cost study is to compare the sum of all costs for consumers who use COS in conjunction with traditional services to the sum of all costs for comparable consumers who use traditional services. However, costs alone are not the sole outcome for the proposed study. We will combine the cost and outcomes studies to determine whether COS are cost-effective. For example, if COS improve outcomes but cost no more than traditional services, then these services are cost- effective. We will be interested in determining which COS models are more cost effective. Traditionally, service costs are divided into direct and indirect costs. Direct costs are costs directly associated with providing the service. For this study, direct costs will include all medical and service costs, including inpatient hospital, outpatient physician, crisis intervention, and emergency room costs. We expect that consumers who use COS will increase some direct costs and decrease others. The difference in costs, or incremental cost, is more important for policy than the absolute level of cost. Direct costs will be measured using claims and survey data collected by sites. Indirect costs are all non-direct costs, including costs of housing, criminal justice, vocational rehabilitation, and income support. Ignoring indirect costs would lead to development of an incomplete picture of costs for those who use consumer-operated service programs, and could bias the results of the cost study, potentially leading to misguided policy recommendations. As for direct costs, differences in indirect cost between consumers who use COS and traditional services and consumers who utilize only traditional services are of greatest interest. Indirect costs will be measured from survey data collected by the sites.

Rationale for Multi-site Study. Evaluations of a single site are limited in the number of different program models, individual characteristics, and environmental variables they can address. Policy makers are, in contrast, typically interested in the widest array of dimensions programs may fall into. The policy space (Cordray & Fischer, 1994) defined by these options refers to the combinations of program models, target populations, environmental contexts, and desired outcomes about which policy makers and service providers desire information. To evaluate the effectiveness of COS, we are interested in learning about how different models work for individuals with different demographic and behavioral characteristics, who are served in programs employing different program models, and who may have taken different paths to receiving these services. Because of the probable reliance of single site studies on a single type of program, single site studies will be unable to address these questions. In addition to this limitation in scope, single site studies also face limitations in sample size, since they may provide services to only relatively small numbers of persons. It is important to recognize that most mental health programs have only small to moderate effects on outcomes (Lipsey & Wilson, 1993). The smaller the true effect size associated with receiving program services, the larger the sample size required to detect the effect. Small numbers of service utilizers make it even more difficult to examine the effects of programs on subgroups such as women, ethnic and racial minorities, or residents of rural communities. Multi-site studies which employ the same design and instrumentation may (under some conditions) be able to pool their data and overcome this limitation. A multi-site evaluation employing a CP can: (1) provide information on a wider variety of program types; (2) aggregate findings from similar programs to more effectively detect smaller program effects; (3) explore the effects of similar programs on different types of individuals who use services because multiple sites can include different types of service recipients and because combining data across sites permits more subgroup analysis; (4) examine the effects of similar programs delivered in different environments. These advantages can be realized, however, only to the extent that sites employ comparable instrumentation in measuring client characteristics and outcomes, and that there is sufficient consistency in other aspects of study design to assure that variability across sites is the result of real differences, rather than variations in the implementation of the study.

Knowledge Regarding Cultural Competence. Consumers from diverse ethnic and cultural backgrounds demand that both research and service programs proactively seek out ways to adopt a culturally competent orientation towards mental health outcomes (Rogler et al., 1987; Neighbors, et al., 1992; Snowden, 1996). Mental health programs, including those demonstrated to be effective when delivered according to standard protocols, often have to be reformed to be sensitive to sociocultural issues. Certain culturally-related characteristics are pertinent: (1) expression of symptoms in terms of indigenous systems of belief (Landrine & Klonoff, 1994); (2) willingness to use alternative providers, including religious and folk healers (Snow, 1993); somatic complaints to express mental health suffering (Escobar et al., 1987); (3) stigmatizing perceptions of mental health problems (Silva de Crane & Spielberger, 1981; Pape et al., 1983); (4) interpersonal sensitivity and specialized norms regarding respect, trust, and authority (Uba, 1994); (5) common sociocultural backgrounds shape a common outlook (Cheung & Snowden, 1990); and (6) culturally distinctive patterns of family burden, responsibility, and satisfaction (Uba, 1994). Claims of cultural sensitivity in service delivery are difficult to evaluate, and often amount to little more than encouragement of ethnic and language matching. When Lopez and Hernandez (1986) reviewed reports of cultural accommodations to clinical practice, it was impossible to tell whether the changes contributed to consumer improvement. While rigorous research could provide clear empirical documentation of ethnic differences in utilization, mis-diagnosis, or poor outcomes, the quality of research on outcomes permits no strong conclusions about the effectiveness of mental health service delivery programs (Neighbors et al., 1992). Snowden (1996) reviewed efficacy studies and found relatively weak effects in programs that evaluated the impact of (1) standard interventions implemented to incorporate ethnocultural elements (Torres-Matrullo, 1982; Comas-Diaz, 1981), (2) ethnic matching and language matching of provider and consumer (Flaskerud & Liu, 1990; Ying & Hu, 1994); and (3) enrollment of ethnic minority consumers in minority oriented service programs (Snowden & Clancy, 1990). The lack of culturally competent instruments to measure the outcomes of mental health services presents a problem. Differences among diverse populations in language, beliefs, and experience may produce inappropriate or misunderstood questions (Hines, 1993). Therefore, some have called for separate development of instruments among culturally diverse groups, while others propose methods for developing instruments with cross-cultural validity or equivalence (Flaherty et al., 1988). Neighbors et al. (1992) offer recommendations for improving mental health service delivery to ethnic minorities that could be applied to this CA including: development of training modules to make non-minority providers aware of the cultural values and practices of ethnic minority groups, and obtaining consultant expertise in these areas.