Section 4: Research Questions and Hypotheses

4.1 Logic Model

The multisite logic model is depicted graphically in Exhibit 4-1. The logic model postulates a range of individual-level baseline factors that characterize enrollees within services, whether traditional or consumer operated. In addition to demographic characteristics, these factors include previous experience with traditional and/or consumer-operated services and baseline indicators of outcomes, such as satisfaction with services, empowerment, physical health, psychiatric symptoms and level of distress therewith, substance use, employment, housing, well-being, social inclusion, and quality of life.

In addition to these factors, three broad categories of community characteristics define the macro-level context in which services are provided and received. The first consists of regional service network characteristics, including: availability of traditional mental health, substance abuse, and consumer-operated services; service system integration; and the mechanisms by which services are paid, e.g., managed care vs. fee-for-service arrangements. The second subsumes socioeconomic indicators of community context including poverty, unemployment, housing costs, and the legal context of mental health services delivery, including authorization for involuntary commitment. The third describes resources and costs related to service provision, such as time, space, equipment, and overhead/administration.

At the level of service provision, the model identifies a set of parameters that is hypothesized to characterize traditional services and a separate, but to some extent overlapping, set of procedures that is hypothesized to characterize consumer-operated services. Traditional services are described in terms of:

(a) Program characteristics (governance, composition of personnel, and organizational structure/hierarchy);

(b) Services provided and received; and

(c) A set of common processes that includes: linkages with other mental health services; continuity of care; professional support; planning, coordination, and monitoring of services; options for commitment, coercion, and control; and a priority placed on safety for program participants.

Consumer-operated services are described in terms of:

(a) Program characteristics (governance, peer vs. mentor approaches, paid vs. voluntary service provision, and organizational hierarchy/structure);

(b) Program models (drop-in center, peer support, or education/advocacy); and

(c) A set of common processes that includes: peer relationships, consumer-run administration, a sense of community, an atmosphere of safety, advocacy, freedom of choice, and quality of consumer participation. A comprehensive list of these indicators is found under the discussion of common ingredients in Section 3.

Based on eligibility criteria set forth in the GFA and elaborated elsewhere in this document, participants are recruited at each site and randomly assigned to receive either TRADITIONAL + CONSUMER-OPERATED SERVICES, or TRADITIONAL SERVICES only. These conditions will be abbreviated as TMHS + COS and TMHS. We hypothesize that specific program characteristics of traditional and consumer-operated act independently, as well as synergistically, to produce intermediate and long-term consumer and cost outcomes. Specific study outcomes are described in greater detail below.

4.2 Multisite Study Research Questions and Hypotheses

The primary goal of the COSP Multisite Research Initiative is to establish the extent to which the addition of consumer-operated services (COS) to traditional mental health services (TMHS) changes selected consumer and cost outcomes for mental health consumers.

4.2.1 Research Question #1: To what extent does participation in a COS program affect selected consumer outcomes for consumers who use traditional service programs?

The selected outcomes specified in the GFA are empowerment, housing, employment, and satisfaction with services. During the first year of them initiative, this list was elaborated in further detail by the Steering Committee, the Research Subcommittee, and Working Groups. Outcomes were specified as being "intermediate" or "long term;" all intermediate outcomes are also considered as long-term outcomes, though not all long-term outcomes are included as intermediate outcomes.

Among the elaborations to the list of outcomes was the specification of three subdomains of empowerment:

Making Decisions (Rogers et al. 1996), subsumes self-efficacy, self-esteem, achievement, and control in life;

Personal Empowerment (Segal, Silverman, and Temkin, 1995) includes the extent to which respondents perceive choice in living situation, how they spend their money, leisure activities, and help seeking behavior, as well as their view of the likelihood that they will have sufficient resources to meet basic needs and wants; and

Organizationally Mediated Empowerment (Segal, Silverman, and Temkin, 1995), subsumes leadership roles in, and influence over, the political processes of organizations.

The GFA-specified domain of consumer satisfaction with services was expanded to include consumer-reported outcomes of services (MHSIP, 1996). These outcomes are defined separately for traditional and consumer-operated programs. New multisite consumer outcomes, subsumed under the broad general heading of "well-being" were added as well. These include hope, self-esteem, subjective quality of life, social inclusion, social acceptance, reduced distress with symptoms, improved physical health, and reduced substance use/abuse. Another major addition in the domain of well-being concerned status of personal recovery.

Hope (Herth, 1992), understood to be important in adapting to illness and promoting wellness, is a key psychosocial outcome for consumers, particularly given the historical presumption that severe mental illness carries a hopeless and dismal prognosis. The addition of social inclusion, defined as active participation in the life of one’s social network and community, reflects the notion that consumer-operated services are particularly successful in increasing participants’ abilities to integrate themselves into their chosen communities. In part, this reflects reduction of stigma as well as increased community-oriented empowerment.

Distress with symptoms subsumes both the occurrence of symptoms and their subjective effects on consumers. Characterizing symptomatology both at baseline and as a study outcome is a necessary aspect of describing the participants in the study. The importance of careful description includes the assessment of consistency of the clinical characteristics of the site populations with GFA guidelines as well as the ability to take appropriate account of these characteristics in analyses and interpretations of study data. Moreover, while a major objective of traditional mental health treatment is symptom reduction, a major tenet of the consumer movement is recovery (see below). Peer counseling and support may be helpful in reducing symptomatology to manageable levels, perhaps reducing the need for aggressive clinical intervention.

Improved physical health as a study outcome reflects the substantial untreated medical morbidity experienced by many mental health consumers. The extent to which this morbidity reflects underlying biological processes also associated with their mental illnesses, or long-term side effects of medication, is unclear. However, consumers’ unmet need for medical care, given considerable medical morbidity, may also reflect the combined effects of poverty and its constraints on access, refusal of general medical providers to take mental health consumers’ physical complaints seriously (Graber et al., 2000), and consumers’ difficulty in advocating for themselves owing to the multiple vulnerabilities associated with co-occurring physical and mental illness. Thus, improved physical health could reflect (a) improved access to (b) clinically appropriate care, as well as (c) improved ability in self-advocacy and (d) increased capacity for self-care. Obviously, to the extent that improved self-advocacy is associated with improved health, empowerment and improved physical health may be interrelated. To the extent that this outcome reflects increased access to and utilization of clinically appropriate care, and perhaps improved capacity for self-advocacy, it could also translate into increases, at least over the short term, in health service costs. However, it also carries the potential for long-term cost savings.

Reduced substance use/abuse is clinically important in its own right as an indicator of clinical improvement for consumers with substance use disorders. However, it is also a critical outcome because the co-occurrence of mental illness and substance use, whether or not the substance involvement meets criteria for an addictive disorder, carries adverse prognostic implications with respect to the mental illness (Drake and Brunette, 1998; Drake et al., 1996).

Recovery is a concept understood in varying ways across programs participating in the COSP Multisite Research Initiative. Instead of focusing primarily on symptom relief, as the medical model dictates, recovery casts a wider spotlight on restoration of self esteem and identity, and on attaining meaningful roles in society (Surgeon General’s Report on Mental Health, 1999). Personal recovery is a central value within the consumer movement, viewed as possible for all consumers irrespective of the nature or severity of their mental illnesses. Further, recovery from the social, economic, political, and legal consequences of mental illness may be more important than recovery from the illness itself. Recovery may or may not mean complete, permanent elimination of symptoms or usage of psychiatric medications; however, increasing recovery is marked by increasing ability to cope with symptoms, as well as increasing sense of purpose in life and personal assumption of responsibility and control over the future (e.g., Corrigan et al., 1999). It is a highly individualized and often nonlinear process which can involve both progress and setbacks. Recovery is fostered both by consumers’ own individual efforts and by interactions with peers.

4.2.2 Research Question #2: To what extent does participation in a COS program affect costs for the following: inpatient hospitalization, crisis intervention, and emergency room utilization, as well as offsetting costs in housing, criminal justice, vocational rehabilitation, physical health care, and income support?

To what extent does participation in a COS program affect costs for the following: inpatient hospitalization, crisis intervention, and emergency room utilization, as well as offsetting costs in housing, criminal justice, vocational rehabilitation, physical health care, and income support?

The basic questions of this study are whether addition of COS significantly increases the effectiveness and benefits of TMHS, and at a cost that is exceeded by the benefits and seems reasonable. These questions can be answered by traditional analyses of variance and calculations of traditional cost-effectiveness and cost-benefit ratios.

In addition to assignment of consumers to either TMHS only or TMHS+COS conditions, the extent to which each consumer in both conditions participates in TMHS and COS is measured. At a more detailed level, the extent to which each consumer participates in program activities, some of which are common to TMHS and COS and others of which may be unique to one or the other program type, is measured. Measuring degree of participation provides a more fine-grained approach than relying solely on random assignment to two conditions.

Experience of a variety of researchers with a variety of programs and populations has shown that not all persons assigned to conditions participate equally in the programs to which they have been assigned. By measuring the degree to which each consumer participates in a variety of program activities, we may be able to discern which of the many activities contribute most to positive outcomes. By also measuring or estimating the costs of those activities, we may determine not only which activities contribute most to positive outcomes, but which also add minimally to total costs or at least which are more affordable.

If sufficiently detailed and valid data on the relationship between procedures and outcomes relationships can be collected, this information can be combined with information on resource -> procedure relationships, the combination of program activities that maximizes outcomes within budget constraints on each type of resource can be determined using linear programming procedures (Yates, 1980, 1996, 1997, 1999).

The specific outcomes chosen for measurement may have profound effects on acceptance of our findings in the consumer and mental health services communities, and by researchers, policy makers, and funders. A variety of outcomes have been chosen, including possible savings in health care, mental health, and criminal justice costs subsequent to participation in TMHS and TMHS + COS. A considerable body of research has shown that participating in TMHS often is followed by reductions in health care and other expenses (e.g., Goodman et al., 2000; Olfson et al., 1999; Simon and Katzelnick, 1997). Analyses of this research have concluded, with varying degrees of certainly, that this relationship between TMHS participation and subsequent service cost reduction is causal and not in response to a third, common factor.

4.2.2.1 Overview of Cost-Procedure-Process-Outcome Analysis (CPPOA) Model

The basic question asked in the planned analyses of cost, cost-effectiveness, and cost-benefit is:

What combination of human and material resources was used to implement which program procedures that caused changes in the biopsychosocial processes that were responsible for the observed outcomes of improved behavior, cognition, affect, and physiology and of reduced use of health care and social services?

The CPPOA model from which these hypotheses and analysis plans are based (Yates, 1980, 1996, 1999) posits that it is useful for both general research purposes and more specific program management decisions to measure the types and amounts of (a) resources, (b) procedures, (c) processes, and (d) outcomes for a human service system. The CPPOA model further posits that to answer the above question, and to use that answer to optimize program performance, relationships between resources, procedures, processes, and outcomes need to be described quantitatively in terms of direction, strength, and deviation from relationships hypothesized by program directors and consumers. Part of this description of relationships involves tests for statistical significance of relationships across a variety of clients, and tests of goodness-of-fit to hypothesized relationships.

To date, efforts have focused on defining and measuring the resources, procedures, processes, and outcomes of COS and, to a lesser degree, TMHS programs. As noted elsewhere in this report, some programs have chosen to measure costs of the program overall per client, and have not itemized the amounts of different types of resources used for different procedures. This will prevent a full CPPOA for all programs. Below we describe both measurement and analysis plans for all programs, and additional plans for programs that are itemizing the amounts of different resources consumed for each procedure.

Resources are the personnel skills and time, the facilities, the equipment and materials, the telecommunications services, the transportation, and all the other things (e.g., financial, security, accounting services) that make the program possible. Resources come in many types; in human services the most expensive generally are personnel time followed by facilities. The value of the resources used in a program is its cost. CPPOA measures and retains information on the amount of each type of resources used in the program, for each participant (i.e., for each consumer). The value of these resources, in terms of local currency, is its cost. "Cost," though, is simply quantity * price. CPPOA attempts to go beyond measuring cost, to understanding in a quantitative and specific manner the mixture of resources that is used by the program to provide an amalgam of services to individual consumers.

Procedures are the specific activities in which consumers are encouraged to engage, and which are supposed to provide benefit. Procedures range from dropping in to a center to sitting in a classroom and listening to lectures and stories. Procedures vary widely between programs. Procedures are observable acts involving consumers and a person (e.g., peer advisor) or piece of equipment (e.g., computer) or materials (e.g., handout, book, manual, newsletter) that are designed to help the consumer achieve program objectives.

Processes are the internal events and ongoing states that, while they cannot be observed directly by anyone other than the consumer, are the links between the activities (procedures) that are supposed to help the consumer and the changes in the consumer's behavior, cognition, affect, or physiology that are the goals of the program (i.e., outcomes, e.g., more pleasant, constructive conversations with family members and fewer acts of aggression, more positive and focused thoughts, reduced anxiety and depression, and a more balanced immune system). For some consumers and for some third parties, changes in these processes are the end-product of the program. The distinction between what is a process and what is an outcome is, to a degree, a matter of the perspective of the interest group with which one is associated.

Outcomes are the results in which interest groups—particularly those supplying the majority of resources that make program procedures possible—are most interested. Achievement of the outcomes often justifies, in the eyes of the contributing interest groups, the use of the resources in this as opposed to alternative endeavors. Outcomes are observable acts or states of the consumer, and can include employment and income produced by employment, (reduced) substance abuse, acts toward others (ranging from aggression to volunteering). A special type of output—reduced use of the services of other programs—can be especially useful in assessing the potential cost-savings benefits of a program. These and other cost savings may more than off-set the value of the resources used by the program (hence the use of "cost-offset").

4.2.2.2 Approaches to CPPOA

 

Simultaneous CPPOA. In addition to describing resources, procedures, processes, and outcomes at various levels of specificity (e.g., per consumer per month for each program, per consumer for their entire period of participation, per month for all consumers, for their program overall), a variety of specific statistical tests can be conducted in the CPPOA model to answer questions of interest to researchers, consumer advocates, policy makers, and funders. In general, the guiding assumption is that outcomes are functions of a complex interaction of processes, procedures, resources, consumer variables, and community variables, i.e.,

O = f (p, d, r, c, m)

 

where O denotes outcome, p = processes, d = procedures, r = resources, c denotes a set of consumer variables, and m denotes a set of community variables. Typically, describing this interaction with precision, and testing its fit to the data, is exceedingly difficult with the relatively small samples sizes available in most social science research (including, possibly, the same size available in this study). For example, path analyses can be conducted only when data for several hundred to several thousand consumers are available, depending on the number of variables included and the number of relationships hypothesized between variables.

Sequential CPPOA. An alternative approach to analyzing data on resources, procedures, processes, outcomes, and consumer and community variables is to test relationships between these elements sequentially, in three sets of analyses. The sequence of independent variable -> dependent variable analyses are a) processes -> outcomes, b) procedures -> processes, and c) resources -> procedures. For example, first relationships between outcomes of interest and the various observed processes can be tested for significance using analyses that include main effects and specified interactions of multiple independent variables. Those processes which were found to be significantly related to outcomes would be included in the next round of analyses. Those processes that were significantly related to outcomes would be treated as the dependent variables in analyses that included procedures as the independent variables that might be related to those processes. Finally, relationships would be described (if not tested statistically) between the various resources and the procedures found to be significantly related to processes (and, hence, to outcomes).

Consumer and community variables can be entered into determination of relationships between each pairing of causally adjacent variables, as either additional main effects and, if suggested by theory and if possible given sample sizes, as elements of interactions. For example, each outcome can be described (and tested for accuracy of description) as being a function of processes, client variables, community variables, and a composite of the interactions of process, client, and community. Similarly, each process may in turn be a function of procedures and client and community variables separately and in interaction with procedure variables:

P = f (d, c, m)

Finally, each procedure in turn may be defined as a function of resources, consumer variables, and community variables:

D = f (r, c, m)

The relationship between resources and procedures seems, at first consideration, to be one that can only be described, not tested (statistically or otherwise). Upon reflection, however, an argument can be made for the possibility and even desirability of testing resource -> procedure relationships. Whether a procedure such as cognitive behavior therapy can only be conducted in an office, and by people who have a certain level of training and certification, is an important question and a testable one. From some perspectives, a primary question of the present study examines whether therapeutic procedures can be performed successfully and with positive impact by persons other than those who typically provide such procedures. (An alternative perspective on this is that entirely different procedures are being used in COS—ones that traditionally trained, educated, and certified professionals could find particularly difficult to carry out.)

4.3 Hypotheses

4.3.1 Short-term/Intermediate Consumer Outcomes

With baseline levels of study outcomes and other potentially confounding respondent characteristics taken into account, compared to consumers receiving only traditional mental health services, consumers receiving both traditional and consumer-operated services will:

(a) show more rapid increases in empowerment and residential stability, more desirable housing situations, more rapid improvement in employment status, and higher levels of satisfaction with services and their outcomes;

(b) demonstrate more rapid improvement in hope, self-esteem, subjective quality of life, social inclusion, and social acceptance; and

(c) experience more rapid reductions in distress with symptoms, improvements physical health, decreases in substance use/abuse, and progression toward recovery.

Alternatively, participants in the two randomization conditions may show differential patterns of improvement with respect to specific outcomes. For example, if TMHS providers operate from an underlying philosophy that treats consumers as patients with an illness to be managed, TMHS alone may have no substantial impact on variables such as empowerment. As such, consumers randomized to TMHS + COS would show greater increases in empowerment than those assigned to TMHS alone. Conversely, to the extent that COS do not specifically focus on symptom reduction, there may be no difference between the two conditions with respect to this outcome variable. These patterns of findings, and their implications, would roughly parallel the results and implications of studies of psychotropic medications and psychotherapy singly vs. combined for conditions such as depression: the case for combining the modalities rests on the extent to which each affects different domains of outcome. In interpreting study results, however, it is also prudent to bear in mind the possibility that improved outcomes in the TMHS + COS condition, relative to those observed with TMHS alone, could reflect a nonspecific benefit of doing more services. However, the design of the study does not allow differentiation of nonspecific benefits of access to more services (TMHS + COS) from specific effects of COS.

4.3.1 Long-Term Consumer Outcomes

With potentially confounding respondent characteristics taken into account, compared to consumers receiving only traditional mental health services, consumers receiving both traditional and consumer-operated services will demonstrate:

(a) more sustained gains, and greater continuation of improvement, in empowerment and residential stability employment status, and satisfaction with services and their outcomes;

(b) demonstrate more sustained and continuing improvement in hope, self-esteem, subjective quality of life, levels of social inclusion, and social acceptance; and

(c) more sustained and continuing reductions in distress with symptoms, improvements physical health, decreases in substance use/abuse, and progression toward recovery.

Alternatively, and as was stated with respect to short-term outcomes, participants in the two randomization conditions may show differential patterns of improvement, and of propensity to sustain and continue improvement over time, with respect to specific outcomes, owing to differences in underlying philosophy between TMHS providers and COS programs.

4.3.3 Research Hypotheses Related to Procedures and Costs

The most basic question in this study is how do costs and outcomes change when COS participation is added to TMHS participation? A more complex set of follow-up questions examines whether the change in outcomes worth the change in costs. Moderators of all these questions include: (1) type of COS (three groupings have been identified), (2) type of TMHS (groupings may emerge as TMHS are examined more closely), (3) degree of consumer participation in the TMHS, and (4) degree of consumer participation in the COS. These moderators can be applied to all of the following breakdowns of these questions in the framework provided by the CPPOA model.

4.3.3.1 Resource -> Procedure relationships

What changes in a) TMHS personnel, b) COS personnel, c) family and friends' time, d) consumer time, e) TMHS facility, f) COS facility, g) TMHS other direct, h) COS other direct, i) TMHS overhead and j) COS overhead costs occur for i) TMHS, ii) COS when consumers are referred for participate in COS as well as TMHS? Are these changes in cost statistically or clinically significant? Do any observed changes depend on the scale of the TMHS or COS, e.g., does a change become reduced or increased as the size of the TMHS or COS increases? Given the potentially large portion of COS costs that may be volunteered or donated, we will want to ask the questions in this section for 1) monetary expenditures only, 2) volunteered and donated resources only, and 3) the total cost of the program, estimated by adding (1) and (2).

For programs that measure resource -> procedure relationships in more detail, additional questions can be asked. In particular, differences in cost between the three groups of programs and between individual programs can be examined in terms of what resources are being used for what procedures. We may find that COS are particularly labor-intensive, relative to TMHS, but that most of their labor costs are volunteered and some of their facilities are donated -- more, by a far margin, that in TMHS. Drop-in programs may have higher "overhead" costs, while advocacy and other programs may be able to adjust their costs more precisely according to the number of consumers wishing to participate in the program. We may find some procedures that are common to most COS and few TMHS, and vice versa. We posit that similar mixtures of different types of resources will be used by programs to implement these procedures. To the degree that programs differ in the resources used to provide similar procedures, we may be able to find the least expensive way to provide a procedure.

4.3.3.2 Procedure -> Process relationships

Basically, we hypothesize that participation in COS will lead to more of the changes described as Common Ingredients elsewhere in this manuscript. We also will examine to which these common ingredient processes occur for consumers participating in TMHS. These relationships may be moderated by the types of clients and communities.

4.3.3.3 Process -> Outcome relationships

We hypothesize that more experience of the changes identified as common ingredients should be related to more positive changes in the major outcomes, including attainment of monetary cost-savings benefits. These predictions, too, should be moderated by consumer and community variables.

Missouri Institute of Mental Health5400 Arsenal StreetSt. Louis, Missouri 63139
Phone: 314-644-8787 Fax: 314-644-8834