As mentioned earlier, the proposed CC will be collaborative partnership between the Missouri Institute of Mental Health (MIMH), R.O.W. Sciences, Inc., the National Empowerment Center (NEC), and the National Mental Health Consumers’ Self-Help Clearinghouse. Much of the work of the CC will take place in and around meetings of the a Steering Committee (SC), which is scheduled to meet 6 times in Year 1, and quarterly in Years 2-4. While the SC will officially decide the major issues of the initiative, we propose that the major work be done through four Task Forces (TF) which focus on areas critical to the initiative: the Methods and Protocol TF; the Cost Analysis TF; the Consumer Research and Provider TF; and the Cultural Diversity and Competency TF. Each TF meeting will be facilitated by a senior staff member from the CC with appropriate expertise.

Establishing a Participatory Process. In response to the emphasis on outcomes measurement and appropriate indicators, this CC proposal will shift toward a multi-stakeholder model based on collaboration (Campbell, 1996; Campbell & Johnson, 1995). Sharing and reconciling divergent views enables people to make conceptual leaps in how they think about the production of mental health measurements and how they can use outcome information to understand and improve services. We will institute a consensus-building process that considers the range of stakeholders, what they value and what they want to know. Effective partnering will depend on the careful application of methods for prioritizing the multiple perspectives. Promoting multi-stakeholder participation has great power if it includes a diversity of stakeholders within an open architecture of knowledge production. As an alternative to polarizing debates, the MIMH CC will engage consumers and professionals in participatory methods for discussion and exchange of perspective. These methods will include the use of collaborative models such as Continuous Quality Improvement, Participatory Action Research (Chesler, 1991; Rogers & Palmer-Erbs, 1994; Leff et al., 1997), Constituency-Oriented Research and Dissemination (Fenton et al., 1993), and Pioneer Dialogues (Loder & Glover, 1992; Blanch et al., 1993).

The collaborative nature of this project means that communication between the study sites and CC will be vital. This demands that the CC think of communication in a new way: one that allows for a constant flow of information back and forth between the people working together on this project. To do NEC will establish a variety of computer-aided and hard copy communication modes to facilitate discussion, collaboratively solve problems, and provide "real time" TA. Vicki Wieselthier, located in St. Louis, will serve as Communications Director (CD), developing and administering the web-based telecommunications system. She will have technical support available on location by MIMH, and knowledge development and editorial support supplemented by NEC staff. Planned electronic communication services include:

CC Web-site: The web site will the CC’s public face--explaining the project and its value to the mental health system and providing a way to quickly disseminate knowledge developed to consumer groups and others working in related areas. The Web will have home pages for study sites as well as a literature review.

Mailing Lists: The CC will create a variety of Internet mailing lists for project participants. Researchers will be able to talk to other researchers, program administrators to other administrators, and TA staff to each other. Consumers in attendance at the study site programs will also have a list. Lists will be established by the CD who will monitor the lists, assist list members, and synthesize the material passing through the lists for used in planning TA. The CC intern will work with the CD to provide needed information. Consultants will participate in the list discussions when alerted to the need by the CD. These lists will be used to communicate study updates, deadlines, other project-related information, will play a large role in development of the CP, and will be archived.

Chat: Real time conferencing will be available through Web-site based chat. A " Java" chat room will be established on the project web page. Regularly scheduled chat sessions will be held and other chats established that are issue or role-based will be arranged as the need occurs. For example, evaluation staff at the various study sites may decide that they want to "talk" to each other in real time every two weeks. If a number of the sites are struggling with an issue, they may get together in the chat room to discuss it. On a monthly basis, the CD will review the chat logs and list archives to synthesize the information and report on the issues under discussion and TA needs that have been identified. This information will be used to generate TA manuals and the project newsletter. It will keep staff involved in the site dialogs informed and current so that scheduled visits to the sites are can be more effective. The CD will also serve as the "help desk" as participants develop skills they need to use Internet technology.

Newsletter: Available in electronic and hard copy, the twice-yearly newsletter will be used to spotlight project accomplishments and to disseminate the knowledge developed. It will provide an opportunity for study sites to report progress as well as provide a mechanism to disseminate basic knowledge on principles of protocol design and evaluation to interested persons and organizations.

Technical Assistance Manuals: TA is viewed as an ongoing, dynamic process. For this reason, the CC will be using technology-dependent assistance via chat rooms and the mailing lists. However, such methods do not take the place of a compendium of information on a particular topic. TA manuals produced as part of this project will bring together in one place, and supplement, the information provided "live" to the study sites. The manuals will become the consumer desk reference for the project. In addition to the knowledge collection, synthesis and dissemination modes discussed above, material and direction for the manuals will be gathered through dialogue sessions with study stakeholders led by Gayle Bluebird, a consumer facilitator, at each study site and a multi-site dialogue session conducted in Year 04. The CC anticipates developing three TA manuals over the life of the project. The on-going communication synthesis activities of the CD, CC interns, CC staff and consultants, and study site evaluation and service staff means that the CC will be able to produce TA manuals in a timely way. Therefore, study sites will have information they need, when they need it.

Plan for Developing a Common Data Set. The first goal of the process of developing a common data set will be to achieve early consensus by the SC on the necessary outcomes that must be collected by all sites to support the multi-site study, a Common Protocol (CP). Developing a CP will be primarily a function of the Methods and Protocol TF, in coordination with the other TFs. As TFs form, their meetings will be facilitated by a senior staff member from the CC with expertise in the TF focus. In the process of developing the CP, Drs. Campbell and Johnsen will take the lead in putting together materials for all SC members and TFs to review prior to the first CC Meeting. The package will include background material on the usefulness of multi-site evaluations and the benefits to be expected from developing a CP. CC staff will also review site hypotheses, interventions, and evaluation plans as stated in proposals. A display of these results will give each site a clear picture of the proposals of the other sites and will guide the development of CPs and multi-site data analysis plans. As the CP is developed and the variables in the protocol are used to operationalize important theoretical constructs, we will ask the sites to develop site-specific logic models about the operation of their consumer-operated program models. These models will permit checks on the adequacy of the developing protocol to measure key aspects of the programs and serve as representations of additional hypotheses that emerge from the collaborative protocol development process.

Consumer Outcome and Service Delivery Data. It is likely that the sites will propose many of the same instruments, but it is also likely that other instruments will be proposed by only some sites. Selection of instruments and items for the CP will be discussed in TF meetings beginning with the second CC meeting. CC study directors will facilitate discussions of the instruments that consider the following criteria: applicability to the target population; psychometric properties; simplicity of implementation; availability of training materials and trainers; respondent burden; cultural sensitivity; sensitivity to consumer concerns; and appropriateness of data source (consumers, providers, records). Our experience as participants in this process indicates that it is likely to result in selecting or adapting the most pertinent sections or subsets of items from some instruments rather than adopting them in their complete published forms. It is also likely that SC members will decide to modify individual questions or develop entirely new items.

Qualitative Data from Ethnographies, Focus Groups, Case Studies, etc. We also expect that sites will wish to collect various types of qualitative data on programs and consumer views of programs. In addition to the qualitative data collection conducted by the CC through a series of dialogues at study sites described elsewhere, we propose that the CC review the proposed qualitative data collection plans and the proposed links with the quantitative data. Some qualitative data collected across sites will be suitable for cross-site qualitative analysis and entry into the project-level data base discussed below.

Interviewer Training and Assessment of Instruments. Once a final draft has been approved, the draft protocol for the consumer outcome component of the CP will be prepared by CC staff with prototype instructions and manuals for administration. Pilot tests will be performed at the sites using current clients not eligible for program or at sites external to the projects. These field tests will help us identify and address any problems. If required, the psychometic properties of the CP will be analyzed. Standardized training of each site's interview trainers will coincide with CP distribution. Each site will send its PI/Research Director and one or more supervisory data gathering staff to receive a 2-day training held at a CC Meeting. We are particularly interested in strategies of data collection that use mental health consumers as surveyors. Although it will then be each site's responsibility to train the personnel who gather data, the longitudinal nature of this project may necessitate periodic "refresher" training as new researchers and other data collectors come into the study. It would be also be advisable to conduct periodic field tests to identify any measurement decay in interviewing procedures. When such problems are discovered, the CC will provide training to remediate them during yearly site visits and check to see whether they have been corrected with subsequent cross-site data analyses. One way to facilitate re-training is to videotape training sessions.

Involving Consumers Target Group. Because we are committed to full involvement of primary consumers in all phases of program evaluation and research, the CC will support their inclusion as TF members. The CC will use the consumer researchers from the MIMH Program in Consumer Studies and Training and the NASMHPD postdoctoral fellow to facilitate the involvement of consumers from diverse backgrounds. In particular, the CC will also urge that composition of the SC include members of ethnic and racial minorities, women, and others.

Building Consensus. Based on our experience with the NIAAA McKinney projects, and the ongoing SAMHSA managed care study, the key to the successful operation of a CC in a multi-site research demonstration is effective facilitation of collaboration among investigators and service innovators involved in the program. This requires a neutral stance by the CC leadership, good group-process skills by the facilitating team, and development of mutual respect. For the COS initiative, the leaders should be knowledgeable about mental health field, especially services for individuals with severe mental illness, and COS, although special knowledge of consumer operated programs will not required of all CC staff involved in building consensus. In addition to Dr. Campbell, a recognized expert in COS, we have asked Dr. Johnsen and Mr. Sonnefeld to serve on the CC because their broad expertise in the field and especially their extensive experience with multi-site evaluations of mental health service innovations. In a variety of group situations, this team has shown an ability to help project leaders achieve consensus on methods, measures, and meaning of research findings.

Minimizing Burden. Throughout the process, we will be concerned with minimizing the burden on the study sites and study respondents, not only limiting the scope and magnitude of the overall CP, but by efficient and intensive support activity by the CC--analyzing common elements, researching alternatives, carefully preparing materials for SC meetings that facilitate cross-site comparisons, disseminating common materials to the sites well in advance of meetings, and by focusing discussions on issues of greatest importance to the multi-site study, while recognizing the concerns of individual site investigators. In this we will draw on our experience in CP development process in multiple previous and ongoing multi-site demonstrations.

Technical Assistance. Sites will have made a commitment to modifying their designs as necessary and will be prepared to implement the data collection plans in their own proposals, but we expect that it will be appropriate to make TA available to some sites related to issues of research implementation. ROW has provided considerable TA on implementing protocols and modifying designs to reflect the vagaries of implementation in multi-site studies, including the NIAAA McKinney demonstrations (R.O.W. Sciences, 1991; 1993) and the ongoing SAMHSA managed-care study. Further, each TF with significant participation by appropriate consultants and study site members will develop material, conduct presentations, and provide TA to the SC and the sites.

Development and Maintenance of Common Data Repository. This section describes both the development of the database that will support the CC’s data collection and analysis responsibilities, as well as the procedures that will be put in place to facilitate data collection from the demonstration sites and to ensure data quality. MIMH will develop databases capable of supporting the CC’s activities. The major functions of the database include: maintaining a roster of active subjects at each demonstration site; tracking data points for individual subjects; generating tracking reports for each demonstration site; supporting the transfer and transmission of data files to and from demonstration sites and between MIMH and R.O.W.; maintaining data files; and, facilitating cross-sectional as well as longitudinal data analysis. These functions may be encompassed in a single database or multiple databases that can be linked.

Database Development. MIMH and ROW will work with the CMHS GPO and members of the SC to identify the functions required of the database. Given these requirements, MIMH will identify the most appropriate software and platform, and will develop a general plan for database development, testing, and implementation. Relational database software in conjunction with statistical analysis software running on a server in a local area network is preferred. The database will contain some combination of individual socio-demographic and clinical characteristics, service needs, service utilization, service outcome and satisfaction. At the project level, the database will contain data on populations, program models and components, implementation problems, local environmental contexts and evaluation methods. These qualitative and quantitative implementation process variables and contextual data will be coded and rated. Once instrumentation of the CP is finalized, the participant-level database will be designed and tested. Individual sites may have their unique requirements for the common database, and these will be accommodated wherever possible. At a minimum, the database will be used to: (1) Maintain a roster of clients who are active at each site: The database must support record additions, updates, and deletions as client status changes; (2) Track the expected dates of individual client data collection points: This is a particularly important and complex task in multi-site studies where clients are enrolled in service on a rolling basis and therefore have different dates at which data are expected to be collected; (3) Generate routine tracking reports: The CC will do everything possible to minimize burden on the individual demonstration sites. This includes generating tracking reports organized by responsible individual to ensure that data are reported on schedule and backlogs do not occur; (4) Support data transfer functions: We anticipate that some sites will transmit data for the CP electronically. The database must have the capacity to integrate batches of data easily and accurately; (5) Maintain data files: All data collected from individual sites will be entered into the database, cleaned and updated as required. Data editing functions accomplished through the database will include tests for outlier values and incomplete data and a program to check for logical inconsistencies; (6) Provide confidential and secure data storage.CC data management staff will develop flexible data transfer plans through SC to accommodate site differences in file management and transfer capability.. With this capacity, data management staff at MIMH will provide TA to sites to help them implement the data collection system.

Collecting and Transporting Data for the Common Protocol. Sites will bear full responsibility for collection of data included in the CP. It is likely that demonstration sites will vary both in their method for and format used to transport data to MIMH. Some may prefer to mail traditional raw data forms. Others will choose an electronic medium. Sites willing to enter data directly into the database format could be provided with a computer based data collection system specifically developed to support the agreed-upon CP by MIMH. Others might be entering data via commercial software packages such as SPSS-DE, the Data Entry Software for SPSS. All of these can be conveyed either on data diskettes or through electronic transfer. MIMH has the capacity to receive data in any of these forms or media. In all cases, data will be entered directly or transferred into the database, edited and cleaned. A comprehensive plan for assuring and maintaining data quality will be developed and presented to the SC during initial months of the study. MIMH has extensive experience on data quality assurance procedure developed for on going outcome study. ROW has also developed such plans for the NIAAA Cooperative Agreement and the SAMHSA Managed Care demonstration (R.O.W. 1991& 1997). The plan will include procedures for checking questionable data with the site updating fields as required. For sites transferring data electronically, a small sample of raw data forms will be requested so that MIMH staff can check the accuracy of data entry. Audit trail and frequency distributions will be run on all variables and sent to each site for verification. Quarterly reports have served several recent CAs well. MIMH staff will work with the SC to determine a reasonable interval.

Access to Data Files. MIMH and ROW will have access to all CP data files. In addition, it is anticipated that sites which do not enter their own CP data directly will wish to have access to these data. MIMH will make these data available routinely following a schedule determined by the SC. MIMH and ROW staff will provide TA to any site requiring assistance in manipulating and analyzing these data. It will be up to the SC to decide whether sites should have access only to their own data or to data from the other demonstration sites as well. This determination should be based in part on decisions concerning whether the cross-site evaluation is primarily formative or summative in nature. If, for example, the cross-site study is defined as a summative evaluation, there is some danger of cross-project contamination if data files are shared prematurely.

Data Security. Two issues are of paramount importance with respect to data security. The first relates to client confidentiality. MIMH routinely employs procedures in all studies to maintain the confidentiality of client-level data. Sites will be asked to strip identifying information from paper forms or electronic files prior to transmitting data to MIMH. The only identifier that will be used is a project ID number. Each site will maintain its own table that links the project ID to personal identifying information. The second issue related to physical security of data files. MIMH routinely uses disk-to-tape back-up procedures in all studies. Data files are backed-up daily during periods of peak activity and weekly during other periods. Tape back-ups of all project files are maintained in secure off-site locations.

Ongoing Reports to Steering Committee. The SC will determine the exact nature and frequency of reports during the data collection period. At a minimum, MIMH will compile and distribute summary reports on the status of data collection at each demonstration site either monthly or quarterly. If the SC desires, more substantive reports can also be generated, ranging from simple frequency distributions to more complex interim analyses.

Plans for Multi-Site Study. We recognize that while we will propose a preliminary multi-site study plan (see Section C), adoption of a final plan for the multi-site study will be subject to approval of the SC. We anticipate that the first four meetings will be devoted to generating agreements on both the outline and specifics of the multi-site study plan. By the end of the protocol development process, we anticipate that there will be a common plan for the multi-site study which includes the following: (1) development of CPs for baseline and follow-up data collection from consumers in the experimental and control conditions which each of the estimated 8 sites will agree to utilize; (2) development of a common timeframe for administering baseline and follow-up instruments to consumers in the experimental and control conditions. We expect that in addition to a baseline interview, there may be as many as three points of follow-up (at 6, 12 and 18 months); (3) development of a taxonomy of sampling and research design approaches which sites fall to to ascertain whether sampling and design decisions confound interpretation of data. This could include use of experimental vs. quasi-experimental designs, use of randomized assignment, and inclusion of more than two groups for comparison; (4) development of a core plan of analysis for the site-level data and an outline for the multi-site data analysis. Generally, each site will be responsible for collection of all individual-level data on service use, outcome and cost. In addition to the elements for which the sites will bear key responsibility for collecting the data, responsibility for implementing several other elements of the evaluation will rest primarily on members of the CC: (1) Collection of data about program implementation and program fidelity at each site, including information about the program model, the intensity of services provided, program staffing, etc.; (2) Collection of data about the system-level context of the COS., including information about funding, the availability of a range of services within the area, program linkages with other programs and agencies within the system, managed care penetration, and the program’s orientation to other programs that could be potentially available to recipients of COS.

Plans for the Cost Study. Like the multi-site study, final plans for the cost study will be a result of an extensive collaborative process which will be led by Brian Yates, Ph.D., a member of the CC team. This collaborative process will include representatives from each study site ,and a resource group of consultants with extensive experience in undertaking cost studies. The development of the cost study will occur concurrently with the development of the outcomes study. While the final form of the cost study will be determined through a group process, we would anticipate that the outline of the study will include several distinct components.

Like the part of the study focused on outcomes, the study will include several different components including: (1) collection of data about service use, employment, housing, vocational rehabilitation and receipt of benefits from respondents at several timepoints as part of the outcome study common protocol; (2) collection of detailed information about program cost from the consumer-operated service programs; and, potentially, (3) collection of administrative or claims data on utilization and costs of mental health and health care costs, to allow for an understanding of the way in which COS affect patterns of service use. We intend to integrate analysis of cost and outcomes, so that we can utilized the types of Cost-Procedure-Process-Outcomes cost models suggested in the literature (Yates, 1995).

Potential problems associated with conducting the cost study. While some elements of the cost study should be straightforward, others present substantial challenges. We anticipate that one section of the common protocol will be devoted to items related to service utilization, cost and benefit which will be utilized as part of the cost study. Collection of detailed information about the overall costs of COS, and development of a unit cost for these services, should be the least problematic aspect of the cost study. However, assignment of precise costs for particular individuals may not be possible in sites where consumers come and go quite freely, without generating reliable records of participation and contact. To overcome this problem, we anticipate that we will provide both a universal form and recommended mechanism for tracking consumer participation in COS within this study. We anticipate that not all of the sites will use the same program model, and that the cost-effectiveness of different program models may vary quite widely. Therefore, one critical aspect of drawing correct conclusions will involve correctly grouping programs with similar models, and using these groupings to rank these alternatives in terms of cost and effectiveness in achieving outcomes.

In addition, we anticipate that the availability of administrative and claims data may vary significantly from site to site. In some sites, managed care organizations providing public sector managed primary and behavioral health care may provide most services of which we will have interest. Because of proprietary considerations, such data may or may not be readily available to all program sites. In other sites, Medicaid claims data may serve as a record of most costs of interest. However, even here, the difficulties in accessing such information, and the difficulties of receiving it on a timely basis may present problems which some sites may find to be insurmountable. Differences between systems financed under managed care and fee-for-service arrangements may confound the distribution of services and the costs associated with COS. It is important to note that while the type of careful studies comparing cost-effectiveness of program models in other areas (for example, psychotherapy) has been more fully developed, there have been few cost studies of COS. Therefore, the strategy of inclusion on the CC team of a group of consultants with broad expertise in conducting cost studies of different types, and led by an individual who has literally written the book in this area, is one which has the highest likelihood of success.

Participatory Process of Proposal Development. The proposal for the MIMH CC was developed based on the principles of participatory design (Schuler & Namioka, 1993). Participatory design makes explicit the critical, and inevitable, presence of values in the development process. To predominant values of quality and work productivity are added broadened participation and skill development. The premise is that these values are closely related, that the productivity of the work is tied to the extent of our involvement, and that the quality of our work is a matter of technological and technical support for continuing and expanding the practices of our partners in this endeavor. The letters in Appendix 6 speak to this participatory process.