Archives   Calendar    Coordinating Center    Multisite Activities    
Project Description  COSP References  Study Sites    Search Our Site   
Table of Contents    Upcoming Meeting Agendas

______________________________________________

Steering Committee Meeting Notes:
December 6, 1999

11:30 a.m. – 6:00 p.m.

Embassy Suites

Washington, D.C.

Gearing Up for Year 2 – The Federal Perspective

Mike English

Mike English, Director, Division of Knowledge and Systems Change for the Center for Mental Health Services (CMHS), indicated that he was impressed with the great progress that the COSP has made.  During the next year or year and a half, CMHS is holding off advertising new studies and using the time to digest what it has learned from the 12 multi-site studies underway. 

The idea of a multi-site study for CMHS is an evolving one.  CMHS did not go to the literature and pick a methodological approach and impose it.  For the ACCESS project, CMHS contracted with a coordinating center and then contracted with the sites.  For the EIDP (employment multi-site study), CMHS established a Steering Committee and it set the policy and direction for the overall study.  He indicated that it is difficult to affect policy with 500 individual studies going on at once.  CMHS wants to affect policy. 

CMHS’ hypothesis is that if you conduct multi-site studies with semi-independent sites with site-specific studies that you get a double bang for your buck.  This creates high opportunity for richness, but it creates problems for sites that have to balance multi-site and site-specific activities.  A second hypothesis is that several minds are better than one and collaboration will yield better approaches.

Mike discussed the following five lessons CMHS has learned about multi-sites studies:

1.      The GFAs are crummy in that local proposals have to be submitted prior to the sites’ involvement in the multi-site study so that reviewers have something to review.  It takes about 16 months to get a multi-site study going.  Mike said CMHS won’t do it this way again. 

2.      Democracy is painful in that it is time-consuming, it creates anxiety and it pits people against one another.  Although joint decision-making is difficult, Mike said he is convinced that the difficult process will yield better decisions.  In these situations, the respect colleagues show for one another can make or break the program. 

3.      Closure must be brought to issues so that they don’t get rehashed.  Participants in multi-site studies are the stewards of significant public resources and they must be able to get their job done.

4.      People don’t have time to get everything done that they want to in their studies.  Most studies have been limited to three years.  Mike assured COSP participants that he would do everything possible to ensure that the opportunities created by the COSP to learn more about this and related topics will be pursued.

5.      The coordinating centers and federal representatives involved in multi-site studies are in a bind.  They are caught between the multi-site and site-specific interests and are pawns in the democratic process.  He said there are questions that need to be answered such as “should the federal representatives have more or less authority in the multi-site studies,” “should the coordinating center principal investigator be a facilitator or be more authoritative,” “how much should we rely on the coordinating center and to what extent do we allow it to give the benefit of its expertise or should it serve as the facilitator of the Steering Committee’s decision-making.”

Mike asked for and received the following ideas on how to improve the multi-site process:

1.      Yvette Sangster said as a consumer she understands more about research and that is cool.  Consumers want to make the project work more than anything because it is more than a research initiative-—it will determine the way consumer organizations live.  She suggested that a copy of the Coordinating Center’s proposal to CMHS be provided to the study sites so that they know everything that will be required up front instead of getting information piecemeal.  It was mentioned to Yvette that the Coordinating Center’s proposal had been made available to the study sites previously.

2.      Sally Rogers said the GFA has improved dramatically, but the tricky part is asking sites to submit a research plan that gets dismantled for the sake of the multi-site study after the sites have gotten support from the stakeholders.  She wishes there is a way to educate people to restructure the research part of the GFA.  She also recommended that a multi-site summit be held before a GFA is issued so that problems and strengths can be explored beforehand.

3.      Carol Mussey spoke positively about the democratic process.  An example of the process working was the preliminary, but brilliant document that defines common ingredients of the consumer-operated service programs involved.  It was painful, but she learned to listen and speak up.  It was empowering.

4.      Tom Summefelt used Pat Corrigan’s words to describe the democratic process as “charming, but challenging.”  The challenge had to do with the GFA and what Yvette spoke to—the lack of clarity about the role of the Coordinating Center and the IRB requirements of the Coordinating Center.  Tom suggested that the multi-site members be the preliminary GFA reviewers.

5.      Greg Teague expressed interest in building on the concept of the multi-site summit.  He said an important part is trying to turn into knowledge developments about systems change.  He suggested that there be an integration of the information being learned from community action grants with that of what is being learned from the multi-site studies. 

6.      Jo-Anne O’Connor said she hoped the summit would include members of the various multi-site Consumer Advisory Panels.

7.      Dianne Cote said she and Sally Clay have been part of the consumer movement for 20 years.  She is glad the federal government is finding out why consumer-operated programs work.  She asked if information was available through exemplary practices so that sites interested in establishing drop-in centers could access the information on what works.  Crystal suggested that this type of information should be on our website.  Many of these interventions are not well documented and there is a need for the manualization of interventions.

Mike accepted the suggestion for a multi-site summit.  He asked the COSP to host the summit of all of the multi-sites in June of 2000.  Steering Committee members agreed to host the event.

Gwendolyn Simpson – Grants Management

Betsy asked Gwendolyn to remind the COSP members about grants management requirements and the role of grants management.  Gwendolyn and Steve Hudak are the grants management specialists working on the COSP.  Either one can be contacted by the study sites or Coordinating Center.  It is best to contact grants management through e-mail. 

Gwendolyn explained that grants management works closely with the project officers to help ensure that the study sites and Coordinating Center succeed and reach all of the objectives of the project.  It monitors fiscal compliance and is a counterpart to the business office in organizations such as those participating in the COSP.

Budget changes can be made without prior approval so long as the amount being changed does not exceed 25% of the total award or $250,000 (whichever is less).  For example, grantees can rebudget from personnel to equipment or supplies, unless they try to move something into a line item that was approved in their original budget.  If an equipment line item was not in the original budget, then prior approval is needed to establish an equipment line item regardless of the amount.  The government considers items that cost less than $5,000 as supplies as opposed to equipment.  The 25% rule applies only to the grantee and not to the subcontractors.

Prior approval is required for alterations or renovations of existing facilities.  Construction of new facilities will not be approved.  Betsy said if sites are going to move a wall or add a stove to a kitchen, they should call grants management for clarification.  Prior approval is also needed for production costs of audiovisual materials that exceed $25,000 or for the printing of a single publication that exceeds $25,000.

The carryover of unobligated funds from one budget period to another is permissible, however, a written request to do so must be submitted to Steve or Gwendolyn and a copy send to Betsy.  These requests should include the grant reference number, a programmatic and budget justification, a copy of the Financial Status Report 9FSR) and signatures from both the principal investigator and program director.  The FSR and requests for use of carryover are due 90 days after the end of the budget period.

Gwendolyn cautioned sites not to do anything related to carryover until a revised notice of award is received from the federal government.  If this is not in writing and the carryover funds are used, the auditors will make you pay the money back.

Reporting requirements include submission of quarterly progress reports to the Coordinating Center, the FSR to grants management at the end of each budget period, and a quarterly financial report (Form 272).  Form 272 might be completed and submitted by the financial persons at the grantee organizations.

Gwendolyn distributed handouts that carried information on where to get OMB circulars, the FSR form, and PHS Grants Policy Statements. 

Comment about Steering Committee Meetings

Jo-Anne O’Connor asked to address the committee.  She said we have a wonderful collaborative process, but the Steering Committee face-to-face meetings are too long and do not include enough breaks.  For health reasons, she asked that consideration be given to how the meetings are handled.  She asked for an actual agenda of topics to be discussed instead of a schedule, that any materials that need to be reviewed be posted to the website at least one week before the meeting, and that the face-to-face meeting time be reserved for the Steering Committee instead of subcommittee and work group meetings.  These issues will be discussed the last day of the face-to-face meeting.

Progress Reports

Project Directors’ Meeting

Jo-Anne reported that the project coordinators met yesterday to discuss the pros and cons of forming a work group.  Coordinators recommended that a work group be formed to focus on the day-to-day implementation of the multi-site study.  The purpose of the group is to exchange information, do day-to-day problem solving, and provide support.  The chair of the work group will change each quarter so as not to burden one site.  The group would work from a private listserv and occasionally have a teleconference, if there is a need.  It might occasionally meet during the face-to-face meeting, if there are pertinent agenda items.

Sally Rogers asked what kind of life the Logistics Subcommittee Group would have after the study starts and wondered if there was a segway into the project coordinators work group for the handling of day-to-day implementation issues.  Tom said there is still some work related to the follow-up common protocol that the Logistics Subcommittee Group needs to undertake.  Sally recommended an interface with the project coordinators.

Qualitative Research Subcommittee Group

Initially the Coordinating Center tried to convene a group to look at issues around the qualitative component of research for the multi-site study.  However, the extent of the work of the group was the two open-ended questions in the common protocol.  The group needs to re-convene to look at the site-specific qualitative research and see if it can be used to inform the multi-site study.  After the first of the year, Jean Campbell will seek volunteers for this work group through the Steering Committee listserv.

Brown Bag Lunch

During lunch, the video from the Well Being Project entitled “People Say I’m Crazy” was shown.  The video was followed by a panel discussion that included Jean Campbell, Ruth Ralph, and Zahira DuVall.  A copy of the video and other Well Being materials were distributed to the consumer representative from each study site.

Cost Study Progress – Site Presentations

California – Bonnie Schell

Bonnie Schell shared some procedures and sample forms that have been developed for her organization’s participation in Track 1 of the cost study.  Among the new forms was a sign-in sheet for the drop-in center that asks for the arrival and departure time for participants.  Bonnie said her center probably won’t be able to get the departure times.  Her center will also provide the number of visits of all consumers to the COSP in a representative month.  There is a form to determine the number of people that came in each day of each week.  Information from the forms can also be used for program management.  Bonnie can see what activities or which staff members are on duty a certain day and whether or not attendance was down.  She can track improvements that are made to the center and see how the improvements affected attendance.

Last year, the center’s total budget was $124,000 and a “visit” cost $20.75.  This year the budget is $184,000 and a cost for a visit is $23.86, which reflects enhanced services and a boost in attendance.  She is also keeping track of volunteer hours in a representative month—this is something new that is being done for the cost study.  A form has been developed to track volunteer services such as the taking out of garbage, providing of transportation, and answering of the telephone.  The supplies that are donated will also now be kept track of.  Supplies include bus passes, donuts, bread, etc.

Maine – David Lambert & Janine Elkanich

David and Janine described their approach to obtaining data for the cost study and said they would let Brian Yates determine which track they best fit.  They have taken care to not overburden the traditional mental health service (TMHS) providers.  David said one advantage in Maine is that 90% of respondents will be on Medicaid and both of the TMHS providers rely on Medicaid billing.  In addition, the state has revamped its billing data so that data are available a month after each quarter as opposed to having a 15-month wait.  The two major billing categories are community support and in-home support, with community support being the biggest.  David said progress note records are reviewed each three months for a certain period specifically for medication management, crisis intervention, supportive counseling, independent living skills, monitoring ISP, and linking to other services.

Janine explained that the TMHS providers offer an array of services that have the same costs.  It is difficult to assign a dollar amount to that piece of the study.  If someone gets 5 units of case management and 10 units of crisis intervention and it flips to 5 units of crisis intervention and 10 units of case management, if you are just using the dollar amount, you will lose a big piece.

On the COSP side, it is more challenging because its space is one wide open space and people access all of the space in their own time.  Janine tried to break up the space into sections that were actually used and grossly estimate what that square footage would be for each section.  For example, for art, the cost was 60 cents per square foot.  The problem is going to be tracking everyone who participates in the COSP.  There are times when people spend 5 minutes there and others spend 5 hours, but the person who was there for 5 minutes might get more from the program than the one who stays for 5 hours.  The art piece is challenging.  For example, if someone does actually create a piece of art, but benefits from participating in the discussion about the art and watches others create, it is a challenge to document this.  The center did not have a sign-in sheet for art before, however, for the cost study, now a voluntary sign-in sheet has been created.

Janine expressed some concern about what will come out of the cost study reports.  If they only look at raw data, we won’t be able to capture certain aspects of the COSP.  People go into COSPs for several different reasons and to be able to capture this will involve some time and effort.  Brian Yates interjected that we just can’t use statistical analysis and that we have to talk to COSP participants to get descriptive results.  He said he does not compile costs and forget about the procedures.  Janine indicated that she has a fear that COSPs are going to be misunderstood.

Janine has priced out meals, computer use, etc. by square foot.  She also has pilot data from five people who participated in pilot interviews.  Their cost data is being analyzed for a six-day period.  Janine pointed out that their center can not bill and that no matter how many people come through the doors, the COSP is committed to serving them.

Iowa – Sally Rogers & Sandy Lint

Sally pointed out that Iowa is different from some of the other study sites in that peer support services are not a placed-based service.  Consumers don’t go to a place, the services are dispersed.  In a rural setting, it is important to factor in transportation time and costs.  A large percentage of service is provided by one TMHS provider—ResCare.  Iowa is still not sure which track of the cost study it falls into.

When Iowa developed its cost study form, it wanted the forms to have multiple uses so that when the study ends, the forms will still be useful.  The form therefore contains information needed for the program funding body.  Sally emphasized that the study site has to have leverage with the people from whom the data are being collected.  In Iowa the peer support specialists will collect the data and they get a stipend from the study site.  People have to be trained on how to use the form and the form must be monitored.  There was concern with subtracting services—they didn’t want research costs in the cost analysis since one of the reasons for a cost analysis is to provide information on the cost for replication of services.

Another issue that had to be addressed was what is bundled into the TMHS.  Because these services are placed-based, volunteer time is relied a lot.  Iowa has to figure out with Brian Yates how to figure volunteer costs.  Some volunteers provide their own resources such as use of their home, telephone costs, and car expenses, and this has to be factored in. 

Iowa is trying to determine how the peer specialists spend their time.  A time can’t be assigned to each procedure, but Iowa will be able to describe the activities that consume most of the peer specialists’ time.  Iowa also has to determine what the peer support supervisor does and a separate form may be needed for that.  There are definitions for each service procedure so everyone is using the same definition. In addition to the information on the form, there are other costs to consider such as those associated with space and overhead.

Sandy explained that there are two components from which records will be obtained.  Information will be requested monthly, however there will be a quarter of lag time for getting the information.  Iowa will get actual reimbursement costs from the billing data reconciled with the payment.  Sometimes a service is provided and the organization does not get reimbursed—these costs will be incorporated into the volunteer category.  Sandy will provide the agencies with a list of names and ask for costs for everything billed and reimbursed for that person.  A form will be developed for the primary coordinator at the traditional site.  She will work up costs for hospitalization and county case management services.  Iowa can not get the actual cost for services, so it must rely on reimbursement.  Reimbursement costs match actual costs because the agency is for-profit and it is breaking even as opposed to making a profit.  To obtain the reimbursement cost per consumer is the best Iowa can do.

Florida – Rich Beaulaurier and Dianne Cote

Florida is collecting cost data under Track 2.  Florida has tried to report on the cost for various sorts of things such as the hours per person and the cost per person based on square feet.  It is too difficult to do actual good and services for each person.  The PEER Center has a 12% overhead rate.  Florida added a section to Brian Yates form to capture direct costs without overhead.  The cost is $8 per person.  The overhead category includes supplies, space, personnel devoted to administrative tasks—the 12% came from an annual audit.  The volunteer time figure also came from the audit.  In-kind services are anything contributed including volunteer time, goods, or services.  The PEER Center provides quarterly reports to the state on the number of consumers served. 

Brian Yates

Brian indicated that the other study sites have also undertaken activities related to the cost study.  He said it has been a pleasure to work with the sites and have them take the ball and run with it.  Sally Rogers asked how to put a price tag on volunteer time if the agency doesn’t have an independent audit and how to unbundle services such as room and board costs for residential programs. Some additional work on these questions will be undertaken so that things are dealt with similarly across the study sites.

Progress Reports

Cost Analysis Subcommittee

Brian Yates reported that the subcommittee has been busy working on the common protocol and that he has been working with each site on the cost study.  He indicated there is need to ensure that we are looking at traditional service costs in the right way and in a way that is common across the study sites; therefore the cost listserv will rise again and a teleconference will be scheduled.  He said he was pleased with the progress made in the cost study.

Data Repository Subcommittee Group

Matthew Hile reported that two issues were discussed: (1) verification of data entry and (2) post-interview observation form.  The Coordinating Center is asking for all data entered into the system to be entered by one person and verified by another.  At those sites that are using laptop computers, the interviewers will be required to be trained by the criteria established by Joe Sonnefeld and re-tested for fidelity in 6 months. 

The group recommended that during the process of the interview that the interviewers have some judgment as to the validity of that interaction and that this information be captured either for each scale in the common protocol or for the entire questionnaire.  Jo-Anne O’Connor had recommended at the interviewer training that the last page of the common protocol consist of a post-interview observation document.  Carol Silverman expressed concern that we will be validating someone’s voice and said input from the CAP was needed.  Kathryn Kidder said she believed the data should stand as respondents report it and that putting a judgment on their information is not ethical.  Jo-Anne emphasized that we would not be judging the person’s answer, but rather judging if the person put thought into each section or if the interviewer thinks the person was hearing and understanding the questions. 

Tom said in Tennessee the very last part of the site-specific protocol consists of an open-ended comment field and rating of the interviewer’s perceived reliability of that particular interview and what sections were difficult for the respondent.  The scales are set up in such a way that could evoke a response pattern.  Matt Johnsen said in looking at the psychometric data, we have relied upon previous folks who developed the scales to deal with these issues.

Jean asked that the discussion be suspended and resumed over the listserv so that it can be brought to the Steering Committee for a vote.  She also asked that the CAP discuss this and provide input.

Consumer Advisory Panel

Zahira DuVall reported that the CAP had a productive meeting Saturday before the beginning of the face-to-face meeting and that it plans to meet the day before each of the future face-to-face meetings. 

The CAP recommended that the common protocol issues be resolved during the December face-to-face meeting so that Tennessee can go into the field the first of January.

She said every COSP in the study has voiced concern to the CAP that they are having difficulty with the way the money has been spent.  There are legitimacy issues in the field, pools are shrinking, and people at the study sites are waiting for the study to begin.

The CAP did some tweaking of the introductory section of the common protocol.

CAP members felt strongly that participation by the COS in the cost study should not alter the service delivery methods of the COS.  There should be no pressure to change service delivery to satisfy the collection of cost data.  COS should be allowed to use the same sign-in sheet they have always used.  To expect that certain procedures will be implemented violates the “do no harm” philosophy.

The CAP recommended that a team of liaisons from each study site works with Vicki Wieselthier over the next three months to develop policies and procedures for what is accessible on the website.  Too much information is available for the general public and levels of security need to be developed.  The CAP would like a report by the March 12 face-to-face meeting.  In the meantime, CAP is recommended that all public access to the listservs on the website be cut. 

The CAP felt strongly that reasonable accommodations at the face-to-face meetings beyond the one day need to be made.  Many of its members take medications and can’t start the day until 10 a.m.  There need to be regular breaks and a more reasonable schedule even if that means adding another half day to the meeting.  If the CAP could have meeting time to caucus during the day instead of always in the evenings, it would be appreciated.

The CAP recommended the COSPs will not be required to change their service procedures in ways that they object to.  COSPs will agree to enter into good faith collaborative efforts to find acceptable alternatives, if possible.  There were no objections.  Consensus was reached.

The CAP recommended that each study site have the opportunity to appoint one liaison to work with Vicki Wieselthier and a federal representative over the next 3 months to make recommendations at the next face-to-face Steering Committee meeting regarding the levels of access to the website and to have public access to the COSP listservs stopped immediately.   The recommendation passed with 17 votes in favor and one abstention by the Coordinating Center.  Sally Rogers suggested that there be some statement to the public about why access was now being denied.

A lengthy discussion about the website took place prior to the vote and the following comments were made:

Yvette Sangster said she had expressed concern about the openness of the website from Day 1.  There is too much information that is accessible and this if prospective respondents have access to the listserv dialogues, this could contaminate the study.

Brian McCorkle said that several major journals have modified their publication policies whereby authors are not allowed to publish something that has been published.  If something is posted on the website, it is considered published and the COSP may be prevented from submitting articles to journals.

Mark Salzer said he was conflicted about this from a process standpoint.  There are thousands involved in mental health who have not had access to information in the past and for the first time we have a federal study looking at consumer-operated services. We could potentially be shutting people out from getting information.

Jo-Anne O’Connor said she did not want the website taken away or shut down, but she did want the listservs private.  The CAP is asking for 3 months to think about what is the right thing to do.  If at the end of 3 months, the decision is to make the listserv public again, then it will be public.  We just want some time to look at this issue.

Paula Stockdale said she agreed that some consumers are well versed in use of the computer and if some prospective participants read some of the listserv discussions, they might be angered.

Fidelity Implementation & Site Visits

Matt Johnsen reported that this work group discussed some information that was made available over the listserv called measuring program fidelity.  This document explained the work of what started out as the Common Ingredients Work Group.  The work involved establishing a list of common ingredients and operationalization and development of the fidelity instrument.  The Research Subcommittee was supportive of the efforts and provided additional suggestions and guidance. 

The name of the instrument was originally the COSP Fidelity Implementation Tool (COST FIT), but it was pointed out that this might not be the best acronym and Sally Clay suggested either COSP True (Tool for Real Use) or COSP Fact (Fidelity Assessment Common Tool).  Some additional effort will be made to identify a name.

Matt said there was a recommendation that those who receive a list of common ingredients understand that this is a list of ingredients commonly found in consumer-operated services rather than a list of ingredients found in all COS.  There also needs to be a disclaimer noting that the document does not represent an overall rating of COSPness.  It was also suggested that the group continue to develop and refine the work that has been done.  Another teleconference is needed to proceed.

In addition, it was suggested that the fidelity instrument be piloted during upcoming site visits.  The group should also involve other stakeholders and not just pilot the instrument during the site visits to look at inter-rater reliability between site visitors and validity between the groups.  Matt acknowledged the significant amount of progress made by the work group.

Betsy complimented Matt for undertaking this enormous task and described the group’s work as exciting and an important part of the COSP.  She said this is a good example of how consumers have contributed and it represents a nice example of collaboration.

Tom indicated that the common ingredients would be a great place to begin working on the first publication to come out of the COSP.  He suggested that a book be developed in which consumers at each study site could describe their program and sum it all up with the common ingredients. 

Yvette moved to form a group making sure each study site and the federal representatives are included to pursue development of a book on common ingredients with Sally Clay, Ed Kramer, and Matt Johnsen serving as editors.  Tom seconded the motion.  There were no objections.  Consensus was reached.

Site Visits- Matt said there was a fair amount of discussion about the site visits.  Three have been completed, and R.O.W. send out a manual and materials to help the sites prepare for the visits.  Some question has been raised as to whether site visits are just site visits or if they are research.  Everyone had a different understanding of research and the group didn’t come to a clear understanding.

The group did decide that local IRBs need to be consulted about the site visits.  From the discussion, the group decided:

1.      From this point forward, the study sites should consult with their local IRBs about the site visits to make sure the visits are being carried out in accordance with local requirements. Sally Rogers said if the IRB has to be consulted the providers in addition to the consumers might have to sign informed consent forms.

2.      R.O.W. will send another copy of the site visit manual to the study sites.

3.      Because sites are being asked to consult their local IRBs and might not be able to get this done prior to the currently scheduled site visits, R.O.W. will negotiate with the sties about the timing and support.  The support issue came up in relationship to the focus groups—some Steering Committee members felt participants might need some additional support in the form of staff members sitting in on the focus groups.  If an individual requests such support, this is a reasonable accommodation.  However, if an individual would like, the opportunity to speak with the site visitors one-on-one would also be allowed.

4.      There is still some disagreement about whether the onsite researchers should sit in on all meetings held during the site visits.  Matt explained that the rationale for not having researchers sit in was that it was not appropriate to the extent that part of what was being determined through the site visit is how the research is going and if there were specific issues related to the research.

Betsy suggested that this be worked out by the sites negotiating with R.O.W.