Archives   Calendar    Coordinating Center    Multisite Activities    
Project Description  COSP References  Study Sites    Search Our Site   
Table of Contents    Upcoming Meeting Agendas

______________________________________________

Steering Committee Meetings
Face To Face: June 5, 2000

 Steering Committee Meeting Notes

June 5, 2000

8:30 a.m. – 4:00 p.m.

Embassy Suites

Washington, D.C.

Jean announced that Pat Corrigan would be the Steering Committee chair for Year 3.

SC Study Blind

A discussion of the blinding of the study was initiated.

Rationale: Advantages: Research findings can become part of an intervention, and a Study Blind helps prevent bias, fraud, or unconscious study manipulation, as well as the danger that presentation or publications based on preliminary results are later contradicted.

Disadvantages: Collaborators are interested in the findings, and the need to publish.  Any problems in logic identified and corrected or altered may limit project-wide collaboration on analysis and authorship.

Brian Y. pointed out that it is important to collaborate while we are being funded for the study, as we will not have the time or resources afterwards.  This study is very unusual.  It doesn’t follow the normal life history of a big project like this.  Normally, we would start out with a very small pilot study and a small grant, then work our way up.  We’re jumping into a very large, multi-site study with no preliminary data and no really good sense of how our instrumentation is working.  If we don’t have the opportunity to check mid-way to see if we guessed right, we are taking a huge risk.

Jean said we have to recognize it isn’t to blind or not to blind, but rather degrees of blinding.  One of the considerations, based on the level of blinding, is that we must do it across all sites for validity. 

Brian M. remarked that he has not heard any discussion on blinding to the baseline, only outcomes, and asked for clarification.

Brian Y. said there was a 4-point motion put out by the CC.  Jean C. said the CC wanted to amend the motion to recommend that it go to a committee to develop a fuller, more nuanced motion.  She recommended that it go to the Logistics Committee and that it come back to the September meeting for a vote.  Jean was asked if she wanted to pull her motion off the table.  She said the motion had already been tabled and reiterated her recommendation that it now be sent to committee for further elaboration  and then have the committee come out with a fully nuanced recommendation. Crystal, Jean and Sita were named to the work group for this task.    

Brian Y. was concerned about being able to look at the 560 baseline interviews and the 80 at his own site.  Jean said the CC motion was only for the multi-site data and that Brian could look at as many as he wanted at his own site.

Betsy M. – Although the policy should address better not two sites get together and compare their data, I think there should be an understanding that this is just another nuance for the “nuance team”.  Betsy also recommended that the SC could instruct the CC to put into place some emergency conditions under which they might break a blind or break part of a blind to this non-equivalence of groups.  It may be necessary to look at mid-course data to decide whether or not corrective actions should be taken. That does not mean that the SC will be scrutinizing the data, but that they could think of all kinds of issues and problems that would come up.  The CC looks for that and reports back.  That might be another way of not breaking a blind but feeling reassured that things are staying with the perimeters of safety.

Tom S. asked if there was any objection to sending the policy to the Logistics Committee for nuancing and articulation?  There were no objections.

The motion on the issue of blinding was tabled for discussion by the SC on the listserve.

It was suggested that the policy be sent to the Logistics Committee for nuancing and articulations. There were no objections.

CC Cost Study

Brian Yates, Matt Johnsen, Jeffrey Merrill

Matt introduced Jeffrey Merrill who has completed a review of the cost study.  Matt began with the review that he and Brian prepared for the revisions to the cost study plans.   One of the concerns raised at the last meeting was the value of the cost study, concerns about cross-site comparability, and other red flags.  In response to critiques, there were several reviews, as part of a scientific review by Goldman and Siegel.  Jeffrey Merrill, who focused on the cost study itself, completed a separate review.

Matt reiterated that the components of the cost study are not a single entity, but rather three points: Input cost (staff, volunteer time overhead, etc.), Procedures (program activities), Cost-related outcomes (reductions in hospital costs, ER costs, housing subsidies).  The overarching approach is the careful consideration of information already being collected in the Common Protocol.  There is a renewed emphasis on Track 1, as well as emphasis on issues of validity and reliability, and the need for more clarity.  There is agreement that the cost study is an important element of the multi-site study.  Questions likely will impact policy.  There is a need to balance maximum scientific rigor and usability in monitoring costs.

Cost related outcomes, measuring outcomes as cost-savings, and direct benefits are valid and very important element.  These data are found in the Common Protocol at the moment.  In terms of input cost, questions about Track 1 and Track 2 and uniformity have raised questions.  Matt said we need to insure Track 1 is embedded within the Track 2 approach.  David Lambert asked for clarification of embedded.  One of the points of non-comparability is that if we estimate them in one way in one track and not in the other, are we measuring the same thing?  We rethought the data elements and concluded that Track 1 is the fundamental basis of the multi-site analyses.

There are differences of opinions about individual level of data collection, i.e. should we take just an average cost per part vs. cost based on use of program?  All reviewers felt that volunteered costs are important.  Matt said that certain analyses could be done with imputed values.  In terms of the cross-site comparison, the suggestion was for a meta-analysis for comparisons.  The areas needing resolution are the role of procedures in analysis, optimization, measurement of consumer participation; cost detail on TMHS as well as COSP, actual vs. estimated costs. 

Brian Yates presented an overview of the modified and clarified (new and improved) cost study research plan.

The cost study goals measure:

·        Outcomes - monetary and monetizable, cost savings, cost offsets, and direct benefits.

·        Input costs (program expenditures, money, volunteer, donated space, etc.).   

·        Consumer participation in programs (overall participation and participation in specific programs).

The Cost-related Outcomes include:

·        Cost savings benefits include: Inpatient hospitalization, ER Dept. visits, Crisis intervention, Social Services, Transfer Payment Reduction, Costs offsets (housing, Criminal Justice, etc.).

·        Direct benefits include: Employment and Volunteering

Brian explained Service Utilization Items 1 & 2 and the need for Service Utilization Studies; Track 1 inputs costs and the how sites can minimize the burden of Track 2 cost measurements; Technical Assistance and support will be available in Year 3; Measuring effectiveness and Consumer Participation; Program Activity Resource Use I & II; Side Studies, Papers and the variety of sites working on side studies; the Status of Cost Study; and, the Next Steps including the logistics of reporting cost study data and the need for collaboration on tasks.  

Jeffrey talked about the cross-site study.  He said we live in a cost conscious world.  In the old days we compared costs between the same intervention, but today we look at one intervention (MH) compared to another (Cancer research).  The carve out of mental health service is proof of this.  Therefore, this type of research is of the utmost importance for mental health services.

Projects like COSP are services that are cost beneficial.  How do you determine if something is cost beneficial and cost effective?  Our Common Protocol gives us this valuable information.  What kinds of services have the greatest impact on cost?  Which of these services have the greatest impact on consumers.  You need good methodology that is both scientifically solid and relevant in order to realize benefits.

There are two levels of analysis: individual site and cross-site.  Each site must collect good Track 1 data.  Any site that can do Track 2 data collection, should do so.  Brian feels if the resources are going to be put in any area, make sure every site is collecting comparable Track 1 information.

There are three reasons cross-site evaluations are important:

       1)  Without cross-site evaluation the study at one site is effective, but results are not                      

             generalizable.

            2) Credibility makes your findings more credible.

            3) Increasing your numbers gives you power.

Joe asked if in cross-site, will you pool the costs?

There is tremendous variability across sites of how costs are calculated.  This is problematic for calculating cross-site costs.  There is no single way of doing this, when you go cross-site.  If all sites are using Track 1 data, by all means use this.  If not possible, then do a meta-analysis, i.e., 30 studies with different costs. An enterprising individual will do a meta-analysis.  It is imprecise, but it is a way of getting around the problem and getting more generalizable results.

Each site could give a good sense of service utilization to clients at that site. But, rather than using cost per service at that site, the site can use an average cost.  Instead of trying to figure out service at each site, use national average of costs. One way of doing this is to make it generalizable.

There are two ways of looking at volunteer service.  Look at it as if it is not a volunteer service, but a fully funded service.  If you can demonstrate in a cost benefit analyses that you are paying $15 an hour for that service and it is still cost beneficial, then you are in a lot stronger position to get somebody to pay for those services, rather than get a volunteer to do those services.  The more data you can collect, the better.

Tom raised the issue of societal costs, and how they can be measured.  The important thing is to make sure that the input costs and benefits accrued are collected with the payer in mind.  Here are the benefits that accrued and the costs associated with them.  David asked how that would be tracked.  Stay with the things you can get from your Common Protocol, put a price on them using community costs that have a range of best case and worst case scenario, and multiply by unit cost.  

Susan says that all of the reviewers have said collect the costs at the particular site.

Matt said that if we can use the efforts in some sites to open the black box at another, it says many things, and can answer some of those things we are curious about.  We can use the details to tell us something about the validity we have, and we can purchase some increased understanding.

Brian said that there are three reasons why the individual sites are so important:

1.      Site needs to have information on its own benefits and services

2.      Cross-site evaluations will have imperfections due to the combining of information.  The individual sites will help validate the cross-site, and the cross-site will help validate the individual sites.

3.       Sites that are able to collect data in-depth will be able to answer more questions that can open the black box and help other sites.

Jeff says we can get more answers from going into the black box.  Across sites the number gets big enough, there is more variance, and we can get a lot of suggestive data.

Greg read the following statement written by Carole Seigel:

“The basis question of the cost analyses is much too complex a question to answer.  I do not think that this study has the capability of disentangling combinations of resources that go into the procedures that impact outcomes.  I do not know of any study that has done so.”  Greg questioned whether we could actually get inside that black box.  The response was a belief that we could and it was worth doing.  

Crystal wanted to speak to the “do ability” of the cost study.  Danielle’s spreadsheet is getting full – which speaks to the “do ability” of the cost study.

Greg says that he is confused about how the unit cost of program activities is estimated.

The explanation given was that one of the things that is helpful for the Track 2 perspective is the information that is collected within the Common Protocol.  So, for each of the COSP program activities there is a cross-site measure. Even if we can’t do Track 2 at a particular site we will in fact have some information from individual participation in different activities.  We realize that there are analyses possibilities that are already in the data whether or not people elect to go to Track 2.

The difference between Track 1.5 and 2 is that Track 2 opens up the box.  Some programs will look at total cost and some programs will look at participation.  It is up to the individual sites to determine their own Track.