Archives   Calendar    Coordinating Center    Multisite Activities    
Project Description  COSP References  Study Sites    Search Our Site   
Table of Contents    Upcoming Meeting Agendas

______________________________________________

Steering Committee Meetings
Face to Face: June 4, 2000

Steering Committee Meeting Notes

June 4, 2000

12:00 p.m. – 5:30 p.m.

Embassy Suites

Washington, D.C.

Welcome

Tom Summerfelt, Consumer-Operated Service Program (COSP) Steering Committee Chair, welcomed new faces, and had those who were at their first meeting introduce themselves.  He said we have worked hard and needed to have a little fun.  On that note, Sita read an amusing piece on COSP diagnoses.

CC Updates

Jean Campbell – Coordinating Center PI

Jean gave an update on personnel changes at the CC.  The follow-up protocol is on the freedrive, and Jean asked that the sites have training updates for their interviewers.  Work continues on issues of deadlines and communication, and the CC is committed to being sensitive to peoples’ needs.  The draft of the brochure is now ready, and each site will be asked for approval.

At this point, every site has submitted their checklist, and has received at least preliminary feedback.  Next week Jean will finish going through all of the submissions and a letter will be sent informing the site that they have completed the process.  This letter is required in the site’s reapplication packet.  There were some issues on exclusion/inclusion criteria that had potential impact for commonality between sites; however, these were resolved.  Jean thanked the PIs for their cooperation in the process.  Mike English suggested that any issues that would impact all of the sites should be brought before the SC.  The following items need to be brought before the Logistics Committee:

1.      Development of uniform process for competency to give consent

2.      Non-respondent bias

3.      Issue of whether we can blind the study to the staff.  The potential to treat in different ways could impact outcomes.

4.      Length of time between baseline and start of the service, as some situations may have changed in an extended time (i.e. housing, etc.).

In Year 3, the CC will provide some technical assistance to the sites.  The CC will hire a part time biostatistician to help sites “snoop the data”.  Money has been allocated to hire an additional economist to help Brian.  The CC will supply site visits if the sites feel they are needed.

Regarding IRB issues: we will be required to report any adverse effects attributed to the study.  There is new thinking about IRBs.  Donna Shalala has made some new regulations, and the information is included in packets handed out.  Some of the changes found in the checklist may require them to go back to their IRB.

Rita gave an update on the data repository and a demonstration of the follow-up data entry program.  A new procedure for totals on the quar4terly reports was outlined, where the total number of respondents interviewed at baseline will be determined by the number of completed and verified records submitted to the data repository.

Multi-Site Research Plan

Matt Johnsen, Rise Goldstein, Joe Sonnenfeld, R.O.W. Sciences, Inc.

Matt Johnsen from R.O.W. said that according to the GFA, the CC was responsible for cross-site analysis.  The research design presentation includes overview of project, the program models (CI), research questions, hypothesis and multi-site research design, and study project.  Scientific Research is based on peer review, and we have to maintain the highest level of rigor, rather than be average.  We want to know if there are any fatal flaws, so it is important to have reviews while we are in the planning stages.  Joe interjected that we want this for publications.  Jean added that it is also to help guide Public Policy.  To accomplish this, we have commissioned several reviews of our efforts, including ones from Howard Goldman, Carole Siegel and Jeffrey Merrill.

Rise started with a recap of the logic model, though she gave credit to others who worked very hard on this before she started with the project.  She discussed the research questions that will be addressed in this study.

RQ1:  To what extent does participation in a COS program affect selected consumer outcomes for consumers who use traditional service programs?  This research question is listed in the GFA.  Some outcomes that will be measured include empowerment, employment, housing, etc.  Beyond the GFA, we will look at other outcomes, such as hope, symptoms, physical health, and substance abuse.

RQ2:  To what extent does participation in a COS program affect costs?   Some of the outcomes we will look at include inpatient days, crisis intervention, housing, etc.

Research Hypothesis: Outcomes short-/intermediate-term, and long term.

Short term Hypothesis: For the experimental group, there will be increases in empowerment, housing, with more rapid improvement in employment status, higher levels of social inclusion, higher levels of satisfaction with services, hope, quality of life and social acceptance.  Also more rapid decrease in symptoms, substance abuse, and physical health problems.

Alternative hypothesis: different patterns by randomization condition: nonadditivity of COSP to TMHS?  Possible examples are empowerment, symptom reduction, and nonspecific benefits of doing more services in TMHS+COSP condition? (Not how much you do, but what you do.)  Joe brought up that there is research to support this, and Jean added that it could be both more and specific, so it will be a statistical challenge.

Long term outcomes: More sustained gains, and greater continuation in improvement, in empowerment, housing, employment, etc.  More sustained and continual improvement in distress with symptoms, substance use, etc.

We will look for differing patterns in improvement over time by randomization condition.  The key scientific issues raised by scientific reviewers:

·        Need to be more precise in research questions and hypotheses

·        Need to specify a priori the primary research questions and hypotheses

Joe R. asked why we needed to do this.  Matt responded that Carole Siegel said this needs to be done a priori for power considerations.  Power is driven by effect size.  We need to decide on the question, and then we can do a clean power calculation based on that.  The power calculation is necessary, as there has not been COSP vs. COSP/TMHS research before.  If there are 20 differences, only 1 in 20 might be significant, so therefore we need to have a clean calculation at outset.

A few more issues raised by reviewers: Carole identified the need to relate specific components of COSPs to particular outcomes within the logic model.  We need to articulate what it is about specific participant characteristics that impact either participation in services or outcomes.  Brian asked if this needed to be done as a multi-site study, and Greg responded that this is a general process that should apply to all good research.  Based on these reviews, we will make the layout and format of logic model design clearer and more readable.  The work of responding to many of the other critiques will be assigned to the Logistics Committee later.

Joe Sonnenfeld discussed issues of the analysis plan.  The CC is responsible for seeing that the multi-site analysis plans are in place to insure that good data will be supplied for analysis.  The reviewers said our plans are general, and that there is a real need to have a research plan in place a priori, to have a valid study.  We are dealing with input from diverse perspectives: site analysts, consumers, providers, policymakers, and reviewers, which presents a real challenge from a research design perspective.  The general features are program-theory guided.  The strategy reflects the logic model, and reflects variation in program characteristics.

Validity is imperative at an individual level and opportunities to be led astray need to be considered.  Some aspects to be considered:

·        Program context effects

·        Variation in participant characteristics

·        Differing perspectives on when it is reasonable to pool data

·        Variation in follow-up intervals

·        Integrating cost components

Joe reminded us that the cost study is not a separate study, but part of the overall research.

An overview of planned analyses was presented, including:

·        A Hierarchy of cross-site analyses

·        Pooled cross-site data

·        COSP clusters

·        Site by site

·        Secondary clusters, subgroups

The sequence of analyses includes the definition and operationalization of the CP variables and scales.  Jean said we need to put together a program theory that says these are legitimate clusters, as this will help in the understanding of COSPs.  There may be regional differences.   Many have said it is not what you do, but what is done in the company of other consumers, that has the greatest impact.

Yvette says that the political atmosphere will impact immature vs. mature Mental Health Systems.

Matt explained that one of the difficulties of this project is looking at all of the possibilities.  Statistically speaking, while we think of this as a big study, there is only so much variation we can consider.  There are a lot of ideas people will toss into the arena, as not all analysis is statistical analysis.  A flow chart of the cross-site analysis was discussed.

Data problems and what to do about missing data, extreme, out of range values, non-normal distributions, lack of variability, and other issues were considered.  R.O.W. will make determinations based on the distribution of data.  In respect as to what to do about missing data: assess the extent and location, cross-site analyses, identify possible causes and ways to minimize, proposed imputation strategies.  We will get feedback from range and other consistency checks.

During the preliminary methodological analyses, we will look at:

·        Did the randomization work within sites?

·        The need to look at data early on, especially covariates, (demographics) outcome variables, and scales.

·        Examine cross-site variation within clusters, as well as overall.

·        Data reduction: Identification of candidate scales/dimensions, logic model and other considerations, statistical approaches.

·        Inter-rater reliability and internal consistency reliability.

·        Validity issues, concurrent within CP, site specific instruments, COSP/FACIT rating scales.

·        Attrition analyses: overall characteristics of those lost to follow-up, ex-examine non-equivalence in followed participants.

Other analyses discussed:

Descriptive population analyses: primary focus of baseline analyses, basic demographics (overall, by site, by randomization, etc.).   Also, control variables for inclusions in models of study outcomes, methods are similar to pilot analyses (frequency/%, simple descriptives), as well as additional analyses.

Outcome analyses: Planning can continue until cleaning of follow-up data is complete, primary analyses as randomized, secondary analyses by level of participation: CP or other data, self-selection of level of participation, difference in program models and implementation.

Pooling and clustering: conduct primary analyses on both pooled and clusters, examine cross-site variation (participant characteristics, variance, outcome patterns), avoid aggregation bias.

Statistical modeling approaches: Build on predictions in logic model.  Statistical methods: regression based approaches, multilevel modeling, longitudinal, after primary, compare alternative models, possibilities depend on state of cross-site data.

Crystal asked if where you have covariate, do you have time varying covariate?  Joe responded that it depended upon the model selected.  Rise added they might get some input from the MIMH biostatician.

Zahira objected to the level of complexity to this presentation.  She said that the CAP was supposed to help in the decision making, but this is over the heads of those with no research background.  Joe R. also, gave his perspective on the difficulty of the presentation and asked for some background as to the reason behind this presentation.

Jean said this entire meeting is an introduction to each site about what kind of homework they need to do.  There will be no votes taken today – all of the ideas and needs identified will need to be worked on over the next year.  Betsy suggested that research terms be described on the website, sort of a Research 101 for those unfamiliar with the terms and concepts discussed.

 Diane said that it would have been helpful if copies of the presentation had been available so people could have made scribble notes on the side as to just what things mean. Yvette, also, gave her perspective about the difficulty in understanding the presentation, and asked that it be brought down a few notches.

Tom clarified that this is the CC’s job, and that this needs to be done.  It will be a learning experience for us all.

Joe continued with some of the potential problems and issues, including:

·        Crossover (participation in COSP by individuals assigned to TMHS alone as well as nonparticipation in COSP

·        Loss to follow-up

·        Beyond the multisite pooling – cluster-specific or other cross-site analyses (COSP models, Non-CP instrumentation, cost study)

Update on power analysis (since December 1999): Joe presented the power analysis based on the projected number of participants per site.  The current projected n is 1,130.

Jean added that the CAP had a long discussion about the concerns of each site on how restrictive the specifications are, and how it may impact recruitment.  Matt said there is a difference in recruitment which makes our numbers go down, however, if the restrictions are lessened, it will impact the effect side.  To maintain power, there is a need for tracking and monitoring recruitment, participation, cross-over, attrition, and loss to follow-up.

The reviewers’ comments included the possible need to have three groups, adding COSP only.  They also thought we might consider getting diagnosis before enrollment, as obtaining this afterwards may result in the loss of some respondents, as they might not qualify as SMI.

Matt commented that the Common Ingredients is a real contribution to the field.

Rise was asked to join the CAP meeting to explain some of the principles discussed – Betsy asked for this to be left to the researchers as it takes years to learn statistical principles.

After break, a discussion was held.

Greg said 1) that no one responded to Betsy’s suggestion that some of the researchers attend the CAP meeting, and 2) he thought the CC did an excellent job of bringing a very difficult discussion together.  He said the Federal Reps and CC want to maintain a uniform research plan, but this may compromise aspects at the site level.  He asked how should they think about these trade-offs, such as having a much lower number, and how this should be handled at the site level.

Jean said any change to what is a dose of COSP requires a 2/3 vote to even open it up for consideration.  Other procedural processes are spelled out in the logistics manual.

Matt said that it seems to him that the strategy that has been employed overall leaves their options open, as we can pool across all sites, and can talk about commonalties, etc.  It is in the preserving our options category.  In the recognition of realities of cross-site studies of this nature, have tried to be sensitive – it is difficult.  There are no optimizing issues as they address issues of numbers, etc.  Reasonable people can come to different places on these issues.

Pat asked if we have to have uniformity of 27 procedures across all sites, and may cost them up to 30 percent of potential clients, is the trade-off reasonable?  You can generalize, but we will lose a number of participants.

Matt responded that there has not been a COSP+TMHS, so we have to estimate these samples, and we will find out through early analyses if they have projected correctly.  Losing 30% of the participants would be devastating to the study.  Crystal reiterated things that Matt said, that its not that we want to take these things into account now, we have always planned on using these processes.

Yvette is confused about some of the choices one makes, as uniformity across sites will impact her site.  If everything is done uniformly, it will drastically impact her program.

Matt recognizes that at least two sites will have different standards based on the population that is targeted, so we can then maybe do clusters.  We can look at six programs in one way and the other two site programs in a different way.  We are not going to fundamentally change the programs.  Jean added that the Logistics Committee agreed on the checklist, and that is what it is based on.  To change a program requires a 2/3 vote.  We have standardized and controlled procedures guided by the logistic manual to remove the wiggle room.  There are some things that cannot be changed because of the specifications from the GFA, so there will be some variability.

Betsy talked about having more than one hypotheses, maybe having a primary, secondary and tertiary.  When you set these a priori you get some power that you wouldn’t get otherwise.

Lorraine invited the PIs and the CC to the CAP meeting so that they could learn what things might impact them.

A motion was made by Greg to switch the CAP and the research subcommittee order for Monday.  No objections so motion passed.  The next motion was to allow researchers at the CAP meeting.  It was agree to have Rise and Greg at the CAP meeting.  Joe would like to understand more about the a priori “thingamajiggy” to see how it would impact the study.

Sally R. expressed some serious concerns about the variation in the cost study.

Matt said there are several pieces to the pie:

·        Cost inputs (what is it that these things cost)

·        Utilization of particular types of programs, with reduced costs to health services, housing, etc. included in the CP.  Being collected at all sites.

·        Procedures, or program activities, imbedded in the CP.

Matt wants to highlight that in the same way we model empowerment, and says we can also use utilization in CP.  Sally is still concerned about calculating the cost inputs.  This discussion was tabled until the following day.

Brian commented that the reviewers’ comments to the plan highlighted the issue of comparability across the sites.  He has found many differences between Iowa and Missouri, and this makes him worry about the differences across the other States.  Tom will address this in the CI meeting on Tuesday.

Tom recognized Sally and Joe Rogers for the rewards they recently received.

The selection of Year 3 face-to-face meeting sites was discussed.  The September 2000 meeting will be held in Washington, D.C.  Site locations were discussed, and each site talked about the highlights in their sites.  It was voted on to have the April 2001 meeting in Florida.  There will be logistic and research committee meetings on an as needed basis.