Introduction
The Fidelity Assessment/Common
Ingredients Tool (FACIT) was initially developed to assess the extent to which
consumer-operated services shared as set of common elements identified by
participating consumer-operated service programs (COSPs) in the COSP MRI.
During an extensive process, a set of common ingredients of
consumer-operated programs were identified, along with a mechanism to measure
the extent to which participating programs expressed the element.
During the first round of site visits (conducted 1999-2001), the
instrument was pilot-tested for use with COSP’s, and inter-rater reliability
was assessed to be reasonable for all but one item.
In addition, during the first set of site visits, the feasibility of
collecting the same kinds of information for traditional mental health services
was assessed by asking some site visit participants to rate traditional services
on the same scales. This
feasibility assessment suggested that it was possible to use the same measures,
with some very slight wording changes to make clear that the subject program was
not necessarily a COSP, to allow the COSP MRI to differentiate between COSP and
traditional mental health services using the FACIT.
In
round two of the site visits, which will begin in November, 2001 and continue
through July, 2002, site visitors will be asked to rate both the COSP and the
traditional services using the FACIT. This
document provides a protocol to be followed by site visitors in making initial
ratings of COSP and traditional services and then in conciliating these
responses. Each site visit will
feature at least two site visitors. Because
site visitors may, for one reason or another not see exactly the same things
during visits, they may make different ratings. After independently rating the program on each item, we will ask site
visitors to use a process of conciliation in which the site visitors will
discuss the item, their initial ratings on the program, evidence leading them to
make this rating, and then to come to consensus on a common rating which
expresses the final assessment of where a program stands on the rating. The steps involved in the initial rating, the conciliation process, and
sharing of data will be detailed in the following pages.
Initial
Rating
Raters
are asked to utilize all available information about a program in making a
determination of a rating. This
includes: responses of all persons interviewed in the course of the site visit,
all documentation provided by the programs under review, information collected
by previous site visitors, information gleaned during tours of the settings, and
information gathered during informal exchanges with program staff and members.
The ultimate purpose of collecting information of different types is to
engage in a type of triangulation, in which the site visitors are able to
generate more valid responses by collecting different kinds of data. There is considerable methodological theorizing to suggest
that this approach is more likely to provide an accurate overall assessment than
relying on a single source. Where
raters find considerable difference in the statements of different respondents
to the same questions, raters must use their best judgment in arriving at an
appropriate rating. Sometimes, for
example, where one respondent provides an answer but expresses uncertainty about
it while another provides a different answer but is quite certain about it, the
rater may decide to give the second answer more weight than the first. In certain rare instances, raters may be confronted with information that
would seem to have little relationship to the reality that the site visitor has
confronted. Site visitors should
delay making judgments about particular ratings until the end of the site visit,
at which point all the information is assembled.
The
suggested procedure for coming up with ratings involves: (1) assembling all
relevant information including interview notes, field notes, documentation
provided prior to the site visit, other documentary material (for example, the
proposal), and the previous site visit report; (2) review all relevant material,
highlighting information that can be used in particular ratings; (3) go through
each FACIT rating making a decision on the level assigned on the basis of the
totality of the evidence available. It is suggested that the rater reviews materials for each
program separately, in order to minimize the risk of a halo effect that may, for
example, lead one to ascribe the good or bad qualities of the traditional
program to the COSP, or vice versa. For
ratings which are difficult, it is suggested that the rater again review
evidence, and consider in some depth the wording associated with each level, in
an effort to rate the item in a manner which is internally consistent.
Raters
are expected to independently rate each
program on each element prior to discussing their ratings with other site
visitors. In the same way that it
is important for raters to avoid the halo effect between programs, it is also
important to make sure that within a program, there is not a halo effect. This is a natural tendency that raters have in making these kinds of
judgments.
The
independent rating of each program by raters will allow us to better understand
issues around interrater reliability of the instrument.
The Conciliation
Process
The process used to put the
impressions of the site visitors together is a process known as conciliation.
Rather than relying on simple averaging of responses from site visitors,
a process of sharing between site visitors with the intention of reaching a
joint decision on a particular rating will be carried out.
The process will involve the following steps:
1)
Review of
independent ratings
2)
If there is
a discrepancy, referring to the FACIT instrument to help understand the issues
intended to be rated under the specific item.
3)
Sharing
information used by each rater in arriving at a decision.
4)
Arriving at
a joint decision about the appropriate rating for the program on the particular
item.
Reporting Data to
Coordinating Center
Each
site visitor should make two
copies of the independent ratings, sending one to MIMH and one to Matt Johnsen
at University of Massachusetts Medical School. In addition, the site visit leader should make two additional copies of
the joint rating and send them to the same locations.
Special Problems
Traditional mental health services
Raters may run into particular problems applying
one or more of the rating items to the traditional services under study. Raters should be particularly sensitive to not ask the questions (or
respond to answers) in a way that traditional services will feel uncomfortable
with the answers that they are providing. While
many traditional programs display some features which might be considered
consumer-oriented or consumer-friendly, these qualities may not extend across
all of the areas covered by the FACIT, so we expect that there may be a wide
range of variability in the scores recorded.
In some cases, more than one traditional mental
health program has been identified in the community or region in which the COSP
is operating. The general
guidelines for the second round site visit are as follows:
1)
Where one or
two traditional mental health providers can be identified, complete a separate
FACIT for each one.
2)
Where more
than two traditional mental health providers can be identified, choose the one
or two programs that are most frequently utilized by COSP members.
3)
Where the
program covers the entire state, complete the FACIT, to the extent possible,
utilizing the state entity (i.e., the state DMH) as the traditional mental
health provider.
Information not available
Raters should, as a rule, attempt to rate each
program on each of the items of the FACIT. In some cases, raters may feel as though they do not have adequate
information in order to rate the program. There
may be several reasons for the lack of information, each of which might lead to
its own specific remedy:
1)
During the
site visit, the team neglected to collect certain information or ask certain
questions of one or more respondents: In this case, team members should attempt,
as possible and appropriate, to recontact the informant or the program to gather
additional information prior to the rating.
2)
The item is
not applicable to the program: In rare cases, the raters may feel that a
particular item is not applicable to a program with a particular model. In this case, it would be appropriate to make this notation in the area
in which the rating is called for, including a brief explanation of why the item
is not applicable.
3)
The team did
not collect any evidence related to the program because the quality that the
rating is being rated is not demonstrated within the program: This is distinct
from the previous rating – if in fact the quality is not in evidence, not
because it is not applicable, but because the program has directed no attention
toward a certain goal, this would be evidence that the program was not
displaying the quality, and hence would likely be judged at level 1. It is incumbent on the raters that they have ruled out the
previous two alternatives before assigning a level 1 (i.e. that they have asked
the appropriate questions, and that the item is applicable to the program).
Actual
rating would be in between two levels detailed in FACIT
There will also be times when the true rating for a
particular item would fall somewhere between two of the levels that are
described in the FACIT. After a
sufficient level of careful assessment as to whether the rating falls more
easily into one level than the other, if the rater is still in doubt, the
operating rule employed by the rater should be “When in doubt, assign the
lower of the two values.” The
highest levels of each item should be reserved for organizations that clearly
meet the criteria for that rating.
Substantial disagreement between
raters
In some rare instances, raters may have substantial
disagreement about the items that may appear to be irresolvable. In this situation, raters are instructed to attempt to conciliate the
rating to the point that each of the raters has expressed fully the rationale
for the disagreement. At the point at which it is clear that there is not going to
be a meeting of the minds on the rating, each rater is asked to write out
his/her justification for the rating, and both of these ratings (along with the
justification) will be sent to the coordinating center. Obviously, it will be most helpful for those who have participated in the
actual site visit to make final determinations about the ratings, rather than
parties that have not participated in the site visit, so raters are urged to
conciliate and to arrive at common agreements whenever possible.
5400
Arsenal Street