Archives   Calendar    Coordinating Center    Multisite Activities    
Project Description  COSP References  Study Sites    Search Our Site   
Table of Contents    Upcoming Meeting Agendas

______________________________________________

COSP Multi-Site Research Overview
Section 12: Quality Assurance Procedures in the COSP Multisite Study

Section 12:  

Rigorous checks and reporting mechanisms have been incorporated into multi-site activities and procedures to assure high quality research, including the means to assure reliability and validity of data collected from study sites.  Areas of Coordinating Center quality assurance monitoring include: (1) human subjects protections, (2) study site field readiness, (3) study site data collection progress, (4) data repository protection and error reduction efforts, and (5) study site interview standardization.

12.1          Human Subjects Protections

Every site must submit documentation of Human Subjects Protections, Internal Review Board approval(s), and the SAMHSA Certificate of Confidentiality to the Coordinating Center. These documents are kept on file at the CC.  In addition, the University of Missouri School of Medicine required IRB approval for the entire multi-site study.  Further, confidentiality oaths are administered to all study site personnel.

12.2          Study Site Field Readiness 

To monitor the readiness of study site ability to begin outcomes and cost data collection, the CC developed a COSP Site Checklist.  This Checklist was the basis of site presentations at the Face-to-Face Steering Committee Meeting in Washington D.C. March 12-14, 2000.  Following these presentations, the CC asked each site to submit hardcopies of the checklist and all support documentation by the end of the second quarter of Year 02.  The CC reviewed these materials for:  (1) IRB approval and related human subjects protections, (2) completion of interviewer training, conduct of debriefing sessions, and taping of practice interviews for CC review, (3) compliance with the Logistics Manual in all recruitment/induction/assignment activities, (4) and the introduction of bias in any materials and/or procedures developed by the study site.  The CC Principal Investigator contacted each study site Principal Investigator and any problems identified by the CC were discussed.

12.3     Study Site Data Collection Progress

Strict adherence to multi-site procedures by study sites is monitored by the Coordinating Center through quarterly reports, and progress per site in recruitment, assignment, data collection, and retention is tracked and reported to the federal representatives.  An Excel workbook was developed that tracks the quarterly quantitative data on study participants.  A template has been sent to each site for this purpose, and augments the Word document that is used to collect information on multi-site activities per site, such as meetings of the local Consumer Advisory Board, relations with traditional mental health providers, problems, and accomplishments.  The sites enter the data collected from their site specific tracking databases, and include the completed worksheet with their quarterly report submissions to the Coordinating Center.  This worksheet is then imported back into the Excel workbook, with an aggregate automatically calculated.  The aggregated totals are included in the Federal quarterly reports submitted by the Coordinating Center.  In addition to these efforts, the Principal Investigator of the Coordinating Center calls every study site Principal Investigator per quarter and discusses the above information.

12.4     Data Repository Protection and Error Reduction Efforts

 

In a multi site study there are a variety of sources of error.  As shown in Figure 1 each site starts with a common protocol and then develops a specific data entry system and database for storage.  Errors can be introduced at and of those three levels.  Typical errors would include recording, data entry, data formatting, and illogical data.

In the COSP study a variety of steps have been taken to reduce the sources of error.  While error can not be completely eliminated the fewer the potential sources or error the fewer errors we will have.

In the COSP project we have developed a common data entry system that precludes entering inconsistent data.  This entry system stores data into a database that has also been created to prohibit both null (empty) and out of range responses.  In addition to those structural changes, we enforce data verification by double entry.  That is, two different staff members must both blindly enter (without seeing what the other entered) the same data.  Disagreements between data entry staff it must be resolved before the record can be marked as completed.  By these methods we have reduced the sources of error by two thirds and greatly enhanced the quality of our information.

The Coordinating Center is strongly supportive of using computer-assisted interviews to collect research data in the COSP project.  However, as described below, the use of computer-assisted data collection increases the level of data verification effort required by the Coordinating Center.

The standard data checking approach the CC has taken to gathering the research information involves a variety of components and steps.

a.        Standard databases with range and validity checking for each individual item.

b.        Standard data entry system that automatically checks and enforces between item validity checking (e.g., If the answer on a question indicates that the next 6 questions are to be skipped all of those questions are automatically set to “not asked.”  Users are prevented from entering information in those fields).

c.  Double entry verification required on all assessments.

d.  Verification queries to look for illogical or divergent information.

For the development of the computer-assisted interviews, the Coordinating Center has provided the standard database (#1) for use.  However, at this point there is no way to be sure if any changes have been made to those tables while they were incorporated into the interactive assessment.

Because the CC standard data entry program will not be used (#2), the CC must write a series of queries and procedures that check the various tables for appropriate skip logic.  Based on the amount of time needed to develop the entry program, these additional steps will also require a substantial programming effort.

Further, with computer-assisted interviews there will be no double entry (#3).  However, this may be less of an issue then it appears.  Double entry is designed to make sure that the correct information is transcribed from a printed form into the computer.  It does not check to make sure that the interviewer wrote down the correct information in the first place.  Because the entry is direct, there are no errors in transcription and therefore no real need for double entry.  Nonetheless, it would be useful to have an independent completion of the data forms based on a video taped recording of a specified percentage of the assessments.  This would enable the accuracy of the entered observation to be checked, clearly an important consideration.

From this analysis, item number 2 is the most serious threat to the comparability of the automated and paper based assessments and will take the most effort on our part to address.

12.5     Study Site Interview Standardization

Data quality can be compromised by biased questioning styles, improper clarification techniques, inaccurate recording of responses, questions being asked out of sequence, or failure to establish rapport with the respondent.  To achieve standardized quality both within the site, and across sites, the CC has established the following objectives: (1) that interviews are conducted in a manner reflecting high quality; and (2) that all interviews at all sites are conducted in the same way.

To achieve these objectives, several procedures have been developed.

a.       To monitor quality at all sites and assure standardization, a position for a multi-site interviewer coordinator was created at the Coordinator Center and charged with this responsibility. 

b.       To ensure standardization and quality of training, the interviewer coordinators from sites were trained by Coordinating Center staff during a "Train-the-Trainer Workshop" or individually by the multi-site interviewer coordinator.  This workshop utilized a training kit, developed by the Coordinating Center, which included: (1) an extensive training manual; (2) a pre-post test to administer to interviewers; (3) a 75-minute training video; (4) a set of slides that can be used to facilitate learning during a training workshop; and (5) a study guide for trainers.  The pre-post test demonstrated a specific level of quality and set of knowledge and skills.  Each interviewer completing the CC training received a certificate of completion.

c.       To ensure quality of individual interviewers across the sites, and that interviewers at each site were administering the protocol in a standardized manner, study sites are required to submit an audio tape by each interviewer conducting one interview per data wave with an individual who is not a study participant.  The Interviewer Coordinator will review these tapes to make certain that interviewers across all sites are asking questions in the same way.

d.       Each site is required to hold interviewer-debriefing sessions periodically to address issues of quality or other concerns among the interviewers, and to provide notes of these debriefing sessions to the Interviewer Coordinator that highlight persistent areas of concern.  These summaries will be collected by the Coordinating Center and compared across the sites to ensure similar levels of interviewing quality and to prevent site specific drift in standardization of protocol administration.

e.       To strategically communicate and discuss issues of interviewer quality at the study sites, an interviewer list-serv was developed.  Representatives from the sites are responsible for sharing information about their experience with interviewing and training either on the list-serv, or directly with the multi-site Interviewer Coordinator. 

f.        Weekly alerts are posted to the listserv by the multi-site interviewer coordinator, assuring that all sites are informed about changes made by the Steering Committee or the Coordinating Center regarding the Baseline Protocol, Follow-Up Protocol, Question-by-Question Guide, and interviewer procedures.  In addition, these updates are recorded in a database maintained by the multi-site interviewer coordinator and are accessible to all sites through the free drive.