 |
 |
Targeting Excellence - Evaluation Outcomes
|
|
 |
 |
 |
The model described in this manual is one of several evaluation models. This performance based model is directed at understanding the effectiveness of programs through the measurement of outcomes and impacts. In this model, program or service evaluation is outcome-driven, where the evaluation attempts to measure the effect of the program or service on its participants and their environment.
The field of evaluation is broad, though generally most evaluation designs can be labeled as fitting into one of four types of evaluations: formative; process; outcome; and impact. These types of evaluation are described here in order to give a sense of the range of evaluation design. Though the evaluation model described in this manual draws on elements of each of the evaluation categories, it lies primarily within the outcome and impact evaluation categories.
Formative evaluation
Process evaluation
Outcome evaluation
Impact evaluation
It is important to remember that evaluation is an evolving process: none of the evaluation types stands completely on its own, information learned in one type of evaluation informs other evaluation designs, and information gathered in one stage of an evaluation helps to determine information needs in other stages. One can simultaneously conduct an outcome evaluation and a process evaluation, given appropriate resources. For example, the evaluation model described here is primarily directed at measuring outcomes. However, process information can easily be collected and analyzed during the evaluation in order to facilitate program improvement.
There are many good reasons for evaluating programs and services. However, when programs and services are evaluated it is often necessary to collect very specific data from individuals in order to determine whether their knowledge, attitudes or behaviors have changed as a result of the educational program, clinical intervention or other provided service.
Program evaluators should be guided by a deep respect for human dignity and personal rights.
Program evaluators often want to link evidence of changes to certain characteristics of clients or participants. A prevention education specialist might want to know if single parents benefited more from a parenting program than two-parent families. A therapist might want to know if a history of childhood abuse changed the effectiveness of a certain kind of counseling program for batterers.
To answer questions like these, individual level data must be collected. No evaluation should proceed with any data collection without a clear understanding of "informed consent" and clear guidelines on maintaining confidentiality of information.
What is Informed consent?
Informed consent requires that in advance of a proposed study or evaluation, the researcher/evaluator obtain permission from the participants to gather data from them.
Informed consent is obtained after the participants have learned about the type of project, what kind of information will be collected, what their role will be, and how the information will be used, protected, analyzed, and reported.
Informed consent is generally secured in one of two ways for program evaluations. (Other kinds of research studies, particularly medical studies or studies where disclosed information could harm the individual, may require more stringent consent procedures).
Active Informed Consent
The evaluator or researcher obtains a signed statement that the individual is willing to participate in the evaluation.
Passive Informed Consent
Descriptions of the study or evaluation are provided to all participants or to the guardians of intended participants (for studies involving minors) and the individuals are asked to notify the researcher or evaluator of they do not wish to participate. That is, the participant returns the forms only if they do not want to participate.
Confidentiality of Data
Assuming that participants have given informed consent, either actively or passively, the researcher/evaluator must still devise numerous safeguards to protect both the appearance and the reality of confidentiality. Thus a research/evaluation project might use anonymous completion of forms, use separate identification and research codes, report data in aggregate form only, etc.
If you have questions about informed consent or confidentiality issues, please contact the Family Violence and Trauma Project, Department of Psychiatry, F. Edward Herbert School of Medicine, Uniformed Services University of the Health Sciences (USUHS), Dr. James E. McCarroll, Bethesda, MD.
1. Determine the outcome or outcomes against which the program should be evaluated.
2. Determine the questions to be asked in order to know whether the program has achieved the outcome.
3. Determine the program outputs which are necessary to answer the questions. These should be measurable indicators of program success which are necessary to reach the outcome.
4. Measure whether the program outputs have been achieved and measure the outcomes of the activities.
5. It is important to know for what purpose the data will be used before embarking on data gathering. It is also important for program improvement to gather data that gets at the process of how the planned activities achieve the specified program output.
6. Gather the data and begin analysis. Data should be analyzed in relation to the program outputs.
7. Write the report. The report should demonstrate that the program outputs were met, and therefore the overall outcome was achieved.
General Overview
Data collection is no longer the purview of administrators or researchers. Increasingly, it is the clinicians responsibility to collect the data in addition to providing the service. At the most basic level the push for improved data collection is driven by the reality of cost containment in the era of managed care. Aside from the economic realities associated with "right-sizing," standardized and consistent data collection can lead to overall improvement in the delivery of services by testing long held but little studied clinical assumptions about the efficacy of various prevention and treatment strategies. Instead of being a source of "extra work," and confusion to the clinician, well organized data collection methods can be a servant to the clinician and not his/her master. In fact, data collection itself serves many masters. The administrator is interested in one type of data so that he or she may better predict and plan for the needs of the organization and its customers (clients and military commands). The clinician can use data to support and guide prevention, evaluation, and treatment objectives. In the age of managed care, providing evidence for program efficacy has become "part of the job" for most clinicians. This has often involved a change from the clinician seeing him/herself strictly in terms of a care-provider to that of a researcher or "numbers collector" as well. The necessity of collecting information pertaining to what the clinician does every day should be viewed as an exciting opportunity to see the forest from the trees.
The task of this seminar is to explore the best data collection methodologies. These methods should be practical and pertinent to the actual field experience and have a direct bearing on program decisions. In this regard the data collection methods should be:
a) Standard: Change for change sake should be avoided. Forms should be not only familiar to the clinicians and but any modifications kept to a minimum. Standard also pertains to standardized. Scales and questionnaires used should be those that have been field tested and proven. This quality not only makes the test more practical for clinical use but allows the data to be compared with a larger body of literature in related areas.
b) Significant: The reliability, validity, and sensitivity and specificity of the measures used should be established whenever possible.
c) Selective: The instrument must be appropriate to: the clinical situation, the training level/experience of the clinician, and the desired use of the data gathered.
d) Long and detailed questionnaires that may be powerful tools in some research protocols will likely prove absolutely useless to the clinician and the administrator hoping to gain information about the services that the organization provides because of the low probability that these forms will be used by the clinician or completed by the clients.
Review of the current procedures used to gather pertinent information needs to be completed in order to best assess what methods might best be used to collect data. Since ultimately the implementation of any research or outcome study is based on the consistent use of standardized programs, evaluations, and treatments across numerous sites, acceptance by the care-providers working in the field and the clerical staff supporting them are key considerations. Three related variables are:
a) Time Constraints: the amount time allotted to perform specific types of commonly rendered services.
b) Resource constraints: number of personnel, training level, financial resources available.
c) Caseload: number and type of programs or cases that are active.
Any recommended modifications of the process could then be made based on the "average" or expectable site conditions and types of program objectives.
There needs to be a natural link between the major overall goals of the organization, the critical success factors that are associated with each of those goals, the programs designed to achieve those success factors, the indicators that concretely measure those success factors, and the instruments/outcomes used to make those measurements. The routine outcome of interest is commonly simply whether or not the specific index event, e.g., spouse abuse or child abuse, recurs. However, beyond the measurement of simple recidivism, there are several other levels on which the impact of care or a specific prevention program may be measured. But in order for outcomes on these levels to be assessed after an intervention, they must first be considered and assessed at baseline, i.e. prior to the intervention. Examples of intervention effects at different levels are:
| Command Level |
| Supervisory perception of overall performance change of the service member(SM): |
|
| Family Level |
|
| Individual Level |
|
Emergent: The clinician's time and energies will be focused almost exclusively on trying to obtain sufficient basic information about the case, sometimes in an emergency room setting, in order to be able to make a judgment regarding the safety of the family member at risk and them make the appropriate disposition. In this setting, using detailed inventories and questionnaires to gather adjunctive information may not be necessary, possible, or productive. Yet, even in this scenario, it still important to have standardized demographic information, descriptions of the index event, and some assessment of common associated conditions that may aide in the initial formulation and recommendations. These forms may already be in use and will require little modifications.
Non-Emergent
Child Abuse Evaluations -- Spouse Abuse Evaluations
CAGE Questions
Questions re: suidicidality; homicidally; major affective or psychotic symptoms risk factors for violence (e.g., weapons in the home, previous violent behavior, etc.- may already be part of standard form.
Child Abuse Potential Scale (CAP)
Inventory of Marital Satisfaction
Family Environment Scale (FES)
CAGE Questions
Inventory of Marital Satisfaction(IMS)
Family Environment Scale (FES)
If children: Child Abuse Potential Scale (CAP) : Short Screening for developmental abnormalities
Denver Developmental
Child Abuse Potential Scale (CAP)
Child Behavior Checklist
Conflict Tactics Scale
Marital Satisfaction Inventory (MSI)
Michigan Alcohol Screening Test (MAST)
Child Abuse Potential Scale (CAP)
Child Behavior Checklist
Quality of Life Inventory
|
|
||
|
_________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ |
 Program/CLINICAL
INTERVENTION Character-istics |
 Outcomes |
 Army Outcomes |
| _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ |
IMMEDIATE _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ INTERMEDIATE _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ LONG-TERM _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ _________________________ |
|
Qualitative research: normally applies to procedures that do not involve a statistical analysis. (Note: this does not apply when statistical techniques are applied to data gathered by qualitative methods.) All of these techniques can be applied as single data gathering strategies or combined with other qualitative or quantitative methods. The following are types of qualitative methods used in social science research.
Interviews: variations in standardization from unstructured and semi-structured (Morgan, 1988).
o informal interview: has no pre-determined format or structure to the questions to be asked. Question arise during a free-flowing discussion that can take place in a social or program environment.
o unstructured interview: used when the researcher knows very little about the topic and is learning about the topic as the interview proceeds and as subsequent participants are interviewed.
o open ended interview: structured around a series of predetermined questions used to gain knowledge about a person's feel
ings, perspectives, motivations, etc. Responses are in the participant's own words and based on personal experience. Because of its personal nature, this method allows respondents to have questions clarified and to give in-depth responses.o semi-structured interview: used when the researcher knows most of the questions, but cannot predict the answers. The questions should elicit discussion and not be close-ended.
Informal or participant observation: can be used to generate descriptive information regarding a project environment, participant activities, and behavior. Observers must be trained to recognized and interpret significant occurrences. Participant observation tends to be done in a setting in which there is something immediately available to observe. Often used for observations of behaviors (Morse & Field, 1995).
Focus groups: are a mainstay of marketing research and the techniques were developed outside the social science arena. It is a form of group interviewing as a research technique. The hallmark of focus groups is the explicit use of the group interaction to produce data and insights that would be less accessible without the interaction found in a group. In marketing, they are almost always applied in conjunction with quantitative methods. Focus groups combine elements of individual interviews and participant observation. Focus groups are used as a preliminary data collection strategy that is later combined with quantitative methods to produce specific predictions, e.g., sales of a product. The general approach of a focus group is not on the alternation between the researcher's questions and the group response. The data are in the form of a transcript and represent the discussions of the group. The researcher takes the role of a moderator. The focus group might conduct a group discussion that would be like the lively conversation among friends or neighbors (Morgan, 1988).
Focus groups are useful for:
o orienting oneself to a new field
o generating hypotheses based on informants' insights
o evaluating different research sites or study populations
o developing interview schedules and questionnaires
o getting participants' interpretations of results from other studies (Morgan, 1988).
o participants that are highly involved in the subject
Advantages of focus groups:
o Opportunity to observe a large amount of interaction on a topic in a limited period of time.
o Used for observation of cognitions and attitudes.
o Requires less preparation than individual interviews.
o Savings in (1) group time and (2) analysis time (only one transcript per group as compared to individual interviews).
o Great ability to explore topics and generate hypotheses.
o Learning about participants experiences and perspectives.
o Useful to understand what participants think and why they think as they do.
Limitations:
o Groups are not usually in a naturalistic setting, but this has little relevance to clinical work where naturalistic groups do not exist.
o Willingness of participants to discuss subject matter.
o Control over interactions is in the hands of the participants rather than the researcher.
o Data collection can be chaotic and vary greatly from group-to-group.
o Not as many ideas produced in groups as in individual interviews.
o Participants may not know enough about the topic.
o Controversy and disagreement are hard to interpret.
o Participants may not feel comfortable about sharing their opinions.
Guidelines for selecting focus groups as a technique:
o how easily and actively would partici pants discuss the topic of interest?
o Can barriers to discussion be overcome?
o Participants should be relatively homogeneous (1) to reduce risk and (2) to increase the amount of interaction. (Heterogeneity may exist between the groups.)
Case studies: comprise a descriptive method which draws on both quantitative and qualitative methods of data collection to describe programs. They capture in-depth information about why a program works and enhance understanding of any unique aspect(s) that may be of special interest to program directors and funders (US Conference of Mayors, 1995).
Questionnaires/surveys: are structured to determine participant characteristics, level of knowledge, attitudes, and behaviors, participant satisfaction with the program, etc.
o the short-answer questionnaire: open-ended questions are used when some of the dimensions of the construct are known, but all possible responses cannot be anticipated. Open-ended questionnaires would be appropriate in a situation where the answers might be sensitive or cause embarrassment to the participant. It is constructed in the form of a short question with one or two lines for the answer (Morse & Field, 1995).
Project service record reviews: supply statistics on items such as number of calls to a hotline, requests for information or services, and any other services provided.
Observational techniques: observation adds breadth to research and answers some contextual questions, but it is time-consuming and expensive (Morse & Field, 1995).
Participant observation: the purpose is to observe participants in as natural a setting as possible. It is the essential data gathering method in ethnography. Observations focus on the content and include the reactions of individuals in the social setting. Forms of participant observation (Morse & Field, 1995):
o complete participation: the researcher's identity is unknown to the participants. It may be a violation of ethical standards and it may be difficult to be involved in work and be an observer at the same time.
o participant-as-observer: participants are aware of the researcher's purpose and researcher negotiates the amount of time devoted to work with a small amount of time set aside for writing notes of observations.
o observer-as-participant: the majority of the researcher's time is spent interviewing participants with a minimal amount of time in the work role. The main disadvantage is that the observer may be considered an outsider and not be confided in by the participants.
o complete observer: in this role, the observer is passive having no social interaction in the setting. This does not permit the observer to interview or clarify issues with the participants.
Histories: use interviewing techniques and can be considered a special form of interviewing (Morse & Field, 1995).
o life histories: used when a researcher wants to explore the history of an individual within a framework of time. It enables the researcher to explore an individual's current attitudes and beliefs in the context of the life span, giving consideration to decisions that were made at an earlier time and place. They may require 50 or more weekly interviews over the course of months (Morse & Field, 1995).
o oral histories: are directed toward specific events and may be obtained over a few hours in one to three short interviews (Morse & Field, 1995).
o diaries: may provide an intimate description of an individual's everyday life. They have been used by biographers to provide insight into an individual's life. Re
searchers can ask participants to keep diaries in a semi-structured or structured format, to record information in response to some event or experience. This technique has been successfully used in health research, but requires a high degree of cooperation among participants (Morse & Field, 1995).o photographs: can provide researchers with a visual image of a context. They can provide visual insights and knowledge of human conditions. They may stimulate a participant's recall of events and experiences leading to greater understanding of an event by the researcher. They allow the participant to tell the story in a spontaneous manner, particularly when the experience focuses on a compassionate dimension, such as in work on caring behaviors (Morse & Field, 1995).
o Letters and personal documents: are helpful in revealing relationships between persons who correspond. The analysis of suicide notes is an example of this type of research (Morse & Field, 1995).
1. Goals of prevention/treatment group
a. Strategy - e.g. prevention/treatment modality
b. Entry criteria
c. Outcome (prevention/treatment objectives)
2. Factors external to prevention/treatment process
a. How many?
b. Outreach?
c. Who reached?
d. Case finding?
e. Self-referral vs. mandated referral
f. Ancillary prevention/treatment
3. Standardized intake data collection
a. History
b. Demographics
4. Pre prevention/treatment measures - qualitative vs quantitative
a. Symptoms (anger, anxiety, depression, well-being)
b. Knowledge, attitudes, and behavior
5. Prevention/Treatment Programs
a. Method (type - clear description of process)
b. Length
c. Time of prevention/treatment per client (weekly/total)
d. Case consultation
e. Referral procedures
- - evaluations for psychology, psychiatry, pediatrics, etc.
f. Prevention/treatment fidelity
- - outside evaluation
- -peer evaluations
g. Compliance and noncompliance with prevention/treatment
h. Difficult case management
i. Drop outs
6. Post prevention/treatment measures - qualitative vs quantitative...
a. Symptoms (anger, anxiety, depression, well-being)
b. Knowledge, attitudes, and behavior
7. Follow-up - short-term and delayed
a. Compare pre and post measures
8. Evaluation methods - formal vs informal evaluations
a. Qualitative
1. Participant observation
2. Open ended interview
3. Focus groups
4. Case studies/presentation
b. Quantitative
1. Questionnaires/Surveys
2. Case record reviews
THIS MATERIAL WAS DEVELOPED FOR THE U.S. ARMY COMMUNITY AND FAMILY SUPPORT CENTER, FAMILY ADVOCACY PROGRAM BY STAFF OF THE FAMILY LIFE DEVELOPMENT CENTER IN COOPERATION WITH CORNELL COOPERATIVE EXTENSION, CORNELL UNIVERISTY, ITHACA, NY. THE MATERIAL IS BASED UPON WORK SUPPORTED BY THE EXTENSION SERVICE, U.S.. DEPARTMENT OF AGRICULTURE, UNDER SPECIAL PROJECT NUMBER 92-EXCA-3-0221.
THIS MATERIAL MAY BE REPRODUCED FOR FAP USE.
Characteristics of
Customers
Resources