Preface |
|
xiii | |
What Is New in the Third Edition |
|
xiv | |
About the Author |
|
xvi | |
1 Program Evaluation: A Prelude |
|
3 | (36) |
|
A Reader's Guide to Chapter 1 |
|
|
3 | (1) |
|
What Is Program Evaluation? |
|
|
4 | (3) |
|
The Program or Intervention |
|
|
4 | (1) |
|
Program Objectives and Outcomes |
|
|
5 | (1) |
|
|
5 | (1) |
|
|
6 | (1) |
|
|
6 | (1) |
|
|
6 | (1) |
|
|
7 | (1) |
|
|
7 | (8) |
|
Evaluation Questions and Hypotheses |
|
|
8 | (1) |
|
Evidence of Merit: Effectiveness, Quality, Value |
|
|
9 | (1) |
|
|
10 | (1) |
|
Selecting Participants for the Evaluation |
|
|
11 | (1) |
|
Collecting Data on Program Merit |
|
|
11 | (1) |
|
Managing Data So That It Can Be Analyzed |
|
|
12 | (1) |
|
Analyzing Data to Decide on Program Merit |
|
|
12 | (1) |
|
Reporting on Effectiveness, Quality, and Value |
|
|
12 | (3) |
|
|
15 | (1) |
|
Baseline Data, Formative Evaluation, and Process Evaluation |
|
|
15 | (4) |
|
|
15 | (1) |
|
Interim Data and Formative Evaluation |
|
|
16 | (2) |
|
Process or Implementation Evaluation |
|
|
18 | (1) |
|
|
19 | (1) |
|
|
19 | (1) |
|
|
20 | (2) |
|
Participatory and Community-Based Evaluation |
|
|
22 | (2) |
|
Evaluation Frameworks and Models |
|
|
24 | (6) |
|
The PRECEDE-PROCEED Framework |
|
|
24 | (2) |
|
|
26 | (1) |
|
The Centers for Disease Control's Framework for Planning and Implementing Practical Program Evaluation |
|
|
26 | (1) |
|
|
27 | (2) |
|
Right-to-Left Logic Model |
|
|
29 | (1) |
|
Left-to-Right Logic Model |
|
|
30 | (1) |
|
Evaluation Reports Online |
|
|
30 | (3) |
|
Summary and Transition to the Next Chapter on Evaluation Questions and Evidence of Program Merit |
|
|
33 | (1) |
|
|
34 | (2) |
|
References and Suggested Readings |
|
|
36 | (1) |
|
|
36 | (3) |
2 Evaluation Questions and Evidence of Merit |
|
39 | (28) |
|
A Reader's Guide to Chapter 2 |
|
|
39 | (1) |
|
Evaluation Questions and Hypotheses |
|
|
40 | (7) |
|
Evaluation Questions: Program Goals and Objectives |
|
|
40 | (3) |
|
Evaluation Questions: Participants |
|
|
43 | (1) |
|
Evaluation Questions: Program Characteristics |
|
|
44 | (1) |
|
Evaluation Questions: Financial Costs |
|
|
44 | (2) |
|
Evaluation Questions: The Program's Environment |
|
|
46 | (1) |
|
|
47 | (11) |
|
|
49 | (1) |
|
|
49 | (3) |
|
Evidence From Expert Consultation: Professionals, Consumers, Community Groups |
|
|
52 | (3) |
|
Evidence From Existing Data and Large Databases ("Big Data") |
|
|
55 | (2) |
|
Evidence From the Research Literature |
|
|
57 | (1) |
|
When to Decide on Evidence |
|
|
58 | (2) |
|
Program Evaluation and Economics |
|
|
60 | (1) |
|
The QEV Report: Questions, Evidence, Variables |
|
|
61 | (2) |
|
Summary and Transition to the Next Chapter on Designing Program Evaluations |
|
|
63 | (1) |
|
|
64 | (1) |
|
References and Suggested Readings |
|
|
64 | (3) |
3 Designing Program Evaluations |
|
67 | (34) |
|
A Reader's Guide to Chapter 3 |
|
|
67 | (1) |
|
Evaluation Design: Creating the Structure |
|
|
68 | (2) |
|
|
70 | (11) |
|
The Randomized Controlled Trial or RCT |
|
|
70 | (5) |
|
|
72 | (1) |
|
|
73 | (2) |
|
|
75 | (2) |
|
|
77 | (4) |
|
|
77 | (1) |
|
|
78 | (1) |
|
Improving on Chance: Stratifying and Blocking |
|
|
79 | (1) |
|
|
80 | (1) |
|
Nonrandomized Controlled Trials |
|
|
81 | (2) |
|
|
81 | (1) |
|
The Problem of Incomparable Participants: Statistical Methods Like ANCOVA to the Rescue |
|
|
82 | (1) |
|
|
83 | (4) |
|
|
83 | (2) |
|
|
85 | (1) |
|
|
85 | (1) |
|
A Note on Pretest-Posttest Only or Self-Controlled Designs |
|
|
86 | (1) |
|
Comparative Effectiveness Research and Evaluation |
|
|
87 | (3) |
|
Commonly Used Evaluation Designs |
|
|
90 | (1) |
|
Internal and External Validity |
|
|
91 | (3) |
|
Internal Validity Is Threatened |
|
|
92 | (1) |
|
External Validity Is Threatened |
|
|
93 | (1) |
|
Summary and Transition to the Next Chapter on Sampling |
|
|
94 | (1) |
|
|
94 | (2) |
|
References and Suggested Readings |
|
|
96 | (5) |
4 Sampling |
|
101 | (18) |
|
A Reader's Guide to Chapter 4 |
|
|
101 | (1) |
|
|
101 | (1) |
|
|
102 | (1) |
|
Inclusion and Exclusion Criteria or Eligibility |
|
|
103 | (1) |
|
|
104 | (6) |
|
|
105 | (1) |
|
Random Selection and Random Assignment |
|
|
105 | (2) |
|
|
107 | (1) |
|
|
107 | (2) |
|
|
109 | (1) |
|
Nonprobability or Convenience Sampling |
|
|
110 | (1) |
|
|
110 | (1) |
|
|
111 | (1) |
|
Power Analysis and Alpha and Beta Errors |
|
|
111 | (3) |
|
|
114 | (1) |
|
Summary and Transition to the Next Chapter on Collecting Information |
|
|
115 | (1) |
|
|
116 | (1) |
|
References and Suggested Readings |
|
|
117 | (1) |
|
|
117 | (2) |
5 Collecting Information: The Right Data Sources |
|
119 | (28) |
|
A Reader's Guide to Chapter 5 |
|
|
119 | (1) |
|
Information Sources: What's the Question? |
|
|
119 | (2) |
|
Choosing Appropriate Data Sources |
|
|
121 | (2) |
|
Data Sources or Measures in Program Evaluation and Their Advantages and Disadvantages |
|
|
123 | (12) |
|
Self-Administered Surveys |
|
|
123 | (2) |
|
|
125 | (1) |
|
|
125 | (3) |
|
|
128 | (1) |
|
|
129 | (1) |
|
Computer-Assisted Interviews |
|
|
130 | (1) |
|
|
131 | (1) |
|
|
131 | (2) |
|
|
133 | (1) |
|
|
134 | (1) |
|
Guidelines for Reviewing the Literature |
|
|
135 | (7) |
|
|
136 | (1) |
|
Identify Inclusion and Exclusion Criteria |
|
|
136 | (1) |
|
Select the Relevant Literature |
|
|
137 | (1) |
|
Identify the Best Available Literature |
|
|
138 | (1) |
|
|
139 | (1) |
|
Consider the Non-Peer-Reviewed Literature |
|
|
139 | (3) |
|
Summary and Transition to the Next Chapter on Evaluation Measures |
|
|
142 | (1) |
|
|
142 | (2) |
|
References and Suggested Readings |
|
|
144 | (3) |
6 Evaluation Measures |
|
147 | (18) |
|
A Reader's Guide to Chapter 6 |
|
|
147 | (1) |
|
|
148 | (3) |
|
|
148 | (2) |
|
|
150 | (1) |
|
A Note on Language: Data Collection Terms |
|
|
151 | (1) |
|
Checklist for Creating a New Measure |
|
|
152 | (4) |
|
Checklist for Selecting an Already Existing Measure |
|
|
156 | (2) |
|
The Measurement Chart: Logical Connections |
|
|
158 | (3) |
|
Summary and Transition to the Next Chapter on Managing Evaluation Data |
|
|
161 | (1) |
|
|
162 | (1) |
|
References and Suggested Readings |
|
|
163 | (2) |
7 Managing Evaluation Data |
|
165 | (22) |
|
A Reader's Guide to Chapter 7 |
|
|
165 | (1) |
|
Managing Evaluation Data Management: The Road to Data Analysis |
|
|
165 | (2) |
|
Drafting an Analysis Plan |
|
|
167 | (2) |
|
Creating a Codebook or Data Dictionary |
|
|
169 | (5) |
|
Establishing Reliable Coding |
|
|
172 | (1) |
|
Measuring Agreement: The Kappa |
|
|
172 | (2) |
|
|
174 | (1) |
|
Searching for Missing Data |
|
|
175 | (5) |
|
What to Do When Participants Omit Information |
|
|
178 | (2) |
|
|
180 | (1) |
|
|
180 | (1) |
|
When Data Are in Need of Recoding |
|
|
181 | (1) |
|
Creating the Final Database for Analysis |
|
|
181 | (2) |
|
Storing and Archiving the Database |
|
|
183 | (1) |
|
Summary and Transition to the Next Chapter on Data Analysis |
|
|
183 | (1) |
|
|
183 | (2) |
|
References and Suggested Readings |
|
|
185 | (2) |
8 Analyzing Evaluation Data |
|
187 | (30) |
|
A Reader's Guide to Chapter 8 |
|
|
187 | (1) |
|
A Suitable Analysis: Starting With the Evaluation Questions |
|
|
188 | (1) |
|
Measurement Scales and Their Data |
|
|
188 | (3) |
|
|
188 | (1) |
|
|
189 | (1) |
|
|
190 | (1) |
|
Selecting a Method of Analysis |
|
|
191 | (3) |
|
Hypothesis Testing and p Values: Statistical Significance |
|
|
194 | (3) |
|
Guidelines for Hypothesis Testing, Statistical Significance, and p Values |
|
|
195 | (2) |
|
Clinical or Practical Significance: Using Confidence Intervals |
|
|
197 | (3) |
|
Establishing Clinical or Practical Significance |
|
|
198 | (2) |
|
|
200 | (4) |
|
Odds Ratios and Relative Risk |
|
|
201 | (3) |
|
Qualitative Evaluation Data: Content Analysis |
|
|
204 | (6) |
|
|
205 | (1) |
|
Learning the Contents of the Data |
|
|
206 | (1) |
|
Creating a Codebook or Data Dictionary |
|
|
207 | (2) |
|
Entering and Cleaning the Data |
|
|
209 | (1) |
|
|
209 | (1) |
|
|
210 | (2) |
|
Summary and Transition to the Next Chapter on Evaluation Reports |
|
|
212 | (1) |
|
|
212 | (2) |
|
References and Suggested Readings |
|
|
214 | (3) |
9 Evaluation Reports |
|
217 | (45) |
|
A Reader's Guide to Chapter 9 |
|
|
217 | (1) |
|
The Written Evaluation Report |
|
|
218 | (14) |
|
Composition of the Report |
|
|
220 | (8) |
|
|
220 | (1) |
|
|
221 | (1) |
|
|
222 | (5) |
|
Conclusions or Discussion |
|
|
227 | (1) |
|
|
228 | (1) |
|
|
228 | (2) |
|
|
230 | (2) |
|
Reviewing the Report for Quality and Ethics |
|
|
232 | (1) |
|
|
232 | (9) |
|
Recommendations for Slide Presentations |
|
|
235 | (6) |
|
|
241 | (1) |
|
|
242 | (1) |
|
Evaluations That Need Institutional Review or Ethics Board Approval |
|
|
242 | (1) |
|
Evaluations That Are Exempt From IRB Approval |
|
|
243 | (6) |
|
|
244 | (1) |
|
|
245 | (4) |
|
The Internet and Ethical Evaluations |
|
|
249 | (2) |
|
Communication Between the Evaluator and the Participant |
|
|
249 | (1) |
|
Communication Between the Participant and the Website |
|
|
249 | (1) |
|
Communication Between the Website and the Evaluator |
|
|
250 | (1) |
|
|
250 | (1) |
|
Sample Questionnaire: Maintaining Ethically Sound Online Data Collection |
|
|
251 | (2) |
|
Example: Consent Form for an Online Survey |
|
|
253 | (1) |
|
|
254 | (1) |
|
|
255 | (6) |
|
|
261 | (1) |
Answers to Exercises |
|
262 | (11) |
Index |
|
273 | |