Preface |
|
xv | |
Acknowledgments |
|
xxiii | |
About the Authors |
|
xxiv | |
Part 1 What Utilization-Focused Evaluation Is And Why It Matters |
|
1 | (82) |
|
Chapter 1 The What, Why, How, Who, When, and Where of Utilization-Focused Evaluation |
|
|
3 | (16) |
|
What Is Program Evaluation? |
|
|
4 | (1) |
|
Utilization-Focused Evaluation |
|
|
5 | (4) |
|
What Is Utilization-Focused Evaluation? |
|
|
5 | (1) |
|
|
6 | (1) |
|
|
6 | (1) |
|
|
7 | (1) |
|
|
8 | (1) |
|
|
8 | (1) |
|
Questioning as an Inquiry and Design Framework |
|
|
9 | (1) |
|
Intended Use by Intended Users |
|
|
10 | (3) |
|
Alternatives and Barriers to Utilization-Focused Evaluation |
|
|
13 | (2) |
|
Evaluation Use as a Value Proposition |
|
|
15 | (1) |
|
|
15 | (4) |
|
Chapter 2 The Essence of Utilization-Focused Evaluation Expressed as Minimum Specifications |
|
|
19 | (28) |
|
MIN SPEC 1. Honor the Personal Factor: Identify and Engage Primary Intended Users |
|
|
21 | (2) |
|
A Personal-Factor Exemplar: Engaging Educational Leaders in Evaluation |
|
|
22 | (1) |
|
MIN SPEC 2. Be Purpose-Driven: Focus on Priority Intended Uses |
|
|
23 | (4) |
|
A Purpose-Driven Exemplar: Evaluation for Improvement |
|
|
26 | (1) |
|
MIN SPEC 3. Facilitate Process Use: Be Active, Reactive, Interactive, and Adaptive in Engaging Users in All Aspects of the Evaluation |
|
|
27 | (7) |
|
The Impacts of Process Use |
|
|
32 | (2) |
|
Process Use as Capacity Building |
|
|
32 | (1) |
|
|
33 | (1) |
|
A Process Use Exemplar: Organization-Wide Learning |
|
|
33 | (1) |
|
MIN SPEC 4. Take a Full-Journey Stance: Focus on Use From Beginning to the End |
|
|
34 | (6) |
|
|
34 | (2) |
|
Pay Attention Along the Way |
|
|
36 | (1) |
|
Endings: Follow Up to Support Utilization |
|
|
36 | (1) |
|
Full-Journey Stance Premise |
|
|
36 | (4) |
|
Readiness: Baseline Associations With and Perceptions of Evaluation |
|
|
37 | (2) |
|
A Full-Journey Stance Exemplar: Developmental Evaluation as Maori |
|
|
39 | (1) |
|
MIN SPEC 5. Adapt to Context Changes: When the Context for an Evaluation Changes, the Evaluation May Have to Change |
|
|
40 | (4) |
|
A Contextual Adaptation Exemplar: The 2020 Global Health Pandemic |
|
|
42 | (2) |
|
|
44 | (3) |
|
Chapter 3 Utilization-Focused Evaluative Thinking |
|
|
47 | (18) |
|
|
47 | (2) |
|
|
49 | (2) |
|
The Logic of Evaluation-With a U-FE Twist |
|
|
50 | (1) |
|
Statistical Analysis as Rigorous Thinking |
|
|
51 | (3) |
|
Evaluative Thinking as a Form of Reasoned Argument |
|
|
54 | (1) |
|
Rigorous Evaluative Thinking |
|
|
55 | (3) |
|
Utilization-Focused Evaluative Thinking |
|
|
58 | (2) |
|
Intersection of the Personal and Professional |
|
|
60 | (1) |
|
|
61 | (4) |
|
Chapter 4 The High Stakes of Evaluation Use, Nonuse, and Misuse |
|
|
65 | (18) |
|
Evaluation for Accountability |
|
|
65 | (2) |
|
Organizational Culture and Evaluation Use |
|
|
66 | (1) |
|
|
67 | (1) |
|
|
68 | (1) |
|
|
69 | (1) |
|
|
69 | (2) |
|
The High Stakes of Evaluation Use |
|
|
71 | (1) |
|
Speak Truth to Power, but What Happens When Those With Power Aren't Listening |
|
|
71 | (1) |
|
The 2020 Global Coronavirus Pandemic: Political Lessons About Evaluation Use |
|
|
72 | (5) |
|
Clear and Credible Communications |
|
|
73 | (2) |
|
Pandemic Evaluation Contributions When Evaluations Were Used |
|
|
75 | (2) |
|
Timeliness of Evaluation Use |
|
|
77 | (1) |
|
Evaluation Use in the Information Age |
|
|
77 | (1) |
|
Use of Knowledge in Daily Life |
|
|
78 | (1) |
|
|
78 | (5) |
Part 2 Utilization-Focused Evaluation Principles |
|
83 | (220) |
|
Chapter 5 Guide to Utilization-Focused Evaluation Principles |
|
|
85 | (10) |
|
Principles Versus Rules and Recipes |
|
|
85 | (1) |
|
|
86 | (1) |
|
Utilization-Focused Evaluation Principles |
|
|
86 | (2) |
|
Overarching and Operating Principles |
|
|
87 | (1) |
|
Guide: Framework for Principles |
|
|
88 | (1) |
|
Guide: Applied to Utilization-Focused Evaluation |
|
|
88 | (2) |
|
Alignment of Personal and Professional Principles |
|
|
90 | (1) |
|
Principles-Focused Evaluation Applied to Utilization-Focused Evaluation |
|
|
90 | (1) |
|
|
91 | (4) |
|
Chapter 6 Contextualize: U-FE Operating Principle 1 |
|
|
95 | (18) |
|
A Situation Analysis Example |
|
|
96 | (2) |
|
The Situation Analysis Challenge |
|
|
98 | (2) |
|
|
100 | (1) |
|
Additional Factors to Assess |
|
|
100 | (2) |
|
Distinguishing Simple, Complicated and Complex Situations, and Interventions |
|
|
102 | (5) |
|
Simple Evaluation Challenges |
|
|
102 | (1) |
|
Complicated Evaluation Challenges |
|
|
103 | (1) |
|
Complex Evaluation Challenges |
|
|
103 | (4) |
|
Special Contextual Challenges |
|
|
107 | (3) |
|
|
110 | (3) |
|
Chapter 7 Personalize: U-FE Operating Principle 2 |
|
|
113 | (16) |
|
10 Tips for Identifying Primary Intended Users |
|
|
115 | (7) |
|
Examples of Working With Primary Intended Users |
|
|
121 | (1) |
|
Facilitation Challenges Working With Intended Users |
|
|
122 | (1) |
|
Diversions Away From Primary Intended Users |
|
|
122 | (5) |
|
|
127 | (2) |
|
Chapter 8 Customize: U-FE Operating Principle 3 |
|
|
129 | (26) |
|
Purpose Decisions Connected to Situation Analysis |
|
|
130 | (2) |
|
Intended Uses Menu: Six Alternative Evaluation Purposes |
|
|
132 | (14) |
|
Purpose 1. Judging Overall Merit, Worth, and Significance |
|
|
133 | (2) |
|
Purpose 2. Learning and Improvement |
|
|
135 | (2) |
|
Purpose 3. Accountability |
|
|
137 | (3) |
|
Purpose 4. Monitoring for Internal Management and Oversight. |
|
|
140 | (2) |
|
Purpose 5. Generating Knowledge and Lessons |
|
|
142 | (2) |
|
Purpose 6. Supporting Innovation and Adaptation: Developmental Evaluation |
|
|
144 | (2) |
|
Applying Purpose and Use Distinctions |
|
|
146 | (6) |
|
From Purpose to Data Collection |
|
|
148 | (2) |
|
Examples of How Purpose Affects Use |
|
|
150 | (2) |
|
Making Menu Selections: Connecting Decisions to Uses |
|
|
152 | (1) |
|
|
152 | (3) |
|
Chapter 9 Align: U-FE Operating Principle 4 |
|
|
155 | (18) |
|
Focusing Evaluation Questions |
|
|
156 | (8) |
|
Generate Actionable Questions |
|
|
157 | (2) |
|
Three Lessons to Guide Question Formulation |
|
|
159 | (2) |
|
Three U-FE Question Formulation Challenges |
|
|
161 | (1) |
|
Generate Actionable Questions |
|
|
162 | (1) |
|
Skill in Question Formulation |
|
|
162 | (1) |
|
The Focus in Utilization-Focused Evaluation |
|
|
162 | (2) |
|
Aligning Users' Questions With Design and Methods |
|
|
164 | (4) |
|
Controversy About U-FE Approach to Methods Decision-Making |
|
|
164 | (1) |
|
Principles of Methodological Engagement With Primary Intended Users |
|
|
165 | (3) |
|
Facilitation Challenges In Making Methods Decisions With Primary Intended Users |
|
|
168 | (1) |
|
|
168 | (1) |
|
Involving Intended Users in a Technical Decision |
|
|
168 | (1) |
|
|
169 | (4) |
|
Chapter 10 Prioritize: U-FE Operating Principle 5 |
|
|
173 | (22) |
|
From MIN SPEC to Principle |
|
|
173 | (1) |
|
Preparation and Readiness |
|
|
174 | (2) |
|
Simulate Use of Findings Before Data Collection |
|
|
176 | (3) |
|
An Ancient Practice Example |
|
|
177 | (2) |
|
Fabricating Data for Simulation |
|
|
179 | (1) |
|
Ongoing Attention to Use During Data Collection |
|
|
179 | (1) |
|
|
180 | (1) |
|
|
181 | (8) |
|
Four Steps to Generating Useful Findings |
|
|
181 | (1) |
|
Answering the What Question: Utilization-Focused Analysis With Intended Users |
|
|
182 | (1) |
|
|
183 | (2) |
|
|
185 | (1) |
|
Facilitating Interpretation |
|
|
186 | (1) |
|
|
187 | (2) |
|
Generating Recommendations |
|
|
189 | (1) |
|
|
189 | (1) |
|
Findings From Data Analysis |
|
|
190 | (1) |
|
|
190 | (1) |
|
|
190 | (1) |
|
|
190 | (1) |
|
|
190 | (5) |
|
Chapter 11 Engage Users: U-FE Operating Principle 6 |
|
|
195 | (18) |
|
Utilization-Focused Evaluation Facilitation |
|
|
195 | (2) |
|
Tips for Being Active-Reactive-Interactive-Adaptive |
|
|
197 | (7) |
|
Facilitation Tips Summary |
|
|
201 | (1) |
|
Regional Economic Development Agency Facilitation Example |
|
|
202 | (2) |
|
|
204 | (6) |
|
Five Operating Principles for Evaluation Facilitation |
|
|
204 | (5) |
|
Principles-Focused Evaluation Example |
|
|
209 | (1) |
|
|
210 | (3) |
|
Chapter 12 Strategize Process Use: U-FE Operating Principle 7 |
|
|
213 | (24) |
|
|
214 | (1) |
|
Five Process Use Facilitation Tasks |
|
|
215 | (5) |
|
Process Use for Organization-Wide Learning: An Illuminative Example |
|
|
220 | (1) |
|
Evaluation Models That Spotlight Process Use |
|
|
221 | (5) |
|
Evaluation Approaches That Center on User Engagement |
|
|
221 | (2) |
|
Principles for Use in Guiding Collaborative Approaches to Evaluation |
|
|
223 | (3) |
|
The Power of Group Processes |
|
|
226 | (1) |
|
Changed Evaluator Role: From Independent Judge to Social Change Facilitator |
|
|
227 | (1) |
|
Process Use as Intervention-Oriented Evaluation |
|
|
227 | (3) |
|
Intervention-Oriented Evaluation |
|
|
230 | (1) |
|
Challenges to Process Use |
|
|
230 | (2) |
|
Being Thoughtful and Ethical |
|
|
231 | (1) |
|
|
232 | (5) |
|
Chapter 13 Adapt: U-FE Operating Principle 8 |
|
|
237 | (16) |
|
|
237 | (1) |
|
Utilization-Focused Evaluation for Adaptive Management |
|
|
238 | (1) |
|
Six Examples of Evaluation Adaptation |
|
|
239 | (2) |
|
Adapting to a Critical Community Incident |
|
|
239 | (1) |
|
|
239 | (1) |
|
Program Participants Change |
|
|
240 | (1) |
|
|
240 | (1) |
|
Adapting to Change in Program Resources |
|
|
240 | (1) |
|
Adapting to COVID-19 Trauma |
|
|
240 | (1) |
|
Evaluator as Adaptive Problem Solver |
|
|
240 | (1) |
|
Adapting Evaluation Designs and Implementation |
|
|
241 | (2) |
|
|
242 | (1) |
|
Risk Assessment to Anticipate Adaptation |
|
|
242 | (1) |
|
Turbulent Environments and Changing Goals |
|
|
243 | (1) |
|
|
244 | (1) |
|
|
244 | (2) |
|
|
246 | (2) |
|
Creating the Conditions for Evaluation Adaptability |
|
|
248 | (1) |
|
|
249 | (4) |
|
Chapter 14 Mobilize Follow-Through: U-FE Operating Principle 9 |
|
|
253 | (16) |
|
Utilization-Focused Reporting |
|
|
254 | (1) |
|
Fear of Negative Findings |
|
|
255 | (2) |
|
Prospect Theory and Evaluation |
|
|
256 | (1) |
|
Generating Actionable Recommendations |
|
|
257 | (3) |
|
10 Guidelines for Generating Useful Recommendations |
|
|
258 | (2) |
|
Follow-Through: Completing The Endgame |
|
|
260 | (1) |
|
Follow-Through as an Evaluation Standard of Excellence |
|
|
260 | (1) |
|
Follow-Through in Practice: Endgame Examples |
|
|
261 | (3) |
|
The Challenge of Meeting Information Needs and Expectations |
|
|
262 | (1) |
|
Example of Follow-Through at an Organizational Level |
|
|
263 | (1) |
|
10 Follow-Through Lessons |
|
|
264 | (1) |
|
|
265 | (4) |
|
Chapter 15 Reflect and Learn: U-FE Operating Principle 10 |
|
|
269 | (16) |
|
|
270 | (1) |
|
Single-, Double-, and Triple-Loop Learning |
|
|
271 | (1) |
|
|
272 | (1) |
|
How Evaluators Influence Cultural Competence and Responsiveness |
|
|
273 | (1) |
|
The Utilization-Focused Evaluator's Role and Values |
|
|
274 | (2) |
|
Learning Over Time: Building Momentum for Use Across Evaluations |
|
|
276 | (1) |
|
The Utilization-Focused Evaluation Flywheel |
|
|
276 | (4) |
|
The Momentum of the Flywheel |
|
|
279 | (1) |
|
|
279 | (1) |
|
|
280 | (5) |
|
Chapter 16 Learning Breakthroughs in Applying Utilization-Focused Evaluation Principles |
|
|
285 | (18) |
|
Lessons Learned About Utilization-Focused Evaluation Principles |
|
|
285 | (11) |
|
|
296 | (3) |
|
Maintain Evaluation Use as the Quintessential Criterion of Evaluation Excellence |
|
|
299 | (1) |
|
|
300 | (3) |
Part 3 New, Emergent, And Urgent Directions For Utilization-Focused Evaluation |
|
303 | (70) |
|
Chapter 17 The High Stakes of Evaluative Thinking and Reality-Testing |
|
|
305 | (16) |
|
Evaluation in the Context of a Post-Truth World |
|
|
305 | (2) |
|
Original High Hopes for Evaluative Thinking |
|
|
307 | (1) |
|
Evaluation and Rationality |
|
|
307 | (1) |
|
Reality Check: Evaluations Largely Unused |
|
|
308 | (1) |
|
The High Stakes of Evaluative Thinking |
|
|
308 | (2) |
|
|
310 | (2) |
|
|
310 | (1) |
|
The Banality of Evaluation |
|
|
311 | (1) |
|
Thoughtlessness in a Post-Truth World |
|
|
312 | (3) |
|
|
312 | (3) |
|
|
315 | (4) |
|
Evaluation in the Epoch of the Anthropocene |
|
|
315 | (1) |
|
Toward a Balanced Assessment |
|
|
316 | (1) |
|
|
317 | (1) |
|
|
318 | (1) |
|
|
319 | (2) |
|
Chapter 18 Using Evaluation to Contribute to a More Equitable and Sustainable World |
|
|
321 | (24) |
|
|
322 | (1) |
|
Stake's Position on Evaluators' Values |
|
|
323 | (2) |
|
Morality and Ethics of Evaluation |
|
|
325 | (2) |
|
Whom Does an Evaluation-and an Evaluator-Serve? |
|
|
326 | (1) |
|
Contributing to Social Betterment as an Intended Use |
|
|
326 | (1) |
|
|
327 | (4) |
|
Equity: No One Left Behind |
|
|
328 | (1) |
|
Evaluation Commitments to Equity |
|
|
329 | (1) |
|
|
330 | (1) |
|
Contextualization (U-FE Principle 1): Equity Situation Analysis |
|
|
331 | (4) |
|
Equity Through a Social Justice Lens |
|
|
332 | (1) |
|
An Evaluation of Systemic Racism |
|
|
333 | (2) |
|
Equity-Focused Evaluation |
|
|
335 | (1) |
|
|
336 | (1) |
|
|
336 | (2) |
|
Contextualization: Sustainability Situation Analysis |
|
|
337 | (1) |
|
Responses to the Climate Emergency |
|
|
337 | (1) |
|
Sustainability as a Universal Evaluation Criterion |
|
|
338 | (1) |
|
|
339 | (1) |
|
Equity and Sustainability as Mutually Reinforcing Criteria |
|
|
339 | (2) |
|
Facilitating U-FE Consideration of the Criteria of Equity and Sustainability |
|
|
340 | (1) |
|
Utilization-Focused Evaluation Approach to Facilitating Criteria Priorities |
|
|
341 | (1) |
|
|
342 | (3) |
|
Chapter 19 Evaluation for Transformation |
|
|
345 | (18) |
|
Transformation Means "No One Left Behind" |
|
|
346 | (1) |
|
Transformation, Transformation, Transformation |
|
|
346 | (2) |
|
Contextualizing: Situation Analysis |
|
|
346 | (2) |
|
Evaluating Transformation |
|
|
348 | (5) |
|
Evaluate Transformation Fidelity |
|
|
348 | (5) |
|
Utilization-Focused Evaluation for Transformation Exemplar |
|
|
353 | (5) |
|
Principles-Focused Transformation and Evaluation |
|
|
355 | (2) |
|
Systems Thinking Integrated With Evaluative Thinking |
|
|
357 | (1) |
|
The World's First Blue Marble Evaluation |
|
|
357 | (1) |
|
Future Utilization-Focused Directions for Transforming Evaluation to Evaluate Transformation |
|
|
358 | (2) |
|
Evaluation for a Transformed Future |
|
|
359 | (1) |
|
|
360 | (1) |
|
|
360 | (3) |
|
Chapter 20 Utilization-Focused Evaluation for Transformation: Toward a More Thoughtful, Equitable, and Sustainable World |
|
|
363 | (10) |
|
Five Decades of Utilization-Focused Evaluation and Forward Into the Future |
|
|
368 | (3) |
|
The Evolution of Evaluation Practice |
|
|
368 | (1) |
|
The Evolution of Utilization-Focused Evaluation |
|
|
369 | (2) |
|
|
371 | (2) |
References |
|
373 | (16) |
Index |
|
389 | |