Foreword |
|
ix | |
Preface |
|
xi | |
|
Part I Theories and Practical Applications of Al Risk Management |
|
|
|
1 Contemporary Machine Learning Risk Management |
|
|
3 | (30) |
|
A Snapshot of the Legal and Regulatory Landscape |
|
|
4 | (1) |
|
|
4 | (1) |
|
US Federal Laws and Regulations |
|
|
5 | (1) |
|
|
5 | (1) |
|
|
6 | (1) |
|
Federal Trade Commission Enforcement |
|
|
7 | (1) |
|
Authoritative Best Practices |
|
|
8 | (3) |
|
|
11 | (2) |
|
Cultural Competencies for Machine Learning Risk Management |
|
|
13 | (1) |
|
Organizational Accountability |
|
|
13 | (1) |
|
Culture of Effective Challenge |
|
|
14 | (1) |
|
Diverse and Experienced Teams |
|
|
15 | (1) |
|
Drinking Our Own Champagne |
|
|
15 | (1) |
|
Moving Fast and Breaking Things |
|
|
16 | (1) |
|
Organizational Processes for Machine Learning Risk Management |
|
|
16 | (1) |
|
Forecasting Failure Modes |
|
|
17 | (1) |
|
Model Risk Management Processes |
|
|
18 | (4) |
|
Beyond Model Risk Management |
|
|
22 | (5) |
|
Case Study: The Rise and Fall of Zillow's iBuying |
|
|
27 | (1) |
|
|
28 | (1) |
|
|
28 | (3) |
|
|
31 | (2) |
|
2 Interpretable and Explainable Machine Learning |
|
|
33 | (48) |
|
Important Ideas for Interpretability and Explainability |
|
|
34 | (5) |
|
|
39 | (1) |
|
|
39 | (5) |
|
|
44 | (3) |
|
An Ecosystem of Explainable Machine Learning Models |
|
|
47 | (3) |
|
|
50 | (1) |
|
Feature Attribution and Importance |
|
|
51 | (12) |
|
|
63 | (5) |
|
Plots of Model Performance |
|
|
68 | (3) |
|
|
71 | (1) |
|
Stubborn Difficulties of Post Hoc Explanation in Practice |
|
|
71 | (4) |
|
Pairing Explainable Models and Post Hoc Explanation |
|
|
75 | (2) |
|
Case Study: Graded by Algorithm |
|
|
77 | (3) |
|
|
80 | (1) |
|
3 Debugging Machine Learning Systems for Safety and Performance |
|
|
81 | (42) |
|
|
83 | (1) |
|
|
83 | (2) |
|
|
85 | (3) |
|
Model Specification for Real-World Outcomes |
|
|
88 | (3) |
|
|
91 | (1) |
|
|
92 | (1) |
|
Traditional Model Assessment |
|
|
93 | (2) |
|
Common Machine Learning Bugs |
|
|
95 | (8) |
|
|
103 | (4) |
|
|
107 | (3) |
|
|
110 | (2) |
|
|
112 | (2) |
|
|
114 | (1) |
|
|
114 | (2) |
|
|
116 | (4) |
|
Case Study: Death by Autonomous Vehicle |
|
|
120 | (1) |
|
|
120 | (1) |
|
An Unprepared Legal System |
|
|
120 | (1) |
|
|
121 | (1) |
|
|
122 | (1) |
|
4 Managing Bias in Machine Learning |
|
|
123 | (36) |
|
ISO and NIST Definitions for Bias |
|
|
126 | (1) |
|
|
126 | (1) |
|
|
126 | (1) |
|
Human Biases and Data Science Culture |
|
|
127 | (1) |
|
Legal Notions of ML Bias in the United States |
|
|
128 | (3) |
|
Who Tends to Experience Bias from ML Systems |
|
|
131 | (2) |
|
Harms That People Experience |
|
|
133 | (2) |
|
|
135 | (1) |
|
|
135 | (2) |
|
Traditional Approaches: Testing for Equivalent Outcomes |
|
|
137 | (4) |
|
A New Mindset: Testing for Equivalent Performance Quality |
|
|
141 | (2) |
|
On the Horizon: Tests for the Broader ML Ecosystem |
|
|
143 | (3) |
|
|
146 | (1) |
|
|
147 | (1) |
|
Technical Factors in Mitigating Bias |
|
|
148 | (1) |
|
The Scientific Method and Experimental Design |
|
|
148 | (1) |
|
Bias Mitigation Approaches |
|
|
149 | (4) |
|
Human Factors in Mitigating Bias |
|
|
153 | (3) |
|
Case Study: The Bias Bug Bounty |
|
|
156 | (2) |
|
|
158 | (1) |
|
5 Security for Machine Learning |
|
|
159 | (30) |
|
|
161 | (1) |
|
|
161 | (1) |
|
|
162 | (1) |
|
Best Practices for Data Scientists |
|
|
163 | (3) |
|
|
166 | (1) |
|
Integrity Attacks: Manipulated Machine Learning Outputs |
|
|
166 | (5) |
|
Confidentiality Attacks: Extracted Information |
|
|
171 | (2) |
|
General ML Security Concerns |
|
|
173 | (2) |
|
|
175 | (1) |
|
Model Debugging for Security |
|
|
175 | (3) |
|
Model Monitoring for Security |
|
|
178 | (1) |
|
Privacy-Enhancing Technologies |
|
|
179 | (3) |
|
|
182 | (1) |
|
|
182 | (2) |
|
Case Study: Real-World Evasion Attacks |
|
|
184 | (1) |
|
|
184 | (1) |
|
|
185 | (1) |
|
|
186 | (3) |
|
Part II Putting AI Risk Management into Action |
|
|
|
6 Explainable Boosting Machines and Explaining XGBoost |
|
|
189 | (42) |
|
Concept Refresher: Machine Learning Transparency |
|
|
190 | (1) |
|
Additivity Versus Interactions |
|
|
190 | (1) |
|
Steps Toward Causality with Constraints |
|
|
191 | (1) |
|
Partial Dependence and Individual Conditional Expectation |
|
|
191 | (3) |
|
|
194 | (1) |
|
|
195 | (1) |
|
The GAM Family of Explainable Models |
|
|
196 | (1) |
|
Elastic Net-Penalized GLM with Alpha and Lambda Search |
|
|
196 | (4) |
|
Generalized Additive Models |
|
|
200 | (5) |
|
GA2M and Explainable Boosting Machines |
|
|
205 | (3) |
|
XGBoost with Constraints and Post Hoc Explanation |
|
|
208 | (1) |
|
Constrained and Unconstrained XGBoost |
|
|
208 | (6) |
|
Explaining Model Behavior with Partial Dependence and ICE |
|
|
214 | (3) |
|
Decision Tree Surrogate Models as an Explanation Technique |
|
|
217 | (4) |
|
Shapley Value Explanations |
|
|
221 | (3) |
|
Problems with Shapley values |
|
|
224 | (4) |
|
Better-Informed Model Selection |
|
|
228 | (1) |
|
|
229 | (2) |
|
7 Explaining a PyTorch Image Classifier |
|
|
231 | (30) |
|
Explaining Chest X-Ray Classification |
|
|
232 | (1) |
|
Concept Refresher: Explainable Models and Post Hoc Explanation Techniques |
|
|
233 | (1) |
|
Explainable Models Overview |
|
|
233 | (1) |
|
|
234 | (1) |
|
|
234 | (1) |
|
Explainable AI for Model Debugging |
|
|
235 | (1) |
|
|
235 | (1) |
|
|
236 | (1) |
|
Other Explainable Deep Learning Models |
|
|
237 | (1) |
|
Training and Explaining a PyTorch Image Classifier |
|
|
238 | (1) |
|
|
238 | (1) |
|
Addressing the Dataset Imbalance Problem |
|
|
239 | (1) |
|
Data Augmentation and Image Cropping |
|
|
240 | (2) |
|
|
242 | (2) |
|
|
244 | (1) |
|
Generating Post Hoc Explanations Using Captum |
|
|
244 | (6) |
|
Evaluating Model Explanations |
|
|
250 | (2) |
|
The Robustness of Post Hoc Explanations |
|
|
252 | (6) |
|
|
258 | (1) |
|
|
259 | (2) |
|
8 Selecting and Debugging XGBoost Models |
|
|
261 | (36) |
|
Concept Refresher: Debugging ML |
|
|
262 | (1) |
|
|
262 | (1) |
|
|
262 | (2) |
|
|
264 | (1) |
|
|
265 | (1) |
|
Selecting a Better XGBoost Model |
|
|
266 | (5) |
|
Sensitivity Analysis for XGBoost |
|
|
271 | (1) |
|
|
272 | (1) |
|
Stress Testing Methodology |
|
|
273 | (1) |
|
Altering Data to Simulate Recession Conditions |
|
|
274 | (2) |
|
Adversarial Example Search |
|
|
276 | (4) |
|
Residual Analysis for XGBoost |
|
|
280 | (1) |
|
Analysis and Visualizations of Residuals |
|
|
281 | (4) |
|
|
285 | (2) |
|
|
287 | (3) |
|
Remediating the Selected Model |
|
|
290 | (1) |
|
|
291 | (2) |
|
|
293 | (2) |
|
|
295 | (1) |
|
|
296 | (1) |
|
9 Debugging a PyTorch Image Classifier |
|
|
297 | (30) |
|
Concept Refresher: Debugging Deep Learning |
|
|
299 | (3) |
|
Debugging a PyTorch Image Classifier |
|
|
302 | (1) |
|
|
303 | (2) |
|
Software Testing for Deep Learning |
|
|
305 | (1) |
|
Sensitivity Analysis for Deep Learning |
|
|
306 | (8) |
|
|
314 | (7) |
|
|
321 | (4) |
|
|
325 | (1) |
|
|
326 | (1) |
|
10 Testing and Remediating Bias with XGBoost |
|
|
327 | (42) |
|
Concept Refresher: Managing ML Bias |
|
|
328 | (3) |
|
|
331 | (4) |
|
Evaluating Models for Bias |
|
|
335 | (1) |
|
Testing Approaches for Groups |
|
|
335 | (10) |
|
|
345 | (4) |
|
|
349 | (1) |
|
|
350 | (1) |
|
|
350 | (5) |
|
|
355 | (4) |
|
|
359 | (3) |
|
|
362 | (4) |
|
|
366 | (2) |
|
|
368 | (1) |
|
|
369 | (30) |
|
|
370 | (1) |
|
|
370 | (1) |
|
|
371 | (2) |
|
|
373 | (2) |
|
|
375 | (4) |
|
|
379 | (1) |
|
|
379 | (4) |
|
Adversarial Example Attacks |
|
|
383 | (3) |
|
|
386 | (1) |
|
|
387 | (3) |
|
|
390 | (4) |
|
|
394 | (1) |
|
|
395 | (4) |
|
|
|
12 How to Succeed in High-Risk Machine Learning |
|
|
399 | (16) |
|
|
400 | (2) |
|
Science Versus Engineering |
|
|
402 | (1) |
|
The Data-Scientific Method |
|
|
403 | (1) |
|
|
404 | (1) |
|
Evaluation of Published Results and Claims |
|
|
405 | (2) |
|
|
407 | (3) |
|
Commonsense Risk Mitigation |
|
|
410 | (3) |
|
|
413 | (1) |
|
|
414 | (1) |
Index |
|
415 | |