Muutke küpsiste eelistusi

Psychometrics of Standard Setting: Connecting Policy and Test Scores [Kõva köide]

2.00/5 (2 hinnangut Goodreads-ist)
(Counseling, Educational Psychology, and Special Education Department, Michigan State Univ., USA)
  • Formaat: Hardback, 260 pages, kõrgus x laius: 234x156 mm, kaal: 453 g, 34 Tables, black and white; 31 Line drawings, black and white; 31 Illustrations, black and white
  • Sari: Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences
  • Ilmumisaeg: 23-Dec-2022
  • Kirjastus: Chapman & Hall/CRC
  • ISBN-10: 1498722113
  • ISBN-13: 9781498722117
Teised raamatud teemal:
Psychometrics of Standard Setting: Connecting Policy and Test ScoresSuurem pilt
  • Formaat: Hardback, 260 pages, kõrgus x laius: 234x156 mm, kaal: 453 g, 34 Tables, black and white; 31 Line drawings, black and white; 31 Illustrations, black and white
  • Sari: Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences
  • Ilmumisaeg: 23-Dec-2022
  • Kirjastus: Chapman & Hall/CRC
  • ISBN-10: 1498722113
  • ISBN-13: 9781498722117
Teised raamatud teemal:
Püsilink: https://www.kriso.ee/db/9781498722117.html
Märksõnad:
This book provides a unifying structure for the activities that fall under the process typically called "standard setting" on tests of proficiency. Standard setting refers to the methodology used to identify performance standards on tests of proficiency. The results from standard setting studies are critical for supporting the use of many types of tests. The process is frequently applied to educational, psychological, licensure/certification, and other types of tests and examination systems. The literature on procedures for standard setting is extensive, but the methodology for standard setting has evolved in a haphazard way over many decades without a unifying theory to support the evaluation of the methods and the validation of inferences made from the standards. This book provides a framework for going beyond specific standard setting methods to gain an understanding of the goals for the methods and how to evaluate whether the goals have been achieved.

The unifying structure provided in this book considers policy that calls for the existence of performance standards, the relationship of proficiency test design to the policy, and tasks assigned to subject matter experts to help them convert the policy to estimates of locations on the reporting score scale for the test.The book provides guidance on how to connect the psychometric aspects of the standard setting process to the intentions of policy makers as expressed in policy statements. Furthermore, the structure is used to support validity arguments for inferences made when using standards. Examples are provided to show how the unifying structure can be used to evaluate and improve standard setting methodology.

Arvustused

"Overall, The Psychometrics of Standard Setting: Connecting Policy and Test Scores is an invaluable resource for professionals involved in developing and evaluating proficiency tests. It is particularly well-suited for policymakers, psychometricians, and test developers seeking to ensure their standard setting processes are scientifically sound and aligned with policy objectives. The books comprehensive approach, combining theory, practical guidance, and examples, makes it a must-read for anyone looking to deepen their understanding of this critical area in psychometrics." -Myanca Rodrigues in ISCB News, May 2024

"Reckase takes us on a deep dive into the theoretical underpinnings of standard setting and achieves his goal of providing a more formal and unified structure to the many facets of standard setting. No light read, this volume is most appropriate for those already quite familiar with the topic or for a graduate course in standard setting. That is, the book will illuminate and provide a different and deeper understanding of the topic for those having already digested the current literature. [ ...] Reckases efforts pay off in a substantial contribution to the measurement literature that strongly complements the existing work. [ ...] We also strongly endorse this text for a graduate course or seminar on standard setting, as it will provide a strong foundation from which to better understand the various standard setting methods and the commonalities present in the larger standard setting process." -Daniel Lewis and Sandip Sinharay in the Journal of Educational Measurement, 2024

Preface xi
Author biography xiii
1 Conceptual Overview for a Theory of Standard Setting
1(16)
1.1 The Prevalence of Standards and Standard Setting
1(3)
1.2 Definitions of a Standard
4(3)
1.3 The Differing Contexts for Setting a Standard
7(1)
1.4 Who Defines the Standard?
8(1)
1.5 Qualifications of the Persons Who Participate in the Standard Setting Process
9(2)
1.6 The Standard Setting Process as a Data Collection Design
11(1)
1.7 Calculating the Value of the Standard
12(2)
1.8 Evaluation of the Results of a Standard Setting Process
14(1)
1.9 Is a Performance Standard an Instructional Goal?
15(1)
1.10 The Goals for This Book
16(1)
2 A Theory of Standard Setting
17(18)
2.1 Introduction fo a Theory of Standard Setting
17(1)
2.2 Basic Concepts
18(9)
2.2.1 Agency
18(1)
2.2.2 Policy Definition
19(1)
2.2.3 Content-Specific Definition and Test Specifications
20(2)
2.2.4 Kernel of the Standard Setting Process
22(2)
2.2.5 Estimate of the Intended Standard
24(1)
2.2.6 Feedback to Panelists
25(1)
2.2.7 Approval of the Estimated Standard
26(1)
2.3 Documentation of the Standard Setting Process
27(1)
2.4 The Validity Argument
27(1)
2.5 Some Fanciful Examples of Standard Setting
28(4)
2.5.1 How Much Oxygen Is Needed for Healthy Living?
28(2)
2.5.2 Understanding of the Functioning of Passenger Automobiles
30(2)
2.6 Summary and Organization of Subsequent
Chapters
32(3)
3 Agency, Policy Definitions, Content-Specific Definitions and Minimum Qualifications
35(12)
3.1 The Agency and Its Role in Setting Performance Standards
35(6)
3.1.1 Involvement
35(1)
3.1.2 Expertise
36(1)
3.1.3 Heterogeneity
36(1)
3.1.4 Categorization of Agencies
37(1)
3.1.4.1 State of Michigan Legislature
37(1)
3.1.4.2 National Assessment Governing Board
38(1)
3.1.4.3 National Council of State Boards of Nursing
39(1)
3.1.5 Influence of Agency Characteristics on Standard Setting
40(1)
3.2 Policy Definition (DP)
41(2)
3.3 Content-Specific Definition (DC)
43(1)
3.4 Minimum Qualifications Definition (minDC)
44(1)
3.5 Summary
45(2)
4 The Construct and the Test Designed to Measure the Construct
47(10)
4.1 The Target for Test Design and Development
47(2)
4.2 The Unidimensional Model for Test Development
49(3)
4.3 Domain Sampling Model for Test Development
52(2)
4.4 Implications of Test Design for Standard Setting
54(1)
4.5 Summary
55(2)
5 The Kernel of the Standard Setting Process
57(36)
5.1 Introduction
57(2)
5.2 Examples of Standard Setting Kernels
59(30)
5.2.1 The Bookmark Kernel
60(13)
5.2.2 The Angoff Kernel
73(6)
5.2.3 The Body of Work Kernel
79(7)
5.2.4 The Contrasting Groups Kernel
86(3)
5.3 Criteria for Selecting a Standard Setting Kernel
89(4)
5.3.1 Criterion Number 1
89(1)
5.3.2 Criterion Number 2
90(3)
6 Panelist Selection and Training
93(22)
6.1 Introduction
93(1)
6.2 The Influence of Panelists' Characteristics on the Estimated Standard
94(3)
6.3 Selecting and Recruiting Panelists
97(2)
6.4 Statistical Considerations for Selecting Panelists
99(4)
6.5 Training of Panelists
103(10)
6.5.1 Presenting the "Big Picture"
106(1)
6.5.2 Training on Policy and Content Definitions (DP and DC)
107(1)
6.5.3 Training on the Language of the Reporting Score Scale
107(1)
6.5.4 Training on the Tasks Required of the Standard Setting Kernel
108(1)
6.5.5 Planning and Timing
109(3)
6.5.6 Evaluating Training
112(1)
6.6 Conclusions
113(2)
7 Feedback and Auxiliary Information
115(30)
7.1 Introduction
115(6)
7.1.1 Categorizations of Feedback and Auxiliary Information
116(2)
7.1.2 Purposes for Providing Feedback
118(3)
7.2 Types of Feedback
121(17)
7.2.1 Feedback about Understanding of the Standard Setting Kernel
122(10)
7.2.2 Feedback about the Consequences of the Use of the Estimated Cut Score
132(3)
7.2.3 Other Types of Feedback
135(3)
7.3 Auxiliary Information
138(3)
7.4 Timing of Feedback
141(1)
7.5 Impact of Feedback on the Standard Setting Process
141(1)
7.6 Conclusions and Recommendations
142(3)
8 Estimating the Standard
145(56)
8.1 Introduction
145(3)
8.2 Assumptions Underlying the Estimation Processes
148(3)
8.2.1 Reporting Score Scale
148(1)
8.2.2 Independence of Information from Panelists
149(2)
8.2.3 Other Assumptions
151(1)
8.3 The Value that Is Estimated
151(45)
8.3.1 Estimation Based on Contrasting Groups Data Collection
154(20)
8.3.2 Estimation Based on Test Item Judgment Data
174(9)
8.3.2.1 Effect of Item Parameter Estimation Error on Estimates of Standards
183(2)
8.3.2.2 Effect of Panelists' Errors in Estimation of Conditional Probabilities
185(2)
8.3.2.3 Including Panelists' Interpretation of Dc and minDc in the Estimation of the Intended Standard
187(2)
8.3.3 Estimation Using Compromise Methods
189(7)
8.4 Recent Developments
196(1)
8.5 Cautions about Generalizability
197(1)
8.6 Summary and Conclusion
198(3)
9 Evaluating the Standard Setting Process
201(28)
9.1 Introduction
201(2)
9.2 Types of Evidence
203(23)
9.2.1 Conceptual Coherence Evidence
203(2)
9.2.1.1 The Policy Definition (DP)
205(2)
9.2.1.2 The Content-Specific Definition (DC)
207(1)
9.2.1.3 The Operational Definition of the Intended Construct, ζ
208(2)
9.2.2 Procedural Evidence
210(1)
9.2.2.1 Kernel Selection and Design of the Process
211(2)
9.2.2.2 Panelist Capabilities and Training
213(1)
9.2.2.3 Implementation and Documentation
214(1)
9.2.3 Internal Consistency
215(1)
9.2.3.1 Consistency between Panelists
215(3)
9.2.3.2 Consistency over Rounds or with Other Data
218(3)
9.2.3.3 Full Replication of the Standard Setting Process
221(1)
9.2.3.4 Decision Consistency
222(2)
9.2.4 External Consistency
224(2)
9.3 The Validity Argument
226(3)
10 Trends in Standard Setting Design and Development
229(18)
10.1 Introduction
229(1)
10.2 Policy and Content Definitions
230(1)
10.3 Setting Multiple Standards
231(8)
10.3.1 Multiple Standards on the Same Test
231(3)
10.3.2 Coordinated Standards for Tests at Different Levels---Vertical Articulation
234(5)
10.4 New Approaches to Standard Setting
239(4)
10.4.1 Embedded Standard Setting (ESS)
239(2)
10.4.2 Item Descriptor (ID) Matching Method
241(2)
10.5 Standard Setting on New Test Designs
243(1)
10.6 Some Final Thoughts
244(3)
References 247(8)
Index 255
Mark D. Reckase has decades of experience developing and reviewing standard setting processes as well as doing advanced work in psychometric theory. He is currently a University Distinguished Professor Emeritus at Michigan State University. He has been the editor of Applied Psychological Measurement and the Journal of Educational Measurement. He has also served as the president of the National Council on Measurement in Education and the International Association for Computerized Adaptive Testing, and as the vice president of Division D of the American Educational Research Association.