π Lecture 20 β Threats to Validity and Related Issues
π Overview: This lecture examines the various factors that can threaten the validity of a psychological test. It details the variables affecting test validity, such as group nature and sample heterogeneity, and provides a framework for evaluating validity coefficients. The lecture concludes with a comprehensive review of the different types of validity evidence.
ποΈ Topics Covered
This lecture covers factors affecting test validity including the nature of the group, heterogeneity of the sample, preselection, and the form of the relationship between test and criterion. It then addresses issues for evaluating validity coefficients, such as changes in relationships, criterion meaning, sample size, range restriction, and differential prediction. The lecture ends with a formal review of content, criterion (predictive and concurrent), and construct validity evidence.
π Lecture Summary
Factors Affecting Test Validity:
There are several variables that can be a source of error in the validity of a test. One must be aware of these variables and their possible effects. The effect of these variables should be controlled in planning, designing, developing, and administering the test.
a) Nature Of The Group: Tests are not administered to identical or very similar groups every time. The group on which the test was administered may not be the same as other groups with whom the test is used. When using a test on the basis of its validity, we need information regarding the details of the sample on which it was validated. A test with a high validity coefficient obtained from one group may not be an efficient predictor with another sample. For example, a test of mathematical ability validated on a sample of people with mixed abilities might not yield similar results when administered to a group of expert accountants or a group of fine artists who haven't studied math since junior school. Test manuals should specify the details of the sample from whom validity was obtained and the population to which the test can be generalized.
b) Heterogeneity Of The Sample: The validity coefficient is a value of correlation. Statistically, the correlation between two sets of scores will be higher if the scores come from a wider range. Therefore, a correlation will be higher in heterogeneous groups as compared to homogeneous groups.
c) Preselection: Preselection can cause a problem when measuring criterion-related validity for a test used in job selection. If the test is validated on a sample of newly selected employees, the fact that they were selected (and are therefore a more capable, homogeneous group) can limit the range of scores and lower the validity coefficient.
d) The Form Of Relationship Between Test And Criterion: One basic assumption in computing the Pearson correlation coefficient is that the relationship between the test and the criterion is linear and uniform across the whole range of scores. If the relationship is not linear and scores are clustered at certain points, as seen in a scatter plot, it will affect the validity coefficient.
Evaluation of Validity Coefficient:
The joint committee of the American Psychological Association, American Educational Research Association, and the National Council on Measurement in Education (1990) described issues to consider when interpreting validity coefficients.
a) Look for changes in the cause of relationships. The cause of the relationship between the test and the criterion should remain the same in the future, but situations may change, altering this relationship.
b) What does the criterion mean? The criterion should be carefully chosen. A test with unknown or dubious validity should not be chosen as a criterion, and the criterion should relate specifically to the use of the test.
c) Review the subject population in the validity study. One should know the type of population on which the test was validated. The test might be used with groups not represented in the population from which the validity coefficient was obtained.
d) Be sure the sample size was adequate. Tests should not be validated on very small samples; the sample should be reasonably sized.
e) Never confuse the criterion with the predictor. At times, test users may confuse the criterion with the predictor, using the criterion first. For example, universities give admission to students who have not passed a graduate ability test but require them to clear it before results are declared. The test (predictor) is then following the criterion (performance), which has already taken place.
f) Check for restricted range on both predictor and criterion. The range of scores should not be restricted. A wider range of scores yields a better correlation/validity.
g) Review evidence for validity generalization. The test user must know if the test can be used with a different population from the one on which it was validated. If not, the user must reconsider the choice of the test.
h) Consider differential prediction. At times, criterion-related validity is not the right choice. If a good criterion is not available, using a weak or wrong criterion is not advisable. In such situations, construct-related evidence for validity is a better option.
Review of Validity and Its Types:
π Definition β Content Validity Evidence: βThe evidence that the content of a test represents the conceptual domain it is designed to coverβ (Kaplan & Saccuzzo, 2001, p.635).
π Definition β Criterion-Related Validity: A procedure where scores on a test are correlated with scores on a criterion. It is defined as βThe evidence that a test score corresponds to an accurate measure of interest. The measure of interest is called the criterionβ (Kaplan & Saccuzzo, 2001, p.635).
π Definition β Construct Validity Evidence: A process used to establish the meaning of a test through a series of studies. The researcher simultaneously defines a construct and develops instrumentation to measure it. Observed correlations between the test and other measures provide evidence for the meaning of the test (Kaplan & Saccuzzo, 2001, p.635).
π Definition β Predictive Validity Evidence: βThe evidence that a test forecasts score on the criterion at some future timeβ (Kaplan & Saccuzzo, 2001, p.638). Predictive validity pertains to prediction over a time interval.
π Definition β Concurrent Validity Evidence: βEvidence for criterion validity in which the test and the criterion are administered at the same point in timeβ (Kaplan & Saccuzzo, 2001, p.635).
Criterion-related validity can thus take the form of predictive or concurrent validity.
β Key Takeaways
The validity of a test is not a fixed property but is influenced by multiple factors including the nature and heterogeneity of the sample, preselection effects, and the linearity of the relationship between test and criterion. When evaluating a validity coefficient, a student must check for changes in relationships, the appropriateness of the criterion, sample size, range restriction, and the possibility of confusing the predictor with the criterion. Finally, it is crucial to remember the three main types of validity: content, criterion (which includes predictive and concurrent), and construct, as each provides different evidence for what a test measures.
π§ Quick Revision Questions
- How does the heterogeneity of a sample influence a testβs validity coefficient?
- What is the problem of "preselection," and how can it lower a validity coefficient?
- List three key issues one should review when evaluating a validity coefficient for a test.
- What is the fundamental difference between predictive and concurrent validity?
- According to the lecture, what type of validity evidence should be sought when a good criterion is not available?
π Lecture 21 β Item Analysis
π Overview: This lecture covers the critical post-administration process of evaluating individual test items through quantitative methods. It explains how item analysis helps test developers identify and refine items that effectively measure the intended construct, ensuring the final test has optimal difficulty and discrimination power.
ποΈ Topics Covered
The lecture covers the definition and purpose of item analysis, followed by detailed explanations of item difficulty index including its calculation, interpretation, and the effects of guessing. It then examines item discrimination index, explaining how to calculate and interpret it using upper and lower scoring groups, along with the shortcut procedure for group selection.
π Lecture Summary
Item Analysis
After test items are written and administered in a pilot study, item analysis is conducted to evaluate each item quantitatively. This process answers questions about which items are too easy or too difficult, whether items differentiate between high and low scorers, and if items are arranged by difficulty level. Items that are "too easy" (everyone gets correct) or "too difficult" (no one gets correct) do not differentiate between those who know and those who do not, making them poor items.
π Definition β Item Analysis: "A set of methods used to evaluate test items. The most common techniques involve assessment of item difficulty and item discriminability." (Kaplan & Saccuzzo, 2001, p. 637)
Item Difficulty
Item difficulty assesses how difficult test items are by calculating the percentage or proportion of test takers who respond correctly. The purpose is to examine whether the difficulty level set during construction was appropriate, identify items that are "too easy" (everyone answers correctly), and identify items that are "too difficult" (no one answers correctly). Such items need to be removed, replaced, or revised.
π Definition β Item Difficulty: "A form of item analysis used to assess how difficult items are. The most common index of difficulty is the percentage of test takers who respond with the correct choice" (Kaplan & Saccuzzo, 2001, p. 637).
π Formula: Item difficulty index (p) = Number of test takers who answered correctly / Total number of test takers. The value ranges from 0 to 1.00. A large p value (close to 1.00) indicates an easy item; a small p value (close to 0) indicates a difficult item. The common acceptable range is between .3 and .8.
π Example: If an item has a difficulty index of .60, this means 60% of test takers answered it correctly (60/100 = .60). An index of .50 means 50% answered correctly (50/100 = .50). An index of 0 means nobody answered correctly (bad item), and 1.00 means everybody answered correctly (bad item).
Average Index of Difficulty of A Test
The average difficulty index for the whole test is calculated by adding all individual item difficulty indices and dividing by the total number of items. For maximum discrimination among test takers' abilities, the optimal average difficulty is approximately .5, with individual items ranging from .3 to .8 (Cohen & Swerdlik, 1999).
π Formula: Average difficulty = (Sum of all item difficulty indices) / (Total number of items)
Taking Care of Guessing
For multiple-choice tests where guessing is possible, the optimal average difficulty is set higher than for free-response tests. The optimal average difficulty is calculated by taking the mid-point between chance success proportion and 1.00. Chance success proportion is the likelihood of guessing correctly (e.g., .25 for 4 options, .20 for 5 options, .33 for 3 options).
π Formula: Optimal average difficulty = (Chance success proportion + 1.00) / 2
π Example: For a test with 4 response options per item, chance success proportion = .25. Optimal item difficulty = (.25 + 1.00) / 2 = 1.25 / 2 = .625. For a test with 5 options, the average proportion correct should be .69 (Lord, 1959).
π‘ Why this matters: Adjusting for guessing ensures that the test accurately measures knowledge rather than luck, especially important in high-stakes selection tests.
Item Difficulty Index In Different Types Of Tests
For achievement tests or intellectual ability tests, item difficulty is based on the proportion of people who gave the correct answer. However, for tests with no right or wrong answers (e.g., personality tests), an item-endorsement index may be used instead. This analysis identifies items that nobody replied to or that received the same response from everyone.
Item Discrimination
Item discrimination measures whether a test item differentiates between those who know (high scorers) and those who do not know (low scorers). A good test item should be answered correctly by more high scorers than low scorers. If an item is correctly answered by low scorers but not high scorers, something is wrong with the item.
Item Discrimination Index
The item-discrimination index (d) is a measure of the difference between the proportion of high scorers answering an item correctly and the proportion of low scorers answering the item correctly. The value of d ranges from -1 to +1. A value of +1 means all high scorers answered correctly and no low scorers did (ideal). A value of -1 means all high scorers failed and all low scorers passed (very poor item). A value of 0 means no differentiation between groups.
π Definition β Item Discrimination Index: "A measure of the difference between the proportion of high scorers answering an item correctly and the proportion of low scorers answering the item correctly; the higher the value of d, the greater the number of high scorers answering the item correctly" (Cohen & Swerdlik, 1999).
π Formula: d = (Number of high scorers answering correctly β Number of low scorers answering correctly) / n, where n = number of high scorers (or low scorers). Alternatively, d = (U β L) / n.
π Example: For item 4 in the table, U = 9, L = 2, n = 10. d = (9 β 2) / 10 = 7/10 = .7. This indicates good discrimination. For item 2, U = 10, L = 0, d = (10 β 0) / 10 = 1.0 (ideal). For item 1, U = 10, L = 10, d = 0 (no discrimination). For item 3, U = 0, L = 10, d = -1 (very poor).
Selecting Upper and Lower Groups
A shortcut procedure (Aiken, 1994) divides examinees into three groups: an upper group consisting of the top 27% of examinees based on total test scores, a lower group of the bottom 27%, and a middle group of the remaining 46%. When the number of examinees is small, upper and lower 50% groups may be used. The item discrimination index is then calculated using only the upper and lower groups.
β Key Takeaways
Item analysis is a crucial quantitative procedure conducted after pilot testing to evaluate individual test items. Item difficulty index (p) ranges from 0 to 1.00, with optimal items falling between .3 and .8; items that are too easy (p=1.00) or too difficult (p=0) should be removed. The average test difficulty for maximum discrimination should be around .5, adjusted upward for multiple-choice tests to account for guessing. Item discrimination index (d) ranges from -1 to +1, with positive values indicating good discrimination between high and low scorers, and values near 0 indicating poor items. The standard procedure uses the top 27% and bottom 27% of scorers to calculate discrimination.
π§ Quick Revision Questions
- What is the purpose of item analysis in test development?
- How is item difficulty index (p) calculated, and what do values of 0, .50, and 1.00 indicate?
- For a multiple-choice test with 4 options per item, what is the optimal average difficulty after accounting for guessing?
- How is item discrimination index (d) calculated, and what does a value of -1 signify?
- What percentage of examinees are typically used for the upper and lower groups when calculating item discrimination?
π Lecture 22 β Item Analysis of Item Distracters
π Overview: This lecture extends basic item analysis to evaluate not just whole items, but the individual response options (distracters) within multiple-choice questions. Understanding distracter analysis is crucial for building tests that accurately measure knowledge by ensuring that only high-knowledge examinees can identify the correct answer.
ποΈ Topics Covered
The lecture explains why analyzing item distracters is important for test quality, then introduces the concept of "response valence" as a method for evaluating each alternative option. It presents data patterns that indicate good distracters, poor distracters, and weak items, along with recommendations for improving or removing problematic options.
π Lecture Summary
Item Analysis of Item Distracters
Item analysis typically uses the item discrimination index to determine how well an item differentiates between high scorers (those who know the material) and low scorers (those who do not). However, it is equally important to analyze the distracters β the incorrect response options in a multiple-choice item. A good item has distracters that appear plausible to examinees who do not know the correct answer, reducing the chance of guessing correctly. If a distracter is too weak (obviously wrong), test-takers will avoid it; if one option is clearly the best, even those who do not know the material may select it. Therefore, each alternative must be evaluated.
π Definition β Item distracter: An incorrect response option in a multiple-choice test that is intended to appear plausible to examinees who do not know the correct answer.
π Definition β Response valence: The number or percentage of examinees in the high-scoring and low-scoring groups who select each response option (including the keyed correct answer).
The lecture explains that while the 'd' value (item discrimination index) shows whether an item discriminates overall, distracter analysis examines how each non-keyed option is chosen. To determine the valence of each option, percentages of responses to the keyed answer and each distracter are calculated. High scorers should predominantly select the keyed answer, while low scorers should be more evenly distributed across the distracters. If a distracter is selected by many high scorers but few low scorers, it is problematic and needs revision or replacement. If all distracters attract roughly equal numbers of test-takers, they are effective.
π Example β Evaluating Distracter Quality: The lecture provides a data table comparing responses from high scorers (n=25) and low scorers (n=25). For a good item where option (a) is correct:
Option (a) β | Option (b) | Option (c) | Option (d) High scorers: 18 | 3 | 3 | 3 Low scorers: 6 | 6 | 7 | 6
Here, the pattern is ideal: most high scorers select the correct answer, and low scorers distribute fairly evenly across all four options.
π Example β A Poor Distracter: In this item, distracter (c) attracts more high scorers (10) than low scorers (7), indicating a problem:
Option (a) β | Option (b) | Option (c) | Option (d) High scorers: 13 | 2 | 10 | 2 Low scorers: 6 | 6 | 7 | 6
Distracter (c) is "too attractive" to knowledgeable students, likely because it appears partially correct. This distracter needs improvement or replacement.
π Example β A Bad Item: This item shows all distracters being selected by more high scorers than low scorers, and low scorers are not evenly distributed:
Option (a) β | Option (b) | Option (c) | Option (d) High scorers: 4 | 5 | 8 | 8 Low scorers: 9 | 3 | 6 | 7
Here, only 4 of 25 high scorers chose the correct answer, while 9 low scorers did. The item is failing entirely and should be removed or completely rewritten.
The lecture concludes that weak items and poor distracters may be improved, but bad items are better removed from the test.
π‘ Why this matters: Distracter analysis prevents the common error of assuming an item is good just because its overall discrimination index is acceptable. Even a discriminating item can have one or more distracters that artificially inflate scores for low-knowledge examinees, undermining test validity.
β Key Takeaways
Distracter analysis is essential for evaluating multiple-choice items beyond the item discrimination index. A good distracter is selected by more low scorers than high scorers, and all distracters should attract roughly equal numbers of low scorers. If a distracter is chosen by many high scorers, it is likely confusing or partially correct and must be revised. If all distracters fail (more high scorers select them than the correct answer), the entire item is bad and should be removed. Effective distracter analysis improves test reliability by ensuring that only knowledgeable examinees consistently select the keyed answer.
π§ Quick Revision Questions
- What is "response valence" in the context of distracter analysis?
- Why is it important that a distracter appear "plausible" to test-takers who do not know the correct answer?
- In a good item with 25 high scorers and 25 low scorers, what pattern of distracter selection indicates a problem with distracter (b)?
- What does it mean if all distracters are selected by more high scorers than low scorers?
- What should be done with a weak distracter versus a bad item according to the lecture?