📘 Lecture 15 — Reliability in Specific Conditions and Allied Issues
📖 Overview: This lecture examines how reliability estimation methods must be adapted for special testing conditions, particularly oral tests and speed tests. It also covers how sample characteristics affect reliability coefficients, introduces the Standard Error of Measurement (SEM) as an alternative expression of reliability, and discusses acceptable reliability values and methods for enhancing reliability.
🗂️ Topics Covered
This lecture covers reliability issues in oral tests including inter-scorer reliability and methods to improve it; the distinction between speed tests and power tests and why split-half reliability is inappropriate for speed tests; the relationship between sample characteristics (variability and ability level) and reliability coefficients; the concept and calculation of Standard Error of Measurement (SEM); acceptable reliability values for research purposes; and strategies for enhancing test reliability by increasing item numbers and improving item quality.
📝 Lecture Summary
Some Factors Influencing Reliability of a Test:
Reliability of Oral Tests:
In oral tests, inter-scorer reliability becomes a significant concern because examiners must assess responses subjectively. Unlike objective written tests where answers are clearly right or wrong regardless of who marks them, oral test scoring is problematic because different raters may evaluate the same response differently. The reliability of oral tests is usually lower than equivalent written tests. To improve reliability, several measures have been recommended: test takers should be instructed to think before responding (Meredith, 1978), electronic recordings of responses should be made for later reevaluation, oral tests should be carefully designed, model questions should be constructed before administration, and more than one rater should be used. Research shows that when these factors are considered, obtained reliability coefficients range from .60 to .70 (Carter, 1962; Levine & McGuire, 1970; Hitchman, 1966).
Reliability of Speed Tests:
Speed tests focus on how quickly a person completes a test — items are uniformly easy, but the time limit is so short that no one can finish all items. As Anastasi & Urbina (2007, p.116) state: "A pure speed test is one in which individual differences depend entirely on speed of performance." In contrast, power tests have time limits long enough for everyone to attempt all items, but the difficulty is steeply graded so no one can get a perfect score. Most tests contain both features to varying degrees.
💡 Why this matters: The nature of the test (speed vs. power) directly affects which reliability methods are appropriate. Using the wrong method can produce misleading, inflated reliability coefficients.
The split-half procedure using odd-even format is NOT suitable for speed tests. Because speed test items have uniform difficulty, a person who attempts X items correctly will likely attempt equal numbers of odd and even items. This can produce an inflated coefficient (possibly r = +1.00) that does not reflect true reliability. Instead, the following approaches should be used: (a) Test-retest reliability, (b) Equivalent-forms reliability, (c) Variations of split-half method where the two halves are administered separately as if they were two tests, with half the total time for each segment.
📐 Formula Application Example: For a test with 50 items and 60 minutes total time → Odd items (25 items, 30 minutes) and Even items (25 items, 30 minutes) administered separately.
Relationship between the Sample Tested and Reliability:
Two sample characteristics affect reliability coefficients:
Variability: Reliability involves correlation, and correlation requires variance in scores. If scores in a group are identical or very similar, the correlation may be zero. Therefore, the sample should have a wide range of scores and diverse test takers. Test manuals must provide detailed information about the sample used for computing reliability.
Ability level: While lack of variability causes problems, too much variation between subgroups can also introduce error. If the sample contains subgroups very different in ability (especially when difficulty level and prior learning matter), this can distort the score distribution from which correlation is calculated. Subgroups within the larger sample should be chosen with great care.
Standard Error of Measurement (SEM):
SEM is defined as "an index of the amount of error in a test or measure. The standard error of measurement is a standard deviation of a set of observations for the same test" (Kaplan & Saccuzzo, 2001). It is a way of expressing reliability.
📐 Formula: SEM = SD√(1 - rᵵᵵ) where SD = standard deviation of test scores and rᵵᵵ = reliability coefficient
📌 Example: If a test has SD = 12 and reliability coefficient = .90, then SEM = 12√(1 - .90) = 12√0.10 = 12 × 0.316 = 3.79. If a person's true score (mean of hypothetical distribution) is 90, then 68% of people may be expected to score within the range of 90 - 3.79 to 90 + 3.79 (approximately 86.21 to 93.79), because in a normal distribution about 68% of scores fall between -1 and +1 standard deviations from the mean.
Acceptable Values of Reliability:
The closer the reliability coefficient is to 1.00, the better. However, for most research purposes, coefficients within the range of .70 to .80 are considered good enough (Kaplan & Saccuzzo, 2001).
Enhancing Reliability:
Increasing the number of items in a test may improve reliability. Additionally, test items should be carefully phrased and it should be ensured that all items measure the same content that the test is intended to measure.
⭐ Key Takeaways
The most critical points from this lecture are: (1) Oral tests require special attention to inter-scorer reliability, and using multiple raters, recordings, and careful design can achieve coefficients in the .60-.70 range. (2) Split-half reliability methods should never be used for speed tests because uniform item difficulty artificially inflates coefficients; instead use test-retest, equivalent-forms, or separately administered halves. (3) Sample characteristics (variability and ability level) significantly affect reliability coefficients — samples must have adequate score range without extreme subgroup differences. (4) The Standard Error of Measurement (SEM = SD√(1-rᵵᵵ)) expresses reliability as a standard deviation of error scores, allowing you to estimate the range within which a person's true score likely falls (e.g., ±1 SEM contains 68% of observations). (5) For research purposes, reliability coefficients of .70-.80 are acceptable, and reliability can be enhanced by increasing the number of items and ensuring all items measure the same content.
🧠 Quick Revision Questions
- Why is inter-scorer reliability a greater concern for oral tests than for written objective tests?
- What distinguishes a pure speed test from a pure power test in terms of item difficulty and time limits?
- Why does the split-half odd-even method produce an inflated reliability coefficient for speed tests, and what three alternative methods should be used instead?
- How do sample variability and ability level influence the reliability coefficient?
- If a test has a standard deviation of 15 and a reliability coefficient of .84, what is the SEM, and within what range would 68% of examinees' true scores fall?
📘 Lecture 16 — Validity
📖 Overview: This lecture defines validity as an essential characteristic of psychological tests, explaining that a valid test measures what it claims to measure. It explores the three main types of validity evidence—content, criterion, and construct—and details procedures for establishing each type, emphasizing specific reporting rather than general labels.
🗂️ Topics Covered
The lecture begins by drawing analogies from everyday life to explain validity and reliability, then defines validity formally. It distinguishes between achievement and predictive test purposes, introduces the three broad categorizations of validity—content, construct, and criterion—and details content-description procedures, including specific steps and applications, before contrasting content validity with face validity and outlining criterion prediction procedures.
📝 Lecture Summary
Validity
Validity is an essential ingredient of a test. The use of a test that is not valid will not only be a waste of time and energy, but it may also result in erroneous judgments and decisions. Validity, just like reliability, is a characteristic and quality that we seek in most life situations where we have to acquire something, make it a part of our life, and base decision making on that. For example, if we need to buy an air conditioner, we would like to make sure that it cools the room, providing the service that it is meant to provide (validity); we would also like to ensure that every time we switch it on it will turn on and start working (reliability). If the machine fails in cooling, and/or does not turn on when switched on, then we do not need it.
In human relationships, we want our friends to be our ‘friends’ in the true sense—warm, sympathetic, understanding, caring, etc., i.e., what a friend ought to be (validity). On the other hand, we like that our friend is there to support us, help us, and be with us every time we need him/her (reliability). Similarly, psychological tests need these features as an essential integral part.
🔑 Definition — Validity: Traditionally defined as “the extent to which a test measures what it was designed to measure” (Aiken, 1994, p.95). Another definition: “The extent to which a test measures the quality it purports to measure. Types of validity evidence include content validity, criterion validity, and construct validity evidence” (Kaplan & Saccuzzo, 2001, p.640).
From these definitions, one can realize that validity of a test is about the nature of a test with reference to the content of the test as well as the content or domain in which it is rooted. It is about what the test measures and how well it measures. Although the title or name of a test gives us a clue of what the test measures, we may not have an accurate idea of what it actually measures until and unless we have appropriate information. This information may be reported in the test manual or may be calculated by us.
Whether or not a test measures a trait that it claims to measure can be determined only through an examination of the objective sources of information and empirical operations utilized in establishing its validity. A test’s validity must be established with reference to the particular use for which the test is being considered. The validity of a test should not be taken casually, and cannot be reported in general terms either. Validity is not, and should not be measured and reported as “highly” valid or “low” validity. It is reported in specific terms.
Procedures for determining test validity are concerned with the relationships between the test content and the domain that it represents; between the test content and the objectives that lead to test construction; or test performance and performance on other independent measures that are known to measure the same phenomenon/trait/ability/domain, etc.
Psychological tests are developed primarily for two reasons:
- Achievement testing: In order to see if a specific content area has been learned and/or mastery is acquired.
- Predictive function: Predicting future performance on the basis of the present test’s performance.
In both cases, the ultimate objective cannot be achieved if the test is not valid. An achievement test is evaluated by comparing its content with the content domain it is designed to assess. These tests may be used as end-of-course examinations in school and by licensing tests for driving a car or qualifying with a skill for a specified occupation. These focus on content domain. The performance tests used to assess and predict future behavior are designed on certain criterion. To measure the validity of such tests, the correlation coefficient between test scores and a direct and independent measure of that criterion is used.
Another area with reference to validity is about testing the construct in a particular test. Constructs are broad categories, derived from the common features shared by directly observable behavioral variables.
💡 Why this matters: Understanding the three types of validity—content, criterion, and construct—is fundamental because each applies to different test purposes, and a test can be valid for one use but not another. Reporting validity in specific, not general, terms prevents misinterpretation.
Content Validity Evidence
“The evidence that the content of a test represents the conceptual domain it is designed to cover” (Kaplan & Saccuzzo, 2001, p.635).
🔑 Definition — Content Validity Evidence: Evidence that the content of a test represents the conceptual domain it is designed to cover.
Construct Validity Evidence
“A process used to establish the meaning of a test through a series of studies. To evaluate evidence for construct validity, a researcher simultaneously defines some construct and develops the instrumentation to measure it. In the studies, observed correlations between the test and other measures provide evidence for the meaning of the test” (Kaplan & Saccuzzo, 2001, p.635).
🔑 Definition — Construct Validity Evidence: A process used to establish the meaning of a test through a series of studies, where a researcher defines a construct and develops instrumentation to measure it, and observed correlations provide evidence for the test's meaning.
Criterion Validity Evidence
“The evidence that a test score corresponds to an accurate measure of interest. The measure of interest is called the criterion” (Kaplan & Saccuzzo, 2001, p.635).
🔑 Definition — Criterion Validity Evidence: Evidence that a test score corresponds to an accurate measure of interest, where the measure of interest is called the criterion.
Content-Description Procedures
Nature:
- A systematic examination of the test content is made in order to determine whether it covers a representative sample of the behavior domain to be measured.
- Such validation procedures measure how well the individual has mastered a specific skill or course of study.
“Content validity is built into a test from the outset through the choice of appropriate items. For educational tests, the preparation is preceded by a thorough and systematic examination of relevant course syllabi and textbooks, as well as consultation with subject matter experts. On the basis of the information thus gathered, test specifications are drawn up for the item writers” (Anastasi & Urbina, 2007, p. 129)
Certain considerations are needed for content validation measures:
- The test should cover all major aspects and items in correct proportion (overloading and under-representation of elements should be avoided).
- The domain under consideration must be fully described in advance rather than after test preparation.
- Content should be broadly covering the major objectives, application and interpretation, and also factual knowledge.
Specific Procedures: The choice of appropriate items is important in content validity. A number of careful decisions are taken at the time of test development:
- For educational tests, items are prepared from relevant course syllabi and textbooks, and in some cases, consultation with subject experts is used.
- Test specifications are given to item writers. They include instructional objectives, relative importance of topics/processes, clearly indicate the number of items for each topic, and may also provide sample material.
- The process of content validation should include description of all procedures in the manual that ensure the test is appropriate and representative. For example, if subject specialists have participated, the names, qualifications, and number of individuals should be stated.
- The test total score and achievement score of an individual can be checked for grade progress.
Application:
- Content validity is basic to the validity of educational and occupational achievement tests.
- This validity measure is applicable to occupational tests designed for employee selection and classification.
- For aptitude and personality tests, content validation is inappropriate.
Content Validity versus Face Validity
Content validity should not be confused with face validity. Many students of psychology take them to be one and the same, whereas actually they are different. Face validity is about the test appearing to be valid. It is not validity in the technical sense; nor is it technically measured.
🔑 Definition — Face Validity: “The extent to which items on a test appear to be meaningful and relevant, actually not evidence for validity because face validity is not a basis for inference” (Aiken, 1994, p.636).
Face validity may be used where prediction or inference are not involved, e.g., a questionnaire or scale for a survey.
Criterion Prediction Procedures
Criterion validity evidence is “The evidence that a test score corresponds to an accurate measure of interest. The measure of interest is called the criterion” (Kaplan & Saccuzzo, 2001, p.635).
🔑 Definition — Criterion Prediction Procedures: Procedures used to establish criterion validity by correlating test scores with a direct and independent measure of the criterion of interest.
⭐ Key Takeaways
A test is valid only if it measures what it claims to measure, and validity must be established with reference to a specific purpose—not reported in general or vague terms. There are three main types of validity: content validity (how well test content represents a domain), criterion validity (how well test scores predict or correspond to an external measure), and construct validity (how well the test measures an abstract concept through a series of studies). Content validity is built into educational and occupational achievement tests through careful item selection and requiring domain description in advance, while it is inappropriate for aptitude and personality tests. Face validity, which is about a test appearing valid, is technically not a form of validity and should not be confused with content validity.
🧠 Quick Revision Questions
- What is the formal definition of validity according to Aiken (1994)?
- What are the two main reasons for which psychological tests are developed?
- How does content validity differ from face validity?
- What are the specific procedures used to ensure content validity in educational tests?
- What is a construct, and how is construct validity evidence established?
📘 Lecture 17 — Criterion Validity
📖 Overview: This lecture explores criterion-related validity, one of the fundamental approaches to establishing test validity. It explains how test scores are compared against external criteria to determine if a test measures what it claims to measure. This lecture is critical because criterion validity is essential for tests used in selection, diagnosis, and prediction of future performance.
🗂️ Topics Covered
The lecture begins by reaffirming validity as the most essential feature of a test, then introduces the three main approaches to validity assessment (content, criterion-related, and construct). It focuses on criterion-related validity, explaining its conceptual foundation through everyday examples. The two main procedures—predictive validity evidence and concurrent validity evidence—are described in detail with their distinctions and applications. The lecture concludes with common criterion measures and the essential characteristics a criterion must possess, including reliability, validity, relevance, and freedom from contamination.
📝 Lecture Summary
Criterion-related Validity
When planning and designing a test, we have a certain standard in mind that we want to meet. We relate scores on our test with those achieved on the criterion or standard. One approach to assessment of validity is through comparing the test results with those on a criterion.
🔑 Definition — Criterion-related validity: A procedure where scores on a test being used are correlated with scores on a criterion. As defined by Kaplan & Saccuzzo (2001, p.635), "The evidence that a test score corresponds to an accurate measure of interest. The measure of interest is called the criterion."
📌 Example: A teacher develops a test of mathematical ability. To determine its validity, she examines the relationship between scores on her test with another authentic test of the same ability or a test of similar skills. By doing so, she is assessing the criterion-related validity of her test.
The criteria used for this purpose may be in the form of scores on psychological tests, mental and behavioral measurement, classifications, grades, teacher's or supervisor's ratings, etc. According to Aiken (1994, p.96), "Traditionally, however, criterion-related validity has been restricted to validation procedures in which the test scores of a group of examinees are compared with ratings, classifications, or other behavioral or mental measurements."
Criterion Prediction Procedures
Two procedures may be adopted for this purpose:
- Predictive validity evidence
- Concurrent validity evidence
Predictive Validity Evidence
Predictive validity evidence: "The evidence that a test forecasts score on the criterion at some future time" (Kaplan & Saccuzzo, 2001, p.638). Predictive validity pertains to prediction over a time interval. Predictive validation yields information that is most relevant to tests that are used for predicting future performance of an individual.
📌 Example: Tests are used for selecting students for an academic program, for personnel selection and job placement, or for choosing staff members for a specific training or higher education. Tests are also used to predict if a candidate is likely to develop an undesirable behavior, characteristic, mental or emotional problem in future.
Concurrent Validity Evidence
In many situations, estimating validity on predictive validation basis is not too practical. It involves a certain time interval and requires a suitable problem selection sample. In such situations, it is preferable to use the concurrent validation approach. In this approach, the test is administered to such a sample whose data on the criterion are already available.
🔑 Definition — Concurrent validity evidence: "Evidence for criterion validity in which the test and the criterion are administered at the same point in time" (Kaplan & Saccuzzo, 2001, p.635). When the validity of a test is measured with reference to a criterion, and both are administered at nearly the same time, then the resulting validity will be concurrent validity.
📌 Example: One of the best examples of concurrent validity is diagnostic tests such as the Minnesota Multiphasic Personality Inventory (MMPI). The test is administered to people in various categories to determine if there are significant differences between the average scores of different categories of individuals.
💡 Why this matters: Anastasi & Urbina (2007) clearly distinguish between predictive and concurrent validation: "The logical distinction between predictive and concurrent validation is based not on time but on the objectives of testing. Concurrent validation is relevant to tests employed for diagnosis of existing status rather than prediction of future outcomes. The difference can be illustrated by asking 'Does Smith qualify as a satisfactory pilot?' or 'Does Smith have the prerequisites to become a satisfactory pilot?' The first question calls for concurrent validation; the second, for predictive validation."
Criterion Measures
There is no limit to the type of criterion that may be used for the purpose of test validation. Some of the criteria mentioned in the lecture include:
- Academic achievement: used for intellectual tests and similar tests
- Matriculation marks: at the time of admission to higher classes, used for selection and screening
- Years of education: for those who have completed or discontinued education
- Performance in specialized training: used for selection, admission, achievement
- Instructor's ratings
- Grades in a semester
Characteristics of a Criterion
According to Cohen (1999), a criterion should be:
- Reliable: The scores on the criterion as well as those on the test should be reliable. The reliability coefficients of the two sets of scores affect the coefficient of validity. This can be understood from the following pattern of relationship:
📐 Formula: R<sub>xy</sub> (coefficient of validity) is affected by r<sub>xx</sub> (test reliability) and r<sub>yy</sub> (criterion reliability). Weak/little/or no reliability will automatically affect validity.
- Valid: It should measure what it is supposed to measure.
- Relevant: It should be relevant to the purpose for which it is being used.
- Uncontaminated: We should try to use uncontaminated criterion. Criterion contamination occurs when the criterion itself has been, partially or fully, based on predictor measures.
📌 Example: You use a test as a criterion for the diagnosis of patients examined in a psychiatric clinic. A probe into the diagnostic procedure indicates that the diagnosis itself involved the use of the same test. In this case, the criterion will be contaminated and will not be a suitable criterion.
⭐ Key Takeaways
Validity is the most essential feature of a test—a test can be reliable without being valid, but it cannot be valid without being reliable. Criterion-related validity involves correlating test scores with scores on an external criterion, and it has two forms: predictive validity (forecasting future performance over a time interval) and concurrent validity (administering test and criterion at the same time for diagnosis of current status). The key distinction between the two is based on the objective of testing, not on time. For a criterion to be useful, it must be reliable, valid, relevant, and uncontaminated—meaning it should not be based partially or fully on the predictor measure itself.
🧠 Quick Revision Questions
- What is the difference between predictive validity and concurrent validity, and how does the objective of testing determine which one to use?
- Why can a test be reliable without being valid, but a test cannot be valid without being reliable?
- What is criterion contamination, and provide an example where it would invalidate the criterion?
- According to Cohen (1999), what are the four essential characteristics a criterion must possess?
- In the example of the foot ruler and kitchen tiles, what served as the criterion, and what validity concept does this everyday example illustrate?
📘 Lecture 18 — Construct Validity
📖 Overview: This lecture introduces construct validity as a third major approach to establishing test validity, distinct from content and criterion-related validity. It explains that construct validity is established through a process of accumulating evidence from multiple sources, using the example of a science concept acquisition test (SCAT). The lecture emphasizes convergent and discriminant validity as key components, culminating in the multitrait-multimethod approach for rigorous validation.
🗂️ Topics Covered
The lecture begins with a concrete example of developing the SCAT test and forming assumptions to test its validity. It then defines construct validity evidence through multiple scholarly descriptions and outlines its underlying assumptions, specifically convergent and discriminant validity. The steps in the construct validity process are listed, followed by an explanation of the multitrait-multimethod approach and how construct validity is computed using correlations and factor analysis.
📝 Lecture Summary
Construct Validity
The lecture opens with the example of the SCAT (Science Concept Acquisition Test), developed to measure how well children in their final school year applied science concepts learned over the previous 4-5 years. To validate this test, four assumptions were formulated:
- SCAT scores will have a high-positive correlation with scores on a test of similar nature and content.
- SCAT scores will have a high-positive correlation with scores on a test of reasoning ability.
- SCAT scores will have a moderate-positive correlation with marks on school science achievement test.
- SCAT scores will have either no or a negative correlation with scores on a test of rote memorization of meaningless material like non-sense syllables.
Four tests were used to test these assumptions: the Dallas Times Herald Test (scale test), Basic Reasoning Skills Test, General Science Test, and a list of non-sense syllables. A sample of school children took SCAT and the other tests, and correlations were computed. This procedure, where other tests are used systematically to provide evidence, represents a third approach to calculating validity called construct validity, distinguishing it from content and criterion-related validity.
Construct Validity Evidence:
Construct validity is measured by examining whether a test measures the particular construct it is supposed to measure. If it does, the test is considered valid; if it does not, it is not accepted as valid.
🔑 Definition — Construct validity evidence: "A process used to establish the meaning of a test through a series of studies. To evaluate evidence for construct validity, a researcher simultaneously defines some construct and develops the instrumentation to measure it. In the studies, observed correlations between the test and other measures provide evidence for the meaning of the test" (Kaplan & Saccuzzo, 2001, p.635).
🔑 Definition — Construct: "An informed scientific idea developed or constructed to describe or explain behavior." Constructs are unobservable, presupposed (underlying) traits that a test developer may invoke to describe test behavior or criterion performance (Cohen, 1999).
The lecture provides an all-encompassing description from Anastasi & Urbina (2007): "The construct validity of a test is the extent to which the test may be said to measure a theoretical construct or trait. Examples of such constructs are scholastic aptitude, mechanical comprehension, and verbal fluency, speed of walking, neuroticism, and anxiety. Each construct is developed to explain and organize observed response consistencies. It derives from established interrelationships among behavioral measures. Construct validation requires the gradual accumulation of information from a variety of sources."
The significant elements of this description are:
- A theoretical construct or trait
- Each construct is developed to explain and organize observed response consistencies
- It derives from established interrelationships among behavioral measures
- Requires the gradual accumulation of information from a variety of sources
Assumptions Underlying Construct Validity:
Two fundamental assumptions underlie the construct validity approach:
a. Test scores will be highly correlated with tests measuring the same or similar construct. This is convergent validity.
b. Test scores will have weak/low (or at times may be negative) correlation with tests meant to measure constructs different from the one measured by the main test. This is discriminant validity.
These assumptions lead to the calculation of two types of construct validity.
🔑 Definition — Convergent Validity Evidence: This aspect of validity shows the extent to which a test measures the same construct/trait/attribute as do other measures that have been designed and developed to measure the same construct/trait/attribute.
🔑 Definition — Discriminant Validity Evidence: Discriminant validity provides information that the test in question does not measure what other tests measure, or it measures something different from what other available tests measure.
Steps in Construct Validity Process:
- The test in question has to be there
- Clearly defining the construct to be examined
- Assumptions about the construct
- Assumptions about the similar constructs and the different constructs
- Identification of measures that will be used to test the constructs in question
- Test administration, may be in various steps
- Calculation of correlations between the test scores
- Analysis of correlations in order to reach final conclusions regarding construct validity
Multitrait-Multimethod Approach to Construct Validity:
One of the most commonly used models of construct validity is the convergent and discrimination model, the multitrait-multimethod approach proposed by Campbell and Fiske (1959). They proposed that while estimating validity of a test, we should look not only at the measures our test is related with, but also the ones it is not related with.
According to this model, the following relationship analysis can provide information regarding convergent and discriminant validity: a. Correlation between the same construct, using the same method b. Correlation between different constructs using the same method c. Correlation between different methods used for same construct d. Correlation between different constructs using different methods
The validation example of SCAT employed this approach.
Computation of Construct Validity:
Primarily, correlations are calculated for all sets of scores. Where a larger number of tests are involved, factor analysis is applied to the obtained statistics. When the multitrait-multimethod approach is employed, a multitrait-multimethod matrix is developed. This is based on a systematic experimental design (Campbell & Fiske, 1959) and is employed when two or more traits are tested by two or more methods.
⭐ Key Takeaways
Construct validity is a process of gradually accumulating evidence that a test measures the theoretical construct it claims to measure, using multiple sources of information. Its core assumptions are convergent validity (high correlations with tests measuring the same construct) and discriminant validity (low or negative correlations with tests measuring different constructs). The multitrait-multimethod approach is a rigorous model that systematically examines correlations between same and different constructs using same and different methods to establish both convergent and discriminant validity. Construct validity computation primarily uses correlations, with factor analysis applied when many tests are involved.
🧠 Quick Revision Questions
- What are the four assumptions used to validate the SCAT test in the lecture example?
- Define construct validity according to Anastasi & Urbina (2007).
- What is the difference between convergent validity and discriminant validity?
- List the eight steps in the construct validity process.
- What is the multitrait-multimethod approach and what four relationship analyses does it examine?
📘 Lecture 19 — How Much Valid Is “Valid”? Decision Theory
📖 Overview: This lecture examines the practical meaning of test validity—specifically, how large a validity coefficient must be to be useful. It introduces decision theory and the Standard Error of Estimate, and explains how base rates, hit rates, and Taylor-Russell tables help evaluate whether using a test improves selection decisions beyond chance.
🗂️ Topics Covered
The lecture begins by discussing the size of validity and its dependence on the test’s purpose, followed by the Standard Error of Estimate as a measure of prediction error. It then introduces decision theory, explaining base rates and hit rates, and the use of cut-off scores for dichotomous decisions. Finally, it covers Taylor-Russell tables as a method for evaluating the net gain in selection accuracy from using a test.
📝 Lecture Summary
How Much Valid Is “Valid”? — The Size of Validity
The magnitude of a validity coefficient is important but less critical than the significance level. According to Kaplan & Saccuzzo (2001, p.138), “there are no hard-and-fast rules about how large a validity coefficient must be to be meaningful. In practice, one rarely sees a validity coefficient larger than .60, and validity coefficients in the range of .30 to .40 are commonly considered high.” The validity coefficient must be statistically significant at the .01 or .05 level to rule out chance.
The acceptable size of validity depends on the test’s purpose. For admissions to academic programs, lower validity may be acceptable because the consequences of misprediction (e.g., admitting an unsuitable student) are less severe. However, for selecting fighter pilots, anesthetists, cardiac surgeons, or for forensic testimony (e.g., lie detectors), even the smallest doubt about validity is unacceptable.
💡 Why this matters: The same validity coefficient can be deemed acceptable or unacceptable depending on the stakes of the decision being made.
Standard Error of Estimate (SEₑₛₜ)
The Standard Error of Estimate (SEₑₛₜ) quantifies the margin of error in predicting an individual’s criterion score due to imperfect test validity. It is defined as “The error of estimate shows the margin of error to be expected in the individual’s predicted criterion score, as a result of the imperfect validity of the test” (Anastasi & Urbina, 2007, p.157).
🔑 Definition — Standard Error of Estimate (SEₑₛₜ): The estimated standard deviation of prediction errors when using a test to predict a criterion score.
📐 Formula: ( SE_{\text{est}} = SD_y \sqrt{1 - r_{xy}^2} ) → This means: the error in prediction equals the standard deviation of the criterion score multiplied by the square root of the proportion of variance not explained by the test.
📌 Example: If a test has a validity coefficient of ( r_{xy} = 0.60 ) and the criterion’s standard deviation is ( SD_y = 10 ), then ( SE_{\text{est}} = 10 \times \sqrt{1 - 0.36} = 10 \times \sqrt{0.64} = 10 \times 0.8 = 8 ). This means the predicted criterion score has a margin of error of ±8 points (at one SE level).
Decision Theory and Decision Analysis
Decision theory asks: Is it worthwhile to use a test for selection? This requires comparing outcomes with the test versus without the test. Two key concepts emerge:
🔑 Definition — Base Rate: “In decision analysis, the proportion of people expected to succeed on a criterion if they are chosen at random” (Kaplan & Saccuzzo, 2001, p.635). It is the success rate without using the test.
🔑 Definition — Hit Rate: “In test decision analysis, the proportion of cases in which a test accurately predicts success or failure” (Kaplan & Saccuzzo, 2001, p.636). It is the success rate when using the test.
Dichotomous Decisions and Cut-Off Scores
When a test is used for dichotomous decisions (select/reject), a cut-off score is set. Scores above it lead to selection, below to rejection. Four possible outcomes arise:
| Decision on test | Future success | Future failure |
|---|---|---|
| Selected | 1. Valid acceptance (right decision) | 3. False acceptance (wrong decision) |
| Rejected | 2. False rejection (wrong decision) | 4. Valid rejection (right decision) |
The test’s value depends on the proportion of right decisions (cells 1 + 4) versus wrong decisions (cells 2 + 3). The hit rate is the percentage of accurate predictions (cells 1 + 4). The base rate is the success rate among those who would be successful even without the test.
Taylor-Russell Tables
Taylor and Russell (1939) developed tables to evaluate the net gain in selection accuracy from using a test. They show the probability that a person selected based on the test will actually be successful, accounting for the base rate.
🔑 Definition — Selection Ratio: The proportion or percentage of applicants who must be accepted.
To use Taylor-Russell tables, three pieces of information are needed:
- The validity coefficient of the test
- The selection ratio (proportion of applicants to accept)
- The base rate (proportion of successful applicants without using the test)
Additionally, a clear definition of “success” must be established.
📌 Example (from table): With a base rate of .60 (60% of applicants would succeed without the test), if the test’s validity coefficient is .30 and the selection ratio is .10 (only 10% of applicants can be accepted), the expected proportion of successes using the test rises to .82 (82% of those selected will succeed). This represents a net gain from 60% to 82%.
💡 Why this matters: Taylor-Russell tables allow test users to determine whether a test provides enough improvement over chance to justify its cost and effort.
⭐ Key Takeaways
Validity coefficients as low as .30 to .40 are commonly considered high in practice, but the acceptable size depends entirely on the stakes of the decision—higher stakes require higher validity. The Standard Error of Estimate quantifies prediction error and is calculated using the validity coefficient and the criterion’s standard deviation. Decision theory compares base rates (success without the test) to hit rates (success with the test) to determine if the test improves selection accuracy. Taylor-Russell tables provide a practical method to evaluate the net gain from using a test, requiring the validity coefficient, selection ratio, and base rate. The cut-off score creates four possible decision outcomes—valid acceptance, false acceptance, false rejection, and valid rejection—and the test’s value lies in maximizing valid decisions.
🧠 Quick Revision Questions
- What range of validity coefficients is “commonly considered high” in psychological testing?
- What is the formula for the Standard Error of Estimate, and what does each symbol represent?
- Explain the difference between a base rate and a hit rate in decision theory.
- List the four possible outcomes when a test is used for dichotomous decisions with a cut-off score.
- What three pieces of information are needed to use Taylor-Russell tables?