📘 Lecture 10 — Test Development: Test Construction
📖 Overview: This lecture details the systematic process of test development, from initial conceptualization through final revision. It explains the critical steps of constructing, trying out, analyzing, and revising test items to create a psychometrically sound instrument.
🗂️ Topics Covered
The lecture covers the five-step test development process: test conceptualization, test construction (including scaling methods and item writing), test tryout, item analysis, and test revision. It details different types of scales (age, grade, stanine), scaling methods (Likert, paired comparisons, sorting, equal-appearing intervals), item formats (selected response and constructed response), scoring models (cumulative, class, ipsative), and the characteristics of good test items.
📝 Lecture Summary
Test Development Process: Test Conceptualization
Test development begins with a test developer's idea to measure a particular construct. The stimulus for creating a test can come from various sources, such as literature on an already developed test that creates a need for further psychometric work, or the emergence of a social phenomenon that requires measurement.
Once the idea is formed, several questions must be answered: What is the test designed to measure? What is its purpose? Is there a need for it? What should the test content and format be? A critical question is "How will meaning be attributed to scores on the test?" which points to the issue of norm-referenced versus criterion-referenced tests.
- In norm-referenced tests, a good item is one that high scorers get right and low scorers get wrong.
- In criterion-referenced tests, development involves pilot work with two groups: one that has mastered the knowledge and one that has not. Items that best discriminate between these groups are considered "good."
🔑 Definition — Norm-referenced test: A test where an individual's score is interpreted by comparing it to the scores of a normative group. 🔑 Definition — Criterion-referenced test: A test where an individual's score is interpreted by comparing it to a predetermined standard or criterion of mastery. 💡 Why this matters: Understanding this distinction determines the entire approach to item selection and test interpretation.
Pilot Work refers to preliminary research surrounding the creation of a test prototype. Test items may be pilot studied to evaluate their suitability for the final instrument. This process involves creating, revising, and deleting many items. Once pilot work is done, test construction begins, though future pilot research may be needed for updates.
Test Construction
Scaling is defined as the process of setting rules for assigning numbers in measurement. In other words, scaling assigns values to different amounts of the attribute being measured.
Types of scales can be categorized by level of measurement (nominal, ordinal, interval, or ratio) or in other ways:
- Age scale: Performance is of critical interest as a function of age.
- Grade scale: Performance is of critical interest as a function of grade.
- Stanine scale: All raw scores are transformed into scores ranging from 1 to 9.
Scaling Methods include several approaches:
- Likert scale: Used to scale attitudes. Each item presents five alternative responses, usually on an agree/disagree continuum. Likert (1932) concluded that assigning weights of 1 through 5 generally works best.
- Method of paired comparisons: Test takers are presented with pairs of stimuli and must select the more appealing one. The test taker receives a higher score for selecting the option considered more justifiable by a group of judges.
- Sorting tasks: Printed cards, drawings, or other stimuli are presented for evaluation. Comparative scaling entails judgments of a stimulus in comparison with every other stimulus. Categorical scaling places stimuli into two or more alternative categories that differ quantitatively.
- Method of equal-appearing intervals: Described by Thurstone (1929), this method is used to obtain data that are presumed to be interval in nature.
All methods except equal-appearing intervals yield ordinal data.
Writing Items involves three important considerations:
- What range of content should the items cover?
- Which item formats should be employed?
- How many items should be written?
For a standardized test with multiple-choice format, the first draft should contain approximately twice the number of items that the final version will contain. This provides a basis for content validity, as the final version must still adequately sample the domain. Items can be written from personal experience, with help from experts, from the sample to be studied, or through literature searches.
Item Formats are divided into two types:
- Selected response format: Presents the examinee with a choice of answers and requires selection of one alternative. Types include multiple choice, matching, and true/false items.
- Constructed response format: Requires the examinee to provide or create the correct answer. Types include:
- Completion item: Requires providing a word or phrase that completes a sentence.
- Short answer item: Written clearly enough for a brief response.
- Essay: Asks examinees to describe a single topic in detail.
Scoring Items uses several models:
- Cumulative model: The most common model. The higher the score, the higher the ability or trait being measured. Test takers earn cumulative credit for responses made in a particular way.
- Class model: Test takers' responses earn credit toward placement in a particular class or category with others whose score patterns are presumably similar.
- Ipsative scoring: The objective is to compare a test taker's score on one scale within a test with another scale within that same test.
Test Tryout
After creating a pool of items, the test developer tries out the test on the sample for which it is constructed. A general rule is no fewer than five subjects, and preferably as many as ten subjects, for every one item on the test. The more subjects in the tryout, the better. The tryout should be executed under conditions as close as possible to the conditions under which the standardized test will be administered, including test instructions, time limits, and atmosphere.
What is a Good Item?
- A good test item should be valid and reliable.
- A good item helps discriminate test takers; high scorers on the test as a whole get it right.
- An item that high scorers do not get right is probably not a good item.
- A good item is one that low scorers on the test as a whole get wrong.
Item Analysis
Item analysis refers to the statistical procedures employed to select the best items from a pool of tryout items. These procedures include an index of an item's difficulty, an item-validity index, an item-reliability index, and an index of item discrimination.
Qualitative Item Analysis is a non-quantitative method that employs verbal rather than mathematical techniques. Through questionnaires or discussions with test takers, valuable information can be obtained on how the test could be improved.
Test Revision
Based on the information gathered during item analysis, some items are eliminated and others rewritten. The test developer characterizes each item by its strengths and weaknesses and may need to balance these across items. For example, if many good items are somewhat easy, the developer may purposefully include some more difficult items. After balancing concerns, the developer produces a test of improved quality. The revised test is then administered under standardized conditions, and if the item analysis is satisfactory, the test may be considered in its finished form.
⭐ Key Takeaways
The test development process is a systematic, five-step cycle: conceptualization, construction, tryout, item analysis, and revision. A critical early decision is whether the test will be norm-referenced (comparing individuals to a group) or criterion-referenced (comparing to a mastery standard), as this determines item selection criteria. During construction, test developers must choose appropriate scaling methods (like Likert or paired comparisons) and item formats (selected-response vs. constructed-response) based on the test's purpose and the number of examinees. A good item must be reliable, valid, and able to discriminate high scorers from low scorers. Finally, item analysis procedures, both quantitative and qualitative, guide the revision process to produce a psychometrically sound final instrument.
🧠 Quick Revision Questions
- What are the five steps of the test development process, and in what order do they occur?
- How does the definition of a "good item" differ between norm-referenced and criterion-referenced tests?
- What is the recommended minimum number of subjects per item for a test tryout?
- Describe the difference between selected-response and constructed-response item formats, providing two examples of each.
- What are the three scoring models discussed in the lecture, and what is the primary purpose of ipsative scoring?
📘 Lecture 11 — Item Writing
📖 Overview: This lecture focuses on the critical process of writing test items during test construction. It covers three primary considerations—content range, item format, and number of items—and provides a detailed examination of various item formats, including selected-response and constructed-response types. Understanding these principles is essential for developing valid and reliable psychological tests.
🗂️ Topics Covered
The lecture begins with three key considerations for item writing: nature and range of content, type of items, and number of items. It then explores item formats as classified by Kaplan and Saccuzzo (2001), including dichotomous, polytomous, Likert, category, checklists, and Q-sorts. The distinction between selected-response and constructed-response formats is discussed, with detailed explanations of each type, including matching items and correction for guessing.
📝 Lecture Summary
Considerations in Item Writing
The process of test construction requires careful planning, with three important considerations: what range of content the items should cover, which item formats to employ, and how many items to write. The test developer decides the nature and extent of content based on the objectives laid down for measurement, ensuring each item measures some aspect of the content area. The type of item format depends on the test type and construct being measured; for example, a projective item may be suitable for a personality test but not for an achievement test. The number of items affects test length and administration (individual vs. group). Whether measuring a single or multiple traits/abilities/domains also impacts item count—fewer items are needed for a single domain, while multiple domains require a larger number. When developing a standardized test with multiple-choice response format, the first draft should contain approximately twice the number of items that the final version will contain. The test developer may write items from personal experience, with help from experts, and through literature searches.
Item Formats
Kaplan and Saccuzzo (2001) describe several item formats: the dichotomous format, polytomous format, Likert format, category format, and checklists and Q-sorts. Test formats are also classified as recall type vs. recognition type, constructed response type vs. identification type, and objective type vs. essay type. Another common division is between selected-response format (where examinees choose from alternatives) and constructed-response format (where examinees create the answer).
Selected Response Format
This format presents the examinee with a choice of answers and requires selection of one alternative (e.g., on achievement tests, selecting the correct option). Types include multiple choice, matching, and true/false items.
a. The Dichotomous Format: This is the alternate response format providing two response options. One option is right and the other wrong, typically scored as one point per correct answer (e.g., ten right options = score of ten). Common variations include "yes"/"no" options (as in personality inventories) and true/false format. The test taker judges whether statements are true or false based on course texts.
🔑 Definition — Dichotomous format: An alternate response format where the test taker is provided with two response options to choose from, typically right/wrong, yes/no, or true/false.
Advantages: Easy to develop (no need for multiple plausible options), quick to develop, and easy to score. Disadvantages: A test taker can accurately answer at least 50% of answers by chance (probability = ½); if equal true and false items exist, marking all true yields 50% correct. Such items may encourage rote memorization rather than conceptual learning. To control disadvantages, use a large number of items covering a wider range of content.
b. The Polytomous (Polychotomous) Format: This format provides multiple response options rather than two. Each item includes a stem, a correct alternative option, and several incorrect alternatives called distracters or foils.
📌 Example:
- Stem: "A psychological test, an interview, and a case study are:"
- Correct Alt: a. psychological assessment tools
- Distractors: b. standardized behavioral samples, c. reliable assessment instruments, d. theory-linked instruments
The stem should clearly state the question or problem. Distractors should each appear to be the correct answer. Using a four-option format reduces chance correct to 25% (1/4); three options yield 33% chance. Testing experts recommend three or four options per item.
📐 Formula: Correction for guessing — Corrected score = R - W / (n - 1) Where R = number of right responses, W = number of wrong responses, n = number of choices per item. Plain English meaning: This formula adjusts the raw score by subtracting a penalty based on wrong answers divided by the number of options minus one.
📌 Example: A test taker scores 40 out of 100 on a four-option test. Corrected score = 40 - 60/(4-1) = 40 - 60/3 = 40 - 20 = 20. This shows guessing can reduce a score substantially when correction is used. The greater the number of wrong answers, the smaller the corrected score. Test developers should include all response options in the same proportions (not favoring one option like 'C' or 'D' more often).
Matching Item: A matching item presents two columns of responses where the test taker determines which item from the first column matches an item from the second column. If both columns have equal numbers, examinees can deduce the right answer by process of elimination. Providing more options than needed minimizes this possibility.
c. The Likert Format: The Likert scale is a popular tool for measuring personality and attitudes. It provides test takers options to endorse their degree of agreement with a statement. Likert used it as part of his method of attitude scale construction. Typically, five response options range from "strongly disagree" to "strongly agree," with "neutral" in between.
📌 Example: "I like to make friends who are older to me." Options: Strongly disagree, disagree, neutral, agree, strongly agree. To avoid the tendency to mark "neutral," six options may be used: Strongly disagree, moderately disagree, mildly disagree, mildly agree, moderately agree, strongly agree. Responses are summed to determine a person's score; negatively worded items are reverse scored and added to the total.
d. The Category Format: Similar to the Likert scale but with more options (commonly a 10-point scale, though the number may vary). Used to rate something (e.g., team performance). Raters may be inaccurate due to contrast effects—rating a player lower when compared to a superb player, or higher when compared to a poor performer. Some authors recommend showing raters videos of performances rated "10" and "1" to control this factor (Kaplan & Ernst, 1983).
e. Checklists and Q-sorts: Adjective checklists contain a list of adjectives; the test taker checks those true about themselves (or others). Used mostly in personality assessment. In Q-sort, the test taker receives statements and sorts them into nine piles based on how true they are of themselves; this can also be used for rating others.
Constructed Response Format
This format requires the examinee to provide or create the correct answer rather than selecting it. Three types: completion item, short answer, and essay.
- A completion item requires the examinee to provide a word or phrase that completes sentences.
- A good short answer item is written clearly so the test taker can respond briefly; there is no strict rule for length.
- An essay requires the examinee to describe a single topic in detail. Skills measured by essays differ from other formats—essays require recall, organization, planning, and writing ability, while other formats only require recognition.
💡 Why this matters: Understanding the differences between selected-response and constructed-response formats is crucial for selecting the appropriate item type based on the construct being measured and the level of cognitive processing required.
⭐ Key Takeaways
The three main considerations before writing items are content range, item format, and number of items—these must align with test objectives and the construct being measured. Dichotomous formats (true/false, yes/no) are easy to develop and score but suffer from high chance correct rates (50%), while polytomous formats (multiple choice) reduce chance but require well-constructed distractors. Correction for guessing using the formula R - W/(n-1) penalizes wrong answers and discourages random guessing. Likert and category formats measure degree of agreement or ratings but may be affected by response biases like "neutral" tendencies or contrast effects. Constructed-response formats (completion, short answer, essay) measure higher-order skills like recall and organization, unlike selected-response formats which only require recognition.
🧠 Quick Revision Questions
- What are the three important considerations a test developer must address before writing test items?
- What is the chance of answering a dichotomous item correctly by pure guessing, and how can this disadvantage be controlled?
- Using the correction for guessing formula, calculate the corrected score for a test taker who gets 35 right and 45 wrong on a 4-option test.
- What is the key difference between the Likert format and the category format in terms of response options?
- What cognitive skills does an essay item measure that are not measured by multiple-choice items?
📘 Lecture 12 — Item Writing: Guidelines For Item Writing
📖 Overview: This lecture covers the fundamental principles and practical guidelines for writing effective test items. It explains the essential steps in test planning and development, including how to avoid common pitfalls like double-barreled items and response sets. The lecture also addresses the specific considerations required for developing different types of tests, such as educational achievement, personality, intelligence, and screening tests, along with an introduction to various scoring models.
🗂️ Topics Covered
This lecture begins with DeVellis's guidelines for item writing, including clarity, item pool development, reading difficulty, item length, avoiding double-barreled items, and mixing positive and negative items. It then covers the initial test plan and design steps, followed by detailed discussions on developing educational achievement tests using taxonomies of objectives, personality tests based on theoretical constructs, intelligence tests considering multiple variables, and screening tests using job analysis. The lecture concludes with test scoring methods, including cumulative, class, and ipsative models, and important cautions about test construction.
📝 Lecture Summary
Item Writing: Guidelines For Item Writing
Every test developer must keep specific aims and objectives in mind while designing a test. DeVellis (1991) provided several guidelines for writing items. First, whatever is to be measured should be clearly defined and items should be as specific as possible. The test developer should prepare a large number of items first, developing 3-4 items for each one to be included in the final version, all representing the content area to be covered.
The reading difficulty level must be appropriate for the test takers, avoiding complex vocabulary that is not part of a layman's everyday vocabulary. Items should be of reasonable length, as exceptionally long items are rarely good. Test developers must avoid "double-barreled" items that include two or more ideas at the same time, such as "I often help the poor because I believe in serving humanity and I am a follower of my leader who himself is known for social service." Such items should be converted into two or more separate items.
🔑 Definition — Acquiescence response set: A tendency of test takers to agree with most items regardless of their content.
💡 Why this matters: This response set can invalidate test results by introducing systematic bias.
To overcome the acquiescence response set, test developers should mix negatively and positively worded items and use them alternately. If all items are negative (e.g., 'I hate liars', 'Most people betray', 'I feel depressed most of the time'), the test taker may develop a response set and mark every statement the same way. Additional considerations include avoiding double negatives (e.g., 'I do not disagree to the fact that people should not be stopped from playing cricket'), as they make it difficult for test takers to understand what is being asked. The cultural background of test takers should also be kept in mind, and cultural, racial, and gender bias must be completely avoided.
Initial Test Plan and Design:
Test development involves several sequential steps: determining and formulating the observations of the test; deciding the domain and content to be covered; deciding about the format of the test and the test items; developing/writing the items; developing much more items than required; trying out the initial version; analyzing the results of the try out; reshaping and refining the first version; another try out if needed; and norm development if required (standardized tests only). Beyond these common steps, there are considerations specific to certain types of tests.
Development of Educational Achievement Tests:
In educational tests, the most important element is the formulation and use of educational objectives stated in behavioral terms. Educational objectives define behaviors that will indicate whether the content has been learned. Several formal systems, known as taxonomies of educational objectives, help test developers.
The most popular system is Bloom's Taxonomy of Educational Objectives: The Cognitive Domain (Bloom & Krathwohl, 1956), with major categories including: Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation. Educational Testing Service's (1965) taxonomy includes: Remembering, Understanding, and Thinking. Gerlach and Sullivan's (1967) system includes: Identifying, Naming, Describing, Constructing, Ordering, and Demonstrating. Ebel's (1979) system covers: Understanding of terminology, Understanding of fact and principle, Ability to explain or illustrate, Ability to calculate, Ability to predict, Ability to recommend appropriate action, and Ability to make an evaluative judgment.
When planning a test, developers often create a table of specification — a two-way table with behavioral objectives on the vertical axis (row headings) and content/topics on the horizontal axis (column headings). This table shows the type of objectives corresponding to specific content areas and the total number of items in each box.
📐 Formula: Table of Specification structure → Behavioral objectives (rows) × Content topics (columns) showing number of items for each objective-topic combination
Development of Personality Tests:
In personality tests, the theoretical approach of the test developer matters most. Items are based on the constructs that are to be measured.
Development of Intelligence Tests:
In planning intelligence tests, several variables must be considered: what aspects of intelligence are to be measured, what criteria will be used (e.g., age, grade, or other), the test format considered suitable, and the target population.
Development of Screening Tests:
Aptitude tests are generally used for screening purposes. The objective is to identify candidates most suitable for the job who fulfill the position's requirements. First, a job analysis or task analysis is conducted, identifying and listing the various components of the job. Test items are based on these tasks and job-related situations or 'critical incidents'.
Test Scoring:
Different tests are scored differently. For objective tests, scoring is done with a scoring key, either manually or using computers with attached scanners. Scoring essay type items is more difficult and can be done using 'holistic' or 'global' scoring (scoring the whole response) or analytic scoring (scoring different components separately), with the latter being the better approach. Involving another person for rescoring can increase objectivity. Standardized tests have their scoring procedures specified in the test manual.
Scoring Items:
There are several scoring models. The most common is the cumulative model, where the higher the score, the higher the ability or trait being measured. For each response made in a particular way, the test taker earns cumulative credit with regard to a particular construct.
The class model is a second model where test takers' responses earn credit toward placement in a particular class or category with others whose score pattern is presumably similar.
A third model is ipsative scoring, where the typical objective is comparing a test taker's score on one scale within a test with another scale within that same test.
🔑 Definition — Ipsative scoring: A scoring model where a test taker's score on one scale is compared with another scale within the same test.
Some Cautions:
Test developers should avoid jargon and use vocabulary that most people can understand. The test should not be too long. It should not be so easy that everyone can do it, nor so difficult that no one can do it. Cultural biases and stereotypical ideas must be avoided. If cultural differences are suspected that may affect test results, nonverbal and culture-free tests should be used. However, even culture-free tests may not be fully culture fair, as people with previous experience and exposure to such items may outperform those from cultures where tasks involving drawings and images are unfamiliar.
⭐ Key Takeaways
The most critical points from this lecture are the specific guidelines for item writing, including clarity of definitions, appropriate reading difficulty, avoiding double-barreled items and double negatives, and mixing positively and negatively worded items to prevent acquiescence response sets. Students must understand the complete test development sequence from determining observations through norm development, and be able to apply Bloom's taxonomy and the table of specification for educational achievement tests. For different test types, remember that personality tests depend on theoretical constructs, intelligence tests require consideration of multiple variables, and screening tests rely on job/task analysis. Finally, know the three scoring models (cumulative, class, and ipsative) and the crucial cautions about test length, difficulty, and cultural bias.
🧠 Quick Revision Questions
- What is a "double-barreled" item and why should it be avoided in test development?
- How does mixing positively and negatively worded items help overcome the acquiescence response set?
- List the six major categories of Bloom's Taxonomy of Educational Objectives for the Cognitive Domain.
- What is the purpose of a table of specification in educational test development?
- What are the three scoring models for test items, and how does each model interpret test taker responses?
📘 Lecture 13 — Reliability
📖 Overview: This lecture introduces the concept of reliability in psychological testing, explaining why it is critical for trustworthy measurement. It covers the definition of reliability, the classical test score theory, sources of error in test scores, and the role of correlation in calculating reliability. Understanding reliability is fundamental because it determines whether a test produces consistent and dependable results for serious decisions.
🗂️ Topics Covered
This lecture begins with the definition of reliability and its importance in psychological testing, distinguishing it from physical sciences. It then explains the classical test score theory (X = T + E) and details various sources of error, including test-related factors, administration process variables, and examinee-related variables. The lecture proceeds to explain the concept of correlation as it relates to reliability, including the coefficient of correlation, its magnitude and direction, and the interpretation of scatter plots for positive, negative, and zero correlations.
📝 Lecture Summary
Reliability
Reliability is one of the three basic qualities of a good psychological test, along with validity and standardization. By definition, reliability means “the consistency of the scores obtained by the same persons when they are reexamined with the same test on different occasions, or with different sets of equivalent items, or under other variable examining conditions” (Anastasi & Urbina, 2007). According to Kaplan & Saccuzzo (2001), reliability is “the extent to which a score or measure is free from measurement error. Theoretically, reliability is the ratio of true score variance to observed score variance.”
The classical test score theory implies that everyone can obtain a true score on any test if the measure is free of error. However, there is no measure that can be considered totally error-free. The scores we obtain are observed scores, which equal the true score plus error. The term error here does not mean a mistake; it refers to the amount and extent of variance that may be expected in results.
🔑 Definition — True Score (T): The score a person would obtain if the measure were completely free of error. 🔑 Definition — Observed Score (X): The actual score obtained on a test, which is the sum of the true score and error. 📐 Formula: X = T + E → The observed score equals the true score plus error. 💡 Why this matters: This formula is the foundation of reliability theory. It means every test score contains some degree of error, and reliability is about estimating how much error is present.
Sources of Error in Test Scores
Psychologists and educationists try to estimate the amount and degree of error that may be expected in measurement. This is why test developers and administrators emphasize uniform testing procedures and controlling possible sources of error.
a. Test Related Factors:
- Difficulty level (too difficult or too easy)
- Length of the test (too long causing fatigue or boredom)
- Domain or content (not suitable for all test takers)
- Items may not be representing the domain or content
- Time limit (may cause stress or a handicap)
b. Test Administration Process:
- Poor and not uniform testing conditions (physical setting and environment)
- Poor, faulty, improperly worded, improperly delivered, not uniform instructions
- Test administrator’s personality (different administrators in different situations)
- Rapport (poor, too much, or too little)
c. Examinee Related Variables:
- Prior learning and experience
- Individual differences (personality styles, stress tolerance, IQ, emotionality, motivation, knowledge)
- Difference from the normative sample
- Within-person differences (changes over time due to life experiences, health, motivation, emotional state)
Test developers try to control or keep these variables constant, but complete consistency is impossible. However, test developers report the coefficient of reliability and characteristics of the normative sample, guiding users about appropriate populations for the measure.
Correlation and Reliability
Reliability is about consistency of scores. Calculation of reliability involves the concept of correlation. The coefficient of correlation is the value yielded by the calculation of correlation, denoted by the letter ‘r’. This coefficient indicates the relationship between two variables and expresses the correspondence between two scores.
The coefficient of correlation tells us two things about a relationship: magnitude (size of correlation) and direction (whether positive or negative). The size of a correlation ranges between -1 and +1, with zero in between. A value of +1 means a perfect positive correlation, -1 means a perfect negative correlation, and zero indicates no correlation. The closer the value is to one, the stronger the correlation.
If scores on two sets increase and decrease together, it is a positive correlation. If scores on set-I increase while set-II decrease, it is a negative correlation. The concept can be understood through a scatter plot. If the lowest scorer on set-I is also lowest on set-II, and highest is highest on both, it is a perfect positive correlation. If the lowest on set-I is highest on set-II, it is a perfect negative correlation.
🔑 Definition — Coefficient of Correlation (r): A numerical value ranging from -1 to +1 that expresses the magnitude and direction of relationship between two variables. 📐 Formula: Pearson Product Moment Correlation (r) → The most commonly used procedure for computing the coefficient of correlation. 📌 Example: Scores from 10 students on IQ and marks in math showed a perfect positive correlation (r = +1.0), as every student’s IQ matched their math marks exactly. Scores on IQ and classes missed showed a perfect negative correlation (r = -1.0), as higher IQ was associated with fewer missed classes. Marks in English and math showed zero correlation (r = 0.0), with no discernible pattern between the two.
The size of the correlation coefficient generally acceptable for reliability is around .80 or .90. A coefficient less than .60 is usually not acceptable. For data from a large number of people, a coefficient less than .80 may also be acceptable.
⭐ Key Takeaways
Reliability is the consistency of test scores and is expressed as the ratio of true score variance to observed score variance, following the formula X = T + E. The three main categories of error sources are test-related factors, administration process variables, and examinee-related variables, which must be controlled to improve reliability. Correlation, specifically the Pearson Product Moment method, is the statistical foundation for calculating reliability, with acceptable coefficients generally above .80. The magnitude and direction of correlation are both important: magnitude ranges from -1 to +1, and direction indicates whether the relationship is positive or negative. A coefficient below .60 is typically unacceptable for reliability purposes.
🧠 Quick Revision Questions
- What is the fundamental formula of classical test score theory, and what does each component represent?
- List three test-related factors that can introduce error into test scores.
- What is the range of possible values for a coefficient of correlation?
- In a scatter plot, what pattern indicates a perfect negative correlation?
- What is the minimum coefficient of correlation generally considered acceptable for reliability?
📘 Lecture 14 — Types of Reliability
📖 Overview: This lecture explores the various methods used to measure the reliability of psychological tests. Understanding these different types is crucial because the choice of method depends on what aspect of consistency is being evaluated—whether stability over time, equivalence between test forms, or internal consistency among items. This knowledge is fundamental for evaluating the quality and trustworthiness of any psychological measure.
🗂️ Topics Covered
This lecture covers five primary types of reliability: test-retest reliability, which assesses stability over time; alternate-form reliability, which examines equivalence between different versions of a test; split-half reliability, a single-administration method measuring internal consistency; Kuder-Richardson reliability, which analyzes inter-item consistency for dichotomously scored items; and coefficient alpha, a general formula for tests with polytomous items. The lecture concludes with a discussion of inter-scorer reliability for subjectively scored tests.
📝 Lecture Summary
Reliability: Definition and Forms
According to Anastasi & Urbina (2007), reliability refers to “the consistency of the scores obtained by the same persons when they are reexamined with the same test on different occasions or with different sets of equivalent items, or under other variable examining conditions”. This definition highlights three key forms of consistency: consistency when the same test is taken again on different occasions, consistency when different but equivalent sets of items are used, and consistency under other variable conditions.
a) Test-Retest Reliability
Test-retest reliability deals with two performances of the same test by the same persons on two different occasions. It measures the consistency and stability of scores over time. The resulting test-retest coefficient is also known as the ‘coefficient of stability’. Scores from the first administration are correlated with scores from the second administration. A major advantage is maximum control over test taker and test item variables, as both the subjects and the items remain the same.
This type of reliability accounts for errors of measurement from several sources:
- Testing conditions may not remain constant (e.g., noise, temperature).
- Changes in the test taker (bodily, emotional, motivational) can occur.
- Fatigue effect or practice effect can introduce error variance.
- The nature of the test may be affected by repeated measurement (e.g., memory of items).
- The length of the time interval between administrations affects the coefficient's magnitude; a shorter interval yields a larger coefficient.
Test-retest reliability is denoted as rₜₜ.
💡 Why this matters: The length of the time interval is a critical factor. A test developer must report the interval between administrations because a reliability coefficient is meaningless without this context.
b) Alternate-Form Reliability
Alternate-form reliability overcomes problems of using the same test twice. The developer creates two alternate or parallel forms that should be completely equivalent in all respects, including same specifications, instructions, time limit, content, number of items, item format, and difficulty level.
When alternate forms are administered in immediate succession (no time gap), the coefficient is the ‘coefficient of equivalence’ or ‘parallel form coefficient’. To compute stability over time (coefficient of stability), the two forms are administered with a time gap. A ‘coefficient of stability and equivalence’ is obtained by counterbalancing the administration of forms:
- Group I takes Form A first, Group II takes Form B first (Occasion I).
- After a time interval, Group II takes Form A, and Group I takes Form B (Occasion II).
This procedure accounts for error from both different sets of items and the time interval. While a good approach, its shortcomings include the difficulty of creating truly equivalent forms and the potential for practice effects across forms.
c) Split-Half Reliability
Split-half reliability avoids problems of constructing parallel forms and the influence of time gaps. In this single-administration approach, the test is divided into two equal halves, and scores on one half are correlated with scores on the other. The obtained value is the ‘coefficient of internal consistency’.
The test can be divided in several ways:
- Dividing by first 50% and last 50% items (problematic due to potential differences in difficulty).
- A better method is dividing into odd-numbered items and even-numbered items, ensuring similar difficulty and content in both halves.
Because the correlation is based on only half the test, it must be adjusted. The Spearman-Brown formula estimates the reliability of the full test.
🔑 Definition — Spearman-Brown formula: A formula used to estimate the internal consistency reliability of a test from the reliability of its two halves, correcting for the fact that the test has been shortened or lengthened.
📐 Formula: rₙₙ = (n * rₜₜ) / (1 + (n-1) * rₜₜ) Where:
- rₙₙ = estimated coefficient for the lengthened/shortened test
- rₜₜ = obtained coefficient from the half-tests
- n = number of times the test has been lengthened or shortened (e.g., for split-half, n=2)
For split-half reliability, the formula becomes: rₜₜ = (2 * rₕₕ) / (1 + rₕₕ) Where rₕₕ is the reliability of the half tests.
d) Kuder-Richardson Reliability
The Kuder-Richardson (K-R) method measures reliability through inter-item consistency using a single test administration. Unlike split-half, it examines performance on each item and is the mean of all possible split-half coefficients. It is only used for tests where items are scored dichotomously (e.g., 0 or 1).
📐 Formula: r = [n / (n-1)] * [1 - (∑pq / SD²ₜ)] Where:
- r = Coefficient of reliability of whole test
- n = number of items
- SD²ₜ = Variance of total scores
- p = proportion of persons who pass each item
- q = proportion of persons who do not pass each item
- ∑pq = sum of the products of p and q for all items
e) Coefficient Alpha
Coefficient alpha (also known as Cronbach’s alpha) is similar to the K-R method but is a general formula that works for tests with more than two scoring weights (e.g., Likert-scale items where 1=never, 2=occasionally, 3=often).
📐 Formula: α = [n / (n-1)] * [1 - (∑s²ᵢ / SD²ₜ)] Where:
- α = Coefficient of reliability of whole test
- SD²ₜ = Variance of total scores
- s²ᵢ = Variance for each item’s scores
- ∑s²ᵢ = sum of the variances of all item scores
Inter-Scorer Reliability / Inter-Rater Reliability
Inter-scorer reliability (or inter-rater reliability) is used when a test requires evaluation by more than one examiner, such as for essay-type or open-ended items. Common procedures include:
- Inter-rater reliability: Two examiners score the test, and the two sets of scores are correlated.
- Intra-class coefficient / coefficient of concordance: Two or more examiners score the performance, and a coefficient is computed from all scores.
⭐ Key Takeaways
The most critical concepts from this lecture are the distinction between methods that require two administrations (test-retest and alternate-form) versus single-administration methods (split-half, Kuder-Richardson, and coefficient alpha). You must remember the specific error sources each method controls: test-retest assesses stability over time but is affected by practice and time interval; alternate-form assesses equivalence across test versions but is difficult to construct; split-half provides a measure of internal consistency using the Spearman-Brown formula. The key difference between Kuder-Richardson and coefficient alpha is that K-R is only for dichotomous items, while alpha is for polytomous items. Finally, inter-scorer reliability is essential for subjectively scored tests.
🧠 Quick Revision Questions
- What is the difference between a "coefficient of stability" and a "coefficient of equivalence"?
- Why is the Spearman-Brown formula necessary in split-half reliability?
- What is the primary limitation of the Kuder-Richardson formula that coefficient alpha overcomes?
- List three potential sources of error variance in test-retest reliability.
- When would an intra-class coefficient be used instead of a simple inter-rater correlation?