📘 Lecture 06 — Test Norms: Interpreting Test Results
📖 Overview: This lecture explains how raw test scores are given meaning through the use of norms, which serve as standards for comparison. It covers the foundational statistical concepts necessary for understanding test score interpretation, including frequency distributions, measures of central tendency, and variability. Understanding norms is critical for evaluating an individual's performance relative to a standardized group.
🗂️ Topics Covered
The lecture begins by defining test norms and explaining their role in interpreting raw scores, introducing the concepts of normative samples and relative standing. It then transitions into essential statistical concepts used in psychological testing, covering frequency distributions, histograms, frequency polygons, the normal bell-shaped curve, and measures of central tendency including mean, median, and mode. Finally, it discusses measures of variability, specifically variance and standard deviation.
📝 Lecture Summary
Test Norms: Interpreting Test Results
Norms are standards used for comparative and normative functions. Test norms are scores on a measure used as standards against which the scores of any test taker are compared. Raw scores on psychological tests have no meaning unless interpreted with additional supporting information. For example, a score of 50 on an intelligence test does not indicate whether a person has an average, below, or above I.Q. level without knowing the average score or norm. Scores become meaningful in the presence of norms developed for that test.
🔑 Definition — Norms: "The test performance data of a particular group of test takers that are designed for use as a reference for evaluating or interpreting individual test scores". Norms provide standards for comparing the results of test takers on different measurements.
Norms are established by administering a test to a normative sample representative of the population of interest. For example, a stress scale for university students is administered to a normative sample of university students. If the average score is 25, this serves as a norm. Another student scoring 30 would be considered above average.
Standardization sample is a group of people whose performance on a test is taken as a standard for comparison. A person's performance is interpreted with reference to the distribution of scores in this sample to discover their relative standing—the position where a person lies in the distribution of scores in relation to other persons.
Raw scores are converted into derived scores for two reasons:
- To learn about an individual's standing in a distribution of scores relative to previously established norms.
- To make a direct comparison of individual scores with standard comparable measures.
These derived scores are expressed in two forms:
- Developmental level
- Relative position within a specific group
💡 Why this matters: Without norms, a raw score is just a number. Norms transform it into meaningful information about a person's performance compared to others.
Statistical Concepts Used In Psychological Testing
The aim of statistical method is to organize and summarize quantitative data. The first step is to tabulate raw scores to give them meaningful shape.
1- Frequency Distribution: A frequency distribution arranges large data in an organized way. Data is grouped into class intervals, and scores are tallied in the appropriate level. The total number of cases is represented by N.
📐 Frequency Distribution Table: For example, raw scores of 100 individuals on an achievement test ranging from 50 to 104 are grouped into class intervals of five points.
| Class Interval | Frequency |
|---|---|
| 50-54 | 2 |
| 55-59 | 5 |
| 60-64 | 7 |
| 65-69 | 10 |
| 70-74 | 14 |
| 75-79 | 13 |
| 80-84 | 20 |
| 85-89 | 11 |
| 90-94 | 4 |
| 95-99 | 6 |
| 100-104 | 8 |
| N=100 |
Frequency distributions can be presented in graphs:
- Histogram: A graph showing distribution of measured scores in the form of class intervals. The horizontal axis (x-axis) presents class interval scores, and the vertical axis (y-axis) presents frequencies. The height of the column presents the number of frequency, and the width covers the length of intervals.
- Frequency Polygon: Scores are presented by taking a mid-point in each class interval. These points are connected by a straight line.
2- Normal Bell-Shaped Curve: Statistical data can be seen in the form of a normal distribution curve. This curve facilitates basic statistical analysis. The curved area covers the average of some value or characteristic (e.g., I.Q., height, weight), and the two sides indicate extreme cases. As the number of people increases, distribution scores are more likely to resemble the original normal curve.
3- Measures of Central Tendency: Measures of central tendency describe raw scores in meaningful ways. They include:
- Mean
- Median
- Mode
- Variability and Standard Deviation
Mean: The mean or average is calculated by adding all scores and dividing by the number of cases (N).
📐 Formula: M = ∑X / N (Where ∑X = sum of all scores, N = number of cases)
📌 Example: Children obtained marks 7, 8, 4, 6, 5, 9, 3 on an English test. Sum of scores (∑X) = 42 Number of cases (N) = 7 Mean (M) = 42 / 7 = 6
Median: The median is the middle-most value or score in a group of data. It is the point that divides the distribution into half above and half below scores.
📐 Formula: Median = (n+1)/2 (where n = number of scores)
📌 Example: For scores 7, 8, 4, 6, 5, 9, 3, first arrange them in order: 3, 4, 5, 6, 7, 8, 9. Median position = (7+1)/2 = 4 The 4th value is 6.
Mode: The mode is the highest frequency value in a set of scores. In a normal distribution curve, it is represented by the highest point.
📌 Example: In scores 7, 8, 7, 7, 9, 4, 7, the mode is 7 because it occurs most often.
Variability and Standard Deviation: Variability is the dispersion of scores around the mean.
Variance: Variance is the "average squared deviation" around the mean.
📐 Formula: Variance = ∑(X - X̄)² / n (Where X = individual score, X̄ = mean, n = number of scores)
To calculate, first subtract the mean from each individual score to find the deviation, then square each deviation value, and finally calculate the mean of these squared deviation values. The sum of deviations around the mean will always equal zero because positive and negative values cancel each other out.
Standard Deviation: Standard deviation (S.D. or σ) is a more adequate measure of variability. It is calculated by taking the square root of the variance.
📐 Formula: Standard Deviation (S.D.) = √Variance
Larger individual differences reveal a larger S.D., while small individual differences indicate low S.D. values. In a normal distribution curve, interpretations of S.D. are very clear.
⭐ Key Takeaways
Norms are essential for interpreting raw test scores, transforming meaningless numbers into meaningful comparisons against a standardization sample. The lecture establishes that understanding basic statistical concepts—including frequency distributions, histograms, frequency polygons, and the normal curve—is foundational for developing and using norms. The three key measures of central tendency (mean, median, and mode) each describe a distribution's center in a different way, with the mean being the most commonly used but sensitive to extreme scores. Variability, measured by variance and standard deviation, describes how spread out scores are around the mean, with a larger standard deviation indicating greater individual differences. For the exam, you must be able to define norms, explain their purpose, and calculate or interpret the mean, median, mode, variance, and standard deviation.
🧠 Quick Revision Questions
- What are test norms, and why are raw scores meaningless without them?
- What is a standardization sample, and what is the purpose of "relative standing"?
- How are a histogram and a frequency polygon different in their graphical representation of data?
- Calculate the mean, median, and mode for the following set of test scores: 10, 12, 12, 15, 18, 20, 20.
- Explain the relationship between variance and standard deviation. If a set of scores has a variance of 36, what is the standard deviation?
📘 Lecture 07 — Types of Norms
📖 Overview: This lecture explains how raw test scores are converted into meaningful derived scores to interpret an individual’s performance. It covers developmental norms (like mental age and grade equivalents) and within-group norms (like percentiles and standard scores), emphasizing their uses, calculations, and limitations in psychological testing.
🗂️ Topics Covered
This lecture covers types of norms including developmental norms (mental age, basal age, grade equivalents, ordinal scales), within-group norms (percentiles, standard scores, normal standard scores, deviation IQ), relativity of norms (interest and longitudinal comparisons), and scales of measurement. Key concepts include how derived scores allow comparison across tests and populations, and the psychometric limitations of each norm type.
📝 Lecture Summary
Types of Norms
Raw test scores need to be converted into relative measures or derived scores to give them meaningful interpretation. These scores reveal an individual’s relative standing and allow comparison across different tests. Derived scores are expressed either in terms of developmental level attained or the relative position of a person within a specific group.
Developmental Norms
Developmental norms are defined as the typical patterns, characteristics, and age-specific tasks or skills of development at any age or stage. They are established based on development and maturation, assuming people can perform at specific levels at different life stages. When most people can perform certain tasks at a given age, this becomes the norm for that age level and is considered the mental age (MA) for that level. For example, if a 13-year-old girl performs tasks that most 13-year-olds can do, her MA is 13. If an adult only performs tasks a 6-year-old can do, his MA is 6, indicating mental deficiency. If a 9-year-old performs tasks meant for a 16-year-old, his MA is higher than his biological age.
However, comparing performance using developmental norms is not always straightforward because people may take tests measuring different abilities, and even subtests of the same test may measure different skills. It is not necessary that everyone attains the same MA across all tests or subtests, making comparison difficult. Although used commonly for descriptive purposes in clinical and research settings, test scores based on developmental norms are not psychometrically sound.
Mental Age
The term mental age became widely used after the development of the Binet-Simon scales, though Binet used the more neutral term “mental level.” Items were grouped in year levels (e.g., items passed by most 8-year-olds were placed in the 8-year level). With frequent use, the problem of scatter was observed — many subjects did not show uniform performance across all subtests, failing items below their age level while passing items above their age level. To address this, the concept of basal age was introduced.
Basal age refers to the highest year at which a person passes all items. For tests passed at higher year levels, the subject is given partial credits in months, which are added to the basal age to yield the child’s mental age. Mental age norms are also used with tests not designed according to year levels. In these tests, the mean raw scores of children of specific age groups in the standardization sample serve as norms. A child’s mental age is determined by comparing her raw score with the age norm. For example, if a 12-year-old girl’s raw score equals the 12-year norm, her mental age is 12.
🔑 Definition — Mental Age (MA): The age level at which a person performs intellectually, based on the average performance of individuals at that chronological age.
A major shortcoming of using mental age is that it does not mean the same thing at different life stages. MA of 4 for a 5-year-old is not equivalent to MA of 24 for a 25-year-old. As age progresses, the unit of MA tends to shrink. A child with MA of 3 at age 4 would be 3 years retarded at age 12, but one year of mental growth from age 3 to 4 is equivalent to three years of growth from age 9 to 12. Therefore, deviation from the norm at different ages is not comparable — deviation at a very young age is much more significant than at older age.
Grade Equivalents
Grade equivalents represent scores on educational achievement tests attained by children in a certain grade. These norms are obtained by calculating the mean raw scores of children in the standardization sample for each grade. If 6th-grade children obtained a mean score of 35 on an arithmetic test, then this raw score has a grade equivalent of 6. A student scoring 35 on the same test is said to have a grade equivalent of 6.
Since most academic years span ten months, successive months can be expressed in decimal points. For example, a grade equivalent of 7.0 refers to average performance of a 7th grader at the beginning of the session, while 7.5 represents average at the middle of the session.
📐 Formula — Grade Equivalent: Mean raw score of children in a specific grade = Grade Equivalent for that raw score.
Grade norms have several limitations:
- Grade units are unequal and these inequalities occur irregularly in different subject matter areas.
- They are only applicable for common subjects taught throughout the grade levels covered by the tests, not for subjects taught for only one to two years.
- Even when the same courses are covered, identical importance and learning cannot be ensured across grades.
- A child may progress more rapidly in one subject than another during the same grade.
- Grade norms tend to be incorrectly regarded as the performance level of all students, ignoring that individual differences in any grade can be so large that scores vary over several grades.
Ordinal Scales
Another approach to developmental norms comes from research in child psychology. Psychologists observed behaviors typical of successive ages in infants and young children, including functions like sensory discrimination, linguistic communication, and concept formation. These observations proved valuable in understanding human development.
An example is the work of Gesell and his associates, who focused on the sequential patterning of early behavior development. The Gesell Developmental Schedules were developed to determine a child’s approximate developmental level in months across four major areas: motor, adaptive, language, and personal-social behavior. Eight key ages from 4 to 36 weeks serve as standards. Gesell claimed children’s development involves: a) orderly progression of behavior changes and b) uniformities of developmental sequence. For example, chronological sequence can be observed in visual fixation and hand/finger movements when reacting to a small object. The scales used are ordinal scales that yield information about the stage where a child stands, with successful performance at one level implying success at all lower levels.
Jean Piaget (1960s) gave his theory of cognitive development, describing stages of cognitive development in a sequence, with age levels being arbitrary. He studied specific concepts including:
- Object permanence: The child is aware of object existence when they are out of sight.
- Conservation: Recognition that an attribute remains constant over changes in perceptual appearance (e.g., liquid quantity remains constant when poured in different shaped containers).
- Perspective: Knowledge that objects appear differently when at a distance.
To assess these, Piagetian tasks are used, designed to reveal the dominant aspect of each developmental stage.
In short, ordinal scales gauge the uniform progression of development through successive stages by measuring attainment of specific functions.
Within-Group Norms
Within-group norms evaluate a person’s performance by comparing their raw score with the most nearly comparable standardization group, such as children of the same age or grade. These norms are so popular that all test scores now provide within-group norms in some form. Within-group scores employ many statistical procedures due to their clearly defined quantitative meaning.
Percentiles
A percentile indicates an individual’s relative position in the standardization sample. The percentage of people in the standardization sample is expressed in terms of percentile scores. For example, if 50% of people obtained a score of 25 on an analytical reasoning test, then this score corresponds to the 50th percentile.
Key points about percentiles:
- They can describe ranks in a group of 100 people — the top person is given rank 1, the bottom person a poorer rank.
- The 50th percentile refers to the median; scores above 50 represent above-average scores, and scores below 50 indicate below-average scores.
- The 25th and 75th percentiles are known as the first and third quartile points (Q1 and Q3), cutting off the lowest and highest quarters of the distribution.
- The difference between percentage and percentile is that percentage is a raw score while percentiles are derived scores.
Standard Scores
Standard scores express an individual’s distance from the mean in terms of the standard deviation of the distribution. Standard scores can be calculated by linear and non-linear transformation of raw scores. Linearly derived scores are known as z-scores.
🔑 Definition — z-score: A standard score obtained by subtracting the mean of the normative sample from the raw score and dividing by the standard deviation of that sample.
📐 Formula — z-score: ( z = \frac{X - M}{SD} )
Where:
- ( X ) = raw score
- ( M ) = mean of normative sample
- ( SD ) = standard deviation of normative sample
📌 Example: If X = 100, M = 80, and SD = 10: ( z = \frac{100 - 80}{10} = \frac{20}{10} = 2.0 )
A raw score equal to the mean results in a z-score of zero. Negative derived scores indicate performance below average; positive scores indicate above-average performance.
Normal Standard Scores
Normal standard scores are standard scores expressed in terms of a distribution transformed to fit a normal curve. They are obtained by finding the percentage of a person in the standardization sample, locating this percentage in the normal curve frequency table, and obtaining the standard score. Normal standard scores can be put in any convenient form.
If normalized standard scores are multiplied by 10 and added to or subtracted from 50, they become T scores (first proposed by McCall, 1922). In a T score scale, a score of 50 corresponds to the mean, 60 corresponds to 1 SD above the mean, and so on.
📐 Formula — T score: ( T = 50 + 10(z) )
Normalized standard scores should be applied when the sample is large and representative and when deviation from normal results is confirmed to be due to drawbacks in the test rather than characteristics of the sample.
Another variation is the Stanine scale (from “standard nine”), developed by the United States Air Force during World War II. Scores run from one to nine, with a mean of 5 and standard deviation of approximately 2.
🔑 Definition — Stanine: A single-digit standardized score system with values from 1 to 9, mean of 5, and SD of approximately 2.
The Deviation IQ
The term IQ (Intelligence Quotient) was introduced in early intelligence tests. The Ratio IQ is obtained by dividing mental age (MA) by chronological age (CA) and multiplying by 100.
📐 Formula — Ratio IQ: ( IQ = \frac{MA}{CA} \times 100 )
If a child’s MA equals CA, IQ = 100 (average performance). IQ below 100 indicates below-average scores; above 100 indicates acceleration.
However, the Ratio IQ has major technical problems — it is not comparable across different age levels unless the SD of the IQ distribution remains constant with age. For example, if a child reads at age 3 (CA) when an average child reads at age 6 (MA), their IQ = 3/6 × 100 = 200. For this reason, ratio IQ was replaced by Deviation IQ.
Deviation IQ is a standard score with a mean of 100 and an SD that approximates the SD of the Stanford-Binet IQ distribution. It compares people of the same age and assumes that IQ is normally distributed.
🔑 Definition — Deviation IQ: A standard score with mean = 100 and SD appropriate to the test, used to compare an individual’s performance to others of the same chronological age.
Relativity of Norms
Interest Comparisons: An IQ score should always be described by the name of the test on which it was obtained. The relative standing of IQs can change when different tests are used. An individual’s relative standing in different functions may be misrepresented by the lack of comparability of test norms. For example, if a verbal test was standardized on a random high school sample but a spatial test was standardized on a selected group, an examiner might erroneously conclude the individual is more verbally able than spatially able.
For longitudinal comparisons (scores on a specific test over time), differences may be due to three reasons:
- Content differences: Intelligence tests with the same label may have different content (e.g., one is verbal, another numerical).
- Scale units incomparability: IQ on one test may have SD of 12 while another has SD of 18.
- Standardization sample composition variation: The same individual will appear to perform better when compared with a less able group than with a more able group.
Scales of Measurement
To describe test scores quantitatively, tests must be designed to yield numeric results — either originally numeric or convertible to that form. Psychological measurement involves rules according to which objects are assigned numbers, and “quality” is expressed in numeric form.
For example, in a personality test, an item asks “do you like to be in the company of young age mates most of the time?” with answer options: “always”, “often”, “could not say”, “rarely”, and “never”. Since the response is qualitative, numbers are assigned (e.g., “never” = 1, “always” = 5). All responses can then be quantified and subjected to statistical treatment. In ability tests where every question has a right answer, the total number of correct responses yields the test score.
💡 Why this matters: Understanding scales of measurement is fundamental because the type of scale (nominal, ordinal, interval, ratio) determines what statistical analyses are appropriate and what interpretations can be made from test scores.
⭐ Key Takeaways
The most critical concepts from this lecture are that raw test scores are meaningless until converted into derived scores using norms, and there are two main categories: developmental norms (mental age, grade equivalents, ordinal scales) and within-group norms (percentiles, standard scores, T scores, Stanines, deviation IQ). Mental age has psychometric limitations because its units shrink with age, making ratio IQ problematic and leading to the adoption of deviation IQ (mean = 100) as a more psychometrically sound alternative. Percentiles provide easily interpretable relative position but are not equal-interval scales, while standard scores (z-scores, T scores, Stanines) allow precise comparison by expressing distance from the mean in standard deviation units. Finally, the relativity of norms means that test scores must always be interpreted in context — considering the specific test used, its standardization sample, and the scale properties — to avoid misinterpretation in both cross-sectional and longitudinal comparisons.
🧠 Quick Revision Questions
- What is the difference between developmental norms and within-group norms, and give one example of each?
- Why is ratio IQ considered technically problematic, and how does deviation IQ address this issue?
- How is a z-score calculated, and what does a z-score of 0, +2, and -1.5 indicate about a person’s performance?
- What are the major limitations of grade equivalents as a type of developmental norm?
- Explain the difference between percentage and percentile, and identify what the 25th, 50th, and 75th percentiles represent in a distribution.
📘 Lecture 8 — Test Norms and Related Concepts
📖 Overview: This lecture explores the concept of standardization in psychological testing through the establishment of test norms. It covers various types of norms—national, national anchor, specific, and fixed reference group norms—and introduces Item Response Theory as an advanced scaling method. Understanding these concepts is critical for correctly interpreting test scores and making valid judgments about test takers.
🗂️ Topics Covered
The lecture covers the process of standardization and its importance, the characteristics and selection of a normative sample including stratified random sampling, national norms based on representative national samples, national anchor norms using the equipercentile method for score equivalence, specific norms for narrowly defined populations, fixed reference group scoring systems using the SAT as a historical example, and finally Item Response Theory as a sample-free measurement approach.
📝 Lecture Summary
Standardization: Test Norms and Related Concepts
Norms are established for the sake of standardization of any test. Standardization is the process whereby a test is administered to a representative sample of population whom the test is meant for, for the sake of establishing norms. A standardized test is one that has normative data, as well as clearly specified administration and scoring procedures.
The Normative Sample
When using test scores, we must ensure that the norms being referred to are representative of the particular population from which the standardization or normative sample was selected. The mean scores of that sample are assumed to represent the parent population. Therefore, if the sample comprised women alone, their mean performance score should be used as a norm for women’s raw scores alone. It is advisable that the standardization sample include maximum characteristics of the population and be of a large enough size to ensure stable values. As the size of sample increases, the chance of making error reduces.
To obtain a true representative sample, careful sampling procedures must be followed. Proportionate stratified random sampling is the best approach, where members from each subgroup/stratum are included in the sample in the same proportion in which they are found in population. A more practical option is to take a purposive sample that we believe contains all characteristics we are interested in catering for. For example, if the standardization sample included children aged 12 to 16 years with six years of schooling, then the norms are meant for a population of children within the same age range and similar educational background. No test provides norms for all sorts of population altogether.
💡 Why this matters: Using tests standardized in western countries with local populations requires great caution, as available norms were not established for such populations.
National Norms
If a test is standardized using a nationally representative sample of the population, it is called a national sample. The sample containing all characteristics of interest is chosen from different geographical regions, communities, socioeconomic status, and institutions. For example, to establish norms for a test measuring achievement of university students in Pakistan, the normative sample must represent university students from all regions of the country.
National Anchor Norms
Many tests measure the same ability or human trait, but a person may obtain different scores on different tests. National anchor norms provide a solution to this problem by creating an equivalency table for scores on various tests of the same ability. Equivalency is calculated using the equipercentile method. Test scores are considered equal only when they have equal percentiles in the group being studied. For example, if a score of 34 on test ABC carries the 85th percentile, and a score of 29 on test XYZ also carries the 85th percentile, then 34 on ABC is equivalent to 29 on XYZ.
🔑 Definition — Equipercentile Method: Test scores are compared and their equivalence is determined with reference to their corresponding percentile scores.
📌 Example: A score of 34 on test ABC carries the 85th percentile, and a score of 29 on test XYZ also carries the same percentile. Therefore, 34 on ABC is equivalent to 29 on XYZ.
💡 Why this matters: National anchor norms should not be used as a single fully dependable source of judgment; every member of the sample must have taken all the tests whose equivalence is being determined.
Specific Norms
An alternate solution to the problem of non-equivalence is to use specific norms, which require that tests be standardized on more narrowly defined populations. Rather than using broadly defined samples, tests can be standardized on narrowly defined samples selected through purposive sampling. In case of using specific samples, the chances of controlling nonequivalence are reduced. When norms for such tests are reported, a clear report of the limits of the normative sample must also be made.
Highly specific norms are useful for most testing purposes. Even when representative norms are available, availability of separately reported subgroup norms is very helpful. For example, while one inventory can measure occupational variables in doctors as one community, separate subscales may be developed for doctors working under different levels of stress and in different working conditions. At times, institutions develop their own local norms. For example, a university may develop norms on its students entering the first year to predict achievement in following years.
Fixed Reference Group
Sometimes non-normative scales are used. One such type is the fixed reference group scoring system. In this system, the distribution of scores obtained from one group of people is used as the basis for calculation of test scores for future administrations. The group from which scores were obtained is called the fixed reference group. This system does not provide normative evaluation of performance but ensures comparability and continuity of scores.
The College Board Scholastic Aptitude Test (SAT) is an example. In 1941, approximately 11,000 candidates took the test. The distribution of scores of this sample was taken as a standard, and all SAT scores were expressed in terms of mean and standard deviation of these candidates. A score of 500 corresponded to the mean of this group; 600 meant one SD above mean, and 400 was one SD below. In 1995, a new fixed reference group began to be used, comprising those who took the SAT in 1990. After April 1, 1995, scores are reported on the "recentered" scale derived from the 1990 reference group.
📐 Formula: SAT scoring system → A score of 500 corresponds to the mean of the fixed reference group; 600 means one SD above the mean; 400 means one SD below the mean.
Item Response Theory
Item Response Theory (IRT) can be understood in terms of latent trait models. Beginning in the 1970s, psychologists became interested in mathematically sophisticated procedures for scaling the difficulty of test items. The availability of high-speed computers made such procedures possible. The basic measure used is the probability that a test taker with a talent trait succeeds on an item of specified difficulty. The latent traits are mathematically derived statistical constructs derived from empirically observed relations among test responses.
The term latent trait model was later replaced by Item Response Theory (IRT) because 'latent trait' created a false impression of a specific trait. The purpose of IRT models is to establish a "sample-free" scale of measurement that is uniform, applicable to individuals and groups having widely varying ability levels, and to test contents varying widely in difficulty levels. Rather than using the mean and standard deviation of a specific reference group, IRT models set origin and unit size in terms of data representing a wide range of ability and item difficulty, obtained from many samples rather than a single sample.
🔑 Definition — Latent Trait: Mathematically derived statistical constructs derived from empirically observed relations among test responses; there is no implication regarding the existence of the trait as such.
⭐ Key Takeaways
The establishment of appropriate norms through a representative standardization sample is fundamental to valid test interpretation. A normative sample must reflect the population characteristics and be of sufficient size, with proportionate stratified random sampling being the ideal method for selection. National anchor norms provide score equivalence across different tests using the equipercentile method, while specific and local norms allow for more precise interpretation within narrowly defined groups. The fixed reference group system, exemplified by the SAT, ensures score comparability over time without requiring normative evaluation. Finally, Item Response Theory offers a sophisticated, sample-free approach to measurement that overcomes limitations of traditional norm-referenced methods by using data from multiple samples across a wide range of ability levels.
🧠 Quick Revision Questions
- What is the purpose of standardization in psychological testing, and what are the key features of a standardized test?
- Describe proportionate stratified random sampling and explain why it is considered the best approach for selecting a representative normative sample.
- How does the equipercentile method work in establishing national anchor norms for score equivalence?
- What is the difference between specific norms and local norms, and when would each be most appropriately used?
- Explain how the fixed reference group scoring system was applied to the SAT, including the significance of the 1941 and 1990 reference groups.
📘 Lecture 09 — Nature and Uses: Domain Referenced Test Interpretation
📖 Overview: This lecture introduces domain-referenced tests as an alternative to norm-referenced tests for interpreting psychological test results. Instead of comparing individuals to a normative group, domain-referenced tests evaluate what a person knows or can do within a specific content domain, making them essential for assessing mastery in education, professional training, and certification contexts.
🗂️ Topics Covered
This lecture covers the definition and purpose of domain-referenced (criterion-referenced) tests compared to norm-referenced tests, including who determines the domain or criterion. It explores content meaning in test interpretation, the process of developing instructional objectives and test items, mastery testing as an all-or-none approach, the relation between domain-referenced and norm-referenced testing, and the use of cutoff scores and expectancy tables for decision-making in minimum qualification contexts.
📝 Lecture Summary
Nature and Uses: Domain Referenced Test Interpretation
So far we have discussed norms for interpreting test results, but norms are not the only way to assess and measure abilities. Tests that use normative data are norm-referenced tests, defined as “a test that evaluates each individual relative to a normative group” (Kaplan & Saccuzzo, 2001). In such tests, a representative sample is tested, raw scores are analyzed, and norms are established. Every individual’s performance is compared to the scores of others — average, below average, or above average relative to the normative sample.
There is another type of test that uses a criterion or standards to describe performance. A person must perform within a domain to be considered proficient in a skill, behavior, or ability. These are called domain-referenced tests. The terms criterion-referenced and domain-referenced are used interchangeably, with the latter being more common. A criterion-referenced test is “a test that describes the specific types of skills, tasks, knowledge of an individual relative to a well-defined mastery criterion. The content of criterion-referenced test is limited to certain well-defined objectives” (Kaplan & Saccuzzo, 2001).
Glaser (1963) first used the term “criterion-referenced testing.” Alternative terms like “domain-referenced” and “content-referenced” were also proposed. The interpretive frame of reference in a domain-referenced test uses a specific content domain rather than a population of persons. Test results are reported in terms of what the test taker knows, how proficient they are, and to what extent they have command over a certain content domain. For example, according to Anastasi (2007), performance may be reported in terms of specific arithmetic operations mastered, estimated vocabulary size, or difficulty level of reading matter comprehended.
🔑 Definition — Norm-Referenced Test: A test that evaluates each individual relative to a normative group. 🔑 Definition — Criterion-Referenced Test: A test that describes specific types of skills, tasks, or knowledge of an individual relative to a well-defined mastery criterion.
Who Decides and Determines the Domain or Criterion?
The domain is primarily derived from the values or standards of an individual or organization. In many real-life and professional scenarios, evaluating a person’s ability relative to a norm is meaningless; what matters is command over the domain. For example, in training surgeons, how much above or below average their skills are is not important — what matters is whether they have acquired the surgical skill to a specified extent. Similarly, in evaluating a pilot, the key question is whether the pilot can fly a plane and how well, not whether they are equal, above, or below others.
Norm-referenced tests assess and describe how well test takers performed in relation to other people. Domain-referenced tests tell us what test takers can do — they focus on potential, while norm-referenced tests focus on comparative performance.
In some contexts, domain-referenced tests are called mastery tests. These are used to assess mastery or achievement of certain skills or contents, with the focus on content rather than a specific population. They became very popular in educational contexts in the 1970s. Performance may be reported in terms of mastery over skills (e.g., specific arithmetic operations, vocabulary size, reading difficulty level), or chances of achieving a designated performance level on an external criterion (educational or occupational).
Domain-referenced tests are commonly used in education, especially in innovations like computer-assisted, computer-managed, and individualized, self-placed instructional systems. According to Anastasi and Urbina (2007), testing is closely integrated in these systems, with instruction introduced before, during, and after each unit to check prerequisite skills, diagnose learning difficulties, and prescribe subsequent instruction. Broad surveys like the National Assessment of Educational Progress in the U.S. also use domain-referenced tests.
Another application area is assessing mastery of a small number of clearly defined job skills, such as in military occupational specialties, or evaluating attainment of minimum requirements (e.g., qualifying for a driver’s or pilot’s license). Familiarity with domain-referenced testing concepts can also help improve traditional teacher-made tests.
💡 Why this matters: Understanding who defines the domain is critical — in high-stakes contexts like surgery or aviation, domain-referenced assessment ensures safety and competence, not just relative ranking.
Content Meaning
The most significant feature of domain-referenced tests is that test performance is interpreted in terms of content meaning. The goal is not relative standing but learning what the person knows and can do. In developing such tests, the content must be carefully chosen, treated, and presented.
A clearly defined domain of knowledge or skills to be assessed is the primary requirement. The content must be important and generally accepted as such. To develop items for assessing mastery, the content is subdivided into smaller units, each defined in performance terms — what behavior will indicate that a content element has been mastered. In educational settings, these units correspond to behaviorally defined instructional objectives. For example: “divides numbers carrying zeros by numbers carrying one zero by canceling one zero at the end,” “can convert grams into ounces,” or “can convert centigrade into Fahrenheit.”
Instructional objectives specify learning outcomes and affect how a course is taught and how assessments are made. After objectives are finalized, the difficult task of item development for each objective follows. Each item must be a good representative of the domain. Careful formulation of objectives, clear statement of concepts and methodologies, and the test developer’s expertise and judgment all matter.
Domain-referenced tests are most useful for testing basic skills like reading and arithmetic at elementary levels. According to Anastasi and Urbina (2007), an ordinal hierarchy is usually adopted for arranging instructional objectives for basic skills — elementary skills must be acquired before higher-level skills. At elementary levels, content and learning sequence are mostly flexible. It is not advisable to formulate highly specific objectives for advanced levels of knowledge in less structured subjects.
🔑 Definition — Instructional Objectives: Behaviorally defined learning outcomes that specify what a student should be able to do, affecting teaching and assessment methods.
Mastery Testing
Mastery testing is an important characteristic of domain-referenced tests. It provides an all-or-none score, meaning the test score indicates the presence or absence of mastery, or the attainment of a pre-established level of mastery. A generally expected level is complete mastery, often set at 80% or 85%.
Another way of reporting mastery is to use a three-way distinction: mastery, non-mastery, and an intermediate, doubtful, or “review” interval.
In mastery tests, individual differences are not a matter of concern. Many educators believe that if suitable instructional methods are used and enough time is given, almost everyone can exhibit complete mastery and achieve instructional objectives. The only area where individual differences may appear is the time taken to learn the content. Therefore, the effect of individual differences can be reduced to a minimum after appropriate training.
Mastery testing is used in many individualized instruction programs. Published domain-referenced tests of basic skills for elementary schools often employ this approach.
Two key issues in constructing mastery tests are: (1) how many items should be used, and (2) what proportion of items must be correct before a reliable assessment is made. Initially, these issues were handled by test developers’ judgment, but now statistical techniques are available.
🔑 Definition — Mastery Testing: A testing procedure that yields an all-or-none score indicating the presence or absence of mastery at a pre-established level (typically 80-85%).
Relation to Norm-Referenced Testing
While mastery testing has advantages, its usefulness is greatest for basic skills at elementary levels. In areas where complete mastery is not the focus, mastery testing is not the best choice. As we go to higher levels with more advanced and less structured subjects (e.g., understanding, critical thinking, appreciation, originality), there is no way to assess complete attainment or mastery — there is no end, limit, or direction to learning. In such cases, norm-referenced tests are useful because there are no cutoff points to show complete absence of these faculties.
Some published tests allow both norm-referenced and domain-referenced applications. For example, Stanford diagnostic tests in reading and mathematics provide both appropriate norms and a system for qualitative analysis of a child’s attainment of detailed instructional objectives.
Some authors note that even when using domain-referenced tests, the concept of norms is still operative — there is an underlying realization that a continuum of abilities exists.
Minimum Qualification and Cutoff Scores
Mastery tests may adopt an all-or-none approach or make a three-way distinction, but there are situations where clear cutoff scores must be specified. These include granting a driving or flying license, selecting workers for a war zone (requiring sharp learning and vision), selecting workers for a nuclear plant, or choosing students for medical school.
Different tests use their own cutoff scores. The following points should be kept in mind when using cutoff scores for decision making (Anastasi & Urbina, 2007): a) Do not use scores from a single test as cutoff — use a band of scores from more than one test. b) Use multiple sources of information for decision making, including relevant performance on other tests from past or present. c) If a panel of judges sets cutoff points, they should be experts in test construction and task performance areas. d) Cutoff scores should be established and verified with the support of empirical information whenever possible. e) Test scores should be obtained from groups that are clearly different from each other on the relevant criterion behavior.
An empirical method for setting cutoff scores is using expectancy tables.
🔑 Definition — Expectancy Table: A table containing the probability of different criterion outcomes for persons who obtain each test score, based on statistical information from past administrations.
📐 Formula/Concept — Expectancy Table Example:
| Score on XYZ | Number of students | Percentage in each grade |
|---|---|---|
| 50-60 | 12 | A: - , B: 1, C: 8, D: 4 |
| 60-70 | 15 | A: 3, B: 6, C: 6, D: - |
| 70-80 | 23 | A: 12, B: 9, C: 2, D: - |
This table shows the relationship between test scores and course grades, helping predict criterion outcomes based on test performance.
⭐ Key Takeaways
Domain-referenced tests focus on content mastery rather than comparative performance, making them essential for assessing specific skills in education, professional training, and certification contexts. The key feature is content meaning — interpreting what a person knows or can do within a well-defined domain. Mastery testing uses an all-or-none approach (typically 80-85% mastery) and is most useful for basic skills at elementary levels. However, for higher-level abilities like critical thinking and creativity, norm-referenced tests remain more appropriate. When cutoff scores are needed for minimum qualification decisions, multiple sources of information and expectancy tables should be used for empirical validation.
🧠 Quick Revision Questions
- What is the fundamental difference between norm-referenced and domain-referenced tests in terms of what they assess?
- According to the lecture, who determines the domain or criterion in domain-referenced testing, and why is this important in professional contexts like surgery or aviation?
- What are instructional objectives, and why must they be defined in performance terms for domain-referenced tests?
- What is mastery testing, and what does an “all-or-none score” mean in this context (including the typical mastery level)?
- What are the recommended guidelines for setting cutoff scores, and how can expectancy tables be used to establish them empirically?