π Lecture 6 β Purpose of Testing-I
π Overview: This lecture covers the fundamental purposes of educational testing, beginning with the various types of educational decisions that test data inform. It then classifies different types of written tests used in classroom assessment and concludes by contrasting two major assessment frameworks: Norm-Referenced Testing (NRT) and Criterion-Referenced Testing (CRT).
ποΈ Topics Covered
The lecture first explores nine types of educational decisions made at different levels (administrative, school, and classroom), including instructional, grading, diagnostic, selection, placement, program/curriculum, and administrative decisions. It then categorizes written tests into eight types based on format (verbal/non-verbal), scoring (objective/subjective), construction (teacher-made/standardized), and time constraints (power/speed). Finally, it introduces Norm-Referenced Assessment (NRT), which compares a studentβs performance to a peer group, and Criterion-Referenced Assessment (CRT), which measures mastery against a fixed standard or criterion.
π Lecture Summary
Topic-018: Educational Decisions Making
Educational decisions are classified into nine types taken at different levels: board/administrative, school management, or classroom by teachers. These decisions rely on test data.
π Instructional Decisions: The nuts and bolts decisions made daily by classroom teachers, such as deciding to spend more time on specific units, regroup students in class for better management, or modify instructional plans.
π Grading Decisions: Made by the classroom teacher but much less frequently than instructional decisions. For most students, grading decisions are the most influential decisions made about them.
π Diagnostic Decisions: Decisions made about a studentβs strengths and weaknesses and the reasons behind them. Teachers make these based on information from informal, teacher-made tests, but standardized tests can also provide diagnostic information.
π Selection Decisions: Involve test data used in part for accepting or rejecting applicants for admission into a group, program, or institution.
π Placement Decisions: Made after an individual has been accepted into a program. They determine where in a program someone is best suited to begin.
π Program or Curriculum Decision: A policy-level decision about whether a lesson, unit, or subject will continue or be abandoned for the next academic session, based on national objectives of education.
π Administrative Decisions: Policy decisions made at school, district, state, or national level. Based on measurement data, this includes financial decisions of schools.
Topic-019: Types of Test
In classroom assessment, different forms are utilized, each with its own benefits and disadvantages. The most common type is written assessment.
π Types of Written Tests: Verbal, Non-verbal, Objective, Subjective, Teacher Made, Standardized, Power, and Speed.
π Verbal tests: Emphasize reading, writing, or speaking. Most tests in education are verbal tests.
π Non-verbal tests: Do not require reading, writing, or speaking ability. Tests composed of numerals or drawings are examples.
π Objective tests: Refers to scoring where two or more scorers can easily agree on whether an answer is correct or incorrect. True/false, multiple choice, and matching tests are examples.
π Subjective tests: When it is difficult for two scorers to agree on whether an item is correct or incorrect. Essay tests are the example.
π Teacher Made tests: Constructed solely by a teacher to be used in the classroom. They are custom-designed according to the needs and issues of a specific class.
π Standardized tests: Constructed by measurement experts over a period of years. They measure broad national objectives and have a uniform set of instructions adhered to during each administration. Mostly, they have tables of norms to which a studentβs performance may be compared to determine where they stand in relation to a national sample of students at the same age or grade level.
π Power tests: Have liberal time limits that allow each student to attempt each item. Items tend to be difficult.
π Speed tests: Have time limits so strict that no one is expected to complete all items. Items tend to be easy.
Topic-020: Norm Referenced Assessment (NRT)
The general purpose of assessment is to gather information to make better and more informed decisions. The utility of that information differentiates among the types of assessments.
π Norm-Referenced Test (NRT): A type of test that tells us where a student stands compared to other students. It helps to determine a studentβs place or rank among a group of similar students.
Dimensions of NRT:
- It provides an estimate of ability in a variety of skills in a much shorter time.
- NRT tends to be general. It measures a variety of skills at the same time but fails to measure them thoroughly.
- It is hard to make decisions regarding the mastery of a studentβs skill in a subject.
- NRT is much more difficult for students to solve. On average, only 50% of students are able to get an item right in a test.
Topic-021: Criterion Referenced Assessment (CRT)
π Criterion-Referenced Test (CRT): A type of test that tells us about a studentβs level of proficiency in or mastery of some skill or set of skills. This is achieved by comparing a studentβs performance to a standard mastery called a criterion.
Dimensions of CRT:
- CRT tends to be specific. It measures a particular set of skills at one time and focuses on the level of achievement of that skill. CRT gives a clear picture regarding the mastery of a studentβs skill in a subject.
- It measures skill more thoroughly, so naturally, it takes more time compared to NRT in measuring the mastery of said skill.
- Items included in CRT are relatively easier. Around 80% of the students are expected to respond to an item correctly in the test.
- CRT compares studentsβ performance to the standards indicative of mastery.
- The breadth of content sampled is narrow and covers very few objectives.
β Key Takeaways
A student must understand that test data serves different educational decision-making purposes at various levels (classroom, school, policy). They must be able to classify tests into their eight written types and distinguish between them. The most critical distinction is between Norm-Referenced Tests (NRT), which rank students against each other (e.g., β50% get it rightβ), are general, and assess broad skills, and Criterion-Referenced Tests (CRT), which measure mastery of a specific skill against a standard (e.g., β80% get it rightβ), are specific, and provide a clear picture of individual proficiency. Remember also that teacher-made tests are custom for a class, while standardized tests have national norms.
π§ Quick Revision Questions
- What is the main difference between a diagnostic decision and a placement decision?
- List three types of written tests that are based on who constructs them (rather than format or scoring).
- Explain the key difference between a power test and a speed test, including the typical difficulty of items in each.
- A test reports that a student scored better than 85% of other students in their grade. Is this likely from a Norm-Referenced Test (NRT) or a Criterion-Referenced Test (CRT)? Why?
- If a teacher wants to know if a student has fully mastered the skill of dividing fractions, which type of assessment (NRT or CRT) would be more appropriate, and why?
π Lecture 7 β Purpose of Testing βII
π Overview: This lecture explores the distinctions between criterion-referenced and norm-referenced testing, examining their unique characteristics, item selection, and scoring approaches. It also introduces formative assessment as a tool for ongoing feedback during the instructional process, which is crucial for both student and instructor growth.
ποΈ Topics Covered
The lecture covers three main topics: the characteristics of criterion-referenced assessment (CRT), a detailed comparison between criterion-referenced tests (CRT) and norm-referenced tests (NRT) across six bases of comparison, and the definition and types of formative assessment used during the learning process.
π Lecture Summary
Topic- 022: Characteristics of Criterion Referenced Assessment
Criterion Referenced Assessment (CRT) evaluates an examineeβs performance against a predefined external standard of competence, rather than comparing it to other examinees. Sampled content in CRT is much more comprehensive, usually three or more items are used to cover a single objective. The meaning of the score does not depend upon comparison with other scores; it flows directly from the connection between the items and the criterion. Items are chosen to reflect the criterion behavior, with emphasis placed upon the domain of relevant responses. Scoring uses the number succeeding or failing or range of acceptable performance.
π Definition β Criterion Referenced Assessment (CRT): A test where an examinee's performance is compared to an external standard of competence, focusing on mastery of specific objectives.
π‘ Why this matters: CRT tells you exactly what a student knows and can do relative to a set of standards, not how they compare to peers.
π Example: "90% proficiency achieved" means the student answered 90% of items correctly on the criterion, or "80% class reached 90% proficiency" means that 80% of students in the class achieved the 90% mastery level.
Topic- 023: Difference between NRT and CRT
This topic compares Norm Referenced Tests (NRT) and Criterion Referenced Tests (CRT) across six bases of comparison:
-
Comparison Targets: In CRT, the examineeβs performance is compared to an external standard of competence. While in NRT, examineeβs performance is typically compared to that of other examinees.
-
Selection of Items: Items included in CRT are of specific nature and designed for the student skilled in particular subjects. In NRT, items are of general knowledge nature; the student should be able to answer it, but superficial knowledge is sufficient to respond to the item correctly.
-
Meaning of Success: In CRT, an examinee is classified as a master or non-master. There is no limit to the number of pass or fail. In NRT, an examineeβs opportunity for success is relative to the performance of the other individuals who take the test.
-
Average Item Difficulty: In CRT, the average item difficulty is fairly high because examinees are expected to show mastery. In NRT, the average item difficulty is lower because tests are designed to spread out examinees and provide a reliable ranking.
-
Score Distributions: In CRT, a plot of the resulting score distribution will show most of the scores clustering near the high end of the score scale. In NRT, a broader spread of scores is expected, with a few examinees earning very low or high scores and many earning medium scores.
-
Reported Scores: In CRT, classification of the examinee is measured as master/non-master or pass/fail. In NRT, percentile ranks or scale scores are frequently used.
π Definition β Norm Referenced Test (NRT): A test where an examinee's performance is compared to that of other examinees, designed to rank students and spread out scores.
π Definition β Master/Non-master: In CRT, an examinee is classified as a master (has met the criterion standard) or non-master (has not met the criterion standard), with no limit on how many can pass or fail.
Topic- 024: Formative Assessment
Formative assessment provides feedback and information during the instructional process, while learning is taking place and occurring. It measures student progress, but it can also assess your own progress as an instructor.
π‘ Why this matters: Unlike summative assessment (which happens at the end), formative assessment happens during learning, allowing for real-time adjustments to teaching and learning strategies.
Types of Formative Assessment include:
- Observations during in-class activities, including studentβs non-verbal feedback during lecture
- Homework exercises as review for exams and class discussions
- Reflection journals that are reviewed periodically during the semester
- Question and answer sessions, both formal (planned) and informal (spontaneous)
- Conferences between the instructor and student at various points in the semester
- In-class activities where students informally present their results
- Student feedback collected by periodically answering specific questions about the instruction and their self-evaluation of performance and progress
β Key Takeaways
CRT assesses mastery against a fixed standard using specific, comprehensive items with high difficulty, reporting pass/fail results. NRT compares performance to other test-takers using general items with lower difficulty to create a wider score distribution, reporting percentile ranks. Formative assessment is a continuous feedback loop during instruction, using various methods like observations, homework, journals, Q&A sessions, and conferences to improve both student learning and teaching effectiveness.
π§ Quick Revision Questions
- On which basis of comparison do CRT and NRT differ regarding the selection of test items?
- In a CRT, what does it mean if a class reports "80% of students achieved 90% proficiency"?
- Why is average item difficulty fairly high in CRT but lower in NRT?
- What is the primary purpose of formative assessment during instruction?
- List three types of formative assessment activities mentioned in the lecture.
π Lecture 8 β Purpose of Testing-III
π Overview: This lecture continues exploring the purpose of testing by focusing on formative and summative assessment. It explains the functions of each, how they differ in timing, focus, and purpose, and why understanding these distinctions is crucial for effective teaching and learning.
ποΈ Topics Covered
This lecture covers the functions of formative assessment, the definition and types of summative assessment, and the functions of summative assessment. It explains how formative assessment focuses on the learning process with ongoing feedback, while summative assessment evaluates the final product for grading and certification.
π Lecture Summary
Functions of Formative Assessment
Formative assessment focuses on a predefined segment of instruction, meaning it only tests what was just taught. A limited sample of learning tasks are addressed, not the entire course content. The difficulty of items varies with each segment of instruction, so early segments may have easier items. Formative assessment is conducted periodically during the instructional process, not just at the end. Its results are used to improve and direct learning through ongoing feedback, helping students correct mistakes immediately.
π Definition β Formative Assessment: Assessment conducted during instruction to monitor student learning and provide ongoing feedback for improvement.
Summative Assessment
Summative assessment takes place after the learning has been completed and provides information and feedback that sums up the teaching and learning process. Typically, no more formal learning is taking place at this stage, other than incidental learning through completing projects and assignments. Summative assessment is more product-oriented and assesses the final product, whereas formative assessment focuses on the process toward completing the product. Once the project is completed, no further revisions can be made. If students are allowed to make revisions, the assessment becomes formative.
π Definition β Summative Assessment: Assessment that evaluates student learning at the end of an instructional unit by comparing it against some standard or benchmark.
π Example: A final exam is summative because it tests the entire course after instruction ends. If a teacher allows students to revise their answers after seeing the correct ones, it becomes formative.
Types of Summative Assessment include:
- Examinations (major, high-stakes exams)
- Final examinations (a truly summative assessment)
- Term papers (drafts submitted during the semester would be a formative assessment)
- Projects (project phases submitted at various completion points could be formatively assessed)
- Portfolios (could also be assessed during its development as a formative assessment)
- Performances
π‘ Why this matters: The key distinction is that the same assignment (e.g., a term paper) can be either formative or summative depending on when and how it is assessed. Drafts are formative; final submissions are summative.
Functions of Summative Assessment
The focus of measurement in summative assessment is on course or unit objectives. A broad sample of all objectives is used, unlike formative assessment which samples only a segment. This type of assessment uses a wide range of difficulty while selecting items for the test. Summative assessment is done at the end of the unit or the course. The most important functions of summative assessment are to assign grades, provide certification of accomplishment, and allow evaluation of teaching.
π Example: A final exam for a semester-long course will include easy, medium, and hard questions covering all 10 units, not just the last unit. The results determine the final grade and whether the student can proceed to the next level.
β Key Takeaways
Formative assessment is conducted during instruction to improve learning through feedback, while summative assessment occurs after instruction to evaluate final outcomes. Formative assessment focuses on a limited segment of instruction with varying item difficulty, while summative assessment covers all objectives with a broad range of difficulty. The same activity (like a term paper or portfolio) can be either formative or summative depending on whether revisions are allowed. The main functions of summative assessment are grading, certification, and teaching evaluation, whereas formative assessment aims to guide ongoing learning.
π§ Quick Revision Questions
- What is the key difference in when formative and summative assessments are conducted?
- List three functions of formative assessment.
- Can a term paper be both a formative and summative assessment? Explain.
- What is the focus of measurement in summative assessment?
- Why does item difficulty vary in formative assessment but use a wide range in summative assessment?
π Lecture 9 β Table of Specification
π Overview: This lecture introduces the Table of Specification, a critical tool for developing test blueprints that ensure content validity and balanced assessment. It explains how teachers systematically allocate questions across content areas and Bloom's Taxonomy cognitive levels, providing practical examples for weighting objectives and constructing a balanced test.
ποΈ Topics Covered
The lecture covers the concept and definition of Table of Specification, its two main functions in test construction, the six elements and appropriateness checklist for developing a comprehensive table of specification, the process of balancing learning objectives through instruction time calculations, and a practical step-by-step example demonstrating how to construct a Table of Specification with proper weightage distribution.
π Lecture Summary
Topic- 028: Table of Specification
The Table of Specification is a tool used by teachers to develop a blueprint for the test. It is the first formal step in test construction and serves as the technical name for the test blueprint.
The blueprint is meant to ensure content validity, which is the most important factor in constructing an achievement test. A unit test or comprehensive exam is based on several lessons/chapters and should reflect a balance between content areas and learning levels (objectives).
π Definition β Table of Specification: A two-way chart or grid relating instructional objectives to instructional content.
π‘ Why this matters: Without a Table of Specification, teachers may unintentionally over-emphasize certain topics while neglecting others, compromising the test's validity.
The Table of Specification performs two important functions:
- It ensures the balance and proper emphasis across all content areas covered by the teacher.
- It ensures the inclusion of items at each level of the cognitive domain of Bloom's Taxonomy.
Topic- 029: Concept of Table of Specification
The Table of Specification helps a teacher in allotting questions to different content areas and Bloom's learning categories in a systematic manner. It ensures that every content area receives appropriate representation and that questions target various cognitive levels appropriately.
Topic- 030: Elements and Appropriateness in Table of Specification
Carey (1988) listed six major elements that should be attended to in developing a Table of Specifications for a comprehensive end-of-unit exam:
i. Balance among the goals selected for the exam (weighing objectives) ii. Balance among the levels of learning (higher order and lower order) iii. The test format (objective and subjective) iv. The total number of items v. The number of test items for each goal and level of learning vi. The enabling skills to be selected from each goal framework
A Table of Specifications incorporating these six elements will result in a "comprehensive posttest that represents each unit and is balanced by goals and levels of learning."
Checklist for Appropriateness of Table of Specification:
- Are the specifications in harmony with the purpose of the test?
- Do the specifications indicate the nature and limits of the achievement domain?
- Do the specifications indicate the types of learning outcomes to be measured?
- Do the specifications indicate the sample of learning outcomes to be measured?
- Is the number of test items indicated for the total test and for each subdivision?
- Are the types of items to be used appropriate for the outcomes to be measured?
- Is the difficulty of the items appropriate for the types of interpretation to be made?
- Is the distribution of items adequate for the types of interpretation to be made?
- If sample items are included, do they illustrate the desired attributes?
- Do the specifications, as a whole, indicate a representative sample of instructionally relevant tasks that fits the use to be made of the results?
Topic- 031: Balance Among Learning Objectives and Their Weighting in Table of Specification
In developing a test blueprint, it is necessary to select learning objectives. Some objectives are more important because more instruction time is spent on them, while others are less important. Therefore, we need to weigh the learning objectives for calculating their relative weightage in the test.
Step 1: Instruction Time To calculate instruction time for columns of the Table of Specifications, the teacher must use the following formula:
π Formula: Percentage of instruction time = Time spent on objective (min) / Total time for instruction being examined (min)
Example: If 250 minutes were spent on a particular objective out of 1000 total minutes of instruction: Percentage of instruction time = 250/1000 Percentage of instruction time = 25%
Step 2: Examination Value The instructor should determine the number of test items/score to be allocated to that objective. Let us assume total marks of the test are 100. Then 25 marks should be allocated to questions related to that objective.
Step 3: Validation Percent of instruction time = Percent of examination value (within Β±2 percent, if not, redo test) 25 Β± 2 = 25 Β± 2
If total marks of the test are 50, then 25% of 50 = 12.5 marks.
π Formula: Point total of questions for objective / Total points on examination = % of examination value
Topic- 032: Balance Among Learning Objectives and Their Weight in Table of Specification: Example
This section provides a practical example of developing a Table of Specification.
Initial Table with Time Weightage:
| Topics/Level | Knowledge | Comprehension | Application | Marks |
|---|---|---|---|---|
| Pakistan Movement (100/500Γ100 = 20%) | ||||
| Geography of Pakistan (150/500Γ100 = 30%) | ||||
| Climate Change (150/500Γ100 = 20%) | ||||
| Industries (50/500Γ100 = 10%) | ||||
| Economy (50/500Γ100 = 10%) | ||||
| Total (Time: 500/Marks: 50) |
Step 1: Distribution of Marks for Each Topic (50-mark test):
| Topics/Level | Knowledge | Comprehension | Application | Marks |
|---|---|---|---|---|
| Pakistan Movement | 10 (20%) | |||
| Geography of Pakistan | 15 (30%) | |||
| Climate Change | 15 (30%) | |||
| Industries | 5 (10%) | |||
| Economy | 5 (10%) | |||
| Total (Time: 500/Marks: 50) | 50 (100%) |
Step 2: Distribution by Cognitive Level According to Bloom's Taxonomy:
| Topics/Level | Knowledge | Comprehension | Application | Marks |
|---|---|---|---|---|
| Pakistan Movement | 5 (50%) | 2 (20%) | 3 (30%) | 10 (20%) |
| Geography of Pakistan | 2 (10%) | 6 (40%) | 7 (50%) | 15 (30%) |
| Climate Change | 7 (50%) | 8 (50%) | 15 (30%) | |
| Industries | 1 (10%) | 1 (20%) | 3 (70%) | 5 (10%) |
| Economy | 1 (20%) | 1 (20%) | 3 (60%) | 5 (10%) |
| Total | 9 (18%) | 17 (34%) | 24 (48%) | 50 (100%) |
π Example Explanation: In the Pakistan Movement topic (worth 10 marks), the teacher allocated 5 marks (50%) to Knowledge level questions, 2 marks (20%) to Comprehension level, and 3 marks (30%) to Application level. The overall test has 18% Knowledge questions, 34% Comprehension questions, and 48% Application questions.
π‘ Why this matters: This systematic distribution ensures the test measures both lower-order and higher-order thinking skills appropriately across all content areas.
β Key Takeaways
A Table of Specification is a blueprint that ensures content validity by systematically distributing test items across content areas and Bloom's Taxonomy cognitive levels. The six elements from Carey (1988) provide a comprehensive framework including balance among goals, learning levels, test format, total items, items per goal, and enabling skills. The weighting of objectives is determined by calculating the percentage of instruction time spent on each topic using the formula (time on objective / total instruction time), and this percentage should be reflected in the examination marks with a Β±2% tolerance. The practical example demonstrates that for a 50-mark test with 500 total minutes of instruction, each topic receives marks proportional to its instruction time, and these marks are further distributed among Knowledge, Comprehension, and Application levels according to the teacher's judgment of each topic's cognitive demands. Finally, the appropriateness checklist helps teachers evaluate whether their Table of Specification aligns with the test purpose, achievement domain, learning outcomes, item types, difficulty levels, and overall representativeness of instruction.
π§ Quick Revision Questions
-
What are the two main functions of a Table of Specification in test construction?
-
How do you calculate the percentage of instruction time for a specific objective, and what is the acceptable tolerance when converting this to examination marks?
-
List the six major elements that Carey (1988) identified for developing a comprehensive Table of Specifications.
-
In the practical example, if the Geography of Pakistan topic received 30% of instruction time and the test is worth 50 marks, why did it receive 15 marks, and how were these marks distributed across the three cognitive levels?
-
According to the appropriateness checklist, what questions should a teacher ask to ensure the Table of Specification's item types are suitable for the outcomes being measured?
π Lecture 10 β SELECTION OF TEST
π Overview: This lecture explains how to select appropriate published tests for educational purposes. It covers the types of published tests available, the standards for selecting them, and the importance of fairness in test selection. This is critical because using the wrong test can lead to invalid assessments and unfair outcomes.
ποΈ Topics Covered
The lecture covers selecting pre-designed published tests, including achievement tests, aptitude tests, reading tests, readiness tests, and placement tests. It then details standards for selecting appropriate tests across two parts, focusing on defining purpose, investigating sources, reading materials, understanding test development, reading evaluations, examining specimen sets, and ensuring skill availability. Finally, it addresses fairness in test selection, including content sensitivity, performance review, identifying bias, and accommodating handicapped test takers.
π Lecture Summary
Topic- 033: Selecting Pre-designed
Published tests are designed and conducted in such a manner that each and every characteristic is pre-planned and known. There are many published tests available for school use. The two most valuable to the instructional program are: i. Achievement tests ii. Aptitude tests
There are hundreds of tests available for each type. Selecting the most appropriate one is an important task. In some cases, published tests are used by teachers. But more frequently, these are used by provincial or national testing programs.
In classrooms, the most used published tests are: i. Achievement tests ii. Reading tests
Published tests commonly used by provincial or national testing programs are:
- Aptitude tests
- Readiness tests
- Placement tests
Topic- 034: Standards for Selecting Appropriate Test βI
Test users should select tests that meet the purpose for which they are to be used and that are appropriate for the intended population.
- First, define the purpose for testing and the population to be tested and select the test accordingly.
- Investigate the potentially useful sources of information, in addition to the test scores, to validate the information provided by tests.
- Read the materials provided by test developers and avoid using tests for which unclear or incomplete information is provided.
- Become familiar with how and when the test was developed and tried out.
π Definition β Standards for Selecting Appropriate Test (Part I): The process of selecting tests by defining purpose and population, validating information from other sources, reading developer materials, and knowing test development history.
Topic- 035: Standards for Selecting Appropriate Test βII
Test users should select tests that meet the purpose for which they are to be used and that are appropriate for the intended population.
- Read independent evaluations of a test and of possible alternative measures.
- Examine specimen sets, disclosed tests or sample questions, directions, answer sheets, manuals, and score reports before selecting the tests.
- Select and use only those tests for which the skills needed to administer the test and interpret scores correctly are available.
π Definition β Standards for Selecting Appropriate Test (Part II): The process of selecting tests by reading independent evaluations, examining specimen sets, and ensuring available skills for administration and score interpretation.
Topic- 036: Fairness in Selecting Appropriate Test
- Evaluate the procedures used by test developers to avoid potentially insensitive content or language.
- Review the performance of test takers of different races, gender, and ethnic groups when samples of sufficient size are available.
- Evaluate the extent to which the performance differences may have been caused by inappropriate characteristics of the test.
- Use appropriately modified forms of tests or administration procedures for test takers with handicapping conditions.
π Definition β Fairness in Selecting Appropriate Test: The practice of evaluating test content for sensitivity, reviewing performance across demographic groups, identifying bias, and accommodating test takers with disabilities.
π‘ Why this matters: Fairness ensures that test scores reflect true abilities rather than irrelevant factors like language bias or disability limitations, leading to more valid and equitable assessments.
β Key Takeaways
Selecting the right test requires defining the testing purpose and population first. Teachers must investigate validation sources, read developer materials, and understand the testβs development history. They should read independent evaluations and examine full specimen sets before deciding. Fairness is essential: test users must review content for insensitivity, analyze performance across groups, identify bias, and accommodate handicapped test takers with modified forms or procedures. Remember: the most commonly used published tests in classrooms are achievement and reading tests, while provincial/national programs use aptitude, readiness, and placement tests.
π§ Quick Revision Questions
- What are the two most valuable types of published tests for the instructional program?
- List the three standards given in Topic-034 for selecting an appropriate test.
- According to Topic-035, what should test users examine before selecting a test?
- What four steps must be taken to ensure fairness in test selection?
- Name the three types of published tests commonly used by provincial or national testing programs.
π Lecture 11 β Characteristics of a Good Test-I
π Overview: This lecture introduces the three essential characteristics of a good test: validity, reliability, and usability. It explains the nature of validity in detail and discusses content validity as one of the three main evidences of validity. Understanding these concepts is critical for evaluating and designing effective assessments.
ποΈ Topics Covered
The lecture covers the three essential characteristics of a good testβvalidity, reliability, and usabilityβfollowed by an in-depth examination of the nature of validity, including its definition as a matter of degree, specificity, and unitary concept. It then discusses content validity as evidence of validity, including its meaning, procedure, and methods for establishing it through classroom instruction, achievement domains, and instructional priorities.
π Lecture Summary
Topic- 037: Characteristics of Good Test: Validity, Reliability and Usability
The most essential characteristics of a good test are validity, reliability, and usability. Validity is an evaluation of the adequacy and appropriateness of the interpretation and uses of results. It determines if a test is measuring what it intended to measure. Reliability refers to the consistency of assessment results. A key relationship exists between reliability and validity: reliability of measurement is needed to obtain valid results, but we can have reliability without validity. Reliability is a necessity but not a sufficient condition for validity.
π Definition β Validity: An evaluation of the adequacy and appropriateness of the interpretation and uses of results; determines if a test is measuring what it intended to measure.
π Definition β Reliability: The consistency of assessment results.
π‘ Why this matters: A test can be reliable (producing consistent scores) but not valid (not measuring the right thing), but a valid test must always be reliable.
Usability refers to the practical requirements of an assessment procedure beyond validity and reliability. These include feasibility, administration environment, and availability of results for decision makers.
π Definition β Usability: Practical requirements of an assessment procedure including feasibility, administration environment, and availability of results.
Topic- 038: Nature of Validity
The nature of validity is described by five key points. First, validity is referred to as "validity of test" but it is in fact validity of the interpretation and use to be made of the results. Second, validity is a matter of degree; it does not exist on an all or none basis. It is best considered in terms of categories that specify degree, such as high, moderate, or low validity. Third, validity is specific to some particular use or interpretation; no assessment is valid for all purposes. For example, an arithmetic test may have a high degree of validity for computational skill and a low degree for arithmetical reasoning. Fourth, validity is a unitary concept; it does not have different types but is viewed as a unitary concept based on different kinds of evidences. Fifth, validity involves an overall evaluative judgment; it requires an evaluation in terms of the consequences of interpretations and uses of assessment results.
π Definition β Validity as a matter of degree: Validity is not all-or-none; it exists on a continuum (high, moderate, or low).
π Definition β Validity as specific to use: No single assessment is valid for all purposes; validity depends on the intended use or interpretation.
π Example: An arithmetic test may have high degree of validity for computational skill and low degree for arithmetical reasoning.
Topic- 039: Evidences of Validity: Content Validity
There are three evidences of validity: content, construct, and criterion. Content validity refers to how well the sample of assessment tasks represents the domain of the tasks to be measured. The procedure for establishing content validity involves comparing the assessment tasks to the specifications describing the task domain under consideration. The method for establishing content validity involves three steps:
- Classroom instruction determines which intended learning outcomes (objectives) are to be achieved by students.
- Achievement domain specifies and delimits a set of instructionally relevant learning tasks to be measured by an assessment.
- Instructional and assessment priorities specify the relative importance of learning objectives to be assessed.
π Definition β Content validity: How well the sample of assessment tasks represents the domain of the tasks to be measured.
π‘ Why this matters: Content validity ensures that a test actually covers the material it was designed to measure, not just a limited or irrelevant subset.
π Example: If a teacher teaches four units (A, B, C, D) and gives equal instructional time to each, but the test has 80% of questions from unit A and only 5% from each other unit, the test would have low content validity because it does not represent the full domain of instruction.
β Key Takeaways
The three essential characteristics of a good test are validity, reliability, and usability, with validity being the most critical for ensuring a test measures what it intends to measure. Reliability is necessary but not sufficient for validityβa test can be reliable without being valid, but cannot be valid without being reliable. Validity is a matter of degree, specific to particular uses, and a unitary concept requiring overall evaluative judgment. Content validity is established by comparing assessment tasks to the full domain of learning outcomes, and its method involves aligning classroom instruction, achievement domains, and instructional priorities.
π§ Quick Revision Questions
- What are the three essential characteristics of a good test?
- What is the relationship between reliability and validity?
- Why is validity considered "a matter of degree" rather than all-or-none?
- Explain why an arithmetic test could have high validity for computational skill but low validity for arithmetical reasoning.
- What are the three steps in the method for establishing content validity?
π Lecture 12 β Characteristics of a Good Test-II
π Overview: This lecture continues the exploration of test quality by focusing on three essential types of validity: construct validity, criterion validity, and consequence validity. These concepts are crucial for ensuring that assessments accurately measure what they claim to measure and produce fair, meaningful results for both students and educators.
ποΈ Topics Covered
This lecture covers three main types of validity evidence. First, it explains construct validity, including its meaning, procedure for development using a test framework, and methods of confirmation through expert judgment and factor analysis. Second, it explores criterion validity, distinguishing between concurrent and predictive validity, along with the procedure and statistical method for verification. Finally, it discusses consequence validity, focusing on how assessment results impact teaching and learning, including considerations and factors within the test itself that can undermine validity.
π Lecture Summary
Topic- 040: Evidences of Validity: Construct Validity
Construct validity addresses how well a test measures up to its claims. For example, a test designed to measure depression must only measure that particular construct, not closely related ideas such as anxiety or stress.
The procedure for establishing construct validity involves developing a test framework, which includes:
- Defining the construct
- Identifying sub-constructs
- Listing indicators of each sub-construct
- Writing test items for each indicator
A detailed example is provided for the construct of Essay Writing:
| Sub-construct | Meaning/Scope | Indicators |
|---|---|---|
| Introduction Paragraph | It introduces the main idea, captures the interest of reader, and tells why topic is important. | 1. Single sentence called the thesis statement is written. 2. Background information about your topic provided. 3. Definitions of important terms written. |
| Supporting Paragraphs | Supporting paragraphs make up the main body of your essay. | 1. List the points about the main idea of the essay. 2. Write a separate paragraph for each supporting point. 3. Develop each supporting point with facts, details, and examples. |
| Summary Paragraph | Concluding paragraph comes after you have finished developing your ideas. | 1. Restate the strongest points. 2. Restate the main idea. 3. Give your personal opinion or suggest a plan for action. |
There are two methods to confirm construct validity of a test:
- Expert judgment: Experts in the field assess the construct validity of the table, and the table is revised under their guidance.
- Factor analysis: Questions are grouped by keeping in view the responses of respondents on them.
Topic- 041: Evidences of Validity: Criterion Validity
Criterion validity demonstrates the degree of accuracy of a test by comparing it with another test, measure, or procedure which has been demonstrated to be valid.
There are two types of criterion validity:
- Concurrent validity: This approach allows one to show the test is valid by comparing it with an already valid test.
- Predictive validity: It involves testing a group of subjects for a certain construct, and then comparing them with results obtained at some point in the future.
The procedure compares assessment results with another measure of performance obtained at a later date (for prediction) or with another measure of performance obtained concurrently (for estimating present status).
The method for determining criterion validity involves statistically correlating the two sets of scores. The resulting correlation coefficient provides a numerical summary of the relationship.
π Formula: Correlation coefficient β A numerical summary indicating the degree of relationship between two sets of scores. π Example: If a new test for job aptitude is administered, and the scores are correlated with actual job performance ratings obtained six months later, the resulting correlation coefficient indicates the predictive validity of the test.
Topic- 042: Evidences of Validity: Consequence Validity
Consequence validity concerns how well the use of assessment results accomplishes intended purposes and avoids unintended effects.
The procedure involves evaluating the effects of the use of assessment results on teachers and students. Both the intended positive effects (e.g., increased learning) and possible unintended negative effects (e.g., dropout of school) need to be evaluated.
Key considerations include:
- Does the assessment artificially constrain the focus of students' study?
- Does the assessment encourage or discourage exploration and creative modes of expression?
Factors in the test or assessment itself that can undermine validity include:
- Unclear directions
- Reading vocabulary and sentence structure too difficult
- Ambiguity
- Inadequate time limits (construct irrelevant variance)
- Overemphasis of easy-to-access aspects of domain at the expense of important, but hard-to-access aspects
- Test items inappropriate for the outcomes being measured
- Poorly constructed test items
- Test too short
- Improper arrangement of items
- Identifiable pattern of answers
π‘ Why this matters: Consequence validity reminds us that even if a test is technically accurate, its use can have harmful side effects that must be monitored and addressed.
β Key Takeaways
The most critical points to remember from this lecture are that construct validity requires a systematic framework defining the construct, its sub-constructs, and indicators before writing test items, and it is confirmed through expert judgment or factor analysis. Criterion validity is established by comparing a test against an already valid measure, with concurrent validity for present status and predictive validity for future performance, using a correlation coefficient to describe the relationship. Consequence validity evaluates both the intended positive and unintended negative effects of assessment use on students and teachers, and many factors within the test itself, such as unclear directions, ambiguity, and inadequate time limits, can undermine overall validity. Finally, a good test must satisfy all three types of validity evidence to be considered truly valid for its intended purpose.
π§ Quick Revision Questions
- What is the difference between construct validity and criterion validity?
- What are the two methods used to confirm construct validity, and how do they differ?
- Explain the difference between concurrent validity and predictive validity, and provide an example of each.
- What is the purpose of consequence validity, and what two types of effects must be evaluated?
- List at least four factors within the test itself that can undermine validity according to the lecture.