π Lecture 31 β Analyzing the Test-II
π Overview: This lecture explores the two major test theories used in item analysis: Classical Test Theory (CTT) and Item Response Theory (IRT). It focuses on the practical application of CTT, specifically how to calculate and interpret item difficulty (p-value) and item discrimination (D) to evaluate and improve test items. Understanding these concepts is crucial for creating valid and reliable assessments.
ποΈ Topics Covered
This lecture begins by introducing Classical Test Theory (CTT) and Item Response Theory (IRT) as frameworks for item analysis. It then delves into the CTT concept of item difficulty, explaining how to calculate the p-value and determine optimal difficulty levels for different item types, including examples. The final major topic is item discrimination, covering its definition, calculation using the upper and lower 27% method, and guidelines for interpreting the discrimination index.
π Lecture Summary
Topic- 107: Test Theories in Item Analysis
This section introduces two fundamental theories for analyzing test items. Classical Test Theory (CTT) is described as having relatively weak theoretical assumptions, making it easy to apply. Its major focus is on test-level information, but it also includes item statistics like difficulty and discrimination. In contrast, Item Response Theory (IRT) presents a more complex model that expresses the association between an individual's response to an item and an underlying latent variable, known as ability (ΞΈ) . This is a continuous, one-dimensional construct, where people at higher levels of the latent trait have a higher probability of responding correctly.
π Definition β Classical Test Theory (CTT) : A test theory with relatively weak theoretical assumptions that focuses on test-level information and item statistics like difficulty and discrimination. π Definition β Item Response Theory (IRT) : A model that expresses the association between an individual's response to an item and the underlying latent variable (ability or trait), often denoted as theta (ΞΈ).
Topic- 108: Item Difficulty in Classical Test Theory (CTT)
This topic explains how CTT measures item difficulty. Within CTT, the difficulty of an item is not a theoretical measure of an individual's ability but an empirical statistic based on a group of examinees. This statistic, known as the p-value or item difficulty index, is the proportion of examinees who answered the item correctly. The p-value ranges from 0.00 to 1.00; higher values (closer to 1.00) indicate easier items, while lower values (closer to 0.00) indicate more difficult items. Values between 0.4 and 0.6 are considered moderate.
π Definition β Item Difficulty Index (p-value) : The proportion of examinees who answered an item correctly, used as an index for item difficulty in CTT. π Formula: p-value = (Number of students who got the item correct) / (Total number of students who answered the item) π Example: For a four-alternative, multiple-choice item, the random guessing level is 1.00/4 = 0.25. The optimal difficulty level is calculated as: 0.25 + (1.00 - 0.25) / 2 = 0.62. P-values above 0.90 are very easy items, and those below 0.20 are very difficult. The optimal difficulty also depends on the number of choices; for instance, it is 0.75 for true/false items and 0.60 for five-alternative multiple-choice items.
Topic- 109: Item Discrimination in CTT
This section describes item discrimination, which is the ability of an item to differentiate between higher-ability and lower-ability examinees. The index ranges from -1.00 to +1.00, with higher positive values indicating better discrimination. A highly discriminating item is one where students who scored high on the total exam also got the item correct, and low-scoring students got it wrong. Values near or below zero are unacceptable and suggest the item is flawed. The discrimination index (D) can be calculated by ranking all students by total test score, then selecting the top 27% and bottom 27%. The index is the difference in the proportion of correct responses between these two groups.
π Definition β Item Discrimination: The ability of an item to discriminate between higher-ability and lower-ability examinees. π Formula: D = (Proportion correct in Upper Group) β (Proportion correct in Lower Group) π Example: To calculate item discrimination, rank all students by their total exam score. Take the top 27% as the Upper Group (UG) and the bottom 27% as the Lower Group (LG). For a given item, if 80% of the UG got it correct and 30% of the LG got it correct, then D = 0.80 - 0.30 = 0.50. This is an excellent discrimination value (D > 0.4). Conversely, if D = 0.10, the item is poor and should be revised. A negative D, e.g., -0.20, means the item is unacceptable and should be checked for error. The acceptable range for D is 0.20 or higher, with values above 0.4 being excellent.
β Key Takeaways
For the exam, you must remember that Classical Test Theory (CTT) and Item Response Theory (IRT) are the two main frameworks for item analysis, with CTT's p-value being the most common measure of item difficulty. The p-value is simply the proportion of students who answered an item correctly, where a value closer to 1.00 is easier and closer to 0.00 is more difficult; the ideal p-value depends on the number of item choices (e.g., 0.63 for 4-choice items). Item discrimination (D) measures how well an item separates high and low achievers and is calculated by subtracting the proportion of the lower 27% group correct from the upper 27% group correct. A D value of 0.20 or higher is acceptable, values above 0.40 are excellent, and any negative or near-zero value indicates a flawed item that should be removed. Finally, remember that items with p-values below 0.20 or above 0.90 require careful review.
π§ Quick Revision Questions
- What are the two main test theories discussed in the lecture, and how do they differ in their approach to item analysis?
- How is the item difficulty index (p-value) calculated, and what does a p-value of 0.80 indicate about an item?
- According to the lecture, what is the optimal item difficulty level for a 3-alternative multiple-choice question?
- What is item discrimination, and how is the discrimination index (D) calculated using the upper and lower 27% groups?
- A test item has a discrimination index (D) of 0.08. According to the given guidelines, what should you do with this item?
π Lecture 32 β Analyzing the Test-III
π Overview: This lecture introduces Item Response Theory (IRT) and its core concepts for analyzing test items. It explains how the Item Characteristic Curve (ICC) serves as the fundamental building block of IRT, and covers the three key item parameters: difficulty, discrimination, and guessing probability.
ποΈ Topics Covered
This lecture covers four major topics: the Item Characteristic Curve (ICC) as the basic building block of IRT with its three methodological properties; Item Difficulty in IRT defined as the ability level where probability of success is 0.5 on a logit scale; Item Discrimination in IRT which reflects how well an item differentiates between high and low ability examinees; and the Probability of Guessing in IRT, including blind and informed guessing types.
π Lecture Summary
Topic- 110: Item Characteristic Curve (ICC) in Item Response Theory
To analyze items using IRT, the main thing to consider is the item characteristic curve (ICC). The ICC is considered the basic building block of item response theory. Methodological properties of an ICC include: 1) Difficulty, which describes how the item functions along the ability scale (an easy item functions among low-ability examinees, a hard item among high-ability examinees); 2) Discrimination, which describes how well an item can differentiate between examinees having abilities below the item location and those having abilities above the item location; 3) An ICC is the graphical representation of the probability of answering an item correctly with the level of ability on the construct being measured.
The ICC gives a picture of the item difficulty, discrimination power, and the probability of answering correctly by guessing. As item difficulties are defined in relation to ability levels on the same scale, if we know person ability then we can predict how that person is likely to perform on an item without administering the item to the person.
π Definition β Item Characteristic Curve (ICC): The graphical representation of the probability of answering an item correctly with the level of ability on the construct being measured. π Example: A typical ICC is shown as an S-shaped curve where the x-axis represents the ability scale (theta) and the y-axis represents the probability of correct response, ranging from 0 to 1.
Topic- 111: Item Difficulty in Item Response Theory
The application of Item Difficulty in IRT is defined as the ability at which the probability of success on the item is 0.5 on a logit scale, which is also known as threshold difficulty. An item that has a high level of difficulty will be less likely to be answered correctly by an examinee with low ability than an item that has a low level of difficulty (i.e., an easy item).
Three item characteristic curves are presented on the same graph, all having the same level of discrimination but differing with respect to difficulty. The left-hand curve represents an easy item because the probability of correct response is high for low-ability examinees and approaches 1 for high-ability examinees. The center curve represents an item of medium difficulty because the probability of correct response is low at the lowest ability levels, around 0.5 in the middle of the ability scale, and near 1 at the highest ability levels. The right-hand curve represents a hard item; the probability of correct response is low for most of the ability scale and increases only when the higher ability levels are reached. Even at the highest ability level shown (+3), the probability of correct response is only 0.8 for the most difficult item.
π Definition β Item Difficulty (IRT): The ability level (on a logit scale) at which the probability of success on an item is 0.5, also called threshold difficulty. π Example: An easy item has its curve shifted to the left on the ability scale, a medium difficulty item has its curve centered, and a hard item has its curve shifted to the right.
Topic- 112: Item Discrimination in Item Response Theory (IRT)
The items on a test might also differ in terms of the degree to which they can differentiate individuals who have high trait levels from individuals who have low trait levels. This property is reflected in the steepness of the item characteristic curve. The steeper the curve, the better it discriminates. Items with steep ICCs are more discriminating compared to relatively flatter curves.
The figure contains three item characteristic curves having the same difficulty level but differing with respect to discrimination. The upper curve has a high level of discrimination since the curve is quite steep in the middle where the probability of correct response changes very rapidly as ability increases. The middle curve represents an item with a moderate level of discrimination; the slope of this curve is much less than the previous curve, and the probability of correct response changes less dramatically as the ability level increases. The third curve represents an item with low discrimination; the curve has a very small slope and the probability of correct response changes slowly over the full range of abilities shown.
π Definition β Item Discrimination (IRT): The degree to which an item can differentiate between individuals who have high trait levels and those who have low trait levels, reflected in the steepness of the ICC. π Example: A highly discriminating item has a steep ICC where probability changes rapidly; a low discriminating item has a flat ICC where probability changes slowly across ability levels.
Topic- 113: Probability of Guessing in Item Response Theory (IRT)
Guessing means giving an answer or making a judgment about something without being sure of all the facts. Guessing is a standard test-taking strategy presented to examinees taking a multiple-choice assessment. If test scores are based simply on the number of questions answered correctly, then a random guess increases the chance of a higher score. In IRT, this parameter of an item is also known as the G (guessing) parameter, which allows detecting the potential possibility of guessing in an item.
Examinees guess because they do not have adequate knowledge or ability to provide the correct answer. There are two types of guessing: Blind guessing, where an examinee chooses an answer at random from among the alternatives offered; and Informed guessing, where the examinee draws upon all his knowledge and abilities to choose the answer most likely to be correct. Item writers should be conscious of guessing and not write items that could be prone to guessing. IRT methods of item analysis should be employed to eliminate those items prone to guessing.
π‘ Why this matters: The guessing parameter in IRT allows test developers to identify items where low-ability examinees have an unusually high probability of answering correctly, indicating the item may need revision or removal.
π Definition β Guessing Parameter (G parameter): An IRT item parameter that allows detection of the potential possibility of guessing in an item, representing the probability of answering correctly by guessing alone.
β Key Takeaways
The Item Characteristic Curve (ICC) is the fundamental building block of IRT, graphically representing the relationship between ability and probability of correct response. IRT defines difficulty as the ability level where the probability of success is 0.5 on a logit scale, and items can be easy, medium, or hard based on where their ICC falls along the ability continuum. Discrimination is reflected in the steepness of the ICC, with steeper curves indicating better differentiation between high and low ability examinees. The guessing parameter (G) in IRT allows detection of items prone to guessing, which can be either blind (random) or informed (using partial knowledge). Understanding these three item parameters is essential for analyzing test quality and improving item construction.
π§ Quick Revision Questions
- What are the three methodological properties of an Item Characteristic Curve (ICC)?
- How is item difficulty defined in IRT, and what is the probability value associated with the difficulty threshold?
- What does the steepness of an item characteristic curve indicate about the item?
- What are the two types of guessing in test-taking, and how do they differ?
- How does an easy item's ICC differ from a hard item's ICC on a graph?
π Lecture 33 β Administering Test
π Overview: This lecture addresses the critical phase of test administration, emphasizing that even a well-constructed test can yield invalid results if not administered properly. It covers key issues like cheating, test anxiety, and poor conditions, then provides detailed guidance on assembling and packaging the test for optimal student performance and reliable data collection.
ποΈ Topics Covered
The lecture covers the Item Characteristic Curve (ICC) in Item Response Theory, issues in test administration such as cheating and test anxiety, the importance of controlling extraneous factors for valid results, recording test items with index cards, and a comprehensive two-part guide on packing the test including grouping similar formats, arranging items from easy to hard, proper spacing, keeping items on the same page, and positioning illustrations near descriptions.
π Lecture Summary
Topic- 114: Item Characteristic Curve (ICC) in Item Response Theory
We have spent a lot of time learning writing instructional objectives, table of specification, selection of questions format, writing questions, aligning questions and objectives to ensure a GOOD TEST. But imagine after all this hard work, if care is not taken while assembling all good work into a test actually presented to students or test administration is not done the way it should be, then whole hard work will be wasted.
Issues in test administration include: 1. Cheating, 2. Poor testing conditions, 3. Test anxiety, 4. Errors in test scoring procedure.
What to do? It is equally important to control all factors other than the test itself to collect trustable evidence of student learning by addressing test administration issues. The way in which the test is administered is very important to meet the goal of producing highly valid, reliable results. Once a test is ready then the next step is to administer it. The teacher has to help the students psychologically by maintaining a positive test-taking attitude, clarifying the rules, penalties for cheating, reminding them to check their copies, minimizing distractions and giving time warnings.
Cheating, poor testing conditions, and test anxiety, as well as errors in test scoring procedures contribute to invalid test results. Accurate achievement data are very important for planning curriculum and instruction. Test scores that overestimate or underestimate students' actual knowledge and skills cannot serve these important purposes. So it is worth a little more time to properly assemble and administer a test.
Topic- 115: Assembling Test: Recording Items
The preparation of test items is greatly facilitated if the items are properly recorded. Recording test items β The card should contain information concerning the instructional objectives, specific objective, difficulty index, discrimination index and the content measured by the item. This card should be prepared for each item to maintain its record.
Topic- 116: Assembling Test: Packing the Test βI
1. Packing the test β Once you have measurable instructional objectives, test blueprint, and written test items matching instructional objectives, so you are ready to package the test and reproduce the test.
Assembling Test (Packaging the Test) β Packing of test involves:
- Grouping together items of similar format
- Arranging test items from easy to hard
- Properly spacing items
- Keeping items and options on the same page
- Position illustration near description (if diagram given)
- Randomness of answer key
- Determine how students record answers (separate sheet or same sheet)
- Providing space for test taker's details
- Proofread the test (typographical and grammatical error)
- Test directions
1. Grouping together items of similar format β Makes easy to understand. Saves response time. Once set of instruction per format is enough.
2. Arranging test items from easy to hard β Increases the possibility of good start for poor starters. Builds confidence. Motivates. Reduces test anxiety.
Topic- 117: Assembling Test: Packing the Test βII
1. Packing the test β Following are some more points.
Space the item for easy reading β Enough space between lines and questions. Suitable answer space. Standard font type and size.
4. Keeping items and options on the same page β Minimizes the likelihood of misprint. Saves respondent from unnecessary hassle. Saves test time.
5. Position illustration near description (if diagram given) β Put all related questions on same page. Diagram above the question.
β Key Takeaways
A properly assembled and administered test is critical for collecting valid and reliable evidence of student learning, as test scores that overestimate or underestimate actual knowledge cannot serve curriculum and instructional planning purposes. Teachers must control factors such as cheating, poor testing conditions, test anxiety, and scoring errors to maintain test validity. When packing the test, items should be grouped by similar format, arranged from easy to hard to build student confidence, and properly spaced for readability with standard fonts. Keeping items and options on the same page minimizes misprints and saves time, while positioning illustrations near their descriptions ensures clarity. Each test item should be recorded on a card containing its instructional objective, difficulty index, discrimination index, and content area for proper record-keeping.
π§ Quick Revision Questions
- What are the four main issues in test administration that can lead to invalid test results?
- Why is it important to arrange test items from easy to hard when packing a test?
- What information should be recorded on a card for each test item during the recording phase?
- List at least five of the ten points involved in packing (assembling) a test properly.
- Why is it recommended to keep items and their options on the same page when assembling a test?
π Lecture 34 β Analyzing the Test-II
π Overview: This lecture continues the discussion of assembling and packing tests, focusing on practical considerations for creating a clear, fair, and well-organized test. It covers randomness in answer keys, student response formats, proofreading, test directions, and the reproduction process, ensuring that the final test is both professional and effective.
ποΈ Topics Covered
The lecture covers packing the test with points on randomness of answer key, determining how students record answers, providing space for test taker details, proofreading for errors, and test directions. It then addresses reproducing the test, including knowing the photocopying machine, specifying copying instructions, and filing the original test.
π Lecture Summary
Topic- 118: Assembling Test: Packing the Test βIII
- Packing the test Following are some more points.
6. Randomness of answer key Check the key for equal distribution of correct answers; use all options equally. This helps avoid guessing and avoid confusion.
7. Determine how students record answers (separate sheet or same sheet) This decision brings uniformity and allows for attempt with clarity. It also prepares for standardized tests where separate answer sheets are common.
8. Providing space for test takerβs details Identification is ensured, and it makes it easy to combine different parts of test (if applicable), such as when a test has multiple sections that need to be scored together.
Topic- 119: Assembling Test: Packing the Test βIV
- Packing the test Following are some more points.
9. Proofread the test (typographical and grammatical error) This helps save time during test by preventing student confusion, avoid clues/confusion that could give away answers, and increase confidence in test by presenting a professional document.
10. Test directions Well-written directions conveys expectation to students, help in time management by explaining how to proceed, and conveys scoring policy and priorities so students know what is expected.
Topic- 120: Assembling Test: Reproducing the Test
In schools it is usually by photocopying, and quality of test copies may vary considerably.
Reproducing the Test Reproduction of test involves:
- Knowing the photocopying machine
- Specifying copying instructions
- Filing original test
1. Knowing the photocopying machine Requires an expert operator who understands the machine. Toner quality must be adequate. The machine should have atomization facility for even distribution of toner. This ensures uniformity of legibility, shades, etc. across all copies.
2. Specifying copying instructions Includes randomly checking of every 19th copy to ensure quality. Also covers paper size, margins, ordering of pages, and stapling to produce a clean, organized final product.
3. Filing original test The master copy is filed for reference and reuse in future. It can be used as reference in random checking against the copies. The original should be kept with you during test to refer to in case of any student questions or disputes.
β Key Takeaways
The lecture emphasizes that a well-packed and reproduced test requires attention to detail before, during, and after printing. Key points include ensuring randomness in answer keys to prevent guessing, deciding how students record answers for clarity and standardization, and always providing space for student identification. Proofreading for errors and writing clear test directions are essential for saving time and avoiding confusion. Finally, knowing the photocopying machine, specifying precise copying instructions, and filing the original master copy are critical steps in reproducing a high-quality, legible test that can be reused effectively.
π§ Quick Revision Questions
- What is the purpose of checking the randomness of the answer key, and how should correct answers be distributed?
- Why is it important to determine whether students record answers on a separate sheet or the same sheet?
- How does proofreading a test for typographical and grammatical errors benefit both the student and the test administrator?
- List the three main steps involved in reproducing a test after it has been assembled.
- Why is it recommended to keep the original master copy with the test administrator during the test?
π Lecture 35 β Analyzing the Test-III
π Overview: This lecture focuses on the critical procedures and best practices for administering tests in educational settings. It provides a comprehensive guide on how to prepare students, maintain test integrity, and ensure a fair and smooth testing environment, covering everything from pre-test attitudes to post-test collection.
ποΈ Topics Covered
The lecture covers four main topics: things to remember when administering a test (Parts I-IV), including maintaining a positive attitude, maximizing achievement motivation, equalizing advantages for students, avoiding surprises, clarifying rules, rotating distribution, reminding students to check copies, monitoring students, minimizing distributions, providing time warnings, and collecting tests uniformly.
π Lecture Summary
Topic- 121: Assembling Test: Things to Remember When Administering Test βI
When a test is ready, the instructor must prepare students for it. There are 11 key things to remember when administering a test: maintain a positive attitude, maximize achievement motivation, equalize advantages, avoid surprises, clarify the rules, rotate distribution, remind students to check their copies, monitor students, minimize distributions, give time warnings, and collect tests uniformly.
Maintain a positive attitude involves assuring students that the test covers material that was taught, addressing students' convenience and limitations, providing guidance to reduce anxiety, and eliminating non-actors that affect achievement.
Maximize achievement motivation requires encouraging students to do their best while mitigating fear, highlighting the value of giving one's best, reducing panic, and encouraging serious thinking.
Topic- 122: Assembling Test: Things to Remember When Administering Test βII
Equalize advantages (test-wise) means discouraging guessing, discouraging leaving answers blank, discouraging multiple answers, and providing instructions such as: don't spend too much time on difficult items, do easy questions first, and must read after completing the test.
π Definition β Equalize advantages: The practice of providing all students with the same test-taking guidance to ensure no student has an unfair advantage based on test-taking skills rather than content knowledge.
Avoid surprises involves giving advance notice of tests, discussing the test structure, and providing ways of preparing for the test. This brings out stable achievement and allows students to perform their best.
Topic- 123: Assembling Test: Things to Remember When Administering Test βIII
Clarify the rules before the test begins. This includes explaining: time limits, restroom policy, special requirements of students, how the test will be distributed, rules about consulting others, and rules about borrowing things.
Rotate distribution of test papers by varying the pattern: left to right, right to left, front to back, back to front, using multiple person distribution, and ensuring all students start at the same time.
Remind students to check their copies before starting. They should verify: the order of pages, the number of pages, the quality of print, request replacement if needed before the test starts, and ensure they have recorded their name and date.
Topic- 124: Assembling Test: Things to Remember When Administering Test -IV
Monitor students (test invigilation) involves enforcing penalties for cheating, preventing disturbance of others, checking seating position/posture, protecting the test from being seen by others, handling cheating materials, and defining the jurisdiction of invigilation staff.
Minimize distributions during the test by avoiding noise, avoiding giving instructions after the start of the test, avoiding in-out movement, and ensuring no talking between invigilation staffβmaintain silence.
Time warning and test collection procedures include: giving warnings at half time, 2/3 time, and at 30, 15, and 5 minute reminders; using minimum words to announce time warnings; giving time for students to close their work; announcing "stop writing"; and collecting tests in the reverse order of distribution.
π Definition β Test invigilation: The process of monitoring students during a test to ensure academic integrity, prevent cheating, and maintain a proper testing environment.
β Key Takeaways
Students must remember the 11-step procedure for test administration, which includes maintaining a positive attitude, maximizing motivation, equalizing advantages, avoiding surprises, clarifying rules, rotating distribution, checking copies, monitoring students, minimizing distractions, giving time warnings, and collecting tests uniformly. The instructor must address both psychological factors (reducing anxiety, encouraging best performance) and logistical factors (distribution patterns, time warnings, collection order). Clear communication of rules before the test and silent, orderly monitoring during the test are essential for fair administration. Cheating prevention through proper invigilation and enforcing penalties is critical for maintaining test integrity. Finally, systematic test collection in reverse order of distribution ensures all papers are accounted for and no student is disadvantaged.
π§ Quick Revision Questions
- What are the 11 key things to remember when administering a test?
- How can an instructor maximize achievement motivation in students before a test?
- What instructions should be given to equalize advantages for test-wise students?
- What specific checks should students perform on their test copies before starting?
- What is the proper procedure for time warnings and test collection at the end of a test?
π Lecture 36 β Scoring Test-I
π Overview: This lecture introduces the concepts of scoring criteria and scoring rubrics for essay-type questions. It explains the purpose and structure of rubrics, detailing their key elements (score, criteria, level of performance, descriptors) and differentiates holistic scoring rubrics from other types. Understanding these concepts is essential for creating fair, reliable, and valid assessments in education.
ποΈ Topics Covered
The lecture first covers scoring criteria and the definition of a scoring rubric. It then details the scoring rubric for essay-type questions, including basic steps for design. Next, it explains the four elements of a rubric: score, criteria, level of performance, and descriptors. Finally, it discusses holistic scoring rubrics, including when to use them and their application to essay questions.
π Lecture Summary
Scoring Criteria: Scoring Rubric
Scoring Criteria involve planning how responses will be scored. This process leads to rethinking and clarifying the questions so that students have a clearer idea of what is expected. Clear specification of scoring criteria in advance of administering essay questions can contribute to improved reliability and validity of the assessment.
A Scoring Rubric is an explicit set of criteria used for assessing a particular type of work or performance and provides more details than a single grade or mark. Score levels identified in a scoring rubric must be descriptive, not merely judgmental in nature.
π Definition β Scoring Rubric: An explicit set of criteria used for assessing a particular type of work or performance, providing more details than a single grade or mark.
π Example: Define the level of rubric as "Writing is clear and thoughts are complete" as compared to "excellent".
Scoring Rubric for Essay Type Questions
Scoring of essay items is a time-consuming and difficult process. Reliability of the test demands that scoring should be consistent not only by the rater at different times but by two independent raters as well. Judgments concerning the quality of a given writing sample may vary depending upon the criteria established by the individual evaluator. Writing samples are just one example of performance that may be evaluated using scoring rubric. Scoring rubric has also been used to evaluate group activities, extended projects and oral presentations.
Basic Steps to design Rubric:
- Identify a learning goal.
- Choose outcomes that may be measured.
- Develop or adapt an existing rubric.
- Share it with students.
Elements of Rubric
A rubric includes four key elements:
- Score
- Criteria
- Level of performance
- Descriptors
Score is a system of numbers or values assigned to a work, often combined with a level of performance. High numbers are for the best performance like 4, 5 or 6, whereas down to 1 or 0 are the lowest score in a performance assessment.
Criteria tells us which feature, trait, or dimension is to be measured and includes a definition and example to make clear the meaning of each trait to be assessed.
Level of Performance uses adjectives to describe the performance levels. These levels tell students what they are expected to do. Descriptors can be used with levels of performance to achieve objectivity, but they can be used without them as well.
Descriptors are details for each level of performance to ensure reliable and unbiased scoring.
π Definition β Descriptors: Details for each level of performance to ensure reliable and unbiased scoring.
Holistic Scoring Rubric
Holistic Scoring Rubrics are good for evaluating overall performance on a task. All criteria are assessed as a single score. As only one score is given, holistic rubrics are easy to score.
When to use:
- There is no correct answer/response to a task, e.g., creative work.
- Focus is on overall quality, proficiency, or understanding of a specific content or skill.
- The assessment is summative, e.g., at the end of the semester or major.
- Assessing significant numbers, e.g., 150 student portfolios.
Holistic Rubric for Essay Type Questions A holistic rubric is probably more appropriate for extended response essay type items involving a student's abilities to synthesize and create and when no definite answer can be pre-specified.
π Definition β Holistic Scoring Rubric: A rubric that evaluates overall performance on a task by assessing all criteria with a single score.
π‘ Why this matters: Choosing between holistic and other types of rubrics depends on the assessment purpose. Holistic rubrics are efficient for large-scale or summative assessments but do not provide detailed, specific feedback on individual components of performance.
β Key Takeaways
Scoring criteria and rubrics are essential tools for improving the reliability and validity of essay question assessments. A rubric's four core elementsβscore, criteria, level of performance, and descriptorsβmust be descriptive, not merely judgmental, to ensure objective and consistent scoring. Holistic rubrics are best used for overall performance assessment on tasks without a single correct answer, especially in summative evaluations or with large numbers of students, as they are easy to score. The process of designing a rubric involves identifying a learning goal, choosing measurable outcomes, developing the rubric, and sharing it with students to clarify expectations. Descriptors are crucial for making each performance level reliable and unbiased.
π§ Quick Revision Questions
- What is a scoring rubric, and how does it differ from a single grade or mark?
- List the four basic steps to design a rubric.
- Name and briefly define the four elements of a rubric.
- What is a holistic scoring rubric, and in which two specific situations is it most appropriate to use?
- Why must score levels in a rubric be "descriptive" rather than merely "judgmental"?
π Lecture 37 β Scoring Test-II
π Overview: This lecture examines two major approaches to scoring constructed response assessments: holistic and analytic scoring rubrics. It explains the different methods for implementing holistic rubrics, details the structure and application of analytic rubrics, and discusses the advantages and disadvantages of each approach. Understanding these scoring methods is essential for educators to provide fair, consistent, and meaningful evaluation of student performance.
ποΈ Topics Covered
This lecture covers four main topics: approaches to applying holistic scoring rubrics including criterion-referenced and norm-referenced methods, the structure and application of analytic scoring rubrics for essay-type questions, the advantages and disadvantages of analytic rubrics, and practical suggestions for scoring constructed response questions effectively.
π Lecture Summary
Topic-129: Approaches to Apply Holistic Scoring Rubric
The first approach to implementing holistic rubric is to decide beforehand on the number of quality categories into which you will sort the student's answers. A second, better way is to craft a holistic rubric that defines the qualities of paper belonging in each categoryβfor example, defining what an "A" paper is, what a "B" paper is, and so on.
A third refinement is to select specimen papers, which are good examples of each scoring category. Then you can compare the student's paper with the pre-specified specimens that define each category level. A fourth way of implementing holistic rubric is to read the answer completely and compare one with another to decide which are the best, the next best, and so on. This results in rough ranking of all papers, but this approach cannot be applied to a large number of papers.
Among these four approaches, the first three are consistent with a grading philosophy of criterion-referenced or absolute quality standards, while the fourth is consistent with norm-referenced or relative standard grading philosophy.
π Definition β Holistic Scoring Rubric: A scoring method that assigns a single overall score based on the general quality of the entire response.
π Example: The holistic scoring rubric provided in the lecture uses a 0-5 scale:
- Score 5: Demonstrates complete understanding of the problem. All requirements of task are included.
- Score 4: Demonstrates considerable understanding. All requirements included.
- Score 3: Demonstrates partial understanding. Most requirements included.
- Score 2: Demonstrates little understanding. Many requirements missing.
- Score 1: Demonstrates no understanding.
- Score 0: No response/task not attempted.
π‘ Why this matters: Holistic scoring allows for quick evaluation and provides an overview of student achievement, but it does not provide detailed information about specific strengths and weaknesses.
Topic-130: Analytic Scoring Rubric
Analytic Scoring Rubric assesses each criterion separately by using different descriptive ratings. Each criterion is given a separate score, and the final score is made up of adding each component part. It takes more time to score but gives detailed more feedback.
When to use analytic rubrics:
- Several faculties are collectively assessing student work
- Outside audience will be examining rubric scores
- Profiles of specific strengths/weaknesses are desired
For developing an analytic rubric for essay type questions, you must first create a list of major elements that students are expected to include in an ideal answer. Next, decide the number of points to award to students when they include each element. Analytic rubric is used to score essay-type questions on different points and works best on the restricted response type essay question.
In developing analytic rubric, identify the certain elements of the answer which are more appropriate to the learning objectives of a course. When assigning numbers to each element, be sure the total points match with the essay's total value in relation to the overall number of points on the test.
Students will get partial credit awarded on a partially correct answer, but partial credits can increase inconsistency in scoring and decrease the reliability of the scoring process. Crafting a partial credit scoring may be difficult after reading a few papers. The pattern of students' errors and misconceptions are emerged then craft a partial credit scoring rubric and use it to score all papers.
π Definition β Analytic Scoring Rubric: A scoring method where each criterion is assessed separately with its own descriptive ratings and separate scores, which are then summed for a final score.
π Structure: The rubric uses four levels for each criterion:
- Beginning (1): Description reflecting beginning level of performance
- Developing (2): Description reflecting movement towards mastery level
- Accomplished (3): Description reflecting achievement of mastery level
- Exemplary (4): Description reflecting highest level of performance
Topic-131: Advantages and Disadvantages of Analytic Rubric
Advantages of analytic rubric:
- More detailed feedback for students
- Scoring is more consistent across students and grades
Disadvantages of analytic rubric:
- Time consuming to score
Suggestions for scoring constructed response questions:
- Prepare an outline of the expected answer in advance
- Use the scoring rubric that is most appropriate
- Decide how to handle factors that are irrelevant to the learning outcomes being measured
- Evaluate all responses to one question before going on to the next one
- When possible, evaluate the answers without looking at the student's name
- If especially important decisions are to be based on the results, obtain two or more independent ratings
π‘ Why this matters: Following these suggestions improves scoring reliability, reduces bias, and ensures that assessment focuses on the intended learning outcomes rather than extraneous factors.
β Key Takeaways
Holistic scoring uses a single overall score based on general quality and can be implemented through four approaches: pre-defined categories, crafted rubrics with descriptors, specimen papers for comparison, or rough ranking through paired comparisonsβwith the first three being criterion-referenced and the fourth norm-referenced. The holistic rubric's main advantage is quick scoring but lacks detailed feedback. Analytic scoring assesses each criterion separately with descriptive ratings across four levels (Beginning, Developing, Accomplished, Exemplary), providing more detailed feedback and consistent scoring though requiring more time. For constructed response questions, educators should prepare expected answer outlines in advance, use appropriate rubrics, evaluate one question at a time, anonymize responses when possible, and obtain multiple independent ratings for important decisions.
π§ Quick Revision Questions
- What are the four approaches to implementing holistic scoring rubrics, and which ones are criterion-referenced versus norm-referenced?
- What are the key differences between holistic and analytic scoring rubrics in terms of feedback quality and time required?
- In the analytic scoring rubric, what are the four performance levels from lowest to highest, and what does each represent?
- When is analytic scoring most appropriate to use according to the lecture?
- List at least four of the six suggestions provided for scoring constructed response questions effectively.