π Lecture 23 β Creating Fixed-Choice Test Items: Matching Exercises-I
π Overview: This lecture introduces matching exercises as a type of fixed-choice test item. It covers the structure, uses, advantages, and limitations of matching exercises, which are designed to measure a student's ability to identify relationships between two parallel sets of items. Understanding how to construct and evaluate these items is crucial for creating effective assessments that test factual associations.
ποΈ Topics Covered
This lecture covers three main topics: the definition and structure of matching exercises, including how they are constructed with premises and responses; the various uses of matching exercises across different subject areas for measuring factual associations; and finally, the advantages and limitations of this test format, highlighting its compactness and susceptibility to irrelevant clues.
π Lecture Summary
Topic- 081: Matching Exercises
Matching exercises consist of two parallel columns: the first column contains premises and the second contains responses, along with directions for matching the two columns. Matching test items are selection items specially used to measure a student's ability to identify the relationship between a set of similar items, each of which has two components, such as words and their definitions, symbols and their meanings, dates and events, or people and their accomplishments.
The matching exercise is an economical method when used with content that has sufficient homogeneous factual information. In developing matching items, there are two columns of material. The exercise is used when measuring a student's ability to identify the relationship between a set of similar items, each of which has two components.
π Definition β Matching Exercise: A test item consisting of two parallel columns (premises and responses) where the student must identify the relationship between items based on directions provided.
π Formula: Premises (List A) + Responses (List B) + Directions β Student matches items based on logical association.
π Example: The lecture provides an example where List A (Premises) contains names like Quaid-e-Azam, Jahanger Khan, Abul Qadeer Khan, Parveen Shakir, and Benazir Bhutto. List B (Responses) contains categories like Player, Prime Minister, Statesman, Poet, Scientist, and Civil Servant. The directions state there can be more than one reason of fame listed for one name, and any given name may not correspond to any of the reasons of fame.
Topic- 082: Uses of Matching Exercises
The typical matching exercise is limited to measuring factual information based on simple associations. It is a compact and efficient method of measuring such simple knowledge outcomes. Examples of relationships considered important by teachers in a variety of fields include: Persons to Achievements, Dates to Historical events, Terms to Definitions, Rules to Examples, Symbols to Concepts, Authors to Titles of books, Foreign words to Local equivalents, Machines to Uses, Plants or animals to Classifications, Principles to Illustrations, Objects to Name of objects, and Functions.
The matching exercise is also used with pictorial materials in relating pictures and words to identify positions on maps, charts, and diagrams. Regardless of the form of presentation, the student's task is essentially to relate two things that have a logical association. This restricts the use of matching exercises to a "small area of student's achievement."
π‘ Why this matters: Matching exercises are efficient for measuring specific factual associations but are limited in scope, meaning they cannot assess higher-order thinking or complex understanding. Teachers must use them strategically for appropriate content.
Topic- 083: Advantages and Limitations of Matching Exercises
Advantages of Matching Exercises
- The major objective of the matching exercise is its compact form, which makes it possible to measure a large amount of related factual material in a relatively short time.
- Another advantage is ease of construction. Poor matching items can be rapidly constructed, but good matching items require a high degree of skill.
Limitations of Matching Exercises
- It is restricted to the measurement of factual information based on rote learning.
- It is highly susceptible to the presence of irrelevant clues that can help students guess answers without actual knowledge.
- There is difficulty in finding homogeneous material that is significant from the viewpoint of our objectives and learning outcomes.
β Key Takeaways
Matching exercises are selection items that assess a student's ability to identify relationships between two sets of homogeneous items, such as premises and responses. Their primary advantage is compactness, allowing for efficient measurement of a large amount of related factual information in a short time. However, they are limited to measuring factual information based on rote learning and are highly susceptible to irrelevant clues. Effective construction requires skill to ensure homogeneous, significant material is used. Teachers must be aware that while matching exercises are easy to construct poorly, creating good ones demands careful planning to avoid irrelevant clues and ensure content validity.
π§ Quick Revision Questions
- What are the two columns in a matching exercise called, and what is the student's task?
- Why is matching exercise considered an "economical method" according to the lecture?
- Name three specific types of relationships that matching exercises can measure, as listed in the lecture.
- Explain one major advantage and one major limitation of using matching exercises.
- What does it mean when the lecture says matching exercises are "highly susceptible to the presence of irrelevant clues"? Give an example.
π Lecture 24 β CREATING FIXED-CHOICE TEST ITEMS: MATCHING EXERCISES-II
π Overview: This lecture continues the discussion on constructing matching exercises by providing detailed suggestions for effective item writing. It covers key principles such as using homogeneous material, ensuring unequal numbers of premises and responses, arranging responses in logical order, and providing clear directions. These guidelines are essential for creating fair, valid, and efficient matching test items that minimize guessing and confusion.
ποΈ Topics Covered
The lecture covers four main topics, each offering a specific suggestion for constructing matching exercises: using only homogeneous material in a single exercise; including an unequal number of responses and premises; arranging response lists in logical order (alphabetical or numerical); and indicating the basis for matching while placing all items on the same page. Each suggestion is accompanied by explanations and examples.
π Lecture Summary
Topic- 084: Suggestions for Constructing Matching Exercises βI
Use only homogeneous material in the single matching exercise. This is described as the most violated rule of developing a matching exercise. Homogeneity is a matter of degree; what is homogeneous to one group may be heterogeneous to another.
π‘ Why this matters: Mixing different categories (e.g., mixing historical figures with scientific terms) makes the exercise confusing and reduces its validity.
Topic- 085: Suggestions for Constructing Matching Exercises βII
Include an unequal number of responses and premises, and instruct the students that responses may be used once, more than once, or not at all. This makes all responses eligible for selection for each premise and decreases the likelihood of successfully guessing. Keep the list of items to be matched brief and place the shorter responses on the right. It is easier to maintain homogeneity in a brief list; four to seven items in each column seems best. Placing shorter responses on the right also contributes to more efficient test tasking.
π Example: Instructions: A list of premises includes a list of prominent Pakistanis and the list of responses give their reason of fame. Match the given names with their respective reason of fame. There can be more than one reason of fame listed for one name and any of the given names may not correspond to any of the reasons of fame.
| List A (Premises) | List B (Responses) |
|---|---|
| Quaid-e-Azam | a. Player |
| Jahanger Khan | b. Prime Minister |
| Abul Qadeer Khan | c. Statesman |
| Parveen Shakir | d. Poet |
| Benazir Bhutto | e. Scientist |
| f. Civil Servant |
Topic- 086: Suggestions for Constructing Matching Exercises βIII
Arrange the list of responses in logical order. Place words in alphabetical order and numbers in sequence. This contributes to the ease with which students can scan the responses in searching for the correct answers. It also prevents them from detecting possible clues from the arrangements of the responses.
π Example: Directions: On the line to the left of each historical event in column A, write the letter from Column B that identifies the time period when the event occurred. Each date in Column B may be used once, more than once, or not at all.
Instruction: A list of premises includes a list of prominent Pakistanis and the list of responses give their reason of fame. Match the given names with their respective reason of fame. There can be more than one reason of fame listed for one name and any of the given names may not correspond to any of the reasons of fame.
| List A (Premises) | List B (Responses) |
|---|---|
| Abul Qadeer Khan | Civil Servant |
| Benazir Bhutto | Player |
| Jahanger Khan | Poet |
| Parveen Shakir | Prime Minister |
| Quaid-e-Azam | Scientist |
| Statesman |
Topic- 087: Suggestions for Constructing Matching Exercises βIV
Indicate the directions for the basis for matching the responses and premises. By following this suggestion, ambiguity and confusion can be avoided. Testing time will be saved; students will not need to read through the entire list of premises and responses and then reason out the basis for matching. Place all of the items for one matching exercise on the same page. This prevents students from missing the responses appearing on another page and generally adds to the speed and efficiency of test administration.
β Key Takeaways
The most critical rules for constructing effective matching exercises are: (1) use only homogeneous material in a single exercise to avoid confusion; (2) include an unequal number of responses and premises and allow responses to be used once, more than once, or not at all to reduce guessing; (3) keep lists brief (4β7 items) and place shorter responses on the right for efficiency; (4) arrange responses in logical order (alphabetical for words, sequential for numbers) for easy scanning; and (5) always provide clear directions explaining the basis for matching, and keep all items on the same page to prevent missing information.
π§ Quick Revision Questions
- What is the most violated rule in constructing matching exercises, and why is it so important?
- How does including an unequal number of responses and premises reduce guessing?
- Why should responses be arranged in alphabetical or numerical order?
- What are the recommended number of items for each column in a matching exercise?
- What are two benefits of placing all items for one matching exercise on the same page?
π Lecture 25 β Creating Fixed-Choice Test Items: Short Questions-I
π Overview: This lecture focuses on developing short answer and completion test items, which are supply-type objective test formats. It explains how these items differ in presentation but share the same response format, and discusses their uses for measuring knowledge of terminology, facts, and procedures.
ποΈ Topics Covered
This lecture covers the development of short answer and completion questions, including their definitions, presentation formats (direct questions vs. incomplete statements), examples across subjects, and their common uses for measuring simple to moderately complex learning outcomes such as data interpretation and problem-solving in mathematics and science.
π Lecture Summary
Topic- 088: Developing Short Answer/Completion Questions βI
Two item formats fall under the category of supply type within objective test items: short answers and completion (also called fill in the blanks). Both are forms of supply items that can be answered with a word, phrase, number, or symbol. They differ only in form of presentation: short answers are presented as direct questions, while completion items are incomplete statements. Both are frequently used for measuring knowledge of terminology, specific facts, principles, and procedures.
π Definition β Short Answer Item: a supply-type test item presented as a direct question that requires a word, phrase, number, or symbol response. π Definition β Completion Item: a supply-type test item presented as an incomplete statement that requires a word, phrase, number, or symbol response.
Topic- 089: Developing Short Answer/Completion Questions βII
Both short answer and completion items are "supply" type test items. Short answers are direct questions; completion items are incomplete statements. π Example:
- Short Answer: What is the name of the man who invented the light bulb? (Thomas Edison)
- Completion: The name of the man who invented the light bulb is __________. This category also includes problems in arithmetic, mathematics, science, and other areas where the solution must be supplied by the student.
π Formula: Short Answer = Direct Question β Student supplies answer π Formula: Completion = Incomplete Statement β Student supplies missing word/phrase/number/symbol
Topic- 090: Uses of Short Answer/Completion Questions
The short answer test item is suitable for measuring a wide variety of relatively simple learning outcomes. Common uses include:
- Simple interpretation of data: i. How many syllables are there in the word Argentina? (4) ii. In the number 612, what value does the 6 represent? (600) iii. If an airplane flying northwest made a 180-degree turn, what direction would it be heading? (Southeast)
More complex interpretations are possible when short answer items measure the ability to interpret diagrams, charts, graphs, and pictorial data. Notable exceptions to the general rule that short answer items are limited to simple learning outcomes are found in mathematics and science, where complex problem-solving can be assessed.
π‘ Why this matters: While short answer/completion items are typically used for simple factual recall, they can effectively assess higher-order thinking in mathematics and science when students must supply solutions to multi-step problems.
β Key Takeaways
Short answer and completion items are both supply-type test formats that require a word, phrase, number, or symbolβdiffering only in presentation (direct question vs. incomplete statement). They are ideal for measuring knowledge of terminology, specific facts, principles, and procedures, as well as simple data interpretation. More complex uses exist in mathematics and science, where they can assess problem-solving skills. Examples include counting syllables, place value identification, and direction changes. These items are limited to relatively simple learning outcomes, with mathematics and science being notable exceptions.
π§ Quick Revision Questions
- What is the fundamental difference between a short answer item and a completion item?
- List three types of learning outcomes that short answer/completion items can measure.
- How would you convert the short answer question "What value does the 6 represent in 612?" into a completion item?
- In what subject areas are short answer items exceptions to the rule of measuring only simple learning outcomes?
- Give an example of a short answer question that tests interpretation of a diagram.
π Lecture 26 β CREATING FIXED-CHOICE TEST ITEMS: SHORT QUESTIONS-II
π Overview: This lecture explores the creation of short answer and completion test items, focusing on their advantages and limitations as assessment tools. It provides specific guidelines for constructing effective items that accurately measure student knowledge while minimizing guesswork. Understanding these principles is crucial for designing fair and valid classroom assessments.
ποΈ Topics Covered
This lecture covers the advantages of short answer/completion questions, including their efficiency for testing large numbers of items and reduced guessing probability compared to true-false questions. It also addresses their limitations, particularly the difficulty in framing questions for a single specific answer and their unsuitability for measuring complex learning outcomes. Key construction suggestions emphasize wording items to require brief and specific answers and avoiding direct textbook statements.
π Lecture Summary
Topic- 091: Advantages of Short Answer/Completion Questions
Short answer and completion questions offer several advantages for teachers. They allow teachers to have students complete a large number of items in a fairly short time, unless the questions involve working complex mathematical problems. A key benefit is the ability to control the possibility of guessing; since the student must generate the answers, the likelihood of guessing correctly is greatly reduced when compared with true-false questions.
However, these question types also have limitations. A potential problem with both short-answer and completion items is that it is difficult to frame questions for one specific answer unless the items are well written. Furthermore, these questions are not usable to measure complex learning outcomes.
Topic- 092: Suggestions for Constructing Short Answer/Completion Questions βI
A primary suggestion for constructing these items is to word the item so that the required answer is both brief and specific. The lecture provides a comparative example to illustrate this principle.
π Definition β Specific Wording: Ensure the question's phrasing clearly points to a single, concise correct answer.
π Example:
- Poor: An animal that eats the flesh of other animals is? (Carnivorous)
- Better: An animal that eats the flesh of other animals is classified as ______ (Carnivorous)
π‘ Why this matters: The "Better" version reduces ambiguity by telling the student the expected form of the answer (a classification term).
Topic- 093: Suggestions for Constructing Short Answer/Completion Questions βII
Another critical suggestion is to not take statements directly from textbooks to use as a basis for short answer items. The lecture provides an example contrasting a poor item derived directly from a textbook with a better, rephrased version.
π Example:
- Poor: Chlorine is a ______. (Halogen)
- Better: Chlorine belongs to a group of elements that combines with metals to form salts. It is therefore called as a ______. (Halogen)
π‘ Why this matters: The "Better" version requires the student to understand the property (combining with metals to form salts) rather than simply recalling a memorized textbook statement.
β Key Takeaways
For the exam, you must remember that short answer/completion questions are effective for efficiently testing a large number of items with reduced guessing compared to true-false questions, but they are difficult to write for a single answer and cannot measure complex learning outcomes. When constructing them, always word the item to require a brief and specific answer, and never lift statements directly from textbooks; instead, rephrase them to require understanding. The key principle is to write items that clearly point to one correct answer by providing enough context.
π§ Quick Revision Questions
- What are two key advantages of using short answer/completion questions over true-false questions?
- What is a primary limitation of short answer/completion questions in terms of what they can measure?
- Why should teachers avoid taking statements directly from textbooks for short answer items?
- What makes a "Better" short answer item, according to the lecture's examples?
- How does the "Better" version for the "Chlorine" question improve upon the "Poor" version?
π Lecture 27 β Creating Fixed-Choice Test Items: Short Questions-III
π Overview: This lecture provides essential guidelines for constructing effective short-answer and completion test items. It focuses on improving question clarity, precision, and fairness by offering practical suggestions for wording, numerical answers, and blank placement. These rules help ensure that test items accurately measure student knowledge rather than confusing them with poor formatting.
ποΈ Topics Covered
This lecture covers three key topics: the superiority of direct questions over incomplete statements for testing understanding, the importance of specifying answer units when numerical responses are required, and best practices for using completion items including limiting blanks and avoiding trivial information. Each topic includes examples of poor versus better question construction.
π Lecture Summary
Topic- 094: Suggestions for Constructing Short Answer/Completion Questions βIII
A direct question is generally more desirable than an incomplete statement when constructing test items. Direct questions provide clear context and reduce ambiguity, making it easier for students to understand exactly what is being asked. Incomplete statements can confuse students about the expected answer format.
π Definition β Direct question: A test item framed as a complete question rather than a sentence with a blank to fill in.
π Example:
Poor: Pakistan gained its independence in ______. (1947)
Better: When did Pakistan gain its independence? (1947)
Best: In what year did Pakistan gain its independence? (1947)
Explanation: The "best" version specifies the type of information required (year), reducing ambiguity.
π‘ Why this matters: Direct questions reduce guessing and better assess actual knowledge by clearly indicating what information students must recall.
Topic- 095: Suggestions for Constructing Short Answer/Completion Questions βIV
If the answer is to be expressed in numerical units, always indicate the type of answer required. This prevents students from providing correct calculations but in the wrong unit or format, which can lead to unfair grading. Specifying units also streamlines the scoring process.
π Rule: Specify the desired unit of measurement in the question itself.
π Example:
Poor: If a watermelon weighs 1kg 200 grams each, how much will 3 watermelons weigh? (3.5kg and 100 grams OR 3600 grams)
Better: If a watermelon weighs 1kg 200 grams each, how much will 3 watermelons weigh? (3.6kg)
Explanation: The better version requests the answer in kilograms, eliminating multiple correct formats and reducing grading confusion.
π‘ Why this matters: Ambiguous answer formats lead to inconsistent grading and penalize students who understood the concept but expressed it differently.
Topic- 096: Suggestions for Constructing Short Answer/Completion Questions βV
When using completion items (fill-in-the-blank), follow three key rules: (1) do not include too many blanks in a single sentence, (2) do not start the sentence with a blank space, and (3) avoid asking trivial information in the blank space. Too many blanks make the item a puzzle rather than a test of knowledge, while starting with a blank and trivial information undermines the question's validity.
π Definition β Completion item: A test item presented as an incomplete sentence where students fill in missing word(s) or phrase(s).
π Example:
Poor: (_____) animals that are born (_____) and (_____) their young are called (_____).
Better: Warm-blooded animals that are born alive and suckle their young are called (_____).
Explanation: The poor version has four blanks, making it confusing; the better version has one blank at the end and tests meaningful knowledge (the term "mammals").
π‘ Why this matters: Well-constructed completion items measure content knowledge, not puzzle-solving skills, and ensure all students have equal opportunity to demonstrate learning.
β Key Takeaways
For effective short-answer and completion test items, always prefer direct questions over incomplete statements to reduce ambiguity. When numerical answers are required, specify the desired unit of measurement to ensure consistent scoring and fair assessment. In completion items, limit blanks to one or two, place them at the end of the sentence, and only ask for meaningful, non-trivial information that reflects key learning objectives. These guidelines help create test items that accurately measure student understanding while minimizing confusion and grading inconsistencies.
π§ Quick Revision Questions
- Why is a direct question considered more desirable than an incomplete statement in test construction?
- What should you indicate when a question requires an answer expressed in numerical units, and why?
- What are three rules to follow when constructing completion items (fill-in-the-blank)?
- In the "poor" example for Topic-096, how many blanks were included, and why is this problematic?
- How does specifying the answer type (e.g., "in years" or "in kilograms") improve the fairness of a test item?
π Lecture 28 β Creating Constructed Response Test Items
π Overview: This lecture introduces constructed response test items, specifically subjective or essay-type questions. It explains the two main types of essay itemsβrestricted response and extended responseβand provides guidelines for constructing effective restricted response essay questions to measure higher-order thinking skills.
ποΈ Topics Covered
The lecture covers the nature of subjective test items, the distinction between restricted response and extended response essay items, the variety of learning outcomes assessable through restricted response questions, and a detailed set of nine guidelines for constructing restricted response essay type items.
π Lecture Summary
Topic- 097: Creating Constructed Response Test Items
Subjective test items (essay questions) are constructed response type questions that can be the best way to measure students' higher order thinking skills, such as applying, organizing, synthesizing, integrating, evaluating, or projecting, while simultaneously providing a measure of writing skills. Essay items can vary from very lengthy (5-10 pages), open-ended, to limited or restricted response (one page or less).
Essay type questions are divided into two types:
- Restricted Response Items
- Extended Response Items
Restricted response items pose a specific problem for which a student needs to recall suitable information, organize it, derive a defensible conclusion, and express it within the given limits of the question.
π Example: "List the similarities and differences in the process of cell division in meiosis and mitosis?"
A variety of learning outcomes can be checked using this format of essay question. Some of these include:
- Analysis of relationship.
- Compare and contrast positions.
- Explain cause-effect relationship.
- Organize data and support a viewpoint.
- Formulate hypotheses.
- Point out strengths and weaknesses.
- Integrate data from various resources.
A teacher can use this type of question under the following conditions:
- When supplying information is required instead of simple recognition.
- When limited numbers of content areas are needed to be tested.
Topic- 098: Guidelines for Constructing Restricted Response Essay Type Items
The lecture provides nine specific guidelines for constructing effective restricted response essay items:
- Statements should not be quoted directly from the text.
- Evaluate essay responses anonymously.
- Frame questions so that the examinee's task is explicitly defined.
- Specify the value and an approximate time limit for each question.
- Employ a larger number of questions that require relatively short answers rather than only a few questions that require long answers.
- Do not employ optional questions.
- Verify a question's quality by writing a trial response to the question.
- Prepare a tentative scoring key in advance of considering examinee responses.
- Score all answers to one question before scoring the next question.
- Make prior decisions regarding treatment of such factors as spelling and punctuation.
π‘ Why this matters: These guidelines help ensure the validity, reliability, and fairness of essay assessments. For instance, avoiding optional questions ensures that all students are tested on the same content, and scoring all answers to one question at a time improves consistency across student responses.
β Key Takeaways
The key distinction between essay questions is between restricted response (short, specific) and extended response (long, open-ended) items. Restricted response items are ideal for assessing higher-order thinking like analysis, comparison, and synthesis within a defined scope. When constructing these items, teachers must explicitly define the task, specify value and time limits, and avoid offering optional questions. For reliable scoring, it is critical to prepare a tentative key in advance, score all answers to one question before moving to the next, and make decisions about handling spelling and punctuation errors. Finally, teachers should always verify the quality of a question by writing a trial response themselves.
π§ Quick Revision Questions
- What are the two main types of essay-type questions?
- List three of the seven learning outcomes that can be assessed using restricted response essay items.
- According to the guidelines, why should teachers not employ optional questions in a restricted response essay test?
- What is the recommended scoring procedure for handling multiple essay questions?
- What is the purpose of a teacher writing a trial response to a question before administering it?
π Lecture 29 β Creating Extended Response Test Items
π Overview: This lecture focuses on extended response essay-type test items, covering their definition, advantages, limitations, and guidelines for writing them effectively. It also provides a detailed exploration of scoring rubricsβboth analytic and holisticβto ensure objective and reliable evaluation of student essays, a critical skill for educators and test developers.
ποΈ Topics Covered
The lecture begins with an introduction to extended response essay type items, explaining their suitability for assessing higher-order thinking skills like synthesis and evaluation. It then provides detailed guidelines for writing such items, followed by a discussion of scoring rubrics used for evaluation. Finally, it contrasts the two main types of scoring rubrics: analytic and holistic, with specific examples and implementation strategies for each.
π Lecture Summary
Topic- 99: Extended Response Essay Type Items
Subjective test items, specifically essay type questions, allow students to determine the length and complexity of the response. These questions are most suitable for measuring higher-level mental process skills like synthesis and evaluation. An example is: "Give examples to identify different forms of governments, which form of government is most suitable in the socio-economic and cultural context of our country? Keep your response limited to 5000-6000 words." Modifications can be made for different grade levels, for example, for grade V: "What are the teachings of Islam about the respect of elders? How do these teachings help us to make a good society? Marks will be given for correct information, your own point of view."
The advantages of essay type items are: they are effective for assessing higher order abilities (analyze, synthesize, evaluate), comparatively less time-consuming to develop, emphasize essential communication skills, and eliminate guessing. The limitations include: scoring is unreliable and time-consuming, and there is limited sampling of the content.
π Definition β Extended Response Essay Type Items: Subjective test items that allow students to determine the length and complexity of their response, ideal for measuring higher-order thinking skills.
Topic- 100: Guidelines for Writing Essay Type Items
The following guidelines are for writing essay type items when developing a test:
- Frame questions so that the examinee's task is explicitly defined.
- Specify the value and an approximate time limit for each question.
- Do not employ optional questions.
- Employ a larger number of questions that require relatively shorter answers rather than only a few questions that require long answers.
- Verify a question's quality by writing a trial response to the question.
- Prepare a tentative scoring key in advance of considering examinee responses.
- Score all answers to one question before scoring the next question.
- Make prior decisions regarding treatment of such factors as spelling and punctuation.
- Evaluate essay responses anonymously.
Topic- 101: Scoring Rubrics for Essay Type Items
Scoring of essay items is a time-consuming and difficult process. Reliability of the test demands that scoring should be consistent not only by the rater at different times, but by two independent raters as well. Scoring rubrics are descriptive scoring schemes developed by teachers or other evaluators to guide the analysis of writing products or processes. Scoring of essay type items focuses on increasing the objectivity (and consequently reliability) of marking. Judgments concerning the quality of a given writing sample may vary depending upon the criteria established by the individual evaluator. Writing samples are just one example of performance that may be evaluated using scoring rubrics; they have also been used to evaluate group activities, extended projects, and oral presentations.
Scoring rubrics provide at least two benefits: they support the examination of the extent to which specified criteria have been reached, and they provide feedback to students concerning how to improve their performance.
π Definition β Scoring Rubrics: Descriptive scoring schemes developed by teachers or other evaluators to guide the analysis of writing products or processes, aiming to increase objectivity and reliability of marking. π‘ Why this matters: Rubrics are essential for making the subjective process of grading essays more objective and fair, ensuring that students are evaluated based on clear, pre-defined criteria.
Topic- 102: Types of Scoring Rubrics
There are two general methods for scoring subject-matter essays: the analytic method and the holistic method.
Analytic Scoring Rubric requires developing a list of major elements that students are expected to include in an ideal answer and deciding the number of points awarded for each element. This type of rubric works best on the restricted response type essay question. When developing an analytic rubric, identify the certain elements of the answer which are more appropriate to the learning objectives. When assigning number to each element, ensure the total points match the essay's total value in relation to the overall test points. When a student gives a partially correct answer, partial credit is awarded, but assigning partial credits can increase inconsistency and decrease reliability. After reading a few papers, patterns of student errors and misconceptions start emerging; then craft a partial credit scoring rubric and use it to score all papers.
The lecture provides a table of factors and sub-factors for an analytic rubric, including:
- Examine problems: Diagnose problem and gather information, Identify facts, Integrate the information and Organize ideas, able to summarize, Define terms.
- Look for evidence: Relevance (evidence pertinent to the issue), Consistency (supporting material consistent with each other).
- Draw conclusions: Make comparative judgments from data and able to adjust opinions when new facts are found and reject information that is incorrect or irrelevant, Analyze and interpretation of data, Combine ideas and information in new ways.
- Make comparisons: Write similarities, Correlate result, Write differences, Look for reasons, Use relationship between phenomena, Present solution, Give well-reasoned conclusion.
- Decision making: Look for plausible alternatives to the conclusion drawn, Explore possibilities, Formulate purposeful judgment, Recognize and correct discrepancies, Reject the wrong answer and make correction.
π Definition β Analytic Scoring Rubric: A rubric that requires identifying specific content elements and assigning points to each, best for restricted response essays. π Example: In the provided table, "Identify facts" under "Examine problems" is a specific element to be evaluated for points.
Topic- 103: Holistic Scoring Rubric
The holistic scoring rubric requires making a judgment about the overall quality of each student's response to an item. No need to mark each specific content element. It is probably more appropriate for extended response essay type items involving a student's abilities to synthesize and create, and when no definite answer can be pre-specified.
One way to implement a holistic rubric is to decide on the number of quality categories into which you will sort the answers. A second way is to craft a holistic rubric that defines the qualities of a paper in each category (e.g., paper "A" or "B"). A third refinement is to select specimen papers that are good examples of each scoring category, then compare student papers to these specimens. A fourth way is to read the answer completely and compare with another to decide which is best, next best, and so on, resulting in a rough ranking. This approach cannot be applied to a large number of papers. Among these four approaches, the first three are consistent with a criterion-referenced or absolute quality standards grading philosophy, while the fourth is consistent with a norm-referenced or relative standard grading philosophy.
π Definition β Holistic Scoring Rubric: A rubric requiring an overall judgment of quality without marking specific content elements, best for extended response essays. π‘ Why this matters: Choosing between analytic and holistic rubrics depends on the test's purpose and the type of response being evaluated.
β Key Takeaways
- Extended response essay items are uniquely effective for assessing high-level cognitive skills like analysis, synthesis, and evaluation, but they suffer from scoring unreliability and limited content sampling.
- When constructing essay questions, guidelines emphasize clarity, pre-defined scoring, and anonymity to ensure fairness and validity.
- Scoring rubrics (analytic and holistic) are essential tools to increase objectivity and reliability in grading subjective essays.
- Analytic rubrics break down the scoring into specific components, making them ideal for restricted response essays, while holistic rubrics provide an overall quality judgment, better suited for extended response and creative work.
- The method of implementing a rubric (e.g., using category definitions, specimen papers, or sorting) should align with the intended grading philosophy (criterion-referenced vs. norm-referenced).
π§ Quick Revision Questions
- What are two main advantages and two main limitations of using extended response essay type items?
- List three of the nine guidelines provided for writing effective essay type items.
- What is the primary purpose of a scoring rubric, and what two benefits does it provide in the evaluation process?
- Explain the key difference between an analytic scoring rubric and a holistic scoring rubric, and provide an example of when each would be most appropriate.
- Describe the four different ways to implement a holistic scoring rubric, and explain which grading philosophy each is consistent with.
π Lecture 30 β Analyzing the Test-I
π Overview: This lecture introduces the two primary theories of test development β Classical Test Theory (CTT) and Item Response Theory (IRT) β and then focuses on item analysis as a key tool for evaluating test item quality. It explains what information item analysis provides, including item difficulty, item discrimination, and distractor analysis, and describes the appropriate stages for conducting these analyses.
ποΈ Topics Covered
The lecture covers two major theoretical frameworks for test development (CTT and IRT), then moves into the practical phase of item analysis. It details the two most common statistics reported (item difficulty and item discrimination), explains distractor analysis, and describes both graphical and numerical presentation methods. Finally, it discusses the two appropriate time stages for conducting item analysis: after administration but before scoring, and after scores have been reported.
π Lecture Summary
Topic- 104: Theories of Test Development
There are two widely perceived theories in psychosocial measurement: classical test theory (CTT) and item response theory (IRT). These two theories represent different measurement frameworks. CTT focuses on finding the true score of an individual on a test and is therefore called true score theory. Its fundamental equation is that the observed score equals the true score plus error. In contrast, IRT is known as latent trait theory because its focus is to find item characteristics as a function of a person's ability level.
π Definition β Classical Test Theory (CTT): A measurement framework focused on obtaining an individual's true score, where the observed score is comprised of the true score plus random error. π Formula: Observed Score = True Score + Error β This equation means that any test score a person receives is a combination of their actual ability level and some degree of measurement error. π‘ Why this matters: CTT provides the foundation for understanding why test scores are not perfectly accurate and why item analysis is needed to minimize error.
π Definition β Item Response Theory (IRT): A measurement framework known as latent trait theory that focuses on how item characteristics (like difficulty) function in relation to a test taker's underlying ability level.
Topic- 105: Item Analysis: Information Provided by Item Analysis
In this phase, statistical methods are used to identify any test items that are not working well. If an item is too easy, too difficult, failing to show a difference between skilled and unskilled examinees, or even scored incorrectly, an item analysis will reveal it. The two most common statistics reported are item difficulty (a measure of the proportion of examinees who responded to an item correctly) and item discrimination (a measure of how well the item discriminates between examinees who are knowledgeable in the content area and those who are not). An additional analysis often reported is the distractor analysis, which provides a measure of how well each of the incorrect options contributes to the quality of a multiple-choice item. In item analysis, analyses are conducted for the purpose of providing information about the items, rather than the test takers. Results can be presented graphically or numerically. The graphic presentation consists of response curves showing the test taker's estimated probability of a particular response as a function of their score on a measure of the general type of skills or knowledge measured by the item. The numerical presentation includes statistics that measure the difficulty of the item and the extent to which it discriminates between strong and weak test takers.
π Definition β Item difficulty: A statistic that measures the proportion of examinees who answered an item correctly. π Definition β Item discrimination: A statistic that measures how well an item differentiates between high-performing and low-performing examinees (those knowledgeable vs. not knowledgeable in the content). π Definition β Distractor analysis: An analysis of how well each incorrect option (distractor) in a multiple-choice item contributes to the item's overall quality.
Topic- 106: Appropriate Time for Item Analysis
There are two stages at which items can be analyzed: after administration but before scoring, and after scores have been reported. After administration, but before scoring, item analysis helps test developers identify errors in the scoring key or serious defects in the itemsβerrors or defects serious enough to exclude the item from scoring. This analysis enables test developers to focus their attention on a relatively small subset of items, helping them make any necessary corrections in the scoring key before the test takers' scores are computed. After scores have been reported, item analysis helps test developers select items for reuse in future forms of the tests, especially if scores on a future form will be linked to scores on the current form through common items. Item analysis is particularly useful in selecting a set of common items that represents the full range of difficulty of the items on the test.
β Key Takeaways
The two major theories of test development are Classical Test Theory (focused on true score = observed score minus error) and Item Response Theory (focused on item characteristics as a function of ability). Item analysis is a critical phase that uses statistics to identify poorly functioning test items, with the two key statistics being item difficulty (proportion correct) and item discrimination (how well the item separates high from low performers). Distractor analysis specifically evaluates the quality of incorrect options in multiple-choice items. The appropriate timing for item analysis is either before scoring (to catch scoring errors and serious item defects) or after scores are reported (to select items for future test forms). Item analysis results can be presented graphically through response curves or numerically through difficulty and discrimination statistics.
π§ Quick Revision Questions
- What is the fundamental formula of Classical Test Theory, and what does each component represent?
- How does Item Response Theory differ from Classical Test Theory in terms of its primary focus?
- What are the two most common statistics reported in an item analysis, and what does each measure?
- What is the purpose of a distractor analysis in item analysis?
- What are the two appropriate time stages for conducting item analysis, and what is the purpose of each stage?