๐ Lecture 30 โ Curriculum Design-ENG519 (Lessons 32โ38)
๐ Overview: This lecture covers the comprehensive process of testing, monitoring, and assessment in language curriculum design, including placement testing, achievement assessment, and proficiency testing. It then transitions into program evaluation approaches, change management strategies, and planning in-service courses, with the final section critically analyzing the Pakistani National English Language Curriculum (2006).
๐๏ธ Topics Covered
The lecture begins with observation of learning and short-term achievement assessment, then moves through diagnostic, achievement, and proficiency assessment with reliability, validity, and practicality criteria. It explores program evaluation types (formative, summative, illuminative), change introduction strategies, in-service course planning, and concludes with a critical appraisal of Pakistan's National English Language Curriculum 2006, including reading goals and textbook design issues.
๐ Lecture Summary
Topic-206: Observation of Learning
Monitoring learners' progress occurs at the level of the learning activity, not assessing learners but directed toward tasks. The purpose is to see if changes to learning activities are necessary. Four questions must be asked when observing learning activities: What is the learning goal? What are the learning conditions that would lead to achievement? What are the observable signs these conditions are occurring? What are the design features that set up these conditions?
For example, in a spoken fluency development activity, learning conditions include learners focused on meaning, limited language demands, and pressure to perform at a higher level. Signs include involvement in communicating, reasonably high speed with few hesitations, and listener comprehension. Design features include opportunity for preparation, task repetition, familiar topics, and time pressure.
Topic-207: Short-term Achievement Assessment
Short-term achievement assessment looks at the product of activities to see if learners are making progress daily or weekly. It provides feedback to teacher and learners and can have motivational purposes. In intensive courses, weekly tests are common. For activities like speed reading or dictation, learners can record performance on graphs to show progress.
A performance objective is a statement with five parts (Brown, 1995): (1) who should achieve the objective (subject), (2) what the person should be able to do (performance), (3) under what conditions (conditions), (4) how performance will be tested (measure), and (5) what level must be reached (criterion).
๐ Formula: Performance Objective = Subject + Performance + Conditions + Measure + Criterion
Good short-term achievement assessment should provide a clear, easily interpreted record of progress, motivate learners, not occupy too much class time, and be a regular expected part of class activity.
Topic-208: Diagnostic Assessment
Diagnostic assessment finds gaps and weaknesses in learners' knowledge so something can be done about them, and also finds what learners know well to avoid wasting time. It is a vital part of needs analysis before and during a course. Findings determine what goes into a course.
The Vocabulary Levels Test (Schmitt, 2002) is an example. It helps teachers decide whether learners should focus on high-frequency vocabulary, academic vocabulary, or low-frequency vocabulary. This is crucial because high-frequency and low-frequency vocabulary require different teaching strategies. For high-frequency vocabulary, every word deserves individual attention. For low-frequency vocabulary, focus should be on vocabulary coping and learning strategies (guessing from context, using word parts, mnemonics, decontextualized learning).
Diagnostic assessment can include analysis of language use (written compositions, reading tasks, spoken performance) and learner self-assessment using checklists or scales, though self-assessment can be subjective.
๐ Definition โ Diagnostic Assessment: Assessment used to find gaps and weaknesses in learners' knowledge so something can be done about them.
Topic-209: Achievement Assessment
Achievement assessment measures both learner achievement during a course and the effectiveness of the course. May include final tests and tests part-way through. Achievement tests are based on material taught, learners usually know what will be asked, and they are criterion referenced (assessed against a pre-determined criterion, not each other).
Mastery tests set a high criterion (e.g., 80% or 90%). Learners can have repeated opportunities to study and sit tests until they reach mastery. Thinking behind mastery tests: if something is learned, it should be learned well; every learner is capable of mastery; teacher's responsibility is to assess learning; mastery is rewarding and motivating.
Achievement assessment is not all done by tests. For composition writing, it is better to base assessment on several pieces of class work. The washback effect refers to how tests affect teaching and learning, motivating and guiding learning.
Topic-210: Proficiency Assessment
A proficiency test draws items not from the course but from the language itself, independent of any course, to show how much learners know of the language. TOEFL and IELTS are examples. TOEFL's internet-based test criterion for undergraduate entry is usually 79 or 80 out of 120. IELTS uses a 9-point scale, with 6, 6.5, or 7 typically needed for university entrance.
A proficiency test can have profound washback effect on a language course, making course content resemble the test format and items. Courses that copy the format of the test are trying to make the proficiency test like an achievement test. Alderson and Hamp-Lyons (1996) show that washback effects are complex, affecting different teachers differently.
๐ก Why this matters: Understanding washback helps teachers design courses that prepare students for proficiency tests without sacrificing sound pedagogical principles.
Topic-211: Good Assessment: Reliability, Validity and Practicality
Three criteria for examining assessment tools: reliability, validity, and practicality.
Reliability: A reliable test gives results not greatly upset by conditions not intended to measure. If the same person sat the test twice, they should get nearly the same result. A test is more reliable if: always given under same conditions, consistently marked, has large number of assessment points, and has clear, unambiguous questions and instructions.
Validity: A valid test measures what it is supposed to measure. Face validity means the test looks like what it claims to be (e.g., a reading test looks like a reading test). Content validity involves checking if the test covers the appropriate content. Validity exists when a test is used for the purpose for which it was designed.
Practicality: Examined by looking at cost, time to administer, time to mark, number of people needed, and ease of interpreting results. Tests can be made more practical by reusable papers, careful formatting, and objectively scored items (true/false, multiple choice). Requirements of practicality, reliability, and validity don't always agree โ short tests are practical but not very reliable or valid. Possible reliability and validity should be preferred over practicality.
๐ Definition โ Reliability: A test gives results that are not greatly upset by conditions that the test is not intended to measure.
Topic-212: What is an Evaluation?
The broadest kind of evaluation looks at all aspects of curriculum design to see if the course is the best possible course to implement. Evaluation requires looking at both results and the planning/running of the course. Most evaluations are more narrowly focused, answering questions about standards, preparation, satisfaction, or cost-effectiveness. Carrying out an evaluation is like carrying out research โ the evaluator must be clear about what question is being asked.
Topic-213: Purpose and Audience of the Evaluation
Weir and Roberts (1994) distinguish two major purposes: program accountability (extent to which those involved are answerable for quality, usually at end points for external audiences) and program development (designed to improve quality during implementation, may involve staff). These are referred to as formative, illuminative, and summative evaluation.
Kiely and Rea-Dickens (2006) make a three-way scope distinction: large-scale evaluations, teacher-led evaluations, and management-led evaluations.
The first critical step in evaluation is finding out whom the evaluation is for and what kind of information they value. This determines confidentiality, what information to gather, and whether data will be provided willingly. The evaluator must also determine if there is a hidden purpose. At the end of the preparatory stage, the evaluator should be able to tell whether the evaluation is worth doing, possible, how long it might take, how much it might cost, whether they are willing to do it, and what evidence will be gathered.
Topic-214: The Type and the Focus of the Evaluation
Formative evaluation has the purpose of forming or shaping the course to improve it. Summative evaluation makes a summary or judgement on quality or adequacy for comparison. These different purposes affect data type, presentation, and timing.
Other distinctions: short term vs long term, process vs product (process focuses on learning and teaching; product focuses on results), and whether evaluation includes cognitive, affective, and resource factors.
It is better to have a small amount of relevant data than a large amount that doesn't address main concerns.
Topic-215: Gaining Support for the Evaluation
Finding weaknesses carries the idea that someone is to blame, which is threatening. For honest data, those involved must feel the evaluation is worthwhile and not personally threatening. This requires meeting with those involved and involving them in planning. Some evaluations involve a respected outsider who makes gaining cooperation a prerequisite.
Generally, assumptions behind evaluation are: this course is worth improving, people running it are capable, they have freedom to make changes, and improvements will make a better course for all.
Topic-216: Gathering the Information
Tools of needs analysis and evaluation are similar but used for different purposes. Evaluation gathers a wider range of data, including performance of teachers and learners, lesson observations, and examination of achievement. It can also look at the environment including administrative procedures, resources, and how outsiders view the course.
Topic-217: Formative Evaluation as Part of a Course
If curriculum design is seen as a continual process, evaluation should be built into a course. This can be done through: negotiation between teacher and learners, periodic peer observation, regular staff meetings, self-evaluation forms, learner course evaluation forms, class time for learner discussion and feedback, and occasional outside evaluators.
Topic-218: Illuminative Evaluation
Illuminative evaluation seeks to find out how different aspects of the program work or are implemented, providing deeper understanding of teaching and learning processes without necessarily seeking to change the course. Questions might include: How do students carry out group-work tasks? What error-correction strategies do teachers use? What reading strategies do students use with different texts?
Example: A teacher teaching a reading skills course finds through student questionnaires that learners perceive the purpose of activities differently than intended. The teacher comes to understand that learners' perceptions reflect what they are most interested in or feel they need most help with.
Topic-219: Summative Evaluation
Summative evaluation seeks to make decisions about the worth or value of different aspects of the curriculum. It is concerned with determining program effectiveness, efficiency, and acceptability. Questions include: How effective was the course? What did students learn? Were the objectives adequate? Were teaching methods appropriate?
Effectiveness can be measured through mastery of objectives (examining each objective and choosing criteria for successful achievement) and performance on tests (unit tests, class tests, formal exit tests). However, mastery of objectives doesn't provide a full picture โ objectives can be achieved despite defects in a course.
Topic-220: Issues in Program Evaluation
Weir and Roberts (1994) propose evaluation characterized by: need for both inside and outside commitment, central interest in improvement and product value, commitment to deeper professional understanding of educational change, systematic documentation, and willingness to embrace both qualitative and quantitative methodology.
Topic-221: The Results of an Evaluation
Results presentation involves ethical issues of confidentiality and consideration for others' feelings. The written report is usually accompanied by an oral report to ensure clear understanding and to say things that couldn't be put tactfully in writing. The report needs to be acted on, with later follow-up to monitor effects and implementation. Surprisingly, evaluation often results in very little change.
Topic-222: Language Program Evaluation: Significance
According to Richards (2001), evaluation is "systematic gathering of information for the purpose of making decisions." Significance includes improving language curriculum design, looking at program worth/procedures/product, assessing effectiveness and efficacy, and conducting a continuous needs analysis for improvement.
Topic-223: Language Program Evaluation: Approaches
Brown's (1995) definition: evaluation is "the systematic collection and analysis of all relevant information necessary to promote the importance of the curriculum and to assess its effectiveness within the context of the particular institutions involved."
Approaches include: Product oriented (focuses on goals and objectives matching instruction), Process oriented (deals with change, innovation, and improvement in implementation), Static characteristic (external agency perspective on overall standing and ranking), and Decision facilitation (cyclic process gathering information about needs, environment, and situation analysis).
Topic-224: Language Program Evaluation: Dimensions
Three dimensions: Formative vs summative (throughout learning vs end of year/unit), Process vs product (how goals achieved vs goals achieved), and Quantitative vs qualitative (descriptive data via questionnaires/textual analysis vs general observation).
Topic-225: Mastery Decisions in Program Evaluation: Student Performance
Program success or failure can be measured through students' performance and achievement. Mastery decisions are based on whether students have reached required competence level, failed, or are underperforming. Thom Hudson (2007) discusses "mastery testing" which helps teachers analyze how to know and what to know, and how to view and what to teach.
Topic-226: Epistemological, Ethical and Technical Issues
Epistemological issues: What students have achieved, how to assess their cognition level and knowledge about language and language learning processes. Ethical issues: How results will be used, ensuring confidentiality. Technical issues: Cut-off points for mastery, test score usefulness for informing whether course is running smoothly.
Topic-227: Implications
Thom Hudson (2007) discusses practical implications of mastery decisions โ making decisions about student success/failure helps analyze overall program evaluation, diagnose weaknesses in course structure, and bring objectivity and neutrality to the program, validity and reliability to the whole course.
Topic-228: Tailoring the Evaluation to Fit the Context: Significance
Warwick B. Elley (1981) discusses molding evaluation process to fit contextual needs. Evaluation is a controlled experiment done systematically. Program evaluation involves variables like assisted principles, trained teachers, and student samples. Ethnographic studies involve external agents observing classroom dynamics to determine program efficiency. This provides firsthand information but involves subjectivity.
Topic-229: Tailoring the Evaluation to Fit the Context: Principles
Principles include deciding about tools (questionnaire, interview sample, textual analysis), knowing about the evaluator (teacher, administrator, or external agent), analyzing sample size, who gathered information, how sample was chosen, design type (qualitative or quantitative), and program aims.
Topic-230: Tailoring the Evaluation to Fit the Context: Selecting and Developing Instruments
Warwick B. Elley (1981) emphasizes selecting and developing instruments (questionnaires, ethnographic studies, structured/semi-structured interviews, textual data) to reshape evaluation. This includes notions of pre-test (testing knowledge before formal evaluation begins), administering tests when students are prepared, marking according to contextual needs, and monitoring instructions.
Topic-231: Introduction to Introducing Change
Curriculum design usually involves change when a new course replaces an existing one. Change or innovation needs consideration so planning work is rewarded by acceptance and appropriate use. An effective way is involving users closely in design and development. Types of change include introducing new course books, changing approaches, new techniques, changing national tests, introducing computer-assisted learning, and moving to standards-based assessment.
Attempts to introduce change are not always successful due to cultural inappropriateness or flawed introduction processes. The cascade model of diffusion (training a few teachers to train others) often doesn't succeed. Two important aspects: change in curriculum and change in minds of people affected. Innovation theory is relevant to curriculum change.
Topic-232: Steps in Introducing Change
Steps to increase likelihood of successful change:
- Make sure change is really needed
- Plan type of change so it's not too great or too small
- Make sure enough people see change is possible
- Use wide range of change strategies
- Be prepared for change to take a long time
Topic-233: Seeing the Need for Change and Deciding on the Size of the Change
Change occurs most easily if people are dissatisfied with the present situation. Stoller's research suggests the size of change should not be too great or too small. Six scales for describing size: explicitness, visibility, originality, compatibility with past practices, difficulty/complexity, and flexibility. The best size is in the middle area of each scale โ the Goldilocks syndrome (not too big, not too small, but just right).
Example: Book Flood experiment (Elley and Mangubhai, 1981): Spending ยพ of class time on extensive reading was a big change from normal teacher-fronted courses. However, ยผ of time remained the same, little training was needed, attractive resources available, and teachers read quietly when learners did, reducing work. One teacher couldn't accept the change and his class's achievement was at control group level.
Topic-234: Realistic Change
Change must be obvious that it's possible and will be an improvement with more gains than losses. Useful to view from several viewpoints: resources to support change, people involved, and people who will receive ultimate benefit (learners). Learners may be directly involved through negotiated syllabus.
Topic-235: Teacher Beliefs
What teachers do in the classroom is determined by what they believe. The notion of teacher as transmitter has been replaced by recognition of teachers' complex mental lives โ teacher cognition โ "the hidden side of teaching" (Freeman, 2002). Introducing change means addressing teacher beliefs. However, teacher beliefs and behavior are not necessarily the same โ contextual factors can facilitate or constrain practice.
Professional development contributes to teacher cognition. Both pre-service and in-service training participants arrive with pre-existing beliefs based on their experiences as learners and teachers.
Topic-236: Using a Variety of Change Strategies
Kennedy (1987) describes three major approaches to change:
- Powerโcoercive: Change through authority, rules, top-down pressure
- Rationalโempirical: Change through explaining, justifying, showing reasons
- Normativeโre-educative: Change through discussion, involvement, negotiation
These should support each other rather than be alternatives. Change is more likely if people see authority support (powerโcoercive), good reasons (rationalโempirical), and feel they participate and "own" the change (normativeโre-educative).
Topic-237: Innovation, Management and Long-Term Support
White (1987) sees innovation as involving management of stages: defining aims, defining end results, gathering information, defining what has to be done, action, and reviewing/evaluating. Markee (1997) provides principles for curricular innovation: innovation is complex, change agents must effect desired changes, good communication is key, strategic approach to managing change is needed, innovation is messy and unpredictable, it takes longer than anticipated, proposals will be misunderstood, implementers need a stake, and change agents should work through opinion leaders.
Topic-238: Introduction and Significance of In-Service Courses
In-service courses involve teacher development after initial training and teaching experience. They may be long-term (Diplomas, Masters) or short-term (hours or days). Planning a short in-service course involves similar decisions to planning a language course, and they are a major way of bringing about innovative curriculum change.
Topic-239: Features of an Effective Workshop
Goals: Workshops can have five goals โ understanding and remembering new ideas, experiencing and evaluating exercises, producing material or exercises, planning units of work, and problem solving.
Content: New information sources include pre-reading, prepared talks, model activities, spontaneous discussion, and feedback.
Presentation: Participants need to be interested and active. Over a two-hour period, there should be about four changes in activity type, group size, focus of attention, or medium. Arrangements for tea/coffee and food are almost obligatory.
Topic-240: Procedures and Activities for Reaching Goals
Ellis (1986) distinguishes experiential practices (actual teaching, real or simulated) and awareness-raising practices (conscious understanding of principles, techniques, issues). Awareness-raising involves choosing from three ranges: input data (videos, transcripts, readings, textbooks, lesson plans, case studies, student work), tasks or operations (comparing, preparing), and workshop procedures (lectures, group discussion, individual work, demonstrations).
Topic-241: Understanding and Remembering Ideas
Understanding enables better-informed experiencing, making, planning, or problem solving. Lamb (1995) found a range of effects one year after a workshop: no uptake, confusion, mislabelling, appropriation, assimilation, adaptation and rejection, and engagement. Lamb's main point: the main focus of short in-service workshops should be teachers' beliefs themselves. Workshops should involve other goals besides understanding, provide written records, and set modest, realistic aims.
Topic-242: Experiencing and Evaluating
An important way of making a technique part of teaching is experiencing it as learner and teacher and reflecting. An interesting principle: "If you want teachers to use the activities you demonstrate, do a poor demonstration" โ a good demonstration makes them think "I couldn't do it that well," while a poor one makes them think "I can do better."
Topic-243: Making Material
Two major approaches: Perfection through gradual approximation (quickly make first attempt, receive feedback, revise) โ advantages include several opportunities and speed as aim. Perfection through preparation (analyze model, study steps, plan, then make) โ advantages include reasonably good first attempt and closer theory-practice relationship.
Topic-244: Planning Lessons and Units of Work
This goal is particularly useful when participants have been introduced to a new activity and must decide how to use it in their own teaching. Harmer (1984) describes an activity using blank timetables and cards with activity names and times.
Topic-245: Sequencing the Components of a Workshop
Palmer found experiencing, problem solving, and making activities were more likely to lead to adoption than understanding. Effective sequencing: experience innovation first, reflect on impact (planning), adapt to own circumstances (making), then evaluate. A major decision is whether speaker input comes at the beginning or after exploring needs.
Topic-246: Evaluating Workshops
Workshops can be assessed using parts of the curriculum design diagram โ looking at needs and whether they were met, whether environmental constraints were considered, whether sensible principles were followed.
Topic-247: Introduction to Integrating Internet-Based Materials
This lesson provides guidance on three approaches to integrating internet-based reading materials. Design is determined by teacher's pedagogical approach, technological expertise, and students' language proficiency. Lesson designs distinguish themselves in: learning resources, scope of learning environment, learning tasks, and degree of teacher/learner involvement.
Topic-248: Developing Internet-Based Lessons: Sound Pedagogical Rationale and Design
Chun and Plass (2000) mention WWW capabilities: universal availability of authentic materials, communication through networking, multimedia capabilities, and nonlinear hypermedia structure. However, there are challenges: limitations on interactivity and bandwidth, students getting lost in hyperlinked structure, no control over quality/accuracy, and little theoretical/empirical research demonstrating how to use materials.
Topic-249: Strengths and Challenges in Integrating Internet-Based Materials
Five key questions for evaluating internet-based reading lessons: Does the design justify use of the medium? Are materials and tasks appropriate for proficiency level? Do activities engage learners in real-world meaningful tasks? How do students demonstrate learning? Are all instructions clearly stated? Are all hyperlinks functional?
Topic-250: Instructional Guidelines
Are students prepared for project-oriented work? Provide clear guidelines on process and nature, allow topic/materials selection, but teacher plays role of guide. Are students familiar with research process? Introduce models like McKenzie's "Iterative Research Cycle" (questioning, planning, gathering, sifting, synthesizing, evaluating). Do students know how to search the Internet? Additional training in search engines and information-seeking skills may be needed.
Topic-251: What Can WWW Offer ESL Teachers? -I
Models and guidelines based on theoretical/empirical research are rare. Focus is on exploration of authentic materials in visual and verbal/textual modes, concentrating on reading skills. Three different approaches to using internet-based resources will be presented.
Topic-252: What Can WWW Offer ESL Teachers? -II
Example: Li and Hart introduced "Exchange" magazine offering platform for teachers and students to exchange views through discussion board. Students wrote for the magazine, developed e-writing skills, and gained cultural awareness by exchanging writings. This incorporated students with diverse demographic backgrounds.
Topic-253: Advantages of Electronic Medium Over Print
Internet-based language learning (IBLL) engages students through emails and blogs, promotes learner autonomy, transforms passive into active learning, helps students engage in collaborative work, provides retrievable feedback mode, and is a positive forum for shy/passive students.
Topic-254: Templates for Interactive Exercises
Templates vary and depend on teacher discretion. Video conferencing exposes learners to native speaking style. Digital text (digital version of printed books) can be listened to, improving listening and reading skills. Talking texts enhance listening proficiency and allow attention to intonation patterns.
Topic-255: Background to English Language Curriculum Policies in Pakistan
In Pakistan, public schools follow the national curriculum while private schools can adopt or devise their own. Textbooks are presumed to be based on the relevant curriculum/syllabus. This lesson critically discusses the 2002 and 2006 versions of the national English language curriculum.
Topic-256: Evaluation of NCEL 2006: Grades I-XII -I
The 2006 curriculum is more comprehensive than 2002. It discusses English's importance in Pakistan and introduces English as a language from grade one, to be used as medium of instruction. However, no attempt examines patterns of English use in the country โ general phrases like "higher learning" and "better career options" indicate an ad-hoc approach.
Topic-257: Evaluation of NCEL 2006 -II
The curriculum has a "product dimension" approach using both "knowledge based" and "skills based" objectives organized as competencies, further classified into standards, composed of benchmarks for different developmental levels (I-II, III-V, VI-VIII, IX-X, XI-XII), specified into student learning objectives grade-wise.
It follows both "language use" and "language analysis" approaches, emphasizing reading and writing skills while dealing with all four skills. It stresses both accuracy and fluency.
Topic-258: Evaluation of NCEL 2006 -III
Textbooks remain one of the most extensively used resources. The curriculum proposes textbooks will not be the only resource for assessment but a "contributing resource." Teachers should "adapt and supplement" material. The curriculum provides guidelines for textbook writers to base each unit on relevant student learning outcomes.
Topic-259: Evaluation of NCEL 2006 -IV
Strengths: focuses on all language aspects (structure, phonology, lexis, discourse, four skills), suggests variety of activities and texts, introduces purposeful reading and writing, deals with functional oral interaction.
Weaknesses: no mention of listening comprehension activities, covert emphasis on reading/writing despite oral skills being necessary in Pakistani job market, 50% of suggested themes are didactic, stress on Pakistani cultural content at lower grades.
Topic-260: Review of Reading Goals in NCEL
Standard 1: All students will search for, discover, and understand a variety of text types through tasks requiring multiple reading and thinking strategies for comprehension, fluency, and enjoyment.
Standard 2: All students will read and analyze literary text to seek information, ideas, enjoyment, and relate experiences to common humanity.
Asghar (2013) notes that due to classroom situations and curriculum development issues, teachers and learners couldn't achieve NCEL goals. Texts don't relate to learners' immediate situations, and language/themes may be above or below learners' level.