π Lecture 15 β Q&RD: Extraneous Factors to Avoid
π Overview: This lecture explains how extraneous variables threaten the internal and external validity of research. It covers the specific types of extraneous factorsβhistory, maturation, control group contamination, testing effects, and researcher/participant biasesβand provides strategies for avoiding them. Understanding these threats is critical for designing rigorous, credible research.
ποΈ Topics Covered
The lecture begins by defining internal and external validity and the role of extraneous and confounding variables. It then examines history and maturation as temporal threats, followed by four types of control group contamination: rivalry, diffusion, compensatory equalization, and demoralization. Five ways testing can spoil results are detailed, including instrumentation, pretest/posttest effects, and time of measurement effects. Finally, the lecture covers the Pygmalion effect, Hawthorne effect, treatment novelty, accumulative treatment effects, treatment fidelity, and treatment strength-time interaction.
π Lecture Summary
Topic-068: Q&RD: Extraneous Factors to Avoid β Internal and External Validity
Any variable you are not intentionally studying in your dissertation is an extraneous variable that could threaten the internal validity of your results. Researchers try to control these extraneous variables so they do not become confounding variables. When an extraneous variable changes systematically along with the variables you are studying, this is called a confounding variable.
Extraneous factors are a research minefield. They may contaminate your data, affecting internal as well as external validity, as internal and external validity are not mutually exclusive. If the results (in the dependent variable, DV) are not only due to the independent variable (IV), then the results are not generalizable.
π Definition β Extraneous variable: Any variable that you are not intentionally studying that could threaten the internal validity of your results.
π Definition β Confounding variable: An extraneous variable that changes systematically along with the variables you are studying.
π Definition β Internal validity: The extent to which the results in the DV are solely due to the IV. If results are contaminated, internal validity is weakened.
π Definition β External validity: The extent to which results can be generalized beyond the study; it is compromised when extraneous factors contaminate the data.
Topic-069: EFTA: History & Maturation
The factor of history includes events taking place at different points in time. Longitudinal studies are more vulnerable and may be affected by such factors. Studies based on natural changes (in society) and physical changes (in personality) may influence the results of the study. External events occurring between pre-test and post-test may influence the results.
Maturation is an internal extraneous factor involving natural processes. It includes changes in physical, emotional, and cognitive structures. For example, in a study of L2 acquisition by young learners using different teaching methods, the IV is the teaching method, and the DV is improvement in L2. However, natural cognitive maturation in young learners could also cause improvement, confounding the effect of the teaching method.
π Definition β History (extraneous factor): External events occurring at different points in time, especially between pre-test and post-test, that can influence study results.
π Definition β Maturation (extraneous factor): Natural, internal processes of physical, emotional, or cognitive change that occur over time and can influence study results independently of the IV.
π Example: In a study testing a new teaching method (IV) on L2 improvement (DV) in young learners, the natural cognitive maturation of the children over the study period is a maturational threat. Any observed improvement could be due to maturation, not the teaching method.
Topic-070: EFTA: Control Group Contamination
Control group contamination occurs when the control group influences the results because of the treatment given to the experimental group. The control group may influence results in four ways:
-
Control group rivalry (John Henry effect): The control group tries to outdo the target (experimental) group. This happens when a group is explicitly labeled as the control group, or if group identities are not kept secret.
-
Experimental treatment diffusion (compromise): Treatment conditions leak when groups are in close proximity and there is a possibility for participants to discuss the kind of treatment they are receiving.
-
Compensatory equalization of treatments: Extra materials or special treatment are given to the control group, distorting the difference between the control group and the experimental group.
-
Demoralization (boycott) of the control group: The control group, feeling resentful about the special treatment of the experimental group, lowers their performance, acting as a potential contaminator.
π Definition β John Henry effect: A form of control group rivalry where the control group tries to outperform the experimental group because they know they are the control group.
π Definition β Experimental treatment diffusion: When treatment conditions spread from the experimental group to the control group because the groups are in close proximity and communication is possible.
π Definition β Compensatory equalization: When researchers provide extra materials or special treatment to the control group to compensate for the experimental treatment, distorting the intended experimental-control group difference.
π Definition β Demoralization (boycott): When the control group becomes resentful and deliberately lowers their performance because the experimental group receives special treatment.
π‘ Why this matters: These four types of contamination can completely invalidate a comparison between experimental and control groups. To avoid them, keep group identities secret and physically separate groups.
Topic-071: EFTA: Testing
Five ways the results may be spoiled by testing procedures:
-
Instrumentation: Different instruments are used for pre-test and post-test. For example, using MCQs for the pre-test and an essay for the post-test to measure English proficiency. The solution is parallel testingβusing equivalent forms of the same test.
-
Measurementβtreatment interaction: The same instrument is used for all types of variables, but it may not be appropriate for all. For example, teaching grammar in an EFL class and then testing it only with MCQs (not essay type) focuses on recognition, not production.
-
The pretest effect: Taking a pretest makes participants aware of the material, giving them an advantage over another group that did not take the pretest.
-
The posttest effect (the click of comprehension effect): Administering a posttest can itself influence results, especially during an oral interview where participants suddenly understand the material because of the test format.
-
Time of measurement effect: Testing immediately after treatment gives a misleading idea about long-term effects. To control this, test participants after a week or so without warning.
π Definition β Instrumentation threat: When different measurement instruments are used at different times (e.g., pre-test vs. post-test), causing differences in results that are due to the instruments, not the treatment.
π Definition β Pretest effect: The awareness of material gained from taking a pretest, which can influence post-test performance independently of the treatment.
π Definition β Posttest effect (click of comprehension effect): The act of taking a posttest itself causes participants to understand or perform differently, especially in oral interviews.
π Definition β Time of measurement effect: Testing immediately after treatment may show temporary effects; results may not be lasting. Delayed testing without warning provides a better measure of long-term effects.
Topic-072: EFTA: Avoiding Various Factors
Several ways in which measuring the dependent variable (DV) can distort study results:
-
Pygmalion effect: The researcher's effectβthe researcher's perception of a participant's behavior influences how they treat the participant, which in turn affects the participant's actual behavior.
-
Hawthorne effect: Participants behave differently simply because they are aware that they are being observed or participating in a study.
-
Treatment intervention: The novelty and disruption of a new treatment can cause temporary effects that are not representative of long-term results.
-
Accumulative treatment effect: When several treatments are given, the order of treatments can affect results. Counterbalanced designs are used to control for this.
-
Treatment fidelity: Whether the treatment was administered in the correct manner, as intended.
-
Treatment strengthβtime interaction: Whether sufficient time was given for a noticeable effect to occur. For example, a teaching methodology needs enough time before any effect can be observed.
π Definition β Pygmalion effect: A researcher's expectation or perception of a participant's behavior influences the researcher's actions, which then causes the participant to behave in accordance with that expectation.
π Definition β Hawthorne effect: Participants alter their behavior because they know they are being studied, not because of the treatment itself.
π Definition β Treatment fidelity: The extent to which the treatment was delivered consistently and correctly as planned by the researcher.
π Example: If a study aims to test a new teaching methodology, the treatment might show no effect if it was not given enough time (e.g., only one class period). This is a treatment strengthβtime interactionβthe treatment needs sufficient duration for a noticeable effect to occur.
β Key Takeaways
Extraneous variables are any unstudied variables that threaten internal validity; when they change systematically with the IV, they become confounding variables. History (external events over time) and maturation (natural internal changes) are two key temporal threats, especially in longitudinal studies. Control group contamination can occur through rivalry (John Henry effect), treatment diffusion, compensatory equalization, or demoralization, so group identities must be kept secret. Testing can spoil results through instrumentation mismatches, pretest or posttest effects, and inappropriate timing of measurement. Finally, researcher and participant biasesβPygmalion effect, Hawthorne effect, treatment novelty, accumulative effects, and treatment fidelity or strength-time issuesβmust be controlled to ensure valid, generalizable findings.
π§ Quick Revision Questions
- What is the difference between an extraneous variable and a confounding variable?
- Explain how maturation could threaten the internal validity of a study on L2 acquisition in young learners.
- List and briefly describe the four types of control group contamination.
- What is the instrumentation threat and how can it be controlled in a pre-test/post-test design?
- How does the Pygmalion effect differ from the Hawthorne effect in terms of who is influenced?
π Lecture 16 β Data Collection Procedures
π Overview: This lecture introduces the two main categories of data collection procedures in applied linguistics research: observational and instrumental. It explains who or what is used to gather data and how different procedures provide answers to research questions.
ποΈ Topics Covered
The lecture covers data collection procedures, focusing on observational procedures (including self-observation, outside observers, and interviewers) and instrumental procedures (including questionnaires, tests, and surveys). Examples of each type are provided, such as protocol analysis, ethnographic studies, Likert scales, and standardized tests like TOEFL and IELTS.
π Lecture Summary
Topic-073: Data Collection Procedures
Once research designs are decided, researchers need to select the data collection procedure. There are many ways for data collection, and the value of a data collection procedure is how well it provides answers to the research questions. Commonly used data collection procedures in applied linguistics fall into two main categories: observational and instrumental.
π Definition β Observational DCP: Data collection procedures that rely on human observation, including participant observation, non-participant observation, sheets, and judges/raters.
π Definition β Instrumental DCP: Data collection procedures that rely on instruments or tools, including questionnaires (closed-open) and tests.
Topic-074: DCP: Observational Procedures
These are data gathering procedures based on observation with many possibilities. The observer can be self (introspection/retrospection) or others (participants β full, non-, or partial). Other roles include interviewers and judges-raters.
Topic-075: DCP: Observational Procedures-Examples
Data gathering procedures based on observation can occur through visual observation using human observation for data gathering. The self as observer uses participants as observers of their own behavior, including protocol analysis (their own internal cognitive states β introspection/retrospection). Outside observers are others who are observing, including full participants, partial participants, and non-participants. This is used in ethnographic studies and conversational analysis. The interviewer conducts observation under highly structured conditions with paper-and-pencil data recording that can be structured, semi-structured, or unstructured.
π Example: In an ethnographic study, a researcher might live within a community as a full participant observer, recording language use patterns in daily interactions.
Topic-076: DCP: Instrumental Procedures
These are data gathering procedures based on instrumentation with many possibilities. They include questionnaires (closed-form or open-form) and tests (discrete items or constructed response).
π Definition β Closed-form questionnaire: A questionnaire with predetermined response options (e.g., yes/no, agree/disagree). π Definition β Open-form questionnaire: A questionnaire allowing free, unstructured responses from participants.
Topic-077: DCP: Instrumental Procedures-Examples
Data gathering procedures based on instrumentation use a range of devices β questionnaires and tests. They can reach maximum participants and are very economical ways to collect data. Surveys may be open or closed. Standardized tests include TOEFL (ETS) and IELTS (UoC). Questionnaires may use Yes-No Questions, Agree-disagree Questions, or the Likert scale (a scale measuring agreement levels). Examples of such tests include the First Certificate in English (FCE), Comprehensive English Language Test (CELT), and McArthur Communicative Development Inventory (CDI). Tests can be norm-referenced (comparing to a norm group) or criterion-referenced (measuring against a fixed standard).
π Formula: Likert scale β A psychometric scale where respondents specify their level of agreement or disagreement with a statement (e.g., Strongly Agree β Agree β Neutral β Disagree β Strongly Disagree). π‘ Why this matters: Choosing between norm-referenced and criterion-referenced tests affects how you interpret results and what conclusions you can draw about your participants' abilities.
π Example: A researcher studying learner attitudes might use a questionnaire with Likert-scale items like "I enjoy speaking English in class" (Strongly Agree to Strongly Disagree), then administer a standardized test like TOEFL to measure proficiency.
β Key Takeaways
Students must remember that data collection procedures are divided into observational (based on human observation, including self, participants, interviewers, and judges) and instrumental (based on tools like questionnaires and tests). Observational procedures vary by the observer's role (full, partial, non-participant) and structure (structured to unstructured). Instrumental procedures include both closed-form and open-form questionnaires, as well as tests with discrete items or constructed responses. The choice of procedure directly impacts how well it answers research questions. Standardized tests like TOEFL and IELTS, alongside instruments like Likert scales, are common in applied linguistics research.
π§ Quick Revision Questions
- What are the two main categories of data collection procedures in applied linguistics?
- What is the difference between full participant observation and non-participant observation?
- What is protocol analysis and who uses it in observational data collection?
- What is the difference between closed-form and open-form questionnaires?
- What is the distinction between norm-referenced and criterion-referenced tests?
π Lecture 17 β Qualities of Good Data-Gathering Procedures
π Overview: This lecture examines the two essential qualities of any data-gathering procedure: reliability and validity. Understanding these qualities allows researchers to critically evaluate the strength of research findings and discern weak from strong studies. The lecture provides detailed explanations, examples, and practical applications of different types of reliability and validity.
ποΈ Topics Covered
The lecture covers the concept of reliability in data collection, including the reliability coefficient and various types such as interrater, intrarater, test-retest, alternate-form, and internal consistency methods. It discusses factors that affect reliability like subjectivity, test length, and item quality, before moving to the Standard Error of Measurement. The second half addresses validity as a global construct, examining trait accuracy and utility, along with criterion-related, content coverage, and face appearance facets.
π Lecture Summary
Reliability
Reliability concerns the consistency of data results. A reliable method gives the same results regardless of who takes the measurement or when it is administered. The most common indicator for reporting reliability is the correlation coefficient, a number that quantifies the degree to which two variables relate to one another. When used for reliability, these are called reliability coefficients, which range between 0.00 and +1.00. A coefficient of 0.00 means no reliability (inconsistent results), while 1.00 indicates perfect reliability (identical results every time). Seldom do coefficients occur at these extremes. The rule of thumb is the higher the better, but what constitutes "adequate" depends on the measurement procedure: observation techniques using judges are satisfactory with coefficients from 0.80 upward, achievement and aptitude tests should have reliabilities in the 0.90s, and interest inventories or attitude scales tend to be lower. Generally, reliabilities falling below 0.60 are considered low regardless of procedure type (Nitko, 2001).
π Definition β Reliability coefficient: A number between 0.00 and +1.00 that represents the consistency of an observation or measurement procedure.
π Definition β Correlation coefficient: A number that quantifies the degree to which two variables relate to one another.
Types of Reliability Coefficients
Different types of reliability coefficients reveal different kinds of consistency, and different measurement procedures require different kinds of consistency.
Interrater or interobserver reliability is required whenever different observers are used to observe or rate participants' behavior. Researchers determine reliability by computing a correlation coefficient or calculating a percentage of agreement. The Bejarano et al. (1997) study used two independent raters and reported interrater reliabilities of 0.98, 0.86, and 0.96, revealing high agreement among raters.
Intrarater reliability addresses whether the same observer/rater gives consistent results if given the opportunity to observe/rate participants on more than one occasion. High agreement is expected within the same person if the attribute is stable and the observer understood the task. Goh's (2002) study on listening comprehension techniques computed an interrater reliability coefficient of 0.76 and an intrarater reliability coefficient of 0.88, showing that she agreed with herself more than with her colleague.
Testβretest reliability measures the stability of the same instrument over time. The instrument is given at least twice, and a correlation coefficient is computed on the scores. This only works if the trait being measured remains stable between administrations. Camiciottoli (2001) used a 22-item questionnaire, gave 20 participants the same questionnaire 6 weeks later, and found a reliability coefficient of 0.89, considered fairly high.
Alternate-form reliability is used when a test has several different forms. Different forms are given to the same individuals with several days or more between administrations, and results are correlated. This tests both stability over time and whether items in different forms represent the same general attribute.
Researchers cannot assume that borrowing items from commercially produced standardized tests inherits the same reliability. Test items often behave differently when put into other configurations, so subtests from larger instruments should be reevaluated for reliability before use.
Internal Consistency Reliability
The last three methods of estimating reliability concern internal consistency β whether all items in an instrument measure the same general attribute. This is important because item responses are normally added to make a total score; if items measure different traits, a total score would not make sense.
Split-half (odd/even) reliability divides the test items in half. Responses on each half are summed and compared. The odd/even method (comparing odd-numbered items with even-numbered items) is preferred because it is not influenced by qualitative changes in items like difficulty or fatigue that often occur in different sections.
Factors Affecting Reliability
Several factors affect reliability. Subjectivity β the more a procedure is vulnerable to perceptual bias, lack of awareness, or fatigue, the lower the reliability. Test length β instruments with fewer items automatically produce smaller reliability coefficients due to mathematical limitations. The Spearman-Brown prophecy formula projects what the reliability estimate would be if the test had more items. Garcia and Asencion (2001) used this procedure, reporting an interrater reliability of 0.98 for a text reconstruction test and a split-half reliability with Spearman-Brown adjustment of 0.73 for a 10-item listening test. Conversely, instruments that are too long cause fatigue, producing inconsistent responses and lowering reliability. Item quality β ambiguous test items, items with more than one correct answer, or trick items produce inconsistent results and lower reliability. Scarcella and Zimmerman (1998) dropped 10 items from their Test of Academic Lexicon because they lowered the Cronbach alpha coefficient.
π Definition β Standard Error of Measurement (SEM): An estimate of how much error there is in a measurement procedure, calculated from the reliability coefficient. Error is any variation in instrument results due to factors other than what is being measured.
π Formula: Perfect reliability (r = 1.00) means no error β all differences between scores are true differences. No reliability (r = 0.00) means no difference between scores can be interpreted as true difference on the trait being measured.
π Example: If a language proficiency test has no reliability, even though scores differ across individuals, the researcher cannot conclude that someone who scored higher has higher proficiency β all differences would be attributed to error from unknown sources.
Validity
Validity refers to the ability of an instrument or observational procedure to accurately capture data needed to answer a research question. Since the early 1990s, previous notions of validity have been subsumed under construct validity (Bachman, 1990; Messick, 1989), with different types now represented as different facets under this global title.
Trait accuracy addresses how accurately the procedure measures the trait (construct) under investigation. Accuracy depends on the definition of the construct being measured. If language proficiency is defined as grammar and vocabulary knowledge plus reading and listening comprehension, the approach must measure all these components. If other researchers define it as oral and writing proficiency, they must assess speaking and writing ability. The degree to which a procedure is valid for trait accuracy is determined by how well the procedure corresponds to the definition of the trait. These definitions should appear in the introduction or methodology section. Gardner et al. (1997) defined language anxiety as "communication apprehension, test anxiety, and fear of negative evaluation" based on the Foreign Language Classroom Anxiety Scale.
Utility is concerned with whether measurement/observational procedures are used for the right purpose. If a procedure is not used for what it was originally intended, the researcher must provide additional evidence of validity for their specific purpose.
π Definition β Construct validity: The overarching concept that subsumes all types of validity, referring to the degree to which a procedure measures the theoretical construct it claims to measure.
Facets of Validity
Three additional facets qualify the main facets of trait accuracy and utility.
Criterion related means the procedure is validated by being compared to some external criterion. It is divided into two types: capacity to succeed (a person having the aptitude to succeed in some endeavor) and current characteristics. Capacity to succeed is typically used for prediction purposes. Predictive utility is determined by correlating measurements from the procedures with measurements on the criterion being predicted. The Modern Language Aptitude Test (Carroll & Sapon, 1959) was developed to predict whether people have aptitude for learning languages. Steinman and Smith (2001) presented evidence that this test is valid for making predictions and has become used as an external criterion for validating other tests.
β Key Takeaways
The two most important qualities of any data-gathering procedure are reliability and validity, and the confidence we place in research findings is directly proportional to how reliable and valid these procedures are. Reliability concerns consistency and is measured using correlation coefficients ranging from 0.00 to +1.00, with different types (interrater, intrarater, test-retest, alternate-form, and internal consistency) revealing different kinds of consistency for different measurement situations. Validity has been reconceptualized as a global concept of construct validity, with trait accuracy and utility as main facets, and criterion-related, content coverage, and face appearance as qualifying facets. Researchers must always report reliability and validity evidence for their instruments, and cannot assume that borrowing items from validated tests inherits the same psychometric properties. The Standard Error of Measurement demonstrates that without reliability, no differences between participant scores can be interpreted as true differences on the trait being measured.
π§ Quick Revision Questions
- What is the difference between interrater reliability and intrarater reliability, and what does each measure?
- Why can a researcher not assume that borrowing items from a commercially produced standardized test will result in the same reliability?
- What is the Spearman-Brown prophecy formula, and when is it typically used?
- According to the modern conceptualization of validity, what are the two main facets of construct validity, and how do they differ?
- Why would using TOEFL results to measure the effects of a 2-week training program be considered invalid?
π Lecture 18 β Understanding Research Results
π Overview: This lecture explains how research resultsβboth verbal (qualitative) and numerical (quantitative)βare presented, analyzed, and evaluated. It emphasizes that numerical data are not inherently more scientific than verbal data and details the processes for ensuring credibility in qualitative research findings. Understanding these evaluation strategies is crucial for critically consuming research.
ποΈ Topics Covered
The lecture begins by addressing the false assumption that numerical data is more scientific than verbal data. It outlines the common procedure of data selection and reduction in all research reports. The main body distinguishes between the presentation and analysis of verbal data versus numerical data, delving into the qualitative "data analysis spiral" and potential biases like holistic fallacy, elite bias, and going native. It then presents a comprehensive list of verification procedures, which are categorized into tactics for evaluating data quality and for evaluating explanations and conclusions. Key strategies for data quality include representativeness, prolonged engagement, clarifying researcher bias, researcher effects, and weighting the evidence. For evaluating explanations, the lecture covers spurious relationships, if-then tests, rival explanations, replicating findings, informant feedback, rich/thick description, and external audits.
π Lecture Summary
Presentation and Analysis of Verbal Data
In qualitative research, the presentation and analysis of data are deeply intertwined. Unlike numerical data, where summaries are presented before analysis, the analysis of verbal data begins at the start of data collection and continues throughout the study. The researcher becomes the primary "measurement device," engaging in what is called the data analysis spiralβa process of engaging, reflecting, noting, organizing, coding, reducing data, and looking for themes. However, published reports rarely describe this process clearly, which raises credibility concerns.
Several analytical biases can threaten the validity of conclusions. These include the holistic fallacy, where a researcher sees patterns that are not actually present; elite bias, which occurs when too much weight is given to articulate or well-informed informants, making the data unrepresentative; and going native, where the researcher becomes so close to the respondents that they adopt their perceptions.
π Definition β Data analysis spiral: A model of qualitative data analysis where the researcher repeatedly engages with the data, reflects, makes notes, organizes, codes, reduces, looks for relationships and themes, checks credibility, and eventually draws conclusions.
Evaluating the Quality of Data
Verbal data cannot be taken at face value. Several strategies help ensure the data is dependable for analysis.
-
Representativeness: This concerns the internal validity or credibility of the data, not just the sample. The veracity of information can be influenced by the choice of respondents or events. A researcher must provide evidence that generalizations are based on appropriate events and an adequate number of observations, so conclusions are not based on the "luck of the draw."
-
Prolonged Engagement and Persistent Observation: The researcher must spend enough time interacting with respondents or events to gather accurate data. While necessary, too much time on site can lead to other researcher effects.
-
Clarifying Researcher Bias: Every researcher has biases. Since analysis begins during data collection, disclosing these biases helps the consumer understand why data is gathered and interpreted in a certain way, and how the researcher arrived at their conclusions.
-
Researcher Effects: There is a reciprocal relationship between the researcher and the participants/events. The researcher's presence can influence the behavior being observed. Consumers should look for evidence that the researcher tried to control for or was at least aware of their own effect on the data and the analysis process. Strategies to avoid this include spending more time on site to become unnoticed and using unobtrusive methods.
-
Weighting the Evidence: Some data are stronger than others. To evaluate data strength, consider: the proximity of informants to the data (closer is stronger), the extent of firsthand observation of actual behavior, and the effort made to check for biases during data gathering.
Evaluating Explanations and Conclusion
This section outlines tactics to evaluate the validity of the researcher's conclusions.
- Spurious Relationships: Not all relationships are direct. A relationship between two variables may be caused by a third, unmeasured variable. A researcher must provide a convincing argument that no other variables are producing an observed relationship.
- If-Then Tests: This is the "workhorse of qualitative data analysis." It involves forming a conditional hypothesis (if X is true, then Y should occur) and testing it on a new sample to see if the predicted consequence appears.
- Rival Explanations: A powerful way to add weight to a conclusion is to formulate and test a plausible competing explanation. The explanation that best fits the data is the most plausible. However, consumers should beware of straw man argumentsβexplanations that are not truly plausible and are easy to refute. The last "standing" explanation may not be the best one.
- Replicating Findings: When the same findings occur across different samples and conditions, confidence in the conclusions increases. A single study from one setting has limited practical use.
- Informant Feedback (Member Checks): Getting feedback from the informants on the study's conclusions can check the plausibility of the researcher's patterns. However, care is needed as respondents may agree to please the researcher or may not understand a technical report.
- Rich, Thick Description: A detailed description of the participants, context, and data-gathering/analysis stages. This allows the reader to decide if the conclusions are transferable to other similar situations.
- External Audits: Hiring an outside expert to evaluate the study is a powerful, though seldom used, method to increase credibility.
π Definition β Spurious relationship: A misleading relationship between two variables that appears to be directly related but is actually caused by a third, related variable.
π‘ Why this matters: These evaluation tactics are essential for the consumer of qualitative research. Without them, it is impossible to trust the conclusions drawn from verbal data, which are inherently subjective.
β Key Takeaways
The critical point from this lecture is that verbal and numerical data are not different in scientific value; numbers are just a transformation of verbal constructs. The core of qualitative research analysis is a spiral process where the researcher is the main instrument, and this process must be clearly reported for the findings to be credible. To achieve credibility, a researcher must actively evaluate the quality of their data using tactics like ensuring representativeness, clarifying their own bias, and managing researcher effects. Finally, the validity of explanations must be rigorously tested using if-then tests, rival explanations, and replication, with results presented through rich, thick descriptions to allow readers to assess transferability.
π§ Quick Revision Questions
- What is the difference between the "holistic fallacy" and "elite bias" as threats to validity in qualitative analysis?
- Explain the "data analysis spiral" as described by Creswell.
- What is a "spurious relationship," and what must a researcher do to rule one out in their conclusions?
- Why is "clarifying researcher bias" important for evaluating the quality of data?
- What is the purpose of rich, thick description in a qualitative research report, and how does it help the consumer?
π Lecture 19 β Understanding Research Results-II
π Overview: This lecture explains how researchers analyze numerical data, covering both descriptive and inferential statistics. It focuses on understanding the shape of data distributions, measures of average and variation, and the crucial concepts of null hypothesis testing and statistical significance. The goal is to equip research consumers to evaluate whether appropriate statistical procedures were used and whether results were interpreted correctly.
ποΈ Topics Covered
The lecture begins with an introduction to presentation and analysis of numerical data, explaining why numerical data is preferred and the distinction between population, sample, statistics, and parameters. It then covers understanding statistics of data, focusing on shape of distribution, measures of average (mean, median, mode), and measures of variation (standard deviation, semi-interquartile range, range). The lecture continues with understanding inferential statistical procedures, explaining the null hypothesis and its role in research. Finally, it covers statistical procedures, dividing them into nonparametric and parametric categories, and explores relationships between variables using chi-square and Spearman rank correlation.
π Lecture Summary
Presentation and Analysis of Numerical Data
Researchers often convert ideas into numerical data because it is easier to work with than verbal data. Statistical procedures quickly identify patterns and relationships in large data sets and can estimate whether findings exceed random chance. The purpose is to introduce common procedures for analyzing numerical data and the basic concepts underpinning them. Understanding formulas is not necessary for research consumers; instead, knowing whether an appropriate procedure was used and whether results were interpreted correctly is essential. Population is the entire number of people to which the researcher wants to generalize conclusions. Sample is a subgroup of that total number. Statistics are quantities gathered on a sample, estimates of what would be found using the whole population. Parameters are quantities gathered directly from the entire population, representing true values that exactly describe the population. Because researchers almost always deal with samples, they use statistics and must make inferences about the population, with statistics helping to understand the chance of making a mistake when inferring from sample to population. Statistics divide into two main categories: descriptive statistics describe a set of data, and inferential statistics use those descriptions to determine whether researchers can generalize findings to a target population.
π Definition β Population: The entire number of people to which the researcher wants to generalize conclusions. π Definition β Sample: A subgroup of the total population used for data collection. π Definition β Statistics: Quantities (or numbers) gathered on a sample that estimate what would be found if the whole population were used. π Definition β Parameters: Quantities gathered directly from the entire population; the true values that exactly describe the population. π Definition β Descriptive Statistics: Statistics that describe a set of data. π Definition β Inferential Statistics: Statistics that produce answers and determine whether researchers can generalize findings to a target population.
Understanding Statistics of Data
There are three basic concerns when using descriptive statistics to describe numerical data: the shape of the distribution, measures of average, and measures of variation. The first concern is whether the data are symmetrically distributed and approximate a normal curve. If a distribution is severely skewed (lopsided), rectangular (no curve), or multimodal (more than one cluster of data), certain statistics should not be used. Based on shape, the second concern is which statistic to use to describe average. There are three: mean, median, and mode. The mean is computed by adding all scores and dividing by the total number of scores. The median is the middle point dividing the number of people in half. The mode is the most frequent score. The mean is the most common estimate of average, but if the distribution does not approximate normal, other indicators are more accurate. The third concern, also affected by shape, is what statistic to use to indicate variance (how much data vary). There are three measures of variation: standard deviation, semi-interquartile range, and range. The standard deviation (SD) is the average deviation of scores from the mean. The semi-interquartile range estimates where the middle 50% of scores are located. The range is the distance from lowest to highest scores. The standard deviation is most commonly used but is only appropriate if the distribution does not vary too much from normalcy.
π Definition β Normal curve: A symmetrical distribution of data that is bell-shaped. π Definition β Mean: Computed by adding up all scores and dividing by the total number of scores. π Definition β Median: The middle point in the distribution of data that divides the number of people in half. π Definition β Mode: The most frequent score in a data set. π Definition β Standard deviation (SD): The average deviation of scores from the mean. π Definition β Semi-interquartile range: A measure of variation that estimates where the middle 50% of scores are located in the data distribution. π Definition β Range: The distance from the lowest to the highest scores in the distribution.
Understanding Inferential Statistical Procedures
Researchers attempt to infer their findings to a population based on a sample, and inferential statistics play a crucial role in this process. The main goal is to describe common inferential statistical procedures, explain why they are used, and provide examples. However, understanding null hypothesis and statistical significance is more important than remembering procedure names.
The Null Hypothesis
Statistical significance directly relates to testing the null hypothesis. Inferential statistics answer two types of questions: are there relationships between variables, or are there differences between groups? The null hypothesis states that there is either no relationship or no difference between groups. Regardless of whether a research hypothesis exists, the null hypothesis is always tested. In exploratory studies with no stated hypotheses, behind every relationship studied is a null hypothesis stating no relationship. For every study exploring differences between groups, a null hypothesis states no real difference. Few published studies explicitly state null hypotheses today, but they are always lurking. An example from Tsang (1996) stated five null hypotheses, one being "There is no significant main effect for nature of program...as a factor in writing performance." The phrase "no significant main effect" means no differences between different programs regarding effect on writing performance. Stating hypotheses in null form is more accurate because inferential statistics test the null hypothesis, not positively stated hypotheses.
π Definition β Null hypothesis: The hypothesis that states there is either no relationship between variables or no difference between groups. π Definition β Statistical significance: A concept directly related to testing the null hypothesis, indicating whether results are greater than random chance.
Statistical Procedures
Inferential statistics divide into two general categories: nonparametric and parametric. Nonparametric statistics are used for analyzing data in the form of frequencies, ranked data, and data that do not approximate a normal distribution. Parametric statistics are used for data that do not stray too far from a normal distribution and typically involve means and standard deviations. Scores on tests and surveys usually fit these criteria. The objectives are to find relationships between variables or differences between groups, with both nonparametric and parametric procedures available under each objective.
π Definition β Nonparametric statistics: Statistical procedures used for frequencies, ranked data, and data not approximating a normal distribution. π Definition β Parametric statistics: Statistical procedures used for data approximating a normal distribution, typically involving means and standard deviations.
Relationships between Variables: Nonparametric Procedures
Two frequently seen procedures are chi-square and Spearman rank correlation. The Pearson chi-square (pronounced Ky-square, symbol ΟΒ²) is the procedure of preference when dealing with data in the form of frequencies or percentages. In its simplest form, chi-square compares observed frequency (or percentages) of different levels of a variable with what would be expected if no relationship existed (the null hypothesis). For example, if a researcher asks whether there is a relationship between gender and success in learning English as a foreign language, they would compare a random sample of males and females on success rate. The null hypothesis would be: no relationship between gender and success rate, therefore no difference between males and females who pass or fail. If true, expected frequency should be 20/20 for each sex. In fictional data, the researcher found 27 females versus 17 males passed, and 13 females versus 23 males failed. Although frequencies appear to differ, the researcher must do a chi-square analysis rather than rely on "eyeball" analysis to determine if they differ from what would be expected under the null hypothesis.
π Definition β Pearson chi-square (ΟΒ²): A nonparametric statistical procedure that compares observed frequency of different levels of a variable with expected frequency under the null hypothesis. π Example: Researcher asks "Is there a relationship between gender and success in learning English?" Null hypothesis: no relationship, so expected frequency is 20/20 for each sex. Observed: 27 females and 17 males passed; 13 females and 23 males failed. The chi-square analysis determines if the observed frequencies differ significantly from expected frequencies under the null hypothesis.
β Key Takeaways
Students must remember that statistics divide into descriptive (describing data) and inferential (generalizing findings) categories. The three concerns when describing numerical data are shape of distribution (normal vs. skewed/rectangular/multimodal), measures of average (mean, median, mode), and measures of variation (standard deviation, semi-interquartile range, range). The null hypothesis always states no relationship or no difference and is what inferential statistics actually test, even when not explicitly stated. Parametric statistics require normal distributions while nonparametric statistics handle frequencies, ranks, and non-normal data. Finally, chi-square is the appropriate procedure for analyzing relationships between variables when data are in frequency form.
π§ Quick Revision Questions
- What is the difference between statistics and parameters?
- Name the three measures of average and briefly define each.
- Under what conditions should the mean and standard deviation NOT be used to describe data?
- What does the null hypothesis always state about relationships or differences?
- When would a researcher use chi-square rather than a parametric procedure?