π Lecture 19 β The Use and Abuse of Psychological Testing
π Overview: This lecture explores the critical issues surrounding psychological testing, including ethical use, privacy, confidentiality, and discrimination. It then introduces the core concept of personality and its assessment through objective and projective methods, detailing major tests like the MMPI-2, MCMI, and NEO-PI-R, along with the Type A/Type B behavior pattern.
ποΈ Topics Covered
This lecture first examines the societal role of psychological testing, focusing on protection, privacy, confidentiality, and discrimination, including the concept of test bias. It then defines personality and introduces personality assessment, contrasting objective and projective tests. The lecture details methods of test construction for objective tests (content validation, empirical criterion keying, factor analysis, construct validity) and provides an in-depth look at the MMPI/MMPI-2, the MCMI, and the NEO-PI-R. Finally, it discusses the Type A and Type B behavior patterns.
π Lecture Summary
The Use and Abuse of Psychological Testing
This section addresses the widespread use and potential for abuse in psychological testing. It highlights that testing is a large enterprise affecting many life areas, necessitating careful scrutiny. Protection is provided by ethical standards (APA, 1992), state licensing, and restrictions on test purchases, though these are not always successful. The Question of Privacy asserts an individual's right to a full explanation of a test's purpose and the use of results, with informed consent being mandatory. The Question of Confidentiality is challenged by data banks and legal decisions like the Tarasoff decision, which mandates breaking confidentiality if a patient poses a danger. Clinicians must explain who will have access to results and obtain consent for any subsequent release. The Question of Discrimination addresses how tests can disadvantage minorities due to lack of exposure, inadequate motivation, or racially unfair test content. Test Bias is defined as a validity issue, occurring when a test predicts more accurately for one group than another. Differences in mean scores do not automatically indicate bias. A test is biased if its predictive validity varies across groups. Bias can be overcome by using different prediction equations for different groups. Computer-Based Assessment offers advantages like cutting costs and standardizing procedures but raises issues of reliability, validity, client acceptance, and misuse by poorly trained individuals.
π Definition β Test Bias: A test is biased to the extent that it predicts more accurately for one group than for another group. π Concept: Mean score differences β Test bias. Test bias = differential validity (different predictive accuracy across groups). π Example: An author developed a hostility inventory where men scored higher than women. The predictive validity (correlation between hostility scores and verbal fights) was similar for both men and women, so the test was not biased for predicting verbal fights. However, the correlation between hostility scores and physical fights might be stronger for men, making the test biased for predicting physical aggression in women if using the male-based prediction equation.
Personality
This section defines personality as the observation that people display consistency in behavior, thoughts, and feelings across different situations and over time. These stable, enduring characteristics are called traits, which are consistent ways of perceiving, experiencing, and interacting with the world. These consistencies may stem from genetic factors, learned patterns, or both.
π Definition β Personality: The observation that people display a certain degree of consistency and structure in the ways that they experience and interact with the world.
Assessment of Personality
This section defines personality assessment as a scientific endeavor to determine important individual differences in personality, develop accurate measures, and explore their consequential meanings. Personality tests are broadly grouped into objective personality tests and projective personality tests.
Objective Personality Tests
This section explains that objective personality tests rely on structured, standardized, self-report measurement devices with straightforward test stimuli and unambiguous instructions (e.g., true/false or Likert scales).
Some Advantages: They are economical for large groups, scoring and administration are simple and objective, a simple score on a single dimension is often possible, and they offer apparent objectivity and reliability.
Some Disadvantages: Items are often behavioral and can be endorsed for different reasons. Inventories may provide a single score for a mixture of behaviors, cognitions, and needs. The transparent meaning of questions can facilitate faking. The forced-choice approach prevents elaboration. Limited reading ability can lead to misinterpretation.
Methods of the Test Construction for Objective Tests
This section describes four strategies for constructing self-report inventories.
Content Validation: The most straightforward approach where clinicians decide what to assess and ask the patient directly. More sophisticated methods involve carefully defining the variable, consulting experts, using judges, and conducting psychometric analyses. Potential problems include varying item interpretation, inaccurate self-report, dishonesty, and disagreement among experts.
Empirical Criterion Keying: This approach makes no assumptions about truthfulness. Items are selected based on their empirical ability to differentiate diagnostic groups. The key assumption is that members of a particular diagnostic group will respond similarly to items.
Factor Analysis: This approach examines inter-correlations among items to reduce them to basic elements or core traits. The exploratory factor analytic approach is atheoretical, while the confirmatory factor analytic approach is theory-driven. Its strength is the empirical demonstration that items measuring a dimension are highly related. A limitation is that it does not prove the items are actually measuring the variable of interest.
Construct Validity Approach: This comprehensive approach combines aspects of the other methods. Scales are developed to measure specific concepts from a given theory. Validation is achieved by demonstrating that the scale measures the theoretical construct. This is the most desirable but most labor-intensive approach, as establishing construct validity is a never-ending process.
π Definition β Empirical Criterion Keying: An approach to test construction where items are selected based on their empirical ability to separate members of a particular diagnostic group from normal controls, with no assumptions about whether the patient is telling the truth. π Definition β Factor Analysis: A statistical approach to test construction that examines inter-correlations among items to reduce them to basic elements or core dimensions of personality.
The MMPI and the MMPI-2
This section introduces the MMPI (Minnesota Multiphasic Personality Inventory), the most widely used and researched objective personality test, developed in 1937 and later updated to the MMPI-2. It consists of over 500 statements to which the subject responds "true," "false," or "cannot say."
Clinical Scales: The MMPI provides scores on 10 standard clinical scales, including Hypochondria (Hs), Depression (D), Hysteria (Hy), Psychopathic Deviance (Pd), Masculinity-Femininity (Mf), Paranoia (Pa), Psychasthenia (Pt), Schizophrenia (Sc), Hypomania (Ma), and Social Introversion (Si).
Validity Scales: To detect response sets like malingering ("faking bad") or carelessness, the MMPI-2 has four validity scales: The ? (Cannot Say) Scale (number of unanswered items), the F (Infrequency) Scale (items seldom answered in the scored direction by normals, suggesting deviance), the L (Lie) Scale (items that put the respondent in an unrealistically positive light), and the K (Defensiveness) Scale (items suggesting defensiveness).
Interpretation: Accurate interpretation requires experience with the test and an understanding of the patient's background. Profile Analysis examines patterns of scores across scales (e.g., elevations on Hs, D, and Hy suggest somatic complaints and depression). Interpretation through Content uses content scales to identify fears, health concerns, cynicism, etc.
π Definition β Malingering: Intentionally faking or exaggerating symptoms for a secondary gain.
Millon Clinical Multi-Axial Inventory (MCMI)
The MCMI is a 175-item true-false inventory developed by Theodore Millon. It allows for scoring on scales representing personality disorders from the DSM (e.g., Avoidant, Dependent, Histrionic, Narcissistic) and scales for less enduring clinical syndromes. It was revised as the MCMI-II in 1987 to be compatible with the revised DSM (IV).
The Revised NEO-Personality Inventory
The NEO-PI-R is a self-report measure of personality based on the five-factor model (FFM) . The five domains (or factors) are Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness. Each domain has six facets (e.g., Neuroticism includes Anxiety, Hostility, Depression, Self-Consciousness, Impulsiveness, Vulnerability). It consists of 240 items rated on a five-point scale.
π Definition β Five-Factor Model (FFM): A model of personality that organizes personality traits into five broad domains: Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness.
Type A - Type B Behavior
Two cardiologists, Meyer Friedman and Ray Rosenman, developed the concept of the Type A behavior pattern, which they hypothesized is a major risk factor for coronary artery disease. The most important aspects are excesses of time urgency and competitive hostility. Type B persons display the opposite qualities: they are relaxed, less aggressive, and unhurried. Interestingly, some data indicate that Type A persons may be less successful than Type B persons.
π Definition β Type A Behavior Pattern: A behavior pattern characterized by time urgency, competitive hostility, and a vigorous striving for achievement, hypothesized as a risk factor for coronary artery disease. π Definition β Type B Behavior Pattern: A behavior pattern characterized by being relaxed, less aggressive, unhurried, and less apt to strive vigorously to achieve a goal than Type A persons.
β Key Takeaways
Ethical testing requires ensuring privacy, confidentiality, and informed consent, while being vigilant about discrimination and test bias. Test bias is a validity issue based on differential predictive accuracy across groups, not just mean score differences. Objective personality tests, like the MMPI-2, MCMI, and NEO-PI-R, are structured, self-report instruments constructed using methods like content validation, empirical criterion keying, factor analysis, or construct validity. The MMPI-2 is the most widely used test, with clinical and validity scales interpreted through profile analysis and content. The Type A behavior pattern, characterized by time urgency and competitive hostility, is considered a risk factor for coronary artery disease.
π§ Quick Revision Questions
- What is the difference between test bias and simple mean score differences between groups on a psychological test?
- Name the four validity scales of the MMPI-2 and the purpose of each.
- What are the five domains of the Five-Factor Model (NEO-PI-R)?
- Compare and contrast the empirical criterion keying method and the factor analytic method of test construction.
- According to the Tarasoff decision, when is it permissible to break confidentiality in a therapeutic relationship?
π Lecture 20 β The Projective Personality Tests
π Overview: This lecture introduces projective personality tests, a radically different approach to personality assessment compared to objective tests. It explains the projective hypothesis, the nature and characteristics of projective techniques, and critically examines major projective tests including the Rorschach, Thematic Apperception Test (TAT), Sentence Completion techniques, and Word-Association techniques, along with their psychometric challenges, advantages, and disadvantages.
ποΈ Topics Covered
The lecture covers the projective hypothesis and the distinguishing characteristics of projective tests, including their unstructured stimuli and indirect methods. It then examines measurement and standardization issues including reliability and validity challenges. Major projective tests are discussed in detail: the Rorschach Inkblot Test (description, administration, scoring, interpretation, reliability, validity, and utility), the Thematic Apperception Test (description, administration, scoring, reliability, and validity), Sentence Completion Techniques (Rotter Incomplete Sentences Blank), and Word-Association Technique. The lecture concludes with advantages and disadvantages of projective techniques.
π Lecture Summary
PROJECTIVE TESTS
Projective tests represent the second broad approach to personality assessment, radically different from objective tests. The format, items, administration, and scoring are all distinct from objective tests. While objective tests require responses to explicit verbal questions or statements, projective tests ask for responses to ambiguous and unstructured stimuli. A major distinguishing feature is the use of a relatively unstructured task that permits an almost unlimited number of responses.
The development and use of virtually all projective personality tests are based on the projective hypothesis. According to this hypothesis, projective techniques were essentially psychological X-rays. When faced with ambiguous stimuli, respondents will project aspects of their personalities onto the stimuli in an effort to make sense of them. The examiner then can work backward from the person's responses to gain insight into personality dispositions.
π Definition β Projective hypothesis: When faced with ambiguous stimuli, respondents will project aspects of their personalities onto the stimuli in an effort to make sense of them.
THE NATURE OF PROJECTIVE TESTS
Projective techniques have the following distinguishing characteristics:
-
Imposition of structure: In response to an unstructured or ambiguous stimulus, examinees are forced to impose their own structure and, in so doing, reveal something of themselves (such as needs, wishes, or conflicts).
-
Unstructured stimulus material: The stimulus material is unstructured. However, this is a tenuous criterion. For example, if 70% of examinees perceive Card V on the Rorschach as a bat, the stimulus is hardly unstructured. Whether a test is projective depends on the kinds of responses encouraged and how those responses are used. The instructions are the important element. If a patient is asked to classify people in TAT cards as men or women, there is great structure; if asked what the people are saying, the task becomes quite ambiguous.
-
Indirect method: To some degree, examinees are not aware of the purposes of the test; the purposes are disguised. Although patients may know the test has something to do with adjustment-maladjustment, they are not usually aware in detail of the significance of their responses. There is no attempt to ask patients directly about their needs or troubles; the route is indirect, hoping this indirectness will make it more difficult for patients to censor the data they provide.
-
Freedom of response: While questionnaire methods may allow only for "yes" or "no," projective techniques permit a nearly infinite range of responses.
-
Response interpretation deals with more variables: Since the range of possible responses is so broad, the clinician can make interpretations along multiple dimensions (needs, adjustment, diagnostic category, ego defenses, and so on). Many objective tests, in contrast, provide but a single score or scores on a fixed number of dimensions or scales.
MEASUREMENT AND STANDARDIZATION
Objective tests lend themselves to an actuarial interpretive approach where norms, reliability, and even validity seem easier to manage. Projective tests, by their nature, seem to resist psychometric evaluation. Some clinicians reject even the suggestion that a test such as the Rorschach should be subjected to psychometrics, seeing this as an assault upon their intuitive art.
Standardization: There are many reasons to standardize projective techniques, including facilitating communication and checking against biases and interpretive zeal of clinicians. Proponents of projective tests usually act as if they have implicit norms. Research problems with projective tests can be formidable. Dissenters argue that interpretations from projectives cannot be standardized because every person is unique, and normative descriptions will be misleading. They argue that standardized interpretive approaches would destroy the holistic nature of projective tests - interpretation is an art.
Reliability: Determination of reliability is not simple. It is too much to expect an individual to produce exactly the same TAT story on two different occasions. One can bypass test responses and deal with the reliability of personality interpretations made by clinicians, but this may confound the reliability of the test with the reliability of the judge. Test-retest reliability may be affected by psychological changes in the individual, particularly with patient populations. Alternate forms reliability is difficult because it is hard to decide if alternate forms for TAT cards or inkblots are equivalent. Even split-half reliability is difficult due to the challenge of demonstrating equivalence of the two halves of each test.
Validity: Because projective tests have been used for many purposes, general questions like "Is the TAT valid?" or "Is the Rorschach a good personality test?" are not useful. Questions must be specific: Does the TAT predict aggression in situation A? Does a Rorschach score correlate with clinical indices of anxiety?
π‘ Why this matters: The psychometric challenges of projective tests highlight the ongoing tension between clinical intuition and empirical science in psychological assessment.
THE RORSCHACH
The prototypic example of projective personality tests is the Rorschach Inkblot Test, developed by Swiss psychiatrist Herman Rorschach in 1921. The Rorschach Inkblot Test has the dubious distinction of being simultaneously the most cherished and the most reviled of all psychological assessment instruments.
Description: The Rorschach consists of ten cards on which are printed inkblots that are symmetrical from right to left. Five cards are black and white (with shades of gray), and the other five are colored.
Administration: The clinician hands the patient the first card and says, "Tell me what you seeβwhat it might be for you. There are no right or wrong answers. Just tell me what it looks like to you." All subsequent cards are administered in order. The clinician takes down verbatim everything the patient says. Some clinicians record the length of time it takes the patient to make the first response and the total time spent on each card. The clinician also notes the position of the card (right side up, upside down, or sideways). Following this phase, the clinician moves to the Inquiry, where the patient is reminded of all previous responses and asked what prompted each response. The patient is also asked to indicate the exact location of the various responses.
Scoring: Scoring converts important aspects of each response into a symbol system related to location areas, determinants, content areas, and popularity.
-
Location: Scored in terms of which portion of the blot was used (e.g., the whole blot, a common detail, an unusual detail, or an area of white space). Attention to the whole blot with accurate form perception reflects good organizational ability and high intelligence. Over attention to detail is common in obsessive and paranoid subjects.
-
Determinants: Reflect the features of the blot that made it look the way the patient thought it looked (e.g., form, shading, colors, movement of humans or animals, inanimate movements). Overemphasis on form suggests rigidity and constriction of the personality. Color responses relate to emotional reactions to the environment and control of affect.
Interpretation: The Rorschach test is particularly useful as an aid in diagnosis. The subject's thinking and association patterns are brought clearly into focus because the ambiguity of the stimulus provides relatively few cues about what are conventional, standard, or normal responses. Proper interpretation requires a great deal of experience. There is high reliability among experienced clinicians. In proper hands, the test is extremely useful, especially in eliciting psychodynamic formulations, defense mechanisms, and subtle disorders of thinking.
Reliability and Validity: Research-oriented clinical psychologists have questioned Rorschach reliability for years. At the most basic level, one should be confident that Rorschach responses can be scored reliably across raters (interscorer reliability). If the same responses cannot be scored similarly by different raters using the same scoring system, the instrument would have little utility. Unfortunately, the extent to which Rorschach scoring systems meet acceptable standards for this basic form of reliability remains contentious. We must also evaluate the consistency of scores across time or test conditions and the reliability of interpretations of scores.
Regarding validity, the Rorschach is not equally valid for all purposes. The problem is not determining whether the Rorschach is valid, but differentiating the conditions under which it is useful from those under which it is not.
Utility of Rorschach: The debate over Rorschach utility in clinical assessment continues. Rorschach is useful when the focus is on unconscious functioning and problem-solving styles. However, critics remain skeptical of the clinical utility of Rorschach scores or their incremental validity.
Rorschach Inkblot "Method": Weiner (1994) has argued that the Rorschach is best conceptualized as a method of data collection, not a test. The Rorschach does not test anything - a test measures whether something is present and in what quantity, but with the Rorschach we do not measure whether people have a personality or how much personality they have. Data generated from the Rorschach method can be interpreted from a variety of theoretical positions. These data suggest how the respondent typically solves problems or makes decisions (cognitive structuring processes) and the meanings assigned to these perceptions (associational processes). This "integrationist" view means the method provides data relevant to both the structure and dynamics of personality. Viewing the Rorschach as a method allows one to fully use all aspects of the data generated, resulting in a more thorough diagnostic evaluation.
THEMATIC APPERCEPTION TEST
The Thematic Apperception Test (TAT) was introduced by Morgan and Murray in 1935. It purports to reveal patients' basic personality characteristics through interpretation of their imaginative productions in response to a series of pictures. Although designed to reveal central conflicts, attitudes, goals, and repressed material, it actually produces material that is a collage of these plus situational influences, cultural stereotypes, and trivia.
Most clinicians use the TAT as a method of inferring psychological needs (achievement, affiliation, dependency, power, sex) and disclosing how the patient interacts with the environment. TAT is used to infer the content of personality and the mode of social interactions. With a TAT, clinicians make specific judgments, such as "This patient is hostile toward authority figures, yet seeks their affection and approval." The TAT is less likely to assess the degree of maladjustment than to reveal the locus of problems, the nature of needs, or the quality of interpersonal relationships.
Description: There are 31 TAT cards (one is blank); most depict people in various situations, but a few contain only objects. Some are useful for boys and men, some for girls and women, and some for both genders. Murray suggested selecting 20 of the 31 cards for a given examinee. The TAT does not appear as ambiguous as the Rorschach, though the figures may clearly be people, it is not always clear their gender, who they are, what they are doing, or what they are thinking.
Administration: Clinicians typically select between 6 and 12 cards for administration. Instructions go something like: "Now, I want you to make up a story about each of these pictures. Tell me who the people are, what they are doing, what they are thinking or feeling, what led up to the scene, and how it will turn out." The patient's productions are transcribed by the clinician.
Scoring: Several scoring systems have been developed. One approach rates each story on scoring categories including unconscious structure and drives, relationship to others, significant conflicts, defenses used, and ego strength. Another scoring system assesses object relations - the respondent's mental representations of people. Four dimensions of object relations are assessed: complexity of representation of people, affect-tone of relationships, capacity for emotional investment in relationships and moral standards, and understanding.
Reliability and Validity: It is very difficult to evaluate TAT reliability and validity in any formal sense due to variations in instructions, administration methods, number of cards used, and scoring schemes. The same methodological issues arise when studying reliability. Personality changes may obscure test-retest conclusions, and there may be uncertainty about equivalent forms. Theme reliability can be investigated, but since word-for-word similarity cannot be expected, one is studying the reliability of judges' interpretations. When there is explicit, theoretically derived scoring instructions, interjudge agreement can reach acceptable proportions. Broad, global interpretations can present problems.
Methods to establish validity include:
- Comparison of TAT interpretations with case data or therapist evaluations
- Matching techniques and analyses of protocols with no additional knowledge about the patient
- Comparisons between clinical diagnoses derived from TAT and psychiatrists' judgments
- Establishment of validity of general principles of interpretation (e.g., tendency to identify with the hero, probability that unusual themes are more significant)
The typical clinical use of TAT suggests it remains a subjective instrument. Adequate interpretation depends on knowledge of the patient's background. The clinician must attend to frequency of thematic elements, unusualness of stories, plot development, misrecognitions, word choice, identifications with plot characters, nature of heroes/heroines, their needs and goals, environmental presses, and general emotional ambiance.
SENTENCE COMPLETION TECHNIQUES
A very durable and serviceable technique is the sentence completion method. The most widely used and best-known version is the Rotter Incomplete Sentences Blank (ISB). The ISB consists of 40 sentence stems - for example, "I like . . .", "What annoys me? ...", "I wish . . .", and "Most girls _____". Each completion can be scored along a 7-point scale to provide a general index of adjustment-maladjustment. The ISB has great versatility, and scoring schemes for various variables have been developed.
The ISB has several advantages:
- Scoring is objective and reliable, due to extensive scoring examples in the manual
- Can be used easily and economically
- Appears to be a good screening device
- Allows considerable freedom of response while being scored objectively
The ISB falls between the two extremes of the objective-projective dimensions. It represents a fairly direct approach that does not require the degree of training necessary for the Rorschach. Some clinicians may be disturbed by the ISB's relative lack of disguise. The ISB does not typically provide information that could not be gleaned from a reasonably extensive interview. In many ways, the ISB provides a cognitive and behavioral picture of the patient rather than a "deep, psychodynamic" picture.
WORD-ASSOCIATION TECHNIQUE
The word association technique was devised by Carl Gustav Jung, who presented stimulus words to patients and had them respond with the first word that came to mind. After initial administration, some clinicians repeat the list, asking the patient to respond with the same words used previously; discrepancies may reveal associational difficulties. Complex indicators include long reaction times, blocking, difficulties in making responses, unusual responses, repetition of stimulus words, misunderstanding of the word, clang association (responding with a word that rhymes), preservation of earlier responses, and unusual mannerisms accompanying responses. Because it is easily quantified, the test has continued as a research instrument, though its popularity has diminished.
π Definition β Clang association: Responding to a stimulus word with a word that rhymes, indicating associational difficulties.
ADVANTAGES OF PROJECTIVE TECHNIQUES
The first advantage is the amount, richness, and accuracy of information collected. Projective techniques are frequently used in individual interviews or focus group discussions (breaking the ice). They help open discussions around socially sensitive issues where the client may be embarrassed or feel a lack of knowledge. These techniques are useful in encouraging freedom and spontaneity of expression in subjects who may hesitate to express their opinion directly for fear of disapproval or when they find topics threatening.
DISADVANTAGES
The primary disadvantage is the complexity of data and the corresponding skills required of the researcher. Interpreters need to be very trained and skilled. Projective techniques are expensive to administer because highly trained staff is needed.
β Key Takeaways
The projective hypothesis is the foundational concept: when faced with ambiguous stimuli, individuals project aspects of their personality. The five distinguishing characteristics of projective tests (imposition of structure, unstructured stimuli, indirect method, freedom of response, and multiple interpretation variables) fundamentally differentiate them from objective tests. The Rorschach Inkblot Test and Thematic Apperception Test are the most prominent projective tests, but both face significant psychometric challenges regarding reliability (interscorer, test-retest, alternate forms) and validity, with ongoing debate about their clinical utility. The Rorschach has been reconceptualized as a "method" rather than a "test," providing data on cognitive structuring and associational processes. Among projective techniques, the Rotter Incomplete Sentences Blank offers a practical compromise with objective scoring while allowing freedom of response, making it useful as a screening device.
π§ Quick Revision Questions
- What is the projective hypothesis, and how does it serve as the theoretical foundation for projective personality tests?
- List and explain the five distinguishing characteristics of projective techniques.
- Describe the four scoring categories (location, determinants, content, popularity) used in the Rorschach Inkblot Test, and explain what each reveals about personality.
- What are the major psychometric challenges (standardization, reliability, and validity) faced by projective tests like the Rorschach and TAT?
- How does the Rotter Incomplete Sentences Blank differ from the Rorschach and TAT in terms of objectivity, disguise, training requirements, and the type of information it provides?
π Lecture 21 β The Observational Assessment and Its Types
π Overview: This lecture explores observation as a primary behavioral assessment technique in clinical psychology. It examines the crucial distinction between naturalistic and controlled observation, detailing specific methods for home, school, and hospital settings. The lecture emphasizes the importance of ensuring reliability and validity in observational data and introduces self-monitoring as a practical alternative for clients in natural environments.
ποΈ Topics Covered
The lecture begins by defining observation as a visual method of gathering information on activities and behaviors. It then introduces observation methods as a primary behavioral assessment technique, followed by detailed examinations of naturalistic observation (including home, school, and hospital settings) and controlled observation (including studies of honesty/deceit, response to stress, and parent-adolescent conflict). The lecture also covers controlled performance techniques (including behavioral avoidance), self-monitoring, and concludes with a comprehensive discussion of variables affecting the reliability and validity of observations, along with suggestions for improvement.
π Lecture Summary
OBSERVATION METHODS
Observation is a visual method of gathering information on activities: of what happens, what your object of study does or how it behaves. To assess and understand behavior, behavioral assessment employs observation as a primary technique. A clinician can try to understand a phobic's fear of heights, a student's avoidance of evaluation settings, or anyone's tendency to overeat. Many clinicians argue that unless people are directly observed in their natural environments, true understanding will be incomplete.
Practically speaking, it is difficult and expensive to maintain trained observers. It is easier to accomplish with children or those with cognitive limitations, and easier to make observations in a sheltered or institutional setting. In some cases, it is possible to use observers who are characteristically part of the person's environment (such as spouse, parent, teacher, friend, or nurse), or even have the client do some self-observation. Ethical concerns require that people are not observed without their knowledge.
For all these reasons, naturalistic observation has never been used in clinical practice as much as it might be. Observation is still more prominent in research than in clinical practice.
NATURALISTIC OBSERVATION
Naturalistic observation is hardly a new idea. McReynolds (1975) traced its roots to the ancient civilizations of Greece and China. Observation, like testing, is useful only when steps are taken to ensure its reliability and validity.
Example of Naturalistic Observation
Many forms of naturalistic observation have been used for specific settings including classrooms, playgrounds, general and psychiatric hospitals, home environments, institutions for those with mental retardation, and therapy sessions.
Home Observation One of the best known systems for home observation is the Behavioral Coding System (BCS) developed by Patterson (1977) and his colleagues. This observational system was designed for use in the homes of pre-delinquent boys who exhibit problems in the areas of aggressiveness and noncompliance. Trained observers spend one or two hours in the homes, observing and recording family interactions, usually immediately before or during dinner. Each family member is observed for two 5-minute periods, with observations made of behaviors in 28 categories every 6 seconds.
π Definition β Behavioral Coding System (BCS): An observational system designed to record family interactions in the homes of pre-delinquent boys, focusing on 28 categories of behavior observed every 6 seconds during 5-minute periods per family member.
π Example: In a study by Patterson and Forgatch (1995), observational data (the sum of multiple categories of aversive behavior such as yelling, humiliating, destructiveness) were coded from home interactions between 67 children and their families. Children's aversive behavior scores at treatment termination significantly predicted future arrests over a two-year follow-up period, while no teacher, mother, or father rating significantly predicted arrests.
π‘ Why this matters: This study demonstrates the superior predictive value of naturalistic observation over traditional ratings by parents or teachers.
School Observation An example of a behavioral observation system used in school settings is Achenbach's (1994) Direct Observation Form (DOF) of the Child Behavior Checklist. The DOF is used to assess problem behaviors that may be observed in school classrooms. It consists of 96 problem items plus an open-ended item. Assessors rate each item according to its frequency, duration, and intensity within a 10-minute observation period. It is recommended that three to six 10-minute observation periods be completed so scores can be averaged across occasions.
Hospital Observation An example of a hospital observation device is the Time Sample Behavioral Checklist (TSBC) developed by Gordon Paul and his associates. It is a time-sample behavioral checklist used with chronic psychiatric patients. Observers make a single 2-second observation of the patient once every waking hour, allowing a daily behavioral profile to be constructed on each patient.
π Definition β Time-sample: Observations are made at regular intervals for a given patient, such as a single 2-second observation once every waking hour.
π Example: Using the TSBC, Menditto et al. (1996) documented how a combination of a relatively new antipsychotic medication (clozapine) and a structured social learning program helped significantly decrease the frequency of inappropriate behaviors and aggressive acts over a 6-month period in a sample of chronically mentally ill patients on an inpatient unit.
CONTROLLED OBSERVATION
Naturalistic observation provides an unfiltered picture of behavior, but specific behaviors of interest may not occur naturally very often. Much time and resources can be wasted, and naturalistic settings often put clinicians at the mercy of events. As a way of handling these problems, clinicians sometimes use controlled observation. These are really situational tests that put individuals in situations more or less similar to those of real life. This is a kind of work-sample approach in which the behavioral test situation and the criterion behavior to be predicted are quite similar.
STUDIES IN HONEST AND DECEIT Early studies by Hartshorne and May (1928-1930) attempted to measure personality by directly sampling behavior. For example, to assess children's honesty, they confronted them with situations where cheating was possible and then observed their responses. Data suggested that children's deceitful behavior was highly situation-specific and should not be construed as reflecting a generalized trait.
RESPONSE TO STRESS During World War II, the U.S. Office of Strategic Services (OSS) used assigned tasks to assess reaction to stress. A sample OSS task involved constructing a large cube out of pegs, poles, and blocks in 10 minutes. Two helpers (actually stooges) interfered, were passive, made impractical suggestions, and ridiculed the candidate. No candidate was ever successful in assembling the cube.
PARENT ADOLESCENT CONFLICT Prinz and Kent (1978) developed the Interaction Behavior Code (IBC) system to assess parent-adolescent conflict. Several raters review and rate audio taped discussions of families attempting to resolve a problem about which they disagree. Summary scores are calculated by averaging scores across raters for negative behaviors and positive behaviors.
CONTROLLED PERFORMANCE TECHNIQUES
Controlled situations allow one to observe behavior under conditions that offer potential for control and standardization. For example, A. A. Lazarus (1961) assessed claustrophobic behavior by placing a patient in a closed room that was made progressively smaller by moving a screen. Bandura (1969) used films to expose people to a graduated series of anxiety-provoking stimuli.
BEHAVIORAL AVOIDANCE The test of avoidance behavior consisted of a series of 29 performance tasks requiring increasingly more threatening interactions with a red-tailed boa constrictor. Subjects were instructed to approach a glass cage containing the snake, touch and hold the snake with gloved and then bare hands, let it loose and return it, hold it within 12 cm of their faces, and finally tolerate the snake crawling in their laps.
π Example: Subjects who could not enter the room received a score of 0. Those who did enter performed the various tasks in the graded series. The avoidance score was the number of snake-interaction tasks the subject performed successfully.
SELF MONITORING
Self-monitoring involves individuals observing and recording their own behaviors, thoughts, and emotions. Clients are asked to maintain behavioral logs or diaries over some predetermined time period. Such a log can provide a running record of the frequency, intensity, and duration of certain target behaviors, along with the stimulus conditions and consequences that followed.
Such data are especially useful in telling how often the behavior occurs, providing an index of change as a result of therapy, focusing the client's attention on undesirable behavior, and helping clients realize connections between environmental stimuli, consequences, and behavior itself.
Problems with self-monitoring include clients being inaccurate, purposely distorting observations, or resisting the procedure. Self-monitoring is usually effective as a change agent only in conjunction with a larger program of therapeutic intervention.
π Definition β Dysfunctional Thought Record (DTR): A record completed by the client that provides the client and therapist with a record of the client's automatic thoughts related to dysphoria or depression. The client completes it when noticing a change in mood, specifying the situation, automatic thought(s), and associated emotions. The final two columns serve as a therapeutic intervention.
VARIABLES AFFECTING RELIABILITY OF OBSERVATIONS
Clinicians must be assured that observational data are reliable β that different observers will produce basically the same ratings and scores.
COMPLEXITY OF TARGET BEHAVIOR The more complex the behavior to be observed, the greater the opportunity for unreliability. Behavioral assessment typically focuses on less complex, lower-level behaviors. Observations about what a person eats for breakfast are likely to be more reliable than those centering on interpersonal behavior.
TRAINING OBSERVERS There is no substitute for the careful and systematic training of observers. Observers must be briefed extensively on definitions and specific behaviors representing each construct.
π Definition β Observer drift: A phenomenon in which observers who work closely together subtly, without awareness, begin to drift away from other observers in their ratings. Over time, they shift their definitions of target behaviors.
To guard against observer drift, regularly scheduled reliability checks (by an independent rater) should be conducted and feedback provided to raters.
VARIABLES AFFECTING VALIDITY OF OBSERVATIONS
CONTENT VALIDITY A behavioral observation schema should include the behaviors deemed important for the research or clinical purposes. The BCS circumvented problems by organizing several categories of noxious behaviors and submitting them for ratings by mothers to confirm clinical judgments.
CONCURRENT VALIDITY This asks whether obtained observational ratings correspond to what others (teachers, spouse, friends) are observing in the same time frame. For example, do observational ratings of children's aggression on the playground made by trained observers agree with ratings made by the children's peers?
CONSTRUCT VALIDITY Observational systems are usually derived from some theoretical framework. For example, the BCS was derived from a social learning framework. The construct validity of the BCS could be demonstrated by showing that children's aggressive behavior declines from baseline after clinical treatment, defined as rearranging social contingencies.
MECHANICS OF RATINGS
A unit of analysis must be specified β the length of time observations will be made, along with the type and number of responses to be considered. For example, every physical movement might be recorded for 1 minute every 4 minutes during a 20-minute recess period.
Ratings can take the form of recording behaviors along a dimension of intensity (how strong was the behavior), duration (how long did it last), or frequency count (how many times in a designated period). A scoring procedure must be developed, ranging from check marks on paper to counters, stopwatches, timers, and laptop computers.
REACTIVITY
Reactivity refers to the fact that patients or study participants sometimes react to being observed by changing the way they behave. The talkative person suddenly becomes quiet. Reactivity can severely hamper the validity of observations because it makes observed behavior unrepresentative of what normally occurs. The real danger is that the observer may not recognize its presence.
SUGGESTIONS FOR IMPROVING RELIABILITY AND VALIDITY OF OBSERVATIONS
- Decide on target behaviors that are both relevant and comprehensive.
- Work from an explicit theoretical framework that will help define the behaviors of interest.
- Employ trained observers.
- Make sure that the observational format is strictly specified.
- Be aware of such potential sources of error as bias and fluctuations in concentration.
- Consider the possibility of reactivity.
- Give careful consideration to how representative the observations really are.
β Key Takeaways
A student must remember that observation is a primary behavioral assessment technique with two main types: naturalistic and controlled. Naturalistic observation provides unfiltered behavior in real settings (home, school, hospital) but is expensive and difficult, while controlled observation (situational tests, work-sample approaches) allows standardization and control of specific behaviors. Self-monitoring is a practical, low-cost alternative where clients record their own behaviors using logs or the Dysfunctional Thought Record. Reliability is threatened by observer drift and complex target behaviors, while validity is threatened by reactivity. To ensure quality observations, one must use trained observers, specify formats, and consider how representative the observations are.
π§ Quick Revision Questions
- What is the main difference between naturalistic observation and controlled observation in clinical assessment?
- What did Patterson and Forgatch (1995) find regarding the predictive value of naturalistic observation compared to parent/teacher ratings?
- What is observer drift and how can it be prevented?
- What is reactivity and how does it threaten the validity of observations?
- How does the Dysfunctional Thought Record (DTR) serve as both an assessment tool and a therapeutic intervention?