π Lecture 39 β Basics of Measurements
π Overview: This lecture introduces the fundamental concepts and challenges of measurement in software engineering. It explores why measuring software attributes is difficult compared to physical sciences, presents the representational theory of measurement as a formal framework, and covers empirical relations, mapping rules, measurement scales, and the distinction between direct and indirect measurement. Understanding these basics is crucial for establishing meaningful and useful software metrics.
ποΈ Topics Covered
This lecture begins by posing four fundamental questions about measuring software attributes, such as complexity and quality. It then introduces the representational theory of measurement from the physical sciences. The concept of empirical relations is explained, using comparisons like height and software product surveys. The lecture details the rules of mapping from the empirical to the mathematical world, the representation condition of measurement, and discusses measurement and models, including direct and indirect measures like programmer productivity and defect density. It concludes with an introduction to measurement scales (nominal, ordinal, interval, ratio, absolute) and prediction systems.
π Lecture Summary
Basics of Measurements
Ordinarily, when we measure physical attributes like length or temperature, we apply sophisticated tools and principles developed over centuries. However, we lack a comparably deep understanding of software attributes and the associated measurement tools. This leads to four difficult questions: 1) How much must we know about an attribute (e.g., complexity) before measuring it? 2) How do we know if we have measured the intended attribute? 3) What meaningful statements can we make from measurements? 4) What meaningful operations can we perform on measures (e.g., averaging productivity)? To answer these, we must establish the basics of a formal theory of measurement, starting from the physical sciences.
The Representational Theory of Measurement
In any measurement activity, rules provide consistency and a basis for interpreting data. Measurement theory codifies these rules, similar to how mathematicians define axioms for geometry. The representational theory of measurement seeks to formalize our intuition about the world: the data we obtain as measures should represent attributes of the entities we observe, and manipulation of the data should preserve the relationships we observe among the entities. Our intuition is the starting point for all measurement, leading to the concept of empirical relations.
π Definition β Empirical Relation: A relation (like "taller than") for which there is a reasonable consensus about which pairs or groups of entities are in the relation.
Empirical Relations
We understand the world by comparing things. For instance, we can observe that Frankie is taller than Wonderman. This "taller than" is a binary empirical relation defined on pairs of people. Empirical relations can also be unary ("is tall") or ternary (comparing groups of three). These relations are mappings from the empirical world (entities and attributes) to the formal mathematical world. If we agree Frankie is taller than Wonderman, then any measure of height must assign a higher number to Frankie.
π Example: A survey of 100 users compared the functionality and user-friendliness of four word processors (A, B, C, D). The table for Functionality shows the percentage who preferred the row's program. For example, 80% of respondents preferred A over B. The table for User-Friendliness shows a different pattern, where preferences are much closer to 50%. The lecture asks if we can make judgments about greater or lesser functionality/user-friendliness from this data. Analyzing such results often clarifies the attribute itself. π‘ Why this matters: This illustrates how we can begin to understand attributes using relatively unsophisticated relationships without special tools, setting the stage for formal measurement.
Measurement is a mapping from the empirical world to the formal, relational world. A measure is the number or symbol assigned to an entity by this mapping to characterize an attribute. Rating formats include: Likert scale (Strongly Agree to Strongly Disagree), Forced ranking, Verbal frequency scale, Ordinal scale, Comparative scale, and Numerical scale.
Rules of Mapping
When mapping an attribute to a mathematical system (the range), we have choices over numbers (real, integers) or non-numeric symbols. A measure must specify the domain, range, and the rule for performing the mapping. Terminology can be imprecise (e.g., "Felix is 11" implies a mapping of age in whole years since birth). In software, measuring "lines of code" requires a clear definition of what constitutes a line.
Representation Condition of Measurement
By definition, each relation in the empirical relational system corresponds via the measurement to an element in a number system. The representation condition asserts that the mapping M must map entities into numbers and empirical relations into numerical relations such that the empirical relations preserve and are preserved by the numerical relations.
π Formula: A is taller than B if and only if M(A) > M(B). The empirical relation is preserved under M as a numerical relation. π Example: For Frankie, Wonderman, and Peter, the representation condition would require that if "Frankie is taller than Wonderman" is true, then M(Frankie) > M(Wonderman). This must hold for all such empirical relations.
Measurement and Models
Direct measurement measures an attribute of an entity itself (e.g., length of source code, duration of testing process). Indirect measurement is derived from other direct measures and is often useful for making interactions visible.
π Definition β Indirect Measure: A measure derived from two or more direct measures.
π Common Indirect Measures in SWE:
- Programmer productivity = LOC produced / person month effort
- Module defect density = number of defects / module size
- Defect detection efficiency = number of defects detected / total number of defects
- Requirements stability = number of initial requirements / total number of requirements
- Test effectiveness ratio = number of items covered / total number of items
- System spoilage = effort spent on fixing faults / total project effort
Where no previous measurement has been performed, direct measurement is natural. However, simple models can lead to more accurate indirect measurement later (e.g., temperature measured indirectly via mercury column length). Measurement for assessment evaluates existing entities (past events). Measurement for prediction uses models to predict attributes of entities that do not yet exist (e.g., early reliability assurance for a future system). A prediction system consists of a mathematical model with procedures for determining unknown parameters and interpreting results.
Measurement Scales and Scale Types
Not all measurement mappings are the same. The differences restrict the kind of analysis we can do. A measurement scale helps us understand which analyses are appropriate.
The lecture mentions five types of measurement scales:
- Nominal
- Ordinal
- Interval
- Ratio
- Absolute
β Key Takeaways
This lecture establishes the foundational theory for software measurement. The key takeaway is that measuring software is fundamentally difficult due to our lack of deep understanding of software attributes, unlike physical sciences. The representational theory of measurement provides a formal framework where measurement is a mapping from empirical relations (like "taller than") to numerical relations, and the representation condition ensures that this mapping preserves the observed relationships. Students must understand the distinction between direct and indirect measures, the definition of key indirect measures like programmer productivity and defect density, and the crucial role of measurement scales in determining appropriate data analysis. Finally, the lecture distinguishes between measurement for assessing existing entities and measurement for predicting the attributes of future ones, setting the stage for model-based prediction.
π§ Quick Revision Questions
- What are the four fundamental questions about software measurement that highlight its difficulty, and how does the representational theory of measurement aim to answer them?
- Define an empirical relation and explain the representation condition of measurement using the concept of "taller than" as an example.
- Distinguish between a direct measure and an indirect measure, and provide two examples of common indirect measures in software engineering with their formulas.
- What is the purpose of a measurement scale, and why is it important to know which scale type (e.g., nominal, ordinal, ratio) a measure belongs to?
- Explain the difference between measurement for assessment and measurement for prediction. What components form a prediction system?
π Lecture 40 β Measurement Scales
π Overview: This lecture covers the fundamental types of measurement scales used in software metrics and introduces a goal-based framework for software measurement. It explains how different scales (nominal, ordinal, interval, ratio, absolute) determine what mathematical operations are permissible, and provides a structured approach to measuring processes, products, and resources in software development.
ποΈ Topics Covered
The lecture begins with a detailed classification of five measurement scales: nominal, ordinal, interval, ratio, and absolute, including their characteristics, admissible transformations, and examples from software engineering. It then explores the concept of meaningfulness in measurement, emphasizing that statements must be invariant under scale transformations. Finally, it presents a comprehensive goal-based framework for software measurement that classifies entities into processes, products, and resources, distinguishes between internal and external attributes, and introduces the Goal-Question-Metric (GQM) paradigm for selecting and implementing metrics.
π Lecture Summary
1 β Nominal Scale
- We define classes or categories, and then place each entity in a particular class or category, based on the value of the attribute. This is nominal measurement. Classes are not ordered. The empirical relation system consists only of different classes; there is no notion of ordering among the classes. Any distinct numbering or symbolic representation of the classes is an acceptable measure, but there is no notion of magnitude associated with the numbers or symbols.
π Definition β Nominal Scale: A measurement scale where entities are placed into unordered classes or categories based on an attribute value. Any mapping that preserves distinct classes is acceptable.
π Example: Capturing the location of software faults (specification, code, design)
2 β Ordinal Scale
- The ordinal scale is often useful to augment the nominal scale with information about an ordering of the classes or categories. The ordering leads to analysis not possible with nominal measures. The empirical relation system consists of classes that are ordered with respect to the attribute. Any mapping that preserves the ordering (that is, any monotonic function) is acceptable. The numbers represent ranking only, so addition, subtraction, and other arithmetic operations have no meaning.
π Definition β Ordinal Scale: A measurement scale where classes are ordered with respect to the attribute, but only ranking is meaningful; arithmetic operations have no meaning.
π Example: Complexity of software modules is described as: Trivial, Simple, Moderate, Complex, Incomprehensible
3 β Interval Scale
- The interval scale carries more information still, making it more powerful than nominal or ordinal scales. This scale captures information about the size of the intervals that separate the classes, so that we can in some sense understand the size of the jump from one class to another. An interval scale preserves order, as with an ordinal scale. An interval scale preserves differences but not ratios. That is, we know the difference between any two of the ordered classes in the range of the mapping, but computing the ratio of two classes in the range does not make sense. Addition and subtraction are acceptable on the interval scale, but not multiplication and division.
π Definition β Interval Scale: A measurement scale that preserves order and differences between classes, but not ratios. Addition and subtraction are meaningful, but multiplication and division are not.
π Example: Complexity of software modules is described as: Trivial (0), Simple (2), Moderate (4), Complex (6), Incomprehensible (8)
4 β Ratio Scale
- We would like to be able to say that one liquid is twice as hot as another, or that one project took twice as long as another. This need for ratios gives rise to ratio scale. It is a measurement mapping that preserves ordering, the size of intervals between entities, and ratios between entities. There is a zero element, representing total lack of the attribute. The measurement mapping must start at zero and increase at equal intervals, known as units. All arithmetic can be meaningfully applied to the classes in the range of the mapping.
π Definition β Ratio Scale: A measurement scale with a true zero point that preserves ordering, interval sizes, and ratios. All arithmetic operations are meaningful.
π Example: The length of software code is measurable on a ratio scale
5 β Absolute Scale
- There is only one way in which the measurement can be made, so M and M' must be equal. The absolute scale is the most restrictive of all. The measurement for an absolute scale is made simply by counting the number of elements in the entity set. The attribute always takes the form "number of occurrences of x in the entity". There is only one possible measurement mapping, namely the actual count. All arithmetic analysis of the resulting count is meaningful.
π Definition β Absolute Scale: The most restrictive measurement scale where measurement is made by counting the number of elements; only one possible mapping exists (the actual count).
π Example: LOC is an absolute-scale measure of the attribute "number of lines of code" of a program. However, LOC is not an absolute-scale measure of length, because there are different ways to measure length (such as thousands of LOC, number of characters, and number of bytes).
Scales of Measurement
| Scale Type | Admissible transformations | Examples |
|---|---|---|
| Nominal | 1-1 mapping from M to M' | Labeling, classifying entities |
| Ordinal | Monotonic increasing function from M to M' | Preference, hardness, air quality, intelligence tests (raw scores) |
| Interval | M' = aM + b (a>0) | Relative time, temperature (Fahrenheit, Celsius), intelligence tests (standardized scores) |
| Ratio | M' = aM (a>0) | Time interval, length, temperature (Kelvin) |
| Absolute | M' = M | Counting entities |
π‘ Why this matters: The permissible transformations for each scale define what statistical operations are valid. Using inappropriate operations (like calculating the mean of ordinal data) can lead to meaningless conclusions.
Meaningfulness in Measurement
- Can we deduce meaningful statements about the entities being measured? A statement involving measurement is meaningful if its truth value is invariant of transformations of allowable scales.
π Example of meaningful statement: "The number of errors discovered during the integration testing of program X was at least 100" π Example of meaningless statement: "The cost of fixing each error in program X is at least 100" (meaningless without reference to a particular scale) π Example of meaningful statement: "A semantic error takes twice as long to fix as a syntactic error" π Example of meaningless statement: "A semantic error is twice as complex as a syntactic error" (not meaningful without clarifying complexity)
A Goal-based Framework for Software Measurement
- Now, we'll talk about a conceptual framework for the diverse software-measurement activities that contribute to an organization's software practices. These practices may include not only usual development and maintenance activities, but also any experiments and case studies. This framework is based on three principles:
- Classifying the entities to be examined
- Determining relevant measurement tools
- Identifying the level of maturity that your organization has reached
Classifying Software Measures
-
The first obligation of any software measurement activity is identifying the entities and attributes we wish to measure. In software, there are three such classes:
- Processes: Collections of software-related activities
- Products: Any artifacts, deliverables, or documents that result from a process activity
- Resources: Entities required by a process activity
-
A process is usually associated with some timescale, where activities are ordered or related in a way that depends on time. Resources and products are associated with the process. Each process activity has resources and products that it uses, as well as products that are produced.
-
Before discussing these entities in detail, we distinguish between internal and external attributes:
- Internal attributes: Can be measured purely in terms of the product, process, or resource itself, separate from its behavior
- External attributes: Can be measured only with respect to how the product, process, or resource relates to its environment
π Examples of Internal and External Attributes:
- Internal: software size, complexity, dependencies among modules (determined without execution)
- External: number of failures experienced by a user, difficulty in navigating screens, database search time (measured when code is executed)
Processes (3.1.1)
- We often have questions about our software development activities and processes that measurement can help us answer. Only a limited number of internal process attributes can be measured directly: the duration of the process, the effort associated with the process, and the number of incidents of a specified type arising during the process.
π Example: Measuring the effectiveness of a requirements review process by measuring the number of requirements errors found during specification, compared to those found during integration testing.
π Formula: Average Cost of each Defect Detected = Cost / Number of Errors Found β This is an indirect measure
π Example: Effectiveness of Software Inspections β measure the average amount of effort expended per thousand lines of code reviewed
π Example: Effectiveness of Software Testing Process β track the number of errors identified in each sub-process (unit testing, integration testing, system testing, acceptance testing) along with the duration and cost
- External process attributes include controllability, observability, and stability.
| Entities | Internal Attributes | External Attributes |
|---|---|---|
| Constructing specification | Time, effort, number of requirements changes | Quality, cost, stability |
| Detailed design | Time, effort, number of specification faults found | Cost, cost-effectiveness |
| Testing | Time, effort, number of coding faults found | Cost, cost-effectiveness, stability |
Products (3.1.2)
-
Products are not restricted to items management commits to deliver to the customer. Any artifact or document produced during the software life cycle can be measured and assessed (prototypes, test harnesses, documents).
-
External Product Attributes: Depend on both product behavior and environment.
- π Example: Measuring reliability of code must consider the machine on which the program runs and the mode of operational usage
- π Example: The understandability of a document depends on the reader's experience
- External attributes include: maintainability, usability, integrity, efficiency, testability, reusability, portability, interoperability
-
Internal Product Attributes: Sometimes easy to measure (size by pages or words). More difficult attributes include code complexity, structuredness, module coupling, and cohesiveness.
π Example: A software system design document consisting of module design documents
- Average module size can be derived by calculating the size of each module
- Size of one Module = No of DFD Bubbles
- Average Module Size = 1/n β(i=1 to n) No of DFD Bubblesα΅’
π‘ Why this matters: Internal attributes (like faults found during inspection) provide insight into external attributes (like reliability based on failures), even though they measure different aspects.
- Quality is multi-dimensional, similar to GNP (Gross National Product) of a country. The GNP is a weighted combination of key goods and services; the weights reflect priorities. Similarly, we measure quality by measuring and controlling a number of internal (structural) product attributes.
| Entities | Internal Attributes | External Attributes |
|---|---|---|
| Specifications | Size, reuse, modularity, redundancy, functionality, syntactic correctness | Comprehensibility, maintainability |
| Designs | Size, reuse, modularity, coupling, cohesiveness, functionality | Quality, complexity, maintainability |
| Code | Size, reuse, modularity, coupling, functionality, algorithmic complexity, control-flow structuredness | Reliability, usability, maintainability |
| Test data | Size, coverage, level | Quality |
Resources (3.1.3)
-
Resources include any input for software production: Personnel (individuals and teams), Materials (including office supplies), Tools (both software and hardware), Methods.
-
We measure resources to determine their magnitude, cost, and quality. These measures help us understand and control the process.
π Formula: Productivity = Amount of Output / Effort Input β A resource measure combining a process measure (input) with a product measure (output)
π Example: Staff attributes that can be measured include experience, age, or intelligence of a developer, which may affect design or code quality
| Entities | Internal Attributes | External Attributes |
|---|---|---|
| Personnel | Age, price | Productivity, experience, intelligence |
| Teams | Size, communication level, structuredness | Productivity, quality |
| Software | Price, size | Usability, reliability |
| Hardware | Price, speed, memory, size | Reliability |
| Offices | Size, temperature, light | Comfort, quality |
Determining What to Measure?
-
Utility is in the eye of the beholder. The Goal-Question-Metric (GQM) approach to process and metrics has proven to be a particularly effective approach to selecting and implementing metrics.
-
Steps of GQM:
- Express the overall goals of your organization
- Generate questions whose answers you must know to determine if goals are being met
- Analyze each question in terms of what measurements you need to answer each question
-
Once you have a list of possible measures, assess your organization to see whether it is capable of providing useful information. The more mature your process, the more that is visible and therefore measurable.
π Example: If developers cannot tell when design ends and coding begins, measuring the duration of the design activity will be impossible.
GQM Paradigm
- A measurement program can be more successful if it is designed with the goals of the project in mind. The GQM approach involves three steps:
- List the major goals of the development or maintenance project
- Derive from each goal the questions that must be answered to determine if the goals are being met
- Decide what must be measured in order to be able to answer the questions adequately
π Example Goal: Evaluate the effectiveness of using a coding standard
- Goal-derived questions: Who is using the standard? How does their productivity compare? How does quality compare?
- Metrics needed: Proportion of coders using the standard, experience profile, productivity (effort/size in LOC or FPs), number of errors found in code
π‘ Why this matters: By deriving measurements from goals, it becomes clear how to use the resulting data. Several metrics might be generated from a single goal, and one metric may support several goals.
β Key Takeaways
The five measurement scales (nominal, ordinal, interval, ratio, absolute) form a hierarchy where each successive scale adds more information and permits more mathematical operations, with absolute being the most restrictive. The meaningfulness of measurement statements depends on invariance under permissible scale transformations β a statement about software attributes is only meaningful if its truth remains unchanged when the measurement scale is transformed. Measurement entities in software are classified into processes (activities), products (artifacts), and resources (inputs), each with both internal attributes (measured within the entity) and external attributes (measured in relation to the environment). The Goal-Question-Metric (GQM) paradigm provides a structured approach to measurement by starting with organizational goals, deriving questions that assess goal achievement, and identifying specific metrics needed to answer those questions. Successful measurement programs require understanding organizational process maturity, as more mature processes have greater visibility and are therefore more measurable.
π§ Quick Revision Questions
- What are the five measurement scales, and what mathematical operations are permissible for each scale?
- According to the GQM paradigm, what are the three steps for establishing a measurement program?
- Distinguish between internal and external attributes, and provide one example of each for software products.
- What does it mean for a measurement statement to be "meaningful," and why is the concept of admissible transformations important?
- How would you classify software LOC (Lines of Code) as a measurement scale, and what attribute is it actually measuring according to the absolute scale definition?
π Lecture 41 β Goal-Question-Metric (GQM)
π Overview: This lecture introduces the Goal-Question-Metric (GQM) approach, a structured measurement framework for software process improvement. It explains how to define measurement goals, derive questions to assess those goals, and associate metrics to answer the questions quantitatively. The lecture emphasizes that GQM ensures measurement is purposeful, traceable, and linked to organizational objectives.
ποΈ Topics Covered
The GQM approach is based on the assumption that organizations must first specify goals, then trace those goals to data, and provide a framework for interpreting data. The resulting measurement model has three levels: Conceptual (Goal), Operational (Question), and Quantitative (Metric). The lecture covers templates for goal definition (purpose, perspective, environment), guidelines for defining product and process-related questions, and provides detailed examples including AT&Tβs inspection process and change request processing. It also explains how GQM complements the entity-attribute measurement framework.
π Lecture Summary
Goal Question Metric Approach
The Goal Question Metric (GQM) approach is based upon the assumption that for an organization to measure in a purposeful way: it must first specify the goals for itself and its projects, then trace those goals to the data that are intended to define those goals operationally, and finally provide a framework for interpreting the data with respect to the stated goals. The result of applying the GQM approach is the specification of a measurement system targeting a particular set of issues and a set of rules for the interpretation of the measurement data. The resulting measurement model has three levels: Conceptual, Operational, and Quantitative.
π‘ Why this matters: GQM ensures that every metric collected directly serves a defined business or project goal, preventing wasteful and meaningless data collection.
1 β Conceptual Level (Goal)
A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view, relative to a particular environment. Objects of measurement are:
- Products: Artifacts, deliverables, and documents produced during the system life cycle (e.g., specifications, designs, programs, test suites).
- Processes: Software-related activities normally associated with time (e.g., specifying, designing, testing, interviewing).
- Resources: Items used by processes to produce their outputs (e.g., personnel, hardware, software, office space).
2 β Operational Level (Question)
A set of questions is used to characterize the way the assessment/achievement of a specific goal is going to be performed based on some characterizing model. Questions try to characterize the object of measurement (product, process, resource) with respect to a selected quality issue and to determine its quality from the selected viewpoint.
3 β Quantitative Level (Metric)
A set of data is associated with every question in order to answer it in a quantitative way. The data can be:
- Objective: If they depend only on the object that is being measured and not on the viewpoint from which they are taken (e.g., number of versions of a document, staff hours spent on a task, size of a program).
- Subjective: If they depend on both the object that is being measured and the viewpoint from which they are taken (e.g., readability of a text, level of user satisfaction). A number of templates are available to aid in generating the goals, questions, and metrics.
π Definition β Objective Data: Data that depends only on the object being measured, not on the viewpoint. Example: staff hours spent on a task. π Definition β Subjective Data: Data that depends on both the object being measured and the viewpoint from which they are taken. Example: level of user satisfaction.
Template for Goal Definition
- Template for Purpose: To (characterize, evaluate, predict, motivate, etc.) the (process, product, model, metric, etc.) in order to (understand, assess, manage, engineer, learn, improve, etc.) it.
- Example: To evaluate the maintenance process in order to improve it.
- Template for Perspective: Examine the (cost, effectiveness, correctness, defects, changes, product measures, etc.) from the viewpoint of the (developer, manager, customer, etc.).
- Example: Examine the cost from the viewpoint of the manager.
- Template for Environment: The environment consists of the following: (process factors, people factors, problem factors, methods, tools, constraints, etc.).
- Example: The maintenance staff are poorly motivated programmers who have limited access to tools.
Goal Question Metric Approach (continue)
The GQM approach also provides separate guidelines for defining product-related and process-related questions. Steps involve defining the process or product, defining the relevant attributes, and obtaining feedback related to the attributes. What constitutes a goal or question may be vague, and several levels of refinement may be required. We can relate the GQM templates to the attribute framework. A goal or a question can be associated with at least one pair of entities and attributes. The use of the measure is determined by the goals and questions, so that assessment, prediction, and motivation are tightly linked to the data analysis and reporting. Thus, the GQM complements the entity-attribute measurement framework. The results of a GQM analysis are a collection of measurements related by goal tree and overall model. However, GQM does not address issues of measurement scale, objectivity, or feasibility. So the GQM measures should be used with care, remembering the overall goal of providing useful data.
Example of GQM
AT&T used GQM to help determine which metrics were appropriate for assessing their inspection process. Their goals, questions, and metrics are shown below:
AT&T Goals, Questions, and Metrics for Inspection Process β Plan
- Goal: Plan
- Question: How much does the inspection process cost?
- Metrics: 1. Average effort per KLOC, 2. Percentage of reinspections, 3. Average effort per KLOC
- Question: How much calendar time does the inspection process take?
- Metrics: 4. Total KLOC inspected
- Question: How much does the inspection process cost?
AT&T Goals, Questions, and Metrics for Inspection Process β Monitor and Control
- Goal: Monitor and Control
- Question: What is the quality of inspected software?
- Metrics: 1. Average faults detected per KLOC, 2. Average inspection rate, 3. Average preparation rate
- Question: To what degree did the staff conform to the procedures?
- Metrics: 4. Average inspection rate, 5. Average preparation rate
- Question: What is the status of the inspection process?
- Metrics: 6. Average lines of code inspected, 7. Percentage of reinspections, 8. Total KLOC inspected
- Question: What is the quality of inspected software?
AT&T Goals, Questions, and Metrics for Inspection Process β Improve
- Goal: Improve
- Question: How effective is the inspection process?
- Metrics: 1. Defect removal efficiency, 2. Average faults detected per KLOC, 3. Average inspection rate, 4. Average preparation rate, 5. Average lines of code inspected
- Question: What is the productivity of the inspection process?
- Metrics: 6. Average effort per fault detected, 7. Average inspection rate, 8. Average preparation rate, 9. Average lines of code inspected
- Question: How effective is the inspection process?
GQM-Goal
A GQM model is a hierarchical structure starting with a goal (specifying purpose of measurement, object to be measured, issue to be measured, and viewpoint from which the measure is taken). So a GQM goal has three coordinates and a purpose.
GQM-Question
The goal is refined into several questions that usually break down the issue into its major components. In general, we will ask at least three groups of questions:
- Group 1: How can we characterize the object (product, process, or resource) with respect to the overall goal of the specific GQM model?
- Group 2: How can we characterize the attributes of the object that are relevant with respect to the issue of the specific GQM model?
- Group 3: How do we evaluate the characteristics of the object that are relevant with respect to the issue of the specific GQM model?
GQM-Metric
Once the questions have been developed, we proceed to associating the question with appropriate metrics. The factors we consider are many, among them:
- Amount and quality of existing data: maximize the use of existing data sources if available and reliable.
- Maturity of the objects of measurement: apply objective measures to more mature objects, and use more subjective evaluations for informal or unstable objects.
- Learning process: GQM models need refinement and adaptation; measures must help evaluate both the object of measurement and the reliability of the model used to evaluate it.
Another Example of GQM
Goal: Improve timelines of change request processing
GQM - Goals A GQM model is a hierarchical structure starting with a goal. A GQM goal has three coordinates:
- Issue: Timeliness
- Object (process): Change request processing
- Viewpoint: Project manager
| Goal | Improve timelines of change request processing |
|---|---|
| Purpose | Improve |
| Issue | The timelines |
| Object | Change request processing |
| Viewpoint | Project Manager |
GQM - Questions The goal is refined into several questions. We try to ask at least three groups of questions.
GQM β Questions-Group 1: How can we characterize the object with respect to the overall goal?
- Example: What is the current change request processing speed?
- Example: Is the (documented) change request process actually performed?
GQM β Questions-Group 2: How can we characterize the attributes of the object that are relevant?
- Example: What is the deviation of the actual change request processing time from the estimated one?
- Example: Is the performance of the process improving?
GQM β Questions-Group 3: How do we evaluate the characteristics of the object that are relevant?
- Example: Is the current performance satisfactory from the viewpoint of the project manager?
- Example: Is the performance visibly improving?
GQM β Metrics Once the questions have been developed, we associate them with appropriate metrics. Factors include: amount and quality of existing data, maturity of objects of measurement, and the learning process.
GQM β Model
| Goal |
|---|
| Purpose: Improve |
| Issue: The timelines of |
| Object: Change request processing |
| Viewpoint: Project Manager |
| Question | Metrics |
|---|---|
| Q1: What is the current change request processing speed? | M1: Average Cycle time, M2: Standard deviation, M3: %age cases outside of upper limit |
| Q2: Is the (documented) change request process actually performed? | M4: Subjective rating by Project Manager, M5: %age of exceptions identified during reviews |
| Q3: What is the deviation of the actual change request processing time from the estimated? | M6: ((current average cycle time β estimated average cycle time)/Current average cycle time)*100, M7: Subjective rating by Project Manager |
| Q4: Is the performance of the process improving? | M8: (current average cycle time/Baseline average cycle time)*100 |
| Q5: Is the current performance satisfactory from the viewpoint of the project manager? | M9: Subjective rating by Project Manager |
| Q6: Is the performance visibly improving? | M10: (Current average cycle time/Baseline average cycle time)*100 |
β Key Takeaways
The GQM approach ensures that measurement is driven by goals, not data availability, and provides a three-level structure: goals at the conceptual level, questions at the operational level, and metrics at the quantitative level. A GQM goal specifies purpose, object, issue, and viewpoint, and is refined into at least three groups of questions: characterizing the object, its relevant attributes, and evaluating those attributes. Metrics can be objective (independent of viewpoint) or subjective (dependent on viewpoint), and their selection depends on data availability, object maturity, and the need for model refinement. The approach is illustrated with two real-world examples: AT&Tβs inspection process (goals of plan, monitor/control, improve) and change request processing (goal of improving timelines), showing how each goal generates specific questions and measurable metrics. GQM complements the entity-attribute measurement framework but does not address scale, objectivity, or feasibility, so measures should be used with care.
π§ Quick Revision Questions
- What are the three levels of a GQM measurement model and what does each level represent?
- What are the three objects of measurement in the conceptual level of GQM? Give one example of each.
- What is the difference between objective and subjective data in GQM? Provide one example of each.
- What are the three coordinates of a GQM goal, and what is an example of a goal definition using them?
- What are the three groups of questions that should be asked when refining a GQM goal? Provide an example question for each group based on the change request processing scenario.
π Lecture 42 β Software-Metrics Data Collection
π Overview: This lecture emphasizes that software measurement is only as valuable as the quality of the data collected. It explores the critical characteristics that define βgood data,β outlines the questions an organization must ask before and during data collection, and distinguishes between raw and refined data in the software measurement process.
ποΈ Topics Covered
The lecture begins by stating that software measurement is only as good as the data collected, citing Moroney (1950) on the necessity of a clear purpose. It then defines what constitutes good data by examining six key qualities: correctness, accuracy, appropriate precision, consistency, association with time/activity, and replicability. Finally, it introduces the two types of data (raw and refined) and illustrates the role of data collection in the overall software measurement framework.
π Lecture Summary
Software-Metrics Data Collection
Software measurement is only as good as the data that are collected and analyzed. In other words, we cannot make good decisions with bad data. Moroney (1950) warned that data should be collected with a clear purpose and a clear idea of how they will be analyzed. We need to ask questions like βWhat constitutes good data?β
What is Good Data?
Measures should be well-defined and valid, meaning that our measures should reflect the attributes we claim they do. However, even with a well-defined measure, we need to ask several questions about the data.
-
Are they Correct? Correctness means that the data were collected according to the exact rules of definition of the metric. For example, if a lines of code count is supposed to include everything but comments, then a check for correctness assures us that no comments were counted. A measure of process duration is correct if time is measured from the beginning of one specified activity and ends at the completion of another specified activity.
-
Are they Accurate? Accuracy refers to the difference between the data and the actual value. For example, time measured using an analog clock may be less accurate than time measured using a digital clock.
-
Are they Appropriately Precise? Precision deals with the number of decimal places needed to express the data. For instance, activity duration need not be reported in hours, minutes, and seconds; hours or days are usually sufficient. Likewise, it is not necessary to calculate the mean cyclomatic number to several decimal places, since cyclomatic number expresses one plus the number of decisions in a module; fractions of a decision to several decimal places are meaningless.
-
Are they Consistent? Data should be consistent from one measuring device or person to another, without large differences in value. Thus, two evaluators should calculate the same or similar function-point values from the same requirements document. Similarly, when the same data value is computed repeatedly over time, the data should be captured in the same way. For example, we often measure the growth in size of a product over time, especially during maintenance. We want to be sure that the size measure is calculated the same way each time, so that the resulting measures are comparable.
-
Are they Associated with a Particular Activity or Time Period? If so, then the data should be time-stamped, so that we know exactly when they were collected. This association of values allows us to track trends and compare activities.
-
Can they be Replicated? Data are often collected to support surveys, case studies, and experiments. These investigations are frequently repeated under different circumstances, and the results compared. At the very least, project histories and study results are stored in a historical database, so that baseline measures can be established and organizational goals set. Unless the data collection from one study can be replicated on others, this amalgamation is impossible. Thus, it is very important to assess the quality of data and data collection before data collection begins. Your measurement program must specify not only what metrics to use, but also what precision is required, what activities and time periods are to be associated with data collection, and what rules govern the data collection (such as usage of a specific tool for data collection). Of critical importance is the definition of the metric. Terminology must be clear and detailed, so that all involved understand what the metric is and how to collect it.
How to Define the Data?
There are two kinds of data with which we are concerned:
- Raw data
- Refined data
Raw data results from the initial measurement of process, product, or resource. But through the refinement process, raw data are transformed into refined data by extracting essential data elements. The analysts can then derive values about attributes.
The Role of Data Collection in Software Measurement
This section appears as a figure in the original text, illustrating the transformation of raw data into refined data for analysis.
β Key Takeaways
A student must remember that software data quality is paramount; bad data leads to bad decisions. The six key attributes of good data are correctness, accuracy, precision, consistency, temporal association, and replicability. Correctness demands strict adherence to the metricβs definition, while precision requires using only meaningful decimal places (e.g., not reporting cyclomatic complexity to several decimals). Raw data is the initial measurement, which is then refined into essential data elements for analysis. Finally, a measurement program must specify not just the metrics, but also the rules, precision, and terminology for consistent data collection across projects.
π§ Quick Revision Questions
- What does it mean for data to be βcorrectβ according to the lecture?
- Why is it unnecessary to report the mean cyclomatic number to several decimal places?
- How does βconsistencyβ apply when two evaluators calculate function points from the same requirements document?
- What is the difference between raw data and refined data in the context of software measurement?
- Why is it critical to assess the quality of data collection before data collection begins?