On the Measurement of Competency

Abstract

This conceptual paper defines competency as a complex ability closely tied to performance in real-world situations, organizing the construct, observation, and interpretation according to the Assessment Triangle. In military job and science performance assessments, the sampling of tasks and observation occasions emerged as major sources of measurement error. The authors propose a model that combines the random sampling of real-world tasks, common scoring dimensions across tasks, and generalizability theory to examine the reliability and interpretability of competency scores.

Introduction

Employers, educators, and policymakers demand 21st-century competencies such as critical thinking, analytical reasoning, problem-solving, and communication. Approaches to measuring these competencies remain insufficiently developed.

Taking the definition provided by Hartig et al. (2008) as a starting point, the author proposed a model for measuring competency. The paper follows the Assessment Triangle of construct, observation, and interpretation. Competency is a hypothetical construct that cannot be directly observed; it is inferred from behavior demonstrated in tasks.

This conceptual paper draws on more than 30 years of research by the author and his colleagues. The study reviews a 1968 astronaut task study, military job assessments from the 1980s, science inquiry assessments from the mid-1980s, and collegiate assessments. For each case, the authors explain how the task domain was established, how tasks were sampled and scored, and how scores were interpreted. Generalizability theory was applied to some of the data.

The construct: competence

Synthesizing web-based definitions and the discussion by Hartig et al. (2008), the authors present seven elements of competency. These include physical and intellectual abilities or skills, a capacity for performance encompassing knowledge and execution, standardized conditions, performance standards for judging adequacy, potential for improvement, underlying complex abilities, and observation in real-world situations.

A complex ability involves mobilizing various skills and capabilities in response to a task’s demands and the supports and constraints of the situation. The combination of abilities mobilized changes as sub-goals are achieved and new goals emerge during performance.

Standardization involves keeping task demands and administration conditions consistent across examinees. The authors identify limitations in using portfolios because tasks, conditions, and authorship are difficult to control. They acknowledge cases in which the portfolio itself fits the criterion situation, as in the assessment of artistic competency. Performance standards specify the level of performance that indicates sufficient competency. The potential for improvement means that performance can change through practice and education.

Observation of performance

Observation tasks are intended to reflect real-world situations and require complex abilities and skills. They should allow for improvement through practice and have definable performance standards. The authors adopted the criterion sampling approach of McClelland (1973). After defining the behavioral domain to be assessed, they list possible tasks and responses, then select a sample of tasks and responses from that population for use in the assessment.

Tasks and responses are considered together. In real life, operant responses, in which examinees generate their own behaviors or answers, are common. Performance involves volition, emotion, daily living skills, and interpersonal skills, along with cognitive abilities.

In the 1968 study, common maintenance tasks were identified and purposively sampled from the Apollo missions of the National Aeronautics and Space Administration (NASA). Accuracy and completion time were recorded for participants with astronaut qualifications. Performance declined as participants moved from Earth’s 1 G to lunar 1/6 G and zero gravity.

In the military job assessment, questions about the validity of using training scores as a criterion prompted the authors to measure actual job performance. In the case of Navy radiomen, 124 tasks were rated on performance, supervision, frequency, complexity, mission importance, and error frequency. Twenty-two tasks purposively sampled by experts were compared with samples of the same size selected using three random sampling methods.

Original Table 2. Comparison of purposive and random sampling

Task characteristic	Infinite population simple random sampling	Finite population simple random sampling	Finite population stratified random sampling
PCTSUP	5.06	6.29	3.11
PCTPERF	5.06	6.25	3.25
IMPORT	1.00	1.20	1.06
ERROR	1.00	1.20	0.29
FREQ	3.12	4.08	1.50
COMP	-1.50	-1.80	-0.89

The values in the table show the difference between purposive and random sampling in standard deviation units. PCTSUP is the supervision rate, PCTPERF is the performance rate, IMPORT is importance, ERROR is error frequency, FREQ is performance frequency, and COMP is complexity.

Purposive sampling overrepresented frequently performed tasks and selected tasks that were less complex than average. The authors favored random sampling because it allows sampling error to be evaluated. Stratified random sampling could also include the most important tasks with a probability of 1.00.

The science inquiry assessment drew its domains from textbooks and materials for teacher-student activities. Tasks were designed to fit the space and safety constraints of classrooms. Hands-on experiments and paper-and-pencil assessments were not readily interchangeable. A student who received a perfect score on a hands-on electrical task scored 0 on a short-answer question about the same task.

Interpretation: from observation to competence

Holistic scoring often uses a 6-point scale, but generic criteria alone make it difficult to diagnose individual performance. Analytic scoring evaluates the details of each task, but makes it difficult to derive comparable scores across tasks. The authors propose a mixed approach that combines common dimensions with task-specific content. This method assesses the reliability and validity of information, reasoning errors, and clarity of expression across diverse tasks.

Trained raters were able to score complex performances consistently. Challenges remained with rater drift over time, costs, administration, and delays in reporting results. Natural language processing software trained on benchmark responses could score large volumes of constructed responses quickly.

Generalizability theory treats scores as a sample drawn from a population of tasks, occasions, raters, and observation methods. Variance components indicate the extent to which each factor and the interactions among factors contribute to score variation. Including observation methods allows both reliability and convergence across measurement methods to be examined.

In the Navy machinist study, performance on 11 job tasks was scored by two raters per task. Variance components for persons, raters, tasks, and their interactions were estimated using a random-effects analysis of variance.

Original Table 3. Generalizability analysis of Navy machinist hands-on performance assessment

Source of variance	Estimated variance component (× 1000)	Percent of total variation (%)
Person (P)	6.26	14.45
Examiner (E)	0.00	0.00
Task (T)	9.70	22.40
P×E	0.00	0.00
P×T	25.85	60.00
E×T	0.03	0.00
P×E×T, error	1.46	3.37

The sum of the percentages exceeds 100% due to rounding.

The person-by-task interaction accounted for 60.00% of the total variation, and reliability with one rater and 11 tasks was 0.72. Adding raters did not change reliability, but adding six tasks could increase it to 0.80. Task sampling imposed cost and time demands.

In the science performance assessment, three tasks—Paper Towels, Electric Mysteries, and Bugs—were scored by two raters across two occasions. The two observation methods were notebook records and direct observation.

Original Table 4. Variance components of the science performance assessment

Source of variance	Estimated variance component (× 1000)	Percent of total variation (%)
Person (P)	330.00	13.11
Task (T)	0.00	0.00
Occasion (O)	100.00	4.14
Method (M)	0.00	0.00
P×T	650.00	25.83
P×O	0.00*	0.00
P×M	10.00	0.50
T×O	30.00	1.03
T×M	120.00	4.67
O×M	0.00	0.00
P×T×O	790.00	31.35
P×T×M	20.00	0.63
P×O×M	0.00	0.00
T×O×M	0.00	0.00
P×T×O×M, error	470.00	18.70

*A small negative variance component was set to 0. The figures and percentages are as in the original text.

The person-by-task and person-by-task-by-occasion interactions, along with the residual, were the primary sources of variation. The authors described occasions as a “hidden facet” that is difficult to detect in a single assessment. While the effect of observation methods was small in this case, larger discrepancies appeared in other studies. They concluded that reliable individual scores require multiple tasks and responses.

Collegiate Learning Assessment (CLA)

The CLA assesses college students’ critical thinking, analytical reasoning, problem-solving, and clear communication. The main reporting units are institutions or programs. Alongside observed performance, it reports value added beyond the level expected from entry scores such as the SAT. It provides individual scores privately to students.

The assessment consists of performance tasks and two types of analytic writing. In the DynaTech task, a student acts as an assistant to a president considering an aircraft purchase. The student evaluates accident risks and decides whether to purchase the SwiftAir 235 by comparing accident articles, accident reports, emails, and performance data. The writing section asks students to make and defend an argument or critique a given argument.

The CLA used the internet, computer scoring, and statistical sampling. Matrix sampling, in which not all students answered all items, reduced the time required. Responding to a subset of tasks still took 90 minutes. Analytic writing was scored by computers. The design provided instructors with similar tasks to support practice in solving real-world problems.

One possible model for measuring competency

The authors’ final model defines a population of tasks and responses based on the competency definition and randomly samples actual criterion situations. It examines score quality using generalizability theory and scoring criteria with common dimensions. The theory is used to determine the number of tasks, raters, and observation occasions required for operational assessments.

The authors presented the model as an idea under development, acknowledging that the cases do not address all elements of their proposed definition of competency. Performance standards and potential for improvement were not covered in detail, and volition and emotion were included alongside cognition in the final definition.

Reading notes

The estimate that adding raters does not change reliability, while adding 6 tasks can increase it from 0.72 to 0.80, provides a basis for allocating resources to observe performance across multiple tasks in this assessment. This result relies on variance components estimated in the Navy engineman study. In that study, 11 job tasks were scored by 2 raters per task, and the person × task interaction accounted for 60.00% of the total score variance. The Navy radioman case draws attention to both how tasks are selected and how many are selected. The 22 tasks purposively sampled by experts from 124 included a disproportionate number of frequently performed duties and were less complex than average. Stratified random sampling can include the most critical tasks with a probability of 1.00 while still allowing sampling error to be assessed.

References

Shavelson, R. J. (2010). On the measurement of competency. Empirical Research in Vocational Education and Training, 2(1), 41–63. https://doi.org/10.1007/bf03546488