Continuous Improvement

Types of Rubrics

Rubrics can be categorized into three broad groups. The first group distinguishes generic from task-specific rubrics. The second group distinguishes analytic from holistic descriptors of performance. The third group distinguishes primary trait from multiple trait rubrics.

Generic and Task-Specific Rubrics

Rubrics can be categorized as generic or task-specific. As is so often the case in assessment, the line between the two categories may overlap creating a combination or hybrid model. Many task-based rubrics are adaptations of generic scales, and many are combinations of generic elements and elements specific to a particular performance task.

Generic rubrics

Generic rubrics can be applied to a number of different tasks within a single mode of communication (Interpersonal, Interpretive, Presentational). ACTFL recommends the following domains be considered when designing rubrics for the three modes of communication: Functions, Contexts/Content, Text Type, Language Control, Vocabulary, Communication Strategies, Cultural Awareness (ACTFL Performance Descriptors for Figure 1Language Learners). The rubric in Figure 1 is a generic Interpersonal rubric designed to be learner-friendly. Learners can ask themselves the questions that will be used to judge their performance in a conversation at the novice level. The questions reflect all of the domains except Contexts/Content which is represented in the choice of topic for the learners to discuss. This generic rubric serves as a guide for novice learners to reference any time they are engaged in an Interpersonal task. They know the characteristics of a strong performance, and know that they must demonstrate a “strong performance” consistently over multiple novice-level topics in order to be considered a strong novice interpersonal communicator.

Task-specific rubrics

Task-specific rubrics are used with particular tasks, and their criteria and descriptors reflect specific features of the elicited performance. Tedick (2002) writes: "While some rubrics are created in such a way as to be generic in scope for use with any number of writing or speaking tasks, it is best to consider the task first and make sure that the rubric represents a good fit with the task and your instructional objectives. Just as a variety of task-types should be used in language classrooms, so should a varietyFigure 2  of rubrics and checklists be used for assessing performance on those tasks" (p. 37). For example, this Presentational Writing task requires learners to refer to a series of pictures to tell a story in the past about a visit to the monuments in Paris, France. The rubric (Figure 2) focuses on control of past tenses, use of story form, and factual information about the monuments.

Hybrid rubrics that combine features of generic and task-specific rubrics are very useful in classroom assessment because they provide feedback to learners on broad dimensions of language production along with their performance on the particular competencies and knowledge targeted by a specific task within a specific unit of instruction. Figure 3Teachers may keep the generic language production elements as they are and change one or two categories to focus on requirements for a specific task. For example, the rubric in Figure 3 for a Presentational Writing task includes categories that are generic (used for all writing tasks) and categories that are task-specific. The task is to write a 5-paragraph essay comparing the importance of biodiversity in France to the United States. In this case, the categories of Organization, Vocabulary, Accuracy, and Culture are elements that are evaluated in all Presentational Writing tasks. In addition to these four categories, a fifth category about the use of a variety of authentic resources is added for this task. It is important to remember that presentational tasks benefit from feedback and subsequent revisions to achieve a polished product.


Holistic and Analytic Rubrics

Rubrics may also be categorized as holistic or analytic. Holistic rubrics describe the characteristics of a performance to give an overall judgment of the quality of the performance. An analytic rubric looks at the individual characteristics of a performance and judges each characteristic separately.

Holistic rubrics:

Figure 4In holistic evaluation, raters make judgments by forming an overall impression of a performance and matching it to the best fit from among the descriptions on the scale. Each band on the scale describes performance on several criteria (e.g. text type + vocabulary + language control). Three or four levels of performance are commonly found in holistic rubrics. Holistic scales may be either generic or task-specific. Figure 4 is a sample of a holistic rubric for a Presentational Writing task.

Advantages of holistic rubrics:

  • They are often written generically and can be used with many tasks.
  • They save time by minimizing the number of decisions raters must make.
  • Trained raters tend to apply them consistently, resulting in more reliable measurement.
  • They are good for summative assessments as they give an overall judgment of performance.

Disadvantages of holistic rubrics:

  • They do not provide specific feedback to learners about the strengths and how to improve performance.
  • Because they lack specific details, they are not useful for formative assessments.

(Tedick (2002), Mueller (2002), and TeacherVision.com (2000-2002).)

Analytic rubrics

Analytic rubrics often combine performance categories from a generic rubric with categories directly related to a task, such as demonstrating understanding of specific lesson content (Moskal, 2000). In practice, the names "analytic rubric" and "multiple trait rubric" may be used interchangeably.

Performance dimensions commonly found in analytic rubrics include the domains in the ACTFL Perfomance Descriptors:

  • Functions
  • Contexts/Content
  • Text type
  • Language control
  • Vocabulary
  • Communication strategies
  • Cultural Awareness

These rubrics may also include task specific items such as:

  • Pronunciation and Intonation
  • Creativity
  • Effective use of technology
  • Successful collaboration
  • Evidence of integration of multiple sources of factual information

Figure 4Figure 5 presents an adaptation of a well-known analytic scale for evaluating ESL writing performance. Describing this rubric, Tedick (2002) writes: "Note that the scale assigns different weights to different features. This allows a teacher to give more emphasis to content than to grammar or mechanics, for example. The option to weigh characteristics on the scale represents an advantage to analytic scoring." (p. 35).

Figure 6Figure 6 shows an analytic scale for role plays and interviews used with students in first-year French courses at the University of Minnesota. This rubric can be used with other languages. In this example, all criteria are weighted equally.

Advantages of analytic rubrics:

  • They provide useful feedback to learners on areas of strength and weakness.
  • Their dimensions can be weighted to reflect relative importance of individual criteria.
  • They can show learners that they have made progress over time in some or all dimensions when the same rubric categories are used repeatedly (Moskal, 2000).

Disadvantages of analytic rubrics:

  • "The whole is greater than the sum of its parts." Tedick (2002) notes: "Separate scores for different aspects of a student’s writing or speaking performance may be considered artificial in that it does not give the teacher (or student) a good assessment of the "whole" of a performance." (p. 36).
  • They take more time to create and use.
  • There are more possibilities for raters to disagree. It is more difficult to achieve intra- and inter-rater reliability on all of the dimensions in an analytic rubric than on a single score yielded by a holistic rubric.
  • There is some evidence that raters tend to evaluate grammar-related categories more harshly than they do other categories (McNamara, 1996), thereby overemphasizing the role of accuracy in providing a profile of learners' proficiency.
  • There is some evidence that "when raters are asked to make multiple judgments, they really make one..." (Fulcher, 2009). Care must be taken to avoid a "halo effect" and focus on the individual criteria to assure that diverse information about the learner's performance is not lost.

 

Primary and Multiple Trait Rubrics

Rubrics can also be categorized as primary trait or multiple trait rubrics. A primary trait rubric is task-specific and evaluates performance based on only one characteristic. A multiple trait rubric evaluates performance based on several characteristics of a specific task.

Primary trait rubrics

Primary trait scoring, as developed by Lloyd-Jones and Carl Klaus (Lloyd-Jones, 1977) was designed to evaluate the primary language function or rhetorical trait elicited by a given writing task or prompt. "Primary trait assessment in its initial formulations focused on the specific approach that a writer might take to be successful on a specific writing task; every task required its own unique scoring guide" (Applebee, 2000, p. 4). In its original form, primary trait scoring would be strictly classified as task-specific, and performance would be evaluated on only one trait, such as the "Persuading an audience" example from Tedick (2002, p. 36) for a task requiring learners to write a persuasive letter to the editor of the school newspaper:

Primary Trait: Persuading an audience
0
Fails to persuade the audience.
1
Attempts to persuade but does not provide sufficient support.
2
Presents a somewhat persuasive argument but without consistent development and support
3
Develops a persuasive argument that is well developed and supported.

Today, you may find that primary trait rubrics vary markedly from their original design and intended use. Applebee notes: "Over the years as primary trait approaches were used more widely, they evolved into a more generic approach which recognized the similarities in approach within broad uses or purposes. The basic question addressed in scoring, however, remained, 'Did the writer successfully accomplish the purpose of this task?' To insure that raters maintained this focus, scoring guidelines usually instructed raters to ignore errors in conventions of written language, and to focus on overall rhetorical effectiveness" (p. 4).

Primary trait rubrics are useful in formative assessments designed to determine how well learners perform a particular language function they have been working on in class. For example, if several lessons have been devoted to working on descriptive language, a culminating writing task might be scored solely on its effectiveness as a description.

Multiple trait rubrics

Hamp-Lyons (1991) coined the term multiple trait scoring for rubrics that she designed, based on the concepts of primary trait scoring, to provide diagnostic feedback to learners and other stakeholders about performance on "context-appropriate and task-appropriate criteria" for a specified topic/text type. She designed her multiple trait rubrics to beFigure 7 applicable across a range of similar tasks. Currently, multiple trait (or multitrait) rubrics are commonlyconsidered to be task-specific, although one or more of their dimensions might also be found in generic, analytic rubrics. Figure 7 illustrates a task and multitrait scoring rubric from a resource for language teachers (Petersen, 1999).

Multiple trait rubrics look like analytic rubrics in that performance is evaluated in several categories, and, in practice, you may find the terms used interchangeably. However, analytic rubrics usually evaluate the more traditional and generic dimensions of language production, while the criteria in multiple trait rubrics focus on specific features of performance necessary for successful fulfillment of a given task or tasks.

Advantages of primary and multiple trait rubrics

  • The rubrics are aligned with the task and curriculum.
  • Feedback is focused on one or more dimensions that are important in the current learning context.
  • With a multiple trait rubric, learners receive information about their strengths and weaknesses.
  • Primary and multiple trait rubrics are generally written in language that students understand.
  • Teachers are able to rate performances quickly.

Disadvantages of primary and multiple trait rubrics

  • Information provided by primary trait rubrics is limited and may not easily translate into grades.
  • Task-specific rubrics cannot be applied to other tasks without adaptation of at least one or more dimensions.

(Hamp-Lyons, 1991; McNamara, 1996; Tedick, 2002.)

 

 

CARLA Mailing List Signup Contact CARLA CARLA Events Donate to CARLA CARLA on Facebook CARLA on YouTube Twitter
Center for Advanced Research on Language Acquisition (CARLA) • 140 University International Center • 331 - 17th Ave SE • Minneapolis, MN 55414