Performance Measurement and Accommodation: Students with Visual Impairments on Pennsylvania's Alternate Assessment
Print edition page number(s) 17-30
Introduction: This study investigated the use of accommodations and the performance of students with visual impairments and severe cognitive disabilities on the Pennsylvania Alternate System of Assessment (PASA)γÇöan alternate performance-based assessment.
Methods: Differences in test scores on the most basic level (level A) of the PASA of 286 students with visual impairments, grouped into one of three functional vision levels, in grades 3/4 or 7/8 were analyzed descriptively as well as inferentially using the non-parametric Kruskal-Wallis and ad hoc Mann Whitney U tests. In addition, teachers of students with visual impairments viewed the videotaped assessments of 257 of these students to record accommodations used. Accommodations data were analyzed for patterns and relationships with test item types. Chi-Squares were calculated.
Results: There was a significant difference in test scores by functional vision level, with students who predominantly use vision performing significantly better than students who use a combination of vision and other senses or use other senses in place of vision. The accommodations used generally matched a student's reported level of functional vision with layout and presentation accommodations being the most prevalent.
Discussion: The types of accommodations selected may be due to the structure of alternate assessments but also raise questions about what is a progression in reading for students who do not access pictures and print with vision. Differences in performance raise questions of accessibility vs. ability and the level of difficulty of test items when accommodations are used. Implications for practitioners: Practitioners should advocate for guidance from alternate assessments on accommodation use that mirrors the intent of test items and consider how state alternate standards translate when tasks addressing those standards are adapted for students with visual impairments. Test developers should consider students with visual impairments when designing and interpreting alternate assessment test items.
This research was supported by a Pennsylvania Alternate System of Assessment contract from the Bureau of Special Education, Pennsylvania Department of Special Education.
Students with severe cognitive disabilities, including students with visual impairments, participate in state assessments through alternate assessments based on alternate achievement standards (AA-AAS). These tests measure progress on established alternate standards in math, reading, and science through the use of portfolios, checklists, rating scales, or performance-based assessments (Quenemoen, Thompson, & Thurlow, 2003; Thompson & Thurlow, 2003). Among these tests, performance-based assessments most closely mirror traditional standardized assessments (Roeber, 2002). Twenty-one states use performance-based tasks or events for alternate assessments (Cameto et al., 2009).
Concerns about performance on assessments by students with and without visual impairments, in general, raise an important question: Are these real differences in ability, or are they related to problems of accessibility (such as format, the quality of adaptations, or the use of accommodations)? Evidence suggests that students with visual impairments may experience challenges with adapted layouts that cover more than one page and test items that involve spatial or multiple comparisons (Caton, 1977; Dimcovic & Tobin, 1995; Stone, Cook, Laitusis, & Cline, 2010). In addition, the intent and the difficulty of items must be considered when items are adapted to another sensory format (such as from visual to tactile or auditory). The lack of opportunity to learn certain concepts (like visually dependent experiences) can also affect performance (Geisinger, 1994; Wyver & Markham, 1999). And when students with visual impairments have additional cognitive challenges, changes in format and opportunities to learn may impose additional barriers to accessibility (for example, braille words in place of pictures).
Principles of universal design for learning applied to AA-AAS can help reduce the number and types of accommodations that students will need. These principles include making sure that vocabulary is appropriate for the grade and testing level, that pictures have clearly defined features, and that assessments allow for changes in format without changing meaning or difficulty (Thompson, Johnstone, Anderson, & Miller, 2005). Universal design for learning does not eliminate the need for accommodations, since some of its principles that may provide greater access for many students may be inconsistent with the principles that will make a test most adaptable to another group (Allman, 2004; Thompson et al., 2005).
All students who take AA-AAS have significant cognitive challenges, with disabilities requiring unique accommodations. Typical accommodations used for students with visual impairments on regular standardized assessments include braille or large print, extended time, tactile graphics or descriptions, items read aloud, responding in the test booklet or via a scribe, and using assistive technology (Allman, 2002, 2004). These accommodations may or may not be appropriate for students with visual impairments and severe cognitive disabilities on AA-AAS. In addition to adaptations in sensory format, the test administrator is typically more involved during an alternate assessment, which may lead to the adaptation of other aspects of the presentation of tests that affect a student's ability to respond. The absence of research in the area of performance on and accommodations for the use of AA-AAS for students with visual impairments and severe cognitive disabilities leaves teachers and test developers without guidance for this population.
The study reported here explored the performance of students with visual impairments and severe cognitive disabilities (referred to as students with visual impairments in the research questions and hereafter) and the assessment accommodations used when taking the Level A Grade 3-4 or 7-8 Pennsylvania Alternate System of Assessment (PASA). The following research questions were investigated:
- Were there significant differences in the scores of the students with visual impairments at different levels of functional vision?
- What accommodations did teachers make to adapt the PASA for the students with visual impairments?
- Were there relationships between the types of accommodations made and the students' level of functional vision or the type of test item?
Data collected from the spring 2005 administration of the PASA were used for the study. The Institutional Review Board at the University of Pittsburgh approved the study. The PASA is a performance-based alternate assessment for students with severe cognitive disabilities consisting of 20-25 test items in both math and reading. Test administrators videotape the assessment of each student and send it in to the PASA headquarters for scoring. During scoring conferences, pairs of trained teachers score test items using a scale of 0-5 according to a rubric based on the accuracy of the response and level of independence. The reliability of scoring is checked by double scoring approximately 15% of the videotapes. Exact agreement reliability averaged 85% for Grades 3 and 8 (PASA, 2005).
The PASA is administered at three levels of complexity. Level A is the most basic and was chosen for the study because 76% of the students with visual impairments took it. Grades 3-4 and 7-8 were analyzed because of the comparison that these two levels provide since they contain similar skills. Progression is measured in Grade 7-8 through the use of pictures instead of objects in reading, and closer discriminations are required in both math and reading. The setup of test items on the Level A PASA involves providing three choices from which to select (objects or pictures) and sometimes an additional target item.
Data from 286 students with visual impairments who took Level A (143 in each grade) were used to analyze performance, and data from 257 of these students were used for analyzing accommodations. If a student (n = 23) took the reading or math assessment at another level, data for that assessment were not used in the analyses.
Students with visual impairments were identified by adding additional questions to the standard demographic form that the test administrators (usually the students' special education teachers) filled out and returned with the recorded assessments. The test administrator indicated if a student had a visual impairment documented in the student's Individualized Education Program. If the student did, additional questions included indicating the visual condition and selecting the student's level of functional vision. Three categories were available: primarily uses vision for most tasks (V), uses a combination of vision and other senses for most tasks (CV), or uses other senses in place of vision for most tasks (NV). Table 1 lists the students by category of functional vision. Cortical visual impairment (48%) was the most frequently reported visual condition. Other conditions included optic nerve hypoplasia or atrophy (12%), retinopathy of prematurity or retinal detachments (9%), cataracts or aphakia (6%), and syndrome-related visual impairment (5%).
The addition of visual impairment-specific questions on the demographic form helped to identify an additional 83% of the students with visual impairments who took the PASA. Only 17% (n = 48) of the students were identified as having a primary disability of visual impairment, all but one of whom attended a school for students who are visually impaired. The largest primary disability designation was multiple disabilities (57%).
Data on performance
The scores (0-5) assigned to each test item were used to analyze the students' performance, with a focus on the means for the total test score for each student. A descriptive and inferential approach was used to investigate research question 1. The percentages of students with visual impairments (as a whole group and by functional vision category) who were classified as "proficient," "novice," and "emerging" on the PASA were calculated. In addition, the Kruskal-Wallis test, a nonparametric version of a one-way analysis of variance, was conducted for each grade and subject (math or reading) to compare differences in the mean scores among the students with visual impairments who were grouped at the three levels of functional vision. A nonparametric test was used to account for the negatively skewed distribution of the PASA test takers. The Mann Whitney U test was conducted as a post hoc analysis to confirm the source of the group differences found on the total score.
Data on accommodations
A total of 257 assessment videotapes were reviewed and coded. Twenty-two certified teachers of students with visual impairments, 36% itinerant, 36% at schools for students with visual impairments, and 28% in other settings (such as resource rooms and a university), were trained (in a three-hour session) to code the videotapes. The accommodations coding sheet contained 29 codes organized by categories: substitutions (like objects for pictures), picture or object enhancement (using colored pictures, for example), layout or setup (creating a defined space, for instance), instructions or directions (such as using alternate wording), and response (like the use of an augmentative or alternative communication device). When no accommodation was used, the teachers of students with visual impairments indicated whether they thought, on the basis of their professional opinions and observations of the students' performance on the tests, the item was "OK as is" or "needed accommodations." The teachers specified their reasoning when the latter was indicated. The first author conducted a reliability check on 10% of the videotapes, making sure to score at least one entire subject test (math or reading) per teacher of students with visual impairments. The overall agreement for the recorded accommodations was 89%.
Patterns of accommodations were analyzed, and chi-square statistics were calculated to answer research questions 2 and 3. More specifically, patterns of accommodations were reviewed for relationships with the types of test items or levels of functional vision. For example, if an accommodation that was being provided to about 10 students consistently across test items suddenly dropped to only 3 students on an item, that item was reviewed to gauge if the change would be expected.
Research question 1
The following mean cut scores (for 2005) were used to establish proficiency levels on the PASA Level A assessment: average score of 5.0 (perfect) rated as proficient, 4.99-3.75 rated as novice, and 3.74-0 rated as emerging. Of the 254 students with visual impairments who took the reading tests and the 272 students with visual impairments who took the math tests for whom proficiency classifications could be calculated, no students achieved proficiency in math, and 6 students (2%), all at Grade 7-8, achieved proficiency in reading. Sixty-three students (25%) achieved novice status in reading, and 75 students (28%) achieved novice status in math. Among the 6 students who achieved proficiency in reading, 5 primarily used vision for most tasks (V). The same pattern was evident for the 63 and 75 students who performed at the novice level in reading and math, respectively, with 56% of those students in reading and 48% in math being classified in the V functional vision group. Table 2 shows the breakdown of students with visual impairments by functional vision and proficiency categories as well as a comparison of the students with visual impairments and all students (with and without visual impairments) who took the assessment.
Table 3 presents the mean scores and standard deviations for each functional vision grouping by grade and subject assessment. The results of the Kruskal-Wallis test indicated a significant difference in the total test score by functional vision classification at the p ≤ .01 level for Grades 3-4 and 7-8 in both math and reading. The .01 level was used to account for the increase in the Type 1 error rate from conducting multiple comparisons. A review of the mean ranks, as shown in Table 4, reveals that the students with visual impairments who primarily used vision (V) scored better overall on the Level A PASA in math and reading at Grades 3-4 and 7-8 than did the students who used a combination of vision and other senses (CV) and those who used other senses in place of vision (NV). These results were confirmed, and significant at the p ≤.01 level, by conducting a post hoc analysis using the Mann-Whitney U test. No significant differences were found at either grade level or on either subject test between the CV and NV functional vision groups.
A closer examination by single test items at p ≤ .01 revealed the same pattern among the V, CV and NV groups on all but 3 test items (1 in math and 2 in reading) in Grade 7-8 and all but 25 test items (11 in math and 14 in reading) in Grade 3-4. Skills that were not significantly different at p ≤ .01 between the V and other groups in reading included selecting related objects or pictures and selecting objects or pictures by function. In Grade 3-4, reading skills that were not significant but whose equivalent (using pictures) were then significant in Grade 7-8 included matching objects, selecting similar objects, and answering literal what or who questions. In math, skills at Grade 3-4 that were not significant included selecting most or least, matching the shortest or longest length, selecting the biggest by area or volume, matching the smallest by volume, selecting a lot, matching a set of 1, and selecting the heaviest or lightest. The equivalent of these skills, which required closer discriminations, were all significant in Grade 7-8, with finding the object (clock) that "shows time" being the only skill not significantly different for the V group compared to the others.
Research questions 2 and 3
Most popular accommodations
Table 5 lists the "top four" accommodations by grade level and subject that were used most frequently with the students in each functional vision category. These accommodations are listed in order for each grouping, with the most frequent first. Overall, the most frequently coded accommodations across grade level and subject assessments were in the layout or setup category. Of the total of 29 accommodations that the reviewers coded, only 12 different accommodations among the grade and subject assessments were used for 10 or more students on the same test item.
Also of interest were cases when accommodations were not provided, but when the teachers of students with visual impairments who viewed the videotapes thought that the student taking this test "needed accommodations." Table 6 shows the percentage ranges of students who did not receive accommodations and those who might have benefited from accommodations.
Chi-square tests confirmed a significant difference at the p ≤ .01 level among the three levels of functional vision for both grades and subjects in terms of the number of students who did not receive accommodations. Post hoc analyses at p ≤ .05 indicated that the V group was significantly different from the CV and NV groups except in Grade 3-4 math for the CV group (p = .858). The CV and NV groups were also not significantly different on Grade 3-4 math (p = .637). In terms of the number of students who the teachers of students with visual impairments thought might benefit from accommodations, significant differences were found among the three levels of functional vision for all assessments except Grade 7-8 reading. A post hoc analysis revealed that the V group was significantly different from both the CV and NV groups on the remaining assessments, but no significant differences were found between the CV and NV groups.
Accommodations by type of item
The majority of accommodations were used consistently for students across all the test items. When the accommodation-distribution pattern changed, the majority of occurrences were expected, particularly in math (for example, using colored pictures would be expected only for items in math involving pictures).
One of the most interesting changes in the pattern for the use of accommodations occurred in the Grade 7-8 reading assessment. The replacement of objects was used for a range of 28-34 students across the three functional vision categories for each test item except one. For the test item "comprehends a 'where' question," the substitution of objects was used for only 11 students. This test item was the only Level A item that contained complex pictures. The pictures--a playground, grocery store, and farm--were named, and the student was asked to select the one where swings can be found. Object replacement in this case would require using objects symbolically (such as the use of a ball to represent a playground). This type of abstract substitution teachers may find problematic to make accessible to students who cannot functionally use the pictures that are provided. Paralleling the decrease in the substitution of objects on this test item, there was a slight increase in the use of the accommodation "made test item auditory in place of pictures or objects." However, the increase did not account for the full drop in the use of objects. A closer inspection of this test item for the CV and NV functional vision groups revealed that pictures were used for some of the students, particularly for those in the CV group, with an increase in the accommodation for using colored pictures apparent. A few CV students were presented with the pictures, but the pictures were described.
It is interesting to note the types of test items for which a pattern emerged even if just for a few students using an aural accommodation. In the Grade 7-8 math assessment, for example, aural substitution was used for a few students for the skills of matching time and matching numbers and for the test item, "selects a daytime or nighttime activity" in which the student was given three pictures--a bed, sunglasses, and a rake--and asked to select the one that people usually use at night. Beyond the "where" comprehension question already discussed, auditory substitution seemed to be used across the board for one or two students. A slight increase in aural accommodations was noted for test items for which the substitution of objects could be more difficult. For example, the use of an aural accommodation was more prominent for the "selects a similar function" test item that presented a target picture of a couch and asked the student to select the one used the same way from three pictures (a cat, tree, or chair).
Performance of students with visual impairments
Students with more functional vision (the V group) performed better overall on the Level A Grade 3-4 and 7-8 PASA compared to the CV and NV groups, and consequently had a higher rate of classification as proficient or novice. The average mean total score for the V group across both grades and subjects was 3.6 compared to 2.7 and 2.3 for the CV and NV groups, respectively. On the PASA, a score of a 3.0 meant that the student selected the correct answer only after being provided with scaffolding to arrive at the correct answer (for example, the student was given more information, or the choices were reduced to two instead of three). A score of 2.0 indicated that the student was actively involved in trying to answer the question but could not arrive at a correct answer. On the basis of the mean total scores, the V group was able to arrive at correct answers with or without support significantly more often than the other groups.
Single items that were not performed differently at both grade levels in reading were more "conceptual." This may be an indication of general cognitive challenges of the group of students with visual impairments regardless of their level of functional vision. The greater number of test items that were not significantly different at Grade 3-4 in both math and reading may also be an indication of the impact of cognition for the lower grade level. However, for Grade 7-8 reading and math, in which more items were significantly different, the question is whether the students with more severe visual impairments found the tasks more cognitively challenging, or if an access issue needs to be considered.
This is a question for the group of students with visual impairments as a whole, since 76% of the students with visual impairments took the easiest level of the PASA. In addition, 70% of the Grade 3-4 or Grade 7-8 students with visual impairments who took the more difficult levels were in the V vision group. At the Level A PASA, the students with visual impairments in Grade 3-4 or 7-8 were more likely to fall in the lowest proficiency category (emerging) than in the middle category (novice), where the majority of all students who took Level A PASA fell. Although a wide range of variables (such as the logistics of manipulating materials and opportunity to learn particular concepts) could have contributed to this discrepancy, analyzing the accommodations used during the assessment can provide information on where changes may need to be made.
Overall, the results for the use of accommodations were not surprising. One would expect that the percentage of students who did not receive or need accommodations on the assessment would be higher for the group with more functional vision (V) and that the teachers of students with visual impairments would more frequently judge a need for accommodations for the CV and NV groups. The top four accommodations by functional vision group also generally made sense for the level.
However, an average of 4 students out of 23 in the NV group ("uses other senses in place of vision") did not receive accommodations, which is surprising, especially for the Grade 7-8 test for reading, in which test items are predominantly picture based. Comments from the teachers of students with visual impairments included "Test administrator stated that the student cannot see the pictures, but there is no way to adapt this test"; "Teacher made no appropriate accommodations"; and "Teacher does not orient, no time to explore" and underscore the need for more guidance regarding adapting assessments in a way that is meaningful to students. On the basis of the recommendations of the teachers of students with visual impairments and some of the layout or presentation accommodations that were used less frequently, better orientation and reorientation to the location of the test materials may support access for some students.
The types of accommodations that were selected are worth discussing. Of the six students at the V vision level for whom objects were used in place of pictures, four attended schools for students with visual impairments. Thus, it is possible that for logistical or practical reasons, the teachers may have made the same accommodations for all their students who were taking the assessment. This is not to say that individualization never occurred. The teachers of students with visual impairments noted several instances on the videotapes when the teachers changed testing items to make them more meaningful.
On the PASA, Level A "reading" is defined as "getting information." Tasks progress from the most concrete (getting information from objects) in Grade 3-4 to more abstract (getting information from pictures) by Grade 7-8. When objects were used in place of pictures in Grade 7-8, an accommodation allowed for students with visual impairments and consistently the top accommodation selected, this theoretical progression in the level of difficulty disappeared. Nevertheless, the use of objects for pictures in Grade 7-8 did not seem to give the CV and NV groups an added advantage or level the playing field with the V group, whose overall performance was better. Do these findings suggest that the "reading" progression for CV and NV students may need to be conceptualized differently from the progression established for PASA Level A? Or is Grade 7-8 more conceptually challenging despite or because of the use of objects? The current data challenge the assumption that "reading" pictures is a more abstract and therefore a harder task than "reading" objects. In fact, the reduction in the use of objects to represent complex pictures (a skill in Grade 7-8) by the test administrators suggests that the accommodation may have been considered too abstract compared to the pictures, and it questions whether attempts to move students with visual impairments and severe cognitive disabilities to this abstract level is routinely attempted as part of a student's functional literacy and communication experiences. Further research is needed in the reading progression for students with visual impairments who have severe cognitive disabilities.
Overall, the lack of selection of some accommodations is probably a direct reflection on the test materials and the population of students to whom the Level A test is geared. For example, tactile representations were generally used only in math for items that did not involve the need to actually identify the items (such as counting). Tactile representations of pictures were not used, probably because there were no tactile symbols with which the students were already familiar (such as for communication) that could be applied to the context of the test item: a random tactile picture would not be meaningful to the students.
The fact that other picture- or object-enhancement accommodations did not make the "top four" list may be due to the fact that pictures in the PASA assessment are already fairly large and bold. However, the lack of use of low vision devices and augmentative communication devices raises important questions. Are these accommodations not typically used in the classrooms of this population of students? Does PASA not lend itself well to the use of these accommodations? Is the PASA accessible enough without them? Further investigation is needed to answer these questions.
Finally, the lack of the use of auditory accommodations is also interesting. It may be that making such accommodations on the assessment is logistically challenging, but their lack of use raises the question again about what comprises a functional literacy program for students with visual impairments: Are objects being used abstractly? Are tactile symbols being used? Is listening for comprehension being taught? And are students learning to interpret environmental sounds?
The results of this study should be interpreted by taking some limitations into account. Although reliability in the identification of the accommodations was high, there is a possibility that some were over- or underreported. Also, recommendations about the need for accommodations were based on the professional opinions of qualified teachers of students with visual impairments, but these teachers did not know the students personally. The teachers were limited to making judgments that were based on observations of the assessment overall and on differences in students' behavior on individual test items.
Finally, the accommodations that were identified in the study reflect access for a particular alternate assessment--the PASA. However, many of the results may apply to other performance-based assessments with similar layouts. The results highlight the need to continue to investigate factors that affect access to and the performance of students with visual impairments and severe cognitive disabilities on high-stakes assessments. They also highlight the need to consider whether standards for alternate assessments used by a state reflect a meaningful progression of skills when applied to students with visual impairments and severe cognitive disabilities. Students who are visually impaired are rarely studied as a separate group, but the differences in performance that we found suggest that closer scrutiny is necessary if the results of assessments are going to be meaningful and useful for instruction. The results from this study have led to the creation of a special version of the PASA for students who cannot access pictures or print visually. The success of this adapted version is an area for future research.
Allman, C. B. (2002). Results of survey on state assessment and accountability initiatives: Inclusion of students with visual impairments [research results]. Retrieved from www.aph.org/tests/results.html
Allman, C. B. (2004). Test access: Making tests accessible for students with visual impairments: A guide for test publishers, test developers, and state assessment personnel (2nd ed.). Louisville, KY: American Printing House for the Blind.
Cameto, R., Knokey, A.-M., Nagle, K., Sanford, C., Blackorby, J., Sinclair, B., & Riley, D. (2009). National profile on alternate assessments based on alternate achievement standards: A report From the National Study on Alternate Assessments (NCSER 2009-3014). Menlo Park, CA: SRI International.
Caton, H. (1977). The development and evaluation of a tactile analog to the Boehm Test of Basic Concepts, form A. Journal of Visual Impairment & Blindness, 71, 382-386.
Dimcovic, N., & Tobin, M. J. (1995). The use of language in simple classification tasks by children who are blind. Journal of Visual Impairment & Blindness, 89, 448-459.
Geisinger, K. F. (1994). Psychometric issues in testing students with disabilities. Applied Measurement in Education, 7, 121-140.
Pennsylvania Alternate System of Assessment. (2005). Technical supplement. Retrieved from http://www.PASAassessment.org:80/Rep2005.jsp
Quenemoen, R., Thompson, S., & Thurlow, M. (2003). Measuring academic achievement of students with significant cognitive disabilities: Building understanding of alternate assessment scoring criteria (Synthesis Report 50). Minneapolis: National Center on Educational Outcomes, University of Minnesota-Twin Cities.
Roeber, E. (2002). Setting standards on alternate assessments (Synthesis Report 42). Minneapolis: National Center on Educational Outcomes, University of Minnesota-Twin Cities.
Stone, E., Cook, L., Laitusis, C. C., & Cline, F. (2010). Using differential item functioning to investigate the impact of testing accommodations on an English-language arts assessment for students who are blind or visually impaired. Applied Measurement in Education, 23, 132-152.
Thompson, S. J., Johnstone, C. J., Anderson, M. E., & Miller, N. A. (2005). Considerations for the development and review of universally designed assessments (Technical Report 42). Minneapolis: National Center on Educational Outcomes, University of Minnesota-Twin Cities.
Thompson, S., & Thurlow, M. T. (2003). 2003 state special education outcomes: Marching on. Minneapolis: National Center on Educational Outcomes, University of Minnesota-Twin Cities.
Wyver, S. R., & Markham, R. (1999). Visual items in tests of intelligence for children. Journal of Visual Impairment & Blindness, 93, 663-665.
Kim T. Zebehazy, Ph.D., assistant professor, University of British Columbia, 2125 Main Mall, Vancouver, BC V6T 2G3, Canada; e-mail: <email@example.com>. Naomi Zigmond, Ph.D., professor, Department of Instruction and Learning, University of Pittsburgh, WWPH 5164, Pittsburgh, PA 15260; e-mail: <firstname.lastname@example.org>. George J. Zimmerman, Ph.D., associate professor, Department of Instruction and Learning, University of Pittsburgh; e-mail: <email@example.com>.
Download braille-ready file
Download ASCII text file (ASCII files are for download only)
Previous Article | Next Article | Table of Contents
There are 0 comments on this article.
JVIB, Copyright © 2012 American Foundation for the Blind. All rights reserved.
If you would like to give us feedback, please contact us at firstname.lastname@example.org.