In 1997, the Organisation for Economic Co-operation and Development (OECD) initiated the Programme for International Student Assessment, known as PISA. PISA is an international OECD’s study-test that was first applied in 2000 and has been carried out every three years since. So far, more than 80 countries and economies volunteered to participate in it.
PISA measures the 15-year-olds’ ability (enrolled in grade 7 or higher) to use their reading, mathematics and science knowledge and skills in the same language that is used in each student’s school (e.g. interpreting texts, problem solving, explaining a phenomenon scientifically). These ratings are linked with data from students, teachers, schools and systems, to realize and understand all the different accomplishments. Based on the results, the effectiveness of a country’s education system is measured.
After feedback from our members, ENIL looked into the inclusiveness of the PISA report. What we have learned is that the PISA tests promote exclusion, rather than inclusion, of disabled people. The sampling method and the design of the PISA test are especially problematic in this regard.
Exclusive rather than inclusive sampling
While, in principle, every 15 year old student can be selected for the PISA test on the basis of a randomized sample, countries are allowed to exclude up to 5% of the student population to sit the exams. The criteria for exclusion are the following:
- Geographical inaccessibility of the school [up to 0.5% of the population]
- Special schools [up to 2.0% of the population]
- Excluding students within schools on the basis of disability or lack of language proficiency [2.5% of the eligible students in the school]
A “qualified staff member ” is in charge of deciding whether a student’s “Special Education Needs (SEN) [are] not severe/serious enough to be a barrier/impediment to their participation” (e.g. OECD, 2014: 85; OECD, 2019: 30).
To define the final ‘target population’, countries must provide information on their sample of students to the OECD. Countries can indicate their “school level” and “within school” exclusions in “form ST7B.” In the sampling instructions of 2012, “Special schools for students with special needs” are the only example used in the instructions on how to fill in the ST7B document for “school level exclusions.” There is also no real punishment for surpassing the threshold. If a country provides the PISA team with a valid reason, they are allowed to exceed the threshold (PISA 2018, The PISA target population).
Test design: from limited to maximum accommodations for access needs
There has been a growing awareness around the importance of inclusion at the OECD. This is illustrated by changes in terminology (the term ‘educable mentally retarded’ was changed to “intellectually disabled students”) and an increasing number of OECD reports around inclusion since 2013. That said, the OECD does not aim for real inclusion in the PISA tests. The 2015 PISA technical report states, for example, that students will be “included when their needs are not serious enough to be a barrier to their participation.” PISA also does not encourage schools to provide accommodations to students with access needs to participate in the test. This is made clear in the 2020 PISA project manual, which states that the test can only allow ‘limited modifications and accommodations to assist students with special educational needs’.
While a recent PISA study of 2018 addressed the topic of accommodations, the main focus of the study was on the feasibility of allowing assistive technology like screen readers during PISA tests. Students with multiple disabilities or access needs other than assistive technology were excluded from the study. This shortcoming was also recognized by the authors of the study, who advised the OECD to develop procedures to allow more types of accommodations (including Personal Assistance) in the future.
Time for a UN CRPD compliant PISA test
The situation outlined above makes it clear that rather than demanding an inclusive survey, PISA only gives a lukewarm encouragement towards inclusion.
By reinforcing the stereotype that only some disabled people can be educated, the PISA test reinforces the idea that it is “natural and expected” for disabled students and others with support needs to be excluded from the society. This is unacceptable and goes entirely against the UN Convention on the Rights of Persons with Disabilities (UNCRPD).
ENIL is finalising a letter to the OECD, to express these concerns. We hope this will be the start of a constructive discussion with the OECD on how PISA can be a tool for inclusion, rather than exclusion.
Kyriaki Efthymiou, ENIL’s volunteer
