Issue 26	Autumn 1999

Social Research Update is published quarterly by the Department of Sociology, University of Surrey, Guildford GU2 7XH, England. Subscriptions for the hardcopy version are free to researchers with addresses in the UK. Apply by email to sru@soc.surrey.ac.uk.

Examining the paradox of achievement gaps

Stephen Gorard is a senior lecturer at the School of Social Sciences, Cardiff University. His research interests include socio-economic segregation in schools, differential attainment by gender, school effects, the role of technology in education, and patterns of lifelong learning. He recently published ‘Keeping a sense of proportion: the “politician’s error” in analysing school outcomes’, British Journal of Educational Studies , 1999, 47, 3.

There are at least two methods of calculating achievement gaps in current use, similar to those used to calculate social segregation and mobility. Each method clearly seems valid to its proponents, yet their results in practice are radically different and often contradictory. This Update considers both methods and some related problems in the calculation of achievement gaps. The issue is one with significant implications for social researchers, as well as commentators in many areas of public policy using similar indicators of performance.

The calculation and discussion of achievement gaps between different sub-groups of students (‘differential attainment’) has become common among policy-makers, the media, and academics. An ‘achievement gap’ is an index of the difference in an educational indicator (such as an examination pass rate) between two groups (such as males and females). In addition to patterns of differential attainment by gender, recent concern has also been expressed over differences in examination performance by ethnicity, by social class, and by the ‘best’ and ‘worst’ performing schools. The concerns expressed in each case derive primarily from growth in these gaps over time.

One of two methods of calculating differential attainment are generally used. The most common uses percentage points as a form of ‘common currency’. Thus, if 30% of boys and 40% of girls gain a C grade in Maths GCSE in one year, and 35% of boys and 46% of girls gain the equivalent a year later, the improvement among girls is said to be greater, in the way that six (46-40) is greater than five (35-30). This is justified by its advocates since percentages are, in themselves, proportionate figures. If true, it would mean that girls were now even further ‘ahead’ of boys than in the previous years. Thus, the gender gap has grown.

The second method calculates the change over time in proportion to the figures that are changing. This approach is advocated by Newbould and Gray in an EOC study of gendered attainment (Arnot et al. 1996, see also Gorard et al. 1999a). For them, an achievement gap is the difference in attainment between boys and girls, divided by the number of boys and girls at that level of attainment. More formally, the entry gap for an assessment is defined as the difference between the entries for girls and boys relative to the total entries.

Entry Gap = (GE-BE)/(GE+BE)

The achievement gap for each outcome is defined as the difference between the performances of boys and girls, relative to the performance of all entries, minus the entry gap.

Achievement Gap = (GP-BP)/(GP+BP) - Entry Gap

The interesting point about the two methods is that they give different results from the same data. For example, Gibson and Asthana (1999) claim that the gap in terms of GCSE performance between the top 10% and the bottom 10% of English schools has grown significantly from 1994 to 1998. Their figures are reproduced in Table 1. This shows the proportion of students attaining five or more GCSEs at grade C or above (the official benchmark), for both the best and worst attaining schools in England. It is clear that the top 10% of schools has increased its benchmark by a larger number of percentage points than the bottom 10%. The authors conclude that schools are becoming more socially segregated over time, since ‘within local markets, the evidence is clear that high-performing schools both improve their GCSE performance fastest and draw to themselves the most socially-advantaged pupils’ (in Budge 1999:3).

This conclusion would be supported by a host of other commentators using the same method (including Robinson and Oppenheim 1998, and Chris Woodhead, in the Times Educational Supplement 12/6/98:5). Similar conclusions using the same method have been drawn about widening gaps between social classes (Bentley 1998), between the attainment of boys and girls (Stephen Byers, in Carvel 1998, Bright 1998, Independent 1998), between the performance of ethnic groups (Gillborn and Gipps 1996), and between the results of children from professional and unemployed families (Drew et al., in Slater et al. 1999).

The second method, using the same figures, might produce a result like Table 2. Although the difference between the deciles grows larger in percentage points over time, this difference grows less quickly than the scores of the deciles themselves. On this analysis, the achievement gaps are getting smaller over time. This finding is confirmed by the figures in the last column showing the relative improvement of the two groups. The rate of improvement for the lowest ranked group is clearly the largest. The bottom decile would, in theory at least, eventually catch up with the top decile (Gorard 1999). The same reanalysis can be done in each of the examples above to show that the gaps between schools, sectors, genders, ethnic groups, and classes are getting smaller over time. This would be the exact opposite in each case to the published conclusions.

To summarise the position so far: using the most popular method of comparing groups over time, there appears to be a crisis in British education. Differences between social groups, in terms of examination results expressed in percentage points, are increasing over time and so education is becoming increasingly polarised by gender, class, ethnicity, and income. However, using the second method, the opposite trend emerges. This is the ‘paradox of achievement gaps’. Both methods are used in different studies. Some writers have even used the equivalent of both methods in the same study (e.g. Levacic et al. 1998, Lauder et al. 1999).

The ‘index wars’

Absolute rates are expressed in simple percentage terms, while relative rates (odds ratios) are margin-insensitive in that they remain unaltered by scaling all the numbers in a row or column (as might happen to the class structure over time, for example). In Table 3, 25% of those in the middle class are of working class origin, whereas in Table 4 the equivalent figure is 40% (from Marshall et al. 1997, pp. 199-200). However, this cannot be interpreted as evidence that Society B is more open than Society A, because the percentages do not take into account the differences in class structure between Societies A and B, nor the changes over time (‘structural differences’).

Given a two-by-two table of the form:

From the point of view of social justice... this is of course both crucial and convenient, since our interest lies precisely in determining the comparative chances of mobility and immobility of those born into different social classes - rather than documenting mobility chances as such (Marshall et al. 1997:193 ).

Comparing indices

Despite the differences, there are many similarities between the methods and their variants (Darroch 1974). For instance, if 100 girls and 100 boys sit an examination, of whom 30 girls and 20 boys achieve a particular grade, the results produced are as in Table 5 (the cross-product ratio is 1.7, etc.). If in a later test 60 of 100 girls and 40 of 100 boys achieve the same grade, the figures from the first and last methods change, while the others remain the same.

The method of percentage points suggests that the gap between girls and boys has doubled from Test 1 to Test 2, whereas the cross-product ratio suggests that the gap has increased less dramatically. The other two methods suggest no change over time.

Resolving the paradox of achievement gaps?

Conclusion

It is not always clear (from literature review and personal experience) that the commentators using the percentage point difference approach are aware that there are other, perhaps better, methods. Without restarting the index wars, it would be wise for this issue to be at least debated in relation to the paradox of achievement gaps. If the multiplicative model is preferred, the consequences would be momentous for much existing research, for the cumulated conclusions of some entire fields of endeavour, the validity of many ‘qualitative’ studies in related areas, for public research funding priorities, and above all for educational policy. As with the earlier debates about class and stratification, other fields of social science investigation would be affected as well.

References

Social Research Update is published by:

Edited by Nigel Gilbert.

Autumn 1999 © University of Surrey

Permission is granted to reproduce this issue of Social Research Update provided that no charge is made other than for the cost of reproduction and this panel acknowledging copyright is included with all copies.