The first PISA results for India: The end of the beginning

By Ajay Shah, on January 5th, 2012

The PISA 2009+ results are the end of the beginning. For the last decade there has been a debate. Some argued the levels of learning inside Indian elementary schools (primary and upper primary) are a national scandal and a threat to the future of India’s society, polity, and economy. Others appeared to believe that the main, if not only, problem with Indian schools was that not enough children attend them and that with more money and more of the same, all would be well. The last five years saw a relentless accumulation of evidence about the crisis of learning. The establishment has tried to deny, deflect, and dismiss the evidence on learning. Eventually the Government of India agreed to participate in the PISA (Programme for International Student Assessment) – but only for two states, Tamil Nadu and Himachal Pradesh – and both sides agreed PISA was the litmus test. The PISA 2009+ results, which are both official and are beyond gain-saying are unspeakably bad. They confirm the worst of what anyone has been saying about the levels of learning in India elementary education.

  • In reading of the 74 regions participating in PISA 2009 or 2009+ these two states beat out only Kyrgyzstan.
  • In mathematics of the 74 regions participating the two states finished again, second and third to last, again beating only Kyrgyzstan.
  • In science the results were even worse, Himachal Pradesh came in dead last, behind Kyrgyzstan, while Tamil Nadu inched ahead to finish 72nd of 74.

But just coming in last (if we can dismiss as a relevant comparator for India a tiny Central Asian state) does not convey the enormity of how bad these results were, as not only was India last, it was far, far, behind its aspirations, both at the bottom and at the top levels of performance.

PISA expresses the levels of performance in two ways, an overall index number and the fraction of students achieving various “levels” of achievement. The PISA index numbers for each subject are scaled so that the typical OECD student is at 500 and the standard deviation across OECD students is 100. The testing of thousands of students allows the results to present not only the average but also the worst (5th percentile) and best (95th percentile) students do in each country/region. PISA also classifies student performance into “levels” that represent different degrees of mastery of the material.

Table 1 compares India’s performance to three groups of countries. The economic superstars have successfully completed the transition from poor to rich economies in just two generations – Singapore, Hong Kong, Korea (China’s only results are just for the city of Shanghai, which are the highest scores of any region tested, but this is too a typical to really be comparable) and India aspires to their sustained success economically. The current super powers are represented by the USA and the OECD average reflects India’s aspirations as a superpower. The rising powers are represented by the BRIC countries of Russia and Brazil which reflect the rise of the emerging markets.

Compared to the economic superstars India is almost unfathomably far behind. The TN/HP average 15 year old is over 200 points behind. If a typical grade gain is 40 points a year Indian eighth graders are at the level of Korea third graders in their mathematics mastery. In fact the average TN/HP child is 40 to 50 points behind the worst students in the economic superstars. Equally worrisome is that the best performers in TN/HP – the top 5 percent who India will need in science and technology to complete globally – were almost 100 points behind the average child in Singapore and 83 points behind the average Korean – and a staggering 250 points behind the best in the best.

As the current superpowers are behind the East Asian economic superstars in learning performance the distance to India is not quite as far, but still the average TN/HP child is right at the level of the worst OECD or American students (only 1.5 or 7.5 points ahead). Indians often deride America’s schools but the average child placed in an American school would be among the weakest students. Indians might have believed, with President Obama, that American schools were under threat from India but the best TN/HP students are 24 points behind the average American 15 year old.

Even among other “developing” nations that make up the BRICs India lags – from Russia by almost as much as the USA and only for Brazil, which like the rest of Latin America is infamous for lagging education performance does India even come close – and then not even that close.

To put these results in perspective, in the USA there has been huge and continuous concern that has caused seismic shifts in the discourse about education driven, in part, by the fact that the USA is lagging the economic superstars like Korea. But the average US 15 year old is 59 points behind Koreans. TN/HP students are 41.5 points behind Brazil, and twice as far behind Russia (123.5 points) as the US is Korea, and almost four times further behind Singapore (217.5 vs 59) that the US is behind Korea. Yet so far this disastrous performance has yet to occasion a ripple in the education establishment.

Table 1: Comparing Indian (Tamil Nadu and Himachal Pradesh) students mastery of mathematics to economic superstars, current superpowers, and rising superpowers
Country/Region 5th mean 95th HP+TN average to comparator average HP+TN average to comparator 5th percentile HP+TN best (95th) to comparator’s average HP+TN best (95th) to comparator’s 95th
Points TN/HP is behind (-)/ahead(+)
Economic Superstars
Singapore 383 562 725 -217.5 -38.5 -99 -262
Hong Kong 390 555 703 -210.5 -45.5 -92 -240
Korea 397 546 689 -201.5 -52.5 -83 -226
Current Superpower
OECD avg. 343 496 643 -151.5 1.5 -33 -180
USA 337 487 637 -142.5 7.5 -24 -174
Rising Superpowers
Russia 329 468 609 -123.5 15.5 -5 -146
Brazil 261 386 531 -41.5 83.5 77 -68
Indian States
Tamil Nadu 241 351 468
Himachal Pradesh 223 338 458
Average of TN and HP 232 344.5 463
Source: PISA 2009 Plus Results, Table B.3.1 for first three columns and author’s calculations.

I have emphasised Mathematics because many believed math was an Indian strong suit. The results for reading and science are similarly bad. Table 2 shows science results in a different format, which shows the proportion of children in various categories of performance. There are three points:

  1. “Below level 1″ doesn’t even have a description as it implies  that so little proficiency is demonstrated it is impossible to
    distinguish from not knowing anything at all. In the USA, even with its socio-economic and racial inequalities and language inequalities and its failing inner city schools, only 4.2 percent are in this category. In HP 57.9 percent of 15 year olds in school cannot be distinguished from not having learned any science at all and in TN 43.6 percent all in this category – ten times as many as the USA.
  2. PISA considers “level 2″ as the minimum level that provides the science competencies that will enable them to participate actively in life situations related to science and technology. Since more than 80 percent of students in both HP and TN
    are level 1 or below this most students in these states have reached age 15 ill-equipped for the century they will face.
  3. While a thin elite that competes for the few highly selective technical institutes are globally competitive, this is a tiny fraction of the population. The estimate of the fraction of TN or HP students at level 6 in science proficiency was zero. Their estimate of the fraction at level 5: also zero. Of course this does not mean there are not such students in these states, of course there are, just that from the samples available in the study the best estimate was so small as to be indistinguishable from zero.
Table 2: Comparison of science proficiency in Tamil Nadu
and Himachal Pradesh to India’s aspirations
Country/Region Below level 1 Level 1 1 Level 5 5 Level 6 6
Singapore 2.8 8.7 15.3 4.6
Hong Kong 1.4 5.2 14.2 2
Korea 1.1 5.2 10.5 1.1
OECD avg. 5 13 7.4 1.2
USA 4.2 13.9 7.9 1.3
Russia 5.5 16.5 3.9 0.4
Brazil 19.7 34.5 0.6 0
Tamil Nadu 43.6 40.9 0a 0a
Himachal Pradesh 57.9 30.9 0a 0a
Source: PISA 2009 Plus Results. Description of levels Table 3.2, percentages Table B.3.4.

1) At Level 1, students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow explicitly from given evidence.

5) At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis.

6) At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they demonstrate willingness to use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.

a) In Table B.3.4 these are reported as blank but the estimated percentages in below 1 to level 4 sum to exactly 100 percent. Obviously this not imply that there are exactly zero students in all of these two states meeting these levels but that with the sample sizes assess students of 1616 in HP and 3210 in TN there was insufficient information to create a non-zero estimate.

These results on PISA 2009+, while tragic for what they imply for Indian youth and perhaps shocking to newcomers to this subject, come as no surprise to those who have been working on basic education in India:

  • Das and Zajonc (2008) used results from Orissa and Rajasthan to create indices on mathematics performance similar to those of TIMSS (Trends in Mathematics and Science Study) and found these states near the bottom of the global rankings.
  • Educational Initiatives carried out an 18 state study using sophisticated testing instruments and found levels of performance on TIMSS comparable items that were stunningly lower. For instance on the open ended question “Write a fraction larger than 2/7″ less than 30 percent of Indian students in standard 8 could answer correctly compared to more than 70 percent internationally.
  • The APRest study led by Karthik Muralidharan and Venkatesh Sundararaman in rural AP asked the same questions of students in grades 2 to 5 and found very slow rates of learning progress.
  • The results year after year from the ASER [2010 2009] study supported by Pratham find that significant fractions of students in Standard 8 cannot master even Standard 2 curricular basics. In rural areas nationwide a third of children in grade 8 could not do a simple division problem and almost 20 percent could not read a level 2 text. The 2011 results, due out in a few weeks will show continued stagnation or even retrogress in learning.
  • Numerous studies by MIT’s JPAL, World Bank, NCAER/University of Maryland and other researchers found levels of performance that were shockingly low compared to curricular expectations.

These PISA 2009+ results are the end of the beginning. The debate is over. No one can still deny there is a deep crisis in the ability of the existing education system to produce child learning. India’s education system is undermining India’s legitimate aspirations to be at the global forefront as a prosperous economy, as a global great power, as an emulated polity, and as a fair and just society. As the beginning ends, the question now is: what is to be done?



2 comments   International Economics   , , ,  

Should We Welcome Globalization or Fear It?

By Winton Bates, on April 16th, 2010

Having just finished reading Gregg Easterbrook’s new book, ‘Sonic Boom’, I think he would say that we should welcome globalization. He sees fantastic potential for social progress, but improved living standards are likely to be ‘wrapped with ribbons of stress, anxiety and dissatisfaction (p.34). His bottom line seems to be that globalization is inevitable and that we just have to learn to live with it.

Easterbrook expects the forces of globalization to grow stronger. That means that the insecurity that people often associate with globalization is likely to accentuate:

‘Job turmoil, the economic roller-coaster, financial bedlam, media superficiality, celebrity inanity, political blather, targeted advertising, scream-and-shout discourse, the paving over of nature – they’re all going to get worse. A lot worse in some cases. Most likely, global economics will be blamed for whatever about coming decades we don’t like.’

He also suggests, however, that much of what people tend to like about life will get better:

‘Prosperity will increase, especially in the less affluent nations where improvement is most needed. Democracy will flourish on five and perhaps six continents … . Information and knowledge will proliferate as never before, while art and culture become available to everyone. Many aspects of this evolving sonic boom will be really terrific.’

Then comes the recommendation:

‘The terrific aspects and the anxiety inducing aspects will be intertwined and we’re just going to have to live with this’ (p. 209).

Why is globalization inevitable? I think Easterbrook discusses this in several places but I have noted one place in particular. (I’m glad I made notes as I read the book because there are few clues offered in the contents page about where to find stuff and the index doesn’t seem to be as helpful as it could be. But I digress!) Easterbrook suggests that we can’t stop global change because it is associated with the spread of freedom – ‘most of the world’s nations are acquiring the same core structures (democracy, free-market economics, emphasis on education) that makes the United States the current world leader … . The more America-like the world becomes, the faster the pace of economic change will be’ (p. 192).

I think Easterbrook is basically right about this. It would probably take a world war to stop globalization and there doesn’t appear to be one of those on the horizon. Perhaps some people said similar things around 1900 – prior to a few decades of disruption in global trade and investment. Even so, the main point is that the forces shaping the future of the global economy are beyond the control of any individual, firm or government. At a national level it is possible to shield some groups from the forces of global change but only by reducing the opportunities available to others.

Easterbrook acknowledges that it is possible for governments to provide a safety net that will provide citizens with some degree of security, particularly in relation to health care. He argues that people in the U.S. suffer more stress than do people in western Europe because of problems associated with the U.S. health care system (pp. 200-202). I don’t know whether or not this is a valid point. Evidence from the Gallup World Poll suggest that people in the U.S. tend to experience more stress than do people in western European countries and Australia. But Mexicans report experiencing a lot less stress than Americans and less stress than Europeans and Australians – so there is probably more involved than health care.

My main reservation about this book, as with Easterbrook’s earlier book ‘The Progress Paradox’ (discussed here), is that I think he overstates the insecurity that people actually feel as a result of the forces of globalization. The book seems to be full of colourful phrases to describe this insecurity. For example, Easterbrook writes of ‘change-based anxiety’ (p.34), ‘Multiple Media Personality Disorder’ which he defines as a ‘a universal low grade nervous tension from which there may be no realistic escape’ (p.70), ‘the Super Bowl of stress’ (p. 72) and ‘collapse anxiety’ (p. 168).

I acknowledge that job insecurity has increased. Easterbrook makes a strong case that each year it gets easier for someone to come along with a superior idea and put an established firm out of business (p. 134). He could be right that in future there will be a greater risk that people who have risen to the middle classes will ‘fall back’ down the economic ladder and end up bitterly unhappy (p. 196). I also acknowledge that the insecurity of modern life is a popular topic of conversation, particularly in the media. But I don’t think insecurity is having a large impact on behaviour and the way people feel about their lives. If a lot of employed people were feeling a high degree of insecurity about their jobs I think they we would see more precautionary saving and less willingness to go into debt than we have seen in recent years. Survey evidence suggests that the vast majority of people in high-income countries feel that they have a great deal of control over their lives.

My conclusion is that there must be a huge gap between the fears that a lot of people express when they talk at a superficial level about the challenges and insecurity of modern life and the deeper feelings that they have about opportunities and threats in their own lives.



Leave a comment   Book Reviews   , , ,  

What Does the Legatum Prosperity Index Measure?

By Winton Bates, on November 17th, 2009

Prosperity. However, my initial reaction when I first looked at the index was that it was more like an index of the quality of life or well-being than of prosperity. My perception of prosperity was too narrow. One of the purposes of the index is apparently to encourage people ‘to take a holistic view of prosperity’. The authors might have a point. Dictionary definitions of prosperity don’t focus solely on economics – they suggest that prosperous is synonymous with flourishing, successful and thriving.

The index has nine components or sub-indexes:
• Economic fundamentals
• Entrepreneurship and innovation
• Democratic institutions
• Education
• Health
• Safety and security
• Governance
• Personal freedom
• Social capital.

Is this a comprehensive list of factors affecting human flourishing? One apparent omission is environmental quality. Although this is considered to some degree in the indicators used for health, I get the impression that the authors’ view of prosperity does not place much value on the enjoyment that people obtain from the natural environment. I wonder whether that view is backed up by research findings.

The main problem I have with this report is that I am not sure how much substance lies behind it. The presentation is incredibly smooth, mainly because it is completely uninterrupted by references to research papers or data sources. The preface suggests that information on data sources is available on the prosperity web site. All I could find on that site, however, apart from the acknowledgement of input from research consultancy, Oxford Analytica, is a statement that the index draws on the Gallup World Poll and other data and includes only factors for which a statistical link with material wealth or life satisfaction can be shown.

The Legatum Institute claims to be a think tank. It seems to me that no organisation claiming to be a think tank should publish research findings without making sure that the underlying research is open to public scrutiny.

However, leaving aside my doubts, this report is worth looking at just to view the chart showing that countries with high prosperity tend to have high performance on all sub-indexes and those with low prosperity have low performance on nearly all sub-indexes.



2 comments   Opinion   , , ,