Last week African Arguments published a story that Prof Morten Jerven, author of ‘Poor Numbers: How We Are Misled by African Development Statistics and What to Do about It’, had been blocked from presenting his research on African statistical capacity at the UNECA. This was due to opposition to his ideas from, notably, the South African Statistician General, Pali Lehohla. The speech Jerven intended to give to the UNECA can be read here. Morten Jerven responds below.

Discussing economic statistics and GDP estimates of African economies is clearly important, but it’s also sensitive. Pali Lehohla and his self-proclaimed union of ‘African Statisticians’ are allied in a self-defeating campaign.

My book Poor Numbers has created an unprecedented argument for investing in the statistical capacity of African countries. Why would Lehohla and his silent supporters go against this? The answer is simple. Pali Lehohla and his counterparts are doing well in the current system. Any change to the status quo in the political economy of statistics in Africa is considered a threat.

The allegations that I am a ‘hired gun’ or ‘that I have not done my research’ are of course ridiculous and entirely false. With Lehohla putting his emphasis on “stopping Jerven in his tracks” before he “hijacks the African statistical agenda” the immediate danger is that good initiatives will be suspended and cancelled. In the long term, statistical offices in the region may struggle for survival.

Any observer of this debate is likely to draw two conclusions: The quality of the statistics are probably not very good and some of the leaders seem to react very aggressively rather than doing something about it.

When Bill Gates read Poor Numbers he concluded that it was time to invest in better GDP statistics. If he or other stakeholders read this particular debate, I don’t think their next logical step is going to be – ‘And I am going to put my money on the angry guy in who is trying to censor public debate’. I fear that the public statements made by Lehola and his supporters seem to observers that statistical offices are part of the problem and not the solution.

I still think any agenda that attempts to provide the public with better statistics must engage with statistical offices and their interests. As I describe below, these are not the only conditions that make it hard to have an objective and open debate on African statistics.

When I first set out to investigate African GDP estimates in 2007, I didn’t think the work would get much of a reception. As I document in Poor Numbers, actual practitioners of national accounts were surprised that a researcher would take interest in their work. Actual data users in the countries concerned, such as Central Banks, were happy that such concerns were being raised, and the consultants were relieved to talk about some of their long-standing worries and the problems they experience.

Data disseminators, on the other hand, were usually quite reluctant to share information, and did not (sometimes understandably) always want to provide information on the methods used to produce the numbers. This was generally justified on the basis of ‘ownership’ of the data and a belief that they shouldn’t share the processes surrounding the production of the final data because of ‘confidentiality’ (which can be frustrating, or as one reviewer put it “[Jerven] relates chilling tales of how his attempts to access raw data behind international institutions’ statistics met with evasion.”

The initial response from many economists working on Africa varied between, ‘so what?…we already know this’, ‘we don’t trust or use official statistics on Africa anyhow’ and ‘I know but what is the alternative?’ Many more scholars in African studies and development studies, who were generally concerned with the long-standing use of numbers on Africa as ‘facts’, were relieved that there was finally someone who sought, not only to unveil the real state of affairs, but genuinely wanted to answer some of the problems that users face when trying to use the data to test their scholarly questions.

My original intent was never to write a book about this. Initially I was genuinely puzzled by some of the discrepancies in GDP data on Africa and, after some deliberation, decided that I should try to systematically collect information on how some African economies are being measured. I conducted the research between 2007 and 2010, and wrote up the manuscript in 2011.

As is documented in the book, my research methodology has been to combine archival work and secondary literature with broad surveys and in-depth interviews. As I document in the book, there is simply no way you can just walk in the door and demand: ‘How poor are your numbers?’ Moreover, one of the key problems is a complete lack of documentation of methods and sources – theoretical manuals do not always match up with the practical process at the office. It has however been possible to collect the information needed by using a combination of methods – evaluating documents like an historian, reading numbers like an economist and collecting metadata through personal observations, triangulation and conversations (like a good journalist would.) In the introduction of Poor Numbers I describe these methods as an attempt to do a ‘political ethnography’ of African economic statistics.

I had an invitation and introduction to all the offices I visited. Date and place of interviews at statistical offices, central banks and donor missions are documented. Between 2007 and 2011 I wrote letters, emails and phoned all statistical offices in Sub-Saharan Africa repeatedly in order to verify information, request access and set up interviews. As anyone who has tried something similar can attest, the response rate is extremely low. For all the places I did go to I had a response, a contact and an invitation.[i]

Upon arrival at all these places I went through the dissemination office to clarify my purpose and research. At all those offices I also requested an interview with Directors and senior management and in every case these requests were ignored. Moreover, as anyone who has done a similar project can tell you, you cannot access these places without an invitation and a name. When permission to enter is given your name is duly recorded in the big book that sits at the front desk of official buildings. I have never needed to sneak in through any backdoor. I was welcomed in at the main gate.

When Pali Lehohla claims ‘that I have not done my research’ it is not only completely wrong, it also stands in striking contrast to the position of the Director in Zambia, who penned a lengthy reaction to Poor Numbers making exactly the opposite argument. His problem is that I have done my research and that I have done it well. I am sure that in retrospect the Director wishes that he had paid closer attention to national account statistics in Zambia so that he might have noticed or answered my responses.

Note that neither Pali Lehohla, nor the Zambian Director, nor any of the other reports that replicated the study I conducted in Poor Numbers, has found any cause for disagreeing with the diagnosis. To understand how African statistical institutions find themselves in the current situation we need to understand the interplay between states, donors and consultants – and how the demand and funding for data affects the quality of data supplied.

From Poor Numbers to statistical tragedy and damned lies

The potentially controversial content of the book and its material became clear to me when I presented part of the work in South Africa in 2011. I documented the upward revisions in Ghana and elsewhere and showed the very uneven application of methods and data in the African region. In response to the talk, Shanta Devarajan, Chief Economist for Africa at the World Bank, declared Africa’s ‘Statistical Tragedy’. It was a bold statement, it was not entirely accurate, but I think it was instrumental in bringing the debate forward.

This declaration, together with the news stories emerging about GDP revisions in Ghana, and forthcoming revision in Nigeria, made it abundantly clear that not only was the knowledge problem bigger than many had thought, these numbers also really matter.

In 2012, I was asked by African Arguments to write a summary of my argument, and explain in layman terms how one country, like Ghana could go ‘from being one of the poorest countries in the world one day to an aspiring middle-income one the next’. My intent with the piece was to demystify the process and to lay bare the basic discrepancies between global standards of measurement and local challenges of availability of data and resources.

That worked to some extent. However, when the Guardian reprinted the story, they smacked the headline: Lies, damn lies and GDP on it. As anyone who has read that piece or my book will know, I go to some lengths to dispel the beliefs that there was a hidden political agenda behind this revision. Indeed, one of the things that has struck me in the course of discussing my book during the past year is how data users manage to maintain an inherent suspicion of any ‘official number’. Meanwhile, critical skills often seem to fail when it comes to thinking about the basic problems of converting a complex reality into simple numbers. A similar misguided gut reaction exists among scholars, who may never trust a number from Sudan, Ghana or South Africa, but would not hesitate to use the same number if the World Bank had recycled it.

The story travelled across to France, where an interview with me was published with the headline ‘Le-Grand-Mensonge’. At this stage it is perhaps not surprising that some actors started to respond to this as a publicity problem. As I just stated, one problem of providing statistics is that you always have to defend your numbers in public – otherwise your institution might suffer a serious credibility problem.

In the case of the African Development Bank, I think some individuals were getting concerned that if too many questions were asked surrounding the accuracy of the GDP numbers, then this might negatively affect the decisions of investors. And rightly so – during this period I got frequent calls from investment banks who wanted to know how big the Nigeria’s GDP really was. In response to the review of Poor Numbers in the Financial Times, Mthuli Ncube, Chief Economist and Vice President, African Development Bank Group, wrote to the newspaper that Africa’s rise was real, and despite the uncertainty around the numbers “For investors, for visitors, for Africans themselves, seeing is believing. The growth is tangible. Come and see for yourself.”

I am somewhat sympathetic to his position. However, the reason we produce statistics is in order to be able to make informed decisions about the magnitude and pace of economic growth. Without these statistics it is impossible to properly analyse the distribution of income or the growth effect on poverty.

It is not often that a book gets mention in the reports of international organizations. Poor Numbers did not only make its way into the references, but the IMF’s Regional Outlook Reports for Africa team, as well as UNECA and AfDB, actually carried out replication studies of my information on the comparable statistics in African countries. The AfDB report spent most of its time responding to the stories in the media, but also found fundamentally the same patterns when it comes to painting a picture of the current situation in provision of economic statistics.

The survey work done for Poor Numbers was undertaken in June 2011, and double checked in November 2011. I spend 14 pages (pp. 123-137) detailing the provenance of each and every observation in my table in Chapter 1. It is remarkable to read the reports in the IMF and AfDB (published in May and June 2013 respectively) and see the differences in information reported. The AfDB has only managed to report information on base years in 34 countries (in the IMF report there is information on 45 countries) but the information in the IMF table and the AfDB report does not cohere.

It is manifest to the knowledge problem on African development statistics that such key disseminators of information cannot agree upon the simple facts in the metadata on national accounts in Africa. However, these reports, Poor Numbers, and even the otherwise so wildly defensive, incorrect and paranoid statements from the directors of statistics in Zambia and South Africa, do not actually at any point disagree with the diagnosis. Their concern is who has delivered it and what implications it will have for their own personal future.

The politics of investing in statistical capacity

When Bill Gates wrote a review of Poor Numbers he said: “[I]t is clear to me that we need to devote greater resources to getting basic GDP numbers right. As Jerven argues, national statistics offices across Africa need more support so that they can obtain and report timelier and more accurate data.” This should be wonderful news, but the road ahead is not that easy. It requires more than simply increased resources – the main problem is the incentives and the political economy surrounding the provision of statistics.

So while Poor Numbers launches the idea of improving the current situation, it does point out that it is not only a question of funding – it is about incentives and global governance of the demand for data. There is a lot of diverging interest among international organizations. The World Bank wants more credit for its new lending programme STATCAP. Paris21 advocates National Statistical Development Strategies. The African Development Bank wants to find funding for a large round of economic censuses, while UNECA wants to oversee the implementation of the newest statistical methods for calculating GDP. These are just a few of the programs out there, and these organizations have sought to engage Poor Numbers to their benefit. Quite naturally, gaining support for ‘their’ program or ‘agenda’ sometimes comes before actually getting the job done: improving measurements.

If you thought that there was a data revolution taking place, with the world united towards achieving a better measurement of development, then I am afraid you are misguided. Lack of firm facts leaves convenient room for negotiation of the numbers when it is needed. Currently the data, if they are available, are not timely or of the quality needed, and hamper any serious agenda for economic governance or development planning. As any reader of Poor Numbers will know, the big story in the book is governance by ignorance.

As I have pointed out, this stands in striking contrast to the demand for data in the development community. The most extreme version of ‘evidence based policy’ comes from those who suggest that we tie financial rewards directly to statistical evidence of success. The trend is that donors are increasingly demanding monitoring and data in return for funding. In this situation, donors can ask states to provide the data for them – with the huge gaps in data combined with blatant incentives to distort it, the average outcome will be ‘policy driven evidence’ rather than ‘evidence based policy’. Alternatively, donors disengage completely from statistical offices and fund their own data collection, evaluation and dissemination. There are marked moves in this latter direction. For example, the USAID funded Demographic Health Surveys are increasingly being viewed as the ‘gold standard’ and the data is increasingly used by economists.

Conclusion

I am concerned about this trend. I worry that while we demand evidence for policy, we forego the opportunity to invest in accountability. I think it is a mistake to think of data as a technocratic search for facts – it has to be viewed as an exercise in building institutions. For all this talk of ‘institutions matter’ and ‘governance’ in development circles, there has been a surprising gap in analysing the statistical office.

We need to rethink the demand for data and how we invest in data in Africa and beyond. My focus has been on Africa because the problem is particularly striking there. To fix the gaps we should first re-think the MDG and other donor agendas for data and do a cost benefit analysis – what are the costs of providing these data and what is the opportunity cost of providing these data? The opportunity cost is often ignored. Local demand for data needs to come into focus. A statistical office is only sustainable if it serves local needs for information. Statistics is a public good, and we need a good open debate on how to supply them.

It is in this light that I am so concerned about the reaction of Pali Lehohla and his silent (or not-so-silent) allies. Ultimately, this may undermine statistical offices in the region. What I am arguing is precisely that we need to strengthen the statistical offices and connect them closer to local needs for information. When one flies in a team of consultants that hire local staff to fill in the questionnaire and the only lasting legacy of the survey is the dataset. At that institution the staff will be waiting for the next payday in the form of a new survey. Maybe Lehohla and other statisticians are worried that I am standing between them and another payday. That is exactly what the Director of Statistics in Zambia is accusing me of in his public letter – he worries that I am trying to distort per diems provisions from him and his colleagues towards foreign consultants.

I can assure him and others that I have no such agenda. My ‘agenda’ has been to draw attention to the unevenness in statistical capacity, and open up a discussion about how to invest in local statistical offices in a way that is suited to local needs – designing incentives and institutions in a manner that matches local institutions. Unfortunately, the recent events may bring support to those who are pessimistic about the possibility of bringing transparency and real reforms to bring better data for development in the future. I accepted the invitation to present my work at PARIS21 in May and at UNECA in September as a part of the public service a scholar can contribute to the exchange of ideas. It is frustrating when other agendas stand in the way of such an exchange. In the meantime, I take some comfort from the many expressions of support that have been relayed to me. I hope, when the dust settles, that we are ready for a free and open debate where all parties feel able to take part.

Morten Jerven is Associate Professor at the Simon Fraser University, School for International Studies. His book Poor Numbers: how we are misled by African development statistics and what to do about it is published by Cornell University Press.

First posted on September 26, 2013 @ AfricanArguments

[i] These countries are Ghana, Nigeria, Malawi, Uganda, Tanzania and Zambia – in Botswana I had no invitation so my study on Botswana is based on archival work. My survey had responses from Burundi, Cameroon, Cape Verde, Guinea, Lesotho, Mali, Mauritania, Mauritius, Morocco, Namibia, Mozambique, Niger, Senegal, Seychelles and South Africa.