1. Reality is a complicated nonlinear relationship in many dimensions. Each randomised
   experiment illuminates the gradient vector in one small region. It’s hard to generalise the results (i.e. low external validity).
2. I am quite worried about the bang for the buck obtained through this strategy. A lot of money is spent, which could
   have other uses in funding dataset creation or research.
3. Economics is a bad field in having low standards of replication. The journals don’t publish replication, which is the
   foundation of science. Randomised experiments, too often, generate proprietary datasets which are controlled by the original
   authors. The scientific progress which comes about from multiple scholars working on common datasets does not come about easily.

Jim Manzi has a great article on the difficulties of obtaining knowledge about social science questions. He tells the story of a field –
Criminology — which experienced the Randomised Experiment Revolution in the 1980s:

In 1981 and 1982, Lawrence Sherman, a respected criminology professor at the University of Cambridge, randomly assigned one of three responses to Minneapolis cops responding to misdemeanor domestic-violence incidents: they were required to arrest the assailant, to provide advice to both parties, or to send the assailant away for eight hours. The experiment showed a statistically significant lower rate of repeat calls for domestic violence for the
mandatory-arrest group. The media and many politicians seized upon what seemed like a triumph for scientific knowledge, and mandatory arrest for domestic violence rapidly became a widespread practice in many large jurisdictions in the United States.

But sophisticated experimentalists understood that because of the issue’s high causal density, there would be hidden conditionals to the simple rule that `mandatory-arrest policies will reduce domestic violence.’ The only way to unearth these conditionals was to conduct replications of the original experiment under a variety of conditions. Indeed, Sherman’s own analysis of the Minnesota study called for such replications. So researchers replicated the RFT six
times in cities across the country. In three of those studies, the test groups exposed to the mandatory-arrest policy again experienced a lower rate of rearrest than the control groups did. But in the other three, the test groups had a higher rearrest rate.

…

Criminologists at the University of Cambridge have done the yeoman work of cataloging all 122 known criminology RFTs with at least 100 test subjects executed between 1957 and 2004. By my count, about 20 percent of these demonstrated positive results: that is, a statistically significant reduction in crime for the test group versus
the control group. That may sound reasonably encouraging at first. But only four of the programs that showed encouraging results in the initial RFT were then formally replicated by independent research groups. All failed to show consistent positive results.

I am all for more quasi-experimental econometrics applied to large datasets, to tease out better knowledge by exploiting natural experiments. By using large panel datasets, with treatments spread across space and time, I feel we gain greater external validity. And, there is very high bang for the buck in putting resources into creating large datasets which are used by the entire research community, with a framework of replication and competition between multiple researchers working on the same dataset.

You might like to see a column in the Financial Express which I wrote a few months ago, with the story of an
interesting randomised experiment. In this case, there were two difficulties which made me concerned. First, this was not randomised allocation to treatment/control: there was selectivity. Second, it struck me as very poor bang for the buck. Very large sums of money were spent, and I can think of myriad ways to spend that money on datasets or research in Indian economics which would yield more knowledge.

I think the austerity debate is unhelpful. There are complicated trade-offs associated with government spending. If the question is framed as “more” or “less”, reasonable people will disagree about costs and benefits that can’t be measured. Even in a depression, cutting expenditures to entrenched interests that make poor use of real resources can be beneficial. Even in a boom, high value public goods can be worth their cost in whatever private activity is crowded out to purchase them. Rather than focusing on “how much to spend”, we should be thinking about “what to do”. My views skew activist. I think there are lots of things government can and should do that would be fantastic. A “jobs bill”, however, or “stimulus” in the abstract, are not among them. If we do smart things, we will do well. If we do stupid things, or if we hope for markets to figure things out while nothing much gets done, the world will unravel beneath us. We have intellectual work to do that goes beyond choosing a deficit level. The austerity/stimulus debate is make-work for the chattering classes. It’s conspicuous cogitation that avoids the hard, simple questions. What, precisely, should we do that we are not yet doing? What are the things we do now that we should stop doing? And how can we make those changes without undermining the deep social infrastructure of our society, resources like legitimacy, fairness, and trust?

–

Elsewhere, in the world of academia, I also noted this piece by Mark Bauerlein, Mohamed Gad-el-Hak, Wayne Grody, Bill McKelvey, and Stanley W. Trimble in the Chronicle of Higher Education on the avalanche of poor research. The authors point towards a growing problem of sub-par research in general pointing to, as far as I can see, three things. First, that the growing amount of poor research is a strain on the system of peer-reviewed work (too many articles to review by too few able reviewers); secondly, that the pressure to produce in academic circles leads to quantity over quality and thirdly that the increasing tendency of money to flow to the amount of publications by default exacerbates the problem.

While brilliant and progressive research continues apace here and there, the amount of redundant, inconsequential, and outright poor research has swelled in recent decades, filling countless pages in journals and monographs. Consider this tally from Science two decades ago: Only 45 percent of the articles published in the 4,500 top scientific journals were cited within the first five years after publication. In recent years, the figure seems to have dropped further. In a 2009 article in Online Information Review, Péter Jacsó found that 40.6 percent of the articles published in the top science and social-science journals (the figures do not include the humanities) were cited in the period 2002 to 2006.

(…)

Our suggestions would change evaluation practices in committee rooms, editorial offices, and library purchasing meetings. Hiring committees would favor candidates with high citation scores, not bulky publications. Libraries would drop journals that don’t register impact. Journals would change practices so that the materials they publish would make meaningful contributions and have the needed, detailed backup available online. Finally, researchers themselves would devote more attention to fewer and better papers actually published, and more journals might be more discriminating.

In the context of the world of academic economics which I am accustomed to I can see most of the issues the authors point. Especially, I would point towards the pressure to produce which is extensive in the context of economics. However, I am not sure about the point that a large bulk of research is bad because it, in itself, takes a lot of time to digest. I like to think that a study which might not be deemed relevant today may find its day in the sun in the future if the consensus and discourse changes.

–

Economist Kartik Athreya from the Richmond Fed (Virginia) is not too fond about econbloggers voicing their opinions on macroeconomic because, as he says, it is a topic much too complicated for econbloggers to understand (the original link to the essay is gone, but FT Alphaville and Scott Sumner provide good coverage and quotes). Now, I don’t even know where to begin here but as both an econblogger and a semi-academic economist I naturally ought to be able to muster some opinion. But really, where do you start here? Well, I especially noted this;

So far, I’ve claimed something a bit obnoxious-sounding: that writers who have not taken a year of PhD coursework in a decent economics department (and passed their PhD qualifying exams), cannot meaningfully advance the discussion on economic policy. Taken literally, I am almost certainly wrong. Some of them have great ideas, for sure. But this is irrelevant. The real issue is that there is extremely low likelihood that the speculations of the untrained, on a topic almost pathologically riddled by dynamic considerations and feedback effects, will offer anything new. Moreover, there is a substantial likelihood that it will instead offer something incoherent or misleading.

Let me be very, very clear here. The ability to solve dynamic optimization problems, to solve complex differential equations, to derive, on paper, various statistical estimators do not make a good economist. You do all this in order to become a part of the initiated crowd and in order to speak a language which dazzles colleagues and the greater public by its complexity and, crucially, is the main reason why economists today still form a gated community. This is natural since it takes half a mathematics degree to say anything which your fellow colleagues will accept as a real economic argument.

But I digress (and rant too). Math is not the problem as such but a symptom of some of the problems with modern economics. In general though, Math makes you smart and helps to build rigorous arguments which helps in any scientific context. As such, I will reciprocate Mr. Athreya’s point; just as the econbloggers are not stupid neither are academic economists (they are devilshy smart for the most part). Yet, the latter have remained stuck too long and too far up the ivory tower to see that the econbloggers are not leeches who prey on the public through simplification of a complex topic, but in fact helps to bring an otherwise unworldly macroeconomic discourse down to earth.

We as economists should encourage this, not move further up the ivory tower.

University, Inc.: The Corporate Corruption of American Higher Education is a necessary book but also a sobering, even depressing one. If you’re concerned about your health, your education or your country, you won’t enjoy what you read in these pages…but you should read University, Inc.

Science is the reason. While numerous theorists since World War II have worked to theorize and complicate the process of knowledge creation in science—Thomas Kuhn’s work comes to mind immediately—the ideal of science as a disinterested arbiter of truth remains. In this vision of what science should be, truth matters. We trust scientific reports because of this aura of dedication, even purity. In these pages Jennifer Washburn documents just how tainted science has become.

Its transformation has not come through malicious intent. Indeed, as Washburn explains, at many points throughout this transformation, science and education have been changed with the best of intentions. As Washburn points out in her historical overview of higher education articulated in the book’s second chapter, scientific research in this country has long had two fairly distinct mandates. One was to pursue pure, abstract research. The other was to engage in practical research, especially applied research that would help the people in a university’s region, as has long been the case with the nation’s agricultural colleges.

However, in progressive steps since World War II—and especially since the 1970s when economic downturns led to a general increase in federal investment in science as a way to spark economic recovery—academic research has shifted not just to the applied and practical but also to a market model. What this means, Washburn explains, is a number of things. First, following the passage of the Bayh-Dole Act in 1980, rather than new ideas flowing freely from universities outward to the community, universities have emphasized research that can be patented—and have retained the patents on these discoveries. This means that universities shift to what Washburn calls “Market-Model U.” In itself, this is not all bad. As someone who has taught in state colleges, I can attest to the waste found in some systems and the inefficiency and the dated slowness with which things are executed. If it were possible to infuse just the efficiency and pace of the finest private enterprises, this would be a good thing.

It hasn’t been possible to do only that, though, Washburn argues. Instead, as federal funding for higher education dried up, universities sought funding elsewhere, producing an increased emphasis on corporate funding. Washburn documents a second effect which is how this has chilled academic research. Stories of graduate students who, due to contractual obligations, can’t share their work, even with their own professors, should make readers blink, as should the anecdotes about faculty members stealing student work. Greed drives dishonesty in Washburn’s account.

This leads to a third damning trend: bias. Any bias in a scientific study should be disturbing, but Chapter 5, in which Washburn documents the systematic distortion of medical studies, should make readers more than a little queasy. These distortions range from the relatively benign (reporting a test drug has a positive effect when it has little or none) to the literally deadly. Deaths have been covered up, as have risks of death, so that drugs can be brought to market for profit. The conclusion is clear: whatever the ideal of science once was, it is not strong enough to stand up to temptation. Self-interest rules.

To be frank, the later chapters are not as strong as these early ones exposing abuses. The attempts to shape new Silicon Valleys and revive regions may fail, but their intentions are good and not necessarily driven by greed (except for prestige and regional pride). Washburn’s discussion of academic employment patterns is not bad, but it is insufficient. What’s more, that she really isn’t that concerned about these is witnessed by the final chapter: all the suggestions for reform focus on intellectual property and clinical research. The attempts to change the nature of education through the use of superstar education may repel some, but it needs to be put into a much larger and more detailed context. It seems more a symptom of the Information Age and what podcasts and computers offer than of the market model per se. How, for example, does MIT’s choice to put its courses online for free fit with this? In other words, the changes detailed in this book are part of a wider redefinition of the ownership of knowledge, a shift linked to everything from music downloads to Wikipedia, and needs to be put in that context.

While my objections matter, and I wish the second half of the book were better (richer, fuller, more inclusive and complex), the expose that fills University, Inc.’s chapters on science and medicine struck fear in my heart, and I suspect they’ll do the same for other readers.

Join the forum discussion on this post - (1) Posts