The Borna Disease Virus Tragedy

The Robert Koch Institute in Berlin (remember Koch’s Postulates?) earlier this year cancelled its research into Borna Disease Virus (BDV). That is really regrettable, because BDV, a neurotropic virus, may cause depression and bipolar disorder. BDV may be transmitted by blood transfusions; research on this is being carried out in Australia. There may even be an effective cure; an antiviral drug, amantadine sulfate, approved for use against the common flu.

Given the prevalence of depression and bipolar disorder, you should have thought that a decent randomized trial would have been carried out. A small trial would not cost more that a couple of hundred thousand dollars. But amantadine is a 30 year old drug, no longer covered by patents. There is practically no incentive for pharmaceutical firms to finance such a study. So far there has been none. Given the potential huge benefits and the low cost, that is a tragedy.

In the meantime, German and Austrian doctors are using amantadine and reportedly getting good results. However, we still need the solid evidence we would get from a well-designed randomized controlled trial.

For background, read this article in Discovery, a more recent paper here. See also

Not another meeting?!

The 12th set of UN climate talks since 1992 is underway in Nairobi. Maybe someone should have sent the organizers of this very large (ministers from 189 countries) and very long (2 weeks) UN meeting a copy of the paper, HOW TO MAKE BETTER FORECASTS AND DECISIONS: AVOID FACE-TO-FACE MEETINGS by J. Scott Armstrong.

His conclusion:

We rely heavily on face-to-face meetings, which are more expensive than alternative approaches, even though it is difficult to find evidence that supports their use. Although evidence-based principles exist for running face-to-face meetings effectively, they are used so rarely that we must turn to more practical solutions. In fact, a pattern of evidence suggests that prediction markets, nominal groups, and virtual teams allow for a more effective use of a group’s collective wisdom. Technology has enhanced the value of these approaches.

Microcredit and Payday Loans

Microcredit loans are popular with poor people because they are payday loans to people without paychecks.

We recently received this comment on a post on microfinance,

May I suggest a few hours at http://www.microfinancegateway.org/ might help answer some of the questions you raise. Certainly credit can lead to problems. But the misuse of anything can. The people I have read and heard who have seen the changes in the lives of people who use their microcredit loans to improve their businesses strongly favor microcredit. Perhaps if you looked more closely, you would too.
John

That site describes itself as an “online resource for the microfinance industry”. An industry site may not be the best place to look for independent evaluations. Nevertheless, this interesting article can be found on the site,

The truth is that microcredit changes poor people’s lives marginally. It is a stretch to go from the modest microcredit impacts that emerge from the little serious research we do have¹¹ to suggesting as the UN’s International Year of Microcredit website does, that road side sellers of a few bananas, used clothes, a few tea bags, or even 50 kilos of rice, are budding entrepreneurs standing at the threshold of participation in the wider economy, and who play a key role in wealth creation. It’s just not so.

Microcredit loans help smooth out fluctuations in income, maintain consumption levels during lean periods, and provide buffers against sudden emergencies. As such they are extremely useful, but much more like payday loans than venture capital investments.

We don’t really know how useful microfinance is. As is usually the case in the aid industry, there is systemic failure when it come to providing rigorous, independent evaluations of costs, benefits, and impacts.

Update: See also this post on the Becker-Posner Blog.

Evidence-Based Economics

Edmund Phelps, recent Nobel Prize winner in Economics, writes about Evidence-Based Economics,

… the supply-siders jumped to the daring conclusion that a permanent cut in tax rates on labor would encourage more work permanently – with no diminution of effectiveness.

In my view, this core tenet of supply-side economics rests on a simple blunder…

We must proceed cautiously, however. In standard analyses, the tax cut brings a reduction in government purchases of goods and services, like defense. But a tax cut could instead contract the welfare state – social assistance and social insurance, which constitute social wealth. In that case, the tax cut, while gradually increasing private wealth, would decrease social wealth. The issue is an empirical one…

Neoliberals are now telling continental Europe that tax cuts on labor can dissolve high unemployment. But the effectiveness of such tax cuts would be largely, if not wholly, transitory – especially if the welfare state was spared. In two decades’ time, high unemployment would creep back. The false hopes raised by cutting taxes would have diverted policy makers away from fundamental reforms that are necessary if the Continent is to achieve the dynamism on which high rates of innovation, abundant job creation, and world-class productivity depend.

Where are the Systematic Reviews we need?

New Scientist writes (subscription necessary to read article),

If you want to know how to preserve biodiversity, do not rely on articles in conservation journals, a new study warns.

IF YOU want to know how to preserve biodiversity, don’t rely on articles in conservation journals. So says a study which argues that conservationists should follow the medical profession’s lead, and ensure that their decisions are objectively based.

“We’re about 30 years behind the medical revolution,” says Philip Roberts of the Centre for Evidence-Based Conservation at the University of Birmingham in the UK. As a standard to aspire to, Roberts and his team took the systematic reviews that are the bedrock of evidence-based medicine. These reviews start with a carefully framed question, and typically list the search terms used to find the studies to be analysed. Strict criteria are then applied to exclude poor-quality research, and finally rigorous statistical tests on the pooled results are used as the basis of an objective guide for doctors to what treatments work best.

The study referred to is Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine.

Abstract,

The Otis Redding Problem in Conservation

Look like nothing’s gonna change
Everything still remains the same
I can’t do what ten people tell me to do
So I guess I’ll remain the same, yes

-Otis Redding, (SITTIN’ ON) THE DOCK OF THE BAY

Why do Integrated Development and Conservation Projects (IDCPs) usually fail to deliver? The problem is what Bob Sutton calls the Otis Redding Problem. Too many objectives.

If a project in a poor country is designed to save endangered species, build local NGO capacity, fight infectious diseases, develop sustainable use of forest products, promote gender equality, and develop ecotourism, it has been designed for failure.

Why the proliferation of badly designed projects where nobody is individually responsible for doing anything for any one result?

Overly complex projects work well for intermediaries and middlemen, including the people who design and manage them. They get to spend the money. And on one hand, IDCPs are attractive to donors, who can say that they are promoting a lot of good things. On the other hand, it is virtually impossible to pin failure on any one person, so nobody will ever be held accountable.

CIAO: Customer feedback, Incentives, Accountability, and, therefore, good Outcomes

CIAO is an acronym William Easterly came up with. His concern is development aid, but his comments readily apply to conservation. He criticizes top down Millennium Development Goals (MDG, link to article in The Economist) and Poverty Reduction Strategy Papers (PRSP) plans.

His recommendation, “Get rid of Utopian goals and plans: just say no to MDGs and and PRSPs”.

He calls for independent evaluations of what works, and he criticizes projects where nobody is individually responsible for doing anything for any one result. In conservation, Integrated Conservation and Development Projects usually suffer from these problems.

You can watch his presentation, and look at his slides in a different window.

Next time you look at a project proposal, think CIAO.

How to fight global poverty

Abhijit Banerjee of MIT’s Poverty Action Lab writes in an article, Making Aid Work,

Randomized trials like these—that is, trials in which the intervention is assigned randomly—are the simplest and best way of assessing the impact of a program. They mimic the procedures used in trials of new drugs, which is one situation in which, for obvious reasons, a lot of care has gone into making sure that only the interventions that really work get approved, though of course not with complete success. In many ways social programs are very much like drugs: they have the potential to transform the life prospects of people. It seems appropriate that they should be held to the same high standards.

Here are the responses to the article. In Banerjee’s response to the comments, he writes,

I should have said more about what is probably the best argument for the experimental approach: it spurs innovation by making it easy to see what works.

(With thanks for the reference to Tyler Cowan of Marginal Revolution)

Doom, Gloom and Protected Areas

In 2003, 11.5% of all land, or 18.8 million square kilometers, were protected areas (source: UNDP-WCMC).

In their comment in Nature discussed in a previous post, Michel Loreau et al. wrote,

“Although protected areas have increased slightly during the past few decades,…”

That is just not true. Protected areas have increased dramatically during the past few decades, both in number of sites and in area.

A protected area is not a magic bullet. Is money better spent, not on new protected areas, but in other ways? On making existing protected areas work? On conservation outside protected areas? Does the very institutions designed to stimulate conservation actually create incentives for biodiversity degradation? (see e.g. this post, or Rupert Gatti et al.). We urgently need to find out. That takes research, not another international panel of experts as recommended by Loreau et al.

Useful and interesting blogs

Take a look at these two blogs on evidence-based practice, Bob Sutton’s Work Matters and Tracy Allison Altman’s Evidence Soup.

Bob Sutton is a professor of management science and engineering at Stanford. He writes about Evidence Soup, “Reading Altman’s blog is like taking an ongoing course in how to make evidence-based decisions and how to take – and evaluate – evidence based actions.”

I have previously mentioned Bob Sutton as the co-author of Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management.

Read e.g. his Management Advice: Which 90% is Crap? (this is a reference to John Wanamaker’s statement that 50% of the money he spent on advertising was wasted, but he didn’t know which 50%).

Evidence-based medicine

I have previously mentioned evidence-based medicine. Here is a cover story on the subject from Business Week.

Data quality problem in systematic review

J.S. Brooks et al. DEVELOPMENT AS A CONSERVATION TOOL: EVALUATING ECOLOGICAL, ECONOMIC, ATTITUDINAL, AND BEHAVIORAL OUTCOMES: REVIEW REPORT is an excellent attempt at a systematic review. Unfortunately, there is a serious problem with lack of quality data.

Main Results
The results of this review are that (1) very few studies provide adequate quantitative measures of success across multiple outcomes to provide a strong test of the hypotheses, and (2) that two separate statistical approaches to the data indicate market selling opportunities are associated with attitudinal outcomes, and community involvement in decision making and implementation is associated with behavioral success.

Conclusions
As regards the first objective, it is clear that without far better monitoring schemes in place it is still impossible to provide a systematic evaluation of how different strategies are best suited to different conservation challenges. First, there is a paucity of high quality data. Second, few studies provide quantitative evaluations of success. Third, few studies evaluate across the full range of relevant outcomes – behavioral, attitudinal, economic and ecological…

This and other papers can be found at the Centre for Evidence-Based Conservation at the University of Birmingham, U.K.

Evidence-based interventions

Examples of systematic reviews, from the Cochrane Collaboration’s recent newsletter,

• Don’t bother with intravenous rehydration for diarrhoea – oral rehydration works just as well

In wealthy countries it is fashionable to prefer intravenous therapy (IVT) over oral rehydration therapy (ORT). A Cochrane Review published, however, shows that ORT is just as effective as IVT.

• Treating water at home is effective in preventing diarrhoea, a major cause of death in young children in developing countries

Supplying clean water to a community helps reduce gastrointestinal diseases, but interventions that kill disease-causing waterborne micro-organisms (or microbes) once it has reached the home can be even more effective. These are the conclusions of a systematic review that considered the outcome of 38 field trials involving more than 53,000 participants.

Before you can carry out systematic reviews, you must have well-designed studies to review. Then you can generalize from those studies. “38 field trials involving more than 53,000 participants”. Sigh. How many useful, well designed field trials do we have in conservation?

Reference-class forecasting

One of the cures for optimism bias, or what Flyvbjerg (see previous post) less diplomatically calls “lying”, is reference-class forecasting.

The idea is simple, see this brief article by Lovallo & Kahneman, and the paper by Flyvbjerg, Procedures for Dealing with Optimism Bias in Transport Planning: Guidance Document,

(1) Identification of a relevant reference class of past projects. The class must be broad enough to be statistically meaningful but narrow enough to be truly comparable with the specific project.

(2) Establishing a probability distribution for the selected reference class. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.

(3) Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

Where, oh where, are the studies we need?

Can we work to reduce poverty and conserve biodiversity at the same time? In a recent paper (Poverty, Development, And Biodiversity Conservation: Shooting in the Dark?) Arun Agrawal and Kent Redford write

the mass of scholarly work on the subject does not permit systematic and context-sensitive generalizations about the conditions under which it may be possible to achieve poverty alleviation and biodiversity conservation simultaneously. The vast sums channeled toward joint achievement of poverty alleviation and biodiversity conservation are all the more remarkable in light of the basic lack of evidence on the extent to which these goals can jointly be reached…

Better research design, based on careful specification of the relevant hypotheses, will likely require panel data from a suite of sites and households to allow systematic comparison across cases and regions…

…before and after studies are likely to prove invaluable in gaining a deeper understanding of the links between different measures of poverty and biodiversity.

For a relevant article, read also Jon Christensen’s Win-Win Illusions. I recommended it.

More on How do we know what works?

It is easy enough to say that we need randomized controlled trials in conservation, but how would you actually design such a study? Here is an example,

Ferraro, P.J., R. Godoy, D. Karlan, W. Leonard, and D. Wilkie. 2002. Field Experiment To Assess The Effects Of Performance Payments On Ecosystem Conservation, Quality Of Life, And Territorial Rights of Indigenous Populations: A Proposal.

How do we know what works?

How do we know what works and what doesn’t work in conservation?

In medicine, the randomized controlled trial is the gold standard. The idea is so simple that you wonder why is wasn’t widely used till after World War II. Steve Stoller of the Kaiser Foundation Research Institute explains,

The pharmaceutical industry really geared up only after WWII. Penecillin was discovered in the 1920s but only produced commercially 20 years later. Synthetic insulin was developed during the 1930s. The subsequent flood of new drug discoveries demanded a rigorous process for testing safety and efficacy.

Randomized controlled trials should become a standard methodology in conservation. The sooner the better.

Evidence-based conservation?

In medicine, the Cochrane Collaboration has pioneered evidence-based medicine through the use of randomized trials and systematic reviews. “Evidence based-medicine?” Does that mean that the practice of medicine is not totally based on evidence? Well, yes. It is a horrifying thought that the medical treatment you receive may not, in fact, be based on real evidence. But at least the medical profession is working on it.

The evidence-based approach has spread to social interventions. The Campbell Collaboration (C2) “is a non-profit organization that aims to help people make well-informed decisions about the effects of interventions in the social, behavioral and educational arenas. C2’s objectives are to prepare, maintain and disseminate systematic reviews of studies of interventions.”

In the field of management, there is a new book by two very well know business school professors, Jeffrey Pfeffer and Robert Sutton, Hard Facts, Dangerous Half-Truths and Total Nonsense: Profiting from Evidence-Based Management.

What is going on in conservation? In a recent paper (hat tip to Marcelino Fuentes) (Money for Nothing? A Call for Empirical Evaluation of Biodiversity Conservation Investments), Ferraro and Pattanayak lament the fact that the field of ecosystem protection and biodiversity conservation lack behind most other policy fields.

They write, “The field of conservation policy must adopt state-of-the-art program evaluation methods to determine what works and when. How many elephants would be poached if there had been no law banning ivory trade?”

Good question. Half-truths and slogans have done a lot of damage in Africa, for example in the militaristic approach to “anti-poaching “. Beware of any organization that wants to collect money for trucks, radios and guns to wage war on the local population in order to protect wildlife. Demand that they show how their projects will make the local “poacher” better off by conserving rather than harvesting wildlife.

There are the beginnings of the evidence-based approach in conservation. For example, the Centre for Evidence-Based Conservation at the University of Birmingham, U.K. “was established in 2003 with the goal of supporting decision making in conservation and environmental management through the production and dissemination of systematic reviews on the effectiveness of management and policy interventions.”