Tag: influence

The Impact of Economics Blogs

David McKenzie (World Bank, BREAD, CEPR and IZA) and Berk Özler (World Bank). Policy Research Working Paper 5783. August 2011. Available as pdf. See also the authors’ blog about this paper.

Introduction: Practically nonexistent a decade ago, blogs by economic scholars have become commonplace. Economics blogs, such as Freakonomics, Marginal Revolution, Paul Krugman and Greg Mankiw, have built large followings – whether measured by subscriptions in Google Reader or by average daily page views (1). Cowen (2008) argues that blogs are the main way that the general public consumes economics in a given day and guesstimates that “…about 400,000 people are reading economics blogs and digesting them” on a daily basis.

These blogs not only give their creators an outlet to disseminate their ideas and work immediately in a format that is more accessible, but also enable instant feedback, are easy to share on the open web, and allow the bloggers a personal style rather than the inaccessible format of academic journals (Glenn, 2003; Dunleavy and Gilson 2011).

Our motivation in examining the impact of economics blogs stems from two observations about blogs and questions that arise from these. First, it seems fair to state that “…informing is the core business of blogging.” (McKenna and Pole 2008, p. 102) This leads to the question of whether blogs improve the dissemination of research findings and whether their readers are indeed more informed (2). On the one hand, coupling the large readership of blogs with the argument of Cowen (2008) that the best ones are written at a level far higher than that of any major newspapers offers the promise that economics blogs may have sizeable effects on the dissemination of economic research and on the knowledge and attitudes of their readers.
Continue reading “The Impact of Economics Blogs”

Assessing the impact of blogs: Some evidence and analysis

See

The Impact of Economic Blogs – Part I: Dissemination by David McKenzie, Berk Özler, 2011-08-05.

  • Question 1: “Do blogs lead to increased dissemination of research papers?””
  • Answer:  “Blogging about a paper causes a large increase in the number of abstract views and downloads in the same month. These increases are massive compared to the typical abstract views and downloads these papers get. However, only a minority of readers click through the blog to the download.” [view paper by McKenzie for more details]

The Impact of Blogs Part II: Blogging enhances the blogger’s reputation. But, does it influence policy? by David McKenzie, Berk Özler, 2011-08-10

  • Question 2: Does blogging improve reputation?
  • Answer: “Regular blogging is strongly and significantly associated with being more likely to be viewed as a favorite economist.”
  • Question 3: Does blogging influence policy?
  • Answer1: This is where we haven’t been able to find much evidence to date [see blog for details of some case examples]
  • Answer2: In response to a case example provided by a reader: “my sense is that:
    i) very few posts actually influence policy
    ii) there are very few readers of blogs who are actually in a position to influence policy, but iii) it only takes one post read by the right reader to potentially make a big difference. This poses enormous problems for statistical inference, since these are likely rare events, but I think it is still useful to see whether there are in fact any plausible candidates.”

The Impact of Blogs Part III: Results from a new survey and an experiment! by David McKenzie, Berk Özler, COMING ON 2011-08-15

  • Including these headings: Survey evidence – why don’t you just ask blog readers?; The Experiment; Impacts on institutional reputation; Impacts on knowledge and attitudes.
  • The Summary:“Using a variety of data sources and empirical techniques, we feel we have provided quantitative evidence that economic blogs are doing more than just providing a new source of procrastination for writers and readers. To our knowledge, these findings are the first quantitative evidence to show that blogs are having some impacts. There are large impacts on dissemination of research; significant benefits in terms of the bloggers becoming better known and more respected within the profession; positive spillover effects for the bloggers’ institutions; and some evidence from our experiment that they may influence attitudes and knowledge among their readers. Blogs potentially have many impacts, and we are only measuring some of them, but the evidence we have suggests economics blogs are playing an important role in the profession.”

RD Comment: Two comments of note towards the end of the paper:

  • “…Table 6 shows that blog readership has not changed many of these attitudes towards methodology, with no significant experimental changes in the full sample. Amongst the subsamples, the most significant change occurs in the male sample, where there is an increase in the proportion that believe that it is difficult to succeed as a development economist on the job market without having a randomized experiment.”
  • “There is also some evidence among the research-focused subsample that more agree with the statement that external validity is no more of a concern in experiments than in most non-experimental studies (something discussed in David’s favorite rant).”
  • RD comment: This may be true, but experimental studies are often held up as being of more value than non-experimental studies. So the lack of difference is a problem, not a non-issue

 

Australasian Evaluation Society 2011 International Conference: Evaluation and Influence

 

Date: 29 August – 2 September (workshops on 29-30th)
Venue: Hilton, Sydney, NSW, Australia

View the Conference website here

View the detailed outline of the program and please click here to view the detailed Pre Conference Workshop Program. Please note the Conference Program is subject to change.

Read more about the Keynote Speakers, their presentations and Conference main streams and ‘hot topics’.

Evaluation and influence

Evaluation claims to influence public policy, professional practice and the management of organisations. What is the nature and extent of this influence? How can evaluations be made more influential?  And conversely, in a rapidly changing world, what are the main influences on evaluation? To what extent is evaluation responding by taking on new approaches and technologies?

With the focus on influence, the conference builds upon the two previous conferences with their themes of evidence (Canberra 2009) and reflections on evaluation (Wellington 2010).

The conference will focus on three sub-themes:

The influence of evaluation on society

How much and in what ways does evaluation impact upon policy, practice and organisations? Where and in what circumstances does it have the most impact, and why? What are other important sources of influence, and how do they compare with evaluation?

Making an evaluation more influential

How can an evaluation be designed and conducted to increase its use and influence?  What are the most persuasive ways of communicating the results of an evaluation?What role can evaluators play in implementing evaluation results? What are the lessons for evaluation from theories of influence and diffusion?

Influences shaping evaluation

How is evaluation changing in response to emerging social, economic and political issues, to increasing complexity and uncertainty, and to new approaches and technologies? What are the important influences on evaluation, and how are they shaping evaluation?

The conference can explore its theme in streams around fields such as education and research, health, human services, justice, international development, Indigenous peoples, natural resource management and the economy. We also expect a stream on design and methodology.

The Elusive Craft of Evaluating Advocacy

Original paper by Steven Teles, Department of Political Science, Johns Hopkins University, and Mark Schmitt, Roosevelt Institute. Published with support provided by The William and Flora Hewlett Foundation. Found courtesy of @alb202

A version of this paper was published in the Stanford Social Innovation Review  in May 2011 and is available as a pdf

“The political process is chaotic and often takes years to unfold, making it difficult to use traditional measures to evaluate the effectiveness of advocacy organizations. There are, however, unconventional methods one can use to evaluate advocacy organizations and make strategic investments in that arena”

Sound expectations: from impact evaluations to policy change

3ie Working paper # 12, 2011, by Center for the Implementation of Public Policies Promoting Equity and Growth (CIPPEC) Emails: vweyrauch@cippec.org, gdiazlangou@cippec.org

Abstract

“This paper outlines a comprehensive and flexible analytical conceptual framework to be used in the production of a case study series. The cases are expected to identify factors that help or hinder rigorous impact evaluations (IEs) from influenc ing policy and improving policy effectiveness. This framework has been developed to be adaptable to the reality of developing countries. It is aimed as an analytical-methodological tool which should enable researchers in producing case studies which identify factors that affect and explain impact evaluations’ policy influence potential. The approach should also enable comparison between cases and regions to draw lessons that are relevant beyond the cases themselves.

There are two different , though interconnected, issues that must be dealt with while discussing the policy influence of impact evaluations. The first issue has to do with the type of policy influence pursued and, aligned with this, the determination of the accomplishment (or not) of the intended influence. In this paper, we first introduce the discussion regarding the different types of policy influence objectives that impact evaluations usually pursue, which will ultimately help determine whether policy influence was indeed achieved. This discussion is mainly centered around whether an impact evaluation has had impact on policy. The second issue is related to the identification of the factors and forces that mediate the policy influence efforts and is focused on why the influence was achieved or not. We have identified and systematized the mediating factors and forces, and we approach them in this paper from the demand and supply perspective, considering as well, the intersection between these two.

The paper concludes that, ultimately, the fulfillment of policy change based on the results of impact evaluations is determined by the interplay of the policy influenc e objectives with the factors that affect the supply and demand of research in the policymaking process.

The paper is divided in four sections. A brief introduction is followed by an analysis of policy influence as an objective of research, specifically, impact evaluations. The third section identifies factors and forces that enhance or undermine influence in public policy decision making. The research ends up pointing out the importance of measuring policy influence and enumerates a series of challenges that have to be further assessed.”

A guide to monitoring and evaluating policy influence

ODI Background Notes, February 2011. 12 pages
Authors: Harry Jones
“This paper provides an overview of approaches to monitoring and evaluating policy influence and is intended as a guide, outlining challenges and approaches and suggested further reading.”

“Summary: Influencing policy is a central part of much international development work. Donor agencies, for example, must engage in policy dialogue if they channel funds through budget support, to try to ensure that their money is well-spent. Civil society organisations are moving from service delivery to advocacy in order to secure more sustainable, widespread change. And there is an increasing recognition that researchers need to engage with policy-makers if their work is to have wider public value.

Monitoring and evaluation (M&E), a central tool to manage interventions, improve practice and ensure accountability, is highly challenging in these contexts. Policy change is a highly complex process shaped by a multitude of interacting forces and actors. ‘Outright success’, in terms of achieving specific, hoped-for changes is rare, and the work that does influence policy is often unique and rarely repeated or replicated, with many incentives working against the sharing of ‘good practice’.

This paper provides an overview of approaches to monitoring and evaluating policy influence, based on an exploratory review of the literature and selected interviews with expert informants, as well as ongoing discussions and advisory projects for policy-makers and practitioners who also face the challenges of monitoring and evaluation. There are a number of lessons that can be learned, and tools that can be used, that provide workable solutions to these challenges. While there is a vast breadth of activities that aim to influence policy, and a great deal of variety in theory and practice according to each different area or type of organisation, there are also some clear similarities and common lessons.

Rather than providing a systematic review of practice, this paper is intended as a guide to the topic, outlining different challenges and approaches, with some suggestions for further reading.”

%d bloggers like this: