The Obama Administration's evidence-based social policy initiatives

Part 6: The Obama Administration’s evidence-based social

policy initiatives: An overview

Ron Haskins, Advisory Board and Jon Baron, President,

Coalition for Evidence-Based Policy

Summary

This paper outlines the Obama Administration’s plan to strengthen the evidence base for US social

policy.

The Obama Administration has created the most expansive opportunity for rigorous evidence to

inﬂuence social policy in the history of the US government.

No president or budget director for

a president have ever been so intent on using evidence to shape decisions about the funding

of social programs as President Obama and former Budget Director Orszag.

The Obama plan to

create evidence-based social policy initiatives turns the normal relationship between policy decision

making and use of social science evidence on its head. Instead of evidence being on the outside

of the decision making process trying to get in, Obama brings evidence inside from the beginning.

The Administration must still convince others that the use of evidence will improve policymaking

and program outcomes, but the argument that evidence deserves a prime role in policymaking is

being made by people inside the Administration and they are arguing to retain an evidence-based

approach as a fundamental part of the President’s legislative agenda, rather than ﬁghting from

the outside to insert evidence-based policies into the decision making process.

Although less

emphasized, the Obama plan for basing program decisions on rigorous evidence can be useful for

cutting spending as well as funding new programs.

Even as early as his inaugural address, the President made it clear that an important goal of his

Administration would be to expand programs that work and eliminate programs that don’t.

Based

on interviews with Administration ofﬁcials and advocates, it is clear that from the earliest days of

the Administration senior ofﬁcials at Ofﬁce of Management and Budget (OMB) were planning

several initiatives to advance the use of evidence-based program models and to generate high-

quality evidence on new program models. When President Obama took ofﬁce, career ofﬁcials at the

OMB, who are often the origin of ideas for increasing government efﬁciency, were already involved

in a formal attempt to encourage federal agencies to conduct high-quality evaluations of their

programs. Building on this effort, by the end of the second year of the Obama Administration there

were six evidence-based initiatives underway.

In the current age of ﬁscal austerity, cuts in many social programs are almost inevitable and

opportunities for new program spending will be limited. In this environment, it will be far better for

the national welfare if the President and Congress cut programs that have minimal or no (or even

negative) impacts rather than successful programs or programs that show promise. The nation’s

social programs are unlikely to be improved until we learn to enact programs based on evidence-

based models, to improve existing programs based on evidence, and to shut down failing programs,

again based on evidence from high-quality program evaluations. Reliable evidence on program

effects can be put to good use both in expanding and cutting programs.

But even when ineffective programs have been identiﬁed, it does not follow that the Administration

or Congress will take action. According to Isabel Sawhill and Jon Baron, since 1990 there have been

ten instances in which a large-scale federal social program was evaluated by a scientiﬁc research

design. In nine of these ten cases (including Job Corps, Upward Bound, 21

Century Community

Learning Centers, and Head Start), popular programs were shown to have modest or no impacts

on their participants.

So far, only the evaluation of Head Start has resulted in signiﬁcant program

changes, and even here the changes are only in the initial stages of surviving the Washington policy

maze and not a single Head Start program has been directly affected yet.

These examples show that the federal government needs to ﬁnd a better way to spend money

on social intervention programs. The Obama initiatives for funding social programs are the most

important attempts so far to ﬁnd this better way and could have a major impact on how social

programs are funded in the future by elevating the role of program evaluations in program

expansion or contraction. Moreover, if the initiatives work, the average impact of social intervention

programs on the well-being of children and families will increase and the nation will be better off.

Based on several interviews with members of the Obama Administration and others inside and

outside Congress knowledgeable about the Obama initiatives, we think the following outline

captures key components of the President’s evidence-based initiatives (although all the initiatives

do not follow every component of the outline):

1. Select an important social problem that would make individual citizens and the nation better

off if the problem could be successfully addressed by social policy.

2. Identify model programs addressed to the problem that have been shown by randomized

trials or other rigorous research to signiﬁcantly reduce the problem.

3. Obtain funds from Congress to scale up evidence-based programs of this type that attack

the problem in accord with the veriﬁed models.

4. Make the funds available to government or private entities with a track record of good

performance to replicate the successful model programs and to develop new model

programs.

5. Continuously evaluate the projects as they are implemented to ensure they are faithfully

implementing the model program and producing good results.

The Obama team at the OMB came into ofﬁce with strong views on the value of rigorous program

evaluation. With a team of powerful OMB ofﬁcials fully committed to the value of experimental

evaluations, the Obama Administration lost little time in launching its initiative to expand evidence-

based social programs.

What follows is an overview of the major characteristics and state of play

for the Administration’s six evidence-based initiatives. Table 1 provides an overview and comparison

of the major characteristics of the six initiatives.

• Home visiting. Home visiting is a service strategy to help families in one or more of three

domains: maternal and child health, early childhood development, and family functioning.

Several home-visiting model programs have been shown by random-assignment studies to

produce signiﬁcant impacts on a variety of parenting behaviors and, less often, child outcomes.

The impacts on mothers include reduced smoking, increased employment, and improved child

rearing practices. The Obama home-visiting initiative, for which Congress has approved $1.5

billion over the 2010-2014 period, is awarding funds in a three-stage process. In the ﬁrst stage,

which has been completed, all states were eligible for a share of funding if they submitted

proposals that met Administration requirements, primarily that they present a plan for conducting

an assessment of the need for home visiting programs in their state. Forty-nine states, the

District of Columbia, and ﬁve territories were awarded funds to enter the second stage. In the

second stage, states are required to complete their needs assessment and submit the results. The

speciﬁc requirements for the third-stage submission have not yet been published, but it seems

possible that most or all states will receive some money and that the best applications will receive

more money.

• Teen pregnancy prevention. The teen pregnancy prevention initiative has proceeded in almost

complete accord with the components of the Obama initiative outlined above. Teen pregnancy

Table 3: The Obama plan for expanding evidence-based programs

Type of

Information

Administering

Agency

Review of

Literature

Amount of

Awards

Review Panel

Selection

Selection of

Proposals

Teen Pregnancy

Prevention

Health and

Human Services

Completed by

Mathematica

$100 million

awarded –

$75 million to

replicate existing

programs, $25

million to test

new strategies

Panels included

both expert peer

reviewers and

federal staff

75 applicants

awarded grants to

replicate existing

programs; 27

grantees awarded

grants to test

new strategies

i3 (Investing in

Innovation Fund)

Education

Although there was

no formal review

of literature, the i3

evidence tiers were

based on a process

for reviewing

evidence developed

by IES over several

years primarily

through its work

on What Works

Clearinghouse and

strengthened.

Up to $650 million

across the three

types of grants

(development,

validation, scale-

up)

315 peer reviewers

were selected

from 1,400

experts in both

subject matter

and research/

evaluation;

reviewers assigned

to panels of ﬁve

people

49 applications

chosen as ‘highest-

rated.’ All secured

matching funds

(20 per cent of

grant amount)

Home Visiting

Health and

Human Services

Completed by

Mathematica

$88 million

awarded in year

one, $1.5 billion

over ﬁve years

Applications

were reviewed

by grants

management

ofﬁcials and

program staff

49 state

governments, DC,

and 5 territories

applied and were

awarded funding

for planning;

second stage of

funding to follow

Social

Innovation Fund

Corporation for

National and

Community

Service (CNCS)

None

$49.2 million

awarded

A total of 60

experts were

drawn from a

pool of experts/

professionals

and the CNCS

reviewer database

of 2,300 people;

reviewers

assigned to panels

of 2-4 people

11 grantees

chosen; must

designate

subgrantees

within six months

TAA Community

College and

Career Training

Program

Departments

of Labor and

Education

$500 million

a year for four

years

Not public

yet. Technical

review panels

will evaluate

all applications

against

evaluation

criteria provided

in application

materials

Workforce

Innovation Fund

Departments

of Labor and

Education

(Administered

through the new

$321 million

Partnership

for Workforce

Innovation)

No funded

appropriated yet.

2011 Budget

requested $321

million in the

Departments of

Education and

Labor

$108 million

Undetermined

is not only a serious national social problem with demonstrated impacts on the mother, the

father, the child, and society, it is also an area of intervention that has a long track record of

creative and diverse programs. A comprehensive review of programs by Douglas Kirby published

in 2001 found eight programs that had what Kirby called: “Strong evidence of success.”

The

Coalition for Evidence-Based Policy, by contrast, has identiﬁed only one program that meets

its criteria for reaching the ‘Top Tier’ of evidence.

There is also a comprehensive review of the

evidence published by the Campbell Collaboration in 2006 that identiﬁed successful evidence-

based programs.

Thus, the ﬁrst two components of our outline – selecting a serious problem

and ensuring that there are evidence-based model programs – have certainly been met by

the Administration in the case of teen pregnancy prevention. Similarly, the Administration

commissioned a literature review from Mathematica Policy Research that was made available

to the public. The review identiﬁed twenty-eight program models that were supported by

high-quality evidence.

Based in part on the Mathematica review, the Administration issued

its request for proposal in April 2010. The applications for funding were reviewed by a panel of

experts based on review criteria published by the Administration. Seventy-ﬁve programs were

selected for Tier 1 funding of $75 million. In addition, $25 million was awarded to 27 Tier 2

projects that have some, but not strong, evidence of success. These projects are now in various

stages of implementation.

• Investing in Innovation Fund (i3). The i3 Fund and the Social Innovation Fund (see below) are

very different from the home visiting and teen pregnancy reduction initiatives in that both fund

a more diffuse set of programs. In the case of i3, virtually any K-12 intervention with evidence of

success or promise could receive funding. Another difference is that the i3 fund recognized three

levels of evidence-based programs. The top tier of funding, called scale up funds, were awarded

for programs supported by evidence from random-assignment evaluations; validation grants were

awarded to programs with some but less evidence of success; development grants were awarded

to programs with a reasonable hypothesis but little or no evidence of success. In order to qualify

for funding, the programs had to improve outcomes for pre-school children, help students qualify

for or succeed in college, help students with disabilities or with limited-English proﬁciency, or

serve schools in rural areas. At $650 million, i3 is the second biggest of the Obama evidence-

based initiatives. Awards for 49 projects were announced in August 2010 for all three categories

of evidence-based programs.

• Social Innovation Fund (SIF). The President has said that solutions to America’s challenges:

“Are being developed every day at the grass roots.” and that his Administration wants to support

those grassroots efforts.

SIF is one method by which the Administration intends to: “Identify

and grow high-performing nonproﬁt organizations.” with experience at the local level.

SIF funds

are being awarded in a two-step process. The Administration ﬁrst allowed ‘intermediaries’ to

accept funds and then in turn to conduct competitions to determine which local organisations

should receive funding. The intermediaries were organizations that had: “Strong track records of

identifying and growing high-performing nonproﬁt organizations.”

Eleven such intermediaries

were awarded $50 million in funding to go with another $74 million they had raised in matching

funds to be distributed to nonproﬁt organizations. The nonproﬁts, which are in the process of

being selected now, will use the money to conduct evidence-based programs addressed to at

least one of three broad areas of social policy: economic opportunity, youth development and

school support, and promoting healthy lifestyles and avoiding risky behavior.

• Community College and Career Training Program. The US Department of Labor had only a

modest commitment to rigorous evaluation at the beginning of the Obama Administration. Even

so, the Administration was eventually successful in getting the Department to sign off on an

evidence-based initiative to provide funds for training of displaced and unemployed workers and

other young adults by the nation’s community colleges. On January 20, 2011 the Department of

Labor, in conjunction with the Department of Education, announced a $2 billion initiative ($500

million a year for four years) for: “The development and improvement of postsecondary programs

of two years or less that use evidence-based or innovative strategies to prepare students for

successful careers in growing and emerging industries.”

An important characteristic of the grants

is that community colleges and other entities receiving the funds are to experiment with existing

employment and training materials in order to adapt them for use with young adults who seek

employment. Given the paucity of post-secondary strategies for career preparation that have a

strong evidence base, it appears that this initiative will be one that focuses on developing new

curriculums and testing them with rigorous designs. The awards will be for between $2.5 million

and $20 million and can be used to support projects using strategies that have been shown

to have: “Strong or moderate evidence of positive impacts on education and/or employment

outcomes.” Evaluation is a central feature of the Challenge Fund: 25 per cent of the assessment

of proposals is based on the evaluation plan; all evaluations must include treatment and control

groups; and the Department of Labor will select some grantees for rigorous evaluation using

random assignment designs.

• Workforce Innovation Fund. This initiative is also being run by the Department of Labor

in conjunction with the Department of Education. Five per cent of the 2011 budgets of the

Workforce Investment Act (WIA) Adult program and the WIA Dislocated Worker program

were set aside to create this fund of nearly $108 million. The fund will be used to create

competitive grants to states or localities to replicate proven practices in training, employment,

and reemployment services, especially for vulnerable groups. Like the other evidence-based

initiatives, the fund will also be used to test promising practices. As with community college

training programs, there are relatively few program models for employment and training programs

with young adults that have been shown by rigorous designs to produce impacts on student

learning. It is anticipated that funds will be focused on ‘learn and earn,’ apprenticeship, and

on-the-job training programs. This initiative is currently on hold because Congress did not

pass any of the 2011 appropriations bills and instead enacted a ‘continuing resolution’ which

funded programs at the 2010 level and suspended all the substantive changes (with a few minor

exceptions).

In addition to these six initiatives, the Administration also included money in the President’s 2011

budget for program evaluation. Administration staffers estimate that there are enough funds in

the 2011 budget to pay for 23 rigorous evaluations of: “The most promising new programs” being

conducted by the various administrative departments. Indeed, the budget has well over $60 million

for the Department of Labor alone to: “Continue to pursue a robust, Department-wide evaluation

agenda.” including rigorous evaluations of WIA performance measures, effects of job counseling,

use of administrative data in workforce programs, incentives for dislocated workers, and effects of

Occupational Safety and Health Administration inspections.

In addition, the White House worked

with the Department of Labor to create a new Chief Evaluation Ofﬁce that will manage the new

evaluations and work with other components of the Department to assist them in conducting

rigorous evaluations of their programs.

These six evidence-based initiatives, plus the new funds for rigorous evaluation across the federal

agencies, constitute the most sweeping and potentially groundbreaking emphasis on rigorous

program evaluation ever pursued by the federal government.

Appendix 1: Coalition for Evidence-Based Policy’s mission and activities 2009-2010

Evidence-Based Reform

Key to Major Gains in Education, Poverty Reduction, Crime Prevention, and Other Areas of Social

Policy

The Coalition is a nonproﬁt, nonpartisan organization, whose mission is to increase government

effectiveness through the use of rigorous evidence about ‘what works.’ Since 2001, our work

with Congressional and Executive Branch ofﬁcials has helped advance important evidence-based

reforms, described below. A recent independent assessment of our work, conducted under our

grant agreement with the William T. Grant Foundation, found that:

“The Coalition has successfully inﬂuenced legislative language, increased funding for evidence-

based evaluations and programs … and raised the level of debate in the policy process

regarding standards of evidence. The Coalition has established a generally positive reputation

as a rigorous, responsive, honest, and impartial advocate for evidence-based approaches,

primarily at the federal level.”

Problems we seek to address

Federal social programs, set up to address important US problems, often fall short by funding

models/strategies (‘interventions’) that are not effective.

When evaluated in scientiﬁcally rigorous studies, government-funded social interventions in areas

such as K-12 education, job training, crime prevention, and poverty reduction are frequently found

to be ineffective or marginally effective. Interventions found to produce sizeable, sustained effects

on important life outcomes do exist, as discussed below, but tend to be the exception. This pattern

occurs in many diverse areas of social policy, as well as other ﬁelds where rigorous studies have

been conducted (e.g. medicine and psychology).

Why it matters

Improving social programs is critically needed. The United States has failed to make signiﬁcant

progress in key areas such as:

• Poverty reduction: We have made no overall progress in reducing US poverty over the past

several decades. The ofﬁcial rate is now 13.2 per cent – slightly higher than in 1973.

• K-12 education: We have made very limited gains in K-12 achievement since the 1970s,

according to the respected National Assessment of Educational Progress long-term trend.

• Economic mobility: We have seen no increase in the ability of youth to move up the economic

ladder relative to their parents since about 1970, according to careful studies.

The opportunity

Rigorous studies have identiﬁed a few highly-effective social interventions.

These interventions are backed by strong evidence of effectiveness – i.e., well-conducted

randomized controlled trials, carried out in typical community settings, showing sizable, sustained

effects on important life outcomes. Although rare, their very existence suggests that a concerted

effort to grow the number of proven interventions, and spur their widespread use, could

fundamentally improve the lives of millions of Americans. Illustrative examples include:

• Nurse-Family Partnership – a nurse visitation program for low-income, ﬁrst-time mothers (in

long-term studies, reduced child abuse and neglect by 50 per cent and, for the most at-risk

children, produced sizable gains in educational outcomes, such as 10 per cent higher GPA).

• Carrera Adolescent Pregnancy Prevention Program – a youth development program for low-

income teens (at age 17, reduced girls’ pregnancies by 40-50 per cent).

• Career Academies – a program for at-risk students in low-income high schools (increased

average earnings by $2,200 per year, sustained through eight years post-graduation).

• Success for All in grades K-2 – a school-wide reform program, primarily for high-poverty

schools (three years after program start, increased school-wide reading achievement in second

grade by 25-30 per cent of a grade level).

Such examples of proven effectiveness are rare in part because rigorous studies, such as well-

conducted randomized controlled trials, are still uncommon in most areas of social policy.

Meanwhile, careful investigations show that the less-rigorous studies that are typically used can

produce erroneous conclusions and lead to practices that are ineffective or harmful.

Precedent from medicine

Rigorous studies – particularly randomized control trials – have led to remarkable improvements in

human health over the past 50 years.

Well-conducted trials have stunned the medical community by overturning widely-accepted

practices, such as hormone replacement therapy for post-menopausal women (shown to increase

the risk of stroke and heart disease for many women), and stents to open clogged arteries (shown

no better than drugs for most heart patients). Such trials have also provided the conclusive

evidence of effectiveness for most of the major medical advances of the past half-century,

including vaccines for polio, measles, and hepatitis B; effective treatments for hypertension

and high cholesterol; and cancer treatments that have dramatically improved survival rates from

leukemia, Hodgkin’s disease, breast cancer, and many other cancers.

Our speciﬁc goal

Incorporate two main reforms into government social programs:

1. Increased funding for rigorous – including randomized – evaluations, so as to grow the

number of research-proven interventions.

2. Strong incentives and assistance for program grantees to adopt research-proven

interventions, and put them into widespread use.

Accomplishments

The Coalition’s work with key Executive Branch and Congressional ofﬁcials has helped inform and/

or shape major new evidence-based policy initiatives, including:

• The Ofﬁce of Management and Budget’s (OMB) Program Evaluation initiative ($100 million

in the FY11 budget for rigorous evaluations to determine what works across 17 federal agencies).

• The Department of Health and Human Services’ (HHS) Evidence-Based Home Visitation

program for at-risk families with young children ($1.5 billion over ﬁve years, enacted in 2010).

• The Education Department’s Investing in Innovation Fund, to scale up evidence-based K-12

educational strategies ($650 million enacted in the 2009 Recovery Act).

• HHS’s Evidence-Based Teen Pregnancy Prevention program ($110 million enacted for FY10).

• The Corporation for National and Community Service’s Social Innovation Fund, to support

public/private investment in evidence-based programs in low-income communities ($50 million

enacted for FY 10).

Endnotes

1. Carol Weiss has recently called this “imposed” use of research. See Weiss, C.H., Murphy-Graham, E. and Birkeland, S.

(2005) An Alternate Route to Policy Inﬂuence: How Evaluations affect D.A.R.E. ‘American Journal of Evaluation.’ 26.

(4):12-30.

2. Orszag’s take on the importance of evidence was succinctly summarized in a now-famous blog post that appeared

in the midst of developing the administration’s various evidence-based initiatives. See Orszag, P. Building Rigorous

Evidence to Drive Policy. ‘Ofﬁce of Management and Budget.’ Blog, June 8, 2009.

3. It would be possible for major players in the legislative branch on either committee with jurisdiction over particular

social programs (either the authorizing committee or the appropriations committee) to try to enact evidence-based

policies. Even the most powerful committee chairman, however, generally has less power and inﬂuence than the

president. Nonetheless, powerful Congressional players could use a set of procedures, like the one we think summarizes

the Obama evidence-based initiatives, to build their own evidence-based initiative.

4. Here is what President Obama said in his inaugural address: “The question we ask today is not whether our government

is too big or too small, but whether it works – whether it helps families ﬁnd jobs at a decent wage, care they can afford,

a retirement that is digniﬁed. Where the answer is yes, we intend to move forward. Where the answer is no, programs

will end.”

5. Funds for an initiative called the Workforce Innovation Fund, which is a mix of systems reforms of the Department of

Labor’s employment and training programs and funds for high-quality evaluations of employment and training model

programs, was in the Obama 2011 budget. However, when Congress was unable to pass the 2011 appropriations bills,

all new discretionary funding (with a few modest exceptions) was suspended. It seems doubtful that the new Congress,

with a Republican controlled House will enact any new appropriations bills for 2011. The Workforce Innovation Fund is

included in the President’s 2012 budget, but again Republican support is questionable.

6. Sawhill, I.V., and Baron, J. (2010) Federal Programs for Youth: More of the Same Won’t Work. ‘Youth Today.’ May 2010,

p.21.

7. Guernsey, L. (2010) Proposed Rules Will Shake Up Head Start. ‘Early Ed Watch.’ Blog. New America Foundation, Early

Education Initiative, September 23, 2010, available at: http://earlyed.newamerica.net/blogposts/2010/proposed_

rules_will_shake_up_head_start-37234.

8. See: www.whitehouse.gov/omb/blog/09/06/08/BuildingRigorousEvidencetoDrivePolicy

9. Stoltzfus, E. and Lynch, K.E. (2009) ‘Home Visitation for Families with Young Children.’ (R40705). Washington, DC:

Congressional Research Service.

10. Howard, K.S. and Brooks-Gunn, J. (2009) The Role of Home-Visiting Programs in Preventing Child Abuse and Neglect.

‘The Future of Children.’ 19, (2): 119-146, see especially Table 2.

11. Kirby, D. (2001) ‘Emerging Answers: Research Findings on Programs to Reduce Teen Pregnancy.’ Washington, DC:

National Campaign to Reduce Teen Pregnancy. p.179.

12. According to the Top Tier Evidence website maintained by the Coalition, the Carrera Adolescent Pregnancy Prevention

Program is: “A comprehensive youth development program for economically disadvantaged teens, a key component

of which is reproduction health care.” The program was evaluated by a large, multi-site randomized control trial. The

evaluation’s key ﬁnding was that girls experienced a 40 to 50 per cent reduction in pregnancies and births three years

after random assignment. Available at: http://toptierevidence.org/wordpress/?page_id=172

13. Scher, L., Maynard, R.A. and Stagner, M. (2006) ‘Interventions Intended to Reduce Pregnancy-Related Outcomes

among Adolescents.’ Oslo, Norway: Campbell Coalition.

14. Given the fact that the Coalition for Evidence-Based Policy, after reviewing a host of teen pregnancy prevention

programs, found only one to qualify as having ‘top tier’ evidence, it is clear that Mathematica used more ﬂexible criteria

for determining that a program model is supported by strong evidence. Under the approach to evidence being taken

by the Obama Administration, the use of more ﬂexible criteria seems appropriate because the Administration wants to

develop new evidence in the case of models that have some but not top-tier evidence of success.

15. Obama, B.H. (2010) ‘Social Innovation Fund.’ Reprinted in Ofﬁce of Social Innovation and Civic Participation. Available

at: www.whitehouse.gov/administration/eop/sicp/initiatives/social-innovation-fund

16. Corporation for National and Community Service ‘Inaugural Social Innovation Fund Grants Awarded to Experienced

Innovators.’ Press Release, July 22, 2010, available at: www.nationalservice.gov/about/newsroom/releases_detail.

asp?tbl_pr_id=1829

17. Corporation for National and Community Service, ‘Social Innovation Fund.’ (2010), available at: www.nationalservice.

gov/about/serveamerica/innovation.asp

18. Department of Labor. News Release: ‘US Labor Department Encourages Applications for Trade Adjustment Assistance

Community College and Career Training Grant Program.’ January 20, 2011.

19. Department of Labor. ‘Budget in Brief: FY2011.’ p.3.