Policy Evaluation

Overview

Effective policy evaluation provides decision-makers with objective analysis of policy options, their likely impacts, and trade-offs. Good policy analysis is evidence-based, considers multiple perspectives, and acknowledges uncertainty.

Policy Analysis Framework

The Eightfold Path (Bardach & Patashnik, A Practical Guide for Policy Analysis, 7th Edition, 2024, SAGE/CQ Press)

1. Define the problem: What exactly needs to be addressed?
2. Assemble evidence: What do we know about the problem?
3. Construct alternatives: What are the possible solutions?
4. Select criteria: How will we judge success?
5. Project outcomes: What will likely happen with each alternative?
6. Confront trade-offs: What are we giving up?
7. Decide: Recommend an option
8. Tell your story: Communicate clearly

Audience note: Bardach's framework was designed for professional policy analysts with dedicated research capacity. For part-time elected officials in small municipalities, consider combining Steps 4-6 into a single weighted comparison matrix, and focusing evidence assembly on peer city comparisons and state municipal league resources rather than academic research.

Problem Definition

Good problem statements:

• Focus on conditions, not solutions
• Are specific and measurable
• Identify who is affected
• Explain why it matters
• Avoid embedded solutions

Bad problem statement: "We need more police officers" Good problem statement: "Response times to priority calls exceed 8 minutes in Districts 3 and 4, compared to citywide average of 5 minutes"

Evidence Gathering

Types of evidence:

• Quantitative data (statistics, metrics)
• Qualitative data (interviews, observations)
• Expert opinion
• Peer community experience
• Academic research
• Historical trends

Assessing evidence quality:

• Source credibility
• Methodology rigor
• Relevance to local context
• Recency
• Potential bias

Evaluation Criteria

Common Criteria

Effectiveness: Does it solve the problem?

• Degree of problem addressed
• Speed of results
• Durability of solution

Efficiency: What does it cost?

• Direct costs
• Administrative costs
• Opportunity costs
• Cost per outcome

Equity: Who benefits and who bears costs?

• Geographic distribution
• Demographic impacts
• Ability to pay considerations
• Generational impacts

Political Feasibility: Can it be adopted?

• Council support
• Stakeholder acceptance
• Public opinion
• Legal constraints

Administrative Feasibility: Can it be implemented?

• Staff capacity
• Technical requirements
• Timeline realism
• Coordination needs

Sustainability: Is it durable?

• Ongoing funding
• Long-term viability
• Environmental impact
• Maintenance requirements

Weighting Criteria

• Not all criteria are equally important
• Decision-makers should assign weights
• Trade-offs become explicit
• Facilitates discussion

Logic Models

A logic model is a visual framework that maps how a program or policy is expected to work, from resources invested through results achieved. It is widely used in program evaluation, grant applications, and performance measurement.

Framework: The Logic Chain

Inputs	Activities	Outputs	Outcomes	Impact
Resources invested	What the program does	Direct products	Short/medium-term changes	Long-term systemic change
Funding, staff, equipment, authority	Services delivered, actions taken	Units of service, participants reached	Behavioral, condition, or status changes	Broad community-level results

Example — Youth Employment Program:

Stage	Description
Inputs	$200K budget, 3 staff, employer partnerships
Activities	Job training workshops, resume coaching, employer matching
Outputs	150 youth trained, 80 placed in internships
Outcomes	60% retain employment at 6 months, improved job-readiness skills
Impact	Reduced youth unemployment rate in target neighborhoods

When to Use Logic Models

• Program evaluation: Map expected vs. actual results to identify where a program is working or breaking down
• Grant applications: Most federal and state grants require or strongly favor logic model submissions
• Performance measurement: Logic models identify which outputs and outcomes to track as KPIs
• New program design: Forces planners to articulate assumptions about how change will happen

Relationship to Bardach's Eightfold Path

Bardach's framework helps choose WHICH policy or program to pursue by comparing alternatives against criteria. Logic models help plan and evaluate HOW a chosen policy is implemented.

Use them together:

1. Bardach (steps 1-7): Define the problem, generate alternatives, evaluate, and select a policy direction
2. Logic model: For the selected option, map the implementation chain from inputs through impact
3. Bardach (step 8, "Tell your story"): Use the logic model to communicate the theory of change to decision-makers

Common Logic Model Pitfalls

• Confusing outputs (things produced) with outcomes (changes achieved)
• Assuming the chain is automatic — each link requires evidence that the connection holds
• Omitting external factors that influence whether outcomes materialize
• Building overly complex models that obscure the core theory of change

Impact Assessment

Fiscal Impact Analysis

• Revenue effects
• Expenditure effects
• One-time vs. ongoing
• Direct vs. indirect
• See Public Finance skill for methodology

Economic Impact Analysis

• Job creation/loss
• Income effects
• Tax base effects
• Multiplier effects
• Regional considerations

Social Impact Assessment

• Affected populations
• Quality of life effects
• Community cohesion
• Access and mobility
• Health impacts

Environmental Impact

• Air quality
• Water quality
• Land use changes
• Energy consumption
• Climate effects

Comparative Analysis

Benchmarking

• Identify peer communities
• Define metrics
• Collect data
• Analyze differences
• Understand context

Selecting peers:

• Similar population
• Similar demographics
• Similar economy
• Geographic proximity
• Comparable services

Best Practice Research

• What works elsewhere?
• Why does it work?
• Is it transferable?
• What adaptations needed?
• What are the risks?

Sources:

• ICMA knowledge network
• Municipal league resources
• Government Finance Officers Association
• American Planning Association
• Academic journals
• Peer direct contact

Stakeholder Analysis

Identifying Stakeholders

• Who is affected?
• Who has influence?
• Who has relevant expertise?
• Who will implement?

Mapping Positions

• Support / Oppose / Neutral
• Intensity of position
• Ability to influence
• Potential to change

Engagement Strategies

• Inform
• Consult
• Involve
• Collaborate
• Empower

Risk Assessment

Types of Risk

• Implementation risk
• Political risk
• Legal risk
• Financial risk
• Reputational risk
• Unintended consequences

Risk Analysis Matrix

Likelihood	Low Impact	Medium Impact	High Impact
High	Moderate	High	Critical
Medium	Low	Moderate	High
Low	Minimal	Low	Moderate

Risk Mitigation

• Avoid (eliminate the risk)
• Transfer (insurance, contracts)
• Mitigate (reduce probability or impact)
• Accept (acknowledge and monitor)

Presenting Analysis

Staff Report Structure

1. Title/Subject
2. Department/Contact
3. Summary/Recommendation
4. Background
5. Analysis
6. Alternatives
7. Fiscal Impact
8. Recommendation
9. Attachments

Decision Matrix

Criteria	Weight	Option A	Option B	Option C
Effectiveness	30%	4	3	5
Cost	25%	3	5	2
Equity	20%	4	3	4
Feasibility	15%	5	4	3
Sustainability	10%	3	4	4
Weighted Score		3.7	3.8	3.6

Communicating Uncertainty

• Be explicit about what we don't know
• Present ranges rather than point estimates
• Identify key assumptions
• Describe sensitivity to changes
• Recommend monitoring approach

Common Pitfalls

Analysis Pitfalls

• Confirmation bias
• Anchoring on first option
• Ignoring opportunity costs
• Underestimating implementation challenges
• Overconfidence in predictions

Process Pitfalls

• Insufficient stakeholder input
• Analysis paralysis
• Ignoring political reality
• Solution in search of problem
• Scope creep

Using Connected Tools

• Use web search for best practice research and peer community examples
• When connected tools are unavailable, work from uploaded documents and web search

Planned connectors (not yet available — plugin works without these):

• document-management — prior analyses and staff reports
• agenda-management — historical decisions and precedents

Municipal Configuration

Check municipal.local.md for:

• Policy priorities
• Budget constraints
• Key stakeholders
• Political context

Output Template

When producing a policy analysis, use this structure:

# Policy Analysis: [Topic]

## Executive Summary
[2-3 paragraph summary]

## Problem Statement
[Clear articulation of the issue]

## Current Situation
[Status quo description]

## Options

### Option 1: [Name]
**Description**: [What this approach entails]
**Implementation**: [What it would take]
**Pros**: [List]
**Cons**: [List]
**Fiscal Impact**: [Estimate]
**Examples**: [Peer communities using this approach]

### Option 2: [Name]
[Same structure]

### Option 3: [Name]
[Same structure]

## Trade-off Analysis

| Criterion | Option 1 | Option 2 | Option 3 |
|-----------|----------|----------|----------|
| Effectiveness | [Rating] | [Rating] | [Rating] |
| Cost | [Rating] | [Rating] | [Rating] |
| Equity | [Rating] | [Rating] | [Rating] |
| Feasibility | [Rating] | [Rating] | [Rating] |

## Key Considerations
[Discussion of important factors]

## Gaps and Uncertainties
[What we don't know]

## Preliminary Assessment
[Analysis of which options seem most promising]

## Recommended Next Steps
[Suggested path forward]

## Sources
[Citations]

Quality Standards

• All claims should be sourced
• Options should be genuinely viable (not strawmen)
• Trade-offs should be presented fairly
• Analysis should be objective, not advocacy
• Uncertainty should be acknowledged

Key Questions for Policy Analysis

1. What problem are we trying to solve?
2. What is the evidence base?
3. What are all the reasonable alternatives?
4. What criteria should we use to evaluate?
5. What are the likely outcomes of each option?
6. Who wins and who loses?
7. What are the trade-offs?
8. What are the risks?
9. Is it implementable?
10. How will we know if it works?

Related Skills

• public-finance: Use for fiscal impact analysis methodology, budget constraints, revenue projections, and pension/OPEB cost implications
• intergovernmental-relations: Use for understanding state and federal policy context, unfunded mandates, grant requirements, and preemption issues that shape local policy options
• ethics-conflicts: Use for conflict-of-interest screening when policies affect officials' financial interests, and for ensuring policy processes meet transparency requirements