Policy design analysis and evaluation

A.Y. 2021/2022
Overall hours
INF/01 SPS/04
Learning objectives
The students will be equipped with the conceptual and empirical tools to understand the most salient features and the most relevant consequences of today's processes in a number of significant social domains.
Expected learning outcomes
By the end of the course students will acquire the ability to critically evaluate and discuss the impact, development and use of the topics covered during the course. The final exam aims to verify the expected learning outcomes in relation to these topics.
Course syllabus and organization

Single session

Lesson period
Second trimester
Course syllabus
Class 00 - The map of the course
The first class illustrates (a) the object and goal of the course; (b) the available resources; (c) the learning path.
Background readings:
A Schneider, H Ingram, Systematically pinching ideas, 10.1017/S0143814X00006851
A Schneider, H Ingram, Behavioral assumptions of policy tools, 10.2307/2131904

Module A - The substance
Instructor: Alessia Damonte

Public policies are solutions to collective problems. As such, they build on two intertwined causal hypotheses. The first considers that the problem follows when some actors adopt certain behaviors in specific situations. The second hypothesis maintains that interventions on the situation or, more directly, on the actors' behaviors can solve the problem, or at least alleviates it.
The module offers analytic tools to develop the two hypotheses, then focuses on the usage of information as a policy instrument.

Class 02 - The focus of public policy: Action situations
PA Sabatier, Theories of the Policy Process, Ch 2
Class 03 - Discussion session
E. Ostrom, Reflections on 'Some Unsettled Problems of Irrigation,' 10. 1257/aer.101.1.49

Class 04 - The making of public policy
PA Sabatier, Theories of the Policy Process, Ch 3, 7
Class 05 - Discussion session
E Aukes, K Lulofs, & H Bressers, Framing mechanisms, 10.1080/19460171.2017.1314219

Class 06 - Information as regulation
JG Kelley, BA Simmons. Politics by number 10.1111/ajps.12119
Class 07 - Discussion session
S Schueth, Winning the rankings game, 10.1017/CBO9781316161555.007

Class 08 - Policy-takers or policy-makers?
L Bini M Bellucci, Accounting for Sustainability, Ch 2 10.1007/978-3-030-24954-0_2
Class 09 - Discussion session
Adrian Zicari and Luis Perera Aldama, Value-Added Statements as a Communication Tool for Stakeholders, 10.1007/978-3-319-62785-4_9

Class 10 - Classwork: analyze and improve actual non-financial accounts
Class 11 - Classwork: analyze and improve actual non-financial accounts

Module B _ The shape
Instructor: Alessia Damonte

Causal hypotheses about the world share a formal structure that especially the lenses of logic can reveal. The module introduces students to the language and the rules of propositional logic and illustrates the features of valid reasoning.

Class 12 - Argument and evidence, An informal analysis of arguments
P Phelan and P Reynolds, Argument and Evidence, Ch 2,4
Class 13 - Patterns of reasoning
P Phelan and P Reynolds, Argument and Evidence, Ch 5
Class 14 - Establishing validity
P Phelan and P Reynolds, Argument and Evidence, Ch 6
Class 15 - Critical analysis in practice
P Phelan and P Reynolds, Argument and Evidence, Ch 7
Class 16 - Assumptions
P Phelan and P Reynolds, Argument and Evidence, Ch 8
Class 17 - Evidence as the ground for the tenability of belief: yet, what counts as evidence?
P Phelan and P Reynolds, Argument and Evidence, Ch 9,10
Class 18 - Test

Module C _ Establishing tenability with sets
Instructor: Alessia Damonte

Qualitative Comparative Analysis (QCA) allows establishing the tenability of causal hypotheses with the set-theoretical rendering of logical models. The module introduces the special assumptions and strategy of the technique with the aid of an actual example.

Class 19 - Attaching numbers to literals: crisp and fuzzy scores
A Dusa, QCA with R, Ch 4
Class 20 - Analytic devices: parameters of fit
A Dusa, QCA with R, Ch 5, 6
Class 21 - Analytic devices: the truth table
Dusa, QCA with R, Ch 7

Class 22 - A QCA from start to finish - Modeling a configurational explanation: hypothesis, cases, and gauges
Class 23 - A QCA from start to finish - Necessity analysis and directional expectations
Class 24 - A QCA from start to finish - Sufficiency analysis and minimizations
Class 25 - A QCA from start to finish - Visualization
Rihoux B, & CC Ragin. 2008. Configurational comparative methods. Ch 2, 3, 5
Damonte A, 2021. Modeling configurational explanations, 10.1017/ipo.2021.2

Class 26 - Group work - Critical replication
Class 27 - Group work - Critical replication
Class 28 - Group work - Critical replication

Class 29 - Students' presentations and discussion
Class 30 - Students' presentations and discussion

Module D _ Establishing tenability with frequentist probability
Instructor: TBC

Either explicitly or implicitly, the goal of most empirical research is to interpret causally the co-occurrence of interesting phenomena. Establishing causality, however, has been notoriously difficult without the luxury of experimental data. After presenting the Potential Outcome as the framework of reference valid causal inference, this module introduces students to the theory and practice of four designs for convincing causal claims without experimental data.

Class 31 - The Experimental Ideal & the Potential Outcomes Framework
Angrist and Pischke: Ch. 1 & 2
Dunning: Introduction, Ch. 5.1 (The Neyman model 5.1.1-5.1.5)

Class 32 - Instrumental Variables
Dunning, Ch. 4
Angrist and Pischke: Ch. 4
Card (1993), 10.3386/w4483
Class 33 - Lab session

Class 34 - The Regression Discontinuity Design
Dunning: Ch. 3
Angrist and Pischke: Ch. 6
Lee (2007), 10.1016/j.jeconom.2007.05.004
Class 35 - Lab session

Class 36 - Matching
Angrist and Pischke: Ch. 3.3.1
Costalli & Negri (2021), 10.1017/ipo.2021.2
Class 37 - Lab session

Class 38 - Difference-in-Differences
Angrist and Pischke: Ch. 5.2 (Differences-in-Differences 5.2-5.2.1)
Card & Krueger (1993), 10.3386/w4509
Class 39 - Lab session

Class 40 - Questions and answers
Prerequisites for admission
The course assumes no special previous knowledge. Nevertheless, students will find module D easier after learning Multivariate Analysis for Social Scientists. Besides, they are warmly suggested to bring their laptops during classes and, especially since Module C, to have the updated version of R and RStudio installed.
Teaching methods
The course develops students' knowledge and competencies through lectures, data lab sessions, instant polls, debates, group-works, and presentations.
Teaching Resources
Module A
- Sabatier Paul A. 2007. Theories of the Policy Process. Boulder, CO: Westview Press. Ch. 2, 3, 7
- Schneider, A., & Ingram, H. 1988. Systematically pinching ideas: A comparative approach to policy design. Journal of Public Policy, 61-80.
- Schneider, A., & Ingram, H. 1990. Behavioral assumptions of policy tools. The Journal of Politics, 52(2), 510-529.
- Ostrom, E. 2011. Reflections on" Some unsettled problems of irrigation". American Economic Review, 101(1), 49-63.
- Aukes, E., Lulofs, K., & Bressers, H. 2018. Framing mechanisms: the interpretive policy entrepreneur's toolbox. Critical Policy Studies, 12(4), 406-427.
- Kelley, J. G., & Simmons, B. A. 2015. Politics by number: Indicators as social pressure in international relations. American journal of political science, 59(1), 55-70.
- Schueth, S. 2015. Winning the rankings game: The Republic of Georgia, USAID, and the Doing Business project. 151-177 in A Cooley, J Snyder (Eds). Ranking the World: Grading States as a Tool of Global Governance, 151-77.
- Bini L., Bellucci M. 2020. Accounting for Sustainability. 9-51 In: Integrated Sustainability Reporting. Springer, Cham.
- Zicari A., Aldama L.P. (2017) Value-Added Statements as a Communication Tool for Stakeholders: The Case of Industrias Peñoles in Mexico. 193-214 In: Freeman R., Kujala J., Sachs S. (eds) Stakeholder Engagement: Clinical Research Cases. Issues in Business Ethics, vol 46. Springer, Cham.

Module B
- Phelan, P. and Reynolds, P. 1996. Argument and Evidence: Critical Analysis for the Social Sciences. London: Routledge.

Module C
- Rihoux, B., and C.C. Ragin. 2008. Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Sage Publications. Ch 2, 3, 5
- Dusa, A. 2019. QCA with R. Cham: Springer.
- Damonte, A. 2021. Modeling configurational explanations. Italian Political Science Review/Rivista Italiana di Scienza Politica, 1-16.

Module D
- Angrist, Joshua, and Jörn-Steffen Pischke. 2009. Mostly Harmless Econometrics: An Empiricist's Companion, Princeton: Princeton University Press.
- Dunning, Thad. 2012. Natural Experiments in the Social Sciences: a Design-Based Approach. Cambridge University Press.
- Card, D. 1993. Using geographic variation in college proximity to estimate the return to schooling. NBER working paper, (w4483).
- Card, D., & Krueger, A.B. 1993. Minimum wages and employment: A case study of the fast food industry in New Jersey and Pennsylvania (No. w4509). National Bureau of Economic Research.
- Costalli, S., & Negri, F. 2021. Looking for twins: how to build better counterfactuals with matching. Italian Political Science Review/Rivista Italiana di Scienza Politica, 1-16.
- Lee, D.S. 2008. Randomized experiments from non-random selection in US House elections. Journal of Econometrics, 142(2), 675-697.
Assessment methods and Criteria
The course equips students with the knowledge and the techniques to establish the logical, causal, and empirical tenability of policy designs as causal arguments.

Students are evaluated for their:

1. active participation.
Lectures include instant polling, lab sessions, discussions, and presentations.
Active participation accounts for up to 6 points.

2. capacity to develop suitable metrics to report social, environmental, and governance achievements.
In the last part of module A, students are organized in workgroups on non-financial reporting.
An effective groupwork earns up to 6 points.

3. capacity to formalize a policy argument.
At the end of module B, students are asked to translate a hypothesis from natural language to Boolean algebra.

4. competence in configurational analysis.
At the end of module C, a replicable published QCA is assigned to each student. Under guidance, they produce a script that answers the following questions:
i) the model: is it configurational?
ii) case and raw variable selection: does it afford proper probation?
iii) calibration: is it replicable?
iv) directional expectations: are they empirically supported?
v) truth table: is there any inconsistent primitive?
vi) solutions: are the deserving ones discussed?
Proper completion accounts for up to 6 pts.

5. competence in quasi-experimental analysis.
At the end of module D, students are asked to replicate a specific quasi-experimental study, then to discuss the strengths and weaknesses of the technique and of its application in the study.
Proper completion account up to 6 pts.

The individual final colloquium discloses the points earned during the whole course and offers the opportunity to review past exercises, discuss shortcomings, and add insights.
The colloquium decides the final mark with a possible adjustment of up to 3 points.
INF/01 - INFORMATICS - University credits: 6
SPS/04 - POLITICAL SCIENCE - University credits: 6
Lessons: 80 hours
Thursday 11:30-12:30 (students) - 13.30-15.30 (thesis students and PhD candidates)
internal building, 2nd floor, room 12 | VirtualOffice channel in Teams