System Evaluation Theory

A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems

By:
Ralph Renger, Just Evaluation Services (JESS), LLC

A volume in the series: Evaluation and Society. Editor(s): Stewart I. Donaldson, Claremont Graduate University. Katrina L. Bledsoe, Education Development Center.

Published 2022

Too often evaluation findings end up on the shelf! Why? Many interventions have several moving parts working together to achieve something each part cannot do independently. Unfortunately, many of the available evaluation approaches oversimplify this reality. A major reason for this is that evaluators do not have a blueprint to plan, and the associated methods to execute, an evaluation fit for this purpose. As such, they revert to using methods with which they are familiar, but are not suitable, for evaluating complex interventions. Consequently, the evaluation findings do not make sense to decisionmakers, so they are ignored. This book provides evaluators of all skill levels with a simple 3-step framework, System Evaluation Theory, that will lead to more actionable recommendations because the methods better capture the reality in which interventions operate; that is as a system.

The book first reviews the limitations of program evaluation approaches that pointed to a need for a different way of thinking, one grounded in systems. After defining a system, the book explains how SET is intentionally aligned to evaluate a system’s two essential properties: interdependence and emergence. Individual chapters are dedicated to explaining how to execute each SET step. Throughout, the author draws on real-world examples and those from his own evaluations to help bridge the theory-practice divide. The reader is then shown how to use SET to develop actionable recommendations.

CONTENTS
Prologue. CHAPTER 1: The Rise of Systems Thinking in Evaluation. CHAPTER 2: What Is a System? CHAPTER 3: System Boundaries, Levels, and Subsystems. CHAPTER 4: Cross-Cutting System Attributes. CHAPTER 5: The Why Behind System Evaluation Theory. CHAPTER 6: System Evaluation Theory Step 1, Defining the Complex Intervention Acting as a System. CHAPTER 7: System Evaluation Theory Step 2, Evaluating System Efficiency (Interdependencies) CHAPTER 8: Evaluating System Interdependencies Using a Family System Case Example. CHAPTER 9: System Evaluation Theory Step 3, Evaluating System Effectiveness (Emergence) CHAPTER 10: Deriving Recommendations. CHAPTER 11: Resistance to System Evaluation.