EvalC3

…tools for developing, exploring and evaluating predictive models of expected outcomes

EvalC3 is an Excel app designed for use in the monitoring and evaluation of the achievements of development aid projects (and parts thereof). But it also has much wider applicability.

What it does

EvalC3 enable users:

• • To identify sub-sets of attributes that describe an intervention & its context, and which are good predictors of the achievement of an outcome of interest.
  • To compare and evaluate the performance of these predictive models,
  • To identify relevant cases for follow-up within-case investigations to uncover any causal mechanisms at work.

These predictions are based on the screening of a data set that (ideally) describes the attributes of a set of those interventions, their context and their outcomes. EvalC3 uses binary data (i.e. 0/1 values) that can represent category membership, or two halves of a range of numeric values.

While EvalC3 enables different forms of systematic quantitative cross-case comparison, its use should be informed by within-case knowledge at both the pre-analysis planning and post-analysis interpretation stages.

The overall approach is based on the view that “association is a necessary but insufficient basis for a strong claim about causation”, which is a more useful formulation than simply saying “correlation does not equal causation”.

INFLUENCES

The design of EvalC3 makes use of two sets concepts and methods:

• • Qualitative Comparative Analysis, a body of methods developed in Political Science. Especially its view of causality (equifinality, asymmetry, conjunctural) and the importance of combining cross-case and within-case analysis)
  • Predictive Analytics, a body of methods used largely for commercial purposes. Especially what is known about different search algorithms and how the performance of prediction models generated by these algorithms can be evaluated.

Four main tools are available to develop these predictions:

1. Manual hypothesis-led inquiry used to explore the predictive power of specific attributes of prior interest. Suitable for data sets of any size. Ideally the first step in the process of analysis using EvalC3
2. Algorithm-based searches
   1. To find the single best predictive model
      1. For quick answers
         1. Cumulative single attribute searches
      2. To avoid the “local optimum problem
         1. Exhaustive searches of multiple attribute combinations
         2. Evolutionary searches, using a genetic algorithm.
   2. To find the best set of predictive models, covering all observed outcomes
      1. Decision Tree searches.

The results are generated instantaneously in the case of manual hypothesis testing, quickly with evolutionary and Decision Tree searches and sometimes much longer with exhaustive searches for combinations of attributes.

A range of performance measures

The results of each search is a predictive model, which describes a subset of attributes that is consistently associated with a specific kind of outcome. The number of the cases identified (and missed) by predictive models is summarised in the form of a truth table, commonly known as a  Confusion Matrix. This table is then used to generate a range of measures of the performance of a given model, which are suitable for use in different contexts.

There is also a model store, where results of any previous model can be accessed: (a) to compare against the design and performance of the current model and (b) reloaded for further exploration.

Supporting tools

The EvalC3 application also two supporting tools:

1. Post cross-case analysis:
   1. A measure of the overall similarity for all the cases within a Confusion Matrix cell.  Similarity/difference measures enable the selection of cases suitable for subsequent within-case investigation in order to identify the nature of any common causal mechanism underlying the case attributes that have been found to be good predictors of outcomes
   2. A chosen-case measure of similarity with all other cases in the data set, covering all cells in the Confusion Matrix.
2. Pre cross-case analysis: Two measures describing the whole data set.
   1. Diversity: The percentage of all possible configurations of the current set of attributes that are present in the data set. The higher the percentage the less likely a current model will be contradicted by new data
   2.  Consistency: The proportion of all the configurations that have consistent outcomes e.g. all present or all absent. Higher levels of consistency will mean models that are found are less likely to have False Positive cases that will require additional attributes to explain their existence.

Additional options

 Analysis of “effects of a cause”: The default setting for EvalC3 is to analyse “causes of an effect” where multiple project attributes may be contributing to an outcome of interest.

However, EvalC3 can also analyse “effects of a cause”, where a particular project intervention (described by a specific attribute in a data set) may be contributing to multiple outcomes.

Triangulation: Data that has been analysed using Qualitative Comparative Analysis(QCA) or Decision Tree algorithms can also be imported and analysed using EvalC3 tools. See the Data Sets page for examples that can be experimented with.

Predictive models first developed by EvalC3 can also be triangulated by later re-analysis using Qualitative Comparative Analysis(QCA) or Decision Tree algorithm

Origins

The original Excel application was designed in 2015 by Rick Davies, and then developed into a more user-friendly and robust Excel version with the help of Aptivate.com. This was done with two purposes in mind: (a) To widen the range of tools available to identify and analyse complex causal configurations, (b) To widen the use of such tools, among the global community of evaluators. A web based version is now under development.

MORE INFORMATION

• https://evalc3.net/

REQUEST A COPY OF EVALC3

• rick.davies@gmail.com