Quantitative analysis: The AHP

Frequent commentor Elizabeth noted on Chirol’s post last week about FP’s Failed States Index

There’s a reason why some of us wouldn’t attempt to make such a index, knowing how foolish and vain it would be to assume we could quantify chaos and anarchy on a global scale and make comparisons.

I rebutted by saying:

Models and indices can provide points of reference that are not arbitrary, even if they lack a certain pinpoint accuracy. There is only so much aid and development money going around, and being able to see how well a country is tracking, and where money could be best spent, could help prevent wasted cash and assist in making an actual difference.

Decisions should be made based upon “proof points” generated from research, rather than simple arbitrariness. I don’t doubt that some decision makers possess uncanny intuition, and are right more than they are wrong. These are the “great men” of history. Mathematicians have tried to develop transparent models for the rest of us mere mortals.

I previously took a look at game theory, and the question of whether or not Bush should say sorry about Iraq. This time I thought it would be interesting to profile one quantitative model used by business consultants and political scientists alike since the early 1970s: The Analytic Hierarchy Process. Following is a (lengthy) description of this highly effective technique in the context of FP’s Failed States Index.

Multi-Criteria Decision Analysis models provide techniques for decision makers to tackle problems involving complex choices and numerous factors for consideration. The Analytic Hierarchy Process, one type of MCDA model developed by American mathematician Thomas Saaty in the early 1970’s, not only supports a decision maker with a robust and transparent framework for evaluating individual factors, it can account for input from multiple decision makers.

The AHP process begins with the development of a hierarchy, which outlines the Objective and a number of Criterion or attributes that must be considered in making a decision. For example, in our case the Objective is the ranking of failed and failing states, the attributes to look for (according to FP) in making a selection include:

• Mounting demographic pressures
• Massive movement of refugees and internally displaced peoples
• Legacy of vengeance ““ seeking group grievance
• Chronic and sustained human flight
• Uneven economic development along group lines
• Sharp and/or severe economic decline
• Criminalisation and delegitimisation of the state
• Progressive deterioration of public services
• Widespread violation of human rights
• Security apparatus as “state within a state”Â?
• Rise of factionalised elites
• Intervention of other states or external actors

The criteria can include a wide variety of data including subjective ratings on some sort of scale, real values such as dollar amounts, or even a binary value (effectively TRUE or FALSE values). Some of FP’s attributes seem fairly subjective but others can be measured by real means using figures on GDP other indexes like the Gini Index.

Once the criteria are defined, the relative importance of each criterion in comparison to the others must be evaluated by a technique called “pairwise comparison.” Each attribute is loaded into a matrix, and individually compared with every other attribute in a one-on-one ranking by an “expert.” Judgements are made on a 1 to 9 scale as experts determine which attribute they think is more important and to what degree.

Pairwise comparison of attributes

The beauty of the AHP is that it can evaluate how consistent your rankings are. Sometimes ranking many different attributes difficult resulting in inconsistent results (eg. A is ranked 3 times more important than B, which is twice as important as C which is ranked 5 times more important than A.)

Furthermore, Saaty’s AHP can harness the disparate evaluations of a team of experts by having each pairwise compare the criterion and then taking the geometric mean of the resulting matrix. While the arithmetic mean gives the average of a sum of quantities, the geometric mean calculates the average factors of a product.

Once the criterion are pairwise compared against each other the scores are normalized. This does two things: 1) it produces a ranking of each of the criteria, and 2) it associates with each criterion a numerical weight. This weighting is used in the next step of the process: evaluating each of the alternatives. In our problem each country will be placed on the left side of a matrix with all the attributes along the top. Each column of the matrix will then be loaded with observed data (whether subjective, real, or binary) for each country.

Loading country values

Every one of these values will be normalized, then multiplied by the weighted values calculated in the previous step. This renders a ranked list of the alternatives, from which decision makers can select the “best” alternative based on the the rational evaluation of each attribute.

We can go one step further by doing a statistical analysis of the resulting rankings by making a histogram, mathematically identifying thresholds and specific categories within the set of rankings. Policymakers could even assign a percentage of aid based on these categories.

Moreover, this is extremely useful for data which changes over time. Each alternative can be measured periodically to determine whether or not they are getting “better” or “worse”, an invaluable benchmark for policymakers to base cost-benefit-type decisions on. Countries that look like they have some momentum towards success might mean more bang for our aid buck, and other countries may be going down too far and too fast, representing a black hole for our cash.

I won’t go through the entire data-entry process since this is worth a master’s degree in itself, but hopefully you all get an idea of how useful such a process can be. For those that would like to learn more, or look at a simpler type of problem (eg. which car to buy based on the criteria of Style, Reliability, and Fuel Economy) tale a look at this illustrated tutorial.