In the intelligence game ambiguity is a relative certainty. Predicting events is an impossibility, but when forecasting it is important to make sure that decision makers know exactly what you mean, and are not confused by what you say. Subjective words like “certain, probable, definite, slam-dunk” can be problematic. Sherman Kent, father of US intelligence, gives some advice on systematically using these difficult words (and their variants) by utilizing a probability table:
| 100% Certainty | ||
| The General Area of Possibility | ||
| 93% | give or take about 6% | Almost certain |
| 75% | give or take about 12% | Probable |
| 50% | give or take about 10% | Chances about even |
| 30% | give or take about 10% | Probably not |
| 7% | give or take about 5% | Almost certainly not |
| 0% Impossibility | ||
This is an interesting technique used often in the intelligence community. By developing arbitrary quantitative values to weigh subjective terms or issues of analysis, better communication can be achieved between analysts and with the consumers. Overall cohesiveness of analysis across a number of analysts will improve as well, since each stakeholder will know how much weight each analyst puts on which factor.
For example: two analysts are tasked with forecasting the probability of the North Koreans doing something to Seoul. Both agree that Kim Jong Il is a maniac and unpredictable, but one analyst says the probability is 60%, and the other 80%. Which one do you beleive? It could be helpful to have each analyst rate the Deer Leader on a crazy scale, say from 1 to 10. Any differences of opinion in sanity will need to be backed up with the proper evidence, which may lead to a re-evalution of KJI’s nutbar factor, and an adjusted probability for attack.
Get the RSS
Comments to this entry
Dan
October 15, 2005
9:46 pm
Gabriel Mihalache
October 15, 2005
10:53 pm
- out of all the possible outcomes (y in number), (x*100/y)% are favorable and ((x-y)*100/y)% are unfavorable.
- if we were to repeat "the experiment" (assuming it's replicable, or maybe that we can turn back time) y times, for a big enough y, we expect that x times the outcome to be favorable and y -x times to be unfavorable.
Now, regarding politics situations, we don't know all possible (and equiprobable) outcomes, to count those that are favorable, nor can we repeat the situation to count the favorable outcomes (neither can we rely on simulations because we don't know all possible outcomes)
Sorry if I'm boring you with Statistics 101 but I'm trying to make the point that an association between phrases and percentages cannot be grounded in the science of statistics, but since it's not the place of the observer (me) to question the use of language, I can only draw the conclusion that this association is purely psychological (our psychological image of the phrase matches/is proportional to our psychological imprint of the percentage)
My point? If 2 analysts give you different percentages, I think you should look at the analysts. If you require another guesstimation from them (e.g. to rate the lunatic from 1 to 10) you are back to the original problem of them giving different answers for no "objective reason".
Having determined a purely psychological aspect of this association between unclear phrases and percentages, it follows that we have not "scientificated" the issue. (This is to be expected, since by language or psychology alone we can't jump from the uncertainty of phrases to the certainty of numbers.)
"75%" might look more authoritative than "probable" but as far as real progress goes, I don't think we're making any.
Curtis Gale Weeks
October 16, 2005
12:07 am
My point? If 2 analysts give you different percentages, I think you should look at the analysts.
Yes, I think I agree. I'm about 90% certain or perhaps 8% uncertain this would be the best route. But the point is that few, few predictions of this sort ever reach 100% certainty -- until Kim Jong Il crosses the physical (and perhaps mental) line. So a standardized measuring system is a very good idea. (Again, approx. 90% certainty of this.)
I'm not at all familiar with the theory. My gut or perhaps that is my mind is telling me that different standards of proof would be asked of analysts who give different levels of certainty. For instance, a 90%-certain guestimate or a 10%-certain guestimate probably wouldn't require as much further fact-finding as a 40-60% certainty level. If this makes any sense (?), then Gabriel's on the right track: analysts which give a 40-60% certainty on any given subject should be fired. (They shouldn't be giving any figure, if they're at those levels, but should be spending their time obtaining the objective facts.)
mark safranski
October 16, 2005
2:38 am
Younghusband
October 16, 2005
4:53 pm
You are right Gabe, this is not scientific at all. That is why I used the word "arbitrary." It is a way of measuring opinion if you like, of looking at the analysts themselves as you stated.
Why is the use of the probability table important? I can give you one perfect example: The Bay of Pigs affair of 1961. If President Kennedy knew that the JCS meant "3 to 1 against" when they told him the invasion had a "fair chance" of success, what would his decision have been then?
lirelou
October 17, 2005
6:00 am
Often, merely asking for the information paper will nudge the analysts into getting together to hammer out a common position, which essentially throws the onus on them to concile their opposing views.
ComingAnarchy.com » Blog Archive » Towards more rigorous analysis
May 11, 2007
12:45 am