Validating Game-Theoretic Models of Terrorism: Insights from Machine Learning

Round 1

Reviewer 1 Report

It advances our understanding of the matter significantly.

There are just some thoughts which the authors may want to consider.

First, in my file there were still some parts of some formulas missing, that would be some software glitch, that you might want to look at.

Second, I feel that the relationship to game-theoretic models could be made a bit better. For instance, why are the interesting results on threshold effects consistent with game-theoretic considerations (a number of games imply  comparative static results with smooth transitions) and thus be better than the usual approches?

Third, I would have a thorough read of the ms. There are still some minor glitches, e.g. some references are not in the list of references. That sort of thing.

Fourth, some more considerations on the relationship between Extreme Bounds Analysis and Machine Learning may (or may not) be appropriate.

Overall, nice paper which needs to get out there.

Author Response

1. We have tried to ensure that all the formulae are visible.
2. We have added a section 6 where we specifically highlight how our results validate game-theoretic model predictions. Here we now show how variable importance can elevate certain theoretical models over others by evaluating the predictive salience of the consequences of these models. We also show how partial dependence plots can validate the non-linear predictions of these models without assuming anything about theory for model specification.
3. We have checked the references. 

Reviewer 2 Report

This paper is poorly motivated and poorly executed. The utilization of machine learning techniques to increase our knowledge of aspects that may not be fully captured by theory or empirics is desirable. However, this paper fails to deliver exactly this. It "aggregates" all theoretical and empirical papers without paying no attention to key distinguishing factors. It argues that empirical models fail to appropriately deal with non-linearities which are implied by predictions of theoretical models. Presumably, the application of machine learning techniques do a better job. However, the "fishing exercise" carried out by this paper - in that it includes a large number of 'explanatory' variables and then let the machine learning algorithm select (fish out) the appropriate relationships - is not intuitive and does not lead to credible conclusions. The exercise does not allow us to learn anything useful about terrorism.

Author Response

The object of the paper was to show how machine learning can provide insights and potentially predictively validate models of terrorism. We show how (a) machine learning can illuminate nonlinear relationships between variables of interest and terrorism without prior theoretical restrictions and (b) use predictive salience as a way to validate models of terrorism.

Game-theoretic models highlight endogenous strategic interactions. These are hard to test using standard parametric techniques since the problem being modeled is by definition endogenous. Machine learning models, by focusing on prediction, can validate game theory-generated hypotheses since prediction does not require the sort of assumptions about the distributions of variables necessary for estimating BLU estimators. 

Reviewer 3 Report

The paper carries out an interesting exercise whose goal is to predict terrorist attacks using a variety of Machine Learning (ML) techniques. It combines data from several databases to create a large set of explanatory variables and then culls the most important variables based on their contribution to the reduction of the mean squared error (MSE). The predictive power of the best ML models seems to be satisfactory, but there are many problems with the data set and the empirical model, which I discuss below:

• The authors use 1970-2014 data from the Global Terrorism Database (GTD) to estimate their models, but do not address some known issues with that data set. GTD records both domestic and transnational terrorism and did not distinguish between the two until 2013. According to Sandler (2015): “A breakdown of terrorism into its two components is essential because the two types of terrorism may affect economic variables and counterterrorism differently.” In addition, GTD has changed its coding conventions a few times, and data for 1993 are incomplete because part of it was lost (Sandler, 2015).
• Some of the explanatory variables used in the paper may be terrorist attacks themselves, like the number of assassinations and guerrilla warfare incidents from the Cross-National Time Series – CNTS. A discussion of how terrorist organizations choose between guerrilla tactics and attacks on civilians can be found in Carter (2016).
• One of the paper’s findings is that the strongest predictors of current levels of terrorism are the history of assassinations and guerrilla warfare. There is an endogeneity problem here, since assassinations and guerrilla warfare can be terrorist acts.
• Variable importance (salience) is measured by how much it helps reduce the MSE. However, one of the shortcomings of ML is inconsistency in model selection. In other words, similar predictions from a ML model can be obtained using very different variables (see, e.g., Mullainathan and Spiess, 2017), making it difficult to learn about the underlying processes that drive terrorism.
• The authors claim that “machine learning algorithms can provide scientifically cross-validated predictions of the likelihood of a terrorist attack to provide national security agencies with an abbreviated, cross-validated list of variables (i.e., policy levers) that can best identify and hopefully deter terrorism.” However, their findings to not lend themselves to this purpose. The output (or dependent) variable in their empirical model is the total number of terror attacks in a country, and the most important predictors (explanatory variables) are assassinations and guerilla attacks. It is hard to see how national security agencies would be able to use these findings to inform their decisions about deterrence measures.

In addition to the issues listed above, it is worth pointing out that the authors seem to be unaware of the large literature in Economics and Econometrics that merges ML and econometric methods (see Mullainathan and Spiess (2017) for a list of papers in this area) and that many ML techniques have been part of the econometrician’s tool kit for quite some time (see Athey and Imbens (2019) for a discussion about the differences and similarities between ML and traditional Econometrics).

In the introduction of the paper, the authors suggest that “the full power of game theoretical insights can be validated by machine learning”, and offer as a key contribution of their paper the introduction of “the emerging methodology of machine learning to the game-theoretic study of terrorism that can, to a great extent, overcome the limitations of classical regression-based methods”. Unfortunately, their paper was not able to accomplish that goal. To identify a group of variables that are successful in predicting terrorist attacks is hardly enough to choose between theoretical models. In fact, a thorough discussion of the different types of game-theoretical models of terrorism is absent from the paper.

References:

Athey, S., and G. W. Imbens. 2019. “Machine learning methods economists should know about,” Stanford Graduate School of Business Working Paper No. 3776.

Carter, D. 2016. “Provocation and the strategy of terrorist and guerrilla attacks,” International Organization 70: 133-173.

Mullainathan, S., and J. Spiess. 2017. “Machine Learning: an applied econometric approach,” Journal of Economic Perspectives 31 (2):87-106.

Sandler, T. 2015. “Terrorism and counterterrorism: an overview,” Oxford Economic Papers, vol. 67 (1): 1-20.

Author Response

1. We have replaced the first paragraph of section 3 with insights from Mullainathan and Spiess, 2017.
   
   
2. We have added section 6 where we specifically highlight how our results validate game-theoretic model predictions. Here we now show how variable importance can elevate certain theoretical models over others by evaluating the predictive salience of the consequences of these models. We also show how partial dependence plots can validate the non-linear predictions of these models without assuming anything about theory for model specification.
   
   
3. We have changed the conclusion and the introduction to reflect changes in points 1 and 2 above.

Round 2

Reviewer 2 Report

No comments.

Reviewer 3 Report

I appreciate the authors' effort to address some of the issues I raised in my first report, in particular the new section discussing the implications of their results for game theory models.

However, three critical problems I mentioned in my first report were not resolved: issues with the Global Terrorism Database (GTD); use of explanatory variables that are themselves measures of terrorist attacks; and the fact that the most predictive variables are not useful for policy-making.

Moreover, the new section on game theory models does not succeed in showing how machine learning can be used to validate theoretical models. It posits supposed empirical implications of a handful of theoretical models found in the literature and claims they are consistent with the predictions of the ML models. Although interesting, this analysis is not enough to evaluate the plethora of models found in the game-theoretic terrorism literature.