Using Natural Language Processing to Analyze Political Party Manifestos from New Zealand^†

Round 1

Reviewer 1 Report

This paper applies NLP methods to analyze political party manifestos and provides interesting results.

The methods used in this study include document similarity, topic modeling, and sentiment analysis. The study found that these NLP methods can provide valuable insights that supplement past work, including detecting the positioning of political parties, objective insights into policy issues, and capturing support for particular policy ideas. However, the study also highlights limitations in these methods, such as the need for fine-tuning and vector representations that capture a party's attitude towards a policy issue. The study suggests that expanding the analysis to more cases and different sources of text could lead to new patterns and insights.

It would be interesting to see a comparison between the conventional NLP methods used in this study and the recent advancements in large language models, such as GPT-3, which have demonstrated impressive performance in various NLP tasks. This comparison could provide a better understanding of the potential of these models in analyzing political texts and possibly lead to further improvements in the field.

Author Response

It would be interesting to see a comparison between the conventional NLP methods used in this study and the recent advancements in large language models, such as GPT-3, which have demonstrated impressive performance in various NLP tasks. This comparison could provide a better understanding of the potential of these models in analyzing political texts and possibly lead to further improvements in the field.

We greatly appreciate this suggestion, but it is a heavy lift because of time constraints. Our goal is to certainly incorporate this suggestion into future work, in which we hope to compare the performance of several different models (including GPT-3), for all the techniques used in this study. Such an approach would, however, require at least several weeks of time, and more likely months. It would also increase the size of the paper by at least a factor of 2, and possibly a factor of 3.

Reviewer 2 Report

A brief summary

This paper explores how NLP techniques can be used to analyze the content of political documents. This research applies these techniques to manifestos developed by New Zealand political parties from 1987 to 2017, a period of significant change in the New Zealand party system.

Specific comments:

There are few relevant studies on the analysis of political documents, so this study has some innovative value. Although only some old and classical NLP techniques are used.

This paper also has a clear motivation and decent experimental results.

I am leaning to give a "minor revisions" based on my current knowledge and understanding of the paper. But I will be willing to revisit the decision after get feedback from the author(s).

In particular, I would be glad if the author could clarify the questions below.

*The number of citations in the article is not sufficient, e.g., TF-IDF, Word2vec, and other techniques are not cited.

*The authors used NLP techniques for the analysis, but were confusingly unable to find parameter settings in the paper.

*Although the authors write in the paper why they do not use the latest techniques such as BERT, I would recommend using BERT to perform a little analysis to compare the existing results.

*There are many tables of experimental results in the paper, and the tables should have arrows to indicate whether the parameter is the larger the better or the smaller the better.

*Based on the fact that there are so many experiments performed, a big summary table should be added at the end of the experimental section to describe the final conclusions.

*More should be described about the academic and practical significance of this study. Having obtained these results, what are the guiding implications ? What specific practical applications will there be in the future?

Author Response

We are grateful for the reviewer’s suggestions and have made an effort to address all of them. 

*The number of citations in the article is not sufficient, e.g., TF-IDF, Word2vec, and other techniques are not cited.

We have now added citations for each technique used in this study and least one other citation.

*The authors used NLP techniques for the analysis, but were confusingly unable to find parameter settings in the paper.

We have now added more information about the models and parameters used. Generally, we relied on default settings, and we are now reporting these settings.

*Although the authors write in the paper why they do not use the latest techniques such as BERT, I would recommend using BERT to perform a little analysis to compare the existing results.

We are unclear about this suggestion. The techniques we used rely on BERT embeddings, particularly for the document similarity analysis and for the topic modeling analysis (using BERTopic).

*There are many tables of experimental results in the paper, and the tables should have arrows to indicate whether the parameter is the larger the better or the smaller the better.

The tables are not presenting results based on changes in parameters. The results presented are generally from one set of parameters for each technique applied – the default settings for the application of BERTopic, for example.

*Based on the fact that there are so many experiments performed, a big summary table should be added at the end of the experimental section to describe the final conclusions.

We have now added the suggested table to the discussion section.

*More should be described about the academic and practical significance of this study. Having obtained these results, what are the guiding implications ? What specific practical applications will there be in the future?

We have added content addressing this suggestion towards the beginning of the paper and we attempt to address these ideas more broadly in the conclusion section. Mainly, if the technologies explored in this paper can be honed and scaled, it can facilitate the analysis of the political information environment across democracies. The long term goal is to identify beneficial or dangerous features of political discourse and possible ways to enhance or mitigate them.

Reviewer 3 Report

1. In the 3.2 Topic Modeling of Manifestos section, you wrote the following:

> The topics produced by this topic modeling approach appear reasonable - > have high face validity.

Can you be more definite about your argument by providing pieces of evidence?

> Table 3 shows a topic cluster that appears to be about transportation and > perhaps more specifically about environmentally friendly transportation:

It is not clear what Table 3 refers to. The topics are from which texts?

2. In the 3.3 Sentiment Analysis of Manifesto Sentences section, you wrote:

> The polarity appears relatively positive, as may be expected from the

> Green Party (GP) on this issue.

What do you want the readers to know about the Green Party to make this statement? I suggest you elaborate on your reasoning behind the use of 'relative positive.'

> The average score for each party, along with the number of sentences

> containing the key words, is reported in table 5 below. The average scores

> appear reasonable.

You used the term "reasonable." Is this a scientific judgment or your subjective opinion? I suggest you substantiate the word 'reasonable' logically or with evidence.

3. In 4. Discussion:

You wrote, "Even if the unsupervised approach used here suffers from some inaccuracies" Can you describe the inaccuracies?

4. In 5. Conclusions:

"First, each type of NLP analysis performed in this study would likely benefit from more appropriate vector representations, especially vector representations that captured a party's attitude towards a policy issue."

What do you envision more appropriate vector representations as, or how do you get them?

Author Response

We are grateful for the reviewer’s suggestions and have attempted to respond to all of them, here and in the paper.

1. In the 3.2 Topic Modeling of Manifestos section, you wrote the following:

> The topics produced by this topic modeling approach appear reasonable - > have high face validity.

Can you be more definite about your argument by providing pieces of evidence?

At the moment, we are relying on subjective evaluations of the topics. These topics appear reasonable to us and sound like the kind of topics politicians would cover during debates or interviews. Most of them are not a random collection of non-sensical words (though there are a few topics like that – out of the hundreds created).

We make some of the topics available so that others can see the results, as well. Throughout the paper we list several topics, with their 10 key words. We can make more available, if desired.

> Table 3 shows a topic cluster that appears to be about transportation and > perhaps more specifically about environmentally friendly transportation:

It is not clear what Table 3 refers to. The topics are from which texts?

We now explain more clearly that the topic comes from the Green Party’s 2017 manifesto. We also explain why we chose to use that specific manifesto. Focusing on one party to create topic models returns more distinct policy topics, than when focusing on all the parties at the same time.

2. In the 3.3 Sentiment Analysis of Manifesto Sentences section, you wrote:

> The polarity appears relatively positive, as may be expected from the

> Green Party (GP) on this issue.

What do you want the readers to know about the Green Party to make this statement? I suggest you elaborate on your reasoning behind the use of 'relative positive.'

We have added content on the Green Party’s position as an environmentalist party that also tends to be to the left of center on economic issues. We also try to be more clear about its position relative to other parties.

> The average score for each party, along with the number of sentences

> containing the key words, is reported in table 5 below. The average scores

> appear reasonable.

You used the term "reasonable." Is this a scientific judgment or your subjective opinion? I suggest you substantiate the word 'reasonable' logically or with evidence.

As mentioned, we can only offer subjective judgements at the moment and make the evidence available so that others may also form an opinion, and hopefully provide feedback. In some cases, the results match what we would expect from the traditional ideological positions of parties. We also see that only a few topics (out of hundreds) are composed of nonsense words. But we agree that in the long term it is desirable to have a more objective way to validate what the NLP techniques are returning.

3. In 4. Discussion:

You wrote, "Even if the unsupervised approach used here suffers from some inaccuracies" Can you describe the inaccuracies?

We now specifically mention issues with overlapping topics and topics that are difficult to analyze with sentiment analysis.

4. In 5. Conclusions:

"First, each type of NLP analysis performed in this study would likely benefit from more appropriate vector representations, especially vector representations that captured a party's attitude towards a policy issue."

What do you envision more appropriate vector representations as, or how do you get them?

Likely (and as we mention briefly in the conclusion), aspect-based approaches are ideal. These would allow us to capture the position of party towards a policy idea with quasi-sentences. There did not appear to currently exist, however, any algorithms that fit our purpose. Thus it is likely that we will need to develop such an algorithm ourselves, in future research.