Finding Evidence of Fraudster Companies in the CEO’s Letter to Shareholders with Sentiment Analysis

Round 1

Reviewer 1 Report

Please see comments in the attached file.

Comments for author File: Comments.pdf

Author Response

Reviewer 1, Thanks for your suggestions. 

line 57. It is not recommended to start a sentence with a quote paper

Wording has been changed to avoid starting a sentence with a quote paper. 

line 110. What newspapers

We have added:  we have used Spanish reference newspapers (for instance El País, El Periódico, El Mundo)

line 122. Perhaps use the same term as in the title: shareholers.
Done!

line 137. 
Wording changed to "converted to plain text, encoding with character set UTF-8." as suggested. 

line 146. This table was never mentioned in the text. (for table 2) 
Reference to the table has been included  In Table \ref{tab2}, we 

line 146. This table was never mentioned in the text. (for table 2) 
Reference to the table has been included  In Table \ref{tab3}, we 

References. All the references have been revised according to MDPI reference guidelines.
https://0-www-mdpi-com.brum.beds.ac.uk/authors/references 

Reviewer 2 Report

The paper is well written and presented making it easy to follow.

I have below minor comments:

1. Some of the sentences are quite long, affecting readability in part. For example, Line 160-163. Please rephrase them into shorter, simpler sentences.
2. Line 145 - 2 should be written as Table 2 (also at other places).
3. It is not clear to me how the required data was obtained? Is it open source and available to others? Are the text documents from only one industry? 
4. Also, it would be useful if the authors made their data available for other researchers to reproduce (just a suggestion).
5. Line 237 - result insufficient ?? to learn the classes...
6. Line 258 - obtained by summing... (or obtained as a sum of)
7. Line 309-314 requires rephrasing.
8. Line 348 - more occurrence...

Author Response

Reviewer 2.  Thanks a lot for your comments and suggestions.

1. Some of the sentences are quite long, affecting readability in part. For example, Line 160-163. Please rephrase them into shorter, simpler sentences.

We have rephrased this and other sentences (see 6)

2. Line 145 - 2 should be written as Table 2 (also at other places).
done, the same for Table 3. 

3. It is not clear to me how the required data was obtained? Is it open source and available to others? Are the text documents from only one industry? 

Thanks for this comment. Indeed we had not fully described the data. Only a short reference in the introduction was given. We have added the following information in the Materials and Methods section.

For our study, a corpus of annual financial reports from different companies was compiled. The publication and wide dissemination of annual financial reports is a legal requirement in Spain in order to promote transparency. The annual financial reports are formal documents with detailed information addressed to shareholders about the financial activities of companies and are meant to be a justification of the management. The information is presented in figures with extensive explanations and a section devoted to management discussion and analysis. Traditionally, they also include, as foreground, a letter addressed to shareholders and signed by the president or the CEO of the company. We collected these letters extracting them from publicly available annual reports.

4. Also, it would be useful if the authors made their data available for other researchers to reproduce (just a suggestion).

Yes, it was already mentioned that data was anonymized to make it sharable. We have rephrase the sentence to make it explicit. 
 
"Additionally, we have anonymized the texts to be able to share the data with other researchers."

5. Line 237 - result insufficient ?? to learn the classes...
Very unfortunate sentence, yes. We have deleted it. 

6. Line 258 - obtained by summing... (or obtained as a sum of)
This long and messy sentence has been rephrased to this one:

"The sentiment score was obtained by first summing positive and negative adjective scores of each of 48 randomly chosen documents. The document score was then used for computing the average value for each class. The computed average value was 78.83 for fraud-documents and for non-fraud documents, 94.62. This class average score confirmed that fraudster documents, like deceptive ones, have a lower positive tone than truthful ones."

7. Line 309-314 requires rephrasing.
Yes, we have rephrased it as follows.

"The results of our classification experiments with fraudsters texts are in line with other similar experiments, although cannot be formally compared with them because using different data sets. For English texts extracted from financial reports, Goel and Uzuner. \cite{goel2016sentiments} reported a 81\% accuracy in detecting fraudster's texts. In their experiment, described in Section \href{Introduction}, they also used an SVM engine but with a very different document representation. They used frequency counts of the different PoS, including number of occurrences of comparative and superlative adverbs. Instead, we have used a bag of words made of a number of adjectives for representing documents."

8. Line 348 - more occurrence...

Rephrased as follows: 

"On the other end, with the lowest  weights, there are differences for other adjectives that seem to occur in both classes."

Reviewer 3 Report

First of all congratulations on submitting the paper. The comments which could improve the paper are given below:
1) All authors are from the same university, department, so no need to write footnote 1,2,3 at all, just one department, and the emails are separated by semicolon.
2) In the abstract, manuscript no need to write always "our research", in some cases it is obvious that is talking about this research.
3) Need to add such keywords like sentiment analysis, SVM;
4) The Introduction section has to be rewritten by splitting it into two separate sections: Introduction and Related works.
5) In the just a general information has to be given about sentiment analysis, about your paper, and I suggest at the end of Introduction to present the structure of the work, by listing the section number and what is inside of them.
6) It would be nice to write at least a paragraph (in the Related works section) what other researchers uses in the sentiment analysis for different languages analysis. In such a way, the authors will show that sentiment analysis can be used in various fields, not only in fraud detection areas. For example: https://0-ieeexplore-ieee-org.brum.beds.ac.uk/abstract/document/6984256; https://0-www-mdpi-com.brum.beds.ac.uk/2076-3417/11/10/4443; etc.
7) It is not a good decision to list references like in lines 45 and 53. Better less, but with some explanation of what is inside of them, the main keys, tasks, results. Now it just a bunch of references.
8) Need a better argument why such dataset splitting is used? Using such a split method the results depend a lot on splitting. If you would use the cross-validation with some k-folds numbers, the real accuracy in an experimental investigation would be seen. Now it more about luck. The obtained results by splitting by themself can lead to the wrong accuracy. At least give a better explanation of why it has been done like that.
9) Need to specify or in Table 1 are given pre-processed text lengths or not.
10) Table 2 with so many citations looks not acceptable in my opinion, it could be redesigned. 
11) Need an argument why SVM is used. There a lot of research where various machine learning methods are used in the sentiment analysis (SVM, Naive Bayes, trees-based algorithms, LSTM, etc.). In your case you have a binary classifier, so maybe it could be the reason, but anyway it is hard to say that this classifier is the best. From personal experience, I would say the other methods could give better results.
12) The experimental investigation performed using a small amount of dataset, so it would be easy to add at least two other methods. But authors have to decide by them-self it is needed or not. In my opinion, it would significantly improve the paper.
13) If you made cross-validation, Table 5 would show the real situation of your classifier. 
14) The experimental investigation is limited, the dataset is very small, but specific, for the Spanish language, maybe in this case it is a novelty. Need to highlight it more from the beginning of the paper. 

Good luck with submitting the paper.

Author Response

Reviewer 3. 

1) All authors are from the same university, department, so no need to write footnote 1,2,3 at all, just one department, and the emails are separated by semicolon.

done

2) In the abstract, manuscript no need to write always "our research", in some cases it is obvious that is talking about this research.
Yes, the abstract has been revised. 

3) Need to add such keywords like sentiment analysis, SVM;

done

4) The Introduction section has to be rewritten by splitting it into two separate sections: Introduction and Related works.

done. 

5) In the just a general information has to be given about sentiment analysis, about your paper, and I suggest at the end of Introduction to present the structure of the work, by listing the section number and what is inside of them.

done. New paragraph about SA at the very beginning, and in 45-56 description of the structure of the paper. 

6) It would be nice to write at least a paragraph (in the Related works section) what other researchers uses in the sentiment analysis for different languages analysis. In such a way, the authors will show that sentiment analysis can be used in various fields, not only in fraud detection areas.

I've added the suggested references with some more explicit description of SA and other tasks. See 58-66.

7) It is not a good decision to list references like in lines 45 and 53. Better less, but with some explanation of what is inside of them, the main keys, tasks, results. Now it just a bunch of references.

In 45, what was intended is to have a list of the different papers that have come to the same conclusion. The goal was to synthesize the information to avoid saying the same for the different references. 
For clarity, we have rephrased the sentence like this: 

"It is also a common observation reported in \cite{npbr03}, \cite{ztqbn03}, \cite{gg12}, \cite{burgoon1996deceptive}, \cite{hancock2007lying}, \cite{hobsonMayew2012}, \cite{liu2012exploring} and \cite{goel2016sentiments} that, in English, liars tend to use..." 

In 53, it is the same case but in relation to different languages. We have included a 'respectively' to make clear that each paper relates to one of the languages.

(as reported in \cite{zs08}, \cite{sm10}, \cite{SchMer2010}, \cite{fp13}, \cite{avc12}, \cite{hauchetal2015} and \cite{mblsh12} respectively) 

8) Need a better argument why such dataset splitting is used? Using such a split method the results depend a lot on splitting. If you would use the cross-validation with some k-folds numbers, the real accuracy in an experimental investigation would be seen. Now it more about luck. The obtained results by splitting by themself can lead to the wrong accuracy. At least give a better explanation of why it has been done like that.

The reference of training and testing cuts has been deleted because this split is only important for the Lexical-based SA experiment. For the ML experiment a 80%-20% evaluation was intended to help in comparing confusion matrices and a cross-validation was also made and it is explained in the text. As explained later, we have added the 10 cross-validation  accuracy in Table 5 to help understanding.  

9) Need to specify or in Table 1 are given pre-processed text lengths or not.
Yes, we add the information in Table 1 that are words.

10) Table 2 with so many citations looks not acceptable in my opinion, it could be redesigned. 

The goal was again to sum up the information and to visualize the consensus about the occurrence of negative words in deceptive texts. We have distributed the references in the corresponding following sections and we have deleted the table. 

11) Need an argument why SVM is used. There a lot of research where various machine learning methods are used in the sentiment analysis (SVM, Naive Bayes, trees-based algorithms, LSTM, etc.). In your case you have a binary classifier, so maybe it could be the reason, but anyway it is hard to say that this classifier is the best. From personal experience, I would say the other methods could give better results.

We have tried with other methods and SVM was the best one by far.  

Naïve Bayes, F1 was 0,727
J48 Decision tree F1 0,57
For LSTM our corpus is too small, as justified in the paper. 

12) The experimental investigation performed using a small amount of dataset, so it would be easy to add at least two other methods. But authors have to decide by them-self it is needed or not. In my opinion, it would significantly improve the paper.

Yes, the experimental part is quite limited, but note that Goel et al. or Almela et al. also used SVMs to build ML classifiers. 

13) If you made cross-validation, Table 5 would show the real situation of your classifier. 

We added the cross-validation accuracy in the table as well. 

14) The experimental investigation is limited, the dataset is very small, but specific, for the Spanish language, maybe in this case it is a novelty. Need to highlight it more from the beginning of the paper. 

Yes, thank you. We have revised the introduction and highlighted the novelty of being Spanish texts at the very beggining. 

Round 2

Reviewer 3 Report

Thank you for your answers, but I still see some mistakes and have some suggestions:
1) Yes, the authors changed the repeated institution and department, but near names of the authors left 1, 2, 3. There is no 2, 3, so need to fix it.
2) The keywords must be in the lowercase, it is not a name or title.
3) In line 18 (and in other places too) author is writing ([1]), in another place same situation (line 27) just [2]. No need to use brackets in the citation, just citation. Need to fix it in the all manuscript.
4) There is any reason why sometimes is used ", in another place ' ? Lines 91-92.
5) Lines 112, 114, sometimes 68,5 %, another line 81.8. Need to revise all manuscript places, where using point instead of a comma.
6) I still think it is a bad idea, to have such paragraphs like from 3.1 to 3.3.3, where just 2-3 sentences. Better to marge some of them to one. Now it is useless to separate them.
7) Some references do not meet the formatting requirements (1, 7, 10, and others).
8) I suggest authors read their manuscript one more time closely.

Need to make the final corrections.

Author Response

Thanks again for your careful review of our paper. We really appreciate your effort to help us to improve the paper. 

1) Yes, the authors changed the repeated institution and department, but near names of the authors left 1, 2, 3. There is no 2, 3, so need to fix it.
Sorry about this ... now fixed. 

2) The keywords must be in the lowercase, it is not a name or title.
Fixed.

3) In line 18 (and in other places too) author is writing ([1]), in another place same situation (line 27) just [2]. No need to use brackets in the citation, just citation. Need to fix it in the all manuscript.
Fixed.

4) There is any reason why sometimes is used ", in another place ' ? Lines 91-92.
Only ' now. 

5) Lines 112, 114, sometimes 68,5 %, another line 81.8. Need to revise all manuscript places, where using point instead of a comma.
Fixed. 

6) I still think it is a bad idea, to have such paragraphs like from 3.1 to 3.3.3, where just 2-3 sentences. Better to marge some of them to one. Now it is useless to separate them.
Ok, we have deleted the subsections and numbered the different phenomena, First,...  

7) Some references do not meet the formatting requirements (1, 7, 10, and others).
We have fix the bugs and we have followed the guidelines in https://0-www-mdpi-com.brum.beds.ac.uk/authors/references 
8) I suggest authors read their manuscript one more time closely.
Yes, we did and we found things to fix. Thanks!