A Comprehensive Framework for Analyzing IoT Platforms: A Smart City Industrial Experience
Round 1
Reviewer 1 Report
Dear authors,
The article is relevant and interesting, reporting a substantial piece of work.
The selection criteria, in particular, are interesting.
There is however a number of improvements, which would be desirable to undertake prior to publication, and which might require a minor second round of reviews.
First, the article is very long, even accounting for its nature (review, tables...). This is detrimental to the usability of this article, as it buries the most relevant parts. A reduction of 10% would be good, but at least a couple of pages should be easily achievable. In particular, there are some repetitions (e.g., regarding the need for such an article), and too many not-so-relevant details (e.g., regarding the backstory), which would be better conveyed in a shortened version.
I suggest moving section 5.3 "Related work" towards the introduction, which is more conventional, and will also help the shortening efforts.
Figure 1 would keep a better quality in a vector graphics format. Watch out for the red spellcheck mark under "IoT-PA".
Page 7, SQ2 looks strange. Indeed, the search "Smart city IoT" is overlapping with "Smart city" and "IoT".
Page 7, "published between 2008 and May 2019". Why those specific dates? Any relation to Table 4 "Timeframe"?
Page 7, lines 261-262, check grammar for "criterion should be ... is an indicator".
Table 3: the title of the column “Resource” is truncated and should be “Resource discovery”.
Table 3: layout problem. The right-hand side of the table is truncated after the "Mobility" column.
Table 3, and other tables: I suggest keeping only the tick √ and remove the crosses × (leave the cell empty) to make it easier to spot the positive features. Should a cell be undetermined (not the case here), use e.g. a question mark.
Table 3, and other multi-page tables: Repeat the header line at the top of a new page.
I am not convinced a binary classification (pass √ vs. fail ×) adequately captures the features of an IoT platform. Some kind of grade (e.g., from 0 to 5) would have been closer to the reality, by arguably more time-consuming to produce.
Then I have picked a few examples (some for which the information is easy to find online) to verify the content of Table 3.
Page 8, FIWARE/OASC [S12] has a × (fail) for "reusability", although this platform seems to pass all the criteria for "reusability" written in page 17, being a collection of various reusable services, each of which can be reused independently https://www.fiware.org/developers/catalogue/ Likewise for several other fails such as "resource discovery", which seems to be the role of the Context Broker https://www.fiware.org/wp-content/uploads/2016/12/3-Day-13-Developers-IoTBroker.pdf
Page 9, taking the example of the "SmartSantander" platform [S54], there is a × (fail) for visualisation. However, it seems to me that this platform does have some good visualisation features, as exemplified online https://maps.smartsantander.eu.
Page 9, another quick verification using the example of CityPulse [S39], there is a × (fail) for storage, although the platform uses an Elasticsearch back-end for storage (cf. https://github.com/CityPulse/IoT-Framework ).
It worries me a bit for the rest of the assessments in Table 3, and although I can understand that some of the assessments can be subjective, I am afraid there might be some kind of bias (towards a specific type of platform maybe) - which I do not believe is intentional. It might boil down to my point higher up about the problem of using a binary classification, but still.
Page 10, line 293, "E1 suggested providing an online version of the framework". Good suggestion. It was unclear to me whether an online too has been published, and if yes, is it available to the readers of the article?
Page 11, line 338, check "THAT" in uppercase.
Page 12, lines 360-367. While this was mostly true many years ago, this description of SQL vs. No-SQL is not so accurate anymore and requires a little update. Indeed, major SQL databases have incorporated standard features for time series, big data with various forms of partitioning and scaling, clustering, parallelisation, storage of unstructured data, e.g. JSON data (including its indexing). Look for instance at PostgreSQL (current version is 13). Furthermore, some No-SQL systems have also incorporated some SQL-like features.
Table 6: layout problem at the top, page 16, 18.
Page 20: section 3.1.2 "Non-functional-related criteria". I do not seem to find a number of criteria, which I believe are quite essential. Either they should be added, or a careful explanation of their exclusion should be added.
- On premise / As a service: Can the platform be installed on a company's own IT infrastructure, or is it limited to an online cloud service?
- There seems to be no legal aspect. Compliance with local legislation is a must. For instance GDPR (General Data Protection Regulation) in Europe.
- Especially in the case when the platform is only available as an online cloud service, under which jurisdiction is it? Including third-party dependencies.
- Licencing: both for "on premise" and "as a service", under which licences are the components available? Are some Open Source? Especially important in order to mitigate the risks if the editor of the IoT platforms stops or modifies the service.
Page 20: section "Interoperability". Here, I think the ontologies are quite essentials. They are named in the section, but I cannot find a direct mapping in Table 6.
Page 22: "Maintainability" from Table 6 is not explained, while all the other fields have their little paragraph.
Page 24, line 656: double-check formula. Having twice "Weight(pi)" looks strange.
Page 33, line 909, check font size.
Page 35, line 996, check grammar for "constitute".
Author Response
Dear assistant editor and reviewer colleagues - Special Issue: Intelligent Edge Computing for Smart Cities (Smart cities, MDPI)
Thank you for giving us the opportunity to improve our manuscript smartcities-1103579 entitled A Comprehensive Framework for Analysing IoT platforms: An Industrial Experience submitted to the special issue Intelligent Edge Computing for Smart Cities of MDPI journal.
We appreciate the effort and time you spent handling the review process. We have revised the manuscript based on your suggestions and three reviewers. We see this revision as a significant improvement over the first version. Instead of making piecemeal changes in response to each concern, we have devoted significant effort to carefully integrate the reviewers’ comments into our proposed framework and reported findings. The comments from the reviewers have helped us tremendously in sharpening and clarifying the motivations, arguments, and contributions of our research. Based on the reviewers’ comments, we have also slightly changed the title of the paper to “A Comprehensive Framework for Analysing IoT platforms: An Smart City Industrial Experience”. In the following pages, we provide specific responses to each comment. We hope that the changes we have made are to your satisfaction. To illustrate how we have responded to each of your comments, we have ordered comments from reviewers based on the sequence in which we have addressed them in the revised version of the manuscript. Our responses and changes to the manuscript are detailed in the following pages of this response document.
Please note that in the second revision of the manuscript, our co-author decided to step out. Due to the large number of comments raised by the reviewers, we decided to the expertise of new co-authors in preparing this rejoinder and improving the manuscript.
For clarity, our revised manuscript and this response letter highlight the changes from the original manuscript in a blue font attached at the end of this document.
Best regards
Asst. Prof. Mahdi Fahmideh (corresponding author), on be half authors
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper proposes an innovative framework for qualitatively assessments of IoT architectures. The proposed approach calls on well-established research techniques.
The problem statement and motivation are convincing. An survey of related works in the literature is provided, though it does not seem exhaustive. The research methods is very rigorous. An application of the proposed framework to assess real IoT platform is also proposed.
Section 2 is verbose (in particular, sub-section 2.2). Much text is repetitive of concepts previously stated. It should be squeezed to provide essential information. No need to sub-sectioning.
Section 2.2's title is "Architecture evaluation frameworks". The reader would expect to see a list/comparison of frameworks for evaluating architectures. No framework other than AHP is mentioned. It is then not clear why authors opted on AHP. Which alternative processes might have been used?
I believe SQ2 query shown in Table 2 is not exhaustive. I understand that smart city is the main target of the article, as requested by the journal scope. Nevertheless, authors are trying to figure out very general (thus, not Smart City-related) functional and non-functional criteria to assess IoT platforms .
I believe that, for an exhaustive definition of criteria, other sectors like smart agriculture, smart industry, etc. should be taken into account. Indeed, at the end of Section 4 authors run a comparison involving very general purpose IoT platforms (AWS IoT Platform, IBM Watson IoT Platform, and Microsoft Azure IoT Platform)
Figures and tables are too often distant from the text referencing them. I suggest revising the whole paper to improve the figures/table positioning, which I believe is something the reader would benefit of.
Concerning non-functional requirement, from what I see in Table 3, they all are typical requirements of generic Software systems. My question is: was the effort worth it? I mean, could it be foreseen that non-functional criteria of IoT platforms perfectly match those of any other generic software system? Or, the other way round, are there software systems' non functional criteria that IoT platforms do not address at all? I guess this point need to be addressed in order to justify the work.
Section 5.1 is too wordy. Please, shrink it to essential facts.
Minor issues:
line 223 : MRs -> MCs
Table 1: expand the acronym DP before using it
Table 3: It overspans the page: some criteria are not shown. Please adjust
line 320: "...and to ..." -> "...to..."
line 606: "...whilst the former.." -> "...whilst the latter.."
line 666: "...in the assessing..." -> "...in assessing..."
Author Response
Please kindly find the attached file to this message.
Author Response File: Author Response.pdf
Reviewer 3 Report
Excellent study and description of how you applied AHP to your problem at hand.
The IoT world is certainly a bit funny and difficult to understand and your paper provides a significant help to decision makers. Please find attached a couple of comments to help you take it even further.
I believe some of my comments could also apply towards a future extension of your platform, where you could focus on other or subsets domains of IoT, eg, protocol layer. There are a lot of players in such field and making an informed decision that fits ones application is no easy job.
On the less positive side, I would have liked to see the tables correctly rendered in the paper and hope you ensure they can be fully read once published. I suggest even publishing them in some format that might be easier for consumption by the reader, eg, drive, onedrive, so on.
There are also a few minor typos throughout the paper and I felt a few sections could be shorter.
Comments for author File: Comments.pdf
Author Response
Please kindly find the attached file to this message.
Author Response File: Author Response.pdf
