Next Article in Journal
Miniaturized On-Chip NFC Antenna versus Screen-Printed Antenna for the Flexible Disposable Sensor Strips
Previous Article in Journal
IoT Network Security: Threats, Risks, and a Data-Driven Defense Framework
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluation of Low-Cost Sensors for Weather and Carbon Dioxide Monitoring in Internet of Things Context

1
Federal Institute of Education, Science and Technology of Rio Grande do Norte (IFRN), Parnamirim 59143-455, Brazil
2
Department of Civil Engineering, University of Minho, 4800-058 Guimarães, Portugal
3
Algoritmi Research Centre, University of Minho, 4800-058 Guimarães, Portugal
*
Author to whom correspondence should be addressed.
Submission received: 13 September 2020 / Revised: 14 October 2020 / Accepted: 21 October 2020 / Published: 23 October 2020

Abstract

:
In a context of increased environmental awareness, the Internet of Things has allowed individuals or entities to build their own connected devices to share data about the environment. These data are often obtained from widely available low-cost sensors. Some companies are also selling low-cost sensing kits for in-house or outdoor use. The work described in this paper evaluated, in the short term, the performance of a set of low-cost sensors for temperature, relative humidity, atmospheric pressure and carbon dioxide, commonly used in these platforms. The research challenge addressed with this work was assessing how trustable the raw data obtained from these sensors are. The experiments made use of 18 climatic sensors from six different models, and they were evaluated in a controlled climatic chamber that reproduced controlled situations for temperature and humidity. Four CO2 sensors from two different models were analysed through exposure to different gas concentrations in an indoor environment. Our results revealed temperature sensors with a very high positive coefficient of determination (r2 ≥ 0.99), as well as the presence of bias and almost zero random error; the humidity sensors demonstrated a very high positive correlation (r2 ≥ 0.98), significant bias and small-yet-relevant random error; the atmospheric pressure sensors presented good reproducibility, but further studies are required to evaluate their accuracy and precision. For carbon dioxide, the non-dispersive infra-red sensors demonstrated very satisfactory results (r2 ≥ 0.97, with a minimum root mean squared error (RMSE) value of 26 ppm); the metal oxide sensors, despite their moderate results (minimum RMSE equal to 40 ppm and r2 of 0.8–0.96), presented hysteresis, environmental dependence and even positioning interference. The results suggest that most of the evaluated low-cost sensors can provide a good sense of reality at a very good cost–benefit ratio in certain situations.

Graphical Abstract

1. Introduction

During the last years, collaborative sensing, a concept well described by the authors in [1,2,3], has been used in several application fields due to paradigm-breaking features such as decentralization, the possibility of enhancing the space-time granularity of a sensing system, reduced costs, and its capability of giving users the power to be a node and become part of a solution to a shared concern or common problem. In short, the main point in collaborative sensing is the use of low-cost devices handled by “non-professional” individuals—citizen scientists—in large quantities to overcome costs and to increase the density of nodes, forming a dense, and even pervasive, monitoring system. The Internet of things, in turn, contributes by providing a resourceful framework to allow participatory sensing to take place [4,5].
As the cost is a constraint to densifying environmental monitoring networks, the Internet of Things and Collaborative Sensing triggered the creation of complementary techniques to assist environmental monitoring in a creative manner. For example, the authors in [6] proposed a sensing campaign to elaborate a noise map of an urban area based on volunteers collecting data using their own smartphones, with a high granularity in space and time, instead of using conventional decibel meters. They obtained very positive results, even if comparing the results to traditional methods. Merging weather observation with the mobile sensing perspective, the authors in [7] proposed a model to estimate the air temperature by applying a mathematical model that used the inner temperature of a smartphone battery as input. They achieved a good overall result while also identifying the need for improvements to enhance the accuracy. The authors in [8] investigated the potential role of smartphones embedded with weather sensors, such as sensors of temperature, humidity and pressure, as mobile nodes in an environmental sensing campaign. They found that the smartphone can act reliably as a node for temperature and pressure only when idle, which requires an application for context detection. In the same context, the author in [9] also proposed a new model to enhance the quality of the data for temperature estimation in smartphones, but the use of smartphones for this purpose still requires an extended field test.
Expanding the considerations regarding the low-cost, technologically capable alternatives that address traditional problems through new approaches, it is worth mentioning the “CanSat” concept. Its main objectives include an increase in environmental awareness and the dissemination of the concepts and objectives involved in space studies among students, enthusiasts and citizen scientists. The CanSat relies on very small aerial probes, with the hardware built inside small cylinders (cans or bottles) to collect data during low-altitude launches by rockets or balloons [10,11]. These devices often carry connected low-cost environmental sensors inside, such as temperature, humidity and atmospheric pressure sensors commonly used in Internet of Things environmental instrumentation [12,13].
Another area of interest in environmental monitoring is the investigation, and monitoring, of urban areas to identify phenomena that may affect people’s health and welfare. A common climatic effect in dense urban centres is the heat island, a localized environment overheating that can increase temperature perception by individuals by up to 10 degrees Celsius [14]. Several studies highlight the importance of continuous temperature monitoring in urban centres to help the investigation of heat island phenomena and corresponding mitigation actions [14,15,16]. This task can also be achieved using low-cost sensors, as reported by the authors in [17].
In addition to the climate, air pollution in urban centres is also an important concern that attracts the attention of researchers, with projects based on low-cost sensing and the Internet of Things. For example, the authors in [18] developed an evaluation framework to classify a city’s air “health” in terms of the concentration of pollutants through a score model system using sensory data as input. This model could be fed by low-cost sensor systems such as the one reported by the authors in [19], which proposed a mobile node to collect data about the air quality in Sydney, Australia, using a low-cost platform boarded in public transportation vehicles, such as taxis and buses, or the IoT sensor for particulate matter described in [20].
The Internet of Things also has a close relationship with smart buildings, providing ways to optimize heating, ventilating and air conditioning (HVAC) systems with connected sensors measuring indoor air quality. Although CO2 is mostly known to be related to the greenhouse gas effect, not being treated as a pollutant, it is considered one of the main indicators of indoor air quality. Moreover, the application field for these sensors is not limited to air-quality monitoring alone. Once humans expel CO2 during breathing, the use of CO2 sensors—associated with proper data analysis, such as machine learning, for example—can detect room occupancy as well [21,22], which can assist in decision-making regarding energy efficiency actions. On the outside of smart buildings, green walls (or living walls) are a common asset used by modern architecture in order to mitigate heat absorption and reflection in buildings, helping to meet energy efficiency requirements and even soothe the heat island effects in the vicinity [23]. In this context, the authors in [24] proposed a solution for the automation of a green wall irrigation system using Internet of Things technologies in conjunction with low-cost environmental sensors and satisfactorily achieved a higher data density for the involved parameters, both in space and time, at the same price of a single system with implied superior quality.
Data quality is a challenge in environmental sensing with low-cost sensors. There is yet no consensus about the use of these cheap, largely available, and ready-to-use sensors as trustworthy devices, since many factors can affect the data quality of a given sensing system [25]. This is precisely what this paper proposes to address. In terms of accuracy, for example, even conventional methods can provide erroneous data [26]. Aware of these obstacles, the authors in [27] proposed a general model to help in evaluating the overall data quality in urban monitoring as a function of its “urban resolution”—a concept that they define as an analogy to “image resolution”, with sensing nodes being related to pixels. The challenge of deploying air-quality monitoring using low-cost sensors and obtaining high data quality at the same time was also observed by the authors in [28], which reported the use of these sensors as promising, but they acknowledge the difficulties in that task due to the technical details involved, such as electronic wiring, sensor calibration and data post-processing. Still regarding data quality enhancement, the authors in [29] presented a robust protocol for overcoming the presence of bad-quality data (e.g., outliers) in non-specific participatory sensing, where individual users may play active roles in the model improvement. The authors in [30] investigated a set of low-cost air-quality sensors from a specific manufacturer and compared their results to those of a reference instrument to check if they can contribute to air-quality measurements. They found that those sensors can achieve excellent results when used in the lab, but their performance decreases when they are placed in outdoor environments. This observation is corroborated by the reports of Borrego et al. [31].
Accuracy enhancement is a possibility to improve the data quality of a sensing system. In this context, the authors in [32] state that the placement of low-cost sensors around professional stations can provide useful resources for calibration and, then, overcoming the problem of data quality for NO2 sensors. The authors in [33] tried different machine-learning algorithms to improve the data quality from low-cost sensors deployed in the field and obtained very good results with the Random Forests algorithm. With similar objectives, the authors in [34] used official datasets generated from professional stations to correct eventual errors in low-cost temperature measurements using Artificial Neural Networks, and managed to correct the mean absolute error by more than 50%, regardless of the many interfering factors inherent to environmental monitoring. The authors in [35] managed to enhance the accuracy of low-cost atmospheric pressure sensors by more than 90% using the Extremely Randomized Trees algorithm.
All these studies can be related, in different ways, to the Internet of Things. Concerning the topic of environmental monitoring, it can be noted that it is imperative that the devices used for collecting environmental data are not only distributed and connected, but also providing accurate data in order to obtain useful information.
Commercially speaking, the expansion of the Internet of Things market, coupled with an increase in environmental awareness, has also contributed to attracting interest in the construction and use of low-cost devices in personal weather stations or even as a supplement to conventional environmental monitoring, which is very reliable but is centralized and significantly more expensive to deploy on a large scale (e.g., in urban centres). Thus, the widespread use of these low-cost environmental sensors in Internet of Things applications, while proving to be a convenient and inexpensive solution, also highlights the need for a close investigation on how these sensors behave under “Do It Yourself” conditions.
The contribution of this paper is to provide a closer look into the raw data of a group of “off-the-shelf” low-cost environmental sensors, subjected to experiments and assessed through objective metrics such as precision, accuracy and trueness. In other words, it is intended to provide information regarding the performance these sensors can achieve in practice when handled by non-specialist individuals building their own “IoT weather station” (such sensors are often sold as “do-it-yourself weather stations” by well-known online vendors) without any further refined data processing, such as machine learning. It is important to highlight that the sensors used in this study were chosen without commercial bias, trying to include a good diversity of brands and models. The tested environmental sensors include temperature and relative humidity sensors, evaluated in controlled conditions, and air pressure and carbon dioxide sensors, evaluated in natural conditions. The performance indicators were obtained by statistically analysing the sensor readings against reference readings, for temperature, humidity and carbon dioxide, and between sensor pairs, for the atmospheric pressure sensors.

2. Materials and Methods

The research approach included a procedure for selecting the sensors for analysis, the planning of the experiments, data collection, data analysis and the discussion of the obtained results, as discussed in the following sections.

2.1. Sensors

The sensors were selected using criteria that an individual would have when engaging in an environmental monitoring project on the Internet of Things. The sensors were chosen considering availability (available in known vendor websites; reachable by a simple internet search), price, ease of use (sensors built in ready-to-use interface boards for Arduino) and nominal parameters suitable for urban spaces with no extreme conditions (trying to cover an “average” situation). In addition to these criteria, the selection of sensors did not favour any specific commercial brand. The selected sensors for climatic variables (temperature, humidity and pressure) are presented in Table 1. The selected sensors for carbon dioxide are presented in Table 2. Three units of each climatic sensor were used, as well as two units of each carbon dioxide sensor.
The climatic sensors the carbon dioxide sensors used in this study are illustrated in Figure 1. The reference sensor used for temperature and humidity was the Lascar Electronics EL-USB-2, with a certificate of calibration available in [36]. The reference sensor used for carbon dioxide was the Vaisala GM70 with a CO2 probe. Both reference instruments are illustrated in Figure 2.

2.2. Performance Metrics

The sensors were evaluated considering three main metrics: accuracy, precision and trueness [37]. The accuracy assessment of the sensors was performed by the analysis of the mean error (ME) and the root mean squared error (RMSE) at different measurand levels. While the mean error provides information regarding the sensor bias, the root mean square error provides a more comprehensive view regarding the sensor accuracy.
As the precision is defined by the statistical dispersion of the measurements of a quantity in similar conditions, the standard deviation computed from each sensor’s readings at stable and reproductible levels was used as a precision metric. Additionally, to illustrate the impact of this dispersion at each stable level, the signal-to-noise ratio (SNR, or the inverse of the variation coefficient, cv−1) was also considered. This parameter, in turn, was computed from the averaged readings of a sensor divided by its standard deviation at that measurand level. A precise sensor would present a standard deviation near to zero and a very high numerical value for the SNR (cv−1).
The information regarding the trueness of the sensors was obtained in terms of the determination coefficient (r2) and the analysis of the residuals in the linear relationship between all the observed and expected values during the quantity variations. The expected output for a hypothetical ideal sensor would be a straight line from the bottom-left to top-right corner, with a y = x relationship. To estimate this metric, we applied a residuals calculation to the linear relationships. For convenience, this metric is referred to as the dynamic residuals (DR) hereafter, and is defined by Equation (1) (with the boundaries determined by the temperature experimental limits):
D R = 1 Δ T T i T f [ y ^ ( x ) y ( x ) ] 2
where y ^ (x) is the averaged linear model output of a given sensor; y(x) is the expected, or ideal, output (y = x); Tf is the final temperature; Ti is the initial temperature; and ΔT is the temperature variation. The interpretation of this parameter is trivial: it is the root mean squared distance between two lines in a defined range, and the closer to 0.0 the value of DR, the better the performance of the corresponding sensor.

2.3. Experimental Protocol

A brief description of the experiments for each group of sensors and equipment used, as well as the reference instruments and expected outcomes, is presented in the following subsections. This description is provided here mostly to enable the reproducibility of these experiments. The experiments were conducted in two stages: one for climatic sensors (Section 2.3.1) and another one for carbon dioxide sensors (Section 2.3.2).

2.3.1. Climatic Sensor Experiment

The experiment for climatic sensors was conducted inside a controlled climatic chamber (Aralab© Fitoclima®) reproducing combinations of temperature and humidity. Two experimental profiles were planned. The first experimental profile had four temperature stages—−5, 10, 25 and 40 °C—and three humidity levels applied in each stable temperature stage: 30%, 50% and 80%. Each temperature stage lasted for eleven hours in stability, and one hour in transition to the next level. Considering that the chamber was unable to control humidity below 0 °C, the second profile was programmed only with positive temperatures—each one with three humidity levels: 30%, 50% and 80%. Each combination of temperature and humidity lasted for five hours after stabilization, each humidity level change lasted for thirty minutes, and the temperature level changes lasted for one hour. Both experimental profiles were executed three times, containing two sets of sensors at a time.
Each sensor group was formed by six different sensor units and was wired into an Arduino Uno device, powered by an external and independent power supply. The Arduino device also had an SD shield attached to it for data logging, and the RTC (Real-Time Clock) service available for time synchronization. The sampling rate was set to one sample per minute. Figure 3 shows a group of climatic sensors (temperature, relative humidity and atmospheric pressure) wired into an Arduino outside and inside the climatic chamber.
Between executions, the generated data were extracted from the SD cards, and one set of sensors was replaced (one set was kept at the end of each execution). Considering 3 sets of sensors, denominated as “A”, “B” and “C”, the pairs subjected to the experiment were made by combinations of two out of three, resulting in three executions to which each set of sensors was subjected twice. The same logic applied to the experimental profile focused on the humidity sensors.
Regarding the atmospheric pressure sensors, their readings reported the natural external conditions during the experiments, since the used chamber did not control its internal pressure, nor was it hermetically sealed.
The evaluated sensors do not differ in terms of the transduction principle for each measurand.

2.3.2. Carbon Dioxide Experiment

Due to the lack of equipment able to control CO2 levels in a controlled chamber, the experiment to assess these sensors was designed to take advantage of the natural conditions in indoor environments with known human occupancy. The experimental plan for carbon dioxide was to place the sensors together with the reference instrument into these environments. Two different indoor environments were planned: an office room with ventilation control during working hours, an approximated volume of 125 m3 and random occupancy during the daytime (a minimum of one person and maximum of 10 people at the same time); a bedroom during the night time, without ventilation, and with an approximated volume of 30 m2 and two occupants during the night time.
The first situation was planned to verify random variations, as people might enter and leave the room at any time. The second situation was planned to verify the sensor response to higher concentrations, since the CO2 can accumulate over time, especially in the absence of ventilation systems (which is a common situation in many residential buildings), as assessed by the authors in [38]. Figure 4 illustrates one unit of each of the tested sensors and the reference instrument.
It is relevant to mention that the carbon dioxide sensors in question use different transduction principles: while the MG-811 is a metal oxide semiconductor (MOS) sensor that needs to heat its sensitive layer, the MH-Z16 uses non-dispersive infra-red (NDIR) to measure the quantity of energy absorbed by CO2 in a specific wavelength. Those sensing mechanisms may imply different performance expectations, as reported in [39,40].

3. Results and Analysis

3.1. Temperature Sensors

An example of one (out of six) experimental output from the temperature sensors is presented in Figure 5. Both accuracy and precision analysis were performed separately for each stable temperature level for all experiments, and considering all sensor units, by using data slices in the regions of interest. The accuracy metrics obtained by averaging all the execution results for the temperature sensors are presented in Table 3.
The farther away from zero these values are, the lower the accuracy of the respective sensor is. These numbers demonstrate that HTU21D presented the lowest overall mean error, whilst the BME280 presented the highest. However, variations in the mean error for different temperature levels may indicate a heterogeneous accuracy along the evaluated temperature range. As the exposed values were averaged from individual sensor unit readings from all the executions, the consistency of the error is represented by the standard deviation of the observed mean errors (σME), for which lower values represent more homogeneous mean errors from different sensor units in different experimental executions (e.g., a constant error from all units of a sensor model would result in σME = 0—in other words, an error purely systematic). On this, it can be noted that the most consistent sensor model was the MPL3115A2, followed by HTU21D. The least consistent sensor models were the DS18B20 and the AM2302.
Expanding the considerations to professional requirements, the World Meteorological Organization (WMO) points out an ideal accuracy of ±0.5 °C in its guidelines for conventional methods for weather observation [41]. Despite all the manufacturer statements, the only model that would meet this requirement, in a “plug-and-play” condition, was the HTU21D. Nevertheless, all the other sensor units would meet the mentioned requirement without greater efforts when using some form of data adjustment.
The computed data about the precision of the temperature sensors are presented in Table 4.
The highest standard deviation was observed at the 40 °C level, where all the sensors exceeded the 0.1 °C mark. This occurrence may be explained by the thermal noise, intrinsically related to the semiconductors. The WMO, regarding precision, recommends a value of 0.2 °C for temperature measurements. It can be concluded that although the precision of the investigated sensors decreases with increasing temperature, they can meet the WMO’s requirement at moderate temperatures. An interesting fact that can be observed in these data is that the sensors do not significantly differ in the values of the standard deviations, as they showed very similar values along the investigated range. This suggests that all the sensor models were affected by noise with similar intensity.
The information presented so far considered the static behaviour of the sensors. However, the performance of the sensors during the temperature variations is also of interest. Scatter plots generated with datapoints during temperature changes are presented in Figure 6, with aggregated data separated by sensor model.
All sensors presented the expected linearity with no anomalies. However, the obtained relationships, which were also used to obtain the dynamic residuals (DR) of all the sensor units (see Table 5; the r2 values were suppressed, since they all exceeded the value of 0.99), demonstrate that some sensor models had a subtle difference—although not critical—in slope and in the intercept point, if compared to the ideal behaviour (y = x).
It is important to emphasize that this metric, if analysed alone, has no descriptive value, as some situations may lead to low DR values, such as—for example—a combination of positive bias and response delay. The DR values were considered as very satisfactory for the AM2302 and BMP180 sensor units. The DR value for HTU21D can also be considered satisfactory, but marginal. The DR value from the other sensor models exceeded 1.0 °C. In terms of sensitivity, it is related to the slope of the averaged linear model. The obtained slopes were near to 1.0 for all the sensor models (minimum of 0.970, for AM2302; maximum of 0.999, for BMP180). In this respect, it can be asserted that these sensors do not present issues regarding their sensitivity.

3.2. Relative Humidity Sensors

The analysis of the humidity sensors required additional attention since their data are more segmented, as diverse combinations of temperature and humidity levels were programmed. Figure 7 illustrates the readings from the humidity sensors inside the controlled chamber during a run of the second experimental profile.
The expected humidity levels were 30%, 50% and 80%. However, it can be noted—from Figure 7—that the reference sensor provided different values at certain moments (e.g., 37% at Hour 28). This reveals that humidity control with different temperature levels is a complex task, even for certified devices.
The accuracy indicators were obtained for each combination of humidity and temperature. The results for the ME and for error consistency ( σ M E ) are presented in Table 6, and the obtained results for the RMSE, with individualized information, are presented as a colourmap in Figure 8.
Regarding the mean error, the higher overall agreement (lower ME) between each sensor model and the reference was observed for the humidity level of 80%. Considering the WMO accuracy recommendations—1% at higher values of relative humidity (80% or more) and 5% at moderate values of relative humidity [41]—the evaluated sensor models reached different levels: the AM2302 met the requirements at all levels, except at 80% at 10 °C, where it exceeded the value of 1%; the HTU21D did not meet the accuracy requirement at any point analysed; the BME280 met the requirement at 30% and 50% levels but, at 80%, only met it at the temperature level of 10 °C. Regarding the error scatter, the lowest value of σME was observed for the HTU21D sensor (1.8%), and the highest value for the AM2302 (2.7%). These values were not considered critical but should not be ignored either, since they are indicative of random error (used for precision metrics).
The precision metrics of the humidity sensor are presented in Table 7, which contains the standard deviations averaged from all sensor units’ outputs.
The overall precision exhibited very satisfactory results, even if compared with the WMO’s requirements (5% at mid-range humidity).
The trueness of the humidity sensors during relative humidity variations was also assessed. Figure 9 contains the scatter plots, with all the datasets from reference and the sensor readings (in grey, all individual readings; in colour, the averages). The respective averaged linear models are presented in Table 8, which also presents the calculated values for the DR.
The grey points scattered around the averaged dataset, in Figure 9, indicate that these sensors were more susceptible to noise during the quantity variations. Regarding the sensitivity, all the sensors presented slope values above 1.0. However, a more significant case is observed for the AM2302 sensor, whose slope exceeded the ideal value by 12%, suggesting that this sensor is more affected by errors than the other models since it was overestimating the relative humidity variations. The DR values agreed with the mean error values observed in static conditions (see Table 6), pointing out—thus—that there was no difference, in terms of overall error magnitude, if these sensors were reading static or dynamic values of relative humidity.
Despite all the observed issues regarding the accuracy of these sensors, either in dynamic or static conditions, there was no evidence that a linear calibration would not produce satisfactory outcomes in a data quality enhancement stage.

3.3. Atmospheric Pressure Sensors

The evaluation of the atmospheric pressure sensors relied on an inter-comparison between the sensor units. Figure 10 contains the atmospheric pressure sensor readings during one experimental execution.
To assess the reproducibility of the barometric sensors, the readings from all the experimental executions were grouped into a single dataset. The comparison was then made between different units of the same sensor model that were used at the same time and under the same experimental conditions (two units, out of three, at a time). The outcomes for these sensors are presented in Figure 11, which illustrates the obtained scatter plot for individual sensor models, and in Table 9, which presents the numerical results of the inter-comparison (the r2 values exceeded 0.99).
In terms of inter-correlation, the linear regression closest to the ideal (1:1) was observed between Units A and B for the BMP180 sensor, the A and B sensor units for the BME280, and between A and C for the MPL3115A2 sensor. However, the RMSE complements this analysis, since its value denotes how good the presented regressions are to describe the pairs of datasets. In this case, both the lowest and highest RMSE values occurred for the BME280 sensor: the lowest value between Units A and C, and the highest value between Units A and B. The observed values point out that the reproducibility of the pressure sensors may be considered moderate to satisfactory for applications where the uncertainty is not critical.

3.4. Carbon Dioxide Sensors

Three experimental executions were performed, two of which took place in an office room (the first and third ones), interspersed with one in a bedroom. The different environments create different expectancies regarding the observed concentration of the gas in natural (non-controlled) conditions.
Whereas an additional step was necessary to extract information from the raw readings of the metal oxide sensors, this section includes two subtopics: one that addresses the issues observed about the calibration of the MOS sensor (Section 3.4.1), and another that presents the results (Section 3.4.2).

3.4.1. Metal-Oxide Sensor Calibration

A detail about the MOS sensor (MG-811) is that it did not provide its readings in values of gas concentration (ppm) but rather in raw voltage values that needed to be converted into ppm. The sensor manufacturer made available an approximated conversion equation, which we observed to be significantly inaccurate. This led to the necessity of a post-conversion of individual sensor data using the reference readings. Figure 12 depicts the nominal conversion curve and the observed calibration curves using data from the first execution in the office room. Additionally, Unit A of the MOS sensor presented a very compressed output, if compared to Unit B: 15 mV of signal excursion, against 40 mV. This, associated with the different baseline values, resulted in very different calibration curves, where the coefficients differed even by an order of magnitude. It was also found that the physical orientation of the sensor affects both its baseline and its sensitivity (the standard orientation used was with the sensitive layer horizontally positioned). Consequently, sudden movements may also affect its readings. In advance, it can be concluded that this sensor model is not feasible for air-quality mobile sensing platforms such as, for example, those boarded on public buses, bicycles, taxis, etc.
Moreover, the calibration of the MOS sensor was found to be very unstable, as each individual sensor required a new conversion process each time it was powered on. This information can be observed in Table 10, which contains the conversion equations obtained from each experiment (the first and third occurred in the office room; the second, in the bedroom).

3.4.2. Results

Figure 13 illustrates the timeseries from readings taken in the office room (first and third executions, chronologically) with mixed occupancy and a ventilation system during the daytime. Figure 14 illustrates the measurements obtained in the bedroom, where it is possible to notice two regions with higher concentrations: the first, from Hour 6 to 18, which corresponded to the occupancy of two persons during night and sleep time; the second region, after Hour 32, corresponded to mixed occupancy with only one person staying during sleep time.
It could be noticed in the data that both units of the MOS sensor had different convergence times (even after the nominal warm-up time), which, in turn, we defined as the time required for the sensor to enter at the first time—and stay—at a maximum distance of 100 ppm from the reference. An example can be seen in Figure 13a, where Unit A of the MOS sensor (in blue) took about 20 h to have its adjusted readings enter the stipulated margin of convergence, whilst Unit B (in green) reached the convergence much earlier: after 4 h. The difference in performance between the metal oxide sensor units might indicate a very poor reproducibility in terms of sensor manufacturing. In terms of calibration, the assessed NDIR units had a straightforward one-button self-calibration process, where the user presses the button while the sensor is exposed to fresh air for a few minutes (where the average CO2 levels are, approximately, 400 ppm). The auto-calibration procedure for these units was performed once before the experiments, in clean air, to verify its robustness (or persistence) over time of use. The accuracy indicators of these sensors are presented in Table 11.
The NDIR sensors presented the best overall accuracy, with very satisfactory results in the first experiment (just after the calibration), and we consider—in advance—this specific model as suitable for citizen science and IoT applications. However, they presented higher errors in subsequent experiments. A more significant bias was found in the last measurement, which occurred in the same conditions as the first measurement, but on different days. It suggests that this sensor unit (Unit B) may be affected by monotonic drift with time (one-directional increasing bias). The MOS sensors presented satisfactory results in the last measurement (the office room experiment repetition), but only with a limited time window due to its stabilization time, which, in turn, is heterogeneous, and it is hard to predict its length before the beginning of the sensor use.
The linearity of these sensors was also evaluated through scatter plots containing the datasets from the sensors and reference. The plots are presented in Figure 15 and Figure 16, for the measurements inside the office room and in the bedroom, respectively. The data containing the linear relationship between the sensors and the reference are presented in Table 12.
The NDIR sensors presented good linearity, with r2 values above 0.97 for all measurements. In terms of sensitivity, they showed a moderate behaviour, with an average slope of 1.11 (minimum of 1.03 and maximum of 1.17). The subplot (b) in Figure 15 clearly depicts a monotonic drift of Unit B of the NDIR sensor, considering that the slope of this device was kept almost the same during the measurements and only the y-intercept point of the linear regression was increased. This phenomenon was also present but less evident in Unit A of the NDIR sensor model MH-Z16.
Summarizing the MOS sensor outcomes, Unit A presented inconsistent stabilization times, making it difficult to establish a linear relationship of its converted output with the reference (its scatter plots was formed with part of its output, only). Considering the stabilized readings, they demonstrated a satisfactory linearity, though. However, the outputs illustrated in Figure 15 and Figure 16, associated with the linear relationships presented in Table 12, led us to assert that this sensor unit also presented issues regarding its sensitivity.

4. Discussion

The evaluated low-cost climatic sensors, subjected to different scenarios inside a controlled climatic chamber, presented a good cost–benefit ratio, considering that most sensors are very cheap and yet present satisfactory performance indicators. However, in terms of accuracy, there was evidence suggesting that these sensors are not calibrated in a full-range manner from the factory, and this condition may mean they do not meet professional accuracy requirements for environmental monitoring, if used in decentralized volunteer-based campaigns, for instance. On the other hand, polluted readings or bad-quality data are not unique to low-cost sensing, as even professional monitoring may provide erroneous or biased readings due to several factors, including the sensors themselves, as discussed by the authors in [26] and in [42].
Regarding the temperature sensors, the best overall performance, in terms of both accuracy and precision, was observed for the HTU21D, and the closest approximation to the reference during dynamic conditions was observed for the BMP180, which presented the lowest value for the DR metric (0.25 °C). The worst case for accuracy was observed for the BME280 and MPL3115A2 sensors, with the error always above 1.0 °C, exceeding their nominal parameters (as stated in the datasheets). In dynamic conditions, the worst performance was achieved by the BME280, with a DR value of 1.9 (perhaps due to its bias), almost double that of the second-worst sensor—the DS18B20, with 1.0. Regarding the precision of the temperature sensors, all the sensors were found to be very precise. In a more general conclusion, we observed that temperature sensors are more likely to present systemic errors (bias) than random errors (noise). Additionally, as the sensors presented good linearity and sensitivity (see the slopes of the linear relationships presented in Table 5), they could detect, very satisfactorily, variations in temperature.
The humidity sensors, in general, demonstrated moderate accuracy and precision. A good linearity was observed, as well as a very strong correlation (trueness) between the sensors and the reference instrument. The slopes of the linear models obtained from the data indicate that these sensors can follow daily humidity variations of the same magnitude considering the experimental boundaries of 30% and 80% air relative humidity. The best overall performance amongst the evaluated sensors was achieved by the BME280 humidity module. These observations can be complemented by the results reported by the author in [43], who found that low-cost humidity sensors using capacitive transduction may present satisfactory results even in extreme conditions of very low temperatures and low vacuum. However, these sensors can also present significantly biased data, as observed for the HTU21D sensor model. Systematic errors can be easily corrected, but the challenge is to identify when the sensor is biased or not. When the sensors are managed by lay individuals, this can be a hindering factor for obtaining high-quality data in the context of citizen science. As a suggestion, when a reference is unavailable for co-location, one can compare the readings from a sensor with the official weather information of the local provider over time and check if the low-cost sensor is following the daily pattern according to the official data. The presence of the random error (noise) was more pronounced in the humidity sensors, yet it did not prove to be critical in relative terms.
The verdict about atmospheric pressure sensors lies in their reproducibility. The obtained results from the inter-comparison of the sensor units allows us to conclude that the evaluated sensors present very good reproducibility. Another finding was an eventual cross-interference from temperature and humidity in the sensor readings, since the deviation observed between the BME280 and BMP180 readings seemed to follow a pattern. In practice, such an issue can be overcome by using machine-learning algorithms using multivariate data to model the error, as demonstrated in [35].
The positive findings about the utilization of the low-cost climatic sensors can be strengthened if the application presented by Mwangi [44] is considered: the author built weather stations with low-cost sensors that were later tested and assessed at NOAA (National Oceanic and Atmospheric Administration of United States of America) facilities, and then managed to deploy them in hard-to-reach sites in Kenya. Moreover, these weather stations used two sensor models that were investigated in this paper: the HTU21D and BMP180 models. Another example of work with interesting achievements using low-cost climatic sensors is the “CanSat” design, launch and data collection experience reported by Colin and Jimenez-Lizárraga [12], which used—with no issues mentioned—the MPL115A2 sensor, a previous version of the MPL3115A2, used in this paper.
In relation to the carbon dioxide sensor outcomes, they led to different conclusions in our assessment in the indoor environment under natural conditions (an expected outcome, since they use different transduction principles). While the NDIR sensor (MH-Z16) showed acceptable accuracy and good linearity, the MOS sensor (MG-811) presented, generally, poorer performance due to its unpredictable stabilization time, inaccurate nominal conversion curve, hysteresis, and higher bias and even the significant interference from physical movements and orientation (both baseline and sensitivity were affected by the physical orientation of the sensor). None of these issues are mentioned in the sensor datasheet.
The analysed data show that MH-Z16 is a ready-to-use sensor with built-in calibration suitable for indoor monitoring. In brief terms, a USD 50 device showed very similar performance to a USD 2000 certified instrument (Vaisala), proving to be a cost-effective device. The MG-811, in turn, considering the options that a citizen scientist would have, for instance, has its application list reduced to basically a carbon dioxide detector, since even movements and physical orientation affected its readings. This sensor may be used in fixed spots, with a stable power supply and with no sudden temperature variations (e.g., exposure to cold wind streams), so it can serve as a complementary asset for indoor surveillance or human presence detectors [21,22,45]. Without a reference being co-located with this sensor, the MG-811 alone is unable to provide information. Accordingly, it can be said that NDIR sensors can achieve superior performance when compared to MOS sensors for carbon dioxide monitoring in the Internet of Things scope. However, the monotonic drift of NDIR sensors still necessitates a very long-term investigation to extract conclusive information on whether they can play an important role in the assessment of climate change, for example, especially in the monitoring of carbon dioxide levels in the atmosphere, which requires constant and reliable measurements over years.
It is convenient to emphasize that although many authors undertake relevant contributions in the field of low-cost environmental sensors, many of the observed contributions are mainly scenario analytics [46,47], field applications [5,17,48,49] and advanced techniques for improving sensor data accuracy [33,34,50]. In view of these observations, this work addressed a little-explored issue when analysing the behaviour and performance of low-cost environmental sensors from the perspective of an individual engaged in using the Internet of Things to build their own platform, justified by the fact that these individuals may not have access to data calibration tools, such as reference sensors or machine-learning algorithms, and yet they remain interested in building and keeping their weather stations running and providing data, either due to particular motivations or by engaging in collaborative sensing campaigns.

5. Conclusions

This paper presents an original experimental study on the performance metrics of sensors commonly used in instrumentation within the scope of environmental monitoring using the Internet of Things, providing a statistical analysis of the raw data obtained from the sensors exposed to scenarios designed to obtain as much performance information as possible (the datasets are given in the Supplementary Materials section). The evaluation included different low-cost environmental sensors for temperature, humidity, atmospheric pressure and carbon dioxide that can be easily obtained at affordable prices from recognized online stores, making the experimental setup easily reproducible for anyone who is interested.
As findings, it turns out that while some of these sensors are ready-to-use devices and can present reliable readings even without previous assessment, in terms of being able to provide a good sense of reality at a very low price, other sensors presented issues regarding accuracy and sensitivity. We also found that the information contained in the sensor documentation sometimes might be partially inaccurate. As an example, we could mention that regarding the temperature accuracy in the datasheets of the BME280 and the MPL3115A2. The nominal levels of accuracy were claimed to be ±0.5 and ±1.0 °C, while the average RMSEs observed were 1.7 and 1.2 °C (from −5 to 40 °C), respectively. Considering the MG-811 MOS sensor, the lack of details in its datasheet was found to be much more critical. This sensor would be completely useless without a reference to calibrate it, since its nominal conversion curve was proven to be incorrect, as well as its susceptibilities to interferences not being stated clearly in the document.
Moreover, to fully meet professional requirements, such as those proposed by the WMO, the individual calibration of these sensors is desired. This suggests that, without efforts in individual sensor calibration, citizen science and Internet of Things approaches for environmental monitoring cannot be compared to conventional methods yet, due to producing biased readings (in temperature monitoring, for instance).
If it was necessary to choose the best sensor for each quantity in “from factory” conditions, from the evaluated ones, we would recommend the HTU21D for temperature, the BME280 for humidity and atmospheric pressure, and the MH-Z16 for carbon dioxide. However, as mentioned, if the operation requires a high-end specification beyond the Internet of Things scope, actions must be taken to obtain higher-quality data information, such as machine-learning methods for real-time calibration.
Aware of the limitations of this work, we can mention, as future work, further investigations to cover the aging of these sensors through the persistence of performance indicators in their long-term use and the robustness of a calibration curve (for the case of metal oxide sensors). In addition, it is also desirable to evaluate the behaviour of climate sensors in an external environment, when they are exposed to natural conditions where the measurements from them can change unexpectedly.

Supplementary Materials

The following are available online at https://0-www-mdpi-com.brum.beds.ac.uk/2624-831X/1/2/17/s1, the datasets containing the sensor readings used to support the findings of this study have been deposited in the Zenodo repository, with the digital object identifier (DOI) “10.5281/zenodo.3560299”.

Author Contributions

Conceptualization, T.A., L.S. and A.M.; data curation, T.A.; formal analysis, T.A., L.S. and A.M.; funding acquisition, L.S. and A.M.; investigation, T.A.; methodology, L.S. and A.M.; project administration, L.S. and A.M.; resources, L.S. and A.M.; supervision, L.S. and A.M.; visualization, T.A.; writing original draft, T.A.; writing review and editing, T.A., L.S. and A.M. All authors have read and agreed to the published version of the manuscript.

Funding

This work was been supported by FCT–Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

References

  1. Burke, J.; Estrin, D.; Hansen, M.; Parker, A.; Ramanathan, N.; Reddy, S.; Srivastava, M.B. Participatory Sensing. In Proceedings of the WSW’06 SenSys ’06, Boulder, CO, USA, 31 October 2006. [Google Scholar]
  2. Goldman, J.; Shilton, K.; Burke, J.; Estrin, D.; Hansen, M.; Ramanathan, N.; Reddy, S.; Samanta, V. Participatory Sensing—A Citizen-Powered Approach to Illuminating the Patterns that Shape Our World; Woodrow Wilson International Center for Scholars: Washington, DC, USA, 2009; pp. 1–15. Available online: https://www.wilsoncenter.org/sites/default/files/media/documents/publication/participatory_sensing.pdf (accessed on 20 October 2020).
  3. Zaman, J.; D’Hondt, E.; Boix, E.G.; Philips, E.; Kambona, K.; de Meuter, W. Citizen-friendly participatory campaign support. In Proceedings of the 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS), Budapest, Hungary, 24–28 March 2014; pp. 232–235. [Google Scholar] [CrossRef]
  4. Hou, F.; Sun, J.; Ma, S. Participatory Sensing Network: A paradigm to achieve applications of IoT. In Managing the Internet of Things: Architectures, Theories and Applications; IET: London, UK, 2016; pp. 121–136. [Google Scholar] [CrossRef]
  5. Santos, P.M.; Rodrigues, J.G.P.; Cruz, S.B.; Lourenço, T.; D’Orey, P.M.; Luis, Y.; Rocha, C.; Sousa, S.; Crisóstomo, S.; Queirós, C.; et al. PortoLivingLab: An IoT-Based Sensing Platform for Smart Cities. IEEE Internet Things J. 2018, 5, 523–532. [Google Scholar] [CrossRef]
  6. D’Hondt, E.; Stevens, M.; Jacobs, A. Participatory noise mapping works! An evaluation of participatory sensing as an alternative to standard techniques for environmental monitoring. Pervasive Mob. Comput. 2013, 9, 681–694. [Google Scholar] [CrossRef]
  7. Overeem, A.; Robinson, J.C.R.; Leijnse, H.; Steeneveld, G.J.; Horn, B.K.P.; Uijlenhoet, R. Crowdsourcing urban air temperatures from smartphone battery temperatures. Geophys. Res. Lett. 2013, 40, 4081–4085. [Google Scholar] [CrossRef]
  8. de Araújo, T.C.; Silva, L.T.; Moreira, A.C. Data Quality Issues on Environmental Sensing with Smartphones. In Proceedings of the 6th International Conference on Sensor Networks—SENSORNETS, Porto, Portugal, 19–21 February 2017; pp. 59–68. [Google Scholar] [CrossRef]
  9. Chau, N.H. Estimation of air temperature using smartphones in different contexts. J. Inf. Telecommun. 2019, 3, 494–507. [Google Scholar] [CrossRef] [Green Version]
  10. Ehikhamenle, M.; Omijeh, B.O. Design and Implementation of CanSat (A Pico-Satellite). Int. J. Sci. Eng. Investig. 2017, 6, 106–111. [Google Scholar]
  11. University Space Engineering Consortium—Japan. Can Satellite (CanSat), 1.0; UNISEC: Tokyo, Japan, 2011. [Google Scholar]
  12. Colin, A.; Jimenez-Lizárraga, M. The Cansat Technology for Climate Monitoring in Small Regions at Altitudes Below 1 km. In Proceedings of the IAA Climate Change and Disaster Management Conference, Mexico City, Mexico, 17 September 2015; pp. 1–10. [Google Scholar]
  13. Ostaszewski, M.; Dzierzek, K.; Magnuszewski, Ł. Analysis of data collected while CanSat mission. In Proceedings of the 2018 19th International Carpathian Control Conference, ICCC 2018, Szilvasvarad, Hungary, 28–31 May 2018; pp. 1–4. [Google Scholar] [CrossRef]
  14. Qaid, A.; Lamit, H.B.; Ossen, D.R.; Shahminan, R.N.R. Urban heat island and thermal comfort conditions at micro-climate scale in a tropical planned city. Energy Build. 2016, 133, 577–595. [Google Scholar] [CrossRef]
  15. Magli, S.; Lodi, C.; Contini, F.M.; Muscio, A.; Tartarini, P. Dynamic analysis of the heat released by tertiary buildings and the effects of urban heat island mitigation strategies. Energy Build. 2016, 114, 164–172. [Google Scholar] [CrossRef]
  16. Salata, F.; Golasi, I.; Petitti, D.; Vollaro, E.d.; Coppi, M.; Vollaro, A.d. Relating microclimate, human thermal comfort and health during heat waves: An analysis of heat island mitigation strategies through a case study in an urban outdoor environment. Sustain. Cities Soc. 2017, 30, 79–96. [Google Scholar] [CrossRef]
  17. Sun, C.Y.; Kato, S.; Gou, Z. Application of Low-Cost Sensors for Urban Heat Island Assessment: A Case Study in Taiwan. Sustainability 2019, 11, 2759. [Google Scholar] [CrossRef] [Green Version]
  18. Silva, L.T.; Mendes, J.F.G. City Noise-Air: An environmental quality index for cities. Sustain. Cities Soc. 2012, 4, 1–11. [Google Scholar] [CrossRef]
  19. Hu, K.; Sivaraman, V.; Luxan, B.G.; Rahman, A. Design and Evaluation of a Metropolitan Air Pollution Sensing System. IEEE Sens. J. 2016, 16, 1448–1459. [Google Scholar] [CrossRef]
  20. Trilles, S.; Vicente, A.B.; Juan, P.; Ramos, F.; Meseguer, S.; Serra, L. Reliability Validation of a Low-Cost Particulate Matter IoT Sensor in Indoor and Outdoor Environments Using a Reference Sampler. Sustainability 2019, 11, 7220. [Google Scholar] [CrossRef] [Green Version]
  21. Szczurek, A.; Maciejewska, M.; Pietrucha, T. Occupancy detection using gas sensors. In Proceedings of the 6th International Conference on Sensor Networks—SENSORNETS, Porto, Portugal, 19–21 February 2017; pp. 99–107. [Google Scholar] [CrossRef]
  22. Jiang, C.; Masood, M.K.; Soh, Y.C.; Li, H. Indoor occupancy estimation from carbon dioxide concentration. Energy Build. 2016, 131, 132–141. [Google Scholar] [CrossRef] [Green Version]
  23. Martins, T.A.L.; Adolphe, L.; Bonhomme, M.; Bonneaud, F.; Faraut, S.; Ginestet, S.; Michel, C.; Guyard, W. Impact of Urban Cool Island measures on outdoor climate and pedestrian comfort: Simulations for a new district of Toulouse, France. Sustain. Cities Soc. 2016, 26, 9–26. [Google Scholar] [CrossRef]
  24. Rivas-Sánchez, Y.A.; Moreno-Pérez, M.F.; Roldán-Cañas, J. Environment Control with Low-Cost Microcontrollers and Microprocessors: Application for Green Walls. Sustainability 2019, 11, 782. [Google Scholar] [CrossRef] [Green Version]
  25. Gitzel, R. Data quality in time series data: An experience report. CEUR Workshop Proc. 2016, 1753, 41–49. [Google Scholar]
  26. Terando, A.J.; Youngsteadt, E.; Meineke, E.K.; Prado, S.G. Ad hoc instrumentation methods in ecological studies produce highly biased temperature measurements. Ecol. Evol. 2017, 7, 9890–9904. [Google Scholar] [CrossRef]
  27. Liu, L.; Wei, W.; Zhao, D.; Ma, H. Urban Resolution: New Metric for Measuring the Quality of Urban Sensing. IEEE Trans. Mob. Comput. 2015, 14, 2560–2575. [Google Scholar] [CrossRef]
  28. Jiang, Q.; Kresin, F.; Bregt, A.K.; Kooistra, L.; Pareschi, E.; van Putten, E.; Volten, H.; Wesseling, J. Citizen Sensing for Improved Urban Environmental Monitoring. J. Sens. 2016, 2016, 5656245. [Google Scholar] [CrossRef] [Green Version]
  29. Chang, S.; Zhu, H.; Zhang, W.; Lu, L.; Zhu, Y. PURE: Blind Regression Modeling for Low Quality Data with Participatory Sensing. IEEE Trans. Parallel Distrib. Syst. 2015, 27, 1199–1211. [Google Scholar] [CrossRef]
  30. Castell, N.; Dauge, F.R.; Schneider, P.; Vogt, M.; Lerner, U.; Fishbain, B.; Broday, D.; Bartonova, A. Can commercial low-cost sensor platforms contribute to air quality monitoring and exposure estimates? Environ. Int. 2017, 99, 293–302. [Google Scholar] [CrossRef] [PubMed]
  31. Borrego, C.; Costa, A.M.; Ginja, J.; Amorim, M.; Coutinho, M.; Karatzas, K.; Sioumis, T.; Katsifarakis, N.; Konstantinidis, K.; de Vito, S.; et al. Assessment of air quality microsensors versus reference methods: The EuNetAir joint exercise. Atmos. Environ. 2016, 147, 246–263. [Google Scholar] [CrossRef] [Green Version]
  32. Mijling, B.; Jiang, Q.; de Jonge, D.; Bocconi, S. Field calibration of electrochemical NO2 sensors in a citizen science context. Atmos. Meas. Tech. 2018, 11, 1297–1312. [Google Scholar] [CrossRef] [Green Version]
  33. Zimmerman, N.; Presto, A.A.; Kumar, S.P.N.; Gu, J.; Hauryliuk, A.; Robinson, E.S.; Robinson, A.L.; Subramanian, R. A machine learning calibration model using random forests to improve sensor performance for lower-cost air quality monitoring. Atmos. Meas. Tech. 2018, 11, 291–313. [Google Scholar] [CrossRef] [Green Version]
  34. Yamamoto, K.; Togami, T.; Yamaguchi, N.; Ninomiya, S. Machine learning-based calibration of low-cost air temperature sensors using environmental data. Sensors 2017, 17, 1290. [Google Scholar] [CrossRef] [Green Version]
  35. de Araújo, T.C.; Silva, L.T.; Moreira, A.C. Deviation Prediction and Correction on Low-Cost Atmospheric Pressure Sensors using a Machine-Learning Algorithm. In Proceedings of the 9th International Conference Sensor Networks—Vol. 1 SENSORNETS, Valletta, Malta, 28–29 February 2020; pp. 41–51. [Google Scholar] [CrossRef]
  36. Lascar Electronics, Certificate of Calibration. (n.d.). Available online: http://www.lascarelectronics.com/pdf-usb-datalogging/data-logger0800188001331301358.pdf (accessed on 28 June 2016).
  37. Inmetro. Vocabulário Internacional de Termos de Metrologia Legal—VIM (2012). Available online: http://www.inmetro.gov.br/inovacao/publicacoes/vim_2012.pdf (accessed on 21 October 2020).
  38. Batog, P.; Badura, M. Dynamic of changes in carbon dioxide concentration in bedrooms. Procedia Eng. 2013, 57, 175–182. [Google Scholar] [CrossRef] [Green Version]
  39. Dinh, T.V.; Choi, I.Y.; Son, Y.S.; Kim, J.C. A review on non-dispersive infrared gas sensors: Improvement of sensor detection limit and interference correction. Sens. Actuators B Chem. 2016, 231, 529–538. [Google Scholar] [CrossRef]
  40. Wang, C.; Yin, L.; Zhang, L.; Xiang, D.; Gao, R. Metal oxide gas sensors: Sensitivity and influencing factors. Sensors 2010, 10, 2088–2106. [Google Scholar] [CrossRef] [Green Version]
  41. WMO. Guide to Meteorological Instruments and Methods of Observation, 7th ed.; Chairperson, Publications Board: Geneva, Switzerland, 2008. [Google Scholar]
  42. Ashcroft, M.B. Which is more biased: Standardized weather stations or microclimatic sensors? Ecol. Evol. 2018, 8, 5231–5232. [Google Scholar] [CrossRef]
  43. Lorek, A. Humidity measurement with capacitive humidity sensors between −70 °C and 25 °C in low vacuum. J. Sens. Sens. Syst. 2014, 3, 177–185. [Google Scholar] [CrossRef] [Green Version]
  44. Mwangi, C. Low Cost Weather Stations for Developing Countries (Kenya). In Proceedings of the 7th United Nations International Conference Space-Based Technol. Disaster Risk Reduct, Beijing, China, 23–25 October 2017. [Google Scholar]
  45. Calì, D.; Matthes, P.; Huchtemann, K.; Streblow, R.; Müller, D. CO 2 based occupancy detection algorithm: Experimental analysis and validation for office and residential buildings. Build. Environ. 2015, 86, 39–49. [Google Scholar] [CrossRef]
  46. Karagulian, F.; Barbiere, M.; Kotsev, A.; Spinelle, L.; Gerboles, M.; Lagler, F.; Redon, N.; Crunaire, S.; Borowiak, A. Review of the performance of low-cost sensors for air quality monitoring. Atmosphere 2019, 10, 506. [Google Scholar] [CrossRef] [Green Version]
  47. Williams, D.E. Low Cost Sensor Networks: How Do We Know the Data Are Reliable? ACS Sens. 2019, 4, 2558–2565. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  48. Gryech, I.; Ben-Aboud, Y.; Guermah, B.; Sbihi, N.; Ghogho, M.; Kobbane, A. Moreair: A low-cost urban air pollution monitoring system. Sensors 2020, 20, 998. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  49. Scholz, L.; Perez, A.O.; Bierer, B.; Eaksen, P.; Wöllenstein, J.; Palzer, S. Miniature low-cost carbon dioxide sensor for mobile devices. IEEE Sens. J. 2017, 17, 2889–2895. [Google Scholar] [CrossRef]
  50. Okafor, N.U.; Alghorani, Y.; Delaney, D.T. Improving Data Quality of Low-cost IoT Sensors in Environmental Monitoring Networks Using Data Fusion and Machine Learning Approach. ICT Express 2020, 6, 220–228. [Google Scholar] [CrossRef]
Figure 1. (a) Climatic sensors used in this study. From left to right: DS18B20, AM2302, HTU21D, BMP180, BME280 and MPL3115A2. (b) Carbon dioxide sensors used in this study. From left to right: MG-811 and MH-Z16.
Figure 1. (a) Climatic sensors used in this study. From left to right: DS18B20, AM2302, HTU21D, BMP180, BME280 and MPL3115A2. (b) Carbon dioxide sensors used in this study. From left to right: MG-811 and MH-Z16.
Iot 01 00017 g001
Figure 2. Reference instruments used in this study for (a) temperature and humidity and (b) carbon dioxide.
Figure 2. Reference instruments used in this study for (a) temperature and humidity and (b) carbon dioxide.
Iot 01 00017 g002
Figure 3. (a) One set of climatic sensors wired into the Arduino device. (b) Two groups of climatic sensors, together with a reference sensor, placed inside the controlled chamber.
Figure 3. (a) One set of climatic sensors wired into the Arduino device. (b) Two groups of climatic sensors, together with a reference sensor, placed inside the controlled chamber.
Iot 01 00017 g003
Figure 4. A single unit of each carbon dioxide sensor placed in the office room (MG-811 sensor in the bottom of the image and MH-Z16 sensor in the middle), and the reference sensor (in the top of the image) for calibration purposes.
Figure 4. A single unit of each carbon dioxide sensor placed in the office room (MG-811 sensor in the bottom of the image and MH-Z16 sensor in the middle), and the reference sensor (in the top of the image) for calibration purposes.
Iot 01 00017 g004
Figure 5. Temperature readings obtained from one experimental execution inside the controlled climatic chamber.
Figure 5. Temperature readings obtained from one experimental execution inside the controlled climatic chamber.
Iot 01 00017 g005
Figure 6. Scatter plot for temperature reference sensor (x axis) and evaluated sensor (y axis): (a) DS18B20; (b) AM2302; (c) HTU21D; (d) BMP180; (e) BME280; (f) MPL3115A2. Grey points refer to all individual datasets aggregated in the same plot, and coloured points refer to the corresponding averaged data.
Figure 6. Scatter plot for temperature reference sensor (x axis) and evaluated sensor (y axis): (a) DS18B20; (b) AM2302; (c) HTU21D; (d) BMP180; (e) BME280; (f) MPL3115A2. Grey points refer to all individual datasets aggregated in the same plot, and coloured points refer to the corresponding averaged data.
Iot 01 00017 g006
Figure 7. Relative humidity readings obtained from one experimental execution inside the controlled climatic chamber.
Figure 7. Relative humidity readings obtained from one experimental execution inside the controlled climatic chamber.
Iot 01 00017 g007
Figure 8. Colourmap representation of root mean squared error (RMSE) calculated for humidity sensor models: (a) AM2302; (b) HTU21D; (c) BME280. The individual sensor unit (A, B or C) with its respective experimental execution (1 or 2) is illustrated on the top edge.
Figure 8. Colourmap representation of root mean squared error (RMSE) calculated for humidity sensor models: (a) AM2302; (b) HTU21D; (c) BME280. The individual sensor unit (A, B or C) with its respective experimental execution (1 or 2) is illustrated on the top edge.
Iot 01 00017 g008
Figure 9. Scatter plot for humidity reference sensor (x axis) and the evaluated sensor (y axis): (a) AM2302; (b) HTU21D; (c) BME280.
Figure 9. Scatter plot for humidity reference sensor (x axis) and the evaluated sensor (y axis): (a) AM2302; (b) HTU21D; (c) BME280.
Iot 01 00017 g009
Figure 10. Atmospheric pressure reading sample obtained from one experimental execution inside the controlled climatic chamber.
Figure 10. Atmospheric pressure reading sample obtained from one experimental execution inside the controlled climatic chamber.
Iot 01 00017 g010
Figure 11. Scatter plots for reproducibility verification of pressure sensors from experimental runs. (a) BMP180; (b) BME280; (c) MPL3115A2.
Figure 11. Scatter plots for reproducibility verification of pressure sensors from experimental runs. (a) BMP180; (b) BME280; (c) MPL3115A2.
Iot 01 00017 g011
Figure 12. Calibration curves of MG-811 sensor from first measurement: (a) nominal; (b) Unit A; (c) Unit B.
Figure 12. Calibration curves of MG-811 sensor from first measurement: (a) nominal; (b) Unit A; (c) Unit B.
Iot 01 00017 g012
Figure 13. Carbon dioxide sensor readings from measurements in an occupied office room. (a) First overall execution; (b) third overall execution.
Figure 13. Carbon dioxide sensor readings from measurements in an occupied office room. (a) First overall execution; (b) third overall execution.
Iot 01 00017 g013
Figure 14. Carbon dioxide sensor readings from the measurement in a bedroom with nightly occupancy without ventilation.
Figure 14. Carbon dioxide sensor readings from the measurement in a bedroom with nightly occupancy without ventilation.
Iot 01 00017 g014
Figure 15. Scatter plots for reference and CO2 sensors from measurements in the office room. From left to right: (a) MH-Z16[A]; (b) MH-Z16[B]; (c) MG-811[A]; (d) MG-811[B]. Datapoints in red correspond to the first measurement, and datapoints in blue correspond to the second.
Figure 15. Scatter plots for reference and CO2 sensors from measurements in the office room. From left to right: (a) MH-Z16[A]; (b) MH-Z16[B]; (c) MG-811[A]; (d) MG-811[B]. Datapoints in red correspond to the first measurement, and datapoints in blue correspond to the second.
Iot 01 00017 g015
Figure 16. Scatter plots for reference and CO2 sensors from measurements in the bedroom. From left to right: the NDIR sensors (a) MH-Z16 [A] and (b) MH-Z16 [B], and the MOS sensors (c) MG-811 [A] (this subplot contains only some of the data, that converged after 18 h of the experiment) and (d) MG-811 [B].
Figure 16. Scatter plots for reference and CO2 sensors from measurements in the bedroom. From left to right: the NDIR sensors (a) MH-Z16 [A] and (b) MH-Z16 [B], and the MOS sensors (c) MG-811 [A] (this subplot contains only some of the data, that converged after 18 h of the experiment) and (d) MG-811 [B].
Iot 01 00017 g016
Table 1. Selected low-cost sensors for climatic variables.
Table 1. Selected low-cost sensors for climatic variables.
SensorParameters 1BrandApproximated Price (EUR) 2
DS18B20TMaxim4.00
AM2302T, HAosong8.00
HTU21DT, HMEAS Sensors12.00
BMP180T, PBosch Sensortech10.00
BME280T, H, PBosch Sensortech20.00
MPL3115A2T, PNXP10.00
1 T—Temperature; H—Humidity; P—Pressure. 2 Prices as seen in 2nd half of 2020.
Table 2. Selected low-cost sensors for carbon dioxide.
Table 2. Selected low-cost sensors for carbon dioxide.
SensorParametersBrandApproximated Price (EUR) 3
MH-Z16CO2Winsen56.00
MG-811CO2Henan HanWei47.00
3 Prices as seen in 2nd half of 2020.
Table 3. Summary of averaged accuracy indicators for temperature sensors (all values in °C).
Table 3. Summary of averaged accuracy indicators for temperature sensors (all values in °C).
Sensor ModelMean Error at σ M E Root Mean Squared Error at
−5 °C10 °C25 °C40 °C−5 °C10 °C25 °C40 °C
DS18B200.590.670.430.160.490.610.700.450.46
AM23020.440.26−0.13−0.590.490.450.340.280.61
HTU21D−0.040.290.210.010.250.200.350.250.25
BMP180−1.08−0.40−0.33−0.400.321.080.430.340.42
BME2801.501.891.661.500.361.671.901.661.51
MPL3115A2−1.40−1.03−1.08−1.250.211.391.051.081.26
Table 4. Summary of averaged precision indicators for temperature sensors.
Table 4. Summary of averaged precision indicators for temperature sensors.
Sensor ModelStandard Deviation (°C) atSignal–Noise Ratio at
−5 °C10 °C25 °C40 °C−5 °C10 °C25 °C40 °C
DS18B200.080.070.060.1264.7167.8416.8342.1
AM23020.060.070.080.1377.6160.1317.7367.2
HTU21D0.070.060.060.1482.2185.8424.5309.5
BMP1800.070.070.070.16106.9147.8373.5265.3
BME2800.060.090.070.1664.1165.5401.5266.2
MPL3115A20.060.070.070.15115.0131.1334.8261.1
Table 5. Averaged linear models for each sensor model and its respective dynamic residuals (DR) indicator.
Table 5. Averaged linear models for each sensor model and its respective dynamic residuals (DR) indicator.
SensorAveraged Linear ModelDR (°C)
DS18B20   y ^ ( x ) = 0.980 x + 1.315 1.007
AM2302   y ^ ( x ) = 0.970 x + 0.658 0.414
HTU21D   y ^ ( x ) = 0.987 t + 0.770 0.567
BMP180   y ^ ( x ) = 0.999 t 0.228 0.250
BME280   y ^ ( x ) = 0.978 t + 2.260 1.901
MPL3115A2   y ^ ( x ) = 0.990 t 0.767 0.943
Table 6. Summary of mean errors obtained for humidity sensors in different climatic combinations (all values as percentual relative humidity).
Table 6. Summary of mean errors obtained for humidity sensors in different climatic combinations (all values as percentual relative humidity).
Sensor ModelMean Error (ME) in σ M E
H = 30%H = 50%H = 80%
10 °C25 °C40 °C10 °C25 °C40 °C10 °C25 °C40 °C
AM2302−4.4−4.6−4.2−2.5−4.2−3.31.50.00.72.7
HTU21D−7.5−7.5−8.3−6.0−7.3−7.9−3.5−5.6−6.91.8
BME280−3.9−2.2−1.4−3.0−2.5−1.20.71.42.22.1
Table 7. Summary of precision indicators obtained for humidity sensors (all values in percentual relative humidity).
Table 7. Summary of precision indicators obtained for humidity sensors (all values in percentual relative humidity).
Sensor ModelAverage Standard Deviation in
H = 30%H = 50%H = 80%
10 °C25 °C40 °C10 °C25 °C40 °C10 °C25 °C40 °C
AM23020.630.270.160.330.090.070.150.120.17
HTU21D0.680.270.150.290.080.050.120.100.10
BME2800.570.270.170.320.130.100.210.120.23
Table 8. Averaged linear models obtained for humidity sensors and their respective DR indicators.
Table 8. Averaged linear models obtained for humidity sensors and their respective DR indicators.
Sensor ModelAverage Linear ModelDR
AM2302   y ^ = 1.118 h 9.626 3.5
HTU21D   y ^ = 1.061 h 10.38 7.1
BME280   y ^ = 1.084 h 6.333 2.3
Table 9. RMSE calculated between sensor pairs of the same model in same experimental conditions.
Table 9. RMSE calculated between sensor pairs of the same model in same experimental conditions.
Sensor ModelUnits Evaluated
A × BA × CB × C
BMP1800.820.550.36
BME2801.150.150.59
MPL3115A20.610.360.87
Table 10. Calibration curves for the metal oxide semiconductor (MOS) sensor MG-811 (x: input in volts; Y: output in ppm).
Table 10. Calibration curves for the metal oxide semiconductor (MOS) sensor MG-811 (x: input in volts; Y: output in ppm).
ExperimentUnit AUnit B
Calibration Curver2Calibration Curver2
1st execution Y = 3.63 · 10 8 · x 15.9 0.71 Y = 2.81 · 10 3 · x 12.1 0.90
2nd execution Y = 1.14 · 10 6 · x 13.8 0.77 Y = 11.5 · 10 3 · x 10.8 0.98
3rd execution Y = 6.36 · 10 16 · x 29.3 0.83 Y = 4.55 · 10 3 · x 14.6 0.91
Table 11. Obtained RMSE for carbon dioxide sensors from described experiments.
Table 11. Obtained RMSE for carbon dioxide sensors from described experiments.
Sensor Type and UnitRMSE (in ppm) from
1st exp.2nd exp.3rd exp.
NDIRA5815023
B38557164
MOS 4A10085759
B5420440
4 This sensor model had fewer reading samples due to the larger stabilization time, as previously suggested by the presented traces in Figure 15 and Figure 16.
Table 12. Correlation and linearity verification of carbon dioxide sensor readings.
Table 12. Correlation and linearity verification of carbon dioxide sensor readings.
Sensor Type and UnitOffice Room (#1)BedroomOffice Room (#2)
Linear Modelr2Linear Modelr2Linear Modelr2
NDIRA y = 1.05 x 83.3 0.99 y = 1.03 x + 55.2 0.99+ y = 1.12 x 75.7 0.99
B y = 1.14 x 91.1 0.99 y = 1.13 x + 238.7 0.99+ y = 1.17 x + 57.6 0.97
MOSA y = 0.81 x + 112.7 0.80 y = 1.80 x 595 , 3 0.90 y = 0.76 x + 121.5 0.96
B y = 0.91 x + 40.6 0.94 y = 1.05 x 100.5 0.99 y = 0.98 x + 80.0 0.90
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Araújo, T.; Silva, L.; Moreira, A. Evaluation of Low-Cost Sensors for Weather and Carbon Dioxide Monitoring in Internet of Things Context. IoT 2020, 1, 286-308. https://0-doi-org.brum.beds.ac.uk/10.3390/iot1020017

AMA Style

Araújo T, Silva L, Moreira A. Evaluation of Low-Cost Sensors for Weather and Carbon Dioxide Monitoring in Internet of Things Context. IoT. 2020; 1(2):286-308. https://0-doi-org.brum.beds.ac.uk/10.3390/iot1020017

Chicago/Turabian Style

Araújo, Tiago, Lígia Silva, and Adriano Moreira. 2020. "Evaluation of Low-Cost Sensors for Weather and Carbon Dioxide Monitoring in Internet of Things Context" IoT 1, no. 2: 286-308. https://0-doi-org.brum.beds.ac.uk/10.3390/iot1020017

Article Metrics

Back to TopTop