Rasterized Data Image Processing (RDIP) Techniques for Photovoltaic (PV) Data Cleaning and Application in Power Prediction

Abstract

Photovoltaic (PV) power generation has attracted widespread interest as a clean and sustainable energy source, with increasing global attention given to renewable energy. However, the operation and monitoring of PV power generation systems often result in large amounts of data containing missing values, outliers, and noise, posing challenges for data analysis and application. Therefore, PV data cleaning plays a crucial role in ensuring data quality, enhancing data availability and reliability. This study proposes a PV data cleaning method based on Rasterized Data Image Processing (RDIP) technology, which integrates rasterization and image processing techniques to select optimal contours and extract essential data. To validate the effectiveness of our method, we conducted comparative experiments using three data cleaning methods, including our RDIP algorithm, the Pearson correlation coefficient interpolation method, and cubic spline interpolation method. Subsequently, the cleaned datasets from these methods were utilized for power prediction using two linear regression models and two neural network models. The experimental results demonstrated that data cleaned using the RDIP algorithm improved the short-term forecast accuracy by approximately 1.0% and 3.7%, respectively, compared to the other two methods, indicating the feasibility and effectiveness of the RDIP approach. However, it is worth noting that the RDIP technique has limitations due to its reliance on integer parameters for grid division, potentially leading to coarse grid divisions. Future research efforts could focus on optimizing the selection of binarization thresholds to achieve better cleaning results and exploring other potential applications of RDIP in PV data analysis.

1. Introduction

With increasing global attention on renewable energy, photovoltaic (PV) power generation has garnered widespread attention and application as a pivotal solution to address the global energy demand, owing to its clean and sustainable nature. However, the operation and monitoring of PV power generation systems often generate a large amount of data, which may contain missing values, outliers, and noise, posing significant challenges for data analysis and application [1,2,3,4,5]. Therefore, PV data cleaning plays a crucial role in ensuring data quality, improving data availability and reliability [6,7].

In recent years, there have been significant advancements in the field of data cleaning, encompassing aspects, such as missing value handling, outlier detection and treatment, and noise reduction. Several studies have proposed targeted methods and techniques for cleaning PV data. For instance, Wang and Deng (2023) introduced a PV data cleaning method based on interpolation and outlier detection [8], effectively handling missing values and outliers. Additionally, Ilyas and Rekatsinas (2022) explored machine learning algorithms for time-series data cleaning [9], which are of significant importance for preprocessing PV generation data. Furthermore, Ray P K et al., (2019) proposed an anomaly detection method for PV power generation data using principal component analysis (PCA) and a support vector machine [10]. Ibrahim M et al., (2022) proposed an anomaly detection method for PV power generation data based on learning techniques [11]. Zhang et al., (2019) developed a hybrid forecasting model for PV power generation data using PCA and an Elman neural network [12].

By comprehensively applying these data cleaning techniques, the quality and reliability of PV power generation data can be effectively enhanced, providing a reliable foundation for subsequent data analysis and applications. For new energy plants, power prediction is a critical task, where the actual total radiation and power serve as key data for model training. To ensure the accuracy of predictions, data cleaning is essential. The aforementioned methods mostly employ statistical and regression-based approaches for data processing.

This study attempts to clean data from the perspective of image processing. After simple cleaning, irradiance power scatter data are rasterized, generating rasterized images with different resolutions based on grid step size. The rasterized images undergo image processing techniques, such as morphological operations, contour extraction, and filtering. Contours satisfying criterion $J_w$ (minimizing dispersion $S_k$ and maximizing area S) are selected as our optimal contours. Scatter data within the contours are retained as the required data. Finally, the effectiveness of data cleaning is validated by comparing the accuracy of different power prediction algorithms.

2. Related Work

Various methods and techniques have been proposed for photovoltaic (PV) data cleaning [13,14]. Sun and Zhang (2018), Wang et al., (2023), and Li et al., (2021) introduced a PV data cleaning method based on interpolation and outlier detection algorithms [1,4,6], which effectively handles missing values and outliers. Despite their advantages, these methods have lower efficiency in handling high-dimensional data and large data volumes, and they may perform poorly on nonlinear and non-stationary time-series data. Additionally, Micheli et al., (2021), as well as Ranjan and Prusty (2019), discussed techniques for time-series data cleaning [15,16], including sliding window smoothing and exponential weighted moving average methods, aimed at reducing noise and fluctuations in data to enhance stability and reliability. However, these methods may lead to data lagging and blurring, thereby affecting the accuracy of subsequent data analysis. Furthermore, Hopwood et al., (2020) employed principal component analysis (PCA) to improve data quality and interpretability by dimensionality reduction and redundant information removal [17]. While these methods have achieved certain effectiveness in data cleaning, they may lose important information, especially when the main variation patterns of data do not conform to linear assumptions. Moreover, Balachandran and Devisridhivyadharshini et al., (2024) combined multiple data cleaning techniques into a comprehensive data cleaning process to achieve thorough cleaning and preprocessing of PV data [18]. Although this method has certain advantages, it requires a reasonable design of the cleaning process to avoid information loss or excessive processing time.

In addition to traditional data cleaning methods, in recent years, image processing techniques have also attracted extensive attention in the field of data cleaning [19,20,21,22,23,24]. Contour extraction in image processing and boundary extraction in data extraction share similarities, involving target detection and extraction, shape and structural information, processing steps, and application domains. The feasibility of using image processing methods [25] to achieve data cleaning is demonstrated by the flexibility, generality, and efficiency of image processing methods, which can be applied to various types of data processing tasks. The application of image processing methods in data cleaning mainly lies in the similarity of data representation, the flexibility of image processing methods, and the efficiency of image processing algorithms. In summary, applying image processing methods to achieve data cleaning will provide new insights and methods for data cleaning.

3. Materials and Methods

3.1. Photovoltaic Anomalous Data Analysis

There exists a close relationship between photovoltaic (PV) power and irradiance data (including global radiation, direct radiation, diffuse radiation, etc.) [26], as the working principle of PV systems involves converting solar energy into electrical energy. Direct radiation refers to solar radiation directly from the sun, whose intensity is influenced by factors such as solar angle and atmospheric transparency. Under ideal conditions, higher direct radiation results in more light energy absorbed by PV cell components, thereby increasing PV power generation. Diffuse radiation refers to solar radiation scattered in the atmosphere before reaching the ground, and its intensity is affected by atmospheric particulates and cloud cover, among other factors. Although diffuse radiation is not as intense as direct radiation, it still provides some light energy to PV cell components, thereby affecting PV power generation.

$$G=D+D_{iff}$$

(1)

where G is global radiation; D is direct radiation; $D_{iff}$ is diffuse radiation.

Equation (1) illustrates that global radiation (G) is the sum of direct radiation (D) and diffuse radiation ($D_{iff}$). Global radiation serves as a crucial indicator for measuring total solar radiation, directly impacting the efficiency of PV cell components in generating electricity [27]. Therefore, there exists a close relationship between global radiation and power output. Higher global radiation results in increased PV power generation.

Figure 1 depicts the relationship between global radiation and power generation at a specific PV station located in Guizhou Province, China. It can be observed that the relationship between PV power generation and total radiation is not simply linear but exhibits certain fluctuating characteristics. Additionally, the data collection process often contains numerous anomalous data points, whose direct utilization in training power prediction models may lead to significant prediction errors.

It is evident from the figure that the relationship between PV power generation and total radiation is not purely linear but demonstrates certain fluctuating characteristics. Furthermore, it is noted that during the data collection process, there are numerous anomalous data points. Utilizing these data directly for training power prediction models undoubtedly leads to considerable prediction errors. Various factors contribute to the generation of anomalous data [28,29,30], including equipment malfunctions, issues with data collection devices, and data quality concerns. Equipment malfunctions represent a significant source of anomalous data generation. Equipment within PV stations, including PV cell components and inverters, may experience performance degradation or failure due to factors such as natural environmental conditions and operational loads, thereby resulting in anomalous data (see red circle B). Additionally, issues with data collection devices commonly contribute to anomalous data. Problems, such as sensor malfunctions, data transmission interruptions, and data recording errors, may lead to inaccurate or anomalous data collection (see red circle A). Moreover, deliberate power limitations or environmental factors often contribute to a significant portion of anomalous data (see red circle C).

3.2. Design and Implementation of RDIP Technology

In order to identify and process anomalous data, we employed the RDIP (Rasterized Data Image Processing) technology method. The specific process for anomalous data analysis is illustrated in Figure 2. To retain the most reasonable data, we devised five sets of cleansing schemes with different resolutions, selecting the most appropriate cleansing scheme based on the maximum contour principle. Initially, the data underwent a simple cleansing process, where records with either total radiation or power generation data being zero were deleted, as these are obvious outlier values.

a.

Data Rasterization and Visualization

The data cleansing process essentially involves retaining frequently occurring data points while eliminating those that occur infrequently. In the Central China region, total radiation values range from 0 to 1500 W/m². For a PV station with an installed capacity of 150 MW, if the data are rasterized using a ratio of 1 W/m² to 1 kW, a maximum of 1500 × 1500 grid blocks will be generated. The PV data mapped to grid blocks were then tabulated to generate a rasterized data table containing data quantities, as depicted in Figure 3.

However, the rasterized data cannot be directly converted into image data as the data quantity for grid block (0, 0) needs correction. Since nighttime data predominantly fall within this grid block, accounting for half of the total data quantity, directly converting the raster table into images would result in distortion. The correction involves substituting the maximum value from other grid blocks. The visualization process essentially involves mapping the raster data to different colors through numerical mapping, akin to converting matrix data into a heatmap. Figure 4 shows the rasterized data transformed into images of different resolutions.

b.

Image Morphological Operations and Contour Extraction

The objective of cleansing is to extract the maximum contours to encompass as much valid data as possible. An essential step in contour extraction is binarizing the grayscale image, with the threshold value being critical. Setting the threshold value too low would include all data points, defeating the purpose of cleansing, while setting it too high would cause excessive pixel separation, hindering contour extraction. Figure 5 illustrates the images decomposed from a 50 × 50 resolution with different heat values.

The red areas represent the contours to be extracted. To ensure the extraction of maximum contours, a common practice in image processing is to perform morphological opening operations on the image before binarization. This involves erosion (Formula (1)) followed by dilation (Formula (2)), aimed at eliminating small pixels, separating contours in narrow areas, and smoothing large contour boundaries.

$$A\otimes B=\left\{x,y|{(B)}_{xy}\subseteq A\right\}$$

(2)

$$A\oplus B=\left\{x,y|{(B)}_{xy}\cap A\ne \varphi \right\}$$

(3)

Furthermore, downsampling the image is another method for contour extraction (refer to Figure 6). It is important to note that the extracted image after downsampling needs to be restored to the original image resolution. Hence, we designed six identical processing workflows, differing only in image resolution. Since the resolution of adjacent images differs by a factor of 4, the threshold selection for morphological opening and binarization should also differ by a factor of 4 for adjacent images.

c.

Maximum Contour Extraction and Resolution Selection

After morphological opening, contour extraction, and contour smoothing, the maximum contour area can be computed. Since the threshold selection between adjacent images is aligned by a factor of 4, the area statistics should be aligned accordingly, meaning that the contour area of lower-resolution images should be expanded by a factor of 4. It can be observed that the selection of threshold H is inversely proportional to the contour area S, and under the same threshold conditions, the image resolution is directly proportional to the contour area. We aim to select the smallest possible threshold to obtain the largest possible contour area; thus, a balance needs to be found between the threshold and the image resolution.

$$S_w=\frac{1}{N}{\displaystyle \sum _{i=1}^k{n}_i{S}_i}$$

(4)

where N is the total number of data points, k is the number of samples, n_i is the number of data points in the i-th sample, and S_i is the dispersion matrix of the i-th sample.

The data cleansing process is essentially a clustering process, and the intra-class dispersion S_w (Equation (4)) can also be used to assess the effectiveness of data cleansing. Minimizing intra-class dispersion is the goal of cleansing, while simultaneously ensuring data quantity as much as possible. Therefore, we use the weighted dispersion matrix S_k (Equation (5)) instead.

$$S_k=\frac{{\displaystyle {\sum }_{i=1}^n{\omega }_i{‖x_i-\overline{x}‖}^2}}{{\displaystyle {\sum }_{i=1}^n{\omega }_i}}$$

(5)

where $S_k$ is the weighted dispersion, n is the number of samples, $x_i$ represents the coordinates of the i-th sample, $\overline{x}$ represents the coordinates of the sample mean, and ${\omega }_i$ represents the weight of the i-th sample. $‖x_i-\overline{x}‖$ denotes the Euclidean distance between the i-th sample and the sample mean.

The weights are determined through the following steps:

• For each numerical value, extract all its coordinates in the matrix to form a coordinate set.
• Then, for all coordinates of that numerical value, consider them as a sample set. When computing the weighted dispersion, use the number of samples as weights. In other words, the weight of each sample equals the number of times that numerical value appears in the matrix.
• Finally, according to the formula for calculating weighted dispersion, use these weights to compute the dispersion, ensuring that the number of samples is correctly applied as weights.

The purpose of this approach is to ensure that different occurrences of each numerical value have corresponding effects when computing the weighted dispersion. Finally, a cleansing criterion function ($J_w$) is designed:

$$J_w=\frac{S}{S_k}$$

(6)

Since S and $S_k$ are both integers, when S takes the maximum value and $S_k$ takes the minimum value, $J_w$ can achieve the maximum value.

3.3. Experimental Design and Evaluation Metrics

An essential aspect of photovoltaic (PV) data cleansing is its impact on power prediction for PV stations, where prediction accuracy serves as the primary metric for evaluating prediction performance. The formula for short-term prediction accuracy (PA) is provided below (Equation (8)). This calculation formula is derived from the “Implementation Rules for Auxiliary Services Management of Grid-connected Power Plants in the Southern Region” and the “Implementation Rules for Grid-connected Operation Management of Power Plants in the Southern Region” (collectively referred to as the “Two Regulations”).

We will employ representative statistical regression prediction models, namely Random Forest [31] and Lasso [32], as well as deep learning prediction models, specifically CNN + LSTM [33] and DWT + EN + GRU [34], to conduct a comparative analysis in the domain of power prediction.

$$\mathrm{NRMSE}=\frac{1}{Cap}\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{\left(P_{fi}-P_{oi}\right)}^2}}$$

(7)

$$\mathrm{P}\mathrm{A}=(1-\mathrm{N}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E})\times 100\mathrm{\%}$$

(8)

where $P_f$ represents the predicted PV power, $P_o$ represents the actual PV power, $Cap$ represents the installed capacity, and PA represents the prediction accuracy.

4. Experimental Results and Analysis

4.1. Data Acquisition and Processing

This study utilized data from a photovoltaic (PV) station with an installed capacity of 150MW located in Guizhou Province, China (Shown in Figure 7). Solar power prediction in Guizhou Province encounters unique challenges distinct from other regions, primarily due to two factors:

• The region’s intricate and varied terrain, encompassing mountainous, hilly, and plain areas, which heightens uncertainty in solar radiation owing to terrain-induced effects;
• The humid climate, susceptible to weather elements like clouds and rainfall, necessitating prediction models to incorporate additional meteorological variables such as precipitation.

A total of 73,824 data points were extracted from this station between 1 February 2021 and 1 October 2023, including time, total radiation, direct radiation, diffuse radiation, and actual power, spanning five dimensions. As total radiation is a key factor influencing actual power, these two types of data must be processed first, as shown in Figure 1.

For the dataset, A and B class abnormal data points are obvious, thus requiring initial cleaning. The cleaned data are illustrated in Figure 8.

To reduce errors during image processing caused by rasterizing the data, it is common practice to maintain the image size as a power of 2. Therefore, this study sets the maximum resolution of the images to 1600 × 1600. Additionally, resolutions of 800 × 800, 400 × 400, 200 × 200, 100 × 100, and 50 × 50 are designed. According to the processing method described, it is known that smaller thresholds in contour extraction can yield larger contour areas. Hence, in this study, only the values 1–6 are analyzed for the threshold. The most reasonable parameters satisfying cleaning criterion Jw are selected. The six different-resolution images corresponding to thresholds 1–6 are shown in Figure 9.

4.2. Experimental Results Analysis

By selecting different thresholds and resolutions, maximum contour extraction is performed after binarizing the images. Calculating the area of the largest contour allows us to obtain parameters, such as area S (scaled with respect to the resolution of 1600 × 1600), sample size N, intra-sample dispersion S_k, and cleaning criterion $J_w$. Refer to

Table 1. Comparison of accuracy with different cleaning methods and different forecast models.

Data Cleaning Method	Forecast Model	MSE	Accuracy
RDIP(Ours)	Random Forest	14.24	90.59%
RDIP(Ours)	Lasso	14.13	90.58%
RDIP(Ours)	CNN-LSTM	13.77	90.82%
RDIP(Ours)	DWT-EN-GRU	13.71	90.86%
Fusion of Pearson Correlation Interpolation	Random Forest	15.30	89.85%
Fusion of Pearson Correlation Interpolation	Lasso	15.57	89.62%
Fusion of Pearson Correlation Interpolation	CNN-LSTM	14.99	90.01%
Fusion of Pearson Correlation Interpolation	DWT-EN-GRU	15.29	89.80%
Cubic Spline Interpolation	Random Forest	18.66	87.56%
Cubic Spline Interpolation	Lasso	21.19	85.87%
Cubic Spline Interpolation	CNN-LSTM	17.83	88.11%
Cubic Spline Interpolation	DWT-EN-GRU	17.55	88.30%

for details.

By applying threshold selection, the binarization of images with different resolutions can be performed, and the corresponding largest contours can be extracted. We consider the power irradiance scatter data contained within the largest contour as valid data and calculate the weighted dispersion S_k of these data. As shown in Figure 10a, the higher the resolution of the image, the greater the weighted dispersion. Except for the rasterized image with a resolution of 1600 × 1600, the weighted dispersion of valid data contained within the largest contours at other resolutions achieves its minimum value when the threshold is 4 or 5. Subsequently, as the threshold increases, the weighted dispersion also increases. This is understandable, since higher thresholds result in fewer valid data points being included (see Figure 10b). Since we aim for the smallest possible weighted dispersion, the threshold selection only needs to consider values less than 6. Additionally, we observe that as the resolution decreases, the number of valid data points increases. When the resolution is reduced to 100 × 100, the number of valid data points tends to stabilize. Our goal is to include as many valid data points as possible; thus, analyzing resolutions of 50 × 50 and above is sufficient.

It can be observed that there is a monotonically decreasing relationship between the threshold and the largest extracted contour (see Figure 10c). By calculating the maximum cleaning criterion coefficient for rasterized images of different resolutions according to Equation (6), it can be found that when the threshold is 1 and the resolution is 200 × 200, the maximum cleaning criterion coefficient is obtained in the rasterized image (see Figure 10d).

The largest contour extracted after binarization of the image with a resolution of 200 × 200 is illustrated in Figure 11.

The opening operation is performed on the maximum contour using a 5 × 5 template to eliminate small objects, separate objects in narrow areas, and smooth large object boundaries. The results are shown in Figure 12.

4.3. Discussion of Results

To validate the effectiveness of our algorithm, we selected a fusion of the Pearson correlation coefficient interpolation method [8] and cubic spline interpolation method [35] for comparison. Additionally, we chose two linear regression models and two neural network prediction models to compare accuracy and mean squared error. As shown in

Table 2. Comparison of prediction accuracy: cleaned vs. uncleaned data.

Month	Forecast Model	Accuracy (Uncleaned)	Accuracy (Cleaned)	Efficiency
April	Random Forest	87.11%	89.13%	2.02%
April	Lasso	88.35%	90.34%	1.99%
April	CNN-LSTM	86.58%	88.68%	2.10%
April	DWT-EN-GRU	87.19%	89.47%	2.28%
June	Random Forest	81.53%	84.33%	2.80%
June	Lasso	83.09%	85.95%	2.86%
June	CNN-LSTM	82.21%	85.11%	2.90%
June	DWT-EN-GRU	82.99%	84.89%	1.90%
September	Random Forest	89.94%	91.13%	1.19%
September	Lasso	88.22%	90.03%	1.81%
September	CNN-LSTM	87.52%	89.42%	1.90%
September	DWT-EN-GRU	89.53%	91.05%	1.52%
December	Random Forest	82.86%	85.63%	2.77%
December	Lasso	84.76%	87.17%	2.41%
December	CNN-LSTM	83.87%	86.86%	2.99%
December	DWT-EN-GRU	82.88%	85.95%	3.07%

, applying the data cleaned by our RDIP algorithm to model training resulted in approximate 1.0% and 3.7% improvements in short-term forecast accuracy, indicating the feasibility of this approach.

To further verify whether our proposed cleaning algorithm improves the subsequent performance prediction, we conducted short-term forecast validation for the months of March, June, September, and December 2023 for the station. These four months were selected to represent the four seasons: spring, summer, autumn, and winter. It is known that the accuracy of photovoltaic power prediction follows a pattern of “high in spring, low in summer, high in autumn, and low in winter”. This is due to stable weather conditions in spring and autumn, which result in higher prediction accuracy, while extreme weather and unstable climate factors in summer and winter lead to lower accuracy.

As shown in Table 2, after data cleaning, the accuracy of all four prediction methods improved by an average of 2.28%. Additionally, we observed that the forecast accuracy in June and December narrowed by nearly 1% compared to April and September. This indicates that the cleaned data better reflect the intrinsic patterns of the photovoltaic power station.

To test the applicability of our cleaning algorithm, we additionally selected two other photovoltaic power stations in Guizhou Province (Yongxin Station and Gangpingzhan) for validation. From the historical data (see Figure 13), we can see that the data quality of these two stations is significantly inferior to that of Zhenliang Station. In fact, these two stations often undergo assessments due to their forecast accuracy not meeting the standards. According to feedback from meteorological departments, these two stations are located in valleys, and the influence of local microclimates causes the coarse-gridded numerical forecast data used to fail to reflect the intrinsic local conditions.

Using our proposed RDIP algorithm for data cleaning, Yongxin Station achieved the best cleaning coefficient, with a threshold of 2 and a resolution of 100 × 160, while Gangping Station achieved the maximum cleaning coefficient, with a threshold of 3 and a resolution of 100 × 160. The cleaning results are shown in Figure 14. Applying the aforementioned four models to calculate the accuracy for these two stations, we found that the accuracy after cleaning increased by nearly 3.7% compared to the accuracy before cleaning. This demonstrates a significant improvement.

5. Conclusions and Future Work

The PV data cleaning method based on RDIP technology proposed in this study approaches data cleaning from a perspective considering both the density and frequency of data distribution. The cleaned data after processing can more effectively ensure that training data reflect the intrinsic patterns of PV data. The accuracy results also validate the effectiveness of data cleaning, laying a foundation for the further development of PV power prediction models and contributing to the safe and stable operation of the power grid. However, the current data gridization process only considers integer parameters, resulting in somewhat coarse grid division. Future work could focus on refining the selection of binarization thresholds to achieve better cleaning results and exploring other potential applications of RDIP in PV data analysis.