Deep Learning-Based Method for Accurate Real-Time Seed Detection in Glass Bottle Manufacturing

Abstract

Glass bottle-manufacturing companies produce bottles of different colors, shapes and sizes. One identified problem is that seeds appear in the bottle mainly due to the temperature and parameters of the oven. This paper presents a new system capable of detecting seeds of 0.1 mm² in size in glass bottles as they are being manufactured, 24 h per day and 7 days per week. The bottles move along the conveyor belt at 50 m/min, at a production rate of 250 bottles/min. This new proposed method includes deep learning-based artificial intelligence techniques and classical image processing on images acquired with a high-speed line camera. The algorithm comprises three stages. First, the bottle is identified in the input image. Next, an algorithm based in thresholding and morphological operations is applied on this bottle region to locate potential candidates for seeds. Finally, a deep learning-based model can classify whether the proposed candidates are real seeds or not. This method manages to filter out most of false positives due to stains in the glass surface, while no real seeds are lost. The F1 achieved is 0.97. This method reveals the advantages of deep learning techniques for problems where classical image processing algorithms are not sufficient.

1. Introduction

The introduction of digital tools that allow the automation of tasks in manufacturing processes is a big challenge and non-trivial issue. The benefits are clear and diverse. Digitalization is therefore ongoing in many industrial manufacturing processes, but there is still a long way to go. Some of the processes that are often automatized are those ones that try to reproduce visual assessments in an objective and repetitive way at high speed. This allows for the assessment and inspection of certain objects with a high degree of precision and accuracy over 100% of production, a process that would require a lot of time for a human being to execute. At the same time, image processing and deep learning techniques have experienced a good level of acceptance in recent years since they have enabled us to tackle some of these challenges. Many computer vision-based applications can be found in the literature for solving different problems in different application fields [1,2,3,4,5,6].

Manufacturing of glass bottles is already a fairly automatized process that still presents some challenges ahead. Glass bottle manufacturers often produce bottles of different colors, shapes and diameters. One of the identified problems is that seeds can appear in the bottle during the manufacturing process, mainly due to the temperature and parameters of the oven. The control of the presence of seeds is important for two reasons. The first reason is the quality of the manufactured element. Drinks manufacturers do not want to sell their products in packages with defects. The second reason is a matter of energy efficiency. If some seeds appear in the glass and their presence increases over time, it is necessary to modify the parameters of the oven to give the bottles the desired quality. Generally speaking, the temperature parameter of the oven should be adjusted to the optimum value so as to manufacture bottles of good quality, on the one hand; and on the other hand, it is advisable not to exceed in energy consumption since it has a high impact on the energy bill. If the temperature of the oven decreases too much, the consumption will be lower, but a higher number of seeds may appear. If the temperature of the oven is very high, the quality of the bottles will be excellent, but there may not have been an efficient use of energy. A compromise must be achieved between these two situations.

The seeds can be categorized into two different types according to their size: seeds mainly refer to small-sized elements, and big-sized seeds are named blisters. Therefore, the availability of tools that may detect the presence of seeds and blisters in the glass are more than welcome. If these tools can differentiate among seeds and blisters and count their number in terms of process indicators, it can contribute to a great improvement in control over the process.

In this work, a new system is proposed whose purpose is to automatically detect the presence of seeds and blisters during the manufacturing process of the bottles. The system can categorize the defects in seeds and blisters according to their size. The system is integrated into the manufacturing line and automatically inspects 100% of the production to locate seeds of up to 0.1 × 0.1 mm² that appear during the manufacture of glass bottles. The system has been validated in three production lines of different sites of Vidrala group, one of the biggest glass bottle manufacturers in the world, and over bottles with a wide variety of size, color and diameter. This production line has a maximum cadence of 250 bottles/min, with a line speed of around 50 m/min.

A first version of the system was deployed in 2017. After three years, in 2020, although the system was performing reasonably well in two production sites (Aiala and Castellar manufacturing sites), Vidrala asked to improve software robustness, mainly in distinguishing seeds from grease stains, which sometimes causes false positives. At that time, deep learning techniques had already demonstrated the capability to correctly execute advanced analysis where traditional image processing techniques would fail. Therefore, a new version of the system was developed with an additional deep learning-based module. This model would finally evaluate whether a detected blob was a seed, a blister or nothing at all. The rejected blobs could be greasy regions, joints or any other object that might appear in the bottle. This deep learning-based model has managed to drastically reduce up to 95% of the false positives detected as seeds by the initial version of the system.

In this work, we propose a new system that can automatically detect and count seeds and blisters in real time in a glass bottle manufacturing line by means of image processing and deep learning-based techniques.

The organization of the paper is as follows: Section 1 provides an introduction to the problem; Section 2 presents a review of the related work; Section 3 details the materials and methods used in this work and describes the proposed solution; results are shown in Section 4; discussion can be found in Section 5; finally, conclusions and future research lines are presented in Section 6.

2. Related Work

In the sector of glass containers, the physical appearance is the main indicator of the product quality, for both aesthetic and physical reasons. Specifically, in the case of glass bottles, there is plenty of literature, as well as multi-inspection machines, available in the market that aim to detect imperfections in several regions of the geometry to meet the quality standards.

The work proposed in this paper is focused on detecting seeds on the walls of the bottles. However, typical defects also include cracks, seams, inclusions or tears in areas like the bottom, mouth and shoulder.

Nowadays, machine manufacturers offer different solutions for the inspection of the glass bottles, both in the hot and cold end areas, replacing manual processes which are very time consuming and dependent on subjective perception among the quality experts. Some of these inspection machines are already a standard in the market, as in the case of MCAL4 [7], OMNIVISION 3 [8], FlexInspect [9], EVOLUTION 12 NEO [10] and others such as Imago Omnia [11], FT system [12] and Linatronics AI [13], allowing multiple configurations and high complexity. Even though they are also capable of detecting seeds in the sidewalls, manufacturers don’t provide detailed information about either the size of the detected seeds or the accuracy to compare metrics and performance with the method presented in this paper. For instance, some claims regarding the number of bottles processed per hour [8,12] report that over 60,000 bottles/h can be managed by a certain system, without specifying the constrains of that specific scenario (number and size of the defects, number of zones inspected, etc.). In general, these systems can be configurated for a broad range of measurements, including not only defects, but bottle dimensions, deformations, cap presence, liquid fill and others.

Meanwhile, the works described in publications reinforces the idea that machine vision is a key technology to tackle the problem of automatizing the inspection task. Ref. [14] lists several use cases where machine vision algorithms are applied to quality control in the glass industry, as well as different techniques widely used to process the images in search of suspicious blobs. In computer vision applications, the first suggested step is related with the definition of the region of interest (ROI) in the image, in order to optimize subsequent analysis. In [15] the authors compare the time reduction in the inspection of glass tubes when the defect detection algorithm is applied to both the original image and the extracted ROI, achieving a performance gain of up to 66%. In terms of processing algorithms, the most common techniques consist in detection of blobs and contours, making it possible to differentiate those pixel clusters that present some quantifiable contrast with the background. Some approaches describe Canny [15,16,17,18], black top-hat [19,20], Watershed [21] and Otsu [22] methods, which successfully extract most of the particles present in the sidewalls. Characterization of the resulting blobs can be achieved analyzing the features [17,20,23] of the connected pixels in those regions. Recently, some works applied artificial intelligence [22,24] for the classification of the blobs, as they provide versatility in the identification of multiple defect classes. To this end, some extracted features from the image were used, which constitute the input of the classifiers.

A critical parameter in the quality control of glass bottles is the range of defect sizes that the system can detect. For instance, ref. [25] sets a lower limit of 36 pixels for the blobs in the sidewalls and bottom, although seeds are not included in the study, whereas [16] managed to classify defects as small as 25 pixels, being able to differentiate seeds from other types of inclusions. A better approach is considered to be the specification of the minimum defect size in physical magnitude, as it is closer to the needs of glass manufacturers. In that regard, the method tested by [21] set the minimum detectable size to 1.5 mm² for defects in the sidewalls of a bottle. A few years later, ref. [17] claimed that knots with a size of 1 mm² can be recognized in syringes and vials, considering that smaller defects do not have a big impact in that market. Finally, ref. [18] presented an algorithm capable of detecting defects with an area as small as 0.15 mm² in the production of glass tubes.

Most of the defects detected in the related works are quite big. Moreover, those systems do not indicate performance in terms of precision or accuracy. They do not indicate how they act when there are other defects that look alike, such as in the case of small seeds and grass stains or the joints of the bottle. This may cause false positives in the detection. It is very difficult to distinguish among these defects using only classical image processing techniques, since the features that describe both the defects and the false positives are similar.

3. Materials and Methods

This section gathers the design of the system, the description of the dataset used for the development of the SW and the proposed SW solution.

The bottles that the system has to inspect have the following characteristics:

• Revolution bottles whose diameters range from 52 to 120 mm.
• Glass thickness between 0.8 and 2.5 mm, approximately, depending on the color.
• Color range: Flint, Green, Dark Green, and Oak. Other hues can be produced.

Different types of bottles can be seen in Figure 1.

Bottles often present grease stains (objects in the bottle that are greasy points but are considered as seeds), which can lead to false positives. The union joint of the bottle can also produce the same effect due to its visual similarity.

The dataset of the system must contain examples with all this variability in shapes, dimensions, hues and greasy vs. clean surfaces.

The design of the system has to guarantee the proper working of this wide variety of bottles that move at a speed of up to 50 m/min in a production line. The number of bottles to be inspected can be up to 250 bottles/min. The minimum size of the detected seed is 0.1 × 0.1 mm². Real-time inspection is necessary.

3.1. System Design

This system has to analyze 100% of the bottles automatically and in real time. According to quality requirements and expert knowledge, it is not necessary to inspect the whole bottle in order to have the total number of seeds. It is enough to have the seed density information in a specified region and analyze its trend over time. An increase in the number of seeds per kilogram may indicate a problem in the furnace (oven) that has to be solved. A low or decreasing trend in the number of seeds per kilogram indicates a good, optimized manufacturing process. To this end, only a specified and fixed region of interest is inspected in every bottle, and the number of seeds per kilogram (or blisters per ton) are the indicators calculated. Bubbles are named in two different ways depending on the size. Seeds refer to bubbles with bounding box with both sides smaller than 1 mm. Blister refers to a big bubble having at least one dimension of the circumscribing bounding box higher than 1 mm.

To this end, a region for inspection approximately 60 mm in height by the diameter of the bottle is established. The seeds/blisters that are displayed in said region will be detected, their shape and size will be evaluated and the results of said analysis will be reported, so that the operator can take the appropriate actions based on the information received. The number of seeds detected in said inspection region will be related to a value of the number of seeds per kilo and per ton, which is the variable usually used in the plant to control the quality of the bottle manufacturing process.

The proposed system includes mechanical, electrical, vision and software development. The vision subsystem and the software subsystem are described in detail in the following sections. The general description of each module is as follows:

• Vision subsystem:

This module includes the set of elements that capture images of the bottles. It is further described below.

• Mechanical subsystem:

This module includes the set of mechanical devices that support the vision subsystem.

• Electrical subsystem:

This module includes the electrical cabinet as well as the additional elements and electrical signals needed for the synchronization of the different modules and the communication with the control computer. The electrical and mechanical elements are housed in a cabinet, as shown in Figure 2. The only element outside the cabinet is the illumination system that is placed opposite the camera for a backlight layout. Moreover, the system receives the encoder signals to synchronize the line scan acquisition with the speed of the manufacturing line in order to properly reconstruct the image of every bottle. An additional photo sensor placed in the line indicates whether a bottle has passed and activates the acquisition process. The PC communicates with the camera through the Ethernet connection.

• Software subsystem:

The SW module was specifically designed for this application and it performs the control and configuration of the camera; the reception, processing and analysis of the images; the launching of the detection process whenever an image is available; and the display and storage of the results and other auxiliary tasks of configuration and verification of the status of the different elements of the system.

The layout of the system and the vision module is shown in Figure 2.

By means of an easy-to-use graphical interface, the user can start the process. The inspection process, once started, will automatically continue until stopped by the user. The software subsystem is responsible for collecting the camera lines to reconstruct the image, estimating the extremes of the diameter of each bottle from the captured images, and analyzing the presence of seeds in each identified bottle. If seeds are detected, their dimensions will be calculated. Likewise, the number of seeds per kilogram will be calculated for bubbles of all sizes, and the number of blisters per ton will be estimated for any seed of more than 1 mm in its largest dimension, calculated based on the known thickness for each model together with the rest of the output data identified as relevant, which will be detailed later. Those relevant values will be displayed in the user interface and delivered to the control system. This information will also be stored in a different file for each reference produced. The user is responsible for providing the reference data of the model of the bottles that are being analyzed. The correct indication of the model of the bottle that is being manufactured affects the final calculation. Every model has a different reference thickness value, indicating the same number of detected seeds may rely on a different number of seeds per kilo and blisters per ton. These are the final expected indicators to manage the process.

The vision subsystem and the software subsystem are described in detail in the following sections.

3.2. Vision Subsystem

The vision module includes the set of elements that capture images of the bottles. These elements are: (1) high resolution monochrome linear camera; (2) optics for the linear camera with high depth of field; (3) high-intensity white backlight illumination.

This vision module is in charge of carrying out the functions related to capturing the images. It manages the communication with the camera, sending it the appropriate configuration parameters and controlling the acquisition of the images. It is also in charge of interpreting the messages coming from the camera (which can provide error information) and of sending the captured images to the processing module. The HW of the capture entity is a high-speed monochrome linear camera with a telecentric lens, where the acquisition of each line is synchronized with a rising edge of the encoder pulse of the manufacturing line where the system is located. The camera will understand that an image is constituted when an established number of lines is grabbed. In this situation, an image is created with 3600 lines. The encoder must be capable of providing around 28,000 lines/s in order to be able to achieve the resolution of 0.03 mm/pulse (resolution = 0.03 mm/pixel) necessary for detection of seeds with a minimum size of 0.1 × 0.1 mm².

The seed detection system in bottles consists mainly of the vision system. The mechanical and electrical HW modules do nothing but provide support and allow the capture and processing of images to be possible. The mechanical system consists of the set of supports that hold the camera and the lighting, which is placed behind the bottle to illuminate it by contrast.

The HW elements chosen for the constitution of the optical system are the following:

• Teledyne Dalsa Monochrome Line Camera (Teledyne Dalsa, Waterloo, ON, Canada) 2048 pixels and 52 kHz, Gigabit Ethernet
• Telecentric optics TC4K120
• Backlight lighting MB-BLL206-W-24

The calculations and reasoning that led to this choice are detailed below.

Operating requirements are that the system must work at a line speed of 50 m/min, detect bubbles of 0.1 × 0.1 mm² in a strip approximately 60 mm high in practically the entire surface of the bottle for diameters between 52 and 120 mm.

The required minimum detectable size determines the resolution of the camera, the capture speed and the resolution of the necessary encoder. Considering three pixels as a minimum default both in the x and y axis, it implies having a resolution in x and y of 0.03 mm/pixel. If the field of view is 60 mm, this resolution is achieved on the x-axis with a 2048-pixel camera. In the direction of movement, which would constitute the y-axis of the image, it is also necessary to have the same resolution of 0.03 mm/pixel, which is equal to 0.03 mm/pulse. At a speed of 50 m/min (0.833 m/s), to have 0.03 mm/pulse, it means providing 27,777 pulses/s to capture at 27,777 lines/s. The camera must capture a minimum of 28 kHz, and the line encoder that will synchronize the capture must provide around 28,000 pulses/s. Taking into account all these calculations, the chosen camera is a linear monochrome camera with 2048 pixels and 52 kHz with Gigabit Ethernet communication protocol.

On the other hand, the fact that the bottle is curved suggests the use of an optic that allows for a good depth of field in order to ensure that the largest surface of the bottle is well focused and, in this way, to be able to conveniently detect seeds in the entire diameter of the circular bottle.

The TC4K120 telecentric lens was chosen for this linear camera, which provides a depth of field of 29.2 mm. The maximum working distance of this optic is 174 mm, and it has a length of 337.3 mm, which must be considered for the correct design of the mechanical support and the selection of the location of the system. The operating range was established and verified for distances in the range 130–160 mm between the end of the lens and the bottle surface.

The vision system module and the related SW is of vital importance and will therefore be executed autonomously in an independent processing thread, so that the capture of new information is not delayed and lines are not lost due to problems in other processing stages.

Taking into account the depth of field of this lens and the range of bottle diameters between 52 and 120 mm, an inspected bottle surface is estimated for a width between 50 mm and 97.46 mm, respectively, by 60 mm height in both cases. That is, practically the whole surface of the middle of the bottle will be inspected. In addition, in this calculation a clearance of 2–3 mm was considered in the positioning of the bottles on the belt, so that they do not get stuck, which penalizes the depth of field. That is, the inspection areas will be 50 mm (width) × 60 mm (height) for bottles with a minimum diameter of 52 mm, and 97.46 mm (width) × 60 mm (height) for bottles with a maximum diameter of 120 mm, approximately.

These elements were chosen to be able to provide the desired resolution in both the x and y axes. Therefore, they must have a fixed relative physical layout. In case of movement, the performance of the system in terms of seeds detection will be different and may even not be detected. The support where the camera will be located on the gantry will allow a certain vertical displacement to adjust the inspection window to the possible different cut and shape of the bottle.

3.3. Proposed SW Solution: The Image Algorithm

The proposed SW solution comprises several stages. These stages are:

• Bottle detection: the bottle is identified in the image; this region constitutes the zone to be analyzed.
• Candidate selection: the potential bubbles are detected in the inspected region; these candidates are the blobs that fulfil the features that may correspond to seeds.
• Filtering by deep learning model: these candidates go through a deep learning-based classification model that finally decides whether they are seeds or not.

The processing pipeline is shown below in Figure 3.

The detailed description of every stage is provided below.

3.3.1. Bottle Detection

The line scan camera is capturing lines continuously. As indicated above, there is a sensor that is activated whenever a bottle crosses the sensor line, and this signal triggers the acquisition of the camera until the total number of lines of the image is acquired. These lines are synchronized with the encoder of the manufacturing line. The reason why a photo sensor is placed to detect the presence of a bottle is to make the processing algorithm easier so that the processing stage is launched only whenever a bottle has passed. The idea is to have a single bottle in every image. However, there are situations where this is not possible, and more than one bottle appears together. Images are 2048 × 3600 pixels in size. Figure 4 shows an example of the input image acquired by the camera.

A key stage in the algorithm is the location of the region of the image to be inspected. Through a global thresholding operation, the darker parts of the images are located. In an image with a single bottle, average ‘y’ coordinate across all the rows in the image represents the diameter of the bottle. The inner region is located and in case its area corresponds to the area range for the bottle of the expected dimensions, the area to be inspected is delimited. In case more than one bottle appears in the image, the algorithm discards the smaller region in the image, not having enough pixel size so as to represent a complete bottle. Images with incomplete bottles, whatever the reason for this might be, are rejected.

3.3.2. Candidate Selection Approach

The baseline of the solution aims at detecting the seeds and blisters present on the surface of the bottle by means of classical image processing techniques. In the case of detecting the presence of seeds, the algorithm will calculate their corresponding height and width dimensions. This was the initially deployed solution that arose from the need for a deep learning-based solution for the filtering of false positives and more accurate output.

This candidate selection approach relies on thresholding, filtering and morphological operations to detect the seeds. The stages of the algorithm are as follows. First, an adaptive and local thresholding takes place. For a segmentation from the light foreground for a pixel at position (r,c), a local threshold T(r,c) is calculated within a window of size mask_size × mask_size by means of the following expression,

$$T(r,c)=\mu (r,c)(1+k(\frac{\theta (r,c)}{R}-1))$$

where $\mu \left(r,c\right)$ is the local mean value within the window and $\theta \left(r,c\right)$ denotes the corresponding standard deviation. The parameter R is the assumed maximum value of the standard deviation (R = 128 for byte images) and k is a parameter that controls how much the threshold value $T\left(r,c\right)$ differs from the mean value. If there is high contrast in the neighborhood of a point (r,c), the standard deviation has a value close to R which yields a threshold value T(r,c) close to the local mean. Dark structures on a light background are segmented. Every pixel p(r,c) whose gray value is smaller than the calculated local threshold $T\left(r,c\right)$ is selected. Value for step is 0.1 and mask size of 21 was fixed.

The blobs above the threshold are connected in a 3 × 3 neighborhood and analyzed. Those blobs not having the features expected for seeds in glass are discarded. The identified features for seeds are sizes ranging between 3 and 200 pixels in both width and height, Euler number being −1 or 0 for detection of inner regions, and not having eccentricity. Relation between width and height is also considered in order to remove the joints or other linear marks in the bottle that cannot be seeds. The blobs that remain after these filtering steps are the final identified potential seeds. Figure 5 shows the processing algorithm.

3.3.3. Deep Learning-Based Classification Model

Although the candidate selection is quite good in clean glass, it can be improved. It was identified that a certain percentage of grease stains and other types of irregularities, such as the seal or joint of the bottle, become the cause of false positives. These types of elements often have an appearance very similar to that of a seed (see Figure 6), and after an analysis at the pixel level and considering shape of the clusters of pixels in a blob, the algorithms are not always able to filter them successfully. As a result, a bottle with abundant grease stains can erroneously increase the count of seeds and blisters detected, especially in the large category, which masks the true measurement in these cases.

This last stage of the proposed method is based on supervised deep learning techniques. In this way, the model will be able to learn the precise morphological characteristics of a bubble, a stain or a joint, and generalize this classification for situations of uncertainty that take place during the inspection. These artificial intelligence technologies allow a detailed analysis of the scene, so that the processing grows in complexity and the algorithms are capable of contextualizing the features extracted from the image, similar to how the brain does. The idea for the classification model is to receive all the candidates detected in an image in the previous stage and to act as the final element responsible for filtering out everything that is not a seed in glass.

For the development of the deep learning-based classification model, the following steps were followed. First, the dataset was carefully reviewed to make sure the annotated images fit in the assigned category. Images with the “doubtful” label are removed from the dataset. Next, the model is trained with different parameters, and best performing weights are chosen. Finally, the metrics for the test subset are obtained. The detailed methodology is as follows.

Dataset for Deep Learning Model

The images collected to train a deep learning model is known as a dataset, and it is, in fact, one of the most critical tasks, since it contains the information that will be transferred to the model. It can be said that, regardless of the complexity of the developed software, the quality of the final classification is strongly dependent on the quality of the examples shown to the model during training. In this case it is a supervised training process, which means that all the images used in the training process were previously annotated, and there was an output associated to an input.

The number of images used in a dataset depends on several factors, such as the number of classes, the homogeneity or heterogeneity of the classes that are intended to be discriminated, or the complexity of the architecture of the network used for training. In addition, it can be stated that an excess of similar images, or the use of unbalanced image sets (in terms of the number of images for each type of class) can cause unexpected behavior in the classification, so that a model with outstanding metrics in training may not be able to generalize when exposed to previously unseen examples.

The dataset to develop the model proposed in this paper was gathered from manufacturing lines of three sites of Vidrala, sites in Castellar and in Aiala—Llodio, both placed in Spain, and another site in Elton, UK. The dataset contains around 1500 images that were acquired with the Line Scan camera used in the system. This dataset was used to tackle the different stages in the processing algorithm. On the one hand, the full image was used for the bottle detection algorithm and candidate selection, and full validation of the system. On the other hand, the ad hoc dataset for the generation of the deep learning-based model was also created.

The dataset for the deep learning-based model contains two classes (or sets) of images, identified as “seed” and “non-seed”. The “seed” class includes a wide spectrum of seeds of different sizes, shapes and orientations, and with a well-defined contour in most cases, while the “non-seed” class has grouped grease stains, marks, joint areas and, in general, any element that, due to its appearance, could lead to a false positive detection. Figure 7 shows examples of images belonging to both classes.

The generation of the dataset was carefully executed. The correct identification of an image to one class or the other is not always easy, since they can often have very similar appearance, and can be confusing even to a trained eye with no more information than that of a small crop with a set of dark pixels. Several revisions were performed to make sure proper assignment of the images is executed. It must be considered that some of the images shown in Figure 6 can be very small, not bigger than 10 × 10 pixels size. As stated above, the good performance of the models lies in the robustness and proper annotation of the dataset. In this case, the annotation of the dataset has consisted of assigning a category label to every image.

The final used dataset contains 3771 images, where 1399 images represent seeds or blisters and 2372 represent other elements not being seeds, such as joints, marks, stains, etc. The dataset was generated from images of bottles having different colors and hues, dimensions, and being circular of squared shape bottles. The aim of this is that the dataset may cover all the possible variability the model might find in the real production stage.

The aim of this approach is that there is a unique model capable of identifying seeds and blisters in the whole range of bottles that are manufactured in all lines of Vidrala sites. The development of separate models for all of the colors of the bottles is not desired. This would add difficulty for the scalability, maintenance and, in the end, adoption of the system by the users.

Training of the Model

The training task consists of showing the dataset to a convolutional network, which will iteratively extract unambiguous characteristics of each of the classes in the dataset. The training process ends when the weights and parameters of that network provide a small error in the prediction, that is, a reduced loss value in the validation subset. The different models trained in the project were generated with the deep learning library of MVTech’s Halcon software. Specific hardware (GPU) was also used to speed up the process, which would otherwise take considerably longer.

The original dataset is split up into three groups (training, validation and test), so that the network is able to assess and correct the quality of learning during the training itself.

Halcon’s deep learning libraries in version 18.11 provide three types of networks, from least to most complex. Based on the tests carried out, the networks with more processing layers introduced greater delay when generating the predictions of the classes, which would cause congestion problems at times with a high number of seed candidates. This problem does not occur with the network in compact format, so this was the final chosen architecture. Other design considerations of the model that imply a preprocessing of the input image are:

• Image rescaling. The networks available in Halcon are configured by default to work with three channel images with a resolution of 224 × 224 pixels, which far exceeds the average size of the candidate clippings produced by the system. This fact negatively affects the system in two ways: on the one hand, the computational cost introduces an additional time to that of the inference of the model itself, being fatal in real time applications; and on the other, rescaling to such a large size introduces unnecessary noise into the image. In fact, it was observed that with smaller image sizes where resolution is barely reconditioned, the error calculated after model training is lower. In addition to testing the resolution of 224 × 224 pixels, we have worked with sizes of 96 × 96 and 48 × 48 pixels, the latter being the option ultimately chosen for the input image.
• Preprocessing. Along with input image rescaling, another series of transformations are applied to the image with the aim of feeding the network with the more meaningful input. These transformations were achieved by means of traditional image processing techniques and are: (1) Contrast enhancement: this operation scales the image within the range of maximum and minimum gray levels, improving the contrast between light and dark areas. The purpose of applying it is so that the model is valid for all the variability of colours and hues of the bottles, that can vary from white to dark-green or oak; (2) Gaussian filter: 50% of the dataset was passed through a 3 × 3 Gaussian filter to cause a certain degree of blurring, both in seeds and in the rest of the elements. This effect makes the system more flexible in situations where the seeds do not appear perfectly focused or defined.

In the training process, data augmentation was applied. Data augmentation allows the dataset to be expanded, generating new cuts with slight variations, based on the original images, and prevents possible overfitting. These data augmentation techniques consist of applying transformations to all or part of the images, such as affine transformations of translation and rotation, flipping the image with respect to some axis or other types of operations. Specifically for this project, transformations were applied to rotate and reflect the images, so that the network understands that the characteristics of one class or another are independent of the position and angle at which seeds and spots appear in the image. In our case, it is not convenient to apply strong transformation since the images are very small in size and the features of the seeds and other regions not being seeds is represented in few pixels.

The networks used in the different trainings belong to the family of residual networks, named ResNet. They were proposed by [26] and are well known in the scientific community. The compact and smaller network was ultimately chosen. Figure 8 shows the ResNet family network architectures.

Finally, the hyper-parameters tuning was tackled in the training process. This set of variables partly determines the configuration of the network and has a direct impact on the quality of the trained model. Some of the used parameters were batch size, learning rate, epochs, momentum, gradient descent and weights initialization. Best results were obtained with batch size of 16 during 100 epochs, learning rate of 0.01, momentum 0.9, and square gradient descent as optimizer of the model in the training process. Training was performed from scratch, and weights of the model were randomly initialized. The curve of the best training is shown in Figure 9.

The model manages to remove all the false positive in the image due to grass stains, joints and other undesirable factors. Final output image with the detected seeds can be seen in Figure 10. The example bottle is specially chosen to see how good the performance is. The bottles do not usually present so many seeds or stains.

3.3.4. Proposed SW Solution: Final Integrated System

Once the model was properly trained and best weights are chosen according to the results over the validation subset, it is necessary to integrate the model into the complete SW solution. The development of the integrated SW was executed on a Pentium IV computer running the 64-bit Windows 7 operating system. The PC needs at least one network card to communicate with the linear monochrome camera with Gigabit Ethernet communication protocol. The software tools for the development of the application are:

• National Instruments LabWindows CVI (version 2012)
• Sapera library SDK (version 8.41) for camera control and image capture
• Halcon library (version 18.11)

The model is part of the final integrated system that inspects the bottle in the manufacturing line in real time, 24 h per day and 7 days per week. As explained above, the processing pipeline has several stages. All suspected bubble candidates detected by the first stage of the process are evaluated by the deep learning-based model for the final decision.

The inclusion of this stage with the classification model hardly penalizes the pipeline processing time by a few milliseconds, so the system fulfils the real-time requirement and allows 100% inspection of the bottles at a production rate of up to 250 bottles per minute. Every image processing action takes around 150–200 ms; this range is due to the number of located possible candidates. The higher the number, the higher the number of elements to be classified by the model, thus the processing time is elapsed.

Figure 11 below shows an image of a bottle correctly grabbed by the camera, with detected seeds framed within the inspected area.

When seeds are ultimately confirmed by the classification model, their dimensions are calculated. Likewise, the number of seeds per kilogram will be calculated for bubbles of all sizes; and the number of blisters per ton will be estimated for any seed of more than 1 mm in its largest dimension, calculated based on the known thickness for each. Glass density is considered constant and equal to 2.52 kg/m².

Those calculated relevant values are displayed in the user interface. The information is stored in a different text file for each reference produced. The user is responsible for providing the reference data of the model of the bottles that are being analyzed. The correct indication of the model of the bottle that is being manufactured affects the final calculation. Every model has a different reference thickness value, and this implies the same number of detected seeds may rely on a different number of seeds per kilo and blisters per ton. These are the final expected indicators to manage the process.

The information related to every model that has to be provided to the system is:

• Reference code: alphanumeric string that identifies the model being manufactured;
• Average glass thickness in mm;
• Color: it can be white or flint, green, dark-green or moss;
• Bottle diameter in mm.

Additionally, in the event that the window presents engravings in the lower or upper part of the inspection zone, an exclusion zone can be created that will indicate that said area should not be processed. If no value is entered (0.00 mm), the system will inspect the 60 mm field of view in the bottle height direction.

The operation of the whole developed application is as follows. As soon as the interface is launched, the program checks that the camera is properly connected. The acquisition parameters of the camera are loaded, the camera is automatically initialized, and it is verified that it is working correctly. If this is the case, the visual indicator of “System Information” located at the bottom of the interface, named as “Ready,” turns green. If the camera was not connected or presented any problem, the visual indicator “Ready” would turn red.

By pressing the Start Inspection button, the visual indicator “Grabbing” changes to green when the camera is operational, and it receives the encoder pulses necessary to record the lines that allow the reconstruction of an image of the established dimensions. This only occurs if the line is in motion and generates such encoder pulses. It is also necessary for the image acquisition to take place that the photocell is on and is correctly detecting the passing of the bottle. In case the line is stopped, there are no bottles passing or the photocell is not detecting them correctly, the camera does not record and this “Grabbing” indicator light is off. In this scenario, there will be no image available that can be processed, and therefore the “Processing” indicator light will be in red. If the camera is grabbing and there is an image, the automatic and uninterrupted processing of all the bottles will be carried out. The “Processing” indicator light will then appear in green. The process continues uninterrupted until Stop Inspection is pressed.

Figure 12 shows the interface of the program in the processing stage, with all the associated information.

Regarding the registration of the information, it is both stored and transferred by TCP/IP protocol to the system that manages the production parameters in the manufacturing site. The information that is transferred is the date and time on which each value is recorded, the average number of seeds/kg and blisters/ton of the last 100 bottles and the average value of the number of seeds/kg and blisters/ton in the last hour.

4. Results

The system has been validated through a set of tests. The objective of the tests is twofold. On the one hand, it is needed to ensure that the system works correctly, without errors occurring in 24/7 operation and in real time processing. The proposed algorithm presents several stages and each of them adds additional time, being especially higher when the glass is dirty, and the number of potential candidates increases. The deep learning-based model for final classification of the candidates takes longer as it has to make a higher number of predictions. On the other hand, it is necessary to validate the performance of the proposed method and to verify the correct detection of seeds and correct rejection of the other elements.

Objective comparison with the current laboratory seeds counting procedure available in the Vidrala site is not possible, strictly speaking, as both methods count seeds differently. The automatic seed detection system inspects a fixed region in the bottle as it passes in front of the capture line of the camera, as explained before. The results are displayed on the user screen through graphs. Likewise, three values are estimated and recorded: seed density value for each bottle, mean value per 100 bottles and hourly mean value. In the laboratory, it is not possible to analyze 100% of the production, so the usual practice consists of inspecting a discrete and random set of bottles (less than 10) every day, normally in the morning. These bottles are inspected with a linear system available in the laboratory (in the Aiala site) or through destructive techniques (in the Castellar site) which allows the inspection of part or all of the surface of the bottle. Through this procedure the values of seeds per kilo (for small ones) and blisters per ton (for large ones, side > 1 mm) are obtained. Both values, the one obtained by the automatic system and the laboratory one, are not strictly comparable because neither the same bottle nor (even if it were the same bottle) the same inspection zone are inspected.

Therefore, to assess whether the system is doing it right or wrong, it is necessary to review one by one the images captured by the system and the real bottles grabbed in those images.

In order to carry out this verification of operation in an objective manner, a procedure was established that allows the reality of each bottle to be compared with that estimated by the system. This procedure has included the following steps:

• A set of dark colored bottles and model 1117/242 has been selected (specifically, 10 bottles have been selected from a larger initial set);
• The bottles have been inspected by the system, in which the original and processed images have been stored;
• An expert of Vidrala has reviewed the processed images and has indicated how many seeds are correctly annotated, how many of them are false, and how many seeds the system has not detected. In case of doubt, the expert has the possibility of checking the original images, as well as the real bottles, to establish whether a seed is large or small (1 mm on a side);
• These values have been compared with the results obtained by the system;

This procedure has been repeated for two situations:

• System with baseline approach, that is, classical image-based solution;
• System with new approach with the deep learning-based classification model.

In this way, it will be possible to conclude whether the deep learning-based model has added value over the baseline and also to quantify the real provided improvement.

For each of the 10 bottles, 10 passes were made, 5 for each face, in order to also verify the repeatability of the system. The difficulty in placing the bottles on the tape in the same position for each iteration means that the exact same area is not seen in the 5 images that correspond to the same side, although it is very similar. Said dispersion has not been quantified, although it has been considered that the system presents high repeatability.

To obtain the validation metrics, a manual review of 20 images has been carried out. The bottles show a really excessive number of seeds, which does not correspond to the usual production situation. This can only occur at the moment of starting the oven, which is when the test bottles have been taken. This selection of bottles with an excessive number of seeds has been made specifically for the execution of the tests and validation of the robustness and performance of the system in real time.

These bottles have been processed by both stages in the algorithm and the number of total seeds and blisters has been obtained for each image. The two stages are the image processing-based process that detects the potential candidates (named as baseline); and the final identified seeds after inclusion of the deep learning-based classification model. The aim is to identify whether the deep learning model contributes to the proper detection of the seeds. On the other hand, the images have been reviewed by a Vidrala expert who, after analyzing the images together with the corresponding bottle, has indicated those seeds that are real, the false positives (grease stains or marks detected as seeds) and the false negatives (seeds not detected).

The performance of the system can be quantified with the following metrics commonly used:

$$True Positive Rate \left(TPR=sensitivity=recall\right)=\frac{TP}{TP+FP}$$

$$False Negative Rate \left(FNR\right)=\frac{FN}{FN+TP}$$

$$Precision=\frac{TP}{TP+FP}$$

$$F1=\frac{2\times precision\times sensitivity}{\left(precision+sensitivity\right)}$$

TPR, precision and F1 with values ranging between 0 and 1, with 1 being optimal; and FNR with values between 0 and 1, with 0 being optimal.

The metrics obtained for both approaches are gathered in

Table 1. Metrics of the deep learning-based system compared with baseline approach.

Method	TPR	FNR	Precision	F1
Baseline (classical image processing)	0.96	0.05	0.49	0.55
With deep learning-based model	1.00	0.02	0.97	0.97

.

Some images of these bottles with the detected seeds and blisters (obtained with both the classical image processing approach and with the deep learning-based approach) are shown below. The large seeds named as blisters are marked with white. Figure 13 and Figure 14 gathers some of the images. Additional examples of detections can be found in Appendix A.

Other tests have been carried out with other models of bottle to verify the correct detection. This verification has been made visually by an expert. Unfortunately, due to unavailability of experts, the same exhaustive procedure of manual annotation of real and false seeds has not been followed. Nevertheless, this visual inspection has also been considered very important for Vidrala, since it may assure that the model can generalize well in other models of bottles of different color and thickness. As explained before, it is not intended to develop a model for seed classification for every color, but a unique model for all the possible colors. This has been addressed in the design of the deep learning-based classification model through the pre-processing stage.

5. Discussion

The results shown in Section 4 above reveal a good performance of the proposed method for seed detection and of the whole system itself. The proposed methodology makes feasible the detection of seeds with a degree of precision close to 1.0 and a False Negative Rate close to 0.0. This means that almost all the seeds are detected correctly, that is, no false positives are detected, and besides, the number of real seeds that are not detected is very small, which is of utmost importance in this process.

The inclusion of the deep learning-based classification model as an additional stage in the processing pipeline outperforms the results obtained by the baseline (the classical image processing-based approach without the deep learning model). This is evident in:

• The number of false positives, understood as the detection and counting of grease stains, marks or other elements that appeared in the image and are attributed to seeds in the past, has been drastically reduced;
• The number of false negatives (undetected seeds) is null in most of the images.

From the observation of the images, it can be appreciated that the seeds that are occasionally lost are those that are very faint, in which there is very little contrast between the circular crown that characterizes them and the hue of the bottle. The observation of the faint seeds mainly happens in light-color bottles with low thickness.

Another aspect that has been verified is that the sizing of the seeds is correct. As explained in introduction, small defects are named as seeds and defects with one dimension larger than 1 mm size are considered as blisters. The image is acquired by a line scan camera that is perfectly synchronized with the electrical signal of the encoder of the manufacturing line and with a verified resolution of up to 0.03 mm/line pulse. Moreover, the processing pipeline calculates dynamically, for every frame, the real resolution in the movement axis, to absorb the possible variation in the speed of the conveyor belt that transports the bottles. This makes feasible the precise estimation of the size in pixels of the seeds detected, and direct conversion to millimeters is achieved through multiplication by real frame resolution. Occasionally, some seeds are not framed perfectly and show a smaller dimension than the real one. This happens in the case of faint bubbles, where part of them, in the faintest area and with less contrast, can be diluted with the background and not be detected. The estimation in size of the seeds is considered to be appropriate. This fact is relevant for the proper calculation of both key indicators in the process, that is, (1) seeds per kilo, where all the detected seeds are considered, and (2) blisters per ton, where only big sized seeds with any dimension larger than 1 mm size are considered.

Regarding the implementation of the algorithm, it has been managed to make different tools and environments work together. The camera is controlled by drivers and API through a C environment. This development environment centralizes the control of the camera and reception of images, the processing pipeline, the communication with the system in the plant to deliver the obtained results and also the user interface to gather the instructions given by the user. The camera also receives the electric signal from the photocell and encoder of the manufacturing line. All this happens at a processing rate of up to 250 bottles per minute in a manufacturing line that can move up to 50 m per minute, which is really fast so as to detect seeds as small as 0.1 mm². This is a real-time system that is working in three sites of Vidrala group 24 h per day and 7 days per week.

It has been possible to integrate a deep learning-based model designed and trained with the Halcon libraries in the existing software written in C without penalizing the baseline approach processing time and without prejudice to the actions feedback and communication with the PLC. The chosen network, as well as its configuration and the size of the input image, have allowed the model inference to add only a few milliseconds to the processing sequence. With this, it is still possible to inspect 100% of the production in real time.

Moreover, the system has been proven capable of detecting seeds in bottles of different colors and thicknesses. This validates the generalization capability of the model, and the fact that it is not necessary to have a model ad hoc for every type of bottle in terms of size, thickness and color, where the visual appearance of the seeds may vary. This fact makes easier the applicability and maintenance of the model in the future, and it does not require specific training for further models of bottles that might be designed and manufactured in the future.

Future work may focus on the further inclusion of additional elements that might appear in the glass of the bottle, such as the infuses. The infuses are the pieces of materials that have not been melted in the furnace (for whatever reason) and that are inside the glass along the manufacturing process. The detection of these elements is not frequent at all but can be problematic since it may cause future cracks in the glass containers.

6. Conclusions

This paper presents a new system capable of detecting bubbles (named seeds) of 0.1 mm² size in glass bottles in their manufacturing process 24 h per day and 7 days per week. These seeds are considered blisters if their dimensions in width or height exceeds 1 mm. The identification of seeds and their dimensions is relevant to control the quality of the process in relation to the quality of the bottle for further filling with liquid, for both aesthetic and quality-related reasons. There are two KPI in the process; the number of seeds per kilo, and the number of blisters per ton, which are relevant to regulate the temperature of the furnace and the process itself.

The bottles are of different sizes, shapes and colors, and they move over the conveyor belt at 50 m/min at a production rate of 250 bottles/min. The detection of the seeds requires a system that processes in real time and that must be fully synchronized with the manufacturing line and its control system.

The system we propose in this paper has met all these criteria. The acquisition of the images is made by means of a high-speed linear camera that has been synchronized with the manufacturing line through a line encoder that indicates the conveyor speed, and a photo sensor that activates whenever a bottle passes in front of the camera, and thus the acquisition starts. The images are processes by means of the new proposed method that includes deep learning-based artificial intelligence techniques and the classical image processing approach. The algorithm comprises three stages.

First, the bottle is identified in the input image to only work in the relevant Region Of Interest (ROI) and to discard the other possible areas of adjacent bottles that might appear in the image. Next, an algorithm based in thresholding and morphological operations is applied over this ROI to locate potential candidates for seeds. This stage usually detects more candidates than real seeds. This is due to the fact that there may appear stains, joints and marks in the surface of the bottles that look very similar to seeds, regarding size, shape and pixel values. Therefore, it is very difficult to distinguish real seeds from these stains and marks, which would be identified as false positives if a third stage is not executed. Finally, a deep learning-based model has been developed that can classify with high accuracy whether the proposed candidates are real seeds or not. The method manages to filter out most of false positives due to stains and others in the glass surface, and at the same time, no real seeds are lost. The F1 achieved is 0.97, which is considered to be more than acceptable by the Vidrala quality team.

The proposed method reveals the advantages of the application of deep learning techniques to problems where the classical image processing algorithms are not enough because of the difficulty in finding meaningful features that help distinguish two classes that are really similar, as it is the case with seeds vs. greasy stains.

The processing method has been embedded in a C environment and it can perform 24/7 in real time at a speed of 50 m/min, with a production rate of 250 bottles/min. It inspects 100% of the production.