Smart Decision-Support System for Pig Farming

Abstract

There are multiple participants, such as farmers, wholesalers, retailers, financial institutions, etc., involved in the modern food production process. All of these participants and stakeholders have a shared goal, which is to gather information on the food production process so that they can make appropriate decisions to increase productivity and reduce risks. However, real-time data collection and analysis continue to be difficult tasks, particularly in developing nations, where agriculture is the primary source of income for the majority of the population. In this paper, we present a smart decision-support system for pig farming. Specifically, we first adopt rail-based unmanned vehicles to capture pigsty images. We then conduct image stitching to avoid double-counting pigs so that we can use image segmentation method to give precise masks for each pig. Based on the segmentation masks, the pig weights can be estimated, and data can be integrated in our developed mobile app. The proposed system enables the above participants and stakeholders to have real-time data and intelligent analysis reports to help their decision-making.

1. Introduction

Substantial risk and uncertainty exist in the agricultural production process and the associated supply chain [1]. For instance, unpredictable virus spread may seriously harm pig growth conditions, resulting in large-scale pig illness. The financial wellbeing of the farmer, credit provider, and insurer may incur damages from these operational risks [2].

To make informed decisions, a smooth flow of modern agriculture information is necessary [3], which requires the collaboration of multiple participants and stakeholders. This consists of a number of stakeholders, including farmers, who serve as the industry’s primary producers; distributors and merchants that store, package, transport, and distribute the harvest to consumers; financial institutions that provide insurance and oversee capital allocation; and regulatory agencies responsible for ensuring food safety, environmental sustainability, and financial stability. Participation in and monitoring of the production process is of interest to each stakeholder. For instance, a farmer may track the pig weights to be aware of their growth conditions; banks may use the data to identify nonperforming loans; and regulatory agencies may adopt the data to evaluate environmental impacts. However, it would be costly if we used traditional methods to collect and distribute such real-time agriculture information.

Making decisions from out-of-date information may have negative effects [4]. The recent epidemic of African swine disease in China is a case in point. The reliance on manual data collection and reporting, which was inefficient and prone to human mistake (whether deliberate or not), caused slow government responses and the virus to spread quickly throughout China in just eight months [5]. Dr. Defa Li, Fellow of the Chinese Academy of Engineering, put the estimated economic cost of the pandemic at 1 trillion Yuan [6]. Moreover, insurance companies also incurred significant losses due to the pandemic [7].

Through the use of Artificial Intelligence (AI) [8], Internet-of-Things (IoT) technologies [9,10], and big data analysis [11], the next wave of the agricultural revolution promises to reduce the aforementioned operational risks and promote the flow of information [12,13]. However, the introduction of such technologies in developing nations in Southeast Asia and elsewhere faces a number of particular difficulties, including the predominance of small and medium-sized farms, unusual farm conditions, inadequate capital support, and a lack of skilled management. The current situation of a few farms in Guangdong, China, is shown in Figure 1. We caution the reader that these are small farms and that the farmers are using low-cost designs. While industrialized economies have had the most success with smart farming and IoT in agriculture, the problems in developing nations have received less attention.

In this paper, we propose a smart pig farming support system. The platform collects data on agricultural productivity using cutting-edge sensors, analyzes the data automatically and interactively, and helps different stakeholders to make well-informed decisions. To address the specific challenges in developing countries, we propose to use unmanned vehicles with sensors to monitor the growth conditions of pigs in a pig farm. Specifically, the unmanned vehicles can be installed at the top of pigsties with fixed rails, and pig images can be taken. The captured images are further processed and analyzed with machine learning techniques, where we attempt to do image stitching, pig instance segmentation and weight estimation. Since the data can be captured and analyzed in an automatic fashion, we are able to monitor livestock farming in real time.

Our contributions can be summarized as follows:

• We propose to use unmanned vehicles based on fixed rails to capture pigsty images, which are low-cost and easy to maintain.
• We propose to apply state-of-the-art AI techniques to conduct data analysis, including image stitching, pig segmentation and weight estimation.
• We propose to develop an app for data fusion, which integrates the collected and analyzed information for stakeholders’ visualization.

In the following sections, we first review the related works (Section 2), including image stitching, pig segmentation, and pig weight estimation methods. Then, in Section 3, we describe each of our proposed smart pig farming system components. We further give experimental results in Section 4.

2. Related Work

2.1. Image Stitching

Image stitching is defined as combining two or more images of the same scene into one full image, which can be called a panoramic image [14]. Due to the advantage of being robust against scene movement and fast processing, many existing works [15,16] adopt feature-based techniques. This method aims to use properly designed features, such as SIFT [17], to get the input image relationships, which are further used to uncover the overlapping areas between input images.

The idea of the feature-based image stitching method is to build the correspondence for points, lines, edges, etc. In order to produce robust stitching results, it is of great significance to adopt scale-invariant and translation-invariant detectors. There are many prevailing feature detectors [18], including SIFT [17], ORB [19], KAZE [20], and AKAZE [21]. To be specific, SIFT [17] is designed based on the Difference-of-Gaussians (DoG) operator, which is invariant to image rotations and scales robustly. Alcantarilla et al. [20] exploit non-linear scale space through non-linear diffusion filtering and propose the KAZE features, which reduce image noise. AKAZE [21] is an improved version of KAZE. Since it uses the modified local difference binary (MLDB) algorithm, AKAZE is a computationally efficient framework. In this paper, we empirically use SIFT as our image stitching approach, since it gives better results than other approach.

2.2. Segmentation Techniques for Pig Images

Semantic image segmentation [22] is a classical computer vision task that aims to classify each of the image pixels into an instance. With the development of deep learning [23], prevailing segmentation models, such as Faster R-CNN [24] and single-shot detection (SSD) [25], perform well on various kinds of datasets. Specifically, DeepLabV3+ [26] improves upon previous work [27] with several improvements, such as adding a simple and effective decoder module to refine the segmentation results. It is proposed to perform multi-class semantic segmentation. Later on, several research works, such as ViTDet [28] and MViTv2 [29], propose to adopt the transformer as segmentation model backbones. MViTv2 [29] is able to perform both image classification and object detection tasks. The aforementioned methods mainly validate their efficacy on the COCO dataset [30].

Pig segmentation is under the umbrella of the general image segmentation task, allowing us to produce the fine-grained shapes and locations of all pigs in given images. The segmented results interpret the complex scene information and give intelligent analysis for the given images. Since the produced results can assist both pig body part identification and behavior recognition tasks [31], many research works [32,33,34] focus on tackling the pig segmentation problem. To be specific, Seo et al. [32] adopt You Only Look Once (YOLO) [35] to separate touching-pigs in real-time. Shao et al. [33] use the segmentation techniques to predict the posture changes of pigs. In this paper, we adopt the pretrained Mask R-CNN [36] model, which is developed on top of Faster R-CNN [24]. We further use our annotated pig images captured in local farms to fine-tune the pretrained model, such that the fine-tuned model is able to give precise masks for pigs.

2.3. Pig Weight Estimation

Weight is an important index in pig rearing [37] and has an effect on managing various stages of the supply chain [38,39]. In terms of the farm scale, one can evaluate a pig’s daily gain and nutritional status through pig weights [40]. Specifically, with real-time pig weight data, farmers can raise pigs in good or bad nutrient status separately to meet the uniform marketing weight standard [37], which brings convenience to the pig farming process. Traditional pig weight measurement requires direct contact with pigs, which is limited by its low efficiency [37]. The non-contact measurement of pig weight is challenging and has drawn much attention.

Some research works [41] adopt additional facilities other than cameras for more weight-related information capture. To be specific, Shi et al. [41] propose to utilize a binocular stereo vision system to analyze and estimate pig body size and weight. Pezzuolo et al. [42] adopt a Microsoft Kinect v1 depth camera to get the pig body dimensions. However, these methods require high-cost implementations and thus are not applicable to our focus scenarios, which are the rural areas of developing countries. Therefore, we refer to the method proposed by [37], where we estimate pig weights based on pig sizes.

2.4. Summary

In

Table 1. Comparison between our proposed method and other methods, where we list our strengths and weaknesses.

Component	Strength	Weakness
Image stitching	Our detected keypoints are dense and scattered generally all over the images.	Our processing time is longer than other methods, because of the descriptor size.
Image segmentation	Our average precision is better than other methods.	Our processing time is longer than other methods, because of the backbone complexity.
Weight estimation	Only monocular camera is required, which is low-cost.	Our estimation accuracy is lower than methods using more sensors.

, we present the comparison between our proposed method and other methods, where we show our strengths and weaknesses. Specifically, our adopted image stitching and image segmentation methods have good performance on the unmanned vehicle captured pig images, where we produce dense detected keypoints to conduct image stitching and achieve high average precision for pig instance segmentation. However, our processing time is relatively longer than other methods. This is acceptable since we do not require instant feedback on the detected results. After we obtain the captured images from our unmanned vehicles, we process these images at our local machines. In terms of the weight estimation component, we utilize a monocular camera, which may yield inferior performance to the depth camera, but it is low-cost and gives low tolerance on the test samples.

3. Methodology

Specifically, our proposed smart pig farming support platform is intended to standardize agricultural procedures, boost farm output, improve pest and disease management, and lessen operational risks in agriculture. This platform enables us to conduct production optimization based on subject-matter expertise and sensor data. Governments, financial institutions, and regulators can also identify potential operational and financial risks and get ready for them. The proposed system primarily supports livestock pig farming. Pig farms can identify pig positions and pig weights and monitor their weight in real time.

We show the proposed smart pig farming support system architecture in Figure 2, which includes three main components. Firstly, we use unmanned vehicles equipped with on-site sensors to collect data at the farm. Secondly, the data are then processed utilizing inexpensive edge computing nodes that do not have issues with sporadic Internet connections. Thirdly, the data are analyzed, the findings are summarized, and they are made accessible on an open platform that supports an ecosystem. The ecosystem supports a number of business and agricultural activities as well as various operations. We present the details of our system components below.

3.1. Data Collection with Unmanned Vehicles

We use rail-based unmanned vehicles that are equipped with surveillance cameras to monitor the livestock pigs. To be specific, the vehicles are installed on the top of pig farms, and the rails are designed to be vertical to the pigsty, where the demonstrations are given in Figure 2. The unmanned vehicles move from one end to another end and take consecutive images simultaneously. We show the consecutive raw images taken by our rail-based unmanned vehicles in Figure 3. They are running at regular intervals automatically so that the pig growth conditions can be monitored in real time.

Since we need to train a new pig segmentation model, we hire annotators to perform semantic polygon mask labeling for pig images. In total, we have 636 and 54 images for training and validation, respectively. Moreover, we also collect real pig weight data to estimate the pig weights. Due to our limited on-site manpower and the number of pigs, we collected 100 real weight samples; then, we split the collected data into 80 training and 20 test samples.

3.2. Image Stitching

It can be observed that the obtained consecutive scanned images overlap with each other. If we directly use these images for pig segmentation and weight estimation without further processing, the same pigs may be present in different scanned images, which makes pig management difficult. To avoid this, we propose using the image stitching approach to stitch all the images in an attempt to remove overlaps and give an overall perspective on the livestock pigs in the captured pigsties. We follow the methods of [15,16] to combine two or more overlapping images to make one larger image, which is realized in four steps.

In the first step, we aim to detect key points and obtain feature descriptors. Here, we use the SIFT [17] detector since it is rotation-invariant and scale-invariant. In the second step, we match key points with features. Specifically, we define a region around each key point, and then compute local features from normalized regions. Based on the local descriptors, we can perform the matching. In the third step, we firstly sample four random potential matches from the second step, which are used to compute the transform matrix H using direct linear transformation. We use the computed H to obtain the projected points $x^{\prime }$ from x, which is denoted as

$$x^{\prime }=Hx.$$

(1)

where x and $x^{\prime }$ are potentially matching pairs. We then count points with a projected distance smaller than a defined threshold, which is set as 0.6. We view these counted points as inliers. We repeat the above process of the third step and return H with the most inliers.

In the fourth step, we aim to stitch two given images, which are depicted as $I_1$ and $I_2$ for demonstration. Since we observe that there are the blurry regions in the final panoramic image, we apply linear blending to reduce this effect. Technically, we first define left and right-stitched margins for blending. We then attempt to define weight matrices $W_1$ and $W_2$ for $I_1$ and $I_2$, respectively. For $I_1$, we define the weights from the left side of the output panoramic image to the left stitched margin as 1, and the weights from the left stitched margin to the right stitched margin are linearly decremented from 1 to 0. For $I_2$, we define the weights from the right side of the output panoramic image to the right-stitched margin as 1, and the weights from the left-stitched margin to the right-stitched margin are linearly decremented from 0 to 1. We further apply the weight matrices and combine $I_1$ and $I_2$. The stitched images $I^{\prime }$ are depicted as

$$I^{\prime }=W_1{I}_1+W_2{I}_2.$$

(2)

3.3. Pig Segmentation

Here, we aim to locate each instance of the farming pigs, such that we are able to perform individual analysis for them. To this end, we adopt the state-of-the-art semantic segmentation algorithm Mask R-CNN [36]. Technically, there are two stages of Mask R-CNN. First, based on the input image, multiple region proposals are generated where there might be an object. Second, the model predicts the object classes, refines the bounding box positions, and also generates a mask in pixel level on the predicted masks from the first step. Both stages are connected to the backbone structure.

Mask R-CNN uses a ResNet [43] as its backbone. It utilizes standard convolution and FC heads for mask and box prediction, respectively. Moreover, the Feature Pyramid Networks (FPN) [44] is also integrated into the ResNet model, which improves the segmentation accuracy without sacrificing the inference speed. To give the segmentation model a good training start point, we use the ImageNet [45] pretrained models to further improve the segmentation performance. Since there is no publicly available model for pig segmentation, we use our labeled data for model training.

3.4. Pig Weight Estimation

Pig weights can be used to monitor pig growth status. With real-time weight data, farmers are able to know if pigs are in a good or bad nutrient status and change their fodder, which brings convenience to the pig farming process. To estimate pig weights according to our generated segmentation masks, we first fit the obtained masks into ellipses. We utilize the major axis and minor axis as pig length h and pig width w, respectively. Then, we follow the equation proposed by [37]. However, considering the particular pig breeds and camera installation conditions in our farms, we need to modify part of the given equation parameters.

Specifically, we denote the weight estimation equations [37] as

$$M=a\times h^b\times w^c,$$

(3)

where M denotes the estimated weights and a, b, and c are parameters that should be calculated from our collected data. In order to fit this equation, we first transform Equation (3) into

$$\log \left(M\right)=\log \left(a\right)+b\log \left(h\right)+c\log \left(w\right).$$

(4)

Since Equation (4) is a linear system, we further adopt the linear regression algorithm to obtain the above a, b, and c parameters. Our final optimized heuristic equation to calculate pig weights is denoted as

$$M=0.0017\times h^{1.1908}\times w^{1.0618}.$$

(5)

The tolerance of Equation (5) on our test samples is 1.8%.

4. Experiments

4.1. Implementation Details

Unmanned vehicle characteristics. The material of our rail-based unmanned vehicle is iron, the size of which is 80 cm × 40 cm × 30 cm. The rails are installed 3 m above the ground, where the vehicle moves forward at 0.1 m/s. The video camera is attached to the vehicle and points vertically to the ground. The camera takes pictures at the interval of 1 s. The camera resolution is 1080p. To control the unmanned vehicle and camera, we use the Firefly-RK3399 (https://en.t-firefly.com/Product/Rk3399/spec.html (accessed on 3 October 2022)) platform.

Segmentation model. Regarding our adopted pretrained segmentation model, we adopt the Detectron2 (https://github.com/facebookresearch/detectron2 (accessed on 3 October 2022)) implementation. We fine-tune the pretrained segmentation model with a learning rate of 0.00025, and the training iteration number is set as 3000.

Operation cost and efficiency. Our whole on-site system implementation for a one-row pigsty costs around 5000 CNY, which is about 702 USD. This is affordable for our deployed local farms. In terms of our operation efficiency, we perform the image stitching and pig weight estimation components in the CPU, and the segmentation component is conducted in the GPU. To be specific, the image stitching and weight estimation components cost about 10 min and 5 s for a pigsty. We use a Nvidia 2080 GPU to obtain the segmented results, and the inference time is 0.3 s per image.

4.2. Image Stitching Results

In Figure 4, we present one resulting image of key points detected by SIFT [17]. We empirically observe that our adopted method gives abundant key point detection results. In Figure 5, we demonstrate a comparison of the matched key points between two images with various features, where we show the key point matching results of ORB [19], KAZE [20], AKAZE [21], and SIFT [17]. Since we do not have ground truth labeling, we only evaluate the performance of these methods qualitatively.

Specifically, it is observed that the SIFT approach produces few intersected matching lines, and the detected key points are diverse. In contrast, ORB and KAZE give many intersected matching lines, and the diversity of AKAZE-generated key points are limited. Key point matching is one of the intermediate steps, and its robust performance is important to our following processing. In Figure 6, we show the qualitative results of the stitched pig images, which cover multiple pigsties under our rail-based unmanned vehicles. The presented qualitative examples demonstrate the efficacy of our adopted method for pig farming image stitching.

4.3. Segmented Results

In

Table 2. Quantitative results for our pig segmentation validation set. IoU denotes Intersection over Union. FPN denotes Feature Pyramid Network. We show average precision (AP) results of various backbone models.

Method	Average Precision
	IoU = 0.50	IoU = 0.75	IoU = 0.50:0.95
DeepLabV3+ [26]	71.1	37.2	36.5
ViTDet [28]	67.0	36.6	37.4
MViTv2 [29]	78.2	40.2	42.3
ResNet50+FPN (Ours)	97.1	87.7	72.4
ResNet101+FPN (Ours)	98.1	90.2	77.5

, we present the quantitative results of our segmentation model, which is evaluated with average precision (AP) under different Intersection over Union (IoU) values. Specifically, we list the quantitative results of various backbone models. DeepLabV3+ [26] improves upon previous work [27] with several improvements, such as adding a simple and effective decoder module to refine the segmentation results. It is proposed to perform multi-class semantic segmentation. However, since we only need to perform segmentation on pig instances, we observe that DeepLabV3+ fails to yield better results than our method. Both ViTDet [28] and MViTv2 [29] adopt the transformer as their model backbones. Based on the experimental results, our adopted ResNet [43] + FPN [44] architecture generalizes to our particular scenarios and gives the best performance among our listed methods. When IoU = 0.50:0.95, our average precision achieves 77.5% with the ResNet-101 model, which indicates that our model performance is consistent under various conditions. ResNet-101 [43] yields better results than ResNet-50 due to its higher complexity.

In Figure 7, we show the pig segmentation results of images taken by our unmanned vehicle, where the results are from various scenarios. The challenge of pig segmentation in our system is that we can only take images from the top view, which suffers from different light conditions. Moreover, the livestock pigs have various poses, and some of them overlap with each other. However, we observe that our adopted Mask R-CNN framework still has achieved over 90% average precision for both IoU = 0.5 and IoU = 0.75, showing the robustness of our system. Specifically, in the second instance of Figure 7, although only parts of the pigs are shown in the image, our model still successfully captures the precise location of these pigs. In the third instance, there are multiple pigs with different poses, and our model also gives correct segmentation results.

4.4. Weight Estimation Results

In Figure 8, we present visualizations for the weight estimation of each pig, giving an overview to farmers. The tolerance of our adopted Equation (5) on our test set is 1.8%. The quantitative and qualitative results both validate the usefulness of our adopted pig weight estimation method. The advantage of our system is that it gives real-time feedback on livestock pigs automatically. With the real-time captured weight data of each pig, farmers are able to know the nutrition status and growth conditions of pigs.

5. Limitations

The proposed system has two main limitations. First, the image stitching results may fail due to pig movement, as shown in Figure 9. This is because our image stitching method assumes that images for stitching contain a static scene and objects only; when a pig moves too much across frames, failure cases may occur. However, our current implementation aims to get the overall estimated pig weights, instead of tracking each pig. Hence, we ran our unmanned vehicle multiple times a day to capture pig images in an attempt to alleviate this issue.

Second, our current implementation only estimates the pig weights based on the segmentation masks, which means we can only obtain an overview of the pig weights. It is useful to uncover some extreme cases. However, we cannot track the weight information and growth status for each individual pig. To improve this, we need to develop pig recognition algorithms so that we are able to track the real-time data and obtain the continuous weight change for each pig. We leave this to our future work.

6. Conclusions

The next phase of the agricultural revolution will increase agricultural output and enhance the sustainability and effectiveness of current farming methods. To support pig farming in developing countries, collecting real-time data and performing relevant analysis is critical. In this paper, we present a smart decision-support system for pig farming, which is low-cost and affordable for developing countries. Specifically, we first adopt on-site rail-based unmanned vehicles to capture all pigsty images at certain intervals. To avoid overlapped pigs in captured images, we conduct image stitching; then, we use an image segmentation model to output pig segmentation masks. Based on the extracted masks, the pig weights can be estimated, and data can be integrated in our developed mobile app. Our proposed system enables pig farming participants and stakeholders to have real-time data reports to help their decision-making.