Reliable Integrity Preservation Analysis of Video Contents with Support of Blockchain Systems

Abstract

In this article, we propose an integrity preservation analysis scheme of video contents working on the blockchain systems. The proposed scheme stores the core points of video contents analysis in the blockchain system permanently so that any user can easily verify the results of the proposed analysis procedure and their reliability. The scheme first examines the codec software characteristics of digital camera devices and video editing tools in advance, and stores them in the blockchain systems. Next, the scheme extracts the codec software characteristic from the target video file and compares it with the prepared characteristics in the blockchain system. With a matched characteristic, the scheme finds out the source camera device or the source video editing tool that generates the target video file. We also propose an integrity preservation trace scheme to record the transformation history of video contents into the blockchain systems. This scheme compares the original video and its transformed video frame by frame, and stores the comparison result with a hash value of the transformed video in the blockchain system. Then, the integrity analysis and transformation history of the target file can be easily searched in the blockchain system, where the hash value of the video file is used as the index of searching operation. We implement the proposed scheme into a practical tool upon a commercial blockchain system, Klaytn. Experimental results show that the proposed scheme carries out the integrity analysis of video contents with 100% accuracy, and provides the transformation history of non-original video contents with 100% accuracy when a proper parameter is given. It is also shown that the proposed scheme completes the integrity analysis within at most one second, and the search operation for transformation history within at most four seconds.

1. Introduction

The video of surveillance digital cameras plays the role of providing crucial evidence for criminal investigation or accident site examinations in many cases. However, many commercial software tools for manipulating video contents are widely distributed, and thus, modifying the video contents becomes very easy. Unless the integrity of video including critical scenes is proved, the video may be useless for evidence or may result in another conflict caused by videos that have been tampered with Moreover, deepfake videos can cause big trouble by spreading false information. Hence, the integrity analysis of video contents has become very important for digital forensics investigations. This article deals with the integrity analysis of video contents and verifies whether the video contents is original or has been tampered with.

A lot of research has progressed in the areas of digital video forensics. The existing techniques of digital video forensics can be classified into three groups: detection of tampered videos, recovery of deleted/hidden videos, and source authentication of videos [1,2,3]. Studies for the detection of tampered videos are roughly divided into an active intrusive approach and a passive blind approach. In the active intrusive approach, a known identifier trace such as a signature or watermark is embedded imperceptibly into the video contents at the recording time, and it is utilized to discover tampering evidence of the forged video [4]. In the passive blind approach, descriptive features such camera sensor artifacts, coding artifacts and material object features are extracted from the video data without the help of signatures or watermarks, and they are analyzed to detect abnormal patterns of forged multimedia [5,6]. The studies for the recovery of deleted/hidden videos design the mechanism of how to search and extract frame artifacts of deleted video upon unallocated storage space of camera devices, and how to assemble found video fragments [3,7]. The studies for the source authentication of video focus on identifying the specific model of source digital devices or video editing tools by examining unique characteristics of digital video devices, such as parameters of software codecs, lens radial distortion, and sensor imperfections [1,8,9]. The proposed scheme handled in this article belongs to the category of source authentication of videos.

In order to verify whether the target video file is generated by a digital camera device or by a video editing tool, the proposed scheme examines the characteristics of codec software used to compress the image stream into a small-size video file. The scheme prepares the codec characteristics of as many camera devices and editing tools as possible, and stores them in a form of signatures database in the blockchain system. Next, the scheme extracts the codec characteristic from the target video file and compares it with the prepared characteristics in the blockchain system. If the extracted characteristic is matched with one of the digital camera devices, the scheme determines if the target video file is original. On the other hand, if the extracted characteristic is matched with one of the video editing tools, the scheme determines if the target video file is edited, i.e., not original. This approach has a drawback: it cannot discriminate in cases in which video contents are preserved without any tampering during the manipulation of video editing tools. To overcome this drawback, we propose another mechanism that traces the integrity preservation of video contents after the target video file is retouched using a video editing tool. Figure 1 shows the trace procedure of integrity preservation for a video file that has been retouched but not tampered with. In the proposed scheme, the front server extracts the software characteristic from the original video and the multimedia contents of original video and those of retouched video. The backend blockchain system verifies the integrity of the original video and stores the integrity verification result of the original video and the comparison result between the original and retouched videos. Then, a user can easily search for the integrity preservation analysis of the original and the retouched video files.

In the proposed scheme, the back-end blockchain system takes the role of keeping the core parts of the proposed scheme immutable, while the front-end server takes the role of processing heavy computation, such as comparison operation of video contents and hash operation of video files. If the whole source codes are located in the front-end server, the analysis result becomes changeable whenever the source codes are updated. In order to guarantee the reliability of the proposed scheme, the core parts affecting the analysis result are located in the back-end blockchain system so that they are not allowed to change.

To evaluate the feasibility of the proposed scheme, we implement the proposed scheme into a software tool working upon a commercial public blockchain system, Klaytn [10]. The software module in the front-end server is implemented with the Javascript language, and the software module in the back-end blockchain system, referred to as smart contract program, is implemented with the Solidity language. In the integrity analysis evaluation, the implemented tool verifies the integrity of two hundred original videos and four hundred retouched videos with 100% accuracy. The implemented tool completes the integrity analysis very quickly, within at most one second, because it examines only the metadata of video files and not their multimedia contents. In the evaluation of integrity preservation trace, we find a noticeable phenomenon: the frames of the original video are not exactly the same as their corresponding frames in the retouched video. Even though there is no manipulation when transforming the original video into the retouched video using a video editing tool, the red, green and blue component values in the original video file are slightly different from those in the retouched video file. So, if any difference between an original video and its retouched video is not allowed, we fail to verify the integrity preservation of the video files which have been retouched but not tampered with in all cases. When a proper difference of red, green and blue components in each pixel is allowed, the implemented tool verifies the integrity preservation of retouched video files with 100% accuracy. When obtaining the comparison result between an original video and its retouched video, the execution time of the implemented tool is affected heavily by the size of multimedia contents, and thus increases proportionally to the file size, When searching the comparison result with a hash value of the input video, the execution time of the implemented tool is affected slightly by the size of multimedia contents, and the search operation is completed within, at most, four seconds in our experiments.

The contributions of this article are following;

• This article proposes a trace scheme of multimedia integrity preservation for retouched but not-tampered videos. To the best of our knowledge, this study is the first to record the transforming history from original videos to their retouched videos in order to trace the integrity preservation of the retouched videos.
• To guarantee the reliability, the proposed scheme stores its main code parts and the transforming history of retouched videos into the public blockchain system. When they are stored in a blockchain system, they become immutable, whereas they can be changed at any time when they are stored in a centralized server. The proposed scheme stores its main codes in the form of smart contract upon the public blockchain system The smart contract is a digital programming code suitably designed for the public blockchain system.
• To evaluate the practical feasibility of the proposed scheme, we implement the scheme upon a commercial public blockchain system and confirm that the scheme verifies the integrity preservation of retouched videos with 100% accuracy within a reasonable execution time. To avoid the maximum size constraint of a single smart contract, the scheme consists of twenty smarts contracts, and they are linked to their blockchain addresses.

The rest of this article is organized as follows. Section 2 surveys the related previous studies and explains the benefit of the proposed scheme compared to the previous works. Section 3 describes the proposed integrity analysis and integrity preservation trace scheme in detail. Section 4 explains the implemented software tool of the proposed scheme and deals with the experimental evaluation results of the implemented tool. Section 5 concludes this article with discussions for further study.

2. Related Works

Due to device manufacturing costs, most legacy camera recording systems were not equipped with intentional authentication information such as an invisible watermark. Therefore, the passive blind approach without any authentication information has been more actively studied compared to the active intrusive approach with authentication information. The passive blind approach extracts descriptive features such camera sensor artifacts, coding artifacts and material object features from the video data, and it analyzes them to detect abnormal patterns of forged multimedia [5,6]. Passive blind techniques are classified again into detection of inter-frame forgery and detection of intra-frame forgery [5]. In the inter-frame forgery, some frames are deleted, inserted or duplicated, while some pixel region within a frame is manipulated in the intra-frame forgery. Most of the previous studies [11] for the inter-frame forgery examine the artifacts of optical flow or motion vectors of objects, and those [12,13,14] for the intra-frame forgery examine the artifacts of noise residue or multiple compressions. All of these passive techniques determine the target video as tampered when the examined artifacts exceed a given threshold. However, it is very difficult to find a proper threshold, and an improper threshold may result in a lot of incorrect detections, such as True Negative detection and False Positive detection.

Recently, deep-learning-based detection techniques [15,16,17] have been actively investigated to find out the tampered videos. These deep-learning-based methods do not require the threshold decision of abnormal patterns and automatically find out the suspicious static images in the target video with the forgery probability. However, this deep-learning approach requires the prelearning procedure with a well-defined training set. The performance of the deep-learning approach heavily depends on the accuracy of its pre-learning procedure.

The recovery approach of deleted videos focuses on searching artifacts of deleted files upon unallocated storage are of file systems, when the deleted video files remain in the unallocated area until they are overwritten with new video files [18]. The searching technique is called carving [19], and it tries to find the unique header or the footer signature of deleted files. Recent carving methods [20,21,22,23] considered the case that a deleted video file is split into multiple fragments and the fragments are spread in non-continuous locations. They search for spread fragments and combine them into a single file based on the information of fragment size and fragment location in the video file.

The source authentication approach focuses on identifying the specific model of source digital devices or video editing tools. The traditional source authentication methods [1] examine the unique characteristics of digital video devices such as parameters of lens radial distortion and sensor imperfections in the artifacts of digital multimedia contents. Recent source authentication methods [8,9,24,25,26] examine software artifacts of video codec in the file metadata, instead of artifacts of digital multimedia contents. These studies consider both software characteristics of video editing tools and digital video cameras. If the extracted metadata are matched with the software characteristics of some camera device, the scheme determines that the given video file has integrity-guaranteed multimedia contents. On the other hand, if the extracted metadata are matched with software characteristics of some video editing tool, the scheme determines that the given video file has integrity-unguaranteed multimedia contents. This approach can quickly verify the name of the video editing tools or camera devices that last touched the target video file. However, this approach is unable to discriminate in cases in which multimedia contents are preserved without any multimedia modification, even though the video is retouched by some video editing tool; for example, file format converting from AVI to MP4. On contrary, the scheme proposed in this article is designed to discriminate in cases in which multimedia contents are preserved without any multimedia modification, even after the video is retouched by a video editing tool.

Due to the immutability of data stored in the blockchain system [27], a few recent studies [28,29,30,31] exploit the blockchain system to trace the integrity of video. A blockchain is a digital ledger system that is duplicated and distributed across the network of computer systems, which makes it quite difficult to break. The most expensive resource in the blockchain system is to store a record permanently, because all computers in the blockchain system cooperate to make a consensus and store the record separately in their ledger storage [32]. The method of Yatskiv et al. [28] stores the hash value of each video file in the blockchain system, and protects unauthorized changes by comparing the hash value of the target video with the hash value stored in the blockchain system. The method of Ghimire et al. [29] stores the hash of video segments in chronological order while recording videos in the digital devices. This approach belongs to active intrusive approach and suffers from frequent expensive record operations of the blockchain system, where authentication information is embedded in the blockchain system instead of multimedia contents. The method of Ghimire et al. [30] stores the hash values of captured scenes and suffers from frequent record operations of the blockchain system, similar to the method of Yatskiv et al. [28]. This method segments the continuous stream of captured scenes, calculates their hash value and stores them periodically in the permissioned private blockchain system. The method of Ghimire et al. [31] automatically selects the key frames of target videos with support of the deep learning technique and stores their hash values in the blockchain system in order to guarantee the integrity of the key frames. However, all of these approaches have drawbacks that cannot discriminate in cases where multimedia contents are preserved without any multimedia modification after the video has been retouched by some video editing tool. The scheme proposed in this article is designed to overcome this drawback by storing the transforming history of videos in the blockchain system. Additionally, the proposed scheme requires one-time record operation with short data for a video file to store its integrity analysis result, whereas most of the previous methods [29,30,31] require frequent record operations with large data on the blockchain system.

3. Proposed Scheme

The proposed scheme consists of two mechanisms. One is to verify the integrity of an original video file by examining the codec software characteristic of the file. The other is to trace the transforming history of a non-original video file by searching the comparison result stored in the blockchain system. The former mechanism is explained in detail in Section 3.1, and the latter mechanism is explained in detail in Section 3.2.

3.1. Analyzing Integrity of Original Video Files

Our previous study [25] deals with the integrity analysis of original video files based on the examination of codec software characteristics. The method given in our previous study prepares two signature databases for video editing tools and digital camera device, respectively. This method extracts the software characteristic from the target video file and compares it with signatures in the two database, as shown in Figure 2a. If any signature in the database of camera devices is matched, the target video file is decided to be original. If any signature in the database of video editing tools is matched, the target video file is decided to not be original. Figure 2b shows a signature example in the database of video editing tools for the editing tool named Movavi Video Editor.

However, the method given in our previous study has a drawback in that the integrity analysis result becomes changeable whenever its source codes are modified. The integrity analysis result of the method is not permanent, and thus, this method does not provide sufficient reliability. To enhance the reliability of integrity analysis result, the mechanism proposed in this article upgrades the method of our previous study [25]. In order to fix the software source codes permanently, the proposed mechanism stores the source codes in the public blockchain system. Considering the expensive record costs of a blockchain system, the proposed mechanism minimizes the data amount stored in the blockchain system. The core parts of the source code are stored in their original forms and the rest non-essential source parts are stored as a shortened form of their entire hash value. That is, the two signature databases shown in Figure 2a and the signature comparison procedure are stored in their original forms in the blockchain system, while the rest of the source codes are stored as their entire hash value. Separation of core source parts prevents the reliability corruption caused by modifications of the rest of the source parts. Even when the rest of the source codes are updated, the key operation of integrity analysis is still immutable, and its analysis result preserves its reliability. Additionally, it is easy to confirm the comparison procedure between the extracted signature and the signatures in the databases, where the signature extracted from the target file is always displayed as a form of log and the signature databases are open to any user through the public blockchain system. If the whole source code, including the databases, is stored as a form of its hash value, the record costs of the blockchain system are reduced, but it becomes difficult to check the signature comparison procedure between the extracted signature and the signatures in the databases.

Figure 3 shows the enhanced procedure (This procedure is not depicted in Figure 1 for the sake of simplicity) to check the source code reliability of the proposed scheme compared to our previous study. The core parts of source codes are constructed as digital programming codes, called smart contracts, in the public blockchain system. The remaining parts of the source codes are constructed in the front server, and their hash value is stored in the blockchain system. Then, the core parts of source codes are fixed permanently due to the systemic nature of blockchain systems, and the remaining parts of the source codes can be checked to see whether they have been updated or not.

Figure 4 shows the detailed structure of the proposed mechanism. The signature databases are stored in their original form in the public blockchain system, and the comparison operation with the extracted signature is constructed as a smart contract program on the blockchain system. The rest of the software modules, such as those extracting the signature from the input file and displaying the ongoing analysis results, are constructed as a stand-alone tool working on a local computer, called the front-end server. The proposed mechanism calculates the hash value of the whole source code in the front-end server and compares it with the hash value stored in the blockchain system. In this structure, whenever the source code in the front-end server is updated, its hash value is stored with its time stamp separately in the blockchain system. Then, it can be checked to see which update version the source codes in the front-end server belong to.

3.2. Integrity Preservation Trace of Reretouched Video Files

Although the mechanism explained in Section 3.1 can quickly verify the name of the video editing tools or camera devices that last touched the target video file, it is unable to discriminate in cases in which video contents are preserved without any modification during retouching of the original video file by some video editing tool. It is common to convert the file format of an original video file into another format suitable for running on portable multimedia devices; for example, from AVI file format to MP4 file format. In this case, the mechanism explained in Section 3.1 determines the converted video file as non-original, but the integrity of the multimedia contents in the converted video file is preserved in its original form.

To overcome this drawback, the proposed scheme provides another mechanism, which is not considered in our previous study [25]. The mechanism compares multimedia contents of the original video and those of the retouched video frame by frame. When all corresponding frames of the original video and the retouched video are the same, this mechanism decides that contents of the two videos are equal. If some corresponding frames of the original video and the retouched video are different, this mechanism decides that contents of the two videos are not equal. The comparison result and the integrity analysis result of the original file are stored in the public blockchain system with a hash value of the retouched video file. Then, the integrity of the retouched video file can be traced with its hash value on the blockchain system. The retouched video file is determined as original only when the comparison result is equal and its corresponding source video file has integrity-guaranteed video contents.

Figure 5 shows the flow chart of the procedure to compare the multimedia contents of two videos. The scheme opens the original video and the retouched video and extracts their Group of Picture (GoP) sequentially. When the frame rates per second (FPS) of the two videos are different, the comparison of each corresponding frame pair in two videos may result in an incorrect comparison result. So, the scheme compares only the first frame of extracted GoPs, called the I-frame, which is not affected by the different FPSs. The scheme calculates the maximum or average difference of the first frames in two extracted GoPs. If the calculated maximum or average difference is smaller than the given threshold values, the scheme extracts the next GoPs of the two videos and continues to compare the first frames of the extracted GoPs until there are no more GoPs in the two videos. If the numbers of GoPs in the two videos are different, and thus the two videos do not reach their ends simultaneously, the scheme returns the result “Not Same”. Only when the two videos reach their ends simultaneously with passes of comparing all GoPs, will scheme return the result “Same”.

Figure 6 shows the detailed structure of the integrity preservation trace mechanism. The function modules for “Video Integrity Analysis” and “Source Code Integrity Analysis” are equal to those in Figure 4. Added function modules are “Integrity Trace Support” and “Stored Integrity Analysis Search”. The proposed mechanism first checks the integrity of the input video file in the module “Video Integrity Analysis”. If the integrity of the file is not proved, the following comparison procedure is skipped. If the integrity is proved, the proposed mechanism compares the frames of an original video and the frames of its retouched video one by one sequentially in the module “Integrity Trace Support”. As well as the comparison result, the integrity analysis result of the original file and the hash value of the retouched video file are stored as a form of the single group variable into the blockchain system. In the module “Stored Integrity Analysis Search”, the mechanism searches for the comparison result of the retouched video and the integrity analysis result of its corresponding original video. This module calculates the hash value of the retouched video file and utilizes it as the index of the searching operation.

The core parts of source code are stored in their original forms, and the rest of the non-essential source parts are stored as a shortened form of their entire hash value, similar to the mechanism explained in Figure 3. The core parts are the signature comparison procedure and the search procedure of stored results, and they are constructed with a program language designed for the blockchain system, called smart contract.

4. Evaluation

For evaluation, we implement the proposed scheme upon a commercial public blockchain system. We choose the Klaytn platform [10], although the Ethereum platform [27] is the most popular commercial blockchain system, because the Klaytn platform is much faster and cheaper than the Ethereum platform. Instead of the main network of the Klaytn platform, we use its non-commercial test network, called Baobab, where its operation costs are free.

4.1. Implemented Software Tool

The proposed scheme is implemented into a software tool (The source codes of our implemented tool can be found at the github site, https://github.com/wanlee02/Klaytn_WhereCamFrom, accessed on 15 August 2022.). The software modules in the front-end server are implemented with the Javascript and HTML languages and managed by the Nodejs package. The software modules in the back-end blockchain system are implemented with the Solidity language, constructed in the form of smart contracts, and managed by the Truffle package. As mentioned in Section 3, the software codes in the blockchain systems are constructed as digital programming codes called smart contracts. Each smart contract has the maximum size constraint: 24 Kbytes on bytecode. So the software codes are divided into 20 modules and each module is implemented into a single smart contract. The 20 smart contract modules are connected with links to their addresses in the blockchain system. Five smart contracts store signature databases of camera devices and five smart contracts store signature databases of video editing tools. The remaining 10 smart contracts compare the extracted signature with two databases and searching the stored result with the transferred hash value.

Figure 7 shows the graphic user interface for the main function of the integrity analysis mechanism, explained in Section 3.1. It is the output display of Chrome browser that is operated by the software modules in the front-end server with an IP address of 201.121.139.90 and a port number of 8080. The button ”File Open” starts the uploading process of an input video file. The button “Analysis” starts the integrity analysis process of the input video file. The software modules in the front-end server just extracts the meta data of the input video file, excluding multimedia contents, and transmits it to the software modules in the back-end blockchain system. Then the software modules in the back-end blockchain system extracts the code characteristic from the transmitted metadata, compares it with the prepared signatures, and determines whether the integrity of the input video file is original, non-original or unknown. The integrity analysis result is displayed as Original, Edited or Unknown as shown in the bottom tree boxes of Figure 7, where Edited represents non-original video contents. The result Unknown occurs when the extracted codec characteristic is not found in the two signature databases, i.e., the signature databases are insufficiently prepared in advance. The decision of integrity analysis is transmitted to the software modules in the front-end server. The matched name of digital camera device or video editing tool is also transmitted. The module in the front-end server displays the transmitted decision as shown in Figure 7.

Figure 8 shows the graphic user interface for supplementary functions of the integrity analysis mechanism. The button “Analysis & Store” stores the integrity analysis result in the blockchain systems with a hash value of the input file. When storing the integrity analysis result is completed, the integrity analysis requests of the same input file can be replaced with the fast search operation, instead of extracting long metadata from the input file and comparing the metadata with prepared signatures. The button “Search” starts the search operation for the stored integrity analysis result with a 32-bytes hash value of the input file.

Figure 9 shows the graphic user interface for the function of integrity preservation trace between an original file and its retouched file, explained in Section 3.2. The button “File Open” provides the uploading process of the original video file and its retouched non-original video file. The button “Compare” starts the process of comparing each pair of corresponding frames between the original video and the non-original video one by one and sequentially. This function operates in the front-end server and takes a long time proportional to the running time of the two video files. To avoid the confusion caused by different frame rates per second between the original video and its retouched video, our software module compares only the first frame in each group of pictures (GoP) instead of comparing all frames in each group of pictures. The comparison result is displayed either Same or Not Same, as shown in the bottom of Figure 9. If the comparison result is Same, the software modules in the front-end server extract the meta data of the two files and transmit them to the software modules in the blockchain system. As explained in Figure 7, the software modules in the blockchain system verify the integrity of the two files with the transmitted meta data. The integrity analysis results of the two files are sent back to the front-end server and displayed on the Chrome browser, as shown in the left bottom box of Figure 9. If the comparison result is Not Same, the software modules in the front-end server provide the specific time when the two videos run differently for the first time, and depict their two corresponding frames, as shown in the bottom right box of Figure 9. Note, in the bottom right box of Figure 9, the right frame contains a hat image marked as a circle, whereas the left frame does not.

Figure 10 shows the procedure used to store the integrity preservation trace in the blockchain system. The button “Analysis” in the left box of Figure 10 starts the storing procedure, where the comparison result of the two files, the comparison time, their integrity analysis results and their hash values are transmitted from the front-end server to the back-end blockchain system. These values are stored as a group variable in the blockchain system. The right box in Figure 10 shows the result of blockchain operation transmitted from the back-end blockchain system to the front-end server.

Figure 11 shows the procedure to search the integrity preservation trace of retouched video files. The button “File Open” starts the uploading process of an input video file. The button “OK” starts the operation to search for the transforming history of the input files stored in the blockchain system. The software module in the front-end server calculates the hash value of the input file and transmits it to the software modules in the blockchain system. The software module in the blockchain system searches for the stored integrity preservation trace with the transmitted hash value as the index key. The search result is transmitted to the software module in the front-end server, and displayed as shown in the bottom box of Figure 11.

4.2. Experiment Results

To examine the performance of the implemented tool, we prepare two hundred original video files and their retouched video files. The two hundred original videos are generated using USB web cameras with Windows 10, Galaxy smartphones with Android and Apple smartphones with iOS. These two hundred video files are retouched with five video editing tools: Movavi Video Editor, VapMix, GomMix, FFmpeg and Pinnacle Studio. We first transform these two hundred original videos without any modification into their corresponding retouched videos, referred to as non-modified videos. Next, we transform these two hundred original videos with some modification into their corresponding retouched videos, referred to as non-modified videos. One hundred files among the two hundred original files are transformed with frame deletion, referred to as inter-frame forgery videos, and the other one hundred files are transformed with image modification within frames, referred to as intra-frame forgery videos. The position of the deleted frame and the location of modified images are selected randomly.

Table 1. Accuracy and execution time of integrity analysis.

Measure	Original Video	Non-Modified	Inter-Frame Forgery	Intra-Frame Forgery
Accuracy	100.0%	100.0%	100.0%	100.0%
Avg exec. time	221 ms	227 ms	228 ms	232 ms

shows the integrity analysis accuracy and the execution time of the proposed scheme. The proposed scheme detects the exact name of camera devices or the name of video editing tools in all cases. Based on the detected name, the scheme classifies the input videos into original videos and retouched videos. However, the scheme does not classify the group of retouched videos: non-manipulated, inter-frame forgery and intra-frame forgery. The execution time of the proposed scheme is less than one second in all cases, whereas the maximum execution time of the scheme is 443 ms. The average execution time for integrity analysis of original videos is 221 ms; for non-modified videos, it is 227; for inter-frame forgery videos, it is 228 ms; for intra-frame forgery videos, it is 232 ms.

Table 2. Comparison accuracy between original videos and retouched videos.

Allowed Difference Ratio	Non-Modified	Inter-Frame Forgery	Intra-Frame Forgery
0.0%	0.0%	0.0%	0.0%
0.025%	99.0%	99.0%	96.0%
0.05%	100.0%	100.0%	100.0%
0.10%	100.0%	100.0%	100.0%
0.20%	100.0%	100.0%	100.0%
0.30%	100.0%	100.0%	100.0 %
0.40%	100.0%	100.0%	100.0%
0.50%	100.0%	100.0%	99.0%
0.60%	100.0%	100.0%	98.0%

shows the comparison accuracy between the original video and the retouched video in the proposed scheme. In Table 2, “Allowed Difference Ratio” means the maximally allowed difference ratio of red, green and blue components in each pixel of the retouched video against those in each pixel of the original video. Even though there is no modification when transforming the original video into the retouched video with video editing tools, the red, green and blue component values in the original video file are slightly different from those in the retouched video file. This variance comes from the re-compression procedure of codecs in the video editing tool, which is different the compression procedure of original codecs in the video recording device. Therefore, if any difference between original video and retouched video is not allowed, the first frame of the original video is determined to not be the same as the first frame of its corresponding retouched video in all cases. As result, when “Allowed Difference Ratio” is 0.0%, the accuracy of comparing original video and retouched videos is 0.0% for all transformations of non-modified, inter-frame forgery and intra-frame forgery videos, as shown in Table 2.

When the “Allowed Difference Ratio” is 0.025%, the accuracy becomes close to 100.0% but not equal to 100.0%. Comparison between original video and retouched video is correct for all static scenes, where objects rarely move in subsequent frames. However, comparison becomes incorrect for some dynamic scenes, where objects moves very quickly in subsequent frames. When the “Allowed Difference Ratio” is set to 0.05%, the accuracy measures of all transforming operations are 100.0%. Until the “Allowed Difference Ratio” is set to 0.40%, the accuracy measures are 100.0%. It is worth mentioning that if the “Allowed Difference Ratio” is set too high, some intra-frame forgery may not be detected during the comparison. For example, when “Allowed Difference Ratio” is set to 0.50%, the proposed scheme determines one intra-frame forgery video as a non-manipulated retouched file among the 100 intra-frame forgery videos. Similarly, when the “Allowed Difference Ratio” is set to 0.60%, the scheme misses two intra-frame forgery videos.

Finally, we examine the processing speed of the proposed scheme against the video file size.

Table 3. Execution time against video size on the blockchain system.

Range of Video File Size (Mbytes)	Analyzing Integrity	Comparing Videos	Storing Result	Searching Result
10∼50	212 ms	19,233 ms	1744 ms	637 ms
50∼100	218 ms	38,568 ms	2824 ms	938 ms
100∼200	221 ms	71,201 ms	3424 ms	1180 ms
200∼500	223 ms	136,119 ms	4026 ms	1315 ms
500∼1024	226 ms	293,744 ms	5725 ms	1974 ms
1024∼1536	233 ms	586,983 ms	7332 ms	2876 ms
1536∼2048	234 ms	722,726 ms	9154 ms	3527 ms

shows the consumed execution time on the Samsung notebook with a i7-core processor with 16 Gbytes memory. The average execution time of videos against the range of file size is depicted. The execution time of analyzing the integrity of input videos is slightly proportional to the file size, but almost close to about 220 ms. The processing time of analyzing the integrity is rarely affected by the file size, because this operation examines only the file header fields, but not the multimedia contents of the video. This processing time is mostly dominated by the speed of the smart contract program in the blockchain system. The execution time of comparing video frames between original videos and retouched videos heavily depends on the file size, as shown in the column “Comparing” of Table 3. This execution time is determined by the hardware performance of the front-end server, because it is carried out on the front-end server. The execution time for storing the comparison result is manifestly proportional to the file size, where the storing operation requires the hash function to compress the input video file into a 32-bytes hash value. The processing speed of storing the comparison result per mega bytes is about 212 (ms)/(Mbytes). This execution time is also affected by the hardware performance of the front-end server. The execution time taken when the comparison result is also manifestly proportional to the file size, because the search operation also calculates the hash value of an input file. The processing speed of searching the comparison result per mega bytes is about 74 (ms)/(Mbytes). The maximum execution time of the search operation is about 3971 ms against the input file with 1.98 Gbytes. The search operation is completed within, at most, four seconds in this experiment. The procedure to obtain the comparison result and store it into the blockchain system is called once for each video, whereas the search operation may be called repeatedly for each video.

The operations of comparing videos and storing result are set for recording the transforming history of a retouched video, while the operation of analyzing integrity is separate and the operation of searching result is separate. The operation of analyzing integrity is completed within, at most, half of a second. The operation of the searching result is completed within, at most, five seconds. The operations of comparing videos and storing the result are completed within, at most, fifteen minutes.

The coin fees required for analyzing integrity and searching result are zero, because these operations do not require data to be permanently stored in the blockchain. On the contrary, the coin fee required for storing the result is about 0.018756 Klay, where the coin fee is the amount of consumed gas × fee per gas ($250\times {10}^{-9}$ Klay). This coin fee is required whenever storing the transforming history of a retouched video. The total coin fee required for distributing and storing our 20 smart contracts in the blockchain system is about 2.365831 Klay, This operation is only executed once.

4.3. Hardware and Blockchain Platform Dependency

The execution time of the proposed scheme depends on both the hardware specification of the front-end server and the platform of the back-end blockchain system. To check the effect of hardware specification of the front-end server, we replace the notebook with a low-performance IoT device, Raspberry Pi 3 with 1.2 GHz Quad Core and 1 Gbytes memory [33,34]. In the Raspberry Pi 3, the execution time of analyzing integrity is similar to that in Table 3. The operation required for analyzing integrity in the front-end servers is to just extract the file header fields, and thus it is rarely affected by the hardware performance of the front-end server. The execution time of comparing video frames in the Raspberry Pi 3 is increased by about 48%, compared to that in Table 3. This operation is heavily affected by the hardware performance of the front-end server. The execution times of comparing video frames and searching for the result in the Raspberry Pi 3 are similar to those in Table 3. These operations are rarely affected by the hardware performance of the front-end server, but are heavily affected by the platform performance of the blockchain system.

Next, we replace the blockchain platform with other public blockchain platforms such as Ethereum [27] and Tron [35] in order to check the effect of the blockchain platform. We use their non-commercial test networks instead of their commercial main networks: Ropsten network for Ethereum platform and Shasta network for Tron platform. We do not consider the private blockchain platform such as Hyperledger Fabric [36] and Quorum [37], because the hardware and network specifications of private blockchain platforms are not fixed or determined by the private platform operator. The private blockchain platform may be not suitable for the proposed scheme because the disclosure of smart contracts and stored data in the blockchain system requires the permission of the network operator in the private platform. In the Ethereum platform, the execution times of analyzing integrity and comparing videos are similar to those in Table 3. The execution time of the searching result is increased by about 740% compared to that in Table 3, and the operation to store results takes about 650 s. In the Tron platform, the execution times of analyzing integrity and comparing videos and searching result are also similar to those in Table 3. The execution times for storing the result and searching for the result are decreased by about 32% and 16%, respectively, compared to that in Table 3. Through these experiments, it is confirmed that the execution speed of comparing video frames depends on the hardware performance of the front server, and the execution speeds of the storing result and the searching result depend on the platform performance of the public blockchain system.

The proposed scheme is applicable to any public blockchain platform supporting two basic functions: storing data permanently and searching for the stored data. When the proposed scheme employs other public blockchain platforms instead of Klaytn, Ethereum and Tron, the metrics of storing time, searching time and the required blockchain operation fee are changed according to the employed blockchain platform. For example, when the proposed scheme employs IoT-oriented blockchain platforms implemented with Raspberry Pi devices [38,39,40], the storing time and searching time of the scheme may be increased due to their relatively low hardware and network performance, while the required operation fee depends on the operation fee policy of platform designer or operator. These IoT-oriented blockchain platforms provide many benefits, such as smaller size of generated blocks, lower energy consumption and lower implementation costs. However these benefits do not affect the performance of the proposed scheme. So comparative evaluation about other benefits and drawbacks of the employed blockchain platform, except storing time, searching time, and the required operation fee, is beyond the scope of this study.

5. Conclusions and Discussions

In this article, we propose an integrity preservation analysis scheme of video contents with the support of blockchain systems. To guarantee the reliability of the proposed scheme, core source codes of the scheme are stored in their original form in the public blockchain system immutably, so that any user can easily verify the core parts of the scheme. The rest of the codes are stored as a short form of their hash value in the blockchain system so that any user can check their modification time.

The scheme first examines the software characteristics of digital camera devices and video editing tools in advance, and stores them in the blockchain systems. Next, the scheme extracts the software characteristic from the target video file and compares it with the prepared characteristics in the blockchain system. With a matched characteristic in the blockchain system, the scheme determines the source camera device or the source video editing tool that generates the target video file. In addition, the proposed scheme stores the transformation history of retouched video contents into the blockchain system. The scheme compares the original video and its retouched video frame by frame, and stores the comparison result with a hash value of the retouched video in the blockchain system. Then, the transformation history of the non-original video file can be easily searched in the blockchain system, where the hash value of the video file is used as the index of the searching operation. We implement the proposed scheme into a practical tool on a commercial blockchain system. Experimental results show that the proposed scheme performs the integrity analysis of video files with 100% accuracy and provides the transformation history of retouched video files with 100% accuracy when a proper parameter is given. It is also shown that the proposed scheme completes the integrity analysis within at most one second, and the search operation for transformation history of retouched video files within at most four seconds.

Although this article deals with only the integrity preservation trace for non-modified transforming from original videos to retouched video, the proposed scheme can be extended to the integrity preservation for modified transforming. Examples of modified transforming with integrity preservation are adding the text subtitle to video frames for language translation and generating a short highlight video from a long video. Additionally, this article addresses only the comparison of image contents, but the proposed scheme is applicable to the comparison of audio contents. This study empirically determines the allowed difference ratio used for comparison between original and retouched videos. We leave the problem of systematically determining the allowed difference ratio for further study. We will also investigate the cost minimization of a blockchain system required to store the core parts in further study.