PROJECT INFORMATION
Project Title: RED Camera Video Format Extraction Utility
Project Start Date: 2016/08
Project Completion Date or Expected Completion Date: 2016/10
Field of Science or Technology: Software Engineering and Technology
Purpose of work
To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.
Experimental Development:
Overcoming Uncertainties, Work Performed, Achievements, and Outcome
Background:
The RED video format is a proprietary RAW format that is used in high quality, professional video production. A client submitted a RAID array that hosted the storage for their video production files. The files had apparently disappeared and at some point, the file system was reformatted and partially overwritten. Metadata and other structures were overwritten therefore the only solution was a raw file recovery.
The format is undocumented therefore we would have to examine and reverse engineer the RED format by utilizing video files provided by the client, using the same equipment. Due to the proprietary nature and hence lack of technical support, we would have to break down the video header and other structures without any assistance or technical documentation.
Goal:
We had the client supply us with a number of RED video files that were produced using the same camera as that of the files to be recovered.
After deciphering the higher-level file structures, we wrote an initial scanning program used to log the location of all RED video file headers on the volume.
The first task was to extract all fragments utilizing the file start locations written to the scan logs. With the assistance of the client, we viewed the extracted fragments to determine which were complete and those that were truncated (due to interleaving of other file data within the file system).
With feedback from the client, we were informed that many fragments had flicker when played using their video editor and many were also much shorter in time in comparison the actual physical size of the video file.
After examining these files further using an openly available RED video viewer, we were able to determine the exact frames that were causing the flicker. In conjunction with the raw video data, we used this information in order to find the approximate location of the corruption within the file. This information then allowed us to accurately locate stream and frame headers, and then further break these structures down. One of the most important data values we were able to extract from the structures was the video frame timestamp and frame size. Using the timestamp and size, we could locate and identify following frames and determine if they were interrupted by other file data or overwrite.
We broke down the frame metadata further and found that there was unique data for each video file that was useful in segregating the hundreds of individual video files created during production. Our scan utility was enhanced to scan the entire volume containing the deleted RED file fragments and find the locations of continuous video frames utilizing the new information we discovered in the frame metadata.
After gathering the locations of all video fragments, we were able to piece together the majority of the lost files. When verifying the recovered files in the RED viewer, we found that some video files would not load and would hang the viewer during the frame indexing operation. After closer examination, we found that the problematic files had missing frames, likely due to overwrite of some frames. We experimented with good files by adding invalid data to the files as well as removing frames. We found that the viewer could tolerate extra invalid data and was able to resync the video, however a missing frame would cause severe issues with playback.
Since we needed to provide working files to our client, we had to devise a method to patch files that contained missing frames. We experimented by adding frames with a recreated frame header and then padded the data portion with zeroes. The solution now allowed the video to be played, so we went further by copying the preceding frame in to the missing frame (with a corrected frame header) and found the video played flawlessly without flicker.
We are successfully able to recover RED video format files which have been deleted or lost from a working file system. We have obtained enough knowledge regarding the internal RED format structures in order to create other enhanced or custom recovery tools depending on a client’s recovery situation.
