A short guide to tuning Comskip

By Erik.

Introduction

Before the tuning of Comskip can start it is essential to understand the basic operation of Comskip. Comskip reads and decodes a recorded mpeg file based on instructions Comskip reads from a so called ini file. The recording is analyzed on various characteristics such as black frames, silences and changes in aspect ratio.

Based on this information Comskip segments the recording in blocks and using heuristics, together with additional information such as the presence of logo, the scene change rate, Close Captioning information and other information sources Comskip tries to determine what blocks of the recording are to be characterized as commercials.

This process can only be successful under a number of conditions. The information to be used for the segmentation must be available in the recording. When, for example, the recording contains insufficient or too many black frames the segmentation process has a high chance to fail. It is therefore essential to instruct Comskip how to recognize the various segmentation information sources to ensure an optimal result.

However, Comskip may not be able to deliver acceptable results on certain recordings from certain broadcasters because the required information is simply not available in the broadcast. The purpose of the tuning is therefore to extract the maximum of information from the recording to enable Comskip to do the best possible job.

General strategies

A simple strategy is to install Comskip and use the comskip.ini file available in the download in the PVR system and let the PVR do its recording. Afterwards, when watching the recording enhanced by the information from Comskip the results can be judged and if not as required some changes to the ini file can be made. The impact of these changes can be judged after the next recording is done.

This strategy works if only limited adaptations are required. It is not the right approach to get the best results for specific broadcasting styles.

It is therefore recommended to use a more elaborate strategy to optimize Comskip.

Setting up the test environment.

Create a separate folder and copy the files from the downloaded Comskip zip file into this directory.

Then copy a number of relevant recordings into this directory and rename them to appropriate short names.

There should now be the following files in the folder. Comskip.exe, comskip.ini, comskip.dictionary and a number of mpeg files. Lets name them for this guide as short.mpg, long.mpg, etc....

One mpeg files should preferably not be very big, about 10 minutes, and contain a mix of show and commercial.

This we will call short.mpg. A second mpeg should be a full recording of about one hour containing a typical sequence of show segments mixed with commercials. This will be called long.mpg.

Now edit the comskip.ini file. Comskip is rather picky on how you format the ini file. No spaces before of after the '=' are allowed. Multiple occurrences of any setting can exist anywhere in the file but only the first will be used. There is no way to 'comment out' a setting.

The following settings should be present when starting the tuning, each on a separate line:

verbose=10

detect_method=47

delete_logo_file=0

intelligent_brightness=0

output_framearray=1

output_debugwindow=1 ; Only for tuning, to be replaced later

non_uniformity=1000 ; Only for tuning, to be replaced later

max_volume=1000 ; Only for tuning, to be replaced later

Now we need to run Comskip on the first mpg file. For the first part of the tuning we will use short.mpg. To run Comskip on the mpeg we create a shortcut icon on the desktop to Comskip.exe. Drag and drop short.mpg on the Comskip icon and watch the processing being done.

First a console window opens and some initial logging appears. Then the debug window opens. To understand the content of the debugwindow please read debugwindow description file.

The frame number in the top left corner should start at 1 and increment with jumps till the last frame number.

Once Comskip reaches the final frame there will be some more logging in the console window and the debugwindow will jump to the first frame.

Comskip can now be terminated by closing the Comskip console window.

The test folder will contain a number of new files, each with its specific content.

The file short.log contains the logging output of the last comskip run.

The file short.txt contains the start end ending frame numbers of the identified commercials, if any.

The file short.csv contains information on brightness, volume, etc... one line per frame. The first line contains the descriptive name of the information. The last figure on the first line is the frame rate times hundred as read from the mpeg file.

The file short.logo.txt contains the logo recognition mask if a logo has been found.

The log file

Now use your favorite editor or text file viewer and open the log file short.log.

The first part of the log file lists some essentials of the comskip run. Then there is information on the frame per frame scanning starting with the line

First Frame

After all frames has been scanned there is a line

Finished scanning file. Starting to build Commercial List.

After this line the global processing with closing frame groups and cleaning up information.

Then the recording is segmented in blocks starting with a line such as

Creating block 0 From 1 (1) to 350 (348) because of Blackframe with 0 head and 2 tail

Then some additional small cleaning up and information generation and the Fuzzy Logic scoring starts with

Fuzzy scoring of the blocks

After the scoring there is the information on the detected commercials starting with

Initial Commercial List

And finally the list of blocks with their details starting with

Block list after weighing

Starting the tuning: the black frame detection.

Five parameters determine the black frame detection process. They are

max_avg_brightness=20

test_brightness=40

max_brightness=60

non_uniformity=1000

max_volume=1000

intelligent_brightness=0

The first parameter to look at is max_avg_brightness, in general test_brightness should be 20 above max_avg_brightness and max_brightness should be again 20 above test_brightness.

A frame is a candidate black frame when the average of the brightness of its pixels if below max_avg_brightness. It is therefore required to identify the black level in the recording and set the max_ avg_brightness somewhat above this level. To do this look in the log file at the list of black frames just below

Black Frame List

There should be lines such as

14 16776 16 35 77

Where 16776 is the frame number, 16 the brightness, 35 the uniformity and 77 the volume. If there are no black frames the max_avg_brightness should be increased with steps of 5, together with test_brightness and max_brightness. Rerun Comskip on short.mpg after each increase.

Look at the minimum brightness in the black frame list that should occur a number of times, and set max_avg_brightness to that level plus five. Rerun Comskip to verify the correct setting. Use the debugwindow to verify the correctness. Use the ‘n’ key to jump to each next black frame and use the cursor keys to scan to the right to verify if a number of the frames behind the current position are indeed black.

Be sure to set test_brightness and max_brightness to respective 20 and 40 above max_avg_brightness.

When one single pixel in a frame has brightness above max_brightness then that frame will not be regarded as black. When more then 5 pixels on a line have a brightness above test_brightness the frame will not be regarded as black.

The detection of the black bars used for the aspectratio calculation uses the same settings so when short.mpg contains parts with different aspectratios is easy to see in the debugwindow if the processing is correct.

Sometimes there is something inside the recording that needs to be ignored. One possibility is to ignore a border around the picture. This is controlled by

border=10

The default border is 10.

Another option is to ignore only a part at the bottom of the video. This is useful when the broadcast contains tickertape at the bottom. The controlling parameter specifies the number of pixels at the bottom to ignore as:

ticker_tape=120

The default is 0, the setting can be checked in the debugwindow because the part of the video that will be cancelled out is shown as black.

Tuning the max_volume.

Black frames with a volume above max_volume will be removed from the black frame list before the final black frame list is shown. This is shown in the log file with lines such as

391 - Removing black frame 41319, from black frame list because volume 1655 is more than 600, brightness 16, uniform 1

Volume levels of –1 signal that Comskip was unable to match the mpeg sound decoding with the picture decoding for that specific frame so these volumes should be ignored.

Key is to identify the volume level that is regarded as silence. This is never at zero but could be anywhere between 10 and 500 depending on the broadcast

Some sequences of subsequent black frames in the black frame list possibly show the volume going down, remain for some frames at a low level and then again going up.

Below you can find a typical sequence from the black frame list

1076 165202 16 148 178

1077 165203 16 141 104

1078 165204 16 148 97

1079 165205 16 110 90

1080 165206 16 105 90

1081 165207 16 95 90

1082 165208 16 78 90

1083 165209 16 72 90

1084 165210 16 38 90

1085 165211 16 48 90

1086 165212 16 4 78

1087 165213 16 0 78

1088 165214 15 0 75

1089 165215 15 0 75

1090 165216 16 0 75

1091 165217 15 0 75

After frame 165214 the final black level is reached with brightness of 15, uniform of 0 and volume of 75.

The lowest volume level that reoccurs on more then one frame sequence is the silence level. The max_volume level should be set at about four times the silence level. There may be black frame sequences that do not show the low volume level of silence but remain at substantial higher volume levels. These sequences should not be regarded as having silence and the max_volume level can be set below these levels to effectively remove them from the black frame list.

Below you will find an example from the black frame list where the frames are perfectly black (brightness is 16 and uniformity is below 5) but the sound level is too high to be silence.

193 82467 16 3 537

194 82468 16 1 537

Acceptable max_volume levels could be in the range of 250 up to 1000.

Again the debugwindow can be used to visually verify the alignment of the black frames and the silences. Use the ‘a’ key to jump to the next block boundary and use the cursor key to verify if the frames behind the current position indeed do have a black frame with silence.

Setting the non_uniformity

Black frames with low brightness and silence do normally also have a low level of non_uniformity, possibly as low as zero. The non_uniformity level is used to recognize frames with uniform brightness, as these could also be block separators. To set the non_uniformity level look again at the black frame list just like what was done for the max_volume but now for the uniformity.

Acceptable levels for non_uniformity could be in the range of 250 up to 1000. However if you record on low quality or have a very high noise level you may have to go up to 2000.

Verifying the settings.

Start Comskip with the adjusted setting on long.mpg and manually verify the black frame detection by using the 'n' and the 'p' keys to step through the detected black frames using the debugwindow.

When no black frame is detected between certain segments use the arrow keys to step frame by frame through the transition. Look at the detected values behind the frame number in the upper left corner and verify that processing done as expected.

Using the short.csv file

For those that have access to spreadsheet software such as excel there is also another way to find the right level.

After Comskip has been run on short.mpg there appears a short.csv file in the test folder. Now open this file with your spreadsheet software.

The first part of the spreadsheet looks like this

frame	brightness	scene_change	logo	uniform	sound	minY	MaxY	ar_ratio	goodEdge	isblack	2500
1	0	0	1	0	-1	10	566	130	495	0
2	64	0	1	27177	-1	10	566	130	495	0
3	64	480	1	27259	1618	10	566	130	495	0
4	64	475	1	27327	1482	10	566	130	495	0
5	64	475	1	27310	2759	10	566	130	495	0
6	64	475	1	27285	1026	10	566	130	495	0
7	64	480	1	27353	2291	10	566	130	495	0
8	64	475	1	27365	2771	10	566	130	495	0

Now select the cell just below brightness and sort the sheet in ascending order on brightness (in excel click the a/z button on the toolbar or select data/sort from the menu). The frames with the lowest brightness will appear at the top of the list and it is easy to see what the black level is. Do the same for the volume (below the word sound) to find the silence level.

To facilitate this process one can download a large Comskip.xls files to be used as a template for more detailed analysis. Copy the short.csv from the open spreadsheet tab (select the whole sheet and then Copy) and Paste it on the input_data sheet (select cell A1 and Paste). Now it is possible to use the autofilter on the input_data sheet and to look at the two graphs on the other sheets.

Using aspect ratio based cutting

Comskip can, next to black frame based cut points, also generate cutpoints when the aspect ratio changes. This optional feature is controlled by

cut_on_ar_change=1

If this is set to 1 Comskip will segment the recording also on changes on aspect ratio. Whether this is a good thing largely depends on broadcast style and personal preferences. Cutting on aspect ratio changes is automatically disabled when insufficient logo is found and will only happen in segments where no logo is found.

Tuning the Logo detection

The presence of logo on a certain segment of the recording guides the heuristics towards interpreting that segment as show. It is therefore important to have good logo detection.

Logo detection is a very complicated process with a large number of tunable parameters. It is therefore advised not to modify most of these parameters.

Only when there are multiple slightly different logo’s in the broadcast and Comskip fails to see the difference between them it is advised to set

aggressive_logo_rejection=1

Some shows have multiple logos. When all logos are at the top or at the bottom of the screen there is no problem. However, when the logo's are on diagonal sides of the screen Comskip will not accept the logo as valid. A possible way to solve this is to set

subtitles=1

This will instruct Comskip not to search for logos at the bottom of the screen.

When Comskip is started and it cannot find a logo file such as short.logo.txt it will start to scan for a logo after 1000 frames. You can change this by setting give_up_logo_search to another value such as

give_up_logo_search=2000

In the debugwindow you can see the appearance of pixels that are candidate for the logo mask. The maximum fraction of the screen size that Comskip accepts as a valid logo is defined by:

logo_max_percentage_of_screen =0.1

This value should be acceptable in most cases but can be increase to 0.35 if needed.

Once the logo has been identified Comskip saves the learned logo edge pixels and start again from the beginning. This can be disabled for slightly faster processing but this is not recommended.

When Comskip has learned the logo it will track the presence of logo as shown in the logo rectangle and in the black bar in the Frame bar at the bottom of the debugwindow. The fraction of edge pixels that should match the mask is defined by

logo_threshold=0.70

When too many false positives are detected this value can be increase in steps of 0.05 till 0.95.

When one logo block was found and after that a long time of no logo, Comskip assumes it has learned a wrong logo and restarts its learning from well within the recording. When after this a logo is found Comskip will restart for a second time.

The following setting controls the deletion of the logo file

delete_logo_file=0

When set to 1 the learned logo file will be deleted after processing is finished.

When no logo is found after about 12 minutes in the show Comskip will give up logo detection and assumes no logo at all.

The amount of logo over time will tell Comskip how to interpret the presence of logo. If more then 50% logo is found Comskip assumes logo to be present on show only. If less then 32% logo is found but more then 15% Comskip assumes logo to be present on commercials only and the logo logic is reversed.

If the amount of logo detected is above a certain threshold defined by

logo_percentile=0.92

it is assumed there is always logo and the logo heuristics will also be disabled.

When the logo detection is assumed to be robust it can be used to suppress false black frames from being detected. This is enabled with the following setting

connect_blocks_with_logo=1

This will connect two blocks into one block if there is logo present at the end of the first block and at the beginning of the second block. An example of a line signaling the deletion of a block due to this connection process is

Joining blocks 20 and 21 at frame 149986 because they both have a logo.

It is also possible to create additional pseudo blackframes some seconds after a logo disappears or before a logo appears The following line in the ini file will create a pseudo blackframe 6 seconds after the logo disappears and 8 seconds before the logo appears.

after_logo=6

before_logo=8

Sometimes the logo continues too long and only disappears during the start of the commercial. This can be solved by shrinking the duration of the logo by using

shrink_logo=10

The effect on the processing is that the logo is assumed to appear 10 seconds later than reality and to disappear 10 seconds before reality. As long as there are no black frames just before the end of the show there is no need to be very precise on this setting

Verifying the Block list

Using all the information gathered from the recoding Comskip will generate a so call Block List. This block list is directly after the black frame list.

An example of a block list:

Creating block 0 From 1 (1) to 2074 (2071) because of Blackframe with 0 head and 3 tail

Creating block 1 From 2075 (2078) to 2339 (2339) because of Blackframe with 3 head and 0 tail

Creating block 2 From 2340 (2341) to 2509 (2508) because of Blackframe with 1 head and 1 tail

Creating block 3 From 2510 (2511) to 2830 (2827) because of Blackframe with 1 head and 3 tail

Creating block 4 From 2831 (2834) to 3651 (3650) because of Aspectratio with 3 head and 1 tail

Creating block 5 From 3652 (3653) to 5279 (5276) because of Blackframe with 1 head and 3 tail

Creating block 6 From 5280 (5283) to 5813 (5810) because of Blackframe with 3 head and 3 tail

Creating block 7 From 5814 (5817) to 6266 (6265) because of Aspectratio with 3 head and 1 tail

Creating block 8 From 6267 (6268) to 6890 (6889) because of Aspectratio with 1 head and 1 tail

Creating block 9 From 6891 (6892) to 10306 (10306) because of Blackframe with 1 head and 0 tail

The reason for segmentation to the next block is indicated as Blackframe or Aspectratio

After the logo processing certain blocks may have been connected as described above leading to a reduced block list. The final block list can be found at the very end of the log file.

Block list after weighing

----------------------------------------------------

# sbf bs be fs fe sc len scc scr cmbnd ar cut bri logo stdev

0)+ 3 0 3 1 2074 0.18 82.920 0 0.00 0 1.30 b 61 0.83 120

1)- 3 3 0 2075 2339 1.78 10.560 0 0.00 2 1.30 b 101 0.00 98

2)- 2 1 1 2340 2509 1.78 6.760 0 0.00 2 1.30 b 86 0.00 108

3)- 4 1 3