HC Predictor - any interest ?

[Darksoul71]

Hi all,

I'm currently in the last "alpha test phase" for my new baby called HC Predictor.
As you might already guess by the name it's a OPV prediction frontend for HCEnc.

My tests with DVD-RB free and normal generated HC.ini files were very promising.
HCEnc sometimes behaves not as predictable as TMPEGEnc, AutomatQEnc or CCE
in regard to target size but I have added some workarounds to meet the target size.

My question to you:
Do you have interest in HC Predictor ?
I'm asking this since DIKO already provides OPV prediction for HCEnc as well as
multicore support via HCEnc^n.

For the average DIKO user HC Predictor provides no true added value. For all other
people using HCEnc together with other tools such as DVD-RB or may be even HC Enc
manually I think HC Predictor provides a great functionality since you do not have
to mess around with any prediction stuff. HC Predictor uses the average bitrate and
movie length to calculate the target size, does the CQ prediction and the final run.

For the user the only difference is that you call hc predictor instead of HCEnc.

Let me know what you think.

Cheers,
D$

P.S.: HCPredictor is compatible with DVD-RB Free by simply renaming the HCPredictor to HCBatch. Then you must of course copy Rejig.exe and HCPredictor.ini to the HCEnc encoder directory of DVD-RB.

P.P.S.: I tested intensively with batchfile (simple commandline calls with ini) and DVD-RB Free

P.P.P.S: For all you AutoIt nerds I´ve included my dsHCEnc.au3 UDF which provides a huge set of functions to work with HCEnc ini files and HCEnc CLI.

[vmesquita]

I would be interested in the methods you would use to maybe improve DIKO prediction engine. I have noticed that DIKO prediction is good with CCE, but with HC it generally oversized

[Mrmscorp]

I would be interested in the methods you would use to maybe improve DIKO prediction engine. I have noticed that DIKO prediction is good with CCE, but with HC it generally oversized

Me too

I have this problem in HC...

Thanks

[Darksoul71]

Good morning, Mrmscorp + VM

I would be interested in the methods you would use to maybe improve DIKO prediction engine. I have noticed that DIKO prediction is good with CCE, but with HC it generally oversized

OK, 1st it is important that you understand that the prediction engine of DIKO works fine.
No matter if you use bisection, Newton Raphson or some else mathematical approach to calculate the CQ value matching your target size.
The calculated CQ value is "right" !

I´m currently using Bisection in HC Predictor:
http://en.wikipedia.org/wiki/Bisection_method

The size deviation is caused by HCEnc itself. HCEnc behaves different for the complete movie than it does for the samples.

Hank himself posted his results in an old discussion over at doom9:
http://forum.doom9.org/showpost.php?p=815244&postcount=637

The important part of this posting:

A Constant Quantizer as used by HCenc is decribed in the ISO document, Q-value and quality-based are "invented" by CCE and TMPEG, I think there's nobody who really knows how it works.

For further details you should follow the full discussion thread:
http://forum.doom9.org/showthread.php?t=93211&page=31

Start reading somewhere at post #610

I´ve written down my observations in the following posts:
http://forum.doom9.org/showpost.php?p=815008&postcount=629
http://forum.doom9.org/showpost.php?p=815039&postcount=631
http://forum.doom9.org/showpost.php?p=815425&postcount=639
 
Now back on the real topic: Why is HCEnc not as predictable in CQ mode as other encoders ?

We´ve seen good working "OPV" prediction with those encoders:
* CCE (pretty much any flavour) with CCEFront, D2SRoba, DIKO, etc
* TMPEGEnc with TOK and other frontends
* AQE with the same frontends as CCE
* NuEnc

The main thing that all those encoders have in common:
They do not really use a constant quantizer mode but a internal calculated value which is translated to a quantizer mode somehow.

In TMPEGEnc it´s a Quality value ranging from 0% to 100%.
So does NuEnc.

CCE and AQE are using something called Q-value.

All those four encoders are very predictable even with 1% samplesize. The are using an unknown function to translate their Q/Quality value to the quantizer values.
I do not know wether this is only done one time (e.g. 70% quality = 5.6 quantizer value) or if the encoder(s) are using their Q/Quality value during the complete encoding
process.

OK, what now follows is wild speculation

I´m neither an encoder developer with intense MPEG2 knowledge, nor do I have superior math knowledge

The function used by TMPEGEnc, AQE or CCE seems to flatten / dampen the wild size deviations we see with HCEnc in CQ mode.

To my understanding the only "true solution" for getting the same size prediction in HCEnc would be adding a Q/quality mode to it.
Sapstar was able to emulate CCE behaviour with AQE by writing a function which translates the Q value given to a CQ value which
can be use with AVCodec lib.

So hank could get the function from Sapstar.

If the calculation is only done at first run, I could integrate it in HC Predictor and work internally with a Q/Quality value instead of CQ.
But I would need the AQE sources and Sapstars help for this.

Currently all this boils down to the following approaches which I use (will use) to fight oversizing:
* Use a safe area for your file:
e.g. 5% safe area means you are only using 95% of your target size for prediction which leaves a few percent buffer.
For one movie per one DVD I think 10-15% safe area are acceptable. For kSVCD/bSVCD and one movie per CD the things are looking different though.
RB Optimizer also provides this feature. Of course one could also simply use a standard value to be substracted from the calculated CQ value.
This would be a similar approach.

* Use Rejig / Requant to shrink the final encoded file:
IMO you´ll not notice a big drop in quality if you shrink a file by 5-15% using a compressed domain trancoder (similar to DVDShrink) when the file is
oversize. In many cases HCEnc will only oversize by 10-15%. This method could be a nice work around. Tylo used this also with D2SRoba IIRC.
HC Predictor for example uses Rejig for shrinking. You can completely disable this feature and define from which oversize percentage Rejig will
be used, e.g. only use Rejig if the movie is oversized > 10%

* Using different / higher sample sizes:
I haven´t done many experiments with the sample size used for prediction but in general HCEnc seems to require
higher sample sizes than CCE/TMPEGEnc. This means rather to use 5-10% sample size compared to 1-2% for CCE.

* Use Ping / Pong approach for prediction:
I´ll think that I´ll also try out Inc´s approach for CQ prediction with his Slicer.avs function.

* Use filtering / limit maximum bitrate:
My experiments have shown that HCEnc will oversize more likely when the source material has lots of details.
Less details = more predictable.
My analogue captures had a higher tendency to oversize whereas DVD sources were more precise.
The same should be true for precompressed material (-> DivX2DVD conversion, DVB material).
So using prefiltering which softens the image and removes details also helps (e.g. kDVD motion adaptive filtering).
Limiting the upper bitrate will also help. All this indicates that the CQ curve simply needs to be "dampened".
I hope you get what I mean.

Unless hank add´s some quality-based mode to HCEnc (which I´ll ask him) I think that the best approach is a combination of the methods above.
Example:
5% safe area plus kDVD filtering and 5% sample size.

Not perfect but should be a workable approach.

I´ve asked hank to add an option to write the DBS file during CQ encodes. This would enable us to do a final sizing pass (second pass of two pass mode) if the CQ encode is way off. Tylo did the same thing for CCEFront.
If hank adds this I´ll support it in HC Predictor.

Once HC Predictor is fully configurable via Ini-File I´ll write some AutoIt script for automated testing and analysis.
Then I´ll try out the effectiveness of the different methods on different material (e.g. analogue captures, DVB-T captures, DVD sources, DivX sources).

Sorry for the lenghty text ! 

Have a nice sunday,
D$

[Darksoul71]

Oh yes, another approach would be of course what Hank and manolito have been using:
* Run a CQ prediction to meet the target size
* break movie into N segments (similar to HCEnc^n)
* Run sequentially through each segment and adjust your calculated CQ value
-> if the last segment oversized, increase the CQ value for the next segment
-> if the last segment undersized, lower the CQ value for the next segment

Have a look at manolitos experiments here:
http://forum.doom9.org/showthread.php?t=138566

The only difference I see between Hank´s and manolito´s approach:
To use always the same starting CQ (which is no good I think) versus calculating
a very well matching CQ value for your target size (which makes more sense I think).

[Darksoul71]

OK folks,

let´s see if I can work out a reasonable approach of manolito´s explanations:
http://forum.doom9.org/showpost.php?p=1166001&postcount=24
http://forum.doom9.org/showpost.php?p=1166032&postcount=26

For me this boils down to this:
* Do a quick CQ prediction with 1% samplesize and a maximum of 5 runs
* Split the movie in "n" segment (identical to HCEnc^n)
* Sequentially encode all segments and adjust the CQ value based on the size of the previous segment
* Merge all segments to final M2V file (identical to HCEnc^n)

Using the same interface as HCPredictor would provide easy usage with DIKO also.

Of course we loose the benefit of running multiple parallel processes but HCEnc has a nice
MT implementation anyway.

I would enable the user to define the "clamp" (as manolito calls it) around the CQ value.
You could finetune how much the CQ value can be adjusted.

How does this sound to you ?

May be hank will add something like this to HCEnc ?

Cheers,
D$

[Mrmscorp]

OK folks,

For me this boils down to this:
* Do a quick CQ prediction with 1% samplesize and a maximum of 5 runs
* Split the movie in "n" segment (identical to HCEnc^n)

Very interesting, because in DIKO for me, many times the prediction, have 50 a 98 runs....

* Sequentially encode all segments and adjust the CQ value based on the size of the previous segment

* Merge all segments to final M2V file (identical to HCEnc^n)

Using the same interface as HCPredictor would provide easy usage with DIKO also.

Of course we loose the benefit of running multiple parallel processes but HCEnc has a nice
MT implementation anyway.

Sounds very good, the usage in DIKO for me is very important because i like this tool for convert BDVD (my litle baby loves watch cartoons)

Thanks...

Sorry my english, a long time i dont read or write en english at least 5 years, my last vacation.....

[vmesquita]

Hi Darksoul71,

Man that was a huge Quality mode analisis! Most likelly one of the most complete I ever read. Thanks! 

A Constant Quantizer as used by HCenc is decribed in the ISO document, Q-value and quality-based are "invented" by CCE and TMPEG, I think there's nobody who really knows how it works.

This is very interesting, I had no idea.

To my understanding the only "true solution" for getting the same size prediction in HCEnc would be adding a Q/quality mode to it.
Sapstar was able to emulate CCE behaviour with AQE by writing a function which translates the Q value given to a CQ value which
can be use with AVCodec lib.

Agreed.

If the calculation is only done at first run, I could integrate it in HC Predictor and work internally with a Q/Quality value instead of CQ.
But I would need the AQE sources and Sapstars help for this.

Exactly. But unfortunatelly SAPSTAR is a little disapeared right now

* Use a safe area for your file:
e.g. 5% safe area means you are only using 95% of your target size for prediction which leaves a few percent buffer.
For one movie per one DVD I think 10-15% safe area are acceptable. For kSVCD/bSVCD and one movie per CD the things are looking different though.
RB Optimizer also provides this feature. Of course one could also simply use a standard value to be substracted from the calculated CQ value.
This would be a similar approach.

This is a simple approache but now very optimal

* Use Rejig / Requant to shrink the final encoded file:
IMO you´ll not notice a big drop in quality if you shrink a file by 5-15% using a compressed domain trancoder (similar to DVDShrink) when the file is
oversize. In many cases HCEnc will only oversize by 10-15%. This method could be a nice work around. Tylo used this also with D2SRoba IIRC.
HC Predictor for example uses Rejig for shrinking. You can completely disable this feature and define from which oversize percentage Rejig will
be used, e.g. only use Rejig if the movie is oversized > 10%

I sometimes do this just because the file has oversize and I am not willing to encode again, but actually it's not very good. I once did a test when people at KVCD.NET was thrilled with the possibility of using this in every encode: a 3% undersized file looks better than a 3% oversized shirinked to perfect fit.

* Using different / higher sample sizes:
I haven´t done many experiments with the sample size used for prediction but in general HCEnc seems to require
higher sample sizes than CCE/TMPEGEnc. This means rather to use 5-10% sample size compared to 1-2% for CCE.

DIKO does this. After finding the best sample with 1% samples, it fine tunes with two 5% samples, to find an approximate linear variantion around the QFactor area and fine tune. But it doesn't seem to help with HC.

* Use Ping / Pong approach for prediction:
I´ll think that I´ll also try out Inc´s approach for CQ prediction with his Slicer.avs function.

Yes this is other idea, I remember Inc explaining this method but I haven't played with it in a long while.

For me this boils down to this:
* Do a quick CQ prediction with 1% samplesize and a maximum of 5 runs
* Split the movie in "n" segment (identical to HCEnc^n)
* Sequentially encode all segments and adjust the CQ value based on the size of the previous segment
* Merge all segments to final M2V file (identical to HCEnc^n)

This is the approach I was thinking when I read your post. The problem is that it work fine with movies with about the same compressibility along the movie. Movies that start slow and becomes more "action packed" throught the end (for instance) would have strange results, since the deviation was already contained in the QFactor selected.

[Darksoul71]

Hi there,

Very interesting, because in DIKO for me, many times the prediction,
have 50 a 98 runs....

This sounds to me as if VM should add something like "MaxRuns" to his prediction loop

OK, may be I should explain a bit better:
Pretty much any CQ prediction boils down to a root problem.

So you can use the typical methods to calculate the CQ:
- Newton Raphson
- Bisection

Event the simplified formula which many tools use shows this:

Q(n+1) = Q(n) * (Estimated size / target size)

You usually loop until Q(n+1) = Q(n)

Normally this converges quite fast (in general 3 runs) but there
are rare cases where this prediction formula gets wild. It simply
doesn´t converge.

This is why a limitation of predictions runs and a smart
stop criteria prevents you from say 100 runs.

The bisection method on the other hand takes something like
5-6 runs to hit the target CQ value but will always converge
since the estimated size only indirectly influeces the calculation
of the next CQ value.

This for example are my current end criterias for HCPRedictor:

Code:

While (Abs ($CQ-$CQ_Alt) > 0.5) And (Abs($DeltaPercent) > 2)

I run the prediction until CQ[n+1] and CQ[n] have no bigger
difference than 0.5 and the percentual size difference between
estimated size and target size is no larger than 2%.

This works fine for me and my bisection method.

In order to use a MaxRun parameter, DIKO could do
the same thing I do:
Save the CQ value and size deviation in % in an array.
Once you´ve reached the maxrun and still haven´t found
a valid CQ you can simply run through this array and
return the CQ value with the lowest size deviation.
Not perfect but works quite nicely.

Another direction I´m currently thinking about:
AQE is using an internal function to translate
the Q-value to quantizer values (which libavcodec
uses IIRC). This is done to mimic the behaviour
of CCE. From my knowledge this works very well since
AQE converges as fast and as exact as CCE.

May be the right approach would be using a Q-value
for size prediction and transform it to a quantizer
using AQE formula.

I wanted to do some experiments anyway to find out
if the size deviations in HCEnc for certain CQ values
allow the development of some sort of correction
function.

Example:
CQ prediction tells me that CQ =2.4 meets the exact
target size but I also know that CQ=2.4 usually is
oversize by 30%. Something like that....

I can tell more if I did some more investigations
on different sources (e.g. DVB, DVD, DivX or
analogue captures).

@Mrmscorp:
No problem ! I can understand perfectly what you
are writing !

Cheers,
D$

[Darksoul71]

Hi Vinicius,

second reply for now since I wrote my last posting yesterday in the train on my way back home.

Man that was a huge Quality mode analisis! Most likelly one of the most complete I ever read. Thanks!

You are welcome ! This was of course only what I was thinking how it works.

Exactly. But unfortunatelly SAPSTAR is a little disapeared right now

I wrote SAPSTAR a PM. I dunno if he has enabled e-mail notification. May be we get some reply soon ?

This is a simple approache but now very optimal

You all will be able to test and judge by yourself very soon. HCPredictor is basically "finished enough" but I still do some changes during my current testing. It is already compatible with HCEnc^n and uses the -cores=1 for prediction as DIKO does

Yes this is other idea, I remember Inc explaining this method but I haven't played with it in a long while.

Hm, the only thing I´m not shure about if this will really help to fight the HCEnc oversizing.

I sometimes do this just because the file has oversize and I am not willing to encode again, but actually it's not very good. I once did a test when people at KVCD.NET was thrilled with the possibility of using this in every encode: a 3% undersized file looks better than a 3% oversized shirinked to perfect fit.

Hm, let me put it this way:
I often use DVDShrink myself when the resulting movies are a bit too big (e.g. 1-5%). I also often use DVDShrink for backup purposes when I only have to compress down to 70-80%. I never saw too big image degration. To me there is a big difference between "not very good" and "i know it´s gonna cost my some quality and live with it".
HCPredictor has an ini-setting to disable the use of Rejig anyway.

DIKO does this. After finding the best sample with 1% samples, it fine tunes with two 5% samples, to find an approximate linear variantion around the QFactor area and fine tune. But it doesn't seem to help with HC.

Ok, I think 1%, 10% or even 20% could provide more exact results but with 3-4 runs at 20% we are close to a normal 2 pass encoding time. So no real reason to use more than 5% IMO.

This is the approach I was thinking when I read your post. The problem is that it work fine with movies with about the same compressibility along the movie. Movies that start slow and becomes more "action packed" throught the end (for instance) would have strange results, since the deviation was already contained in the QFactor selected.

I don´t think so. The first CQ prediction run already gives you a good starting point.

Read manolito´s last posting on what he call hybrid mode (which is what I´ve discribed above):
http://forum.doom9.org/showpost.php?p=1166032&postcount=26

<snip>
This CQ value is used as the initial CQ for the adaptive mode. The first 40% of the movie are encoded with this CQ value. From then on I do allow CQ correction, but only in a very limited way. I set a clamp around the initial CQ so it can only get bigger by 0.60 and smaller by 0.50. This is enough to take care of prediction inaccuracies while still keeping an almost constant quantizer. Only for the last 10% of the movie I switch to "Aggressive Mode" where the CQ gets corrected without restrictions (almost: I still enforce an upper CQ limit of 8.00 unless the initial CQ is above 6.00).
<snip>

I think that I´ll write a simple combination of HCPredictor and HCEnc^n provide a functionality similar to what manolito has added to DVD2SVCD. I´ll let the user set the upper and lower "clamp" values for the CQ rate control. I would rather break a movie into 10% pieces. This allows better rate control over the CQ value.
Let´s see if this works out. Having something like this would be nice for DVD-RB also.

But first things first:
I want to finish HCPredictor first prior thinking about something like a HCHybrid tool.

Cheers,
Holger

[Darksoul71]

Oh yeah, one last thing:
If you feel like it I could release an intermediate version with a quick "howto configure".
Might still need one or two weeks for this but feel free to post your thoughts.

Hint:
That´s why I´ve named this thread "HC Predictor - any interest ?"

[vmesquita]

The bisection method on the other hand takes something like
5-6 runs to hit the target CQ value but will always converge
since the estimated size only indirectly influeces the calculation
of the next CQ value.

This for example are my current end criterias for HCPRedictor:

Code:

While (Abs ($CQ-$CQ_Alt) > 0.5) And (Abs($DeltaPercent) > 2)

I run the prediction until CQ[n+1] and CQ[n] have no bigger
difference than 0.5 and the percentual size difference between
estimated size and target size is no larger than 2%.

This works fine for me and my bisection method.

In order to use a MaxRun parameter, DIKO could do
the same thing I do:
Save the CQ value and size deviation in % in an array.
Once you´ve reached the maxrun and still haven´t found
a valid CQ you can simply run through this array and
return the CQ value with the lowest size deviation.
Not perfect but works quite nicely.

Looks like a good generic solution. It would work with any encoder.

Another direction I´m currently thinking about:
AQE is using an internal function to translate
the Q-value to quantizer values (which libavcodec
uses IIRC). This is done to mimic the behaviour
of CCE. From my knowledge this works very well since
AQE converges as fast and as exact as CCE.

May be the right approach would be using a Q-value
for size prediction and transform it to a quantizer
using AQE formula.

I wanted to do some experiments anyway to find out
if the size deviations in HCEnc for certain CQ values
allow the development of some sort of correction
function.

This is an interesting idea. I would take some work but could lead to some interesting results. If we can perform a few encodes with different matherial at different quantisizer and plot a graph, we would be able to see how its function behave. Maybe it's possible to approach using some sort of sinus or log function (for instance)

second reply for now since I wrote my last posting yesterday in the train on my way back home.

I was wondering why the identation  BTW I also love mobile internet. I'm a huge Opera Mini fan,I use it a lot in my good and old Nokia 6600. But I don't do a lot of typing since it's keyboard is only numeric.

I wrote SAPSTAR a PM. I dunno if he has enabled e-mail notification. May be we get some reply soon ?

Let's hope so! 

You all will be able to test and judge by yourself very soon. HCPredictor is basically "finished enough" but I still do some changes during my current testing. It is already compatible with HCEnc^n and uses the -cores=1 for prediction as DIKO does

Cool! 

To me there is a big difference between "not very good" and "i know it´s gonna cost my some quality and live with it".
HCPredictor has an ini-setting to disable the use of Rejig anyway.

Definatelly. The quality difference falls in "i know it´s gonna cost my some quality and live with it" in the tests I did back that time. You can only notice if you magnify the same frame in the two situations.

Ok, I think 1%, 10% or even 20% could provide more exact results but with 3-4 runs at 20% we are close to a normal 2 pass encoding time. So no real reason to use more than 5% IMO.

Agreed.

I don´t think so. The first CQ prediction run already gives you a good starting point.

Read manolito´s last posting on what he call hybrid mode (which is what I´ve discribed above):
http://forum.doom9.org/showpost.php?p=1166032&postcount=26

<snip>
This CQ value is used as the initial CQ for the adaptive mode. The first 40% of the movie are encoded with this CQ value. From then on I do allow CQ correction, but only in a very limited way. I set a clamp around the initial CQ so it can only get bigger by 0.60 and smaller by 0.50. This is enough to take care of prediction inaccuracies while still keeping an almost constant quantizer. Only for the last 10% of the movie I switch to "Aggressive Mode" where the CQ gets corrected without restrictions (almost: I still enforce an upper CQ limit of 8.00 unless the initial CQ is above 6.00).
<snip>

I think that I´ll write a simple combination of HCPredictor and HCEnc^n provide a functionality similar to what manolito has added to DVD2SVCD. I´ll let the user set the upper and lower "clamp" values for the CQ rate control. I would rather break a movie into 10% pieces. This allows better rate control over the CQ value.
Let´s see if this works out. Having something like this would be nice for DVD-RB also.

I still think that the "difference action level" issue would be a problem, but I had an interesting idea, combinig with what you said. Something like this:
1) Find a good starting point using bissection, with 1% samples and 3 or 4 interactions.
2) Encode a 1% sample taking only the first 20% of the movie and calculate expected average.
3) Encode 20% of the movie at the calculated Q.
4) Compare final size with the expected average in (2), which is not the full movie average. The deviation information is used to increase or decrease the Q level for the next part to compensate. But the compensation is done always from the full movie Q, not last part Q.
5) Encode a 1% sample taking only parts between 20% and 40% at the new Q and calculate the expected average.
6) Encode the portion between 20% and 40%. Compare with the expected average and increase or decrease the Q.
And so on...

Of course, the last 20% would have no further other parts to compensate. But its "potential to oversize" is much less than the whole movie, of course. To deal with this part we could:
1) Leave some room as you suggested previously
2) Use the 80% already encoded as "real data" to adjust Q. I'm not sure if this is trustable.

I don't think being agressive in the last 10% as suggested is a good, because the last part of the movie is generally important to look bad 

Oh yeah, one last thing:
If you feel like it I could release an intermediate version with a quick "howto configure".
Might still need one or two weeks for this but feel free to post your thoughts.

Sure I'm a little busy this week but I'll try 

[Darksoul71]

Hey folks,

it´s late in good old germany (23:10h) but I wanted to
share the first version of HCPredictor:

Grab it here:
http://rapidshare.com/files/135377570/hc_predictor_alpha_package.zip.html
or here:
http://www.megaupload.com/de/?d=2II7SA3I

OK, first a BIG FAT WARNING:
This is alpha software. So do not play around with it if you are afraid to break your computer

Installation:
Simply grab the package and unzip it somewhere....

At the first run of the HCPredictor.exe you´ll be asked to browse for HCEnc.exe (or HCEnc^n.exe).

Then HC Predictor exits and writes a HCPredictor.ini

Configure it as described below and you are set 

HCPredictor supports HCEnc^n with only one limitation:
You have to set the number of your cores in the HCEnc^n.ini

HCPredictor will parse but ignore the -cores parameter set by DIKO. I´ll change this in a future version

OK, now a short explanation on the ini settings and what the do:

The ini will look like this:

Code:

[Common]
Debug=0
HCEnc=C:\Eigene Dateien\AutoIt Projekte\HCEnc^n v0.1.6c\HCENC^n.exe
Safearea=0
CQ_BFACTOR=1
CQ_PFACTOR=1
CQ_MAXBITRATE=0
SampleSize=5
UseRejig=0
RejigPercent=5
MaxRuns=10
PredictionMode=1
StartCQ=8
[Expert]
CQ_Delta=0.5
Size_Delta=2
CQ_Precision=2

Debug=0/1
Enables extended logging to a local file called "Prediction.log" (which is stored in the HCPredictor directory) if set to 1

HCEnc=C:\Eigene Dateien\AutoIt Projekte\HCEnc^n v0.1.6c\HCENC^n.exe
This should be obvious....the path where HCEnc or HCEnc^n is stored

Safearea=0
This enables you to set a safe area to the final filesize to prevent oversize. Example: If you set this to 5 (= 5%) then HCPredictor will only use 95% of the true target size for size prediction.
So if you set this to 10-15(%) you should be pretty save from oversizing. You´ll loose some bitrate though for better quality.

CQ_BFACTOR=1
CQ_PFACTOR=1
CQ_MAXBITRATE=0
All those three ini settings correspond with the HCEnc.ini settings. If CQ_MAXBITRATE is set to 0 it will not be added
to the HCEnc.ini

SampleSize=5
This sets the sample size used in % for the size prediction. I think 5% is quite nice for HCEnc. 0.5 or 1% is definitely too small for HCEnc from my observations but anything > 2% should do.

UseRejig=0/1
This enables Rejig to recompress if the final encoded M2V is too big. Costs you some quality but is fast and saves you from re-encoding the movie.
If you can live with the quality that DVDShrink provides at > 75% compression level then try it.

RejigPercent=5
Defines at what oversize level Rejig kicks in. Default is 5%. So if your resulting CQ encoded movie oversizes bigger than 5% it will be "rejiged"

MaxRuns=10
Defines how many runs for prediction are acceptable

PredictionMode=1/2
Defines which prediction mode is use to calculate next CQ
1= Bisection method
2= Q(n+1) = Q(n) * (Estimated size / target size)

Other prediction modes might follow

StartCQ=8
Defines the starting CQ value for prediction. For bisection this is very important since a good startvalue will tremendiously cut down the number of prediction runs but can cause a major increase in number of prediction runs if choosen bad. Hank named 8 as his start value. So I do not think that this will hurt. Feel free to use another

[Expert]
CQ_Delta=0.5
Size_Delta=2
CQ_Precision=2

OK, the expert value are that the Ini-Tag says: For expert only !!!!!!

The first two values define the exit criteria for the prediction loop.
I think the AutoIt code is pretty selfspeaking:

Code:

While (Abs ($CQ-$CQ_Alt) > $CQ_Delta) And (Abs($DeltaPercent) > $Size_Delta) And ($Run < $MaxRuns + 1)

CQ_Delta defines the value required as difference between CQ(n) and CQ(n+1) in order to exit the prediction loop.
Size_Delta does the same for size. Prediction will loop until the absolute value of the predicted size doesn´t defer more then the defined percent value from the target size.
So Size_Delta = 2 means 2%. The Prediction will loop until the file is within 98% to 102% of the target size

CQ_Precision defines to how many digits the calculated CQ values are rounded.
CQ_Precision = 0 would limit you to integer value. So you would have 1 to 32 CQ values (similar to QuEnc)
2 leaves you two digits (e.g. CQ=2.31)
1 leaves you one digit (CQ=2.1)

HCEnc uses a precision of 3 digits internally. So setting the value to more than 3 makes no sense.
Any value smaller than 3 digits will cause quicker prediction but also means less exact prediction results.
I think 2 digits is a nice compromise.

Now feel free to play

Cheers,
D$

[Darksoul71]

Hi VM,

ok, just some notes prior I head to bed:

Looks like a good generic solution. It would work with any encoder.

I think this could help the DIKO prediction routine to avoid something like 50-100 runs in rare cases and still gives you the best bang for your buck (as the U$ say )

I was wondering why the identation  Grin BTW I also love mobile internet. I'm a huge Opera Mini fan,I use it a lot in my good and old Nokia 6600. But I don't do a lot of typing since it's keyboard is only numeric.

It was not really mobile internet. More downloading the forum thread and answering offline by typing in Notepad++ on my notebook
Mobile internet is still not really cheap here in Germany and unless my company pays me a HDSPA card plus all resulting costs I will
definitely not be browsing the www from train by any means :-)

This is an interesting idea. I would take some work but could lead to some interesting results. If we can perform a few encodes with different matherial at different quantisizer and plot a graph, we would be able to see how its function behave. Maybe it's possible to approach using some sort of sinus or log function (for instance)

Agreed but let´s wait for SAPSTARs reply (see below). If it doesn´t work for HCEnc then I´ll do further investigations. At least I hope that my idea is not completely crap.

Let's hope so!

Jup, already got PM from SAPSTAR the very same day. He promised me to take a look at his sources and tell me what he did to translate Q-value to quantizers.

I don't think being agressive in the last 10% as suggested is a good, because the last part of the movie is generally important to look bad

I second that. That was the first thought when Manolo kindly explained what he is doing. I think that CQ "rate control" should take place
over the complete movie. Better do smaller changes over a bigger period than doing "wild" changes for only a small period.

I still think that the "difference action level" issue would be a problem, but I had an interesting idea, combinig with what you said. Something like this:
1) Find a good starting point using bissection, with 1% samples and 3 or 4 interactions.
2) Encode a 1% sample taking only the first 20% of the movie and calculate expected average.
3) Encode 20% of the movie at the calculated Q.
4) Compare final size with the expected average in (2), which is not the full movie average. The deviation information is used to increase or decrease the Q level for the next part to compensate. But the compensation is done always from the full movie Q, not last part Q.
5) Encode a 1% sample taking only parts between 20% and 40% at the new Q and calculate the expected average.
6) Encode the portion between 20% and 40%. Compare with the expected average and increase or decrease the Q.
And so on...

Hm, nice....I like the idea of doing segment based CQ prediction. In sum doing a prediction for each segment shouldn´t take more time than doing a CQ prediction for the complete movie since we talk only about 1% of 20% (for example)
.
The last approach I thought about combined with yours would look like this:
1) Segment movie to n segments
2) Calculate segment target size (= complete target size / # of segment)
3) Do CQ prediction for first segment
4) Get segment size
5) New complete target size = full target size - segment size
6) Calculate next segment target size (= new complete target size / # of remaining segment)
7) Do CQ prediction for second segment
Get segment size for second segment
9) start with step 5 until no segments remain

I hope you get what I mean . This way oversize segments mean less space for the remaining
segments whereas undersized segments leave more space for remaining segments.

Short example:
Target size 4000 MB
Using four segment.
Target size for first segment 1000 MB
File size for first segment after CQ run: 850 MB
New target size for next three segments: 4000-850 MB = 3150 MB
Target size for second segment = 1050 MB

I think this could work very well....

We´ll see.. let´s finish HCPredictor first *ggg*

Now I´m off to bed.

TTYL,
D$

[vmesquita]

It was not really mobile internet. More downloading the forum thread and answering offline by typing in Notepad++ on my notebook
Mobile internet is still not really cheap here in Germany and unless my company pays me a HDSPA card plus all resulting costs I will
definitely not be browsing the www from train by any means :-)

Cool I also love NotePad++.
BTW Mobile Internet is not very cheap here in Brazil too. But I have a 1Mb monthly traffic plan (you read right, it's 1 Mb not 1 Gb), which is more than enough for casual Opera Mini surfing and Google maps mobile once in a while. I pay $ 12 monthly for this plan.

Agreed but let´s wait for SAPSTARs reply (see below). If it doesn´t work for HCEnc then I´ll do further investigations. At least I hope that my idea is not completely crap.

Ok.

Hm, nice....I like the idea of doing segment based CQ prediction. In sum doing a prediction for each segment shouldn´t take more time than doing a CQ prediction for the complete movie since we talk only about 1% of 20% (for example)
.
The last approach I thought about combined with yours would look like this:
1) Segment movie to n segments
2) Calculate segment target size (= complete target size / # of segment)

Here's the problem (I think): segment target size is not complete target size / # o f segments (or at least should not be IHMO). Why? The same "action packed vs peaceful issue). Many content have calmer areas and high movement areas. This way the calmer areas will receive as much bitrate as the high movement areas. That's why I think it would be better to encode a sample of the segment. This way we will know the true expected average according to a sample, and we prevent calmer areas to look better than action areas. This can also be negligible, we would need some movie with this caracteristics (slow pacing in the begining, fast movind scenes later) to be sure.

I also saw the HC predictior release, but since I am at work and here the block rapidshare and megaupload, I'll have to wait to download at home