HCShredder - the guided CQ encoding frontend for HCEnc

[Darksoul71]

Hello *,

this is the first release of HCShredder - the guided CQ encoding frontend for HCEnc

Grab it here:
http://www.megaupload.com/de/?d=XN196BQ9
or here:
http://rapidshare.com/files/140067487/HCShredder.zip.html

Readme follows here:
What is / does HCShredder ?
HCShredder is somewhat a combination of a segmented
encoder (= HCEnc^n) and a CQ prediction tool (= HCPredictor).
It´s based on the ideas of manolito.

Have a look at the following thread for more details:
http://forum.doom9.org/showthread.php?t=138566

I just implemented what manolito in reply to my question discribed here:
http://forum.doom9.org/showpost.php?p=1166011&postcount=25
http://forum.doom9.org/showpost.php?p=1166011&postcount=26

Think of this as a "guided CQ" mode. manolito called it hybrid mode.

Essentially it´s just 1 pass VBR using size prediction and CQ encoding.

Since we are only adjusting the quantizer value slightly I think we
can still call this CQ encoding.

HCShredder is configured via the HCShredder.ini file.
It will be generated at the first run. Keep HCenc and
DGPulldown available prior starting HCShredder.

Now more details on the HCShredder.ini:

Stuff in [common]:

Debug=0/1
Enables debugging (= extended logging) when set to 1.
Default is 1.

HCEnc=C:\YourPath\HCenc_023.exe
Stores the path where you installed HCEnc.

Safearea=0
This enables you to set a safe area to the final filesize to prevent oversize.
Example: If you set this to 5 (= 5%) then HCPredictor will only use 95% of the
true target size for size prediction. So if you set this to 10-15(%) you should be
pretty save from oversizing. You´ll loose some bitrate though for better quality.
Not really needed for HCShredder but gives you some safe area.

CQ_BFACTOR=1
CQ_PFACTOR=1
CQ_MAXBITRATE=0
All those three ini settings correspond with the HCEnc.ini settings. If CQ_MAXBITRATE
is set to 0 it will not be added to the HCEnc.ini

SampleSize=5
This sets the sample size used in % for the size prediction. I think 5% is quite nice for HCEnc.
0.5 or 1% is definitely too small for HCEnc from my observations but anything > 2% should do.

MaxRuns=5
Defines how many runs for prediction are acceptable. Personally I wouldn´t cut down this
too mutch because otherwise the starting Q value might be way off.

PredictionMode=1
Defines which prediction mode is use to calculate next CQ
1= Bisection method
2= Q(n+1) = Q(n) * (Estimated size / target size)

Other prediction modes might follow

StartCQ=8
Defines the starting CQ value for prediction. For bisection this is very important since a
good startvalue will tremendiously cut down the number of prediction runs but can cause a major
increase in number of prediction runs if choosen bad. Hank named 8 as his start value. So I do not
think that this will hurt. Feel free to use another value.....

FixM2V=1
Enables the fixing of the incorrect timecodes with DGPulldown (similar to HCEnc^n).

DGPulldown=C:\Eigene Dateien\AutoIt Projekte\HCEnc^n v0.1.6\DGPulldown.exe
Stores the path where you installed DGPulldown.

Use_2Pass=0/1
This will enable two pass mode for the last segment inf set to 1.
This was VMesquita´s idea to hit the target size exactly. I like
this idea and will add this functionality in one of the next versions.

Stuff in [Expert]
CQ_Delta=0.5
Size_Delta=2
CQ_Precision=2

OK, the expert value are that the Ini-Tag says: For expert only !!!!!!

The first two values define the exit criteria for the prediction loop.
I think the AutoIt code is pretty selfspeaking:
Code:

While (Abs ($CQ-$CQ_Alt) > $CQ_Delta) And (Abs($DeltaPercent) > $Size_Delta) And ($Run < $MaxRuns + 1)


CQ_Delta defines the value required as difference between CQ(n) and CQ(n+1) in order to exit the prediction loop.
Size_Delta does the same for size. Prediction will loop until the absolute value of the predicted size doesn´t
defer more then the defined percent value from the target size. So Size_Delta = 2 means 2%. The Prediction will
loop until the file is within 98% to 102% of the target size

CQ_Precision defines to how many digits the calculated CQ values are rounded.
CQ_Precision = 0 would limit you to integer value. So you would have 1 to 32 CQ values (similar to QuEnc)
2 leaves you two digits (e.g. CQ=2.31)
1 leaves you one digit (CQ=2.1)

HCEnc uses a precision of 3 digits internally. So setting the value to more than 3 makes no sense.
Any value smaller than 3 digits will cause quicker prediction but also means less exact prediction results.
I think 2 digits is a nice compromise.

FilterAVS=
Here you can specify an AVISynth script file which will be automatically added during encoding.
This could be a general purpose denoiser, your favourite automated cropping and risize via GripFit, etc.
Simply specify the path to the script and the name of your script here.

HCValues=
Here you can specify the path to a text file with your custom HC ini settings. This
could be a custom matrix, the *AQ parameter, etc.
Mind you that HCShredder doesn´t do any input validation or script parsing.
So make shure your custom settings do not interfere with the HC.ini values
given via commandline.

Segments=10
Defines the number of segments used for the encoding process.

Q_Low_Border=0.8
Defines the "lower" border for Q-values

Q_High_Border=0.8
Defines the "higher" border for Q-values

Q_Is_Percent=0/1
Determins wether the Q-border values given are in percent or absolute.
If the to 1 then Q_High_Border and Q_Low_Border values are treated
as percent values, e.g. 20 = 20%

If set to 0 then the values are treated as absolute values. I first experimented
with values around 1.5 but lowered them to 0.8 for now.


manolito said that he only used very small absolute values for this
<snip>
so it can only get bigger by 0.60 and smaller by 0.50.
<snip>

Just try out yourself. YMMV.....

[vmesquita]

Great, Darksoul71! 

I have added to the news page. 

[Darksoul71]

@VM:
You are welcome !

@all:
please adjust the clamp values in your ini:

Code:

Q_Low_Border=0.8
Q_High_Border=0.8
Q_Is_Percent=0

you may also experiment with smaller values (e.g. 0.6)

[mikneadia]

Exciting piece of software.

If I may, I think that prediction method no2 may be improved ( I do not know how ) based upon:
a) Take into account all Q calculations (from 0 to N) to predict N+1
b) It seems that the optimal Q (the one that gives you the desired size) is an inflexion point ( method 2 will overestimate Qopt if Q(N) is smaller than Qopt and underestimate it if Q(N) is higher than Qopt. It might be the other way around (I have done those test long time ago).
You may have a adjustement to make (positive or negative depending if this is an oversize or undersize) based on method 2.

Q(N+1)=(1+K)*Q(N)*S(N)/S(desired)

with K depending of the amount of over or undersizing (empirical lookup table)
or if we are lucky
a=(S(N) - S(desired))/abs(S(N) - S(desired))  ; a=1 if S(N)>S(desired) or -1 if S(N)<S(desired)
b=max(S(N);S(desired))/min(S(N);S(desired))  ; b=S(N)/S(desired) if a=1 or S(desired)/S(N) if a=-1
alpha is a constante (regression analysis based on empirical data)
1+K=((1+(alpha*b))power(a))   ; if for ex alpha=1 then K=b if a=1 or 1+K=1/(1+b) if a=-1

The idea is that the graph is symetrical ( on a log basis) with regards to oversizing and undersizing (an oversizing of 100 % (S(1)= 2*S(desired)) should lead to similar decisions than an undersizing of 50 % (S(desired)=2*S(1)), at least inside a reasonable interval (b between 1 and 2).
I did not check the accuracy of the above formulas (just a lead).
If at one point, we have two iterations that represents a small oversizing and a small undersizing , we can check that the next prediction make sense from the left side and the right size of the curve (the oversized one and the undersized one). May be assuming that the left slope and the right slope are identical will lead to a better prediction (all of this on a log graph (Log(Q) vs (Log(S)).

Great job.

P.S. Another approach will be to make at each iteration a prediction based on a educated interpolation of two non-perfectly correlated estimates. One that will most likely undershoot ( like your no 2 method for ex) and one that will most likely overshoot (like the one used in Rb-opt for ex).

[Darksoul71]

Hi mikenadia,

thanks for your posting !
I guess I should have more clearly stated that those two functions are only used
in the very first place during the CQ prediction to find the Q-value of the first
segment.

I second that the optimal Q is an inflexion point. Honestly I thought about using
Newton Raphson for the CQ prediction in the first place but was to lazy to do so.
Esp. since even bisection converges after five iterations. For HCShredder I do not
need a fully perfect prediction. Just a good starting value.

For the rest of the encoding I simply alter the CQ value directly based on the
size deviation of the last segment but limited by the clamp values. Here is where
I want to put a bit more brain in it. May be I/we can work out a good quantizer
rate control algorithm. For now using clamps worked fine though. I had a few
undersizes by 5-20% maximum (pretty similar to QuEnc) and mostly hit the target
size by +/- 2% max.

Not bad at all I think in the first run....

VMs idea of using a size change per quantizer makes my mind tick.
May be I should first calculate how many kB the resulting segment
changes by altering the quantizer value by 0.1 for example.

Anyway we are also talking about the slope in this point of the function.

I´ll have to think about this some time. Usually good ideas pop up if
I ride to work on my bike....

I´ve sent hank a PM over at doom9 on this.....
Hopefully he continues his experiments on this hybrid, OPV, VBR encoding mode....
I see a lot of potential in this. It´s a pitty I didn´t pay attention to manolitos
experiments earlier since they save us from the biggest problem of HCenc in OPV
mode: Oversized files

The penalties are very limited:
Parallel CQ encoding with multiple HCEnc instances via HCEnc^n might be a bit faster
but multithreaded encoding is very fast also.....

At five prediction runs with 1% sample we save 95% time compared to a two pass encoding.

Well let us experiment a bit with HCShredder and different values. Then we can see if
things work out...

Cheers,
D$

P.S.: It´s amazing to see what manolito has done simply based on batch files with a
few external tools. Have a look here for HC_D2S:
http://forum.doom9.org/archive/index.php/t-133066.html
Haven´t used DVD2SVCD for ages but was an avid user a few years ago. Esp.
with D2SRoba.

[mikneadia]

Hi, Darksoul71

This place is quieter than doom9 (I love it!!!).
Keep on biking. You are doing great.
You may have little use for fast Q converging but others might be interested.
If I have an idea concerning the quantizer rate control algo, I 'll post it.
And yes, congratulations to Manolito ( and you, too!)

Presently, I am using Rb-opt for redistribution (Main movie in MPEG Standard and all extras in AVAMAT7).
For a given segment and a given size, AVAMAT7 gives me the lowest Q.Even if the extra segment is interlaced and the main movie progressive, most of the time,the Qopt for the Extra segment (in AVAMAT7) is lower than the Qopt for Main movie (in MPEG standard).
By doing a redistribution on the  whole DVD (in one pass) I am able to get a satisfactory bitrate distribution.
The issue with Rb-opt algo is that for Extras (when Q are high or when the initial Q is far away from Qopt) it implodes (the Q are almost diverging, not converging) and it makes for that segment a very wild guess when the maximum number of iterations is reached.
Q init depends upon the matrix, the bitrate but I am not sure if it takes into consideration whether the segment is interlaced or not.
It is difficult to use in the current state, but...

Thx a lot for your efforts and program and let's go back to THE QUANTIZER RATE CONTROL ALGO.

Basic thoughts.
1) Concerning the size change per quantizer, may be look at size change (in %) per quantizer change ( in %).
2) Also may be considering a decrease in segment size at each iteration ( an error in estimating Q in one of the last segment will have less and less impact and will  have time to be corrected even taking into account the "Clamp" constraints (you will have a lot of small segments after 80 or 90 % of the encode) even if you have to increase samplesize accordingly to maintain a sufficient nb of frames.The first segment might even be 40 % of the total encode!
3) I think the issue is similar to size betting (How much should I bet to be sure (99.9%) that I will not lose more than half my money after 30 bets, then 29...).Repeat the problem after each encode for the last 30 ones.
At every segment encoding, compute the error in size estimation ( in %), study and update the distribution of errors and if the mean is negative consistantly, we may suspect saturation and adapt our algo accordingly. In the last segments, we could look at the variance of those errors and with the mean , modify the algo so we are sure (mean + 2 or 3 STD) that the remaining segment encodings will not create total oversizing (if we want to avoid that at any cost). We may have to assume that all encodings are independent events (true or false?) and use cumulative Bernouilli (or whatever) distribution (n=30 for the A -30 segment (if we do not want the cumulative to exceed with 99.9% confidence, we can deduct p and 1-p for one event and estimate the Q for that event (encode), update error distribution, n=29 for the next segment... where A is the total number of segments to encode). It is getting too complex for me ( do not know how to handle the fact that the last 30 encodes have different number of frames...), but if you think that this may lead somewhere, I can go back to the library (or even better , a younger and brighter member doing the "real" work).

[weiguo]

Thank U So Much For your Great Tools!. I'm not a expert like you, and then I'd like to ask you. Is there any command or instruction similar to the CCEFront "-size_mb MB"?. If not, how could I work with different size predictions?, like: 2000, 4000 or 8000 MB.

TIA javascript:void(0);

[Darksoul71]

Hi all,

wow, it´s getting crowded here

@mikenadia:
>>This place is quieter than doom9 (I love it!!!).
So do I ! The general discussion "tone" seems to me
much friendlier and more personal than over at D9.

In regard to your basic thoughts about q rate control:
I have to think about this. In general I dislike adjusting
the segment length on the fly and making the calculations
too complex. I want the source code still be handable
for someone without math diploma

..but in general I agree that there is a lot of improvement
possible (depending on your math knowledge). My
math is *eerr* well, lets say limited 

If one tells me how to code the algorithm I can do it though   

Tylo explained once this Newton Raphson stuff to me very well but,
hey, he is the author of D2SRoba. If he wouldn´t know who
would it be then ?

@weiguo:
You are welcome !
I do not consider myself really an expert. I just collect ideas
from other peoples and make something out of it.

Ok, back to your question(s):
>Is there any command or instruction similar to the CCEFront "-size_mb MB"?.
There is no "size_mb" command since I didn´t want to break CLI compatibility
with HCEnc but there is a -filesize parameter which lets you specify the target
file size in kbyte. I could parse this and use this size directly as target size for
the prediction run. What do you think ?

>If not, how could I work with different size predictions?, like: 2000, 4000 or 8000 MB.
Well this would be a typical functionality of the frontend calling HCShredder, e.g.
DIKO. Or you could use any bitrate calculator out there to calulate which birate
is neccessary to hit your target size.

HCShredder currently reads the bitrate from HC.ini, extracts the frame count of
the source movie and simply calculates the targetsize like this:
Targetsize = (number of frames / FPS) * Bitrate

Cheers,
D$

[Darksoul71]

@weiguo:
Fresh from the AutoIt compiler:
http://www.megaupload.com/de/?d=WS473FCW

Has support for -filesize <your size in kbyte>

I did a quick run and it seemed to work

[mikneadia]

Hi, Darksoul71

1) About segment length, I thought that it was a math concept, that each actual segment was virtually subdivided.
2) It is my understanding that manolito is still using a  predictor algo (predictive mode method with only one iteration) to guess the next segment Q (based on the cumulative size and probably a weighted cumulative Q), so the predictor algo is used more than for computing the initial Q.
3) For quality purposes, it may be interesting to favor undersizing (to keep some bitrates for the end), which means use a conservative approach (no overshooting: the actual method I think) if the cumulative size is below the desired cumulative size (decrease Q for next segment) and an aggressive method (like the one used in rb-opt that will cause probably overshooting) if it is above (increase Q for next segment).
4) we could also do the following: After encoding segment N, we have a cumulative size, a desired cumulative size and a weighted cumulative Q. We assume that cumulative Q will produce the same error (in %) and we make two adjustments (first to compensate for that error: predictive mode method) and the second one to adjust for the new desired size ( we compensate the error in size of the encode of the first N segments in adjusting the size of  the remaining ones) , at least when the system is not in aggressive mode (first 90 % of the total encode).
5) the clamp may be in %.

[Darksoul71]

Re-Hi,

The issue with Rb-opt algo is that for Extras (when Q are high or when the initial Q is far away from Qopt) it implodes (the Q are almost diverging, not converging) and it makes for that segment a very wild guess when the maximum number of iterations is reached.

This seems to indicate that RB-opt should use the approach I discribed here:
http://www.vmesquita.com/forum/index.php/topic,8865.msg57178.html#msg57178
<snip>
Save the CQ value and size deviation in % in an array.
Once you´ve reached the maxrun and still haven´t found
a valid CQ you can simply run through this array and
return the CQ value with the lowest size deviation.
Not perfect but works quite nicely.
<snip>

This way we can use the CQ with the lowest size deviation
even if the CQ prediction algorithm doesn´t converge.

Thats a thing I like about bisection:
I must converge

HCShredder leaves you the option to choose between
size-based calculation of Q(n+1) or bisection.

1) About segment length, I thought that it was a math
concept, that each actual segment was virtually subdivided.

No, currently segment length is static but I will experiment a bit with this...
Problem here: I´m simply using the segmentation code of HCEnc^n which generates the
segments prior any run. Changing this to a dynamic approach would require a major
rework of HCShredder (= somewhere on the lower end of the todolist)

2) It is my understanding that manolito is still using a predictor algo (predictive mode method with only one iteration) to guess the next segment Q (based on the cumulative size and probably a weighted cumulative Q), so the predictor algo is used more than for computing the initial Q.

Yes, agreed ! I just found it in his explanations. I forgot about this somehow.

It´s here in his posting:
http://forum.doom9.org/showpost.php?p=1166001&postcount=24

Unfortunately I have no real clue how this weighing should look like for HCShredder.
How do I calculate Q(N+1) by taking the previous segments into account ?

Suggestions ?

3) For quality purposes, it may be interesting to favor undersizing (to keep some bitrates for the end), which means use a conservative approach (no overshooting: the actual method I think) if the cumulative size is below the desired cumulative size (decrease Q for next segment) and an aggressive method (like the one used in rb-opt that will cause probably overshooting) if it is above (increase Q for next segment).

I think the user can already choose between a more conservative approach and a more agressive approach.
By setting the lower limit to 0 we allow only increasing the Q-value but not decreasing it below the
start Q. This way we limit the risk to oversize. Allowing raising and lowering the Q-value is more
"aggressive". We may hit the target size more precisely and get no undersize but on the other hand
we may experience oversizing.

A simple approach for "conservative encoding":
- Allow only increasing the quantiser but not lowering it below the start Q.
- Do 2 pass encoding for the last segment.

What do you think ?

4) we could also do the following: After encoding segment N, we have a cumulative size, a desired cumulative size and a weighted cumulative Q. We assume that cumulative Q will produce the same error (in %) and we make two adjustments (first to compensate for that error: predictive mode method) and the second one to adjust for the new desired size ( we compensate the error in size of the encode of the first N segments in adjusting the size of  the remaining ones) , at least when the system is not in aggressive mode (first 90 % of the total encode).

Could you please elaborate this ? I´m a kind of hands on type and always require
some examples prior understanding / using something.

TIA....

5) the clamp may be in %.

You can choose wether to use relative clamp value (%) or absolute clamp value in the ini.
Relative clamp values mean a wider range of absolute clamp values for bigger quantizer
values. For small Q-values (e.g. < 10) there is no big difference between using 20%
clamp value and something like 1.6 Q as clamp value.

I mean: In the very first place HCShredder is ment to be experimental software.
I´m expecting and want people to experiment with clamp values, sample sizes, etc.

Regards,
D$

[mikneadia]

Hi, D$

I am not very good with using "Quote" function and I did not read all manolito and your posts; I am sure I did not understand fully all ALGOs, but this is for discussion purposes.

1) your question 1.
Do not want to argue but do not think why it has to be dynamic. You can divide segment 1 in 10 equal subsegments (length L/10) or (L/5,L/6...L/15). I did not do the math (they do not add up to L). I am not sure that this will be an improvment.
End of discussion.

2) your question 2
My understanding.
a) you encode segment 2 at Q(2). It gives you S(2).you compute CQ(2), which is the weighted cumulative Q after encoding segment 2.
CQ(2)=((Q(1)*Nb(1))+(Q(2)*Nb(2)))/(Nb(1)+Nb(2)) where Nb(1) and Nb(2) are the number of frames of segment 1 and 2
I was arguing that this is wrong if Q(1) and Q(2) are very different
http://forum.doom9.org/showthread.php?p=1071747#post1071747  (post 736)
but as of now, I will shut my mouth (I won't: generalized concept in next post ( using again predictive mode 2)).

b) Now, we have a combination of segment 1 and 2 encoded at CQ(2), actual size S(1)+S(2), desired size Sdesired(1) +Sdesired(2)
there is an error that you want to correct (predictive mode 2).but instead of applying that correction to the same encoded segment (1+2), you apply it to segment 3.
therefore Q(3)=CQ(2)*errorsize(2)
with errorsize(2)=(S(1)+S(2))/(Sdesired(1) +Sdesired(2))
Now you encode segment 3 at Q(3) with Sdesired(3)

I may have made some mistake between Nb, S,....  but that is the idea.

3) your question 3
a) I was thinking of HCshredder taking the decision itself between conservative    (predictive mode 2) and aggressive method (bissection...) based on the cumulative size critera (above or under the cumulative desired size).
It is like a dog on a leash. If he behaves well, you treat him conservatively . If he goes over the fence, you bring him back aggressively (your decision, not the user one).
That decision is made by the ALGO before encoding each segment (or subsegment...).
b) the 2-pass encoding is a good idea (last resort if possible) and the decision to do it or not should be done by the ALGO based on conditions at the end of the before last encode.

4) your question 4
I am going to try.
you have actual segment to encode subdivided in 3 subsegments 1,2,3.
They have each a desired size.
At the end of encode 2, I know the actual size of (1+2) with errorsize(2) (compared to Sdesired(1) +Sdesired(2)).
In your question 2 (cf above) , I am computing Q(3) based upon errorsized 2 for Sdesired(3).***
But now I want to change Sdesired(3) to take into account that "actual size of (1+2+3)= Sdesired of(1+2+3)".
NewSdesired(3)= Sdesired(3) - errorsize(2)
LineD$: Therefore NewQ(3)=Q(3)*Sdesired(3)/NewSdesired(3)  ; Q(3) has been computed 3 lines above:***

Here, errorsize(2) is a difference. In question2, it was a ratio. I should have renamed one of them.
(3) represents all the remaining subsegments to be encoded (if they were more than 3).

If you go that road, predictive mode algo (used in LineD$ just above) might have to be improved ( discussed before) because it will be used at least twice (may be 4 times if  used for the algo in 2a) per segment encode and the errors are cumulative and may not cancel each other. We should even give him a different name (best predictor...) because it is only a one_iteration algo.

I probably made some mistakes and I hope I was clear . Do not hesitate to ask for clarifications or corrections or improvments.

[mikneadia]

Trying to find the Weighted Cumulative Q of two segments.
We assume that we know Q1opt and Q2opt and their encoded sized S1 and S2.
Will not used that if Q1 opt and Q2 opt have been wildly guessed (no encoding) based on previous estimates.
This is the case for non converging attempts.
I assume that Q (weighted cumulative Q of Q1opt and Q2opt) is close enough to Q1opt and Q2opt so that I can use:
Q=Q1opt*S1/S1(new)    and Q=Q2opt*S2/S2(new)  where S1(new) and S2(new) are estimates of the size of segments 1 and 2 encoded at Q.
I have S1(new)+S2(new)=S1+S2
((Q1opt*S1)/Q)+((Q2opt*S2)/Q))=(S1+S2)
LineD$: Q= ((Q1opt*S1)+(Q2opt*S2))/(S1+S2)

which is very similar to the previous one but can be adapted if we find a better approximation of Q around the optimum or can be generalized to situations far from the optimum if we find a better estimation (cf previous discussion) or when Q1opt and Q2opt are not the optimum and we know S1 and S1(desired). At the optimum, S1=S1(desired).

P.S.  It will be important to know when we can use  Q=Q=Q1opt*S1/S1(new) :best predictor equation
How far from the optimum and check that this is the "best" approximation around the optimum.
In the above-mentionned example, we should check a posteiori that the value Q compute in LineD$ is within the usabililty interval of "best predictor equation" for both segments. If Q1opt and Q2opt are too far apart and similar size, Q will be in the middle and too far from Q1opt and Q2opt.

[weiguo]

@weiguo:
Fresh from the AutoIt compiler:
http://www.megaupload.com/de/?d=WS473FCW

Has support for -filesize <your size in kbyte>

I did a quick run and it seemed to work

Thank U So Much!, I'm trying...

[mikneadia]

Hi, D$

This thread is very interesting (missed it before)
http://www.vmesquita.com/forum/index.php/topic,8865.msg57178.html#msg57178

It will be nice if you could resume what is pertinent ( for newcomers to this thread coming from D9 or else); could be used as a sticky to give newcomers an update on issues (solved and unsolved) concerning HCPredictor.
Now, after having read the thread, I have some questions.
1) How do you choose the first Q (before doing any encoding). Same number all the time or based on some information you can get from the source (average bitrate, interlaced or not....).
2) You choose bissection to ensure convergence .The "false position method" and "Brent's method" looks interesting (always converging but read " analysis " paragraph of the False position method: both end-points should move at each iteration).
http://en.wikipedia.org/wiki/False_position_method
http://en.wikipedia.org/wiki/Brent's_method
3) As of now, I understood that you have two methods (the conservative one : predictive method 2) that may converge after 3 iterations and the aggressive one (bissection) that will converge after 5 or 6 iterations.
in the bisection, what is your algo to make sure that the first iteration is of opposite sign?
The main goal being convergence and the secondary one being reducing number of iterations,may be we can try a combination of both.

We assume f(a)>0; The conservative method will probably predict b1 such as f(b1)>0 (No overshooting).
The aggressive method will probably predict b2 such as f(b2)<0 (Overshooting).

First iteration: you take weights that favors the conservative approach:
You take b3=(b1+ c*b2)/(1+c) where c is between 0 and 1.
If f(b3)<0, you are sure to converge, if not you do another iteration with new values for b1 and b2 (they can be recomputed at each iteration: it should b1(N) and b2(N)), but this time you increase c above 1.
If you still did not have negative number, go all the way for bisection (or higher and higher c).
4)Other approach:For the bisection, are you taking the midpoint or making educated guess that may increase convergence.We could use the interpolated value (cf 2 lines above) as the intermediary point as soon as we have two iterations with opposite signs.
5) Because I did not know and did not read your (and VM) previous works, my suggestions may have been tried before, found erroneous or be contradictory. I apologize for that .

May be we should merge the two threads, close this one with only the first 3 posts and a link to the other thread.
Great work (too bad I missed it before).