Would a HCEnc^n be off interest for you ?

[Darksoul71]

Hi all,

do you think a tool similiar to QuEnc^n would make sense for HCEnc ?
Would this be something you widely use ? Or is one encoder "middleware"
which enables you to run multiple instances of QuEnc on your multicore
system enough ?

Best regards,
D$

[vmesquita]

Hi Darksoul71,

Did you read my mind?  I was thinking exactly about this yesterday. QuEnc is great but it's based on avcodec, so it has the same flaws freeenc has: a bugged rate control that sometimes delivers wrong filesize and bitrate spikes, which causes muxing issues and playback problems on SVCDs (which are more restrictive on max bitrate). HC is free and written from scratch, so it doesn't suffer from this issue. Of course I don't know about the complexity of doing this... 

[Darksoul71]

Hi VM,

not really...
I had HCEnc^n in mind for quite some time.

in regard to complexity:
Consider HCEnc^n as work in progress 

[vmesquita]

Consider HCEnc^n as work in progress 

CooL! 

[NathanX]

That would open a plenty of possibilites, so... OH YEA :-)

[danpos]

QuEnc is great but it's based on avcodec, so it has the same flaws freeenc has: a bugged rate control that sometimes delivers wrong filesize and bitrate spikes, which causes muxing issues and playback problems on SVCDs (which are more restrictive on max bitrate). HC is free and written from scratch, so it doesn't suffer from this issue. Of course I don't know about the complexity of doing this... 

Hi V.! I'm a few far away from 'scene' but last time that I checked kvcd.net forum out, I did see a post from Sagittaire in which he commented out about progress on avcodec library, with a fix onto VBV rate control and others critical fixes.

Just For Your Information.

Regards,

[Darksoul71]

Hi all,

ETA for HCEnc^n is next weekend.
I think I´m able to finish the code today or tomorrow since the majority of the
functionality was already implemented with QuEnc^n. So my coding is essentially more an adaption of those parts of the code which are specific for QuEnc.

There will be some more testing to do though prior I release HCEnc^n. Thus -> next WE 

@NathanX:

That would open a plenty of possibilites, so... OH YEA :-)

Could you be more specific ?

My main reason for asking this:
I´ve some ideas in mind which wait for being implemented in HCEnc^n.

The initial release of HCEnc^n will be very simplistic, which means that will be segmented encoding only. No adaptive bitrate distribution as in QuEnc^n. I´ve planed to add this for a future version but for now it will be a KISS (Keep it stupid and simple) implementation. This is also true for the CLI support. The first release will only support "-ini C:\My HC.ini"

All the other CLI switches for specifying infile, outfile, etc. might follow in a future version although I´m unsure about this.

One of the next versions will feature ADB (= adaptive bitrate distribution) and a more future version might provide CQ prediction using Inc´s Slicer.avs

I´ve already received the source code for the HCEnc CQ prediction which Mr_Odwin used for FAVC.

May be my idea is stupid and I should leave out CQ prediction but I have the feeling (since HCEnc seems to be more predictable now) that a multi-instance CQ encoding with HCEnc will be very fast.

My main motivation to code HCEnc^n were my tests yesterday where HCEnc showed less ringing and seemed to produce a sharper image than all AV-lib based encoders which I used for reference (FreeEnc, QuEnc, NuEnc).

QuEnc definitely looks good a lowest bitrates but I have to test now how it competes to HCEnc. I´ll do some "One CD KSVCD" tests very shortly.

Best regards,
D$

[Darksoul71]

Wohaaaa....HCEnc^n up and running ! 

Unfortunately HCEnc isn´t able to run multi-instanced from within the same directory since it writes the video informations from the first pass into his dbs file. Every instance of HC requires to be run in a separate directory. To avoid this problem I create a temporary HCEnc directory for each core which are sequentially numbered, e.g. HC_000, HC_001 and so on.

Not perfect but the only way to deal with this unless Hank adds a switch to specify the dbs file used 

[vmesquita]

Hi Darksoul71,

Looks like a good workaround to me!  I am planning on creating some sort of "middleware" that will detect the number of processors and launch regular HC or HC^n according to the processor. This would be a great speed improvement in DIKO! 

[NathanX]

@Vmesquita,

it would be really a nice "feature", if Diko could work with QuEnc^n or HC^n. That would open the possibility to use even more advanced filter chains without losing too much encoding speed when you own a dual-core CPU. And the existing (custom) filter chains would be even faster than they are at the moment.

@Darksoul
i have made a lot one disk K/BSVCD in the last years. To my eyes QuEnc has its advantages at low bitrates (lets say below 1100 Kbit), at bitrates above that value HC offers better results. That is why I use HC more often now. So HC^n would be really a great improvement in speed and could be THE recommended encoder for Diko.

[Darksoul71]

@VM: I agree with NathanX. Supporting QuEnc^n as well as HCEnc^n would be a definite plus. Both encoders have their strengths and their weaknesses. HCEnc doesn´t produce bitrate spikes, keeps more details but might not be the optimal encoder for ultra-low bitrate encodes. QuEnc is IMO vise versa. Both of my multicore frontends are CLI compatible with the corresponding encoder and do not require any adoption in DIKO other than launching a separate exe on a dualcore / multicore machine when detected.

BTW: Does CQ prediction within DIKO work well with HCEnc ?

@NathanX: I agree ! QuEnc seems to "soften" more at lower bitrates although I did a KCVD (352x576) one CD version of Punisher using your General Purpose LS script (I really begin to like this ) at 730 kBit/s average with HCEnc and the result looked really good. Unfortunately I didn´t have much time since I only checked the M2V prior heading to work. I´ll do a SVCD and play it at my SAP this evening. Frankly I don´t expect any surprises since my 19" TFT reveals any compression artefacts very well whereas my big 4:3 TV (No, I haven´t got money to buy a big fancy 16:9 TV ) normally smoothes out a lot.

In general I think ist wiser to support QuEnc^n and HCEnc^n. So the user can decided which encoder suits his viewing habbits and personal preferences. For example I like a softer picture whereas a good friend of mine always goes for maximum sharpness and would never encode below 704x576.

Requirements vary from user to user and I feel that the frontend shouldn´t limit the user here if possible.

Two final comments:
1) Keep in mind that the available CPU time is the bottleneck for AVISynth as well as for the encoder(s). In other word: The more CPU power your need for your AVS, the less CPU power is available for the encoder(s). I could even imagine that a single encoder assigned to one core and a complex CPU-hungry script assigned to another core could run faster than running two encoder instances with two segments for the complex script. Neverthelesse from my observation QuEnc^n behaves quite linear: Two instances -> doubled speed compared to a single instance (under optimal conditions).  This is only true on a dual-core machine. Hyperthreading machines should behave different here.

2) The QuEnc and the HCEnc encoder templates for DIKO need to be verified. I didn´t do a lot of testing yesterday evening (since it was late) but HCEnc^n crashed when being called from DIKO (Free 2.30). A quick check of the template showed that there was a value (from TMPEGEnc) stored for progressive. May be a cut, copy, paste error ?

I don´t know and have to check again.

Best regards,
D$

[Darksoul71]

Hi VM,

there is definitely a problem with the HCEnc template. I re-installed DIKO Free with Version 2.32.

Here is a part of the original HC.ini from DIKO Free:

Code:

[main]
...
PROGRESSIVE=MPEGVideoEncoder_VideoEncodeMode_Progressive
...

Look for the progressive part. When you look into TMPEGEnc.ini you find the following vallues:

Code:

PROGRESSIVE=MPEGVideoEncoder_VideoEncodeMode_Progressive
INTERLACED=MPEGVideoEncoder_VideoEncodeMode_Interlaced

Looks pretty much the same as for HCEnc, doesn´t it ?   

I removed the Progressive settings in HC.ini it worked but there is still either a problem with the template or may be the approach you replace those placeholders in the aa2 file. I show you what I mean. This is the HC.aa2:

Code:

*wait  0
*profile       best
*MAXBITRATE %%MAXBITRATE%%
*OUTFILE %%N%%.mpv
*INFILE %%N%%.avs
*BITRATE %%BITRATE%%
*aspect        %%ASPECT%%
*gop           %%GOP_LENGTH%% 2
%%PROGRESSIVE%%
%%TOPFIELDFIRST%%
*closedgops
%%MODE%% %%QFACTOR%%
*dc_prec       9

*custommatrix
%%MATRIX_INTRA%%

%%MATRIX_INTER%%

Here is the ini which came out generated by DIKO for a progressive Divx:

Code:

*wait  0
*profile       best
*MAXBITRATE 8000
*OUTFILE D:\DIKO Working\movie0.mpv
*INFILE D:\DIKO Working\movie0.avs
*BITRATE 3032
*aspect        4:3
*gop           15 2

*TFF
*closedgops
 1.000000
*dc_prec       9

Leaving out *INTERLACE for a progressive video is fine for HCEnc since there is no special progressive parameter. What makes me wonder is the *TFF value. Why does DIKO add a value for the field order when the video is specified progressive ?
What also makes me wonder is the 1.0000 below *closedgops. Is this based on the CQ encoding part ?

Also quite important (as well as for CCE) is the scanmethod, in other words the way the encoder evaluate the image.
For progressive video this should be set to zigzag (*SCANMETHOD ZIGZAG) whereas it should be alternating (*SCANMETHOD *ALT) for interlaced. This parameter is not required but I would rather set it to the "optimal" value matching the source video.

What do you think ?

Best regards,
D$

[NathanX]

@VM: I agree with NathanX. Supporting QuEnc^n as well as HCEnc^n would be a definite plus. Both encoders have their strengths and their weaknesses. HCEnc doesn´t produce bitrate spikes, keeps more details but might not be the optimal encoder for ultra-low bitrate encodes. QuEnc is IMO vise versa. Both of my multicore frontends are CLI compatible with the corresponding encoder and do not require any adoption in DIKO other than launching a separate exe on a dualcore / multicore machine when detected.

BTW: Does CQ prediction within DIKO work well with HCEnc ?

@NathanX: I agree ! QuEnc seems to "soften" more at lower bitrates although I did a KCVD (352x576) one CD version of Punisher using your General Purpose LS script (I really begin to like this ) at 730 kBit/s average with HCEnc and the result looked really good. Unfortunately I didn´t have much time since I only checked the M2V prior heading to work. I´ll do a SVCD and play it at my SAP this evening. Frankly I don´t expect any surprises since my 19" TFT reveals any compression artefacts very well whereas my big 4:3 TV (No, I haven´t got money to buy a big fancy 16:9 TV ) normally smoothes out a lot.

In general I think ist wiser to support QuEnc^n and HCEnc^n. So the user can decided which encoder suits his viewing habbits and personal preferences. For example I like a softer picture whereas a good friend of mine always goes for maximum sharpness and would never encode below 704x576.

Best regards,
D$

Nice to hear that you like the LS script :-) 352x576 is a pretty low resolution as well as 730 Kbit/s is a low bitrate. I would expect QuEnc in 2 pass mode to achieve an even better result than HC does due to the used matrix (the QLB matrix is perfect for slower motion and darker colours!).

Regarding the support of Quenc^n and HCEnc^n I 100% agree with you. A DIKO supporting multicore-CPUs in the best possible way AND with free encoders would be a real benefit to my eyes.

Greetings
NathanX

[vmesquita]

Both of my multicore frontends are CLI compatible with the corresponding encoder and do not require any adoption in DIKO other than launching a separate exe on a dualcore / multicore machine when detected.

Yes that's the approach I was thinking of.

BTW: Does CQ prediction within DIKO work well with HCEnc ?

It works, but sometimes it can take a while to converge, the method currently is more optimized for CCE.

In general I think ist wiser to support QuEnc^n and HCEnc^n. So the user can decided which encoder suits his viewing habbits and personal preferences. For example I like a softer picture whereas a good friend of mine always goes for maximum sharpness and would never encode below 704x576.

I agree. I need to implement "memory" of the executable path in templates so I can bundle both encoders in DIKO package (plus FreeEnc for those who don't want to upgrade) and allow the user to easily switch between them.

1) Keep in mind that the available CPU time is the bottleneck for AVISynth as well as for the encoder(s). In other word: The more CPU power your need for your AVS, the less CPU power is available for the encoder(s).

I agree with this, but tweaking scripts and encoder for a multicore system can be a complicated job. I think that only being able to multithread the encoder will produce a speed increase which will be much more signicant than finding the optimal combination...

2) The QuEnc and the HCEnc encoder templates for DIKO need to be verified.

Yes definatelly. Most templates can be outdated due to the updating of the encoders adding new functions and changing older ones. Plus only the CCE and FreeEnc templates have been extensivelly tested.

there is definitely a problem with the HCEnc template. I re-installed DIKO Free with Version 2.32.

Here is a part of the original HC.ini from DIKO Free:

Code:

[main]
...
PROGRESSIVE=MPEGVideoEncoder_VideoEncodeMode_Progressive
...

You're right, I probably adapted Tmpgenc template and forgot to change this.

Leaving out *INTERLACE for a progressive video is fine for HCEnc since there is no special progressive parameter. What makes me wonder is the *TFF value. Why does DIKO add a value for the field order when the video is specified progressive ?
What also makes me wonder is the 1.0000 below *closedgops. Is this based on the CQ encoding part ?

DIKO always adds the TFF value no matter if it's progressive or interlaced... But as far as I know it doesn't cause issues in any encoder.
The 1.000 is the CQ, DIKO also always adds this parameters even if it's not a CQ based encode. But as I see this may cause problems with HC, or not?

Also quite important (as well as for CCE) is the scanmethod, in other words the way the encoder evaluate the image.
For progressive video this should be set to zigzag (*SCANMETHOD ZIGZAG) whereas it should be alternating (*SCANMETHOD *ALT) for interlaced. This parameter is not required but I would rather set it to the "optimal" value matching the source video.

You're right. This parameter is currently not implemented in DIKO encoder template interface, but it's certainly missing (I forgot about this). I need to add this.

[vmesquita]

Ok, regarding HC Preditcion: I found why it was taking a long time to converge. The template was configured to search values in all range possible. This is not desirable: we want only the values that yield good quality (like CCE QMax=40), otherwise it's a waste of time to use scripts and good quality encoders. I did a little test and found that a size similar to CCE QMax=40 is obtained with HC QMax=6. So I tweaked HC template and voi lá: prediction now converges sometimes in 1 or 2 cycles. Much better 

Edit: There was a bug in fine tune routines for encoders with fractional Q (such as HC). I fixed for the next release, and changed the routine so it skips the fine tune process when the 5x sample comes out OK (faster Q search!)