having a hard time constraining bitrate.

May 21, 2013 at 5:22 PM
hey guys,

i've been using mcebuddy though it's various and improving versions for at least a year now with zero issues... mostly it's been converting cartoons for the kids and has been pretty much flawless...

lately i've tried to make it do a little more and it seems like i'm running into an unexpected problem...

currently i'm trying to convert a bunch of mkv's into mp4's... the conversion goes fine, but the bitrate of the output file seems to be all over the place.... i've tried a couple of profiles - high mp4, and high m4v (both 2 pass) with similar results... i even tried a modified m4v profile where i manually set the bitrate to 1500, but that appeared to have no effect...

basically, my output files come out at a bitrate much higher then specified.... for example, high m4v has a default bitrate of 2000, but my files come out at an arbitrary bitrate of anywhere between 3000-4000, one file came in almost at 5.... manually setting the bitrate higher/lower seems to make no difference....

i do not have these issues when converting wtv files, so all the converted wtv comes out at whatever bitrate i specify... i'm guessing either mkv's are treated differently, or i have something setup wrong...

please help, and thank you :)
Coordinator
May 22, 2013 at 12:52 AM
Please upload your conversion log that will tell the story of what bitrate is being used and where it is changing

May 22, 2013 at 1:56 PM
just in case, i reran the same job on a different machine... i recently reinstalled shark's codec pack on my main htpc where i originally ran the job, so i wanted to make sure that i wasn't seeing this problem due to some wacky codec conflict or some other nonsense like that...

nope, same results...

https://dl.dropboxusercontent.com/u/6317570/mcebuddy/Dexter%20-S03E03-%20The%20Lion%20Sleeps%20Tonight%20%281080p.BluRay%29.mkv-mkv%20test%20again-2013-05-22T00-48-10.0494550-04-00.log

https://dl.dropboxusercontent.com/u/6317570/mcebuddy/Dexter%20-S03E04-%20All%20In%20the%20Family%20%281080p.BluRay%29.mkv-mkv%20test%20again-2013-05-22T02-41-20.9718736-04-00.log
Coordinator
May 22, 2013 at 3:34 PM
The bitrates are increasing because MCEBuddy profile bitrates are optimized for 720 width.
When you increase the width in the conversion task:

Max Width -> 1280


MCEBuddy compensates by increasing the bitrate (inthis case to 3550). So if you want a lower bitrate then reduce the bitrate in the profile OR you can specify FixedResolution=true in the profile, in which case MCEBuddy won't change the original resolution or bitrate.
Coordinator
May 22, 2013 at 4:20 PM
I forgot, the easiest option, reduce the quality slider in the conversion task and it'll reduce the bitrate. no other changes required
May 22, 2013 at 4:28 PM
ok, that makes sense... i'm guessing this mostly applies to mkv's, since i've had no issues locking bitrate when converting wtv's at 1280...

i would be curious if you could help me with a work around or sorts....

here is what i'm trying to accomplish...

i have a large pool of drives serving as tv storage.... the pool totals just under 11tb, and i have just over 10tb of data there so i'm running out of space... a huge chunk of that space is being taken up by 1080p mkv's... i'm trying to free up some space by shrinking/converting them to mp4....

currently the tool i use for this is called HBBatchencoder... it's a very small, simple program that serves as a gui to trigger the handbrake cli to do the batch conversions... it does what i want it to do, and does it relatively well, but i would like to have mcebuddy take over that role... the problem with HBBatchencoder is that unlike mcebuddy it's "not smart" so there is no logic that you can use to specify the files you want to target - it just takes anything with an mkv extentiion and processes it... but the bigger problem is that it doesn't "play nice"... there always seems to be a point where it runs away with the resources and makes the pc so sluggish it's virtually useless... mcebuddy, by comparison, has been running on that same pc and i can't even tell it's there... whatever you did to make sure that mcebuddy doesn't hog resources works, and works very well...

so again, my goal is to have mcebuddy take over from HBBatchencoder... so far i have run into two issues when trying to do that....

1 - i cannot control the output bitrate when converting to 720p
2 - mcebuddy won't even start the conversion and service crashes if i point it at the main pool... 10tb is a lot of data and there are a lot of files, and i think that the sheer volume simply overwhelms it...

i am not sure if there is a workaround for number 2, but i think there maybe a workaround for number 1...

currently i use a modified appletv2 preset with HBBatchencoder and it uses this string
-f mp4 -4 - T -X 1280 --loose-anamorphic -e x264 -b 2000 -2 -r 29.97 --pfr -a 1,1 -E faac,copy:ac3 -6 dpl2,auto -R Auto,Auto -B 160,auto -D 0.0,0.0 -m --verbose=1 -O -O

i figure there has to be a way for create a profile that would skip most of the conversion logic reserved for wtv (remuxing etc) and just trigger the cli with these parameters... this would still allow me to use mcebuddy's filtering options to trigger only specific files/shows, and will allow me to dump HBBatchencoder for good...

i know this is not what mcebuddy was created for, so i will understand if you can't answer me....
Coordinator
May 22, 2013 at 11:28 PM
so first up, this logic applies to ALL profiles that are bitrate based. WTV profiles are quality based so it does not affect them (-b vs -qscale)

Coming to you points, it is veyr much possible to do what you're doing and MCEBuddy has scaled upto 40TB according to another user so there is no issue in that. In one of the earlier version there was a bug where the number of sub directories caused the GUI to crash which has since been fixed and I've tested it with upto 150,000 entries.

Regarding point 1, yes it can be done but you will need to create your own custom profile to do so. My question however is why are you using MP4 and not MKV? MKV is a great format. Look at the sample profile in profiles.conf like VCD which have fixed bitrates and resolutions and will not vary. Use that as a base to create your own custom profile, you can use the encoder of your choice. After you've created it you can post it here and I will validate the same for you.

Regarding point 2, that's VERY surprising, What version of MCEBuddy are you using? Can you post MCEBUDDY.LOG here for me to analyze. I don't see any reason why it cannot the process the files.


May 23, 2013 at 2:58 AM
ok, so i tried a couple of experiments tonight...

fist i tried i set the watch directory to the main tv pool, and mcebuddy hung with "service unavailable"... i checked the log and i saw that there was an error... it looks like between all the nested folder names and the filenames, some files had more then 256 characters (or whatever the max is) in the name...

i then went down a level in the folder tree and mcebuddy hung with service unavailable again... i got the log and actually started typing here, and was about good five to six paragraphs deep when my htpc screen flickered and suddenly mcebuddy came back to life and started to display the list of queued files... it hasn't started the conversions yet, but at least it's finding the files... then it hung again...

i rechecked the log right now and i do see that it's being populated, so something is happening and it looks like despite the "service unavailable" mce buddy is working.... it may be struggling a bit, but it's working :)

i'll let it run a little longer and see what happens...

as far as mp4's...

i agree with you that mkv is a more modern, and probably a better container, but i still prefer mp4 for a variety of reasons.... there are some minor reasons, like the fact that mp4's tend to stream much better, especially to extenders or using the Plex/Roku combo, but the biggest reason i use mp4's is because they are completely headache free when it comes to compatibility....

mp4's just work, and they work on anything and everything without the need to install any apps or codecs/plugins etc... you want to stick a thumb drive in your tiv - it works.... you want to play them on a dvd player made when hammer pants were the all the rage - it works... there pretty much isn't a device out there that doesn't play this format natively.... i have two small kids that travel between grandparents a lot, so having things that "just work" is a big deal for me... saves me lots of headaches :)

i find handbrake's appletv2 preset to be ideal for me, even though i don't have or even ever seen an appletv... the preset automatically limits the max resolution to 720p, which is where i want to be for backups, and it creates dual audio tracks - AAC and AC3 and does it well even if the source has DTS... having dual audio tracks does bump the file size a bit, but it also ensures that older devices will play the file just fine (using the aac track) and devices that support AC3 will play that track... i am yet to run into a device that couldn't play the mp4's i converted using this preset, so i'm trying to keep as much of my library in that format as possible...

i'll take your advice and start looking through sample profiles to see if i can figure out a way to frankenstein one of my own... let's see what happens :)

thank you very much for your help so far, rboy... i'll post with some updates soon...
Coordinator
May 23, 2013 at 1:24 PM
You haven't told me which build you're using and MOST importantly you haven't posted the mcebuddy.log file, I need to see the log file to see where and what is causing issue. Will be unable to help without the actual mcebuddy.log


Coordinator
May 23, 2013 at 1:59 PM
FRom what you're mentioning, the GUI flickering, sounds like an old version of MCEBuddy.
May 23, 2013 at 4:19 PM
Edited May 23, 2013 at 4:20 PM
my apologies, i forgot to attach the log - https://dl.dropboxusercontent.com/u/6317570/mcebuddy/mcebuddy.log

i'm using 2.3.12

the word "flicker" is not really accurate to describe what i was seeing, my apologies for using it.... the gui was hanging with the "service unavailable", but then would come back to life - sort of... basically, it would be about 5-10 (much longer than that the first time) minutes of hang, and then it would pop up the queued files list for a minute, then hang again.... rinse/repeat.... this kept going for a while... even when the gui did show the files, there was no progress bar or any other indication to show that an actual conversion was taking place, but it turns out there was.... some files have completed successfully even though the gui looked like mcebuddy was hanging or nothing was happening etc...

so it works, i'm happy :)

i had much less luck with profiles, though... actually, let me rephrase that - i had great luck, and could manipulate the handbrake cli pretty much any which way i wanted to when the width was set at 720... however, once i upped it to 1280 my luck ran out... i simply could not get mcebuddy to lock on to the set bitrate no matter what i tried.... you mentioned earlier that mcebuddy is setup for 720 and processes 1280 differently, so i'm guessing my luck had something to do with that and that i won't be able to fix this problem.... so i gave up - for now...

i'm glad that i found out that mcebuddy can handle a large pool of files, but that alone was half the battle for me and it wasn't even the biggest half... the fact that i can't lock the bitrate once i up the width kind of defeats the purpose of scanning that large pool to begin with... i know you mentioned using the quality slider instead, but i'm after a consistent filesize and that's something that will be very tricky to achieve by playing with quality...

i guess i'll have to stick to using HBbatchencoder, or look for some other tool to handle this particular task.... that said - i absolutely love mcebuddy... i already mentioned that it's been running problem free (through various versions) on my htpc for well over a year now... no hickups, no issues, i don't even remember that it's there sometimes... i don't have the skill or the knowledge to make it do everything i want i to do, but the things it's meant to do it does very well, and i am very happy with it... so happy i just donated :)
Coordinator
May 23, 2013 at 11:08 PM
1. I don't see any crashes in the log file
2. I noticed you're using an old version of MCEBuddy 2.3.12 which has bugs in it for handling large volumes of files and data which makes the GUI unresponsive and engine slow. There is a new architecture in 2.3.13 for handling large volume which makes the GUI more responsive and engine faster. I would suggest upgrade to the latest BETA and it would solve your problem.

Coming to the profie, I don't understand what you're trying to do, here is a simple profile that comes with MCEbuddy with a constant bitrate and fixed resolution. You can customize the same for your own requirements. this one is ffmepg based with 960 width, auto cropping and 2500k bitrate. Similarly you write a custom profile using handbrake (you will need to know how to write handbrake paramters but it looks like you do) and you're good to go. Remove the 2ChannelAudio if you want multi channel audio, change the aac to ac3 codec (see other profile samples).

[iPhone Hi-Res]
Description=iPhone 4,4S,5 compatible (H.264 MP4 width 960 pixel video and AAC stereo audio).
order=ffmpeg
ffmpeg-general=-threads 0
ffmpeg-video=-ss 3 -vf yadif=0:-1,hqdn3d,crop=iw:ih:0:0,scale=min(960\,iw):trunc(ow/dar/2)*2 -vcodec libx264 -b 2500k -x264opts ref=6:subq=6:subme=7:qcomp=0.8:b-pyramid=strict:no-weightb=0:me=umh:direct=auto:level=3:partitions=p8x8i4x4:no-cabac=1:trellis=1:no-dct-decimate=1:threads=auto
ffmpeg-audio=-acodec aac -ab 160k -strict experimental -cutoff 15000
ffmpeg-audioac3=-acodec aac -ab 160k -strict experimental -cutoff 15000
ffmpeg-ext=.mp4
ffmpeg-audiodelay=skip
2ChannelAudio=true
FixedResolution=true
PreConversionCommercialRemover=true


May 24, 2013 at 5:33 AM
yes, you're right - there was no crash.... the gui was unresponsive and was hanging with "service unavailable" for a long time, which is unusual... typically, mcebuddy springs right up and just starts pumping.... so when it hangs like that for a while it makes it look like it's crashed, or just not functioning.... this is what made me think it wasn't working....

as far as profiles, i'm trying to do a very simple thing - take some 1080p mkv's and convert them to 720p mp4's.... nothing overly complicated, and nothing that mcebuddy shouldn't be able to handle.... i've been using handbrake for a while and i kind of learned how to manipulate it to my needs, so i tend to use handbrake based profiles... usually i just modify one of the preset apple based profiles that comes with mcebuddy to suit my needs....

i seem to have zero issues manipulating the handbrake profiles when the width is set at 720.... each and every setting i change reflects in the output, so any changes to video/audio bitrate and pretty much any other setting that can be manipulated via the cli works just fine... that is, again, as long as the width is 720....

once i set the width to 1280 (720p), things go south... the bitrate i set is no longer recognized... so if i set the file to convert at -b 2000 it should convert at 2000, instead i got anywhere from 3000-4500.... this makes it very difficult to control output size....

for example, here is the profile i just used on this conversion - https://dl.dropboxusercontent.com/u/6317570/mcebuddy/Dexter%20-S01E06-%20Return%20to%20Sender%20%281080p.BluRay%29.mkv-mkv%20test%20again-2013-05-23T21-58-54.5440851-04-00.log

[M4V Apple High Quality]
Description=Apple compatible 2 pass MP4 (H.264/AAC) high quality conversion. 4GB file size limit. More sensitive to errors in the source video.
order=handbrake
2pass=true
handbrake-general=--decomb --denoise="weak" --loose-anamorphic --verbose=2 -T -f mp4
handbrake-video=--start-at duration:3 -e x264 -b 1600 -x cabac=0:ref=2:bframes=0:weightp=0:8x8dct=0:trellis=0:subq=6:me=hex
handbrake-audio=-E faac -6 auto -R auto -B 160 -D 0
handbrake-audioac3=-E faac -6 auto -R auto -B 160 -D 0
handbrake-ext=.m4v
handbrake-audiodelay=skip
PreConversionCommercialRemover=true
FixedResolution=true

it's basically the stock profile that came with the 2.3.13beta with one line that you suggested added at the end... the bitrate in this profile is set to 1600... the output file is at 2800+, nowhere close... other files, converted with 2.3.12 and similar settings converted at much higher bitrates often reaching into 4000's and creating a file size much larger then what i'm after... again, at width 720, everything is ok... at 1280, the bitrate setting just seem to be ignored...

this is what i'm struggling to work around right now.... i tried a number of variations of this profile with a number of different settings and the same pattern repeats... zero issues on 720, no luck at 1280....

i know mcebuddy was not build to be a batch mkv to mp4 converter, so the fact that i can't make it do what i want is not that big of a deal... it would be nice if i could dump HBbatchencoder simply so i have one less tool that needs to run on my htpc, but again - it's ok.... as long as mcebuddy keeps crunching my wtv recordings flawlessly, like it's been doing, i am happy :)
Coordinator
May 24, 2013 at 6:44 PM
Edited May 24, 2013 at 6:45 PM
The bitrate is changing is you haven't set the size in teh profile. FixedResolution just tells MCEBuddy not to limit the resolution but it will compensate for changes in resolution by adjusting the bitrate to maintain quality (otherwise you may have a video at 1080p trying to encode at 1000kbps which will be disaster).

The answer is to add the resolution to your profile, so your profile will look like (note the -w 1280):
[M4V Apple High Quality] 
Description=Apple compatible 2 pass MP4 (H.264/AAC) high quality conversion with 1280 width. 4GB file size limit. More sensitive to errors in the source video. 
order=handbrake 
2pass=true 
handbrake-general=--decomb --denoise="weak" --loose-anamorphic --verbose=2 -T -f mp4 
handbrake-video=--start-at duration:3 -e x264 -b 1600 -x cabac=0:ref=2:bframes=0:weightp=0:8x8dct=0:trellis=0:subq=6:me=hex -w 1280
handbrake-audio=-E faac -6 auto -R auto -B 160 -D 0 
handbrake-audioac3=-E faac -6 auto -R auto -B 160 -D 0 
handbrake-ext=.m4v 
handbrake-audiodelay=skip 
PreConversionCommercialRemover=true 
FixedResolution=true 
However what I will do is that in the next BETA release I will change the logic so that if the user has specified FixedResolution it will also fix the bitrate and not adjust it. If the user wants to use the original resolution he can increase teh max width slider in teh conversion task page