HOWTO conveniently convert videos using mencoder, x264

[fangorn]

Not a bad idea.

But I see two problems:

1. Each user will have different feeling what is "relevant information" and for sure have
   different ways to express them
2. The "ask the user for the unrecognized tokens" approach does not match with my principles
   of "no questions asked" and "just do the right thing"

[Akovia]

Thanks for the reply fanghorn.

Each user will have different feeling what is "relevant information" and for sure have
different ways to express them

I agree when it comes to mp4, as it has many fields that are supported across many clients. With mkv, I've only seen players that support the title tag, which is the most relevant in each container anyway. I also recognize that these scripts aren't really the right approach to detailed tagging for any container, but being able to set the title on conversion would cover the most important while being very convenient. I would hazard to guess that most people are only setting the title field anyway and this would be a great way to do that. As far as supporting other metadata fields, I knew it would be great for my needs and figured if you don't ask it will never happen. =D

The "ask the user for the unrecognized tokens" approach does not match with my principles
of "no questions asked" and "just do the right thing"

Personally I am just fine with this approach, which is why I made the suggestion of a fallback to use the filename as title if a match isn't found. I was just trying to give suggestions for usability of users not quite comfortable with regex, and avoid the inevitable of converting a 4gb folder full of files that don't get tagged they way you thought. I would be fine with just an echo as it's processing, (or into the avi2mkv_command.log) so you could Ctl+C if it's not what you expected.


On another note, as I stated I am still quite new to your scripts and I've run into a snag. For some reason I am unable to batch process of all things. Everything I try only processes the first file then quits.

Code: Select all

avi2mkv *.avi

I've even tried xargs, but nothing seems to work. Any suggestions?

[fangorn]

Yes batch processing multiple files is in the queue for menc at the moment.
Including running multiple jobs in parallel to max. out serious hardware.

avi2mkv at the moment does not support multiple input files.

But you can always let the shell do the job for you. (as long as you are not on Windows, that is)

Code: Select all

for i in * ; do avi2mkv "$i" ; done

Take care of the syntax. And only use avi2mkv options that you want to apply to all files.

[Akovia]

Yes batch processing multiple files is in the queue for menc at the moment.
Including running multiple jobs in parallel to max. out serious hardware.

avi2mkv at the moment does not support multiple input files.

But you can always let the shell do the job for you. (as long as you are not on Windows, that is)

Code: Select all

for i in * ; do avi2mkv "$i" ; done

Take care of the syntax. And only use avi2mkv options that you want to apply to all files.

Thank you fangorn. I finally got a chance to try it out and it works great. Just what I needed.

[Akovia]

Well I ran into some trouble now =/

Tried to batch convert a folder of ogm files, and a few ended up without audio. (Glad I checked them) I can't tell if it's a script problem or not, but I can ogmdemux manually and re-mux with mkvmergeGUI without any trouble.

I captured the terminal output, but it seems to have conflicting information that I can't decipher.

Here is part of the output:

Code: Select all

$ avi2mkv Seikai\ no\ Monshou\ -\ 06.ogm
using Seikai no Monshou - 06.ogm as source
video stream is 1
audio streams are 0
Movie Seikai no Monshou - 06.ogm with title Seikai no Monshou - 06 has a geometry of 640 x 480 and a framerate of 23.976
Video stream is 1, audio stream(s) are 1 

And the logfile:

Code: Select all

Logfile created at 2012-03-27 19:37:56
dumping necessary information listings
/usr/bin/mkvmerge  -i "Seikai no Monshou - 06.ogm"  2>&1
/usr/bin/mplayer "Seikai no Monshou - 06.ogm" -identify -frames 0 -vc null -vo null -ao null -nocache 2>&1
avi2mkv_audio_test_codec 1 Seikai no Monshou - 06.ogm
avi2mkv_audio_test_codec 1 Seikai no Monshou - 06.ogm -aid 0

/usr/bin/mkvmerge  --output "Seikai no Monshou - 06.mkv"  -v --title "Seikai no Monshou - 06"    --default-duration 0:24000/1001fps   --default-track 1:yes --forced-track 1:no  --language 1:eng --default-track 1:yes --forced-track 1:no  -a 1 -d 1 "Seikai no Monshou - 06.ogm"   --track-order 0:0,1:0

I can post the rest of the terminal output if you wish, just leery of over spamming code if not asked to.
Any thoughts?

Edit: Also wanted to ask if there was a way to have the script exit on a fail?

[fangorn]

I did not test ogm that much. But the command line you posted uses a
track order that needs external audio file. If avi2mkv did not extract the
audio to an extra file there is something wrong.

As A shot from the hip I would replace 0.1,1.0 by 0.1,0.0

Why it does not use this I have to test.

[Akovia]

The command line I used was

Code: Select all

for i in *.ogm ; do avi2mkv "$i" ; done

I didn't insert any extra options.
After doing some more research I did stumble upon something that may be related. I ran ffprobe on all the files and noticed a pattern.
On the files that muxed correctly, ffprobe shows 2 errors/warnings.

Code: Select all

[ogg @ 0x9a88b80] Page at 202 is missing granule
[vorbis @ 0x9a8f7a0] Extradata missing.

On the ones it skipped muxing the audio I only get a single error/warning

Code: Select all

[ogg @ 0xb03bb80] Page at 143 is missing granule

The hex addresses vary per file, but the errors are identical otherwise. I did a google search on the errors and found a reference to a ffmpeg bug
http://www.mail-archive.com/ffmpeg-issu ... 08585.html
but I can't see how it would be related. I was thinking your script was just demuxing with ogmdemux, then re-muxing with mkvmerge which I am able to do manually without any problems. I don't see where ffmpeg would play any part.

I am using a freshly compiled version of ffmpeg from git less than 2 weeks old following the instructions here.
http://ubuntuforums.org/showpost.php?p= ... count=1289

As I said before, it may have nothing to do with your script, but I thought you should know. I would just like to figure out a way to know if the muxing failed somehow so I don't end up deleting my sources thinking they were converted properly. If there is any way I can help, just let me know.

Thanks

[fangorn]

The error is a little bit more complex. I cannot reproduce.

I do not use OGM movies and just created one using avi2mkv. It does not show such errors.

avi2mkv uses ffmpeg only for media type detection (simple audio streams).

That is most likely not the problem.

When writing MKV it uses mkvmerge, which can thanks to Moritz Bunkus read OGM
directly. ffmpeg would only be part of the process if you had external audio (a second
audio stream or replacement audio) to identify the audio format. For the rest of the
detection process it uses mplayer. If you wanted to convert the audio streams before
muxing the process would get a little bit more complex.

As I cannot reproduce: How were the OGM files created? What formats do video
and audio streams have (mediainfo can give a good list). As you talked of a
ffmpeg bug, were they written by ffmpeg? Do you know the command line that
created them?

Edit: Do you use the default settings for avi2mkv (in /home/<username>/.fangorn/config)?

[Akovia]

I do not use OGM movies and just created one using avi2mkv. It does not show such errors.

I'm sure there is something about the files I'm using that is the problem. They are many years old and I have no idea what the encode settings were when it was created.

avi2mkv uses ffmpeg only for media type detection (simple audio streams).

If ffmpeg is being used to detect audio streams and throws that error, might that not explain why it excludes it?

When writing MKV it uses mkvmerge, which can thanks to Moritz Bunkus read OGM
directly. ffmpeg would only be part of the process if you had external audio (a second
audio stream or replacement audio) to identify the audio format. For the rest of the
detection process it uses mplayer. If you wanted to convert the audio streams before
muxing the process would get a little bit more complex.

I didn't realize that mkvmerge could handle OGM directly. After reading this I tried to mux the offending files directly by dropping them into mkvmergeGUI without demuxing first and it worked fine. The only noticeable hitch was that it puts the audio as the first track instead of the video.
It also says here that ffmpeg is only used when there are external audio files. Am I missing something from that statement above where you say it's used for media detection?

What formats do video
and audio streams have (mediainfo can give a good list).

Code: Select all

General
Complete name                    : /Seikai no Monshou - 06.ogm
Format                           : OGG
File size                        : 211 MiB
Duration                         : 25mn 5s
Overall bit rate                 : 1 177 Kbps

Video
ID                               : 0 (0x0)
Format                           : MPEG-4 Visual
Format profile                   : Advanced Simple@L3
Format settings, BVOP            : Yes
Format settings, QPel            : Yes
Format settings, GMC             : No warppoints
Format settings, Matrix          : Default (H.263)
Codec ID                         : XVID
Codec ID/Hint                    : XviD
Duration                         : 25mn 5s
Bit rate                         : 1 010 Kbps
Width                            : 640 pixels
Height                           : 480 pixels
Display aspect ratio             : 4:3
Frame rate                       : 23.976 fps
Resolution                       : 8 bits
Colorimetry                      : 4:2:0
Scan type                        : Progressive
Bits/(Pixel*Frame)               : 0.137
Stream size                      : 181 MiB (86%)
Writing library                  : XviD 0.0.09 (UTC 2003-03-25)

Audio
ID                               : 1 (0x1)
Format                           : Vorbis
Format settings, Floor           : 1
Duration                         : 25mn 5s
Bit rate mode                    : Constant
Bit rate                         : 112 Kbps
Channel(s)                       : 2 channels
Sampling rate                    : 48.0 KHz
Stream size                      : 20.1 MiB (10%)
Writing library                  : libVorbis 1.0 (UTC 2002-07-17)

Do you use the default settings for avi2mkv (in /home/<username>/.fangorn/config)?

The only change I made was to the language section.

I really doubt that your script is the cause of the problem here. If anything I'd bet it's one of the tools being used. Maybe this isn't worth troubleshooting much further. The only real issue I see is not getting any feedback when a stream fails to mux properly. This obviously hasn't been a problem in the past, so this might an isolated case. Either way I wanted you to be aware that it can happen.

I appreciate the time you've taken to help me troubleshoot this but I certainly don't expect you to take it any further unless you want to out of curiosity. If so I would be happy to make the source file available to you on my file server. Otherwise, I'll just do my OGM files manually one at a time with mkvmergeGUI. If I run into any other snags with my AVI files I'll be sure to let you know.

Many Thanks,
Ako

[fangorn]

Sync problem from .ts source files

mencoder seems to have a serious problem with MPEG Transport Stream (.ts) source files.
The audio sync is lost completely (due to an increase in video stream playtime).

Workaround: mux the .ts source files to .mkv using avi2mkv and run menc on the .mkv.
This seems to get around this problem.


I am still in the process of developing some new features (processing multiple inputs in
parallel in menc and riptoh264, bluray-info to optionally feed avi2mkv and output
anything avi2mkv can directly, ...)

I did not test anything of this on anything but gentoo so far and testing is pretty slow
when you test encoding three 1080p fulltime movies in parallel. So it will be
some time until these features hit the streets.

[fangorn]

Hi,

after a long break some new features for my scripts.

menc and riptoh264 can now process lists of input files, and even can process more than one input in parallel to utilize faster machines fully using the –jobs <num> option.

menc can use about any input and call riptoh264 with your configuration for DVD directories/ISO images or bluray-info with your configuration for BluRay directories. Lists of input can be generated by shell expansion on Linux or by menc on Windows. Optionally all subdirectories are searched for input recursively.

avi2mkv can now process multiple inputfiles, one after the other. Wildcards are usable on UNIX/Linux as the shell will process them.

bluray-info now calls avi2mkv at the end if an output container different from TS is requested.

menc and riptoh264 are now capable to convert only parts of the input source. If chapter information is available (internally or in an external file) you can specify chapter ranges. Or you can specify start and/or end time.

Also there are more little improvements.

Have fun with the tools. And if there are still some problems in the code, don’t hesitate to Email me.

Further improvements for avi2mkv are in progress now, but will take a while.

fangorn

[wannabegeekster]

So how do I use these program's together? Do I have to run the choose_video_parameters before I convert each video? The help of the application says that it is mandatory to specify the input video width and height. Not all videos have the same width and height.

Is it possible to convert all of my videos to 480p?

I ultimately want to do something like this and have all of the videos converted in one go:

find /path -size +800M -exec menc {} \;

While I do enjoy the ability to run something like for f in *.avi; do menc $f; done. I don't really need that for the most part.

Thanks for the scripts! Can't wait to try them all out.

[fangorn]

You only think of the video type you want to convert.

For video conversion:
DVD -> riptoh264
BluRay -> bluray-info (maybe followed by menc)
everything else -> menc

For Container format changes and/or audio conversion:
avi2mkv

These programs call the others if needed. With exception of bluray-info that does at the moment not
call menc automatically for recompression of the HD video.

To convert to 480p you have three choices:

• --targetheight 480
  keeps the aspect ratio and forces 480 lines
• --targetheight 480 --expand_if_cropped
  forces a 16/9 video with black borders added if necessary
• --targetwidth 854 --targetheight 480
  forces an aspect ratio. with --crop you can remove black borders before scaling.

Your command line is absolutely possible. I would just set the necessary parameters on the command
line or in the configuration file (~/.fangorn/config).

One problem with this approach: menc does by intention not delete the original. With this method you
will double your movies on the harddisk and maybe will not find them again. Use the --outputpath to
redirect the converted movies to one directory. --tmppath will restrict the temporary files to a location
you can easily clean if something does not work as expected.

have fun,
fangorn

[fangorn]

Hi,

after about six months of development I have decided to do an open Beta of avi2mkv and menc version 4.0.0.

avi2mkv was rewritten next to completely and after that the feature list has tripled. Not counting all the
automatic glue that makes it convenient. I have tested the basic functionalities on my Gentoo box. The DVD
muxing capabililty aside they are working so far. But I cannot test all the combinations possible. So I will
be thankful for any bug reports.

The possibilities are next to endless now. So if you can't figure out how to do something, just ask. The documentation
most likely can be improved also.

There is a new installation process that is more convenient (and should work on more distribtuions). This
was necessary as the number of dependencies has also increased. You can find the new instructionns here: http://blog.fangornsrealm.eu/install/.

To give a hint what I am so excited about, here is the list of new features for avi2mkv-4.0.0:

Version 4.0.0
Major redesign of next to anything. Previous functions should still work, though.

General usage changes:
- accepts single audio streams as input now.
Conversion and output to either raw data stream or selected container is supported.
- accepts single subtitle stream as input now.
Conversion and output to SRT Text format, DVD VobSub format or BluRay SUP format is supported.
- accepts source files not in the working directory
- accepts wildcards as input even on Windows (parses all wildcards internally if not done by the shell)
- optionally specify a directory for temporary files using --tmpdir option
(Default: use current working directory)
CAUTION: Directory --tmpdir must not include whitespace!
Whitespace will be automatically replaced and the directory with _ instead is used!
In some constellations this leads to an unwritable temporary directory!
- optionally specify a directory for the output file using --outputdir option
(Default: use current working directory)
- optionally specify a filename for the output file (may include the output path) (
Default: get from input file)
- accepts file- and directory names now that include escaped whitespace (\ ) as done by UNIX
shells when not in ""

Container changes:
- automatically sends unprocessable input containers to menc and lets them process with default
configuration
DVD
- optionally get MPEG2 video in DVD format from menc with --dvdmux
muxes the input streams to a DVD data structure.
- optionally force nonanamorphic DVD when in DVD mode
Default: use anamorphic target video size when scaling
AVI
- muxing to AVI container now supports all audio options
MP4 and OGM
- MP4 and OGM muxing finally can reverse the order of the first two audio streams
ALL supported containers
- optionally concatenate movies to single output file
This requires identical parameters for video and audio!
So it is most useful when prepared by an menc run with the same parameters.
- optionally split the output container at a specified size
all audio options are supported
automatically uses only relevant chapter marks and corrects the time codes to the new movie length
automatically uses only relevant parts of subtitles and corrects the time codes to the new movie length
- automatically uses chapter information in files named <moviename>_chapter.txt (as written by menc, riptoh264 and bluray-info)
- automatically extracts chapter information from source if available and nothing specified externally
- automatically generates a chapter mark every five minutes if no chapter marks are available otherwise
- automatically uses <movie>.jp(e)g or <movie>.png as coverart when muxing to Matroka (MKV) or MP4.

Audio processing changes:
MPlayer is dropped from audio conversion for now. So it is no longer necessary to have all audio streams
in one source file. This also means the previously supported channel and pan mappings are not supported
any more - at least for the moment.
- optionally export audio streams to PCM WAV file in original or in CD Audio sampling.
- optionally extract the lossy core stream out of lossless DTS-HD (DTS) and Dolby TrueHD (AC3) audio streams
- use lavc encoder if oggenc is not available (internal OGG encoding requires a special version of ffmpeg)
- use lavc encoder for FLAC audio (removed the dependency on flac)
- automatically tests container format against supported audio formats and converts if necessary
- Internal audio streams now get handled different. By default all internal audio streams of a selected
language are used.
- optionally you can override this behaviour with the --use_one_audio_stream_per_language option
- optionally you can also deactivate the behaviour to only use as many internal audio languages as
are NOT specified as external audio stream by the --use_all_internal_audio (surplus external streams
get a default language setting then)

Subtitle changes:
- --subtitlelist now accepts an input file containing the list of subtitles (written by subtitle extraction routines)
- automatically uses a file <moviename>_subtitle_list.txt (written by subtitle extraction routines) if nothing else is specified
- automatically extracts subtitles from MKV, TS, OGM, MP4 sources if available and none of the other options matches
- now able to mux subtitles to TS container (SUP and SRT formats)
- now able to mux subtitles to OGM container (SRT format)
Text is written in UTF-8 format, so it is automatically converted to a local encoding table for ogmtools.
(ogmtools package necessary, so not available under Windows)
- now able to mux subtitles to MP4 container (VobSub and SRT formats)
- optionally convert subtitles to a format supported by the output container with --convert_subtitles
BluRay SUP can be converted to smaller DVD VobSub (sub/idx) format
DVD VobSub (sub/idx) format can be converted to SRT text format by OCR with tesseract or gocr
Text formats can be converted to the most supported SRT format.
In lack of good examples only part of the formats found are implemented.
Especially the more complex formats are not implementable without a real world example.
If you have a text subtitle in a format that is not (correctly) converted
contact me and provide an example. I will try to implement it.
SRT text format can be converted to VobSub (sub/idx) format
SRT subtitles get converted to SUP format (tsMuxeR function)
- optionally convert any subtitle formats to SRT format with --convert_any_to_srt
Default: convert only other text formats (plain text or XML) to SRT automatically
Warning: Conversion of SUP to VobSub and from VoSub to SRT takes a long time.
- optionally convert any subtitle format to DVD VobSub format
This is only supported for Matroska container.
Warning: This will take a long time and might even fail for some subtitles.
- optionally force conversion BluRay SUP subtitles to DVD VobSub with --convert_sup_to_sub
- optionally force conversion DVD VobSub to SRT text format with --convert_sub_to_srt
- optionally force conversion SRT text format to DVD VobSub with --convert_srt_to_sub

[_______0]

do these scripts have their full-version non-scripted commands?? I find those more interesting.

Let's say I do some conversion, do the script show the original ffmpeg (whatever uses) command on the console?

[fangorn]

Yes they do, but not on the console as it would pass by very quickly.
They write a log file that contains each and every command line to call an external tool.
This log file is deleted by default when the program run ends. You have to activate the
--debug option to keep the tools from deleting their temporary files/directories.

[fangorn]

Finally I found the time to create a full set of ebuilds.

http://progs.fangornsrealm.eu/install/ebuilds.tar.bz2

If you need any further instructions, go to the install page.

http://blog.fangornsrealm.eu/install


menc is out of beta in the meantime, as I managed to reduce the known bugs to zero.

riptoh264 has gotten some love also, after a long time.

avi2mkv has had some bugfixes and is really stable for "normal" usage. But a few of the
new features are still widely untested or known to be buggy, so it's official status stays beta.

Have fun,
fangorn

[gabrielg]

Can anybody tell me which version of imagemagick they are successfully running with menc? It fails compiling after I activated the truetype flag, which is required, this is the error:

Code: Select all

libtool: link: (cd ltdl/.libs/libltdlc.lax/dlopen.a && x86_64-pc-linux-gnu-ar x "/var/tmp/portage/media-gfx/imagemagick-6.7.8.7/work/ImageMagick-6.7.8-7/ltdl/.libs/dlopen.a")
libtool: link: x86_64-pc-linux-gnu-ar cru ltdl/.libs/libltdlc.a ltdl/loaders/.libs/ltdl_libltdlc_la-preopen.o ltdl/.libs/ltdl_libltdlc_la-lt__alloc.o ltdl/.libs/ltdl_libltdlc_la-lt_dlloader.o ltdl/.libs/ltdl_libltdlc_la-lt_error.o ltdl/.libs/ltdl_libltdlc_la-ltdl.o ltdl/.libs/ltdl_libltdlc_la-slist.o ltdl/.libs/lt__strl.o ltdl/.libs/libltdlcS.o  ltdl/.libs/libltdlc.lax/dlopen.a/dlopen.o 
libtool: link: x86_64-pc-linux-gnu-ranlib ltdl/.libs/libltdlc.a
libtool: link: rm -fr ltdl/.libs/libltdlc.lax
libtool: link: ( cd "ltdl/.libs" && rm -f "libltdlc.la" && ln -s "../libltdlc.la" "libltdlc.la" )
{standard input}: Assembler messages:
{standard input}:11005: Warning: end of file in string; '"' inserted
x86_64-pc-linux-gnu-gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugs.gentoo.org/> for instructions.
make[1]: *** [wand/wand_libMagickWand_la-mogrify.lo] Error 1
make[1]: Leaving directory `/var/tmp/portage/media-gfx/imagemagick-6.7.8.7/work/ImageMagick-6.7.8-7'
make: *** [all] Error 2

and my version is 6.7.8.7, tried ~amd64 as well with similar luck (different error).

I may just need to open a bug, but want to make sure it isn't me.

Thanks!

[fangorn]

imagemagick is a dependency of avi2mkv only if you want to use the dvd USE flag.
Which is known not to work properly, yet. The test cases I used produce a DVD with
menus, but the menus do not work correctly at the moment. DVD without menus
is still not implemented. I am a little stuck with these features for a while now.

So if you can't get it to compile, deactivate dvd support in avi2mkv and it should work.

I am concentrating on getting audio conversions to work stable on all platforms and
am at the moment evaluating the best ways to convert multichannel audio streams
into all other formats. In this resort the platforms are quite different, and I want to have
the same features on all of them. The current version is working on Linux and Windows,
but not on Mac OS. And Windows version has compeletely different feature set.

[gabrielg]

Thanks, fangorn - just for completeness, while my problem started with the truetype use flag, that was pulled by the dvd use flag, so removing those solved the emerge issue.

Now, sadly, I get an error at runtime:

Code: Select all

$ menc --help
Bareword found where operator expected at /usr/bin/toolbox_fangorn.pm line 4650, near ") unl"
        (Missing operator before unl?)
syntax error at /usr/bin/toolbox_fangorn.pm line 4650, near ") unl
"
Missing right curly or square bracket at /usr/bin/toolbox_fangorn.pm line 4650, at end of line
Compilation failed in require at /usr/bin/menc line 200.
BEGIN failed--compilation aborted at /usr/bin/menc line 200.

My perl is pretty non existent, so I tried to add a } before the last unl, but that created another error, so I thought I'd post it here before playing any further.

[fangorn]

I had problems with my provider and the last transfer of the toolbox was never finished.
So the version you have might be incomplete.

In doubt download / emerge the toolbox again.