Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ffprobe while mv
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Multimedia
View previous topic :: View next topic  
Author Message
Nicias
Guru
Guru


Joined: 06 Dec 2005
Posts: 446

PostPosted: Wed Mar 05, 2014 11:36 pm    Post subject: ffprobe while mv Reply with quote

I need to do the two following commands

Code:
$ ffprobe -select_streams v -count_frames input.mkv
$ mv input.mkv /some/other/device/.

Now both commands involve reading the whole file. Is there a way to combine those. My thought was something like:
Code:
$ cat input.mkv | tee /some/other/device/input.mkv | ffprobe -select_streams v -count_frames -
$ rm input.mkv

Is there any reason this wouldn't work?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Thu Mar 06, 2014 1:32 am    Post subject: Reply with quote

Nicias ...

linux caches disk access so the first read is enough to make the subsequent read noticably faster (assuming mm hasn't had need to evict the page) ... this is why memory use grows with uptime (the saying is "a page of free RAM is wasted RAM"). The bottleneck in the above will be the write to '/some/other/device/' ... and 'cat |tee' won't be helping in that regard.

best ... khay
Back to top
View user's profile Send private message
Nicias
Guru
Guru


Joined: 06 Dec 2005
Posts: 446

PostPosted: Thu Mar 06, 2014 1:59 am    Post subject: Reply with quote

I would agree with you, except that the file is typically over 2G and my machine only has 2G RAM, so I don't think caching will help.

I did both with a 2G test file. Using tee: 2:55. Using ffprobe then cp: 3:44. The two files were identical (according to diff)

This is part of a script that takes about an hour to run. I'm not sure what I'll do. I'm inclined to do the two step method for safety even though it takes an extra 49 seconds and wears the disk out a bit more.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Thu Mar 06, 2014 6:51 am    Post subject: Reply with quote

Nicias wrote:
I would agree with you, except that the file is typically over 2G and my machine only has 2G RAM, so I don't think caching will help.

Nicias ... well, some of that 2G of memory will still be caching some part of the file, but yes, its not going to overcome physical memory limits.

Nicias wrote:
I did both with a 2G test file. Using tee: 2:55. Using ffprobe then cp: 3:44. The two files were identical (according to diff)

Thats interesting as I've not thought about such combinations previously.

Nicias wrote:
This is part of a script that takes about an hour to run. I'm not sure what I'll do. I'm inclined to do the two step method for safety even though it takes an extra 49 seconds and wears the disk out a bit more.

I'm not sure there is any issue with output being guarenteed safe ... the 'tee' should be seen by the shell as one job so the parent shell (which forked the 'cat') shouldn't close until both outputs are completed.

Anyhow, in zsh one can do the following ...

Code:
# {cat input.mkv} > output.mkv | ffprobe -select_streams v -count_frames -

... the '{.....}' will wait for both output to be written as the cat isn't forked from the current shell (note that for zsh a pipe is implicitly a redirection, hence the 'tee' is implicit).

Not sure how bash behaves in this regard ... sorry.

best ... khay
Back to top
View user's profile Send private message
Nicias
Guru
Guru


Joined: 06 Dec 2005
Posts: 446

PostPosted: Thu Mar 06, 2014 2:53 pm    Post subject: Reply with quote

In terms of the memory, I would think that the mm would dump the beginning of the file to read the end of the file and then vice versa when reading the second file, hence not actually using any of the cache.

In terms of safety, you are saying that the cat | tee -> file part will always do the "cp" even if the ffprobe part misbehaves? That might be true, but I don't know if ffprobe can always take a piped file. It might need to seek sometimes. That was what I meant by safety.

I think I will stick with the two step process as the one step doesn't use as much more time as I thought.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Sat Mar 08, 2014 6:56 pm    Post subject: Reply with quote

Nicias ... sorry, I seem to have overlooked the notification that you'd responded ...

Nicias wrote:
In terms of the memory, I would think that the mm would dump the beginning of the file to read the end of the file and then vice versa when reading the second file, hence not actually using any of the cache.

Yes, your probably right.

Nicias wrote:
In terms of safety, you are saying that the cat | tee -> file part will always do the "cp" even if the ffprobe part misbehaves? That might be true, but I don't know if ffprobe can always take a piped file. It might need to seek sometimes. That was what I meant by safety.

I was looking it from the perspective of the parent ... that is what my zsh example was about ... the stdout streams are spawned after the 'cat' is spawned from the parent shell and because there are two multiple outputs the parent may not wait for multios to finish ... that is why the cat is run as a job in the current shell, to prevent this happening. The children are of course prone to 'misbehave' but I assume that 'tee' will do the right thing and complete whether one or other output fails. The point I was making is that as far as the parent is concerned the 'tee' is one output and so the parent shouldn't care what the ultimate outcome of the redirection is ... its non-interfering. If one stream fails, I imagine 'tee' will report this to stderr, and like any task there is no guarantees, the best you can do its test $? and do something based on success/failure.

best ... khay
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Multimedia All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum