I need "read file until" tool

[Zucca]

I have a specific need for a tool which can work in a pipe with other programs.

I need to read a (binary) file until a string (more preferably regex) of characters (in hex or ascii) are encountered.

I've used something like this before:

Code: Select all

((offset=$(grep --byte-offset --only-matching --text "<regex>" "<file>" | head -n 1 | grep -Eo '^[0-9]+')+1))
tail -c +"$offset" "<file>"

... although this example outputs the data after the regexp. Easily modified to output the data before. But that cannot work in a pipeline. Unless one could use "tee" with some strange magic. I don't think its possible.

Anyway. Does such tool exist? Or can it be done with some shell (bash) trickery? Maybe (g)awk?

I want this because to avoid using temporary files.

[mike155]

You could use sed:

Code: Select all

sed '/PATTERN/q' FILE

See: https://unix.stackexchange.com/question ... -the-match

Sed works at least for text files. Maybe it can be tweaked to work with binary files.

[mike155]

dev-util/bbe is a sed for binary files.

You could try sometghing like:

Code: Select all

bbe -b '/STRING/:$' -e "D" 

Bbe doesn't seems to understand regular expressions though...

[Zucca]

dev-util/bbe is a sed for binary files

Thanks!
I might get away using it... It surely is the closest thing I'm looking for.

[Zucca]

This bbe looks like a wonderful tool.
Shame it does not support regex or even patterns (kinda).

The d command would for perfectly if it could take negative values as first parameter. I guess it would complicate the program by order of magnitude.

But that gave me a thought... What if I could give sed different record (block) splitter than a newline?
Then again I could do the same with awk. Problem is then if you give give it a huge binary file... It could eat up my RAM very quickly and with it goes down the performance.

Oh well. I'll take one more closer look at bbe, test it out, then decide if it is the real answer.

[ff11]

Not every binary has strings encoded in the same way (some will need the search to be done relative to the position of each character on the string, others will need a special character table). Could you tell us what you need this for?

EDIT: Just for information, how a string can get a format that makes several of these tools infeasible on the binary: Hazards of Converting Binary Data To A String

[Zucca]

ff11: I'm trying to concatenate cpio archives on-the-fly (pipe). Normally you'd need to store the earlier archive on disk before you can concatenate another one. I've solved that problem already.

Now I'm just interested generally about the topic. bbe is quite a tool, but lacks regexp features.
String format therefore isn't an issue either. I think strings -tool is for that kind of usage.

[ff11]

ff11: I'm trying to concatenate cpio archives on-the-fly (pipe). Normally you'd need to store the earlier archive on disk before you can concatenate another one. I've solved that problem already.

Now I'm just interested generally about the topic. bbe is quite a tool, but lacks regexp features.
String format therefore isn't an issue either. I think strings -tool is for that kind of usage.

I'm sorry, but I didn't get it right. For some reason I think you are trying to recreate the app-arch/tar using strings and concatenation.

[Zucca]

<OT>
I'm making a wrapper around cpio, which has few features to identify entries given to it (from stdin) so that it can be used also as a initramfs image creation tool (my primary goal).
It can be also dropped into /etc/kernel/postinst.d so it can auto create initramfs image when a new kernel is installed.

It however does not have any more automation than identifying the entries, therefore it can be used also as a general cpio tool. The automatic part of it is only to deal with library collecting and module dependencies (optionally including firmware files). Everything else is left to the user:

• linuxrc/init script
• listing of required
  • binaries
  • modules
  • ... and optionally firmware files

So basically it solves the dependencies for initramfs and packs them along other files specified by the user.
</OT>

[ff11]

<OT>
I'm making a wrapper around cpio, which has few features to identify entries given to it (from stdin) so that it can be used also as a initramfs image creation tool (my primary goal).
It can be also dropped into /etc/kernel/postinst.d so it can auto create initramfs image when a new kernel is installed.

It however does not have any more automation than identifying the entries, therefore it can be used also as a general cpio tool. The automatic part of it is only to deal with library collecting and module dependencies (optionally including firmware files). Everything else is left to the user:

• linuxrc/init script
• listing of required
  • binaries
  • modules
  • ... and optionally firmware files

So basically it solves the dependencies for initramfs and packs them along other files specified by the user.
</OT>

Sorry, but i'm a little lost here.
Isn't cpio an archive that is a concatenation of files and directories preceded by a header giving the file name and other file system information (like the size)?
Isn't it enough to use the command normally?
If the issue is with stdin, wouldn't using named pipe be enough?
And if the cpio command is missing features, wouldn't it be better to discuss it with the GNU people? They probably have a much broader view on using cpio than the folks here.
( on the mailing list: https://www.gnu.org/software/cpio/cpio.html ).

[Zucca]

You misunderstood and I didn't explain very well.

Features of my little project which are out of cpio feature scope:

• Handles dependencies of modules and their firmware files
• Handles directories differently
  • cds into the directory and adds all the files and directories using realtive paths (add them into cpio root)
• Can work as a drop-in file inside /etc/kernel/postinst.d
• Compresses the archive

All the files with absolute paths are added as-is but binaries are inspected if they need any dynanic libraries. Those library files are automatically added to the cpio.
Modules listed with their name only (no path given) are inspected for dependencies also.
If --auto-firmware is being used then all the firmware files which are associated with listed modules are also added automatically.

All these features are not something cpio would need built-in.

The reason I want to concatenate cpio archives is to make it a little neat.
Normally there's an EOF marker inside every cpio archive: A pseudo file named 'TRAILER!!!' with zero size. I'm removing this marker. It's not really neccessary in case of initramfs images. But normally you'd need to run cpio inside a loop to be able to extract or list the files it contains.

So yes. cpio archives are files concatenated, but cpio still has the EOF marker.

man 5 cpio is really interesting read in its own right. I've been reading it quite many times now. :P

Anyway... Too much off-topic now.

[szatox]

I haven't inspected the binary format of CPIO archive, but you might even get away with using head.
Like in head -c -10 (skip the last 10 bytes)

[Zucca]

I haven't inspected the binary format of CPIO archive, but you might even get away with using head.
Like in head -c -10 (skip the last 10 bytes)

I actually tried it. It didn't work properly, because cpio adds trailing padding depending of which cpio format is being used.