How can I extract ABI and version info from binary libs?

[Zeault]

Greetings,

I am trying to use crossdev to build new packages for an existing Linux system that I have a limited ability to change. The problem is I don't know what exactly my target ABI is or the different versions of the compiler and libraries that I should use.

The full story follows:

I have an older ASUS all-in-one router product which has served me well over the years, but my network is becoming more sophisticated and I need more features. Rather than replace it with a new device I would like to try and modify the firmware myself. This particular router is NOT supported by any of the open source WRT Linux distributions due to its use of proprietary Broadcom hardware and firmware. Despite that, the platform is quite open. I can access it with SSH and I have gathered a lot of system info that way. ASUS also supplies GPL source tarballs for their routers, but the one I downloaded for my model seems to be missing some files. I am still working to figure that out because it has very limited documentation, but so far I don't believe I can simply build an image using the ASUS makefiles as-is.

As for the proprietary things I can't change we have:

• Some kernel modules in /lib/modules
• Some programs in /bin (and maybe they use proprietary shared libs I don't know)

What I think I want to do instead of trying to build the whole image is simply extract the existing root squashfs image, add some things to it, and then write it back using mtd-write. If my (admittedly quite incomplete) knowledge of Linux is correct, then all I should need to add my own custom programs is cross build them using the same arch, ABI, kernel headers, uclibc version, and kernel version, (and compiler version?). Here lies my problem. I don't know all of these things. I'm asking to see if there is an 'easy' way I could find them out by examining the binary files that I can extract from the firmware.

This may not come off as a strictly Gentoo question, but my plan is to slowly transform this router into an embedded Gentoo system. I will use crossdev configured with all the above info to build all of the packages I want, and then I'll use the Squashfs tools to create a blob I can write to the MTD partition when I want to do an update. If I keep all the ABI and versions the same then I should be able to do this without disturbing the existing proprietary software. I just have to keep the modules and bins in their existing places right?

OH and yes for the record I am aware of the risks of messing up my router beyond repair. I am also aware that this undertaking may require a lot of expertise that I clearly don't have, but 'outside of my comfort zone' is one of my favorite places to be.

Thank you

[NeddySeagoon]

Zeault,

There may be a lot of information lying around in in /proc and /var/logs

What does

Code: Select all

cat /proc/cpuinfo

Routers are often MIPS or arm of some flavour.

What libc does it use?
It need not be glibc. There are several others. you will need that to build a cross toolchain.

What does

Code: Select all

uname -a

say about the kernel?

Is there a /proc/config.gz

What does dmesg have to say about the hardware?

As it has evil binary kernel modules, rebuilding the kernel is not possible.

If you unpick the firmware pieces, can you boot it in a QEMU VM on some more mainstream hardware?
That lets you do a good deal of testing off the real hardware too.

I suspect that it has a serial console but you may need to open the box to get to it. It may only be solder pads.

What does it use for permanent storage?
I'm thinking ahead to unbricking it :)

[pingtoo]

Zeault,

I have tendency of over engineering, So I spent a lot of time doing extra works. I wonder in this case there maybe a simpler solution.

Have you consider build static so you don't need to concern about ABI?

Depend on how much more free storage you have and what additional features you want to put on the router, if it is just simply add DHCP/DNS/PROXY you may be find a package that can easily build in statically linked binary than you don't need to care about ABI as long as you have free space to hold the binary. And with statically linked binary there maybe some other way to shrink the binary for example use app-arch/upx

[GDH-gentoo]

• Some programs in /bin (and maybe they use proprietary shared libs I don't know)

You can also get useful information from the output of readelf -lhd /bin/some-of-those-programs.

Have you consider build static so you don't need to concern about ABI?

Why? You still have ABI compatibility issues at build time when linking to static libraries.

[pingtoo]

Have you consider build static so you don't need to concern about ABI?

Why? You still have ABI compatibility issues at build time when linking to static libraries.

My mistake, Sorry that I was not clear. My idea is having a separated build environment, "crossdev" for example. so as long as the build environment target the router's architecture then it should be possible to create binary run on the router without care of ABI. but still under one condition that the required function(s) is not hardware function. If the desired function require hardware support than it is the kernel need to support which might have some sort of compatibility issues.

[Zeault]

Here is my uname

Code: Select all

Linux RT-AC1200G 2.6.36.4brcmarm #1 PREEMPT Fri Jun 19 13:33:26 CST 2020 armv7l GNU/Linux

and here is my CPU

Code: Select all

Processor	: ARMv7 Processor rev 5 (v7l)
BogoMIPS	: 1795.68
Features	: swp half thumb fastmult edsp 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 5

Hardware	: Northstar Prototype

The system uses uClibc. This whole time I was under the impression that Gentoo supported uClibc, but I just checked and it does not. I know that what I am doing isn't really supported by ANYONE either, but I would like to take advantage of crossdev and portage to reduce my own burden of maintenance.

Have you consider build static so you don't need to concern about ABI?
...
... And with statically linked binary there maybe some other way to shrink the binary for example use app-arch/upx

That is a good suggestion, although I would like room to grow: adding more packages that might not like static linking out of the box. However this got me thinking. If I avoid uClibc and go use musl instead, what if I just had both installed at the same time? A quick web search suggests this is possible. I'd just keep my musl libs in /libmusl instead of /lib or something similar. I've never tried this before, and now I want to test it out on one of my other systems.

And oh yeah talk about over engineering. This is sort of just a fun and learning project though. The device itself isn't mission critical, and I have a backup plan for my network if it goes down. As long as I don't have to rewrite too much code to keep it running then I'll consider it sustainable. I'm not above writing or maintaining code, but I used to be way too ambitious with things or take on way too much responsibility. Now that I'm older and hopefully slightly wiser I've realized that my time is really limited and I just can't commit to that stuff. So anyways, hopefully most of the work on this project is just setup and it doesn't end up being to difficult to maintain.

As for storage/unbricking:

Code: Select all

dev:    size   erasesize  name
mtd0: 00040000 00010000 "boot"
mtd1: 00fb0000 00010000 "linux"
mtd2: 00d7d274 00010000 "rootfs"
mtd3: 00090000 00010000 "jffs2"
mtd4: 00010000 00010000 "nvram"

Everything is partitioned neatly. I was only going to touch rootfs. The stock firmware hardly uses the jffs2 partition but it does work. I wanted to try fuse-overlayfs or mergerfs to get the two partitions mounted together on / so I can write config files over ssh. The stock firmware takes the configs that you give it in the web interface, marshalls them into an odd squashed code and stores them as variables in nvram. Sure that works but it is not easy for me to edit if I try to change a config over ssh.

I think the boot partition contains some version of uboot with the tftp bootloader backup mechanism. ASUS provides a "rescue" utility that runs on windows and they explicitly state that it is capable of remotely flashing a router in rescue mode if it was damaged due to a failed firmware upgrade or the installation of third-party firmware.

Sorry this has strayed so far off the original question of finding out the ABI/version of already compiled libs. I have continued trying to research this and it is still not clear to me if the different versions of uClibc have incompatible ABI differences. Also I have not found an easy way to figure out the version of a compiled uClibc binary. It does not have the -v option that glibc has. I am going to mess around with musl for now and I might end up spinning this off into another topic. Thank you.

[NeddySeagoon]

Zeault,

Raking about in the gentoo-x86-historical git repo

Code: Select all

commit 0a2a2b02a59a7467052aa9868543dfe6a94c3d84
Author: Robin H. Johnson <robbat2@gentoo.org>
Date:   Tue Aug 3 17:22:33 2010 +0000

    kernel-2 deblob support for 2.6.35; bug #324505: deblob-check usage.
    (Portage version: 2.2_rc67/cvs/Linux x86_64)
     (Unsigned Manifest commit)

So your 2.6.36.4brcmarm kernel is from 2010 sometime.

ulibc has been long dead. It was alive and well in 2018 as there are some stage 3 tarballs still around.
That's one example.

There is selection of ebuilds in gentoo-x86-historical. The hard bit will be finding source files.

Likewise, for any contemporary software you want build. All the ebuilds from the start of CVS in 2002 to present day are only a git clone away.
Finding sources may be a challenge.

[Zeault]

Oh crap. Are you implying that in order to emerge a package from portage I need to either have a more up-to-date kernel or try and find the package source from way back then? I wasn’t going to assume that everything would just work, but I thought since the kernel is mostly hardware support that I could just build the system library with those kernel headers and then almost every package could be built from the latest source. As long as it’s not some kind of low level package that needs a newer kernel feature then shouldn’t it not care what version of the kernel I have?

Like I said. A lot of expertise I clearly don’t have. But this is how I like to learn new things. Thank you for your help

[NeddySeagoon]

Zeault,

If you will dynamically link things, it needs to be able to link to the installed libraries that you have on the target hardware. That means you need them in your crossdev target root.
uclibc was removed from portage by

Code: Select all

commit a311268af50644d4f15372d550c51e5f435b79b6
Author: Anthony G. Basile <blueness@gentoo.org>
Date:   Sun Oct 14 07:59:08 2018 -0400

    sys-libs/uclibc: final cleanup from tree

Your uclibc is from between 2010, when your kernel was made and the above commit, so which uclibc do you need?
It need not be exact, as long as the things you link in your crossdev target root have all the bits they need on the target hardware.

With static linking, if it can be achieved, the binary is almost self contained. It still needs to make calls to the kernel but that's a much smaller problem space.

You know than its

Code: Select all

Processor   : ARMv7 Processor rev 5 (v7l)

and uclibc.
Starting with the Embedded Handbook build yourself a cross toolchain for that CPU and uclibc.
That's going to fail as uclibc is no longer in the tree, so crossdev won't find it, hence you need a suitable uclibc ebuild and sources.

[Zeault]

Ahhh okay. I misunderstood your previous post. I think I am going to abandon the idea of linking to the existing uclibc that's on there. My new plan is to create another prefix in the squashfs and install the newest version of musl that will work with those old kernel headers. Then I can build my newish software and leave the old stuff mostly undisturbed. ....maybe. ....hopefully.

I've never made this type of musl prefix before but there is a lot of chatter about doing it across the web. e. g. people making a prefix in their home directory to compile and install software on a system they do not have administrative privileges for. I am going to try it out on a few of my other linux machines before I attempt it on the router.

Thank you very much.

[pingtoo]

Zeault,

Have you try the ASUS support page for your device? I found you can download source code for your device under "Driver & Utility" Tab and select "Select OS" with "Other".

In the downloaded zip/tarball, I found in the toolchain directory have uClibc sub-directory, which in turn have 0.9.30.1 and 0.9.29 sub-directories.

The entire tarball looks to me resemble openwrt early date release source code tar ball. May be you can set up a openwrt build system or use buildroot and adopt the downloaded source code to make a complete environment.

It seems to me that there are not much of space on the internal storage, I guess it is a 32MB model it would be hard to use modern linux kernel because its size.

the "linux" partition is around 15MB, I think it is holding kernel and possible initrd.
the "rootfs" partition is around 12MB, I think you can work on this one.
the "jffs2" parition is around 576KB, I think this is where is hold your configuration changes.
the "boot" partition is is 256KB, I think it is holding u-boot.