| View previous topic :: View next topic |
| Author |
Message |
transpetaflops
Apprentice
Joined: 16 May 2005
Posts: 157
|
 Posted: Tue Mar 21, 2023 12:39 am Post subject: Debugging src_compile phase in app-backup/znapzend ebuild |
 |
|
Good evening all. I'm not an ebuild developer but I can usually hack my way forward by studying examples, reading error messages and adjust accordingly. But now I've hit a snag and I'm in dire need of some knowledge transfer.
BACKGROUND:
Well known MRTG developer Tobi Oetiker has a utility to automate ZFS snapshots and replication, which I have used with success in FreeBSD. https://github.com/oetiker/znapzend
An ebuild exists in the tgerczei overlay but has been abandoned for several years and it's stuck at EAPI 5. https://git.gerczei.eu/tgerczei/gentoo-overlay/src/branch/master/app-backup/znapzend
I've been trying for days to migrate it to EAPI 7 and have had partial success.
CURRENT PROGRESS:
The only things I've had to do so far is to add a depend for sys-block/mbuffer (not strictly necessary), adjusted the syntax for the patches and also regenerated the patches (it was only a matter of updating the version number of znapzend).
Pastebin of my diff: https://pastebin.com/raw/5wHQ6Jw0
ebuild znapzend-0.21.1.ebuild clean - WORKS
ebuild znapzend-0.21.1.ebuild prepare - WORKS
ebuild znapzend-0.21.1.ebuild configure - WORKS
ebuild znapzend-0.21.1.ebuild compile - FAILS
And here is where it gets tricky for me. As the elog message in the ebuild states, znapzend needs some Perl modules not currently in Portage and will install these automatically from CPAN and this is the part that fails. But it only fails when I let Portage do the compiling. If I manually enter the working directory when the compile fails, and simply type make, it works! I can see from the output that it complains over not finding some Perl modules (Carton and Date::Parse) but they are there in the working directory and are correctly picked up during a manual compile. I can't figure out from the console output what goes wrong.
Pastebin of output from ebuild znapzend-0.21.1.ebuild compile: https://pastebin.com/raw/ksU89FQf
Pastebin of output from manually typing make in the working directory: https://pastebin.com/raw/VpAG48uC
Here's a few lines from the first log when it fails and the same part from the working output when I type make manually:
| Code: | Making all in thirdparty
make[1]: Entering directory '/var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty'
echo "** Installing Dependencies using Carton install"
** Installing Dependencies using Carton install
test -f cpanfile-5.36.snapshot && cp cpanfile-5.36.snapshot ../cpanfile.snapshot || true
test -x carton/bin/carton || PERL_CPANM_OPT= PERL_CPANM_HOME=/var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty /usr/bin/perl bin/cpanm -q --notest --local-lib-contained /var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty/carton Carton Date::Parse
! Couldn't find module or a distribution Carton
! Couldn't find module or a distribution Date::Parse
make[1]: *** [Makefile:435: cpanfile-5.36.snapshot] Error 1 |
| Code: | Making all in thirdparty
make[1]: Entering directory '/var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty'
echo "** Installing Dependencies using Carton install"
** Installing Dependencies using Carton install
test -f cpanfile-5.36.snapshot && cp cpanfile-5.36.snapshot ../cpanfile.snapshot || true
test -x carton/bin/carton || PERL_CPANM_OPT= PERL_CPANM_HOME=/var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty /usr/bin/perl bin/cpanm -q --notest --local-lib-contained /var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty/carton Carton Date::Parse
Successfully installed Path-Tiny-0.144
Successfully installed Class-Tiny-1.008
Successfully installed Module-CPANfile-1.1004 |
Is there any way to get more verbose output during src_compile so I can see what's going on? Or is there some major difference between EAPI 5 and EAPI 7 I need to correct for? Any input would be greatly appreciated.
Regards
Morgan |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21593
|
| Posted: Tue Mar 21, 2023 1:24 am Post subject: |
|
|
| transpetaflops wrote: | | Is there any way to get more verbose output during src_compile so I can see what's going on? |
I think your problem is that the upstream build system is failing, but doing so quietly. You could change src_compile to instruct the upstream build system to be more verbose, assuming it has a way to request greater verbosity. If it has that capability, the way to request it is build-system specific, so without reviewing the upstream build system, I cannot tell you whether it supports verbosity at all, and if so, how.
Fortunately, from the other things you have said, I can make an educated guess. | transpetaflops wrote: | | And here is where it gets tricky for me. As the elog message in the ebuild states, znapzend needs some Perl modules not currently in Portage and will install these automatically from CPAN and this is the part that fails. |
This is a bug in the upstream build system. Build systems are to build things, not to download them. Accessing the Internet during src_compile is forbidden, so any build system that wrongly assumes it can download from the Internet will just fail. Most of them fail a bit more obviously than this one did, though. The typical behavior is for them to fail complaining that they cannot resolve some public hostname. | transpetaflops wrote: | | But it only fails when I let Portage do the compiling. If I manually enter the working directory when the compile fails, and simply type make, it works! |
This would fit with my theory above. Portage enforces a sandbox. Your manual invocation of make does not. | transpetaflops wrote: | | I can see from the output that it complains over not finding some Perl modules (Carton and Date::Parse) but they are there in the working directory and are correctly picked up during a manual compile. |
That is a bit odd. Without examining the package, my best guess is that the artifacts you found are not full copies of the packages, or they were put there by your manual make downloading from the Internet. | transpetaflops wrote: | | I can't figure out from the console output what goes wrong. |
In my opinion, that is another bug. If the error messages are so vague that the user cannot determine what the error is trying to say, then they need to be improved. You should not need to contact the developer, or ask someone to read the source, just to understand what is wrong. | transpetaflops wrote: | Here's a few lines from the first log when it fails and the same part from the working output when I type make manually: | Code: | echo "** Installing Dependencies using Carton install"
** Installing Dependencies using Carton install |
|
The phrasing here is suspicious. As I said above, build systems are to build things. They are not to install dependencies. They can check that dependencies have been installed (provided they do not engage in automagic dependencies). | transpetaflops wrote: | | Code: | | test -f cpanfile-5.36.snapshot && cp cpanfile-5.36.snapshot ../cpanfile.snapshot || true |
|
This looks wrong, since it will never fail. If the recipe line must not fail, a leading - to tell Make to ignore errors would have been simpler. However, for the apparent purpose, I would have gone with: | Code: | | if ! test -f cpanfile-5.36.snapshot; then cp cpanfile-5.36.snapshot ../cpanfile.snapshot; fi |
In my opinion, this is better since it will return success if the file exists, return success if the file did not exist, but was successfully copied, and return failure if the file did not exist and the copy failed. The recipe you showed never exits failure, even if the copy fails. This then allows Make to continue on, which may produce more confusing error messages when a later recipe assumes the file was copied. | transpetaflops wrote: | | Code: | | test -x carton/bin/carton || PERL_CPANM_OPT= PERL_CPANM_HOME=/var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty /usr/bin/perl bin/cpanm -q --notest --local-lib-contained /var/tmp/portage/app-backup/znapzend-0.21.1/work/znapzend-0.21.1/thirdparty/carton Carton Date::Parse |
|
You might get better behavior if the ebuild deletes carton/bin/carton before beginning, so that downloads are not attempted. | transpetaflops wrote: | | Code: | ! Couldn't find module or a distribution Carton
! Couldn't find module or a distribution Date::Parse |
|
This error message needs to be improved. In what ways did it not find the package? Where did it look? What did it try? The error message tells us none of these things. | transpetaflops wrote: | | Code: | Successfully installed Path-Tiny-0.144
Successfully installed Class-Tiny-1.008
Successfully installed Module-CPANfile-1.1004 |
|
These are also lacking, in my opinion. From where were these installed? What was done to validate the install? The success message does not tell us.
To recap: your problem is probably the network-sandbox. The correct solution is to package and install the dependencies. An ugly workaround, which would not be accepted in the main tree and is discouraged generally, would be to override the network sandbox, so that the upstream build system can download arbitrary blobs from the Internet. Among other reasons, this is bad since the build may work today and fail tomorrow, seemingly with no changes. |
|
| Back to top |
|
|
transpetaflops
Apprentice
Joined: 16 May 2005
Posts: 157
|
| Posted: Tue Mar 21, 2023 2:09 am Post subject: |
|
|
Thank you, Hu.
I'm inclined to agree with you on your conclusions. My thoughts have wandered in the same direction. The original ebuild from the overlay is from 2020 though and has presumably worked at one point judging from the lack of bug reports. I would assume that the sandboxing restrictions have applied way longer than that so I don't see how the original author circumvented it.
In FreeBSD the required Perl modules exists in Ports so their Makefile appears to completely delete the thirdparty part in the configure part of the build system.
| Code: | pre-configure:
@${REINPLACE_CMD} -e '/thirdparty\/Makefile/d' ${WRKSRC}/configure.ac |
Arch Linux has a package for znapzend but their PKGBUILD doesn't give me any useful hints. Their build system simply does a configure + make so I suppose everything gets bundled in their final package.
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=znapzend
It's far beyond my skill level to start creating patches for Oetiker's build system. Maybe I could create ebuilds for the missing Perl modules and go the FreeBSD route. I'd have to look into some other Perl module ebuilds to see if they could be easily adapted or if they perhaps already exist in some overlay. In worst case I might have to install Pacman and pull the package from Arch or just install znapzend manually. But Portage is a blessing and I'd really like to have everything managed by it.
Once again, thank you for your valuable insights Hu.
/Morgan |
|
| Back to top |
|
|
transpetaflops
Apprentice
Joined: 16 May 2005
Posts: 157
|
| Posted: Wed Mar 22, 2023 8:19 pm Post subject: |
|
|
In case someone stumbles upon this thread through Google, I'll add my working ebuild below. Please feel free to suggest improvements because I'm very much an ebuild newbie.
It's based on the abandoned version found at:
https://git.gerczei.eu/tgerczei/gentoo-overlay/src/branch/master/app-backup/znapzend/znapzend-0.20.0.ebuild
It also uses simple sed commands, taken from the FreeBSD Ports version, to prevent the thirdparty CPAN modules from being installed from within the znapzend build script.
https://cgit.freebsd.org/ports/tree/sysutils/znapzend/Makefile?id=3d112f58dc8dac57deb3481b684b4d4a8dba16a9
Instead I use the necessary Perl modules directly from Portage with one exception, Mojo::Log::Role::Clearable, which unfortunately is missing in Portage. It needs to be installed with g-cpan before emerging znapzend. Instructions can be found in the remarks in the ebuild.
The latest release version of znapzend (0.21.1) uses a deprecated Perl module function call and rather than installing those deprecated modules I chose to let the ebuild install the latest git version where this is corrected.
Don't forget you also need the initscript from the original ebuild in addition to my ebuild.
https://git.gerczei.eu/tgerczei/gentoo-overlay/src/branch/master/app-backup/znapzend/files/initd_znapzend
/Morgan
app-backup/znapzend/znapzend-9999.ebuild
| Code: | # Copyright 1999-2023 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
EAPI=8
DESCRIPTION="ZFS backup with remote capabilities and mbuffer integration"
HOMEPAGE="https://github.com/oetiker/znapzend"
if [[ ${PV} == "9999" ]] ; then
inherit git-r3
EGIT_REPO_URI="https://github.com/oetiker/znapzend.git"
else
SRC_URI="${HOMEPAGE}/releases/download/v${PV}/${P}.tar.gz"
fi
LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~amd64"
IUSE="mbuffer"
# perl-gcpan/Mojo-Log-Clearable is currently unavailable in Portage and
# needs to be installed with app-portage/g-cpan before emerging znapzend.
# Create a local repository first unless one already exists.
#
# 'eselect repository create local'
# 'g-cpan -i Mojo::Log::Role::Clearable'
RDEPEND="
mbuffer? ( sys-block/mbuffer )
dev-perl/IO-Pipely
dev-perl/Mojolicious
dev-perl/Role-Tiny
virtual/perl-Data-Dumper
virtual/perl-Getopt-Long
virtual/perl-Scalar-List-Utils
virtual/perl-Sys-Syslog
virtual/perl-Text-ParseWords
perl-gcpan/Mojo-Log-Clearable
"
src_prepare() {
default
./bootstrap.sh
# Remove thirdparty Perl modules and use the system provided instead (fix from FreeBSD)
sed -i '/thirdparty\/Makefile/d' ${S}/configure.ac
}
src_configure() {
PERL5LIB="/usr/share/${PN}/perl5" \
econf --libdir="/usr/share/${PN}/perl5"
# Remove thirdparty Perl modules and use the system provided instead (fix from FreeBSD)
sed -i 's/thirdparty//g' ${S}/Makefile
}
src_compile() {
emake
}
src_install() {
default
insinto /etc/default
newins init/${PN}.default ${PN}
newinitd ${FILESDIR}/initd_${PN} ${PN}
} |
|
|
| Back to top |
|
|
Kuhrscher
Guru
Joined: 29 Dec 2003
Posts: 498
Location: Hamburg, Germany
|
| Posted: Wed May 03, 2023 7:54 pm Post subject: |
|
|
To make znapzendzetup work, I had to install Mojo-IOLoop-ForkCall and Mojo-IOLoop-Delay via gcpan.
But generally your ebuild works like a charm. Thank you very much. |
|
| Back to top |
|
|
transpetaflops
Apprentice
Joined: 16 May 2005
Posts: 157
|
| Posted: Wed May 03, 2023 10:49 pm Post subject: |
|
|
Thank you. Those are the deprecated Perl modules I mention above. They were used up until version 0.21.1 and shouldn't be needed for the git version. Anyway, there's now a 0.21.2 release and you can fix the ebuild for it with this patch:
| Code: | --- znapzend-9999.ebuild 2023-04-19 19:58:07.583788530 +0200
+++ znapzend-0.21.2.ebuild 2023-04-19 19:43:45.580584164 +0200
@@ -10,12 +10,12 @@
inherit git-r3
EGIT_REPO_URI="https://github.com/oetiker/znapzend.git"
else
- SRC_URI="${HOMEPAGE}/releases/download/v${PV}/${P}.tar.gz"
+ SRC_URI="${HOMEPAGE}/archive/refs/tags/v${PV}.tar.gz"
fi
LICENSE="GPL-3"
SLOT="0"
-KEYWORDS=""
+KEYWORDS="~amd64"
IUSE="mbuffer"
# perl-gcpan/Mojo-Log-Clearable is currently unavailable in Portage and
|
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
