Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Slackware's make_world.sh
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
psycho
Guru
Guru


Joined: 22 Jun 2007
Posts: 534
Location: New Zealand

PostPosted: Fri Feb 04, 2022 1:39 am    Post subject: Slackware's make_world.sh Reply with quote

This is from the Slackware (15) release announcement:

"For the first time ever we have included a 'make_world.sh' script that allows automatically rebuilding the entire operating system from source".

Has anyone seen the script yet? I'm guessing it'll just be a systematic run through the source tree dumping binaries somewhere for installation, with no centralised patching system (let alone dependency resolution for e.g. rebuilding just part of the OS). On the other hand, if it at least provides some kind of useful mechanism for managing local patches, it could make Slackware even more interesting as a stable platform onto which other stuff can be built from source (it's already one of the best for that, with its policy of sticking as close as possible to upstream and layering on the bare minimum of additional software to make things cohere)...and I guess even if it doesn't do that, the mere provision of a make_world.sh script makes a full source rebuild an officially supported activity now.

Of course the slightest bit of patching, or changes to CFLAGS or whatever, and the official support would likely be withdrawn. Portage is really something, the way it supports such a vast amount of variation: the amount of human time and knowledge coded into Gentoo's package management (including e.g. distributed through all the ebuilds) is staggering. Perhaps unfairly as I haven't seen it yet, I'm thinking Slackware's "make_world.sh" will be to "emerge world" roughly what a honk is to a symphony.
Back to top
View user's profile Send private message
InvisibleRasta
Apprentice
Apprentice


Joined: 30 Mar 2011
Posts: 181

PostPosted: Fri Feb 04, 2022 2:06 am    Post subject: Reply with quote

Here is the script http://ix.io/3OAd
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Fri Feb 04, 2022 3:58 am    Post subject: Reply with quote

I do not trust Slackware shell scripts.
They give me the impression to be sloppy.
Back to top
View user's profile Send private message
psycho
Guru
Guru


Joined: 22 Jun 2007
Posts: 534
Location: New Zealand

PostPosted: Fri Feb 04, 2022 5:45 am    Post subject: Reply with quote

Thanks for posting the script InvisibleRasta. LOL...check out line 38:
Code:
# Slackware is not Gentoo.

:lol:
Back to top
View user's profile Send private message
ndavilam
n00b
n00b


Joined: 09 Feb 2022
Posts: 1

PostPosted: Wed Feb 09, 2022 11:46 pm    Post subject: Reply with quote

Leonardo.b wrote:
I do not trust Slackware shell scripts.
They give me the impression to be sloppy.



Slackware:
Initial release: July 16, 1993

Gentoo:
Initial release: March 31, 2002

Slackware is a set of hard work and dedication and as a Slackware user and in general vision I have no reason to distrust the build IMHO they are readable enough to be able to afford to fix a line of code yourself. If something is not in your possibilities to solve it, there is a passionate community that helps inexperienced people like me.
But thanks to this community little by little one learns many tricks.

I don't know if you have used Slackware or if you are afraid to use this distribution, if you haven't, I invite you to use it.
Me, I won't say anything about Gentoo because I haven't used it.
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Thu Feb 10, 2022 1:23 am    Post subject: Reply with quote

Ok.
Back to top
View user's profile Send private message
adcdam
Apprentice
Apprentice


Joined: 15 Jan 2015
Posts: 160

PostPosted: Thu Feb 10, 2022 4:49 pm    Post subject: Reply with quote

i used both distros, both outanding distros, i used Funtoo first and Gentoo for about 5 years, im using Slackware current on my pc and slackware 15 on my laptop. Next year Slackware will be 30 years old, if the distro isn't as well made it wouldnt lasted almost 30 years.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Fri Feb 11, 2022 12:23 am    Post subject: Reply with quote

Leonardo.b wrote:
I do not trust Slackware shell scripts.
They give me the impression to be sloppy.
Anything in particular? I was considering using Slackware as one of my options for a binary distro.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Fri Feb 11, 2022 3:20 am    Post subject: Reply with quote

Nothing against Slackware, I am not unraccomanding it.

This particular build_world.sh does not fit my tastes.
I do not like pkgtools scripts either - those scripts minated my trust.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Fri Feb 11, 2022 5:50 am    Post subject: Reply with quote

I was just curious what about them you didn't trust.

I looked at the one linked, and while the style is lacking some space that would help readability, nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Fri Feb 11, 2022 5:15 pm    Post subject: Reply with quote

pjp wrote:
nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)
I have been summoned.
http://ix.io/3OAd:
     1   #!/bin/bash
    47   cd $(dirname $0) ; CWD=$(pwd)
  • First non-comment line is not set -eu or similar. This may let errors pass undetected later.
  • Failed to quote parameter, allowing an error to occur. Due to the lack of set -e, this error will not stop the script. The script will blunder on, running in the wrong directory.
    Code:
    $ cat /tmp/demo/sub\ dir/a.sh
    #!/bin/sh

    cd $(dirname $0)
    pwd
    $ bash -x /tmp/demo/sub\ dir/a.sh
    ++ dirname /tmp/demo/sub dir/a.sh
    + cd /tmp/demo dir
    /tmp/demo/sub dir/a.sh: line 3: cd: too many arguments
    + pwd
    /tmp
  • Useless assignment of $CWD. It is never read. Even if it were read, it likely could be skipped through use of $PWD.
http://ix.io/3OAd:
    54   LOGDIR=$TMP/make_world
    56   mkdir -p $LOGDIR/lock
Failed to quote parameter, allowing a surprising and almost certainly wrong result to occur.
Code:
$ TMP=/tmp/demo/sub\ dir/ bash -x a.sh
+ LOGDIR='/tmp/demo/sub dir//make_world'
+ mkdir -p /tmp/demo/sub dir//make_world/lock
Now dir/make_world/lock exists, and '/tmp/demo/sub dir//make_world' does not.
http://ix.io/3OAd:
    62     # Make sure this ends in '/':
    63     if [ ! "$(echo $SLACKWARE_SOURCE_DIRECTORY | rev | cut -b 1)" = "/" ]; then
The script explicitly requests to be run by /bin/bash, so it would not be unreasonable to use a Bash transform here.
http://ix.io/3OAd:
    77   # Be kind, don't hit control-c! If you do, you might leave broken packages,
    78   # logfiles, and locks in $TMP that will cause problems for you later. If you're
    79   # not in a huge hurry to quit, create this file (replace with $TMP if needed):
    80   # /tmp/make_world/lock/abort
    81   # This will cause all instances of make_world.sh to exit when they complete the
    82   # task they are working on.
    83   rm -f $LOGDIR/lock/abort
This seems fragile. It would be better to use a trap handler to recover from a control-c, or to arrange for orphaned logfiles and locks to be recognizable as such by the later run.
http://ix.io/3OAd:
    85   BUILDLIST=${BUILDLIST:-$LOGDIR/buildlist}
    86   if [ ! -r $BUILDLIST -a ! -r ${BUILDLIST}.lock ]; then
Failed to quote argument.
Code:
$ a=/tmp/demo/sub\ dir/a.sh
$ if [ ! -r $a -a ! -r $a.x ]; then echo a; else echo b; fi
-bash: [: too many arguments
b
http://ix.io/3OAd:
    90     touch ${BUILDLIST} ${BUILDLIST}.lock
Quoting, again. I will stop flagging this particular error now.
http://ix.io/3OAd:
    95         if [ "$uname -m)" = "x86_64" -a "$(basename $(echo $script | cut -f 1 -d ' '))" = "isapnptools.SlackBuild" ]; then
uname -m could be cached, since this script will not migrate to a different architecture while it runs.
http://ix.io/3OAd:
   107     # You might want to do this to comment some build scripts out first, or if
   108     # you'd like to sort it into a "magic build order". ;-) This script is pretty
   109     # good at just brute-forcing things, though (with a few runs).
That suggests the packages aren't actually built in an enforced safe ordering, so odd failures and wasted work could result.
http://ix.io/3OAd:
   117     while [ 0 ]; do
   118       if [ ! -r ${BUILDLIST}.lock ]; then
This could be simplified as while [ -r "$BUILDLIST.lock" ], rather than needing a separate if+break.
http://ix.io/3OAd:
   174       cat $BUILDLIST | grep -v "^$" | grep -v "^#" | sort | uniq | cut -f 1 -d ' ' | rev | cut -f 1 -d / | rev > $LOGDIR/tmp-pkgs-to-build.$$
Useless Use of Cat. Unnecessary chaining of grep.
http://ix.io/3OAd:
   193           for findconfigure in */configure ; do
   194             rm -rf "$(dirname $findconfigure)"
   195           done
   196           for findautogen in */autogen.sh ; do
   197             rm -rf "$(dirname $findautogen)"
   198           done
Some cleverness with globs and text transforms could collapse all this into a single aggregation loop and a single rm on numerous directories.
http://ix.io/3OAd:
   227         HELD_BY="$(cat $LOGDIR/lock/lock.global)"
As a Bash script (rather than sh script), this is a Useless Use of Cat. $(< filename) can get the same result.
http://ix.io/3OAd:
   230           sleep 10
This is a rather long delay. Clever use of inotify tools could allow an instant reaction. Even without that, polling at 1-2 second intervals, without printing on every loop, would probably be fine.
http://ix.io/3OAd:
   235           if [ ! "${HELD_BY}" = "$(cat $LOGDIR/lock/lock.global)" ]; then
   236             HELD_BY="$(cat $LOGDIR/lock/lock.global)"
Time of Check / Time of Use. The file ought to be read once, and its value reused, rather than re-read repeatedly.
http://ix.io/3OAd:
   243       if [ -r $OUTPUT_LOCATION/$(PRINT_PACKAGE_NAME=foo $buildscript | head -n 1) -a $FORCE_BUILD = no ]; then
Checking $FORCE_BUILD is cheaper. It ought to be done before running the build script.
http://ix.io/3OAd:
   247       # Use flock to only allow one instance of this script to work on a given
   248       # SlackBuild script at a time.
If we assume use of a working flock, why use a polling loop above instead of just acquiring the lock?
http://ix.io/3OAd:
   260         # pkgtools, x11, and KDE all trigger the detection above, but none of them
   261         # really need the global lock. So only request the lock if the build
   262         # script is not one of those.
That is not particularly confidence inspiring.
http://ix.io/3OAd:
   268         ( flock 9 || exit 11
   271         ) 9> $LOGDIR/lock/lock.global
Using a fixed file descriptor is messy. Since use of Bash is assumed, the shell can be asked to pick a free descriptor and provide it.

The descriptor leaks into all the individual build scripts. This might be considered a feature since it keeps the lock held as long as any build script process is alive. It might be considered a bug since it lets those scripts easily write garbage into the lock file.
http://ix.io/3OAd:
   272         # Remove lock file:
   273         rm -f $LOGDIR/lock/lock.global
Deleting the lock file here largely negates the benefit of leaking the lock into the individual build scripts.

No comments on the remainder of the script.
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Fri Feb 11, 2022 9:27 pm    Post subject: Reply with quote

Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Permanent link to the script: http://mirror.slackware.hr/slackware/slackware64-15.0/source/make_world.sh
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Fri Feb 11, 2022 9:28 pm    Post subject: Reply with quote

Hu wrote:
pjp wrote:
nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)
I have been summoned.

:)
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Sat Feb 12, 2022 3:20 am    Post subject: Reply with quote

Leonardo.b wrote:
Hu wrote:
pjp wrote:
nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)
I have been summoned.

:)
++


Hu wrote:
First non-comment line is not set -eu or similar.
Or similar? Because you've mentioned using -eu, I've been doing that for a while now. I won't go line by line, but I've also tried to quote everything. Thanks for the script reviews that you do. I try to absorb what I can.


Leonardo.b wrote:
Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.
Is there a better way than using 'sudo -u ... -g ... /bin/sh -c ...'? su probably works I suppose... not sure why I went with sudo, maybe habit.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Sat Feb 12, 2022 5:47 pm    Post subject: Reply with quote

set -o pipefail can also be a good option, in addition to set -eu, but is slightly more situational. There may be cases where a non-rightmost process is expected to "fail" (exit nonzero) even when it does everything that it should, so adding pipefail has a slightly higher risk of breaking scripts. In a new script, I would almost certainly set it. When reviewing someone else's existing script, I might not, since determining whether it is suitable requires an analysis of the pipelines used.

In my opinion, it is far more common that set -e is a good thing than that it is a bad thing. I believe that most scripts are safer with set -e, and that if they terminate early because a child process exited with an unhandled error[1], that terminating is almost always better than continuing on after a child process reports failure.

[1]: set -e roughly implies || exit $? after the statement. If the script explicitly detects the failure, it can survive a failed command.
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Sat Feb 12, 2022 8:38 pm    Post subject: Reply with quote

pjp wrote:
Leonardo.b wrote:
Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.
Is there a better way than using 'sudo -u ... -g ... /bin/sh -c ...'? su probably works I suppose... not sure why I went with sudo, maybe habit.

Boh. I don't know. su is granted to be installed for everyone.

To drop privileges, it is also required to tweak makepkg.
Currently, it does not support creating packages as user (tar --owner instead of chmod?).

Are you interested in using the scripts yourself?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20048

PostPosted: Tue Feb 15, 2022 8:44 pm    Post subject: Reply with quote

Leonardo.b wrote:
Are you interested in using the scripts yourself?
Not this one specifically. Since you mentioned dropping privileges, I thought I'd ask if there was a better way than I've been doing it.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
mid-kid
n00b
n00b


Joined: 24 Aug 2014
Posts: 16

PostPosted: Fri Apr 08, 2022 9:09 pm    Post subject: Reply with quote

Slackware's source is a collection of standalone scripts with no inter-dependent working, except when sometimes one script calls another.
They're all written with many shell-y assumptions in mind, as many, many of the scripts in various source tarballs will break with spaces and special characters in path names, anyway. It's assumed you run them in a sane environment, and make_world.sh additionally assumes you're running on a system you don't mind clobbering up, as some scripts will remove packages or do other things during compile. (one of the build scripts, linuxdoc-tools, is notorious for using slacktrack, a tool that first "touch"es the entire filesystem and then packages any files with newer timestamps)
Errors are checked when necessary, not throughout the entire script. This is true for many gentoo scripts as well, as it allows more freedom in how errors are handled.

The beauty of slackware scripts is mostly in their brutalist simplicity. They do one thing. No fancy options or flags, no "source"d scripts, PATH modifications or other implied behavior. Everything you see is everything you get. This makes them very easy to edit and reason about.

Leonardo.b wrote:
Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Permanent link to the script: http://mirror.slackware.hr/slackware/slackware64-15.0/source/make_world.sh


This is true for all slackware package building scripts since the dawn of time. Originally because files in the final tarball had to be owned as root, back when fakeroot didn't exist, and fakeroot can't really set some permissions either. Might as well do everything as root instead of juggling with root permissions or whatever. Are you starting to get why it's called "slack"ware yet?

Also, portage executes the src_install() stage as root as well, there's not much to be gained except for preventing build scripts from clobbering /root/.
Back to top
View user's profile Send private message
Leonardo.b
Apprentice
Apprentice


Joined: 10 Oct 2020
Posts: 294

PostPosted: Sat Apr 09, 2022 5:04 pm    Post subject: Reply with quote

mid-kid wrote:
Errors are checked when necessary, not throughout the entire script. This is true for many gentoo scripts as well, as it allows more freedom in how errors are handled.

I like most those scripts that politely insult me, every time I ask them to do odd things.

To ignore an error produced by a command (after set -e) I usually do:
Code:
command || true


Quote:
Originally because files in the final tarball had to be owned as root, back when fakeroot didn't exist, and fakeroot can't really set some permissions either.

The metadatas in the tarball must say root:root; it is enough to ask tar to ignore the ownerships while creating the tarball. No need to really touch the files.
There used to be a version of makepkg able to run as user, somewhere on GitHub.

Quote:
Are you starting to get why it's called "slack"ware yet?

Yes. I don't complain.
Back to top
View user's profile Send private message
mid-kid
n00b
n00b


Joined: 24 Aug 2014
Posts: 16

PostPosted: Sun Apr 10, 2022 11:49 am    Post subject: Reply with quote

Leonardo.b wrote:

The metadatas in the tarball must say root:root; it is enough to ask tar to ignore the ownerships while creating the tarball. No need to really touch the files.


Until very recently (somewhere in the last 3ish years) it still used tar-1.13 for compatibility reasons. I'm not sure if that version supported the option.
But even if it did, I don't think you can use it to set SUID bits, make files owned by different system users, etc...

I've used makepkg through fakeroot before, it usually works fine if you don't need all that, but SlackBuild scripts are expected to be ran as root anyway, in case they do...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum