Slackware's make_world.sh

[psycho]

This is from the Slackware (15) release announcement:

"For the first time ever we have included a 'make_world.sh' script that allows automatically rebuilding the entire operating system from source".

Has anyone seen the script yet? I'm guessing it'll just be a systematic run through the source tree dumping binaries somewhere for installation, with no centralised patching system (let alone dependency resolution for e.g. rebuilding just part of the OS). On the other hand, if it at least provides some kind of useful mechanism for managing local patches, it could make Slackware even more interesting as a stable platform onto which other stuff can be built from source (it's already one of the best for that, with its policy of sticking as close as possible to upstream and layering on the bare minimum of additional software to make things cohere)...and I guess even if it doesn't do that, the mere provision of a make_world.sh script makes a full source rebuild an officially supported activity now.

Of course the slightest bit of patching, or changes to CFLAGS or whatever, and the official support would likely be withdrawn. Portage is really something, the way it supports such a vast amount of variation: the amount of human time and knowledge coded into Gentoo's package management (including e.g. distributed through all the ebuilds) is staggering. Perhaps unfairly as I haven't seen it yet, I'm thinking Slackware's "make_world.sh" will be to "emerge world" roughly what a honk is to a symphony.

[InvisibleRasta]

Here is the script http://ix.io/3OAd

[Leonardo.b]

I do not trust Slackware shell scripts.
They give me the impression to be sloppy.

[psycho]

Thanks for posting the script InvisibleRasta. LOL...check out line 38:

Code: Select all

# Slackware is not Gentoo.

[ndavilam]

I do not trust Slackware shell scripts.
They give me the impression to be sloppy.

Slackware:
Initial release: July 16, 1993

Gentoo:
Initial release: March 31, 2002

Slackware is a set of hard work and dedication and as a Slackware user and in general vision I have no reason to distrust the build IMHO they are readable enough to be able to afford to fix a line of code yourself. If something is not in your possibilities to solve it, there is a passionate community that helps inexperienced people like me.
But thanks to this community little by little one learns many tricks.

I don't know if you have used Slackware or if you are afraid to use this distribution, if you haven't, I invite you to use it.
Me, I won't say anything about Gentoo because I haven't used it.

[Leonardo.b]

Ok.

[adcdam]

i used both distros, both outanding distros, i used Funtoo first and Gentoo for about 5 years, im using Slackware current on my pc and slackware 15 on my laptop. Next year Slackware will be 30 years old, if the distro isn't as well made it wouldnt lasted almost 30 years.

[pjp]

I do not trust Slackware shell scripts.
They give me the impression to be sloppy.

Anything in particular? I was considering using Slackware as one of my options for a binary distro.

[Leonardo.b]

Nothing against Slackware, I am not unraccomanding it.

This particular build_world.sh does not fit my tastes.
I do not like pkgtools scripts either - those scripts minated my trust.

[pjp]

I was just curious what about them you didn't trust.

I looked at the one linked, and while the style is lacking some space that would help readability, nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)

[Hu]

nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed.

I have been summoned.

Code: Select all

     1	#!/bin/bash
    47	cd $(dirname $0) ; CWD=$(pwd)

• First non-comment line is not set -eu or similar. This may let errors pass undetected later.
• Failed to quote parameter, allowing an error to occur. Due to the lack of set -e, this error will not stop the script. The script will blunder on, running in the wrong directory.

  Code: Select all

  $ cat /tmp/demo/sub\ dir/a.sh
  #!/bin/sh
  
  cd $(dirname $0)
  pwd
  $ bash -x /tmp/demo/sub\ dir/a.sh 
  ++ dirname /tmp/demo/sub dir/a.sh
  + cd /tmp/demo dir
  /tmp/demo/sub dir/a.sh: line 3: cd: too many arguments
  + pwd
  /tmp

• Useless assignment of $CWD. It is never read. Even if it were read, it likely could be skipped through use of $PWD.

Code: Select all

    54	LOGDIR=$TMP/make_world
    56	mkdir -p $LOGDIR/lock

Failed to quote parameter, allowing a surprising and almost certainly wrong result to occur.

Code: Select all

$ TMP=/tmp/demo/sub\ dir/ bash -x a.sh 
+ LOGDIR='/tmp/demo/sub dir//make_world'
+ mkdir -p /tmp/demo/sub dir//make_world/lock

Now dir/make_world/lock exists, and '/tmp/demo/sub dir//make_world' does not.

Code: Select all

    62	  # Make sure this ends in '/':
    63	  if [ ! "$(echo $SLACKWARE_SOURCE_DIRECTORY | rev | cut -b 1)" = "/" ]; then

The script explicitly requests to be run by /bin/bash, so it would not be unreasonable to use a Bash transform here.

Code: Select all

    77	# Be kind, don't hit control-c! If you do, you might leave broken packages,
    78	# logfiles, and locks in $TMP that will cause problems for you later. If you're
    79	# not in a huge hurry to quit, create this file (replace with $TMP if needed):
    80	# /tmp/make_world/lock/abort
    81	# This will cause all instances of make_world.sh to exit when they complete the
    82	# task they are working on.
    83	rm -f $LOGDIR/lock/abort

This seems fragile. It would be better to use a trap handler to recover from a control-c, or to arrange for orphaned logfiles and locks to be recognizable as such by the later run.

Code: Select all

    85	BUILDLIST=${BUILDLIST:-$LOGDIR/buildlist}
    86	if [ ! -r $BUILDLIST -a ! -r ${BUILDLIST}.lock ]; then

Failed to quote argument.

Code: Select all

$ a=/tmp/demo/sub\ dir/a.sh
$ if [ ! -r $a -a ! -r $a.x ]; then echo a; else echo b; fi
-bash: [: too many arguments
b

Code: Select all

    90	  touch ${BUILDLIST} ${BUILDLIST}.lock

Quoting, again. I will stop flagging this particular error now.

Code: Select all

    95	      if [ "$uname -m)" = "x86_64" -a "$(basename $(echo $script | cut -f 1 -d ' '))" = "isapnptools.SlackBuild" ]; then

uname -m could be cached, since this script will not migrate to a different architecture while it runs.

Code: Select all

   107	  # You might want to do this to comment some build scripts out first, or if
   108	  # you'd like to sort it into a "magic build order". ;-) This script is pretty
   109	  # good at just brute-forcing things, though (with a few runs).

That suggests the packages aren't actually built in an enforced safe ordering, so odd failures and wasted work could result.

Code: Select all

   117	  while [ 0 ]; do
   118	    if [ ! -r ${BUILDLIST}.lock ]; then

This could be simplified as while [ -r "$BUILDLIST.lock" ], rather than needing a separate if+break.

Code: Select all

   174	    cat $BUILDLIST | grep -v "^$" | grep -v "^#" | sort | uniq | cut -f 1 -d ' ' | rev | cut -f 1 -d / | rev > $LOGDIR/tmp-pkgs-to-build.$$

Useless Use of Cat. Unnecessary chaining of grep.

Code: Select all

   193	        for findconfigure in */configure ; do
   194	          rm -rf "$(dirname $findconfigure)"
   195	        done
   196	        for findautogen in */autogen.sh ; do
   197	          rm -rf "$(dirname $findautogen)"
   198	        done

Some cleverness with globs and text transforms could collapse all this into a single aggregation loop and a single rm on numerous directories.

Code: Select all

   227	      HELD_BY="$(cat $LOGDIR/lock/lock.global)"

As a Bash script (rather than sh script), this is a Useless Use of Cat. $(< filename) can get the same result.

Code: Select all

   230	        sleep 10

This is a rather long delay. Clever use of inotify tools could allow an instant reaction. Even without that, polling at 1-2 second intervals, without printing on every loop, would probably be fine.

Code: Select all

   235	        if [ ! "${HELD_BY}" = "$(cat $LOGDIR/lock/lock.global)" ]; then
   236	          HELD_BY="$(cat $LOGDIR/lock/lock.global)"

Time of Check / Time of Use. The file ought to be read once, and its value reused, rather than re-read repeatedly.

Code: Select all

   243	    if [ -r $OUTPUT_LOCATION/$(PRINT_PACKAGE_NAME=foo $buildscript | head -n 1) -a $FORCE_BUILD = no ]; then

Checking $FORCE_BUILD is cheaper. It ought to be done before running the build script.

Code: Select all

   247	    # Use flock to only allow one instance of this script to work on a given
   248	    # SlackBuild script at a time.

If we assume use of a working flock, why use a polling loop above instead of just acquiring the lock?

Code: Select all

   260	      # pkgtools, x11, and KDE all trigger the detection above, but none of them
   261	      # really need the global lock. So only request the lock if the build
   262	      # script is not one of those.

That is not particularly confidence inspiring.

Code: Select all

   268	      ( flock 9 || exit 11
   271	      ) 9> $LOGDIR/lock/lock.global

Using a fixed file descriptor is messy. Since use of Bash is assumed, the shell can be asked to pick a free descriptor and provide it.

The descriptor leaks into all the individual build scripts. This might be considered a feature since it keeps the lock held as long as any build script process is alive. It might be considered a bug since it lets those scripts easily write garbage into the lock file.

Code: Select all

   272	      # Remove lock file:
   273	      rm -f $LOGDIR/lock/lock.global

Deleting the lock file here largely negates the benefit of leaking the lock into the individual build scripts.

No comments on the remainder of the script.

[Leonardo.b]

Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Permanent link to the script: http://mirror.slackware.hr/slackware/sl ... e_world.sh

[Leonardo.b]

nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)

I have been summoned.

:)

[pjp]

nothing stood out as immediately bad. I'm no expert on "bad," but at least it wasn't bad enough that I noticed. :)

I have been summoned.

:)

++

First non-comment line is not set -eu or similar.

Or similar? Because you've mentioned using -eu, I've been doing that for a while now. I won't go line by line, but I've also tried to quote everything. Thanks for the script reviews that you do. I try to absorb what I can.

Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Is there a better way than using 'sudo -u ... -g ... /bin/sh -c ...'? su probably works I suppose... not sure why I went with sudo, maybe habit.

[Hu]

set -o pipefail can also be a good option, in addition to set -eu, but is slightly more situational. There may be cases where a non-rightmost process is expected to "fail" (exit nonzero) even when it does everything that it should, so adding pipefail has a slightly higher risk of breaking scripts. In a new script, I would almost certainly set it. When reviewing someone else's existing script, I might not, since determining whether it is suitable requires an analysis of the pipelines used.

In my opinion, it is far more common that set -e is a good thing than that it is a bad thing. I believe that most scripts are safer with set -e, and that if they terminate early because a child process exited with an unhandled error[1], that terminating is almost always better than continuing on after a child process reports failure.

[1]: set -e roughly implies || exit $? after the statement. If the script explicitly detects the failure, it can survive a failed command.

[Leonardo.b]

Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Is there a better way than using 'sudo -u ... -g ... /bin/sh -c ...'? su probably works I suppose... not sure why I went with sudo, maybe habit.

Boh. I don't know. su is granted to be installed for everyone.

To drop privileges, it is also required to tweak makepkg.
Currently, it does not support creating packages as user (tar --owner instead of chmod?).

Are you interested in using the scripts yourself?

[pjp]

Are you interested in using the scripts yourself?

Not this one specifically. Since you mentioned dropping privileges, I thought I'd ask if there was a better way than I've been doing it.

[mid-kid]

Slackware's source is a collection of standalone scripts with no inter-dependent working, except when sometimes one script calls another.
They're all written with many shell-y assumptions in mind, as many, many of the scripts in various source tarballs will break with spaces and special characters in path names, anyway. It's assumed you run them in a sane environment, and make_world.sh additionally assumes you're running on a system you don't mind clobbering up, as some scripts will remove packages or do other things during compile. (one of the build scripts, linuxdoc-tools, is notorious for using slacktrack, a tool that first "touch"es the entire filesystem and then packages any files with newer timestamps)
Errors are checked when necessary, not throughout the entire script. This is true for many gentoo scripts as well, as it allows more freedom in how errors are handled.

The beauty of slackware scripts is mostly in their brutalist simplicity. They do one thing. No fancy options or flags, no "source"d scripts, PATH modifications or other implied behavior. Everything you see is everything you get. This makes them very easy to edit and reason about.

Also, the script runs all the buildings as root; it would be good to drop privileges at a certain point.

Permanent link to the script: http://mirror.slackware.hr/slackware/sl ... e_world.sh

This is true for all slackware package building scripts since the dawn of time. Originally because files in the final tarball had to be owned as root, back when fakeroot didn't exist, and fakeroot can't really set some permissions either. Might as well do everything as root instead of juggling with root permissions or whatever. Are you starting to get why it's called "slack"ware yet?

Also, portage executes the src_install() stage as root as well, there's not much to be gained except for preventing build scripts from clobbering /root/.

[Leonardo.b]

Errors are checked when necessary, not throughout the entire script. This is true for many gentoo scripts as well, as it allows more freedom in how errors are handled.

I like most those scripts that politely insult me, every time I ask them to do odd things.

To ignore an error produced by a command (after set -e) I usually do:

Code: Select all

command || true

Originally because files in the final tarball had to be owned as root, back when fakeroot didn't exist, and fakeroot can't really set some permissions either.

The metadatas in the tarball must say root:root; it is enough to ask tar to ignore the ownerships while creating the tarball. No need to really touch the files.
There used to be a version of makepkg able to run as user, somewhere on GitHub.

Are you starting to get why it's called "slack"ware yet?

Yes. I don't complain.

[mid-kid]

The metadatas in the tarball must say root:root; it is enough to ask tar to ignore the ownerships while creating the tarball. No need to really touch the files.

Until very recently (somewhere in the last 3ish years) it still used tar-1.13 for compatibility reasons. I'm not sure if that version supported the option.
But even if it did, I don't think you can use it to set SUID bits, make files owned by different system users, etc...

I've used makepkg through fakeroot before, it usually works fine if you don't need all that, but SlackBuild scripts are expected to be ran as root anyway, in case they do...