Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Per-package CFLAGS seem already to be part of Portage!
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Tue Sep 19, 2006 7:17 pm    Post subject: Per-package CFLAGS seem already to be part of Portage! Reply with quote

Hi all,

While searching for a solution how to define per-package CFLAGS and sifting through various postings on that issue, I noticed that a simple solution to the problem seems already to be built into Portage.

And it works right out of the box, without adding any custom scripts to /etc/portage/bashrc!

Well, at least it does in the 2006.1 profile, because the feature is actually a part of the profile, not of Portage itself.

But there are good chances it might work for older profiles as well, because the feature is part of the base profile, which all other profiles are based upon.

I found the feature when examining the files in the various profile directories, where I encountered the file

/usr/portage/profiles/base/profile.bashrc

which obviously sources script files from

/etc/portage/env/category/package

For instance, when I added a file /etc/portage/env/app-text/unix2dos

containing the line

Code:
CFLAGS="-Werror"


then this flag was obviously passed to the compiler during the ebuild's compile phase, because then the compiler bailed out with an error. (This was intended for testing: -Werror makes all warnings errors, so I could notice the difference whether or not my CFLAGS have been taken into account. And it seems they were.)

So, if you want to use different CFLAG/CXXFLAGS/FEATURES (or other variables set in /etc/make.conf) for a specific package, create a text file

/etc/portage/env/category/package

and add your overrides in exactly the same way you would otherwise do in the command line manually.

For instance, instead of doing a
Code:
CFLAGS="-O3" emerge --oneshot gcc

you could instead create a file /etc/portage/env/sys-devel/gcc
containing the line
Code:
CFLAGS="-O3"

and then just use
Code:
emerge --oneshot gcc

in order to achieve the same effect.

The next I will try to do is creating symlinks instead of real files in /etc/portage/env.

This should allow me to share a single file with non-standard stronger optimizations settings in the /etc/portage/env subdirectories without a need to copy the same file into several subdirectories.

It will also be easier to modify the settings if they are stored only in a single place.

I will thus create only a single physical settings file for each group of related options (one file for stronger optimization, another one for buggy packages to be compiled with FEATURES="nostrip", etc), and refer to them only via symlinks from within /etc/portage/env.

Has anyone else already found and used that obviously undocumented system profile feature?

I would really like to learn of any experiences, potential drawbacks, and generally the opinons of others before I make heavy use of this /etc/portage/env feature in my live system!

Anyway, it looks quite promising so far.

What I still have to find out is whether the per-package variables set that way are additive, or whether they replace the default values from the current profile or /etc/make.conf.

UPDATE: I have now written a minimalist ebuild for testing the behaviour when specifying flags in /etc/portage/env.

The result is: The settings in the /etc/portage/env files completely replace all settings from the matching variables in /etc/make.conf.

However, at the time when the /etc/portage/env files are called (are sourced as shell scripts), the /etc/make.conf settings are already available in the respective environment variables.

Which means you can use code like
Code:
CFLAGS="$CFLAGS -O3"
CXXFLAGS="$CXXFLAGS -O3"

in order to add options to the end of the /etc/make.conf options. (As far as I know, later options can override previous mutually-exclusive options in the GCC command line. Thus "-O2 -O3" should be the same as "-O3" alone.)

Alternatively, all the power of the shell's scripting language can be unleashed to filter out / replace unwanted flags from the original /etc/make.conf settings.

Or, just choke on the /etc/make.conf settings and simply specify all the CFLAGS/CXXFLAGS you want to use, completely replacing the /etc/make.conf settings.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32434
Location: 56N 3W

PostPosted: Tue Sep 19, 2006 10:04 pm    Post subject: Reply with quote

Guenther Brunthaler,

According to the guys in #gentoo-portage this is equivelent to modifying your /etc/make.conf with one exception.
When something breaks, the actual CFLAGS used will not be reported in emerge --info

Please do not report bugs that occured while this feature is in use for the failed package.
Clear it first and rebuild with your standard /etc/make.conf CFLAGS.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Tue Sep 19, 2006 10:08 pm    Post subject: Reply with quote

yeah this feature is realy handy , especially when it comes to the EXTRA_ECONF variable available for some ebuilds. Really really handy, I think you said enough about it, but, it's not a well known feature (probably due to what NeddySeagoon said about emerge --info), and it's a shame ! ;=
_________________
The End of the Internet!
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Tue Sep 19, 2006 11:53 pm    Post subject: Reply with quote

NeddySeagoon wrote:
According to the guys in #gentoo-portage this is equivelent to modifying your /etc/make.conf


Thanks! That's exactly that kind of feedback I have been waiting for.

NeddySeagoon wrote:
do not report bugs that occured while this feature is in use for the failed package.
Clear it first and rebuild with your standard /etc/make.conf CFLAGS.


Thanks also for pointing out this important fact.
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 178
Location: Europe/Prague

PostPosted: Sat Dec 02, 2006 12:58 pm    Post subject: Reply with quote

Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 2:59 pm    Post subject: Reply with quote

sliwowitz wrote:
Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*)


Well, not automatically, I'm afraid.

But of course it is possible to write a script to aid in such cases.

I wrote a simple one named "set-optimization-level":
Code:
#! /usr/bin/perl -w
# Adds a list of package descriptions in the format "category/package"
# to the list of heavily optimized packages by creating symlinks
# in /etc/portage/env.
#
# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/set-optimization-level $
# $Author: root $
# $Date: 2006-10-15T11:45:12.625789Z $
# $Revision: 448 $


use strict;


my $outdir= '/etc/portage/env';
my $target= '../heavily-optimized';


my($d);
foreach (@ARGV ? @ARGV : <>) {
   s/^\s*|\s*$//g;
   substr($_, 0, 0)= "$outdir/";
   die unless ($d)= m{^(.+)/[^/]+$};
   mkdir $d or die "Cannot mkdir '$d': $!" unless -d $d;
   if (-l) {
      next if readlink eq $target;
      unlink($_) == 1 || die "Cannot remove old '$_': $!";
   } elsif (-e) {
      # Note: '-l' also works for symlinks where '-e' will fail!
      die "'$_' already exists and is not a symlink";
   }
   symlink $target, $_ or die "Cannot symlink '$_': $!";
   print "symlinked '$_'.\n";
}


In order to use this script, you have to
  • Create a directory "/etc/portage/env".
  • Create a script "heavily-optimized" within that directory. That script should set up the CFLAGS etc as you want them for heavily optimized packages.
  • Run my script for each "category/package"-name you want to be optimized. The script will automatically create the required symlinks. It also recognizes whether a symlink has already been set up and will not complain if you later specify a category/package accidentally again.
  • The category/package specifications should not contain a version specification; just the generic category and package name, such as "x11base/xorg-x11".
  • The script also takes multiple arguments, which means it can be used with xargs.
  • If no arguments are specified, the script expects a categoy/package list on standard input, one package specification per line. Which means it's easy to redirect something like /var/lib/portage/world as input to my script.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 3:09 pm    Post subject: For those who like my script above Reply with quote

Here is my version of the "heavily-optimized" script, and it's counter-part script "debugging" which sets special CFLAGS for debug-only versions of packages.

There is also a script "default" which simply does nothing. I use it to redirect a symlink to a neutral script, if I do not want special optimization for a package. As my script will never re-create or change an existing symlink, it will not overwrite such a symlink with one pointing to "heavily optimized" if I accidentally specify the package as an argument to my script later.

Script "/etc/portage/env/heavily-optimized":
Code:
#! /bin/false
#
# Symlink target for "/etc/portage/env/<category>/<package>".
# Changes compiler options for heavy optimization,
# unless the "debug" USE-flag is set for the package, in which case
# the debugging options will be set instead.


# Whether space-separated string $1 contains token $2.
contains_ncm9q70alywv346gx4ug8hwrq() {
   test "${1#$2}" != "$1" && return
   test "${1%$2}" != "$1" && return
   test "${1% $2 *}" != "$1" && return
   return 1
}
# Check for debugging override.
if contains_ncm9q70alywv346gx4ug8hwrq "$USE" "debug" \
   || contains_ncm9q70alywv346gx4ug8hwrq "$FEATURES" "noclean"
then
   unset -f contains_ncm9q70alywv346gx4ug8hwrq
   . /etc/portage/env/debugging
else
   unset -f contains_ncm9q70alywv346gx4ug8hwrq
   optimizer_filter_ncm9q70alywv346gx4ug8hwrq() {
      local S1 S2 OLD NEW T REST M SET
      S1=1; S2=1
      while true; do
         case $S1 in
            1) # Start variable modification.
               case $S2 in
                  1) OLD="$CFLAGS";;
                  2) OLD="$CXXFLAGS";;
                  3) OLD="$FEATURES";;
                  4) OLD="$USE";;
               esac
               SET=
               case $S2 in
                  1 | 2) SET="-O3 -DNDEBUG -fomit-frame-pointer -fno-stack-check";;
               esac
               OLD="$(echo "$OLD" | tr -s '[[:space:]]' ' ' | sed -e 's/^ *//; s/ *$//')"
               NEW=
               S3=1
               S1=2
               ;;
            2) # Modify next token if necessary.
               if [ -z "$OLD" ]; then
                  S1=3
                  test -n "$SET" && NEW="$NEW${NEW:+ }$SET"
                  continue
               fi
               REST="${OLD#* }"
               if [ "$OLD" = "$REST" ]; then
                  T="$OLD"; OLD=
               else
                  T="${OLD%$REST}"; T="${T% }"; OLD="$REST"
               fi
               M=
               case $S2 in
                  1 | 2)
                     case "$T" in
                        -O[0-9]) M=1;;
                        -g*) M=1;;
                        -DNDEBUG | -DDEBUG | -D_DEBUG) M=1;;
                        -fomit-frame-pointer | -fno-omit-frame-pointer) M=1;;
                        -fstack-check | -fno-stack-check) M=1;;
                        -fno-eliminate-unused-debug-symbols | -feliminate-unused-debug-symbols) M=1;;
                     esac
                     ;;
                  3)
                     case "$T" in
                        nostrip | keeptemp | keepwork | noclean) M=1;;
                     esac
                     ;;
                  4)
                     case "$T" in
                        debug) M=1;;
                     esac
                     ;;
               esac
               if [ -n "$M" ]; then
                  # $T matches a debugging-relevant token.
                  test -z "$SET" && continue
                  T="$SET"; SET=
               fi
               NEW="$NEW${NEW:+ }$T"
               ;;
            3) # Complete variable modification.
               case $S2 in
                  1) CFLAGS="$NEW"; S2=2;;
                  2) CXXFLAGS="$NEW"; S2=3;;
                  3) FEATURES="$NEW"; S2=4;;
                  4) USE="$NEW"; return;;
               esac
               S1=1
               ;;
         esac
      done


   }
   optimizer_filter_ncm9q70alywv346gx4ug8hwrq
   unset -f optimizer_filter_ncm9q70alywv346gx4ug8hwrq
   echo "OVERRIDING FLAGS: heavily-optimized"
   #echo "CFLAGS=$CFLAGS"; echo "CXXFLAGS=$CXXFLAGS"; echo "FEATURES=$FEATURES"; echo "USE=$USE"
fi


Script "/etc/portage/env/debugging":
Code:
#! /bin/false
#
# Symlink target for "/etc/portage/env/<category>/<package>".
# Changes compiler options for source-level debugging.


debugging_filter_ncm9q70alywv346gx4ug8hwrq() {
   local S1 S2 OLD NEW T REST M SET
   S1=1; S2=1
   while true; do
      case $S1 in
         1) # Start variable modification.
            case $S2 in
               1) OLD="$CFLAGS";;
               2) OLD="$CXXFLAGS";;
               3) OLD="$FEATURES";;
               4) OLD="$USE";;
            esac
            SET=
            case $S2 in
               1 | 2) SET="-O1 -DDEBUG -ggdb3 -feliminate-unused-debug-symbols -fno-omit-frame-pointer -fstack-check";;
               3) SET="noclean nostrip";;
               4) SET="debug";;
            esac
            OLD="$(echo "$OLD" | tr -s '[[:space:]]' ' ' | sed -e 's/^ *//; s/ *$//')"
            NEW=
            S3=1
            S1=2
            ;;
         2) # Modify next token if necessary.
            if [ -z "$OLD" ]; then
               S1=3
               test -n "$SET" && NEW="$NEW${NEW:+ }$SET"
               continue
            fi
            REST="${OLD#* }"
            if [ "$OLD" = "$REST" ]; then
               T="$OLD"; OLD=
            else
               T="${OLD%$REST}"; T="${T% }"; OLD="$REST"
            fi
            M=
            case $S2 in
               1 | 2)
                  case "$T" in
                     -O[0-9]) M=1;;
                     -g*) M=1;;
                     -DNDEBUG | -DDEBUG | -D_DEBUG) M=1;;
                     -fomit-frame-pointer | -fno-omit-frame-pointer) M=1;;
                     -fstack-check | -fno-stack-check) M=1;;
                     -fno-eliminate-unused-debug-symbols | -feliminate-unused-debug-symbols) M=1;;
                  esac
                  ;;
               3)
                  case "$T" in
                     nostrip | keeptemp | keepwork | noclean) M=1;;
                  esac
                  ;;
               4)
                  case "$T" in
                     debug) M=1;;
                  esac
                  ;;
            esac
            if [ -n "$M" ]; then
               # $T matches a debugging-relevant token.
               test -z "$SET" && continue
               T="$SET"; SET=
            fi
            NEW="$NEW${NEW:+ }$T"
            ;;
         3) # Complete variable modification.
            case $S2 in
               1) CFLAGS="$NEW"; S2=2;;
               2) CXXFLAGS="$NEW"; S2=3;;
               3) FEATURES="$NEW"; S2=4;;
               4) USE="$NEW"; return;;
            esac
            S1=1
            ;;
      esac
   done
}
debugging_filter_ncm9q70alywv346gx4ug8hwrq
unset -f debugging_filter_ncm9q70alywv346gx4ug8hwrq
echo "OVERRIDING FLAGS: debugging"
#echo "CFLAGS=$CFLAGS"; echo "CXXFLAGS=$CXXFLAGS"; echo "FEATURES=$FEATURES"; echo "USE=$USE"


Script "/etc/portage/env/default":
Code:
#! /bin/false

# This is a symlink dummy target.
# It does absolutely nothing.
#
# Use it to temporarily remove special optimizations
# from some target, without a need to remove the
# target's symlink completely.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 3:21 pm    Post subject: A helper script that might be useful to you Reply with quote

I have also written a simple helper script, which extracts the "category/package"-specifications from the output of "emerge" in a form suitable as input for my "set-optimization-level" script.

Script "/usr/local/sbin/extract-category-and-package":
Code:
#! /usr/bin/perl
# Extracts <category>/<package> from an emerge-generated package list.
#
# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/extract-category-and-package $
# $Author: root $
# $Date: 2006-10-30T20:34:30.295020Z $
# $Revision: 494 $


while (defined($_= <>)) {
   print if s{^.*] (\S*?)-\d.*?$}{$1};
}


This script is handy if you want to optimize a package you are about to emerge as well as all the helper packages it depends on.

For instance, let's assume you wanted to install "games-fps/industri" and make sure it runs fully optimized.

Just do a
Code:
# emerge --pretend --emptytree games-fps/industri | extract-category-and-package \
  | set-optimization-level
and my scripts will mark the package as well as all of its dependencies for subject to heavy optimization.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 3:31 pm    Post subject: Finally another goodie Reply with quote

Always wondered what a specific dependency of a package was good for?

Of course, it is always possible to manually run esearch or a similar utility in order to get a description.

But that's laborious if you want to do this for all the packages in a rather long ist.

That's why I wrote another little filter script for this.

It's called comment-packages, takes category/package list on standard input, writing the same list to standard output, with package descriptions appended to each line.

For instance, to comment all the dependencies of a package "games-fps/industri", just run
Code:
# emerge --pretend --emptytree games-fps/industri \
  | extract-category-and-package | comment-packages


and my script will display:
Code:
virtual/libiconv:Virtual for the GNU conversion library
sys-devel/gettext:GNU locale utilities
sys-libs/gpm:Console-based mouse driver
sys-libs/ncurses:console display library
virtual/libintl:Virtual for the GNU Internationalization Library
sys-apps/texinfo:The GNU info program and utilities
sys-apps/groff:Text formatter used for man pages
sys-devel/gnuconfig:Updated config.sub and config.guess file from GNU
sys-libs/db:Berkeley DB
sys-libs/gdbm:Standard GNU database libraries included for compatibility with Perl
sys-devel/libperl:Larry Wall's Practical Extraction and Reporting Language
dev-lang/perl:Larry Wall's Practical Extraction and Report Language
perl-core/Test-Harness:Runs perl standard test scripts with statistics
app-shells/bash:The standard GNU Bourne again shell
app-admin/perl-cleaner:User land tool for cleaning up old perl installs
perl-core/PodParser:Base class for creating POD filters and translators
perl-core/Test-Simple:Basic utilities for writing tests
virtual/perl-Test-Simple:Virtual for Test-Simple
dev-perl/Locale-gettext:A Perl module for accessing the GNU locale utilities
sys-apps/help2man:GNU utility to convert program --help output to a man page
sys-devel/autoconf-wrapper:wrapper for autoconf to manage multiple autoconf versions
sys-devel/m4:GNU macro processor
sys-devel/autoconf:Used to create autoconfiguration files
sys-devel/automake-wrapper:wrapper for automake to manage multiple automake versions
sys-devel/automake:Used to generate Makefile.in from Makefile.am
sys-apps/man:Standard commands to read man pages
sys-apps/findutils:GNU utilities for finding files
sys-devel/binutils-config:Utility to change the binutils version being used
sys-devel/binutils:Tools necessary to build programs
sys-devel/libtool:A shared library tool for developers
dev-util/pkgconfig:Package config system that manages compile/link flags
x11-misc/util-macros:X.Org autotools utility macros


Caveat: As my script is very simple, it requires that you have the eseach package installed, and will run esearch for each category/package combination in the input list. And as esearch will only list descriptions which it can find, packages without any description will be silently skipped on output by my comment-packages script.

Finally, here is the script.

Script "/usr/local/sbin/comment-packages":
Code:
#! /bin/sh
# Takes a list of <catagory>/<package> and adds ":<comment>".
#
# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/comment-packages $
# $Author: root $
# $Date: 2006-10-30T20:34:30.295020Z $
# $Revision: 494 $


while read PKG; do
   esearch --fullname --nocolor --own '%p:%d\n' "$PKG" | head --lines 1
done
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 5:20 pm    Post subject: Some explanations for my heavily-optimized script Reply with quote

In case you might be wondering what my heavily-optimized and debugging scripts actually do, here is some explanation.

Both scripts work in a similar way: They examine the currently defined CFLAGS, CXXFLAGS, FEATURES and USE settings as specified in you /etc/make.conf.

That is, they will be using the settings from your system if you use the scripts, not mine.

Then the scripts scan those settings for particular items which are considered significant for determining whether we are building an optimized or debugging (or normal) version of a package.

Such items might be CFLAGS such as "-O<something>", "-f-omit-frame-pointer", "-g" etc.

All those items will be removed from the current CFLAGS, FEATURES etc. in the first pass of my script.

Any other settings, such as "-march", will not be touched and be left as they are.

In the second pass my scripts will add the settings which are specific for a "heavily-optimized" or "debugging" build.

For instance, in "heavily-optimized" CFLAGS like "-O3", -f-omit-frame-pointer" and "-DNDEBUG" will be added.

In contrary, the script "debugging" will add CFLAGS like "-O1", "-ggdb", -f-no-omit-frame-pointer" and "-DDEBUG".

If you want to modify the scripts to disable/enable different options, just look at the case with the "SET=" lines: The case specifies which environment variable should be modified (CFLAGS, FEATURES, ...), and the assignment specifies the options to add.

Similarly, the case with those "M=1" statements must be modified to filter out any flags which you want to be modified.

For instance for optimized builds it's a good idea to use CFLAGS "-DNDEBUG" and remove any CFLAGS like "-DDEBUG", "-D_DEBUG".

In order to implement this (which already has been done in my scripts), all those variants have to be filtered out first (in the "M=1"-lines of my script), and then only the variants which refer to optimized builds have to be added in the "SET=" lines of the script.

In any case, it's important that any setting you want to be modified has to have a case in both parts of my script, in the "SET="-section as well as in the "M=1" section.

If you just added an item to the "SET=" section but not also a corresponding case to the "M=1" section, it might happen that both types of CFLAGS are set at the same time: The original ones as well as your modified ones. This will generally be a bad idea.

You might also notice that my script can easily be modified to change items of any environment variable, not just CFLAGS, CXXFLAGS, FEATURES and USE.

In order to modify different environment variables as well, just add more cases to the sections which load and save the environment variables.

You will especially have to change the lines
Code:
case $S2 in
                  1) CFLAGS="$NEW"; S2=2;;
                  2) CXXFLAGS="$NEW"; S2=3;;
                  3) FEATURES="$NEW"; S2=4;;
                  4) USE="$NEW"; return;;
               esac

to something like
Code:
case $S2 in
                  1) CFLAGS="$NEW"; S2=2;;
                  2) CXXFLAGS="$NEW"; S2=3;;
                  3) FEATURES="$NEW"; S2=4;;
                  4) USE="$NEW"; S2=5;;
                  5) OTHERVAR="$NEW"; return;;
               esac
in order to allow modification of $OTHERVAR within the script.

I hope you get the idea.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sat Dec 02, 2006 5:29 pm    Post subject: Oh and I forgot to mention Reply with quote

If my "heavily-optimizing" script detects that the "debug" USE flag is set for a package, it will invoke the "debugging" script automatically instead.

This allows one to toggle beween heavily-optimitzing and debugging via the debug USE flag for all packages on which my set-optimization-level script has been applied already, without a need to change the symlink in such cases.
Back to top
View user's profile Send private message
mark_alec
Bodhisattva
Bodhisattva


Joined: 11 Sep 2004
Posts: 6066
Location: Melbourne, Australia

PostPosted: Sun Dec 03, 2006 7:10 am    Post subject: Reply with quote

Moved from Portage & Programming to Documentation, Tips & Tricks.
_________________
www.gentoo.org.au || #gentoo-au
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 178
Location: Europe/Prague

PostPosted: Sun Dec 03, 2006 10:26 am    Post subject: Reply with quote

Guenther Brunthaler, thanks a lot for your scripts. They are exactly what I was dreaming about. Using them gives you imho even more power than a modified portage bashrc. I would send you some "thumbs up" smileys, but they aren't in forum's assortement...
Back to top
View user's profile Send private message
mudrii
l33t
l33t


Joined: 26 Jun 2003
Posts: 789
Location: Singapore

PostPosted: Tue Dec 05, 2006 10:12 am    Post subject: Reply with quote

nice scripts I will go and try to check it

Regards
_________________
www.gentoo.ro
Back to top
View user's profile Send private message
Element Dave
Tux's lil' helper
Tux's lil' helper


Joined: 10 Nov 2006
Posts: 82

PostPosted: Tue Jan 09, 2007 8:12 am    Post subject: Reply with quote

Guenther Brunthaler wrote:
sliwowitz wrote:
Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*)


Well, not automatically, I'm afraid.

But of course it is possible to write a script to aid in such cases.

I wrote a simple one named "set-optimization-level":
Code:
#! /usr/bin/perl -w
# Adds a list of package descriptions in the format "category/package"
# to the list of heavily optimized packages by creating symlinks
# in /etc/portage/env.
#
# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/set-optimization-level $
# $Author: root $
# $Date: 2006-10-15T11:45:12.625789Z $
# $Revision: 448 $


use strict;


my $outdir= '/etc/portage/env';
my $target= '../heavily-optimized';


my($d);
foreach (@ARGV ? @ARGV : <>) {
   s/^\s*|\s*$//g;
   substr($_, 0, 0)= "$outdir/";
   die unless ($d)= m{^(.+)/[^/]+$};
   mkdir $d or die "Cannot mkdir '$d': $!" unless -d $d;
   if (-l) {
      next if readlink eq $target;
      unlink($_) == 1 || die "Cannot remove old '$_': $!";
   } elsif (-e) {
      # Note: '-l' also works for symlinks where '-e' will fail!
      die "'$_' already exists and is not a symlink";
   }
   symlink $target, $_ or die "Cannot symlink '$_': $!";
   print "symlinked '$_'.\n";
}



Here is a cleaner and "safer" modification of your above script:
Code:
#!/usr/bin/perl -w

use strict;
no warnings qw(uninitialized);

my $outdir= '/etc/portage/env';
my $target= '../heavily-optimized';

while (chomp(my $atom = <>)) {

   next if $atom =~ /^(?:#|\s*$)/;
   $atom =~ s/^\s*|\s*$//g;
   [^\w\+\/-]
   if ($atom !~ /^[\w\+-]+\/[\w\+-]+$/) {
      warn("Skipping invalid package specification: $atom\n");
      next;
   }
   my $category = (split(/\//, $atom))[0];
   my $target_dir = join '/', $outdir, $category;
   if (-!d $target_dir) {
      mkdir($target_dir) or die "Cannot create \"$target_dir\": $!\n";
   }
   my $link = join '/', $outdir, $atom;
   next if -e $link; # don't touch existing links/files
   symlink($target, $link) or die "Cannot symlink $link => $target: $!\n";
   print "symlinked $link => $target\n";
}


I personally prefer to keep everything in a single file in conjunction with a custom /etc/bashrc. I much prefer external scripting languages to bash/shell scripting, so my bashrc does almost nothing itself, instead calling a perl script to do the real work.

BTW, is there an official specification for what characters may appear in an "atom base" [as described in ebuild(5)]; i.e., in the format category/package, without the version? The character classes in the expression [\w\+-]+/[\w\+-]+ are sufficient to match all packages currently in portage, but I'd still like to know for the sake of completeness.


Last edited by Element Dave on Wed Jan 10, 2007 4:12 am; edited 1 time in total
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 3036
Location: The Peanut Gallery

PostPosted: Tue Jan 09, 2007 11:26 am    Post subject: Reply with quote

Thanks Guenther. As ever, you provide the glue to make the system stay up!
Back to top
View user's profile Send private message
swimmer
Veteran
Veteran


Joined: 15 Jul 2002
Posts: 1291
Location: Netherlands

PostPosted: Tue Jan 09, 2007 1:47 pm    Post subject: Reply with quote

Hmm - nvidia-drivers/alsa-driver/ndiswrapper all have problems with sandbox violations during emerge on 2.6.19 kernels so I thought I could use this feature and placed this line in /etc/portage/env/media-sound/alsa-driver:
Quote:
FEATURES="-sandbox"

Same for x11-drivers/nvidia-drivers & net-wireless/ndiswrapper ...

But I still get sandbox violation errors :-/

What am I missing?

Greetz
swimmer
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Tue Jan 09, 2007 7:13 pm    Post subject: Reply with quote

swimmer wrote:

FEATURES="-sandbox"


Does the FEATURES variable support removing single features by prefixing them with "-" at all?

I'm not so sure. It's similar to USE, of course - but does it really work the same? I don't know.

But even if this might be the case in /etc/make.conf or when running emerge from the command line: In /etc/portage/env scripts, variable settings work differently.

Settings in /etc/portage/env-scripts are not cumulative; they are executed by the shell, not by the python parser of portage.

Which means, if you don't want "sandbox" in $FEATURES, just write something like

FEATURES=""

which does not set "sandbox".

However, it might be better to set FEATURES to the same settings as in your /etc/make.conf, and just omit "sandbox" from that list of features.
Back to top
View user's profile Send private message
swimmer
Veteran
Veteran


Joined: 15 Jul 2002
Posts: 1291
Location: Netherlands

PostPosted: Wed Jan 10, 2007 2:18 am    Post subject: Reply with quote

Hi Guenther,

Code:
FEATURES=-sandbox emerge alsa-driver

works flawless - it's more likely another problem ...

Greetz
swimmer
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Jan 10, 2007 5:51 pm    Post subject: Reply with quote

Hi swimmer,

swimmer wrote:
works flawless - it's more likely another problem ...


You are referring to the paragraph
Quote:
but generally it won't, same for most other variables usually found in make.conf as they are used before /etc/portage/env is parsed (which btw isn't done by portage itself).
are you?

Well, then we are out of luck regarding FEATURES in /etc/portage/env I guess!

Too bad. :-(

So we have to look for a better solution it seems.

But at least the /etc/portage/env approach works for normal shell variables like CFLAGS.

BTW, as it seems you have troubles with distcc: Have you ever tried to leave FEATURES=distcc alone and to disable distributed compilation indirectly?

That is, override
Code:
MAKEOPTS=j1
in the /etc/portage/env snippet.

This should serialize compilation even without disabling distcc... perhaps it helps?

However, this can work only if the /etc/portage/env snippet is parsed before make is called... which I don't know.
Back to top
View user's profile Send private message
swimmer
Veteran
Veteran


Joined: 15 Jul 2002
Posts: 1291
Location: Netherlands

PostPosted: Wed Jan 10, 2007 6:27 pm    Post subject: Reply with quote

Hu? I don't know exactly why people think I have problems with distcc but I can assure you I do *not* have any problem with distcc since I don't use it ;-)

And indeed it seems that "FEATURES=-sandbox" is not support in /etc/portage/env :(

Out of luck
swimmer
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Jan 10, 2007 6:55 pm    Post subject: Reply with quote

swimmer wrote:
Hu? I don't know exactly why people think I have problems with distcc


Oops - sorry; I somehow mismatched your posting with those from the referenced thread, where someone was having issues with distcc.

swimmer wrote:
Out of luck


Yes.

Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.

From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.

Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited.

But at least, script languages protect against buffer overruns and similar simple but common errors, which certainly is an advantage.

Anyway, if you ever should happen to find a solution to the per-package FEATURES problem, please let me know by posting here.

I'll do the same.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 3036
Location: The Peanut Gallery

PostPosted: Thu Jan 11, 2007 3:42 am    Post subject: Reply with quote

AFAIC portage should just be written in bash.
Back to top
View user's profile Send private message
Dralnu
Veteran
Veteran


Joined: 24 May 2006
Posts: 1919

PostPosted: Thu Jan 11, 2007 4:20 am    Post subject: Reply with quote

steveL wrote:
AFAIC portage should just be written in bash.
iirc, one of the first versions was in pure bash.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner.
Back to top
View user's profile Send private message
Conan
Guru
Guru


Joined: 02 Nov 2004
Posts: 360

PostPosted: Thu Jan 11, 2007 5:08 am    Post subject: Reply with quote

sourcemages package manager is written in pure bash.

It's kind of silly.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum