Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2168
Location: The Peanut Gallery

PostPosted: Tue Oct 03, 2006 12:49 pm    Post subject: Reply with quote

count_zero wrote:
My script looks for packages that require the failed package to be built before proceeding. For instance, continuing with my example above:

if xbitmaps were to fail, the script would perform "equery depends x11-misc/xbitmaps", producing these packages as dependents:
x11-base/xorg-server-1.1.1
x11-apps/xsetroot-1.0.1
x11-libs/openmotif-2.2.3-r9

These aren't automatically removed from the emerge queue. Rather, the output of emerge -p is performed on each of them. If xbitmaps were really needed to proceed, portage would include xbitmaps in the output of emerge -p. However, since I already have an old version of xbitmaps, portage doesn't require that the new version be installed before proceeding with these emerges. Thus, the script only removes packages which must be built before proceeding.

I hope I explained this okay.

Makes perfect sense, thanks!
It seems your approach is stronger than just what depends on this, ie by using the -p flag. Why can't we just use that approach in the bash part of Guenther's script?
@Guenther - thanks for all the work! The simulator results were interesting.
Would we be able to use the algorithm count_zero outlines (ie if APPB depends on FAILED_APP, do an emerge --pretend of APPB to see if it would actually require FAILED_APP to be emerged first, and only if so move APPB to the failed list.)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Tue Oct 03, 2006 9:10 pm    Post subject: Reply with quote

steveL wrote:
ie if APPB depends on FAILED_APP, do an emerge --pretend of APPB to see if it would actually require FAILED_APP to be emerged first, and only if so move APPB to the failed list.


We could - if only Portage dependency information was to be trusted!

It's still the same problem: emerge --pretend also uses Portage's dependency information, which might be wrong. It's not only emerge --emptytree which is affected.

To be fair: Actually, it's not Portage's fault at all!

Not portage makes a mistake in such cases, but the ebuilds simply provide incorrect dependency information.

That is, they specify more dependencies than they really need.

Or - even worse - they do not specify dependencies which they actually do require.

In the first case, nothing disastrous happens (more packages are compiled than necessary), unless that incorrect dependency leads to a circular dependency graph.

In the second case, the missing dependency will go undiscovered as long as the actually required package happens already to be installed by coincidence (or rather due to dependency requirements of other packages which are already installed).

The algorithm in my script will handle both cases, because it is not really dependent on Portage's dependency information. While Portage's information will be used, it actually only serves as a speed-up for the algorithm.

So the problem is clearly in the ebuilds.

Unforunately, incorrectly specified dependencies are not necessarily easy to detect.

It's not always just so that a package says: "I depend on this and that package, period."

Rather there are typically conditional dependencies involved: "If this USE-flag is set, and that USE flag is not set, then I'll require that first dependency. Otherwise, if that third USE flag is set, and ..."

Furthermore, dependencies cannot only be triggered by USE-flags, but also on other types of conditions like package version number ranges. Actually, the full power of the shell programming language can be used to evaluate conditions, which means the complexity of dependency specification is potentially unlimited.

And if that was not complicated enough, the complexity is further increased by the fact that ebuilds can be inherited from other ebuilds, including their dependencies.

Summing up, dependency specifications can get pretty complex and error-prone in ebuilds, and one should not be too surprised if some ebuilds contains incorrect dependency information as a consequence.
Back to top
View user's profile Send private message
Lloeki
Guru
Guru


Joined: 14 Jun 2006
Posts: 437
Location: France

PostPosted: Wed Oct 04, 2006 9:41 am    Post subject: Reply with quote

just to be clear, dependency terminology is as follows:

dependencies are packages that a package needs
reverse dependencies are packages that need this package

out of the blue, here's a basic recursive dep' resolution algorithm, which I bet emerge (and lots of other package managers) uses, or something alike (maybe not recursive, but principle is the same):

Code:
function install(list)
foreach pkg in list
 insert(pkg)
build buildlist


Code:
function listofdeps(pkg)
return deps of pkg


Code:
function insert(pkg,revdep)
if pkg is not in buildlist,
 if pkg has deps,
   foreach listofdeps(pkg) as dep
     if dep not in revdep
      insert(dep,revdep+pkg)
 add pkg to buildlist


example:

let a, b, c, d some packages
a depends on b
b depends on d
c depends on d
d depends on a

so we have a cycle a>b>d>a, and an innocent package c.

we run install(a c), here's the running pass, done by hand:
Code:
insert(a) is called
 a not in buildlist
 pkg has dep b
 b not in revdep
 insert(b,a) is called
  b not in buildlist
  pkg has dep d
  d not in revdep
  insert(d,a b) is called
   d not in buildlist
   pkg has dep a
   a is in revdep, a ignored
   add d to buildlist
   buildlist is d
   return
  add b to buildlist
  buildlist is d b
  return
 add a to buildlist
 buildlist is d b a
 return
insert(c)
 pkg has dep d
 d in buildlist, ignored
 add c to buildlist
 build list is d b a c
 return
build d b a c


now run install(c a), you will end up with a different order of a, b and d (it's: b a d c). which one is right? hell if I know. they may even be both right or both wrong. all the fault of a package seemingly not related to the cyclic dependency of a b d. so when you do emerge world, the order of the content of any package of the world file influences the order of cyclic dependencies. now, it's insane to try to twist around a thousand unrelated packages just to build things in the right order.

what the algorithm does inside, is building an acyclic oriented graph, and move recursively inside it, building the buildlist along the path.

with graph theory, it is provable that:
1. each package not in a dep cycle will have all of its deps in the right order
2. it is impossible to solve circular deps with the given information, as it results in an infinite loop (thus the passage of redep, which eventually breaks the n-cycle at n-1 steps)

the only way to solve cyclicness is to have more information. this specific information is the extra bit of contextual knowledge we have over the machine:
- linux-headers do not compile, so they can be installed before gcc
- you already have some other gcc that can build glibc, so glibc depends on gcc in a less important manner gcc depends on glibc
this will result in the instinctive order of linux-headers glibc gcc. so how did we solve cyclicness inn the end? we didn't. cyclicness is unsolvable. what we added is more information to the graph, rendering it somehow acyclic. what we did is prevailing one path over another on the graph. we said one was dependent in a 'less important manner', so what we did is that we weighted the graph. by assigning a weight, a value, a priority to each dependency of a package, we could sum them while walking the graph, and take the path with the bigger (smaller, whatever distinctive) priority, thus favoring an order.
this way, install(a c) and install(c a) will yield the same, and right, order of a b and d.

so, "if only Portage dependency information was to be trusted" is a bit harsh ;)
in the end, emerge does a great job dealing with dependencies. it just lacks information.
_________________
Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F*** ;)


Last edited by Lloeki on Wed Oct 04, 2006 2:13 pm; edited 1 time in total
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2168
Location: The Peanut Gallery

PostPosted: Wed Oct 04, 2006 12:28 pm    Post subject: Reply with quote

Well, this is all really informative. In terms of implementation, it comes down to doing the best we can. In this case I reckon that comes down to using emerge --pretend, as that's the best info we have. After all it takes into account the user's current USE flag set, and does give a list of what we'll need to pull in as against what we might.

I prefer the algorithm in count_zero's script so I reckon I'll try building the list (for ordering) with Guenther's script and then doing the actual install with count_zero's.

One question: what does portage do if it gets a circular dependency like you both outline? Just that I've never seen it happen. It's not hard to detect such a situation in any case.
Back to top
View user's profile Send private message
Lloeki
Guru
Guru


Joined: 14 Jun 2006
Posts: 437
Location: France

PostPosted: Wed Oct 04, 2006 2:23 pm    Post subject: Reply with quote

I don't know, seeing how emerge -pve gcc and emerge -pve glibc give out the same result. Anyway, emerge has no other choice than to break the cycle, whether it does it haphazardly or willingly at some place is beyond my knowledge: a look at the source would be much helpful to understand the black magic, but I'm fairly new (albeit learning) to python.

Guenther, have you considered python as a language for your script? this is really a great language (even if I miss curly braces: they really help in readability, all the more with pair highlighting in IDEs), and one certainly has to worry if an upgrade breaks python, since it will break portage too... so yes, relying on perl is not the thing to do, but python may be a good choice.
_________________
Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F*** ;)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Wed Oct 04, 2006 11:35 pm    Post subject: Reply with quote

Lloeki wrote:
Guenther, have you considered python as a language for your script?


Well, yes and no: I have no experience with python, while I have much experience with Perl.

Another language to learn... where learning a new language is usually the smallest obstacle: Learning the associated libraries/APIs and run-time-environment is the real challenge.

For instance JAVA: The core language is very small and can easily be learned in a couple of hours (especially with experience in C++). But the wealth of runtime libraries available surely need considerable time to master.

And I'm afraid, the same will hold for python, which also provides numerous runtime-libraries for the programmer to use.

Being a Perl guy, I admit python has a smaller, easier to learn core language, a more consise syntax, and generally looks more elegant than Perl.

I especially like its generators and the fact that all of its runtime libraries are based on exception handling.

But there are also drawbacks.

When first trying python on a Windows host, I encountered many serious problems in its UNICODE support, rendering it almost useless for internationalized applications. (Perhaps they have fixed that since then?)

I also dislike its habit of prefixing/postfixing everything with underscores (such as constructor names) for special-purpose functions. To me, it's rather a hint what features the language designers forgot to think about in the first place, than a well-designed syntactic construct.

Another issue are its rather idiosyncratic string-quotation mechanisms which I also consider not the most elegant solutions to the problem.

Finally, I like to make heavy use of anonymous functions in Perl.

And the lambda functions or python are so crippled that they are not really a replacement for that.

Summing up, I consider python to be good, but not excellent.

BTW, I think JavaScript is a largely underestimated language (regarding its potential).

It has amicably stolen many of the best features of Perl, but avoided to also inherit Perl's bloat. It has a cleaner syntax than Perl which is closer to C or JAVA, and allows easier integration into host applications than most other script language.

And in comparison to other popular scripting languages like LUA, JavaScript provides much more expressive power.

I would really be happy to see JavaScript as a standard shell language instead of the Bourne shell.

But reality is different... ;-)

Lloeki wrote:
so yes, relying on perl is not the thing to do, but python may be a good choice.


Agreed.

But to be honest: I'm just too lazy to learn a new scripting language, only to improve a script which already works and does it's job.

Also, this whole problem only arised after another improvement has been added to the script: Interruption-free operation.

Before that, the script simply stopped as soon as it encountered the first failing package, and the user was free to decide whether to edit the script file and remove any packages (such as dependent packages, a.k.a . "reverse dependencies") from it.

And finally, we still have count zero's excellent script!

I therefore suggest the following approach:

  • For the common cases, where the emerge -auDN world at the beginning of my guide worked without any problems, the current version of my script will do fine. This will usually be the case for installations where administrators run "emerge --sync && emerge -auDN world" in regular intervals anyway in order to keep their systems up-to-date.
  • For more specific cases, such a partially inconsistent systems or if rebuilding some packages can be expected to fail in advance, use count zero's excellent script which uses my script only as a dependency list generator backend.
  • For the hardest cases where nothing else works, use BadPenguin's method of doing a complete stage-1 reinstall.


Always remember: In Gentoo it's all about choices!
Back to top
View user's profile Send private message
gerard82
Advocate
Advocate


Joined: 04 Jan 2004
Posts: 2181
Location: Netherlands

PostPosted: Thu Oct 05, 2006 11:00 pm    Post subject: Reply with quote

Hi all,
I ran the program (script) after having gone through the necessary preliminaries according to the
guide.
The list to be recompiled contained some 800 packages.
I know this is a lot,but sometimes you try something once and leave it in.Pure lazyness on my part
and having 250 GB of which only 8% is used.
Some people wondered how long it would take:
from oct.2 21:17 to oct.5 02:48 hr.
My box: athlon-xp @1447 Mhz,1G RAM IDE HD.
I had a problem after reboot:
The kernel was compiled with gcc 3.4 , the rest with gcc 4.1. so X didn't work.
I recompiled the kernel and then did
Code:
module-rebuild rebuild

When the nvidia-driver finished compiling there was a message on the screen:
Code:
nvidia: Unknown symbol remap_page_range
nvidia: Unknown symbol pci_find_class

and X still couldn't start.
I emerge synced and a new nvidia driver was available: 1.0-8774.
I emerged that one,same error message.
Apparently something like this has happened before to people and they always solved it by emergeing a newer driver.
For me this didn't work as you can see.
I desperation I downloaded the same driver from the nvidia site as a file which is selfextracting
and ran it.No error messages and lo and behold it worked!
The kernel module is now located at different folder.
I would like to know if I did anything wrong,if so what?
Gerard.
Back to top
View user's profile Send private message
lorebett
Tux's lil' helper
Tux's lil' helper


Joined: 08 Oct 2004
Posts: 106
Location: Firenze, Italy

PostPosted: Thu Oct 05, 2006 11:02 pm    Post subject: Reply with quote

Quote:
By the way, the generated script will do this in a recoverable way:
It can be aborted at any time by you, and will continue where it left off
when you re-run it. (The package where the script was interrupted will
have to be compiled again from its beginning, though.)


Isn't there a way to avoid this (The package where the script was interrupted will
have to be compiled again from its beginning)?

This would be quite useful for huge packages, e.g., OpenOffice...
_________________
HOME: http://www.lorenzobettini.it
BLOG: http://tronprog.blogspot.com
MUSIC: http://www.purplesucker.com
Back to top
View user's profile Send private message
gerard82
Advocate
Advocate


Joined: 04 Jan 2004
Posts: 2181
Location: Netherlands

PostPosted: Thu Oct 05, 2006 11:09 pm    Post subject: Reply with quote

One more remark:
Now when it boots X doesn't start until eth0 is finished.
It used to start before the starting of eth0 was visible on VT1.
Gerard
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Oct 06, 2006 1:38 am    Post subject: Reply with quote

Gerard van Vuuren wrote:
I had a problem after reboot:
The kernel was compiled with gcc 3.4 , the rest with gcc 4.1. so X didn't work.


This should normally not be a problem, because the difference between gcc 3.4 and 4.1 is only significant for C++ programs and the whole kernel is written in C only (not C++).

Hoewever, who knows what sort of object code binary drivers contain... perhaps the guys at NVIDIA actually did use C++, and require linkage against C++ runtime libraries.

This will then result in a requirement to re-emerge the NVIDIA drivers.

Same is true if the kernel is recompiled and the option has been enabled for the kernel build process to store version information within the modules.

Gerard van Vuuren wrote:
When the nvidia-driver finished compiling there was a message on the screen:
Code:
nvidia: Unknown symbol remap_page_range
nvidia: Unknown symbol pci_find_class

and X still couldn't start.


I remember having had the same problem a couple of months ago.

The reason for the error message was quite simple: The NVIDIA-drivers I was using at that time were outdated, and so a driver update was required in order for the NVIDIA drivers to work with the new kernel. (The "unknown symbol" problems arise from the fact that the NVIDIA drivers wants to use old kernel functions which have been removed from newer versions of the kernel.)

So it's neither a problem of Portage nor my script; you just need to update your NVIDIA drivers.

Gerard van Vuuren wrote:
I emerge synced and a new nvidia driver was available: 1.0-8774.
I emerged that one,same error message.


I'm not using binary NVIDIA drivers on the box where I'm writing this posting, so I can't check. But I remember there was an important change in the NVIDIA driver packages a couple of months ago: Before the change, there were two different ebuilds, one for OpenGL/GLX and one for the NVIDIA kernel driver. And since the change, a new ebuild has been provided (with a different package name) which combined both ebuilds into a single one.

Or was it exactly the opposite of what I've written here? Can't remember.

Anyway, look for a those new ebuilds. The NVIDIA Hardware Acceleration Guide has also been updated accordingly, I presume.

Gerard van Vuuren wrote:
For me this didn't work as you can see.


Because you only updated your old driver; but you need to unmerge it and install the completely new driver.

Gerard van Vuuren wrote:
No error messages and lo and behold it worked!
The kernel module is now located at different folder.


The new ebuild will also eliminate that problem.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Oct 06, 2006 1:53 am    Post subject: Reply with quote

Gerard van Vuuren wrote:
One more remark:
Now when it boots X doesn't start until eth0 is finished.
It used to start before the starting of eth0 was visible on VT1.
Gerard


That could be the result of some bug fix in the service startup scripts.

Unless you have modified your X startup files to pass the -nolisten tcp option to the X server, the X server will listen on port 6000 for inbound connection requests from remote applications which want to use your display. (The xauth cookies should normally ensure that only authenticated remote applications can get access to your display.)

But in order to listen to network interfaces, the network drivers must have brought up before. Which means network has to start before X.

BTW, I always use -nolisten tcp in my X startup scripts, as I am not using X via network connections other than through SSH tunnels (which will bypass the port 6000 connection method anyway; i. e. that port 6000 listening is not required at all for SSH X11 forwarding).

Note that X connections via port 6000 are also totally insecure, because they are not encrypted and thus only make sense (if at all) within a completely trusted LAN.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Oct 06, 2006 2:17 am    Post subject: Reply with quote

lorebett wrote:
Isn't there a way to avoid this (The package where the script was interrupted will
have to be compiled again from its beginning)?


Unfortunately, not when using emerge.

The whole emerge/ebuild stuff is essentially nothing else than a bunch of scripts which run the Makefile of a package with the right arguments and environment variable settings.

However, it is sometimes possible to restart a Makefile's actions from "within the middle" if the emerge's actions are done manually.

That is, in order to emerge some package xxx, instead of doing a
Code:
emerge xxx
you can do the following:
Code:
ebuild $(equery which xxx) compile

If there is an error during the compile, you can go into the ebuild's working directory (usually /var/tmp/xxx/work) and run make again there (after fixing the problem), which should continue to compile the remaining source files.

As soon as the compilation finished successfully, you need to run the following commands to complete the job:
Code:
ebuild $(equery which xxx) qmerge

which will install the compiled binaries into your live filesystem
Code:
ebuild $(equery which xxx) clean

which will remove the build directory
Code:
emerge --noreplace xxx

which will register the package in the world list of installed packages
Code:
emerge --ask --clean xxx

which will remove old versions of the package

Unfortunately, this assumes the Makefile of the package has been written in a clean way, which will usually be the case. But there might also be some packages where the above approach will not work; in such cases it is not possible to restart a make "from the middle".
Back to top
View user's profile Send private message
Lloeki
Guru
Guru


Joined: 14 Jun 2006
Posts: 437
Location: France

PostPosted: Fri Oct 06, 2006 8:09 am    Post subject: Reply with quote

Quote:
That could be the result of some bug fix in the service startup scripts.

you might just be right, there's that:
Code:

$ cat /etc/init.d/xdm
(snip)
depend() {
        need localmount

        # this should start as early as possible
        # we can't do 'before *' as that breaks it
        # (#139824) Start after ypbind and autofs for network authentication
        after bootmisc readahead-list ypbind autofs openvpn gpm netmount
        before alsasound net.lo

        # Start before X
        use acpid hald xfs
}
(snip)

so, if you have netmount, which evidently depends on net, x will start after netmount, thus after net.
you can adjust some things in /etc/conf.d/rc, namely RC_NET_STRICT_CHECKING, but htis might badly affect some things.
or remove netmount altogether from startup, or if x really doesn't need netmount but you mount some remote folder, remove netmount from the after line.
_________________
Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F*** ;)
Back to top
View user's profile Send private message
cyclohexan
n00b
n00b


Joined: 11 Nov 2004
Posts: 50
Location: Germany

PostPosted: Thu Oct 12, 2006 7:19 am    Post subject: Reply with quote

Thanks for your guide, Guenther. But one question:
You don't mention /sbin/fix_libtool_files.sh (described here)
Is this tool not necessary, while using your guide?
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Thu Oct 12, 2006 5:32 pm    Post subject: Reply with quote

cyclohexan wrote:
You don't mention /sbin/fix_libtool_files.sh (described here)
Is this tool not necessary, while using your guide?


It's not necessary to run it with my guide. This script searches for outdated libraries, but as my guide will recompile the entire system, no outdated libraries will be left to be found.
Back to top
View user's profile Send private message
Doogman
Apprentice
Apprentice


Joined: 24 Sep 2004
Posts: 241

PostPosted: Thu Oct 12, 2006 7:48 pm    Post subject: Reply with quote

Last weekend, I used the script to update my system to the new GCC and everything seems to be working fine! I especially like the hands-free aspect and it's definately a time saver. No more baby-sitting portage with "--skip-first" to get around the troublesome packages.

Although I did manage to find one package that would trip-up the script: wargus. When you emerge it, the ebuild halts while waiting for the game CD. Luckily it was at the end of the compilation run and I noticed it. I didn't have the game CD handy, however, so I altered the state file of your script to skip ahead a package. Perhaps a "--skip-first" option would be handy after all?
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Oct 13, 2006 6:17 am    Post subject: Reply with quote

Doogman wrote:
the ebuild halts while waiting for the game CD.


Your encountered an interactive ebuild... Hmm.

I'm not sure how to handle this from within a batch script. If there were a --batch switch for emerge or ebuild, I would use it.

I could redirect the output of yes as the standard input for the ebuilds, or just send newline characters.

But that won't help in such cases, because operator action (such as inserting CDs) is actually required; not just pressing Enter.

Doogman wrote:
so I altered the state file of your script to skip ahead a package.


Which is the right thing to do in such cases.

Doogman wrote:
Perhaps a "--skip-first" option would be handy after all?


That's a good point - I'll consider this. But skipping is not exactly what is wanted in this scenario: If a package is skipped by simply incrementing the index from the state file, it is considered as being "compiled and up to date". This means the admin has to keep track about such skipped packages, and manually re-emerge them later.

Perhaps I should create another package list, where manually skipped packages will be recorded as a reference for the administrator later?

However, until I implement something, incrementing the state file entry manually is the easiest way to skip a package.

I'll also post a description of the state file contents to this forum.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Oct 13, 2006 6:58 am    Post subject: Hacking the script - state file contents Reply with quote

Hi all,

Motivated by a posting from Doogman I decided to publish a description of the state file which is maintained automatically by my script for the reference of those who want to hack the script or modify its behaviour.

The state file is based on the same basename and filesystem location as the generated script, but has the string ".state" appended to the basename, and a dot (".") prepended to the basename.

The state file is a normal UNIX format text file.

The format is simple:
  • The first line contains the state file version number. This version number must match the internal version number of the script which is in control of the state file. The script will consider the state file to be outdated if the version numbers do not match, and will ignore its contents.
  • The remaining lines of the state file have only a specific meaning in the context of a specific version number in the first line.


In the following, there is a description of the state file contents for state file version number 1.13:
  • Line 1: The version number as stated above.
  • Line 2: The GCC version which is currently used for compilation. If this does not match the actual default compiler version, the state file contents will be ignored and recompilation of all packaged will restart from the beginning. This is a safeguard for situations where a compiler version update was made before the script finishes its execution in the first place, which should normally not be the case.
  • Line 3: A line containing 3 unsigned integer values, separated by a single space character. The values have the following meaning:

    • Progress counter. This is the item index number of the last package emerged successfully. When re-running the script after an interruption, it will skip all packages with item index numbers less than or equal to this value. This means, an effect similar to emerge's --skipfirst can be emulated by just manually incrementing this state file value before re-running the script.
    • The number of packages recompiled successfully in the current pass of the script.
    • The number of failing packages in the current pass of the script. If this number is nonzero when the current pass of the script is done, another pass will be run for the failing packages. But this will only be done if a least one package has been rebuilt successfully in the last pass. This constraint has been added to eliminate the possibility for infinite loops.


The package index numbers mentioned above refer to a specific package name and version each.

The associations are directly listed in the generated script file in lines starting with the word item.

The format of those lines is simple:

item index package_name_and_version

where index is the item index number, and package_name_and_version is the full package name and package version.
Back to top
View user's profile Send private message
fxtl
n00b
n00b


Joined: 03 Oct 2006
Posts: 21

PostPosted: Sat Oct 21, 2006 9:20 am    Post subject: Reply with quote

(Sorry, this might be in a wrong thread, but I was reinstalling gcc and after that using Guenther's script..)

Ok, I have a really weird problem.. emerge --pretend --update --deep --newuse world wanted to recompile gcc 4.1.1 with "GTK" flag, so I did that :P My system is built with 2006.1/desktop profile, so there was no gcc upgrade, just a reinstall. After rebooting, I noticed that ntfs-3g mounts are no longer working, I get an error message:

"Error: Volume name could not be converted to current locale: Invalid or incomplete multibyte or wide character"

It still mounts the volume, but "ls -l -a" doesn't show anything, not even the dot-directories. The volume is not corrupted, I checked it in Windows. Also kernel's read-only ntfs mount works fine.

And here are two other symptoms that are perhaps related (they appeared at the same time):

While booting:

---
Starting Automounter..
Glib: Cannot convert message: Cannot convert fallback '?' to codeset 'ISO-8859-15'
---

Also when I run emerge --sync, I get partly this kind of output:

---
MOTD brought to you by motd-o-matic, version 0.3

\#162\#145\#143\#145\#151\#166\#151\#156\#147\#040\#146\#151\#154\#145\#040\#154\#151\#163\#164\#040\#056\#056\#056\#040\#144\#157\#156\#145
\#164\#151\#155\#145\#163\#164\#141\#155\#160\#056\#143\#150\#153
...
---

Seems to be some kind of character set problem/Glib problem, whatever 'Glib' is. Emerge and the rest of the system still works fine, these are the only problems I've found so far.

I decided to recompile everything using Guenther's instructions and script, that went without problems, but no help, nothing changed :(

I also checked the Gentoo localization guide and checked my locale settings, they were ok and unchanged. I also ran locale-gen, no help.

A newbie like me can't fix problems like these.. Linux/GCC Gods, please help :)
Back to top
View user's profile Send private message
fxtl
n00b
n00b


Joined: 03 Oct 2006
Posts: 21

PostPosted: Sun Oct 22, 2006 9:58 am    Post subject: Reply with quote

Yes, I got it working by switching to UTF8 and changing the default NLS option in kernel :) (http://www.gentoo.org/doc/en/utf-8.xml)
Back to top
View user's profile Send private message
Doogman
Apprentice
Apprentice


Joined: 24 Sep 2004
Posts: 241

PostPosted: Sun Oct 22, 2006 7:25 pm    Post subject: Reply with quote

Looks like a new setting for interactive ebuilds is being proposed.

http://thread.gmane.org/gmane.linux.gentoo.devel/43195

Maybe this would help with the problem I had with the script being stuck at a game CD ebuild.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2766
Location: Pittsburgh, PA, USA

PostPosted: Mon Oct 23, 2006 4:01 am    Post subject: Reply with quote

Thanks for this script. Its working quite well to upgrade my dual processor amd64 box to gcc-4.1.1.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
allengwinn
n00b
n00b


Joined: 21 Aug 2006
Posts: 3

PostPosted: Thu Oct 26, 2006 2:41 am    Post subject: cyrus-sasl -- the never ending problem package Reply with quote

If you're going to run this script with cyrus-sasl, you might want to read this first and patch. Somebody fix this!

http://bugs.gentoo.org/show_bug.cgi?id=152544
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2766
Location: Pittsburgh, PA, USA

PostPosted: Mon Oct 30, 2006 4:22 am    Post subject: Reply with quote

It was not easy but I got it done. I am not sure exactly why but gcc-config-1.3.14 did not fully activate gcc-4.1. Although it did say gcc-4.1 was default. The problem was parts of kde were still looking in the /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6 folder for libraries and complaining that the ABI was wrong. After installing eselect-compiler-2.0.0_rc2-r1 (said that gcc-3.4.6 was default) and using it to select gcc-4.1 everything worked. I did have this script rebuild my packages again just to be sure and all but 5 packages emerged:

item 894 =sys-apps/net-tools-1.60-r12
item 895 =sys-fs/reiserfsprogs-3.6.19
item 896 =net-fs/nfs-utils-1.0.6-r6
item 897 =dev-libs/cyrus-sasl-2.1.22
item 898 =dev-cpp/gnome-vfsmm-2.12.0

I believe the first 3 are because I am using linux-headers-2.6.18 and I am not sure of the other 2.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
[n00b@localhost]
Apprentice
Apprentice


Joined: 30 Aug 2004
Posts: 211
Location: London, UK

PostPosted: Mon Oct 30, 2006 2:16 pm    Post subject: Reply with quote

I have just used Guenther's script for a second time to recompile my entire system (once for gcc 4.1/gentoo 2006.1 and another time for glibc 2.5). Whilst it was compiling I developed the following script to find out how long it would take to finish.

Code:
#!/bin/bash
# Script to find out how long it will take to finish recompiling
# all the packages in Guenther Brunthaler's recompile-remaining-packages
# script (http://forums.gentoo.org/viewtopic-t-494331.html)
#
# Gary Macindoe
# 29-Oct-2006

export recompile_script="/root/recompile-remaining-packages"

export genlop_c="$(mktemp)" || exit 1

genlop -nc > "${genlop_c}"

# Functions to add dates together

# parse_date <string>
# Takes a lingual date as the first argument and parses
# it into environment variables.
# The variables set by this function are:
# SECS:  the number of seconds
# MINS:  the number of minutes
# HOURS: the number of hours
# DAYS:  the number of days
# WEEKS: the number of weeks
# YEARS: the number of years
# The function returns 0 on success, 1 on failure.
function parse_date () {
   local number token

   export YEARS=0
   export WEEKS=0
   export DAYS=0
   export HOURS=0
   export MINS=0
   export SECS=0

   for token in $1;
      do
      case $token in
         [0-9]*)
            number=$token
            ;;
         year|years)
            export YEARS=$number
            ;;
         week|weeks)
            export WEEKS=$number
            ;;
         day|days)
            export DAYS=$number
            ;;
         hour|hours)
            export HOURS=$number
            ;;
         minute|minutes)
            export MINS=$number
            ;;
         second|seconds)
            export SECS=$number
            ;;
         and)
            ;;
         *)
            echo "Error: Unrecognised token $token"
            return 1
            ;;
      esac
   done

   return 0

}

# add_dates <days1> <hours1> <minutes1> <seconds1> <days2> <hours2> <minutes2> <seconds2>
# Takes two dates and adds them. Carries over seconds, minutes and  hours to minutes, hours and days.
# The variables set by this function are:
# TOTAL_DAYS:  the total number of days
# TOTAL_HOURS: the total number of hours
# TOTAL_MINS:  the total number of minutes
# TOTAL_SECS:  the total number of seconds
# Returns 0 on success, 1 on failure.
function add_dates () {
   if [ $# -ne 8 ];
      then
      return 1
   fi

   export TOTAL_SECS=$[ $[ $4 + $8 ] % 60 ]
   export TOTAL_MINS=$[ $[ $3 + $7 + $[ $[ $4 + $8 ] / 60 ] ] % 60 ]
   export TOTAL_HOURS=$[ $[ $2 + $6 + $[ $[ $3 + $7 + $[ $[ $4 + $8 ] / 60 ] ] / 60 ] ] % 24 ]
   export TOTAL_DAYS=$[ $1 + $5 + $[ $[ $2 + $6 + $[ $[ $3 + $7 + $[ $[ $4 + $8 ] / 60 ] ] / 60 ] ] / 24 ] ]

   return 0;

}

# format_date <days> <hours> <minutes> <seconds>
# Takes a date and formats it. Fields that are zero are excluded.
# Returns 0 on success, 1 on failure.
function format_date () {
   local output

   if [ $# -ne 4 ];
      then
      return 1
   fi

   output=""

   if [ $4 -ne 0 ];
      then
      if [ $4 -eq 1 ];
         then
         output=" [1;32;40m$4  [0;37;40msecond"
      else
         output=" [1;32;40m$4  [0;37;40mseconds"
      fi
   fi

   if [ $3 -ne 0 ];
      then
      if [ $3 -eq 1 ];
         then
         if [ "$output" ];
            then
            output=" [1;32;40m$3  [0;37;40mminute and $output"
         else
            output=" [1;32;40m$3  [0;37;40mminute"
         fi
      else
         if [ "$output" ];
            then
            output=" [1;32;40m$3  [0;37;40mminutes and $output"
         else
            output=" [1;32;40m$3  [0;37;40mminutes"
         fi
      fi
   fi

   if [ $2 -ne 0 ];
      then
      if [ $2 -eq 1 ];
         then
         if echo "$output" | grep "and" &>/dev/null;
            then
            output=" [1;32;40m$2  [0;37;40mhour, $output"
         elif [ "$output" ];
            then
            output=" [1;32;40m$2  [0;37;40mhour and $output"
         else
            output=" [1;32;40m$2  [0;37;40mhour"
         fi
      else
         if echo "$output" | grep "and" &>/dev/null;
            then
            output=" [1;32;40m$2  [0;37;40mhours, $output"
         elif [ "$output" ];
            then
            output=" [1;32;40m$2  [0;37;40mhours and $output"
         else
            output=" [1;32;40m$2  [0;37;40mhours"
         fi
      fi
   fi

   if [ $1 -ne 0 ];
      then
      if [ $1 -eq 1 ];
         then
         if echo "$output" | grep "and" &>/dev/null;
            then
            output=" [1;32;40m$1  [0;37;40mday, $output"
         elif [ $output ];
            then
            output=" [1;32;40m$1  [0;37;40mday and $output"
         else
            output=" [1;32;40m$1  [0;37;40mday"
         fi
      else
         if echo "$output" | grep "and" &>/dev/null;
            then
            output=" [1;32;40m$1  [0;37;40mdays, $output"
         elif [ $output ];
            then
            output=" [1;32;40m$1  [0;37;40mdays and $output"
         else
            output=" [1;32;40m$1  [0;37;40mdays"
         fi
      fi
   fi

   echo "$output."

   return 0

}

# date-add "<date1>" "[date2]"
# Adds two lingual dates and prints the result. If date2 is omitted it is read from stdin
# Returns 0 on success, 1 on failure.
function date-add () {
   local input input1 input2
   local secs1 mins1 hours1 days1
   local secs2 mins2 hours2 days2

   if [ $# -lt 1 -o $# -gt 2 ];
      then
      echo -e "Usage: $0 <date1> [date2]\nIf date2 is omitted it is read from stdin"
      return 1
   fi

   input1=$( echo $1 | tr -d "[:punct:]" )

   if [ $# -eq 2 ];
      then
      input2=$( echo $2 | tr -d "[:punct:]" )
   else
      input2=$( read $input && echo $input | tr -d "[:punct:]" )
   fi

   if parse_date "$input1";
      then
      secs1=$SECS
      mins1=$MINS
      hours1=$HOURS
      days1=$DAYS
      if parse_date "$input2";
         then
         secs2=$SECS
         mins2=$MINS
         hours2=$HOURS
         days2=$DAYS
         add_dates $days1 $hours1 $mins1 $secs1 $days2 $hours2 $mins2 $secs2
         format_date $TOTAL_DAYS $TOTAL_HOURS $TOTAL_MINS $TOTAL_SECS
      fi
   fi

   return 0

}

# Genlop -c functions

# current_package_name
# Parses the genlop -c output for the name of the package being compiled
function current_package_name () {
   egrep --colour=never "^ \* [a-z]+-[a-z]+/[-_+0-9a-z]+-[.0-9]+[a-z]?(_alpha[0-9]*|_beta[0-9]*|_pre[0-9]*|_rc[0-9]*|_p[0-9]*)?(-r[0-9]+)?" "${genlop_c}" | cut -c4-
}

# current_package_number
# Finds the number of the package currently being compiled from the list of packages to be compiled
function current_package_number () {
   egrep --colour=never "^item [0-9]+ ="$(current_package_name) "${recompile_script}" | gawk '{ print $2 }'
}

# current_package_time_left
# Parses the genlop -c output to find the time left for the package being compiled
function current_package_time_left () {
   egrep --colour=never "^       ETA: " "${genlop_c}" | cut -c13-
}

# total_packages
# Finds the total number of packages to be compiled from the package list
function total_packages () {
   egrep --colour=never "^item [0-9]+ =[a-z]+-[a-z]+/[-_+0-9a-z]+-[.0-9]+[a-z]?(_alpha[0-9]*|_beta[0-9]*|_pre[0-9]*|_rc[0-9]*|_p[0-9]*)?(-r[0-9]+)?" "${recompile_script}" | tail -n1 | gawk '{ print $2 }'
}

# time_left
# Fakes an emerge -ep output into genlop -p to find the time left to compile the rest of the packages in the list
function time_left () {
   local packages_left=$[$(total_packages) - $(current_package_number)]
   (echo -e "\nThese are the packages that would be merged, in order:" &&
   echo -e "\nCalculating dependencies   ... done!\n" &&
   for x in $(egrep --colour=never "^item [0-9]+ =[a-z]+-[a-z]+/[-_+0-9a-z]+-[.0-9]+[a-z]?(_alpha[0-9]*|_beta[0-9]*|_pre[0-9]*|_rc[0-9]*|_p[0-9]*)?" "${recompile_script}" | tail -n$[ ${packages_left} - 2 ] | cut -d' ' -f3 | cut -c2-);
      do
      echo "[ebuild   R   ] $x"
   done) | genlop -np | egrep --colour=never "^Estimated update time: " | cut -d':' -f2 | cut -c2-
}

# if the recompile script cannot be found, exit
if [ ! -r "${recompile_script}" ];
   then
   echo "Cannot read ${recompile_script}"
   rm -rf "${genlop_c}"
   exit 1
fi

# current_package_name returns nothing if there is no emerge in progress
if [ -z "$(current_package_name)" ];
   then
   echo "No emerge in progress"
   rm -rf "${genlop_c}"
   exit 1
fi

echo " [0;37;40mCurrently compiling  [0;31;40m$(current_package_number) [0;37;40m of  [0;31;40m$(total_packages) [0;37;40m:  [0;34;40m$(current_package_name) [0;37;40m"

current_package_time_left="$(current_package_time_left)"

# Replace words with numbers for adding
if [ "${current_package_time_left}" = "less than a minute." ];
   then
   current_package_time_left="1 minute"
elif [ "${current_package_time_left}" = "any time now." ];
   then
   current_package_time_left="0 minutes"
fi

# Add the time left for the current package to the time left for the rest of the package list
time_left=$( date-add "$(time_left)" "${current_package_time_left}" )
echo "Time left: ${time_left}"

# Clean up
rm -rf "${genlop_c}"

(The ^[ characters are generated by ctrl+v esc in vi. I have posted a bzipped version in case copy and pasting doesn't work.)

Example output:
Code:

garyslaptop ~ # recompile-status
Currently compiling 805 of 811: sys-libs/db-4.2.52_p4-r2
Time left: 32 minutes and 47 seconds.


Hope somebody else will find this useful!

Gary
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 7 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum