Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Update 0.1.4.0_beta10 [git] --toolchain, --perl
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 9, 10, 11  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Sep 05, 2013 11:58 pm    Post subject: couple of bugfixes Reply with quote

Ah I see what you mean now Geki. I've been working on update today, and just pushed some fixes to stuff that was stopping me upgrading. (I could have got round them with emerge, but that would have meant I lost the test-cases to fix the bugs with.)

I'm glad I took the time to go slowly with the harfbuzz thing: it turned out I needed to change the default cacheDir setting, since portage has been md5-cache only for quite a while. That's what was triggering it looking for the SLOT in the ebuild, instead of the cache, which is where it was picking up the comment from (that's sorted now as well, ofc.) So that should make things a bit quicker for all of us.

I think that just got missed, as I remember spending ages waiting for md5-cache to be the default. Also, there was a problem with blockers, which are slotted, which I hit with php. The ebuild was gone from tree, so I spent a while thinking it was another issue with getSlot. The fixes were pretty small in the end, as usual, but that area has always been hard to debug; so I made it a bit easier for next time.

Anyhow, I've just hit it not allowing me to proceed with toolchain upgrade because I need to upgrade my kernel before I can do kernel-headers. Picking up on --exclude is an option, although i feel loathe to add yet more code, when my inclination is to keep the rigour of the check, as it only kicks in if linux-headers is in the list, and upgrading to it would be unsafe. Other than that, if you want to do something past the envelope, you should be doing it directly with emerge. I'll have a think about it, as I get my machine updated.
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2387
Location: Germania

PostPosted: Fri Sep 06, 2013 6:03 am    Post subject: Reply with quote

if I get you right, I just have to mask newer linux-headers. lets see ...
_________________
hear hear
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Fri Sep 06, 2013 1:03 pm    Post subject: Reply with quote

geki wrote:
if I get you right, I just have to mask newer linux-headers.

Yeah that will work, in terms of it not coming up in the list, though I'm not sure it's something you want on a permanent basis.

There is always the SKIP envvar for temporary situations, or ofc the noCompile array in /etc/update if you never want update to handle a package, unless you tell it to explicitly on the command-line (see: update -h config.) So if you just want to avoid linux-headers for one run, you'd use SKIP=sys-kernel/linux-headers update or just export it in the terminal session you're running update from. It sounds more like you want it in noCompile though, and that should definitely work or it's a bug that needs fixing immediately.
Code:
** Skipping sys-kernel/linux-headers-3.9 - in $SKIP

SKIP's been there since the beginning (as has noCompile), and it could no doubt use some tweaking, so if you find a way it could be better, tell me and I'll try to implement it. It was only ever meant as an emergency measure, so it requires the full category: looking at it, it would be more useful if it could skip on package name as well. Back when we added it, we actually had CATX iirc for category exclusions, though we don't any more, and the idea was to do more on skipping, but then we added the dialog (E) to skip stuff, and left SKIP as a "left-string" match; eg:
Code:
export SKIP='dev-perl/ sys-devel/gcc-4.7. dev-lang/python-2'

BTW if you want to sort things out a bit more quickly at whatever point, login to IRC: chat.freenode.net and /join #friendly-coders; my nick is igli. IME it usually helps to chat real-time about what needs changing, do it there and then, and then rework in collaboration with someone who actually needs the use-case; they tend to know exactly what needs to happen, and what is or isn't needed.

Sometimes it's hard to get that right, when you're just making your best effort to guess what would be useful, and experience leads you to be cautious; you can end up implementing a whole chunk of code that no-one ever uses, which is a waste of CPU time as well as your own. ;)

I'm online most of the time; if not, Griz64 is the QA meister for update, though he's really our infra guru, so he knows it inside out and is always on the latest git version (at least half the bugfixes start with him poking me on IRC) and there's a quite a few Gentoo users in there (the people who started it are all #gentoo-chat regulars.)

For ref, if you need to take something out of the command-line target, edit /root/emerge/target (/root is default dir setting); I just did that to get the SKIP output, though I started with update linux-headers glibc (since SKIP is a glob match, targets specified on the cli, are not considered. I guess we could tweak that to always skip exact package name matches.)
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 14, 2013 4:28 pm    Post subject: updating after a few months Reply with quote

Well, I've finally finished the first cut of --toolchain; it's currently in git (along with a few other fixes.) Oh I also enabled update --perl in line with --python and --haskell, since Griz was on-hand to QA it.

My last --deep world, according to /var/log/emerge.log was on January 06, 2013, (ie 8 months ago!) and I've finally set it off again, after upgrading my kernel, since linux-headers were blocked as above (by design), then the following from update --toolchain
Code:

** Resolving --toolchain..

These are the packages that would be merged, in order:

Calculating dependencies  .... ... done!
[ebuild     U  ] sys-devel/gnuconfig-20130111 [20120818] 51 kB
[ebuild   R    ] sys-devel/gcc-config-1.7.3  0 kB
[ebuild   R    ] sys-devel/binutils-config-3-r3  0 kB
[ebuild   R    ] sys-devel/patch-2.6.1  USE="-static {-test}" 0 kB
[ebuild     U  ] sys-devel/autoconf-wrapper-13 [12] 0 kB
[ebuild     U  ] sys-devel/automake-wrapper-9 [7] 0 kB
[ebuild   R    ] sys-devel/m4-1.4.16  USE="-examples" 0 kB
[ebuild   R    ] dev-libs/gmp-5.0.2_p1  USE="cxx -doc -static-libs" 0 kB
[ebuild     U  ] dev-libs/mpfr-3.1.1 [3.0.1_p4] USE="-static-libs%" 0 kB
[ebuild     U  ] dev-libs/mpc-1.0.1 [0.8.2] USE="-static-libs%" 610 kB
[ebuild     U  ] sys-devel/gettext-0.18.2 [0.18.1.1-r3] USE="-acl -cvs cxx -doc -emacs git -java nls openmp -static-libs" 15,330 kB
[ebuild     U  ] sys-devel/make-3.82-r4 [3.82-r3] USE="nls -static" 0 kB
[ebuild     U  ] sys-devel/flex-2.5.37 [2.5.35_p10] USE="nls -static {-test}" 1,273 kB
[ebuild   R    ] sys-devel/bison-2.4.3  USE="nls -static" 0 kB
[ebuild   R    ] dev-lang/perl-5.12.4-r1  USE="berkdb -build -debug -doc gdbm -ithreads" 0 kB
[ebuild     U  ] sys-devel/autoconf-2.69:2.5 [2.68:2.5] USE="-emacs -multislot%" 1,187 kB
[ebuild  NS    ] sys-devel/automake-1.13.4:1.13 [1.11.6:1.11, 1.12.6:1.12] 1,416 kB
[ebuild   R    ] sys-devel/libtool-2.4-r1:2  USE="{-test} -vanilla" 0 kB
[ebuild   R    ] sys-libs/zlib-1.2.7  USE="minizip -static-libs" 0 kB
[ebuild     U  ] dev-lang/python-3.2.5-r2:3.2 [3.2.3:3.2] USE="-build -doc -examples gdbm -hardened% -ipv6 ncurses readline sqlite ssl threads tk (wide-unicode) -wininst xml" 9,020 kB
[ebuild     U  ] sys-devel/binutils-2.23.1 [2.22-r1] USE="cxx -multislot -multitarget nls -static-libs {-test} -vanilla zlib" 20,966 kB
[ebuild     U  ] sys-apps/portage-2.2.1 [2.1.11.50] USE="-build -doc -epydoc (ipc) (-pypy2_0) -python2 -python3 (-selinux) -xattr" LINGUAS="(-pl%) -ru" PYTHON_TARGETS="(-pypy2_0) -python2_6% python2_7%* python3_2%* (-python3_3) (-python3_4)" 881 kB
[ebuild   R    ] sys-devel/gcc-4.6.3:4.6  USE="(-altivec) cxx -doc (-fixed-point) fortran -gcj -graphite gtk (-hardened) (-libssp) -lto% mudflap (multilib) -multislot nls -nopie -nossp nptl -objc -objc++ -objc-gc openmp -regression-test% (-test%) -vanilla" 24 kB
[ebuild     U  ] sys-kernel/linux-headers-3.9 [3.4-r2] 3,614 kB
[uninstall     ] media-sound/alsa-headers-1.0.25
[blocks b      ] media-sound/alsa-headers ("media-sound/alsa-headers" is blocking sys-kernel/linux-headers-3.9)

Total: 24 packages (13 upgrades, 1 in new slot, 10 reinstalls, 1 uninstall), Size of downloads: 54,368 kB
Conflict: 1 block
* 24 in total
! media-sound/alsa-headers-1.0.25 seems to have gone from tree, no changelog in overlays or portage
@ media-sound/alsa-headers: blocks sys-kernel/linux-headers-3.9
* Uninstall media-sound/alsa-headers(1.0.25) before sys-kernel/linux-headers(3.9)
[Toolchain] sys-kernel/linux-headers-3.9 [3.4-r2]
[Toolchain] adding glibc-2.15-r3
[TC] sys-devel/gcc-4.6.3 first built: Sat Feb  9 22:46:54 2013
+ adding dev-libs/gmp-5.0.2_p1: Wed Feb 29 05:15:16 2012 [used by gcc]
[Toolchain] dev-libs/gmp-5.0.2_p1
@ gcc deps changing ABI [requires glibc]
+ dev-libs/mpfr-3.1.1 [required by gcc]
[Toolchain] dev-libs/mpfr-3.1.1 [3.0.1_p4]
+ dev-libs/mpc-1.0.1 [required by gcc]
[Toolchain] dev-libs/mpc-1.0.1 [0.8.2]
* Checking dependencies for sys-devel/gcc-4.6.3..  done.
[Toolchain] sys-devel/gcc-4.6.3
+ adding sys-libs/zlib-1.2.7: Mon Oct  1 03:26:06 2012
[Toolchain] sys-libs/zlib-1.2.7
+ adding sys-devel/gnuconfig-20130111
[Toolchain] sys-devel/gnuconfig-20130111 [20120818]
+ adding sys-devel/gcc-config-1.7.3: Mon Aug 13 06:19:06 2012
[Toolchain] sys-devel/gcc-config-1.7.3
+ adding sys-devel/autoconf-2.69
[Toolchain] sys-devel/autoconf-2.69 [2.68]
+ adding sys-devel/autoconf-wrapper-13
[Toolchain] sys-devel/autoconf-wrapper-13 [12]
+ adding sys-devel/automake-1.13.4
[Toolchain] sys-devel/automake-1.13.4
+ adding sys-devel/automake-wrapper-9
[Toolchain] sys-devel/automake-wrapper-9 [7]
+ adding sys-devel/binutils-config-3-r3: Wed Jun 27 21:50:18 2012
[Toolchain] sys-devel/binutils-config-3-r3
+ adding sys-devel/binutils-2.23.1
[Toolchain] sys-devel/binutils-2.23.1 [2.22-r1]
@ ABI change of sys-devel/binutils [2.22-r1 -> 2.23.1] requires glibc
+ adding sys-devel/flex-2.5.37
[Toolchain] sys-devel/flex-2.5.37 [2.5.35_p10]
+ adding sys-devel/patch-2.6.1: Wed Feb 29 03:00:13 2012
[Toolchain] sys-devel/patch-2.6.1
+ adding sys-devel/m4-1.4.16: Wed Jun 27 21:50:09 2012
[Toolchain] sys-devel/m4-1.4.16
+ adding sys-devel/make-3.82-r4
[Toolchain] sys-devel/make-3.82-r4 [3.82-r3]
+ adding sys-devel/bison-2.4.3: Sun Jul 31 04:35:43 2011
[Toolchain] sys-devel/bison-2.4.3
+ adding sys-devel/libtool-2.4-r1: Tue Oct 11 06:48:08 2011
[Toolchain] sys-devel/libtool-2.4-r1
+ adding sys-devel/gettext-0.18.2
[Toolchain] sys-devel/gettext-0.18.2 [0.18.1.1-r3]
+ adding sys-apps/portage-2.2.1
[Toolchain] sys-apps/portage-2.2.1 [2.1.11.50]
+ adding dev-lang/python-3.2.5-r2
[Toolchain] dev-lang/python-3.2.5-r2 [3.2.3]
+ adding dev-lang/perl-5.12.4-r1: Mon Oct 10 02:01:39 2011
[Toolchain] dev-lang/perl-5.12.4-r1
[Toolchain] adding glibc-2.15-r3 [again]
[TC] 26 toolchain packages
* C compiler will be installed
* OS headers will be installed, followed by libc
* libc will be installed again post-toolchain
@ About to cancel? Y to proceed, E to edit list(N y e) Y
TC: upgrading linux-headers - before glibc
[TC] Upgrading linux-headers from 3.4-r2 to 3.9
^C

update: user cancelled: install sys-kernel/linux-headers
* Terminating...done
I cancelled because it wasn't uninstalling alsa-headers before trying to emerge linux-headers.
This turned out to be an oversight in toolchain handling, so it was important to fix it. And to be sure it worked across resume, since when we have a toolchain target (that is update --toolchain with no named target, much like we default to world/@system @world in normal usage) there is no package data at all in the usual list files. Again this fixed an issue with previous code, since the custom -e handling uses the same setup.

If you use update -r --toolchain then it will only do the toolchain packages in the saved list. Don't forget you can use a numeric flag ie -[0-9] in wildcard terms, for a temporary output run, with its own set of list files etc, exactly as under your main /root/emerge directory (or whatever you set dir to in the config).

Similarly, if you give it a target with --toolchain, it will resolve that target (or targets or sets) and then do only the toolchain packages in that list. You can see above that the default target, assesses the state of all toolchain packages against the compiler. linux-headers upgrades always pull in glibc (as you've probably seen, since that's been the case for a few years.) In the above case, many things which are important to compilation had not been built for ages, let alone with the current compiler (that's what the dates are about. Yes this may be superfluous for *-config, but it's easier not to do a check than bolt-it in afterward.)

compiler deps (gmp, mpfr, mpc by default for gcc) must also have been in place when the compiler was last built, as well as checking for ABI changes (minor upgrades are simply built without triggering a rebuild of gcc.) Other packages are tested against its first-built time, and the sequence is as we define it in our configuration (cf: update -h config.)

libc deps (default to binutils as seen above) changing ABI trigger a further rebuild of glibc. This is the modern equivalent of the old "build everything twice" metaphor applied only to gcc and glibc. You can set libc_deps to an empty value in /etc/update but note that a compiler rebuild as above will still trigger a glibc rebuild.

Sorry, but if you want to play it unsafe, use emerge at the command-line to do it. KK, press E and skip the last glibc (or anything else;)

Hmm the above looks a bit verbose; think I'll swap "Toolchain" with "TC" in the output soon.

[edit:] Remove DEBUG sequence, since it's in next post. typos
update -P does not work correctly; stick with update @preserved-rebuild for now.
typo last update was in Jan 2013, not 2012. ;)


Last edited by steveL on Mon Sep 23, 2013 9:57 pm; edited 5 times in total
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 14, 2013 4:42 pm    Post subject: update --toolchain: resume Reply with quote

So I got the resume working:
Code:
# update -r
** These are the packages in the emerge list:
 Toolchain: Resuming..
[ebuild     U  ] sys-kernel/linux-headers-3.9 [3.4-r2] 3,614 kB
[ebuild   R    ] sys-libs/glibc-2.15-r3  USE="[TC] Automated rebuild"
[ebuild   R    ] dev-libs/gmp-5.0.2_p1  USE="cxx -doc -static-libs" 0 kB
[ebuild     U  ] dev-libs/mpfr-3.1.1 [3.0.1_p4] USE="-static-libs%" 0 kB
[ebuild     U  ] dev-libs/mpc-1.0.1 [0.8.2] USE="-static-libs%" 610 kB
[ebuild   R    ] sys-devel/gcc-4.6.3:4.6  USE="(-altivec) cxx -doc (-fixed-point) fortran -gcj -graphite gtk (-hardened) (-libssp) -lto% mudflap (multilib) -multislot nls -nopie -nossp nptl -objc -objc++ -objc-gc openmp -regression-test% (-test%) -vanilla" 24 kB
[ebuild   R    ] sys-libs/zlib-1.2.7  USE="minizip -static-libs" 0 kB
[ebuild     U  ] sys-devel/gnuconfig-20130111 [20120818] 51 kB
[ebuild   R    ] sys-devel/gcc-config-1.7.3  0 kB
[ebuild     U  ] sys-devel/autoconf-2.69:2.5 [2.68:2.5] USE="-emacs -multislot%" 1,187 kB
[ebuild     U  ] sys-devel/autoconf-wrapper-13 [12] 0 kB
[ebuild  NS    ] sys-devel/automake-1.13.4:1.13 [1.11.6:1.11, 1.12.6:1.12] 1,416 kB
[ebuild     U  ] sys-devel/automake-wrapper-9 [7] 0 kB
[ebuild   R    ] sys-devel/binutils-config-3-r3  0 kB
[ebuild     U  ] sys-devel/binutils-2.23.1 [2.22-r1] USE="cxx -multislot -multitarget nls -static-libs {-test} -vanilla zlib" 20,966 kB
[ebuild     U  ] sys-devel/flex-2.5.37 [2.5.35_p10] USE="nls -static {-test}" 1,273 kB
[ebuild   R    ] sys-devel/patch-2.6.1  USE="-static {-test}" 0 kB
[ebuild   R    ] sys-devel/m4-1.4.16  USE="-examples" 0 kB
[ebuild     U  ] sys-devel/make-3.82-r4 [3.82-r3] USE="nls -static" 0 kB
[ebuild   R    ] sys-devel/bison-2.4.3  USE="nls -static" 0 kB
[ebuild   R    ] sys-devel/libtool-2.4-r1:2  USE="{-test} -vanilla" 0 kB
[ebuild     U  ] sys-devel/gettext-0.18.2 [0.18.1.1-r3] USE="-acl -cvs cxx -doc -emacs git -java nls openmp -static-libs" 15,330 kB
[ebuild     U  ] sys-apps/portage-2.2.1 [2.1.11.50] USE="-build -doc -epydoc (ipc) (-pypy2_0) -python2 -python3 (-selinux) -xattr" LINGUAS="(-pl%) -ru" PYTHON_TARGETS="(-pypy2_0) -python2_6% python2_7%* python3_2%* (-python3_3) (-python3_4)" 881 kB
[ebuild     U  ] dev-lang/python-3.2.5-r2:3.2 [3.2.3:3.2] USE="-build -doc -examples gdbm -hardened% -ipv6 ncurses readline sqlite ssl threads tk (wide-unicode) -wininst xml" 9,020 kB
[ebuild   R    ] dev-lang/perl-5.12.4-r1  USE="berkdb -build -debug -doc gdbm -ithreads" 0 kB
[ebuild   R    ] sys-libs/glibc-2.15-r3  USE="[TC] Automated rebuild"
[TC] 26 toolchain packages
* 26 in total
! media-sound/alsa-headers-1.0.25 seems to have gone from tree, no changelog in overlays or portage
@ media-sound/alsa-headers: blocks sys-kernel/linux-headers-3.9
* Uninstall media-sound/alsa-headers(1.0.25) before sys-kernel/linux-headers(3.9)
[TC] Uninstalling =media-sound/alsa-headers-1.0.25
media-sound/alsa-headers: 1.0.25 none none

All selected packages: media-sound/alsa-headers-1.0.25
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in:  5 4 3 2 1
>>> Unmerging (1 of 1) media-sound/alsa-headers-1.0.25...

TC: upgrading linux-headers - before glibc
[TC] Upgrading linux-headers from 3.4-r2 to 3.9
* Checking gentoo-headers-base-3.9.tar.xz .. [OK]
* Checking gentoo-headers-3.9-1.tar.xz .. [OK]
+ Unpacking gentoo-headers-base-3.9.tar.xz
+ Unpacking gentoo-headers-3.9-1.tar.xz
* Patching:..............done
+ Compiling Source ...
+ Completed compile of linux-headers-3.9
[TC] sys-kernel/linux-headers-3.9 installed successfully.
Packages installed so far: 1 of 1.      25 remaining
[TC] TC: rebuilding sys-libs/glibc- after linux-headers
[TC] Rebuilding sys-libs/glibc-2.15-r3
* Checking glibc-2.15.tar.xz .. [OK]
* Checking glibc-ports-2.15.tar.xz .. [OK]
* Checking glibc-2.15-patches-23.tar.bz2 .. [OK]
% Checking gcc for __thread support ...
% Checking kernel version (3.10.7 >= 2.6.9) ...
% Checking linux-headers version (3.9.0 >= 2.6.9) ...
+ Unpacking glibc-2.15.tar.xz
+ Unpacking glibc-ports-2.15.tar.xz
+ Unpacking glibc-2.15-patches-23.tar.bz2
* Patching:................................................................................................................................done
% Using GNU config files from /usr/share/gnuconfig
+ Compiling Source ...
% Building glibc for ABIs: x86 amd64
%             ABI:   x86
%          CBUILD:   x86_64-pc-linux-gnu
%           CHOST:   x86_64-pc-linux-gnu
%         CTARGET:   x86_64-pc-linux-gnu
%      CBUILD_OPT:   i686-pc-linux-gnu
%     CTARGET_OPT:   i686-pc-linux-gnu
%              CC:   x86_64-pc-linux-gnu-gcc -m32
%         ASFLAGS:
%          CFLAGS:   -pipe -g3 -ggdb -march=native -O2 -fno-strict-aliasing

Sit back and wait and my toolchain is all nicely consistent. At this point info ld stops working so I take the hint, and reboot.

Next I found a bug in revdep library detection (not handling quotes properly due to a typo, but let's take the opportunity to deal with spaces and pick up revdep libs more consistently), and this led to the circular revdep issue; I tracked that down to revdep-rebuild wanting to rebuild libraries with preserved-rebuild libs in them; these are held-over from the last installation, but counted as part of the current (or next) version of the package: so ofc rebuilding them doesn't work since portage continues to preserve the lib, and revdep continues to think it's a dud.

This only applies when you follow the emerge warning to use revdep-rebuild --library '/usr/lib64/libfoo.so.1' && rm .. for example; I think the manual message is to avoid a preserved-rebuild loop. Luckily the 3 packages with preserved libs were precisely the gcc deps, so when the other two came up as the only consumers of libgmp.so.3 I knew something was up (since they were upgraded after it was rebuilt with new(ish;) gcc.) I've raised it in #gentoo-portage and hopefully it'll be fixed soon (revdep should not treat a preserved lib as being a current object.) The python rewrite will be default from next version, so after that's bedded down hopefully this bug will get sorted.

edit: this isn't standard preserved-rebuild message.


Last edited by steveL on Sat Sep 14, 2013 5:46 pm; edited 1 time in total
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 14, 2013 5:42 pm    Post subject: updating after a few months: part 2 Reply with quote

So what came next, once I was sure there was no weird linkage going on with toolchain things? (And I'd got some sleep;-) The usual tree-chasing basically, just a few things compressed into one upgrade, including an improvement in USE flag editing.

Standard update -s came up with a couple of blockers. First off, ncurses got a new tinfo flag, and openrc needed to be compiled with it off (I think; I lost the terminal output, so I'm going off emerge.log and eix output for what's there now.) update ncurses took care of that.

texlive-langgerman-2012 was blocking the upgrade to texlive-2013, despite the new version being in the list, so for the time being I simply used emerge -Cq texlive-langgerman (remember: update doesn't have an uninstall option: we don't shield you from emerge, since if you don't know how to use it properly you will not be able to troubleshoot anything at all.) Note that -cq is the normal uninstall switch, which is dependency aware. However since it's dependency-aware, it won't remove a package that's in-use (ie depended on by another package, or in world/system.)

Additionally sysvinit was holding back openrc and udev (iirc) and util-linux needed upgrading for something else, so I simply hit update sysvinit util-linux to get past those. (update does not install to world unless you tell it to with -i.)

Then I tried update -aU system but only 2 packages showed up (note i'd forgotten -u ;) and it was worse with --bdep, so I just used update system and let it skip useless rebuilds, using the old logic. That got the list down from 40 to 16 packages and meant I could proceed knowing my base system was up to date. (Stuff like bash; again a reboot was in order.)

The next bit was me playing with USE-flags, basically, and seeing what new stuff was coming in. I had a slight issue with qtsql, since the KDE desktop profile now sets the mysql flag on for that package. This was a suspect area, since we had logic to detect "default" in UI (dialog) as what the package is currently coming in with, ie the default if we don't change anything, but the logic for deciding what flags to set is separate, and uses IUSE from the ebuild. I'd wrestled with it before, but came to a hackish compromise that allowed me to at least switch off a profile-set flag. Even if I do force this one off globally at some point. Basically we now always accept an off setting from the user if the package is currently going to have the flag on, according to emerge. Previously (as here) the global being off, would be enough for it to assume the setting would be off.

That needs a bit of refinement, as ever, but it is an iterative improvement on what was there before. (or I would have had to set the flag manually, by editing package.use.) I also fixed up error handling under --jobs: if you skip an ebuild then that can cause emerge to error out. In my case, I have firefox in noCompile, so it only gets updated when I use update firefox, not on every upgrade that comes in (since it takes forever and uses over 15G of disk space.) But the installed firefox had a dependency on sqlite with fts3 USE-flag, which sqlite-3.8.0 does not have. So I was getting an error, only I couldn't see it, apart from "0 Jobs of compiled" and then a die message that no packages were compiled. That's fixed; (I actually only noticed it as I was trying to update udev without lvm2, which didn't work either. So that was my testcase; it was only when I got the error output that I saw the firefox/sqlite issue.)

And off we go:
Code:
** Running: emerge --changed-use --with-bdeps y -uD1 --jobs=3 @system @world
  Emerge  256 Upgrading qtxmlpatterns from 4.8.2 to 4.8.4

* done: dev-vcs/git-1.8.1.5
Jobs: 1 running (255/476 done.) Load: 1.18, 1.29, 1.44
:-)
So my advice is: get the machine to a point where emerge can do update -uU system usually by doing --toolchain first (if it's been a while: that's what it's intended for, as well as ensuring everything is built with the current compiler, headers, libc etc.)
Then see what's stopping the world update and preferably upgrade pieces that are causing conflicts; in the worst case remove a dependency that isn't important (and if you don't know, ask in #gentoo.) You may have to skip an additional package or two: for instance I ended up simply skipping sqlite in addition to firefox, and letting everything else go.

Note skipping does not (usually) mean use of SKIP; it just means deselecting the package from the dialog list. Hit 'e' (or 'E') to see that when it asks if you want to proceed with the build.

And make sure you run dispatch-conf before you reboot, as well as revdep-rebuild. (update makes sure you do that, of course, though you can't always get through revdep-rebuilds, so try to find out what's happening under the hood: man revdep-rebuild. In my case I confirmed the issue with ldd /usr/lib64/libmpfr.so.1 for example, which told me it was the preserved lib that had the linkage to the old gmp lib.)

There are of course things to improve with toolchain handing, especially slotted packages like python, which applies to all types of update. Now we have --perl in place, that's also in toolchain, and they trigger the relevant cleaner/update when they change ABI. ghc is rebuilt when gcc changes ABI. Again, that needs to be added so --haskell will be picked up even when gcc isn't being rebuilt.

Constructive feedback and ideas for improvement welcome.

HTH,
steveL.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 21, 2013 6:50 pm    Post subject: Reply with quote

Minor bump, as I fixed a couple of bugs.

If it's working okay for everyone for the next couple of weeks, I'll wrap the changes into an ebuild. Though we really should set up an overlay, since waiting a year between releases is no good.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Apr 05, 2014 1:18 am    Post subject: Reply with quote

Just so you don't think I'm neglecting this (or an ebuild) I am very close to finishing off --toolchain. The recent perl difficulties, led me to wanting this dependency conflict handled automatically, which entailed handling all of them (3 types that I know about so far) generically, including when ABI warnings happen for packages that aren't in the resultant list. This shows the problem:

Code:
~ # update -aT
** Resolving --toolchain..

These are the packages that would be merged, in order:

Calculating dependencies  ....... ... done!
[ebuild     U  ] sys-devel/gnuconfig-20131128 [20130111] 45 kB
[ebuild   R    ] sys-devel/gcc-config-1.7.3  0 kB
[ebuild   R    ] sys-devel/binutils-config-3-r3  0 kB
[ebuild   R    ] sys-devel/patch-2.6.1  USE="-static {-test}" 0 kB
[ebuild   R    ] sys-devel/autoconf-wrapper-13  0 kB
[ebuild   R    ] sys-devel/automake-wrapper-9  0 kB
[ebuild   R    ] sys-devel/m4-1.4.16  USE="-examples" 0 kB
[ebuild   R    ] dev-libs/gmp-5.1.2  USE="cxx -doc -pgo -static-libs" 0 kB
[ebuild   R    ] dev-libs/mpfr-3.1.2-r1  USE="-static-libs" 0 kB
[ebuild   R    ] dev-libs/mpc-1.0.1  USE="-static-libs" 0 kB
[ebuild     U  ] sys-devel/gettext-0.18.3.2 [0.18.2] USE="-acl -cvs cxx -doc -emacs git -java ncurses%* nls openmp -static-libs" ABI_X86="(-32) (64%*) (-x32)" 15,810 kB
[ebuild   R    ] sys-devel/make-3.82-r4  USE="nls -static" 0 kB
[ebuild   R    ] sys-devel/flex-2.5.37  USE="nls -static {-test}" 0 kB
[ebuild   R    ] sys-devel/bison-2.4.3  USE="nls -static" 0 kB
[ebuild   R    ] dev-lang/perl-5.12.4-r1  USE="berkdb -build -debug -doc gdbm -ithreads" 0 kB
[ebuild   R    ] sys-kernel/linux-headers-3.9  0 kB
[ebuild   R    ] sys-devel/autoconf-2.69:2.5  USE="-emacs -multislot" 0 kB
[ebuild   R    ] sys-devel/autoconf-2.13:2.1  0 kB
[ebuild   R    ] sys-devel/automake-1.13.4:1.13  0 kB
[ebuild   R    ] sys-devel/automake-1.12.6:1.12  0 kB
[ebuild   R    ] sys-devel/libtool-2.4.2:2  USE="-static-libs {-test} -vanilla" 0 kB
[ebuild   R    ] sys-libs/zlib-1.2.8-r1  USE="minizip -static-libs" ABI_X86="(-32) (64) (-x32)" 0 kB
[ebuild   R    ] dev-lang/python-2.7.5-r3:2.7  USE="-berkdb -build -doc -examples gdbm -hardened -ipv6 ncurses readline sqlite ssl threads tk (wide-unicode) -wininst xml" 0 kB
[ebuild     U  ] dev-lang/python-3.3.3:3.3 [3.3.2-r2:3.3] USE="-build -doc -examples gdbm -hardened -ipv6 ncurses readline sqlite ssl threads tk -wininst xml" 11,788 kB
[ebuild     U  ] sys-devel/binutils-2.23.2 [2.23.1] USE="cxx -multislot -multitarget nls -static-libs {-test} -vanilla zlib" 20,949 kB
[ebuild     U  ] sys-apps/portage-2.2.8-r1 [2.2.7] USE="-build -doc -epydoc (ipc) (-pypy2_0) -python2 -python3 (-selinux) -xattr" LINGUAS="-ru" PYTHON_TARGETS="(-pypy2_0) -python2_6 python2_7 -python3_2 python3_3 (-python3_4)" 855 kB
[ebuild   R    ] sys-devel/gcc-4.6.3:4.6  USE="(-altivec) -awt% cxx -doc (-fixed-point) fortran -gcj -graphite (-gtk%*) (-hardened) (-libssp) (-lto%) -mudflap* (multilib) -multislot nls -nopie -nossp nptl -objc -objc++ -objc-gc openmp -regression-test -vanilla" 0 kB
[ebuild   R    ] sys-devel/gcc-4.7.3-r1:4.7  USE="(-altivec) -awt% cxx -doc (-fixed-point) fortran -gcj -go -graphite (-gtk%*) (-hardened) (-libssp) (-lto%) -mudflap* (multilib) -multislot nls -nopie -nossp nptl -objc -objc++ -objc-gc openmp -regression-test -vanilla" 43 kB

Total: 28 packages (5 upgrades, 23 reinstalls), Size of downloads: 49,487 kB

! Dependency conflict:
! conflict: dev-lang/perl:0
! dev-lang/perl:0: dev-lang/perl-5.16.3 blocked [ebuild scheduled for merge]
!! TC: dev-lang/perl-5.16.3 is ABI blocked
! dev-lang/perl: conflict: dev-perl/HTTP-Negotiate-6.0.0 [installed]
! dev-lang/perl: conflict: dev-perl/HTML-Parser-3.690.0 [installed]
! dev-lang/perl: conflict: dev-perl/Net-Daemon-0.480.0 [installed]
! dev-lang/perl: conflict: dev-perl/PlRPC-0.202.0 [installed]
! dev-lang/perl: conflict: dev-perl/libwww-perl-6.30.0 [installed]
! dev-lang/perl: conflict: dev-perl/File-MimeInfo-0.150.0 [installed]
! dev-lang/perl: conflict: dev-perl/WWW-RobotRules-6.10.0 [installed]
* 28 in total
[TC] sys-devel/gcc-4.7.3-r1 first built: Thu Oct 17 14:15:54 2013
* checking gcc-config 4.7.3
[TC] sys-devel/gcc-4.7.3-r1  last built: Sat Dec  7 10:19:57 2013
~ adding dev-libs/gmp-5.1.2: Sat Dec  7 07:49:42 2013 [used by gcc]
[TC] dev-libs/gmp-5.1.2
~ adding dev-libs/mpfr-3.1.2-r1: Sat Dec  7 07:50:58 2013 [used by gcc]
[TC] dev-libs/mpfr-3.1.2-r1
~ adding dev-libs/mpc-1.0.1: Sat Dec  7 07:51:37 2013 [used by gcc]
[TC] dev-libs/mpc-1.0.1
> skipping sys-libs/zlib-1.2.8-r1: Sat Dec  7 10:20:43 2013
> skipping dev-lang/perl-5.12.4-r1: Sat Dec  7 10:26:56 2013
~ adding sys-devel/gnuconfig-20131128
[TC] sys-devel/gnuconfig is being upgraded
[TC] sys-devel/gnuconfig-20131128 [20130111]
> skipping sys-devel/gcc-config-1.7.3: [perl]
> skipping sys-devel/autoconf-2.69: [perl]
> skipping sys-devel/autoconf-2.13: [perl]
> skipping sys-devel/autoconf-wrapper-13: [perl]
> skipping sys-devel/automake-1.13.4: [perl]
> skipping sys-devel/automake-1.12.6: [perl]
> skipping sys-devel/automake-wrapper-9: [perl]
> skipping sys-devel/binutils-config-3-r3: [perl]
~ adding sys-devel/binutils-2.23.2
[TC] sys-devel/binutils is being upgraded
[TC] sys-devel/binutils-2.23.2 [2.23.1]
> skipping sys-devel/flex-2.5.37: Sat Dec  7 10:39:32 2013
> skipping sys-devel/patch-2.6.1: Sun Feb 16 07:00:11 2014
> skipping sys-devel/m4-1.4.16: Sat Dec  7 10:41:01 2013
> skipping sys-devel/make-3.82-r4: Sun Feb 16 07:02:15 2014
> skipping sys-devel/bison-2.4.3: Sat Dec  7 10:42:52 2013
> skipping sys-devel/libtool-2.4.2: Sat Dec  7 10:43:49 2013
~ adding sys-devel/gettext-0.18.3.2
[TC] sys-devel/gettext is being upgraded
[TC] sys-devel/gettext-0.18.3.2 [0.18.2]
> skipping dev-lang/python-2.7.5-r3: Sat Dec  7 10:50:58 2013
~ adding dev-lang/python-3.3.3
[TC] dev-lang/python is being upgraded
[TC] dev-lang/python-3.3.3 [3.3.2-r2]
~ adding sys-apps/portage-2.2.8-r1
[TC] sys-apps/portage-2.2.8-r1 [2.2.7]
[TC] 8 toolchain packages
* python will be installed
@ About to cancel? Y to proceed, E to edit list(N y e)?
update: user cancelled

Note that it's processing the conflicts and flagging perl as a toolchain ABI warning (I added 5.16 and 5.18 to local version). But it's not running the fix yet as perl-5.16 is not in the list. So the perl deps aren't being added either, as emerge is saying it's just going to rebuild 5.12. You might also notice the dates on gcc: it's rebuilding its deps, since they were built just before the last build of gcc, and not with a build of gcc linked with those versions. That was built just after, so this should be the last build of them (for this sequence.) "Confused? You should be." ;)

To actually fix it I need to add support for: clean(virtual/perl-Test-Simple) and perhaps re-resolve; turns out we already had something slightly more complex, for unmerges that happen before a package is installed (which needs to be handled for warning packages as well, ofc.) I didn't write much of this part, but most of it is really crufty, and hasn't been touched for several years. Anyhow I've rewritten some of the old warning code to be more efficient structurally (which I now need to change the rest to using.) It should be more useful for upgrades of old machines as well, once I do, since it will do them in the order in the file, ie chronologically as tree changed.

So not gone, not forgotten :) Error output should use more colour after I committed the setup of them; we were using derived code in shell, so imported it back, and found everything was a tangle of spaghetti so had to clean that up first. The actual ids I've been using for a while, so you should see a few places brighten up (unfortunately mostly errors ofc..:) Once I'm happy with toolchain, I'll switch to working on the installer again.

OFC I'm far behind the tree. I always stop for big fixes like this, that hit my install if I can. It means I'm forced to work on getting the thing working correctly. It also reassures me that update copes with several months between world upgrades, as that's what I usually end up doing.
Back to top
View user's profile Send private message
brownandsticky
n00b
n00b


Joined: 19 Nov 2014
Posts: 4

PostPosted: Wed Nov 19, 2014 9:32 am    Post subject: Terrific Gentoo helper script for package updates Reply with quote

'been using this script for a week or so now and it's awesome. I used my own little script for years but wanted something more polished and complete. This fits the bill nicely.

Many thanks +SteveL

As a stop gap for overlay. Could you possibly make the ebuilds available via git (in app-portage directory)?
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Nov 20, 2014 6:30 pm    Post subject: Reply with quote

brownandsticky wrote:
'been using this script for a week or so now and it's awesome.

Ah thanks mate; that's exactly what motivates me to improve it (flattery;) Well it's actually that people find it useful, as that's the biggest compliment in my book.
Quote:
I used my own little script for years but wanted something more polished and complete. This fits the bill nicely.

Many thanks +SteveL

You're very welcome :-)
Quote:
As a stop gap for overlay. Could you possibly make the ebuilds available via git (in app-portage directory)?

Yeah was going to put the ebuild in the git repo, though I've changed it as well as the makefile for prefix support, so wanted to test it a bit first in case I've broken something.

I can easily make an overlay available; it just seemed odd to do it for literally one package, and even then I'm only really happy supporting the git version, unless someone else is happy to deal with the QA process; bug wrangling principally. Decision-making as to what's required right now, and veto over any changes, belongs to Griz64. If we had an overlay and were putting out versions that people reported on, decision-making could move to votes I guess, but not veto. Somehow I doubt enough people care to make that worth doing.

If we were to put it in an overlay, I'd rather use sys-apps, btw, as it was designed to support all 3 package manglers, from the beginning. But with a git ebuild, that's not necessary (which is why I like it;)

Again, thanks for the positive feedback: it's given me the buzz I need to tackle the last bit of toolchain (putting back a function I deleted a while ago so backup of packages we're about to remove, works again.)
Back to top
View user's profile Send private message
jamtat
Apprentice
Apprentice


Joined: 09 Aug 2003
Posts: 162

PostPosted: Tue Jul 07, 2015 11:22 pm    Post subject: Reply with quote

I found the script discussed in this thread while trying to find a semi-automated way of keeping my Gentoo/MythTV system updated/upgraded. Could I ask that the author of this script take a look at the thread I created at https://forums.gentoo.org/viewtopic-t-1021528.html?sid=6b54a4ec784b5aa3fe0d4ae52a8571bd and offer input as to whether this script could serve my purposes? Thanks in advance for any input on this: I hope this might provide the answer I'm seeking.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Jul 16, 2015 5:24 pm    Post subject: Reply with quote

Sorry I didn't reply sooner; this didn't come up in "My Posts", for some reason.
jamtat wrote:
I found the script discussed in this thread while trying to find a semi-automated way of keeping my Gentoo/MythTV system updated/upgraded. Could I ask that the author of this script take a look at the thread I created and offer input as to whether this script could serve my purposes? Thanks in advance for any input on this: I hope this might provide the answer I'm seeking.

Yes it would; it's designed to be used automated, under the control of another script or program:

The -x flag means "automated", wherein it bails out on many things it would ask a human about; also happens if it's being piped, and there's no terminal in sight.
cf: POSIX test: [ -t 0 ] though we use [[ in bash.

The -H flag means "use colours for the terminal, as you're being run for a Human, even though you're being piped: I'm piping your output to a terminal"; it also implies automated mode.
Note it's undocumented, though discussed here before, and deeper semantics are subject to change, since it's meant for use with chroot installation, and was implemented precisely for our installer. The relevant script variable (easily searchable) is useTerminal.
Having said that, it hasn't changed for years; you can always complain if and when something does change with it, though if you do use it, it's likely better to tell me here as well, so I'll post if we do some more work on the installer that requires such a change in semantic. It's unlikely since its effects are covered by -x, apart from forcing colour display.
Oh I think it doesn't ensure certain deps like dialog either, because it's designed for use in early-install.

Additionally, it has specific exit codes, documented in the script, only about 7 or 8 afair, so that the calling script can know about certain outcomes. Haven't added any for ages, since no-one's ever even mentioned using the ones that are there, let alone ask for others.

Talking of dialog, I quite like the -E flag to go straight to Editing the list, which might be useful in certain situations for a calling script (eg retrying, when you know you're interactive, and the user wants to tweak.)

For more info, run update -h and update --help (which is a different option, not aliasing the first.) You might like update -h error as well, if you plan to do much custom-tweaking. Thank drescherjm for that ;) (the base function does what he mentioned there, or is meant to, afair.)
update -h config will always show you the latest config settings (directly from the script), and their defaults.

HTH,
steveL

(I fixed the topic link above for you; you can edit your post and mod it to use the same; select the text, and click the "Topic" button, or type the tags.)

PS: there's a bug with autounmask when you skip the package, in unstable portage, which is much tighter and does not accept full CPVs, only pkg or pkg:slot, for exclusion. It doesn't affect the majority of exclusions, but thought I should mention it; should be corrected within a week or two, depending on when Griz next shouts at me about it. ;p
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Jul 18, 2015 1:11 pm    Post subject: trac snafu Reply with quote

Just in case anyone tries to register on the trac, sorry but we've closed that for a few days, due to a deluge of fake account spam, till infra sort out a captcha or something they're happy with.

Shame, as I've always hated captchas, but can't see any other real option. In any event, if you need an account (unlikely I know ;) then feel free to pm me, or ask (Monkeh, igli or Griz64) in #friendly-coders on IRC: chat.freenode.net
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2387
Location: Germania

PostPosted: Sat Sep 12, 2015 9:34 am    Post subject: Reply with quote

finally got time to merge git version. set noCompile array in config wrt kernel, its headers, grub and perl to skip these for manual upgrades.

just a great experience updating, thank you again, steveL! :o
_________________
hear hear
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Mon Sep 14, 2015 3:29 pm    Post subject: Reply with quote

Hey you're welcome geki :-)

Thanks for the encouragement. :D
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 9, 10, 11
Page 11 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum