Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP] Calculate your world compile time
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 226
Location: Brazil

PostPosted: Sat Mar 10, 2018 4:50 pm    Post subject: [TIP] Calculate your world compile time Reply with quote

I was struggling very hard to find an utility that calculates the total time I'd compiled all my packages, so I developed a command. Here are the explanation:

1. Get a list of all installed packages without the version at the end.
2. Redirect this to show building times in seconds.
3. Print only building times.
4. Sum all lines of building times.
5. Redirect the sum to a file.
6. Evalute the sum from the file.
7. Print the evaluated sum in HH:MM:SS format.
8. Delete the file to keep the system clean ;-)

Code:
qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime


Mine is 15h:51m:28s right now.


EDIT: steveL did a big enhancement over my command, eliminating steps 3, 5, 6 and 8:
Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'



EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.

1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation:
Code:
qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

2) Discovering the problematic build times (maybe report a bug?). I want to point out which packages build times are bigger than 5400 seconds (1h30min):
Code:
qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'

_________________
Emerging en gentoo


Last edited by rudregues on Tue Apr 10, 2018 5:20 pm; edited 5 times in total
Back to top
View user's profile Send private message
albright
Advocate
Advocate


Joined: 16 Nov 2003
Posts: 2508
Location: Near Toronto

PostPosted: Sun Mar 11, 2018 12:56 am    Post subject: Reply with quote

OK, that is fun

my desktop i7-6700: 16h:36m:10s


my thinkpad x1 carbon (4th gen / i7-6600U): 21h:42m:15s

very similar installations (but distcc probably skews result)
_________________
.... there is nothing - absolutely nothing - half so much worth
doing as simply messing about with Linux ...
(apologies to Kenneth Graeme)
Back to top
View user's profile Send private message
fedeliallalinea
Bodhisattva
Bodhisattva


Joined: 08 Mar 2003
Posts: 19441
Location: here

PostPosted: Sun Mar 11, 2018 8:22 am    Post subject: Reply with quote

You can skip to write in a file
Code:
qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' | eval && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60))

_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5122
Location: The Peanut Gallery

PostPosted: Sun Mar 11, 2018 11:18 am    Post subject: Re: [TIP] Calculate your world compile time Reply with quote

Luke.. use the awk, Luke ;)
Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
(untested, as not in gentoo atm.)
Back to top
View user's profile Send private message
Juippisi
Apprentice
Apprentice


Joined: 30 Sep 2005
Posts: 275
Location: /home

PostPosted: Sun Mar 11, 2018 5:30 pm    Post subject: Reply with quote

Is this one of those cases where "I copied a bash line from the internet and now my / is empty"?

Just kidding,
Code:

# qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
11h:50m:56s


Sure felt a lot longer when I did -e world a while back. i7-2700k with '-march=native -O3 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -flto=8 -fuse-linker-plugin -pipe -fstack-protector-all -mindirect-branch=thunk'
Back to top
View user's profile Send private message
Fitzcarraldo
Veteran
Veteran


Joined: 30 Aug 2008
Posts: 1501
Location: United Kingdom

PostPosted: Sun Mar 11, 2018 6:59 pm    Post subject: Reply with quote

It's a rather academic exercise, though, isn't it? It doesn't take into account one's time spent investigating between failed merges due to blockers and bugs in packages and/or ebuilds. In many cases I have had to spend hours trying to fix a problem and get a package to merge, and its build time may be a few minutes. Over time I have spent more time debugging than the actual time it took to build the packages.

Code:
clevow230ss /home/fitzcarraldo # qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
40h:18m:44s

_________________
Clevo W230SS: amd64, OpenRC, nvidia-drivers & xf86-video-intel.
Compal NBLB2: ~amd64, OpenRC, xf86-video-ati, dual booting with Win 7 Pro 64-bit.
KDE on both laptops.

Fitzcarraldo's blog
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12735

PostPosted: Sun Mar 11, 2018 7:10 pm    Post subject: Reply with quote

Juippisi wrote:
Is this one of those cases where "I copied a bash line from the internet and now my / is empty"?
Any time you see an eval in the program, that's a good question to ask. Although sometimes useful, incautious use of eval is extremely dangerous.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5167

PostPosted: Sun Mar 11, 2018 7:34 pm    Post subject: Reply with quote

Code:
 ~ $ qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
5055h:12m:42s



My output's a bit skewed by this one entry...
Code:
openscenegraph: 18068512 seconds average for 1 merges


Probably related to this bug in qlop
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1346
Location: KUUSANKOSKI, Finland

PostPosted: Sun Mar 11, 2018 8:26 pm    Post subject: Re: [TIP] Calculate your world compile time Reply with quote

steveL wrote:
Luke.. use the awk, Luke ;)
Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
(untested, as not in gentoo atm.)
Works. Returns the exact same time.

My time is 76h:19m:7s... FX-8350 shows its age.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1699

PostPosted: Sun Mar 11, 2018 9:36 pm    Post subject: Reply with quote

Quote:
103h:16m:18s

My poor hamster powered pc :lol:
I bought this box nearly a decade (sic!) ago and it was dirt cheap already.... And I don't really feel like replacing it any time soon. That APU A8-3870 works well enough.
Back to top
View user's profile Send private message
Goverp
Guru
Guru


Joined: 07 Mar 2007
Posts: 589

PostPosted: Mon Mar 12, 2018 10:08 am    Post subject: Reply with quote

What would be more useful is similar arithmetic in emerge --ask to estimate the time for the command in question. Then this script becomes essentially emerge -ea world.
_________________
Greybeard
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1699

PostPosted: Mon Mar 12, 2018 8:31 pm    Post subject: Reply with quote

Goverp, this script completed in 20 seconds, which is significantly faster than 4 days emerge -aev @world would take.
Also, each of us has different @world, different set of USE flags and different hardware which means we simply _cant_ compare results anyway. There is no way to get any meaningful data from it besides "how badly screwed I am when my hard drive dies". And the answer is "you better verify your backups"
Back to top
View user's profile Send private message
Goverp
Guru
Guru


Joined: 07 Mar 2007
Posts: 589

PostPosted: Tue Mar 13, 2018 10:40 am    Post subject: Reply with quote

szatox, I'm not sure what you think I was suggesting, but nothing that would take 4 days. My point was that emerge could do the same arithmetic, and tell you how long the command would take if you say "Yes", thereby at least telling you whether to take a coffee, go out to the cinema, or take a small vacation while portage recompiled whatever.
_________________
Greybeard
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1346
Location: KUUSANKOSKI, Finland

PostPosted: Tue Mar 13, 2018 11:56 am    Post subject: Reply with quote

That would be useful... Also the estimations for each package in the listing, since by calcutaling total time the time for each packge is already known and wouldne add much overhead.

This can be done wia processing --pretend output, but wastes time.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
tuggbuss
Apprentice
Apprentice


Joined: 20 Mar 2017
Posts: 208

PostPosted: Fri Mar 30, 2018 9:28 am    Post subject: Reply with quote

Code:
3h:35m:50s


    i7 - 6900K
    64GB RAM
    NVMe
    Nvidia blob
    Xfce4
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 226
Location: Brazil

PostPosted: Fri Apr 06, 2018 9:17 pm    Post subject: Reply with quote

tuggbuss wrote:
Code:
3h:35m:50s


    i7 - 6900K
    64GB RAM
    NVMe
    Nvidia blob
    Xfce4
This is humiliation...
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6817

PostPosted: Sat Apr 07, 2018 6:30 am    Post subject: Reply with quote

Ant P. wrote:
My output's a bit skewed by this one entry...
Code:
openscenegraph: 18068512 seconds average for 1 merges

Can't beat me :)
Code:
qlop -vgH fuse
fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds
fuse-2.7.4: Sat Sep 19 03:21:32 2009: 22 seconds
fuse-2.8.1: Wed Nov 11 00:44:36 2009: 18 seconds
fuse-2.8.1: Wed Jun  2 23:19:18 2010: 14 seconds
fuse-2.8.1: Wed Sep 15 21:20:45 2010: 34 seconds
fuse-2.8.5: Mon Oct 11 05:01:42 2010: 23 seconds
fuse-2.8.6: Wed Sep 28 02:18:14 2011: 18 seconds
fuse-2.8.6: Mon Dec  5 17:28:47 2011: 23 seconds
fuse-2.8.7: Mon Apr 16 02:45:26 2012: 38 seconds
fuse-2.9.2: Sun Dec  2 17:55:34 2012: 58 seconds
fuse-2.9.3: Sun Dec 22 10:38:25 2013: 15 seconds
fuse-2.9.5: Thu Apr 14 12:07:27 2016: 16 seconds
fuse-2.9.7: Sun Jan 29 21:19:25 2017: 25 seconds
fuse-2.9.7-r1: Sun Mar 25 03:44:15 2018: 9 seconds
fuse-2.9.7-r1: Sun Mar 25 08:28:48 2018: 14 seconds
fuse: 15 times
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 312
Location: Stockholm

PostPosted: Sat Apr 07, 2018 8:17 pm    Post subject: Reply with quote

Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

502h:0m:44s :!:
Linux localhost 4.12.10-gentoo #2 SMP Thu Aug 31 09:20:09 GMT 2017 x86_64 Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz GenuineIntel GNU/Linux
Something is wrong!

[Moderator edit: added [code] tags to preserve output layout. -Hu]
Back to top
View user's profile Send private message
josephg
l33t
l33t


Joined: 10 Jan 2016
Posts: 694
Location: usually offline

PostPosted: Sat Apr 07, 2018 9:41 pm    Post subject: Reply with quote

think i just topped everyone 8O and i run a tight ship too!
Code:
$ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
777h:50m:29s
$ uname -a
Linux localhost 4.9.76-gentoo-r1-josephg #7 SMP Sun Feb 18 20:38:56 GMT 2018 i686 Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz GenuineIntel GNU/Linux

and this is my fastest machine :roll: sigh i don't keep up with the rest of the world
atleast it's faster runtime than my neighbours' shiny new windoze laptops :lol:
_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6817

PostPosted: Sat Apr 07, 2018 10:06 pm    Post subject: Reply with quote

i didn't first output my result because it is just bork, but here it is to make everyone a few more happy with their "tiny" 500-700hours to build :D
Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
341606371735829h:20m:0s
Linux beleg 4.14.11 #2 SMP PREEMPT Wed Jan 10 15:11:16 CET 2018 x86_64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz GenuineIntel GNU/Linux
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6087
Location: Room 101

PostPosted: Sat Apr 07, 2018 10:47 pm    Post subject: Reply with quote

krinn wrote:
Code:
qlop -vgH fuse
fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds

krinn ... hehe, just over 3 million years, that makes you gentoo's oldest user, hows that beard of yours? :)

Code:
% printf "%'.f\n" $((1158050441 / 364))
3,181,457

... well, if you don't calculate leap years.

best ... khay
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2577
Location: Illinois, USA

PostPosted: Sat Apr 07, 2018 11:03 pm    Post subject: Reply with quote

Athlon X4 (bulldozer) Not bad for a $50 cpu
Code:
 qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
2h:17m:38s

EDIT:Can't believe Kaveri is this fast.
Code:
 # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
0h:0m:0s
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 226
Location: Brazil

PostPosted: Mon Apr 09, 2018 9:59 pm    Post subject: Reply with quote

I edited my first post in order to help people with huge build times.

Don't know though what to do with Tony0945's Kaveri...
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
<3
l33t
l33t


Joined: 21 Oct 2004
Posts: 908

PostPosted: Tue Apr 10, 2018 12:04 am    Post subject: Reply with quote

can we make an ebuild of this to put in portage or add this to gentoolkit?
Back to top
View user's profile Send private message
ali3nx
Guru
Guru


Joined: 21 Sep 2003
Posts: 563
Location: Winnipeg, Canada

PostPosted: Tue Apr 10, 2018 2:29 am    Post subject: Reply with quote

dual cpu 10 core hyperthreaded xeon :roll:

Total: 1153 packages (1153 reinstalls), Size of downloads: 0 KiB

Code:
fenrir ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
9h:46m:11s
fenrir ~ # uname -a
Linux fenrir 4.16.0-gentoo #3 SMP Tue Apr 3 03:50:21 CDT 2018 x86_64 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz GenuineIntel GNU/Linux

_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum