| View previous topic :: View next topic |
| Author |
Message |
mani001
Guru
Joined: 04 Dec 2004
Posts: 485
Location: Oleiros
|
 Posted: Sat Jul 22, 2023 2:37 pm Post subject: Historically increasing compilation times |
 |
|
Hi,
my desktop computer is growing old and I've noticed that compilation times of most packages increase significantly over time. The quintessential example would be given by qtwebengine
| Code: |
Fri Aug 18 20:19:13 2017 >>> dev-qt/qtwebengine-5.7.1-r2
merge time: 42 minutes and 4 seconds.
Mon Jul 10 22:11:46 2023 >>> dev-qt/qtwebengine-5.15.10_p20230623
merge time: 3 hours, 38 minutes and 46 seconds. |
One could argue that browsers (I guess compiling qtwebengine is similar to compiling chromium) have been getting more and more sophisticated in recent years...BUT the same thing can be observed for any package I checked, e.g.,
| Code: | Sat Apr 9 11:56:00 2016 >>> sys-libs/glibc-2.22-r4
merge time: 6 minutes and 20 seconds.
Thu Jun 29 17:27:37 2023 >>> sys-libs/glibc-2.37-r3
merge time: 23 minutes and 42 seconds. |
or
| Code: | Thu Mar 10 21:35:49 2016 >>> x11-base/xorg-server-1.17.4
merge time: 54 seconds.
Fri Jun 23 22:34:48 2023 >>> x11-base/xorg-server-21.1.8
merge time: 2 minutes and 7 seconds. |
I't all good but some years ago I was kind of hoping that in the future packages would compile in almost no time as we switch to more powerful CPUs 
I guess the reason is not only every piece of software getting more and more complex but also compilers getting slower due to more features and/or checks. Any takes on this?
Cheers. |
|
| Back to top |
|
 |
dmpogo
Advocate
Joined: 02 Sep 2004
Posts: 3267
Location: Canada
|
| Posted: Sat Jul 22, 2023 4:44 pm Post subject: |
|
|
The was some recent regression in sandbox-2.32 that increased compilation time at least for some c++ code by almost three times. Sandbox-2.37 which came out recently s supposed to revert some regressive changes, but it is not yet clear if it is a full solution.
See https://bugs.gentoo.org/910273
Saying that, modern compiles indeed look notably slower, trying to do more fancy optimizations
Last edited by dmpogo on Sat Jul 22, 2023 4:48 pm; edited 3 times in total |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Sat Jul 22, 2023 4:45 pm Post subject: |
|
|
You're not the first to notice. One example, although I'm pretty sure qtwebengine has generated others:
qtwebengine taking more than a day to emerge
As for more capable CPUs, that too has been noticed by many. Andy and Bill's law:
what Andy giveth, Bill taketh away or sometimes what Intel giveth, Microsoft taketh away.
That could also be updated to 'javascript websites taketh away."
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
stefan11111
l33t
Joined: 29 Jan 2023
Posts: 922
Location: Romania
|
| Posted: Sat Jul 22, 2023 6:43 pm Post subject: |
|
|
| pjp wrote: |
That could also be updated to 'javascript websites taketh away." |
For that, we have noscript.
_________________
My overlay: https://github.com/stefan11111/stefan_overlay
INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd /usr/lib/modules-load.d *udev* /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus /usr/bin/gdbus /lib/udev" |
|
| Back to top |
|
|
ChrisJumper
Advocate
Joined: 12 Mar 2005
Posts: 2390
Location: Germany
|
| Posted: Wed Jul 26, 2023 9:23 pm Post subject: |
|
|
Its because of the complexity. Companies need to monetize the Internet of things, try to sell information and enhance the complexity and the privacy and data drain through applications, energy usage and try to selling data. A Web page in the 1990 fit on some Floppy Disk and today its lager than old video games like doom.
Sad to say but, yes. You should buy a new computer ;D
Its faster and need less energy too and will have more Memory. And with gentoo it will collect less data. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA
|
| Posted: Sat Jul 29, 2023 8:07 pm Post subject: |
|
|
Don't forget that gcc is getting slower too - from additional optimizations to additional safety checks (like compile time buffer overflow checks) the simplest programs are taking longer to build even with the same source code.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3628
|
| Posted: Sat Jul 29, 2023 8:20 pm Post subject: |
|
|
| eccerr0r wrote: | | Don't forget that gcc is getting slower too - from additional optimizations to additional safety checks (like compile time buffer overflow checks) the simplest programs are taking longer to build even with the same source code. |
Expect binary size to grow as well, most of the time.
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here.
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. "
Last edited by CaptainBlood on Wed Aug 02, 2023 3:44 am; edited 2 times in total |
|
| Back to top |
|
|
ChrisJumper
Advocate
Joined: 12 Mar 2005
Posts: 2390
Location: Germany
|
| Posted: Sat Jul 29, 2023 11:15 pm Post subject: |
|
|
| eccerr0r wrote: | | Don't forget that gcc is getting slower too - from additional optimizations to additional safety checks (like compile time buffer overflow checks) the simplest programs are taking longer to build even with the same source code. |
Thank you eccerr0r,
but it feels like a shitmove. But you are right about it. I am not sure if this is about complexity. Trying to drain information from free software, or about newbies which just use daily programming tools.
First i think about the best case, newbies loving open source and use just commercial programed free tools. Like the Linux Kernel got maintained by 95% of big tech companies or developers working and got payed there. But still share and develop open source software.
I have still hope. And think its a mix between. In some free time try to read, understand and talk about that important code, too!
Personally i am sad about my last post to exist, but it is important and do not trust your (personalized) assistant/friend of your smartphone and computers. Cause it likely now you better, cause on your leakage of information then your best friends and parents. Still we can code better software! |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA
|
| Posted: Tue Aug 08, 2023 3:08 am Post subject: |
|
|
| CaptainBlood wrote: | | eccerr0r wrote: | | Don't forget that gcc is getting slower too - from additional optimizations to additional safety checks (like compile time buffer overflow checks) the simplest programs are taking longer to build even with the same source code. |
Expect binary size to grow as well, most of the time. |
Yes I'd expect some runtime checks added to help with bloat, but I would hope coupled with code optimization and using newer CPU instructions, code size could also go down. Alas the runtime check code along with processor bug avoidance, these trounce any improvements from better instruction scheduling.
Also another thing is C++ ... where the compiler tries to guess what you're doing leading to massive bloat because you "might" need the code.
| ChrisJumper wrote: | but it feels like a shitmove. But you are right about it. I am not sure if this is about complexity. Trying to drain information from free software, or about newbies which just use daily programming tools.
|
I think also it's part of trying to stay relevant. People are rushing to Python/Ruby/Haskell because they do away of a lot of potential exploitable issues, and then you have rust on the other side which (tries to) enforce correct by design (and its bloated checks to do this). The C standard is written in stone that you're allowed to hang yourself when you shoot yourself in the foot. There are some ways to try to at least warn you you're about to do these things that the other languages simply don't let you do... and the PHB decides security holes are too big of a risk?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3347
Location: Rasi, Finland
|
| Posted: Thu Aug 10, 2023 7:14 pm Post subject: |
|
|
Some posts have been split into "complex CPU instructions increase binary size?" because the topic somewhat derailed and the discussion in the split posts is more in depth.
Fixed the link -- NeddySeagoon
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1547
|
| Posted: Fri Aug 11, 2023 6:01 am Post subject: |
|
|
| mani001 wrote: | Hi,
my desktop computer is growing old and I've noticed that compilation times of most packages increase significantly over time. The quintessential example would be given by qtwebengine
| Code: |
Fri Aug 18 20:19:13 2017 >>> dev-qt/qtwebengine-5.7.1-r2
merge time: 42 minutes and 4 seconds.
Mon Jul 10 22:11:46 2023 >>> dev-qt/qtwebengine-5.15.10_p20230623
merge time: 3 hours, 38 minutes and 46 seconds. |
One could argue that browsers (I guess compiling qtwebengine is similar to compiling chromium) have been getting more and more sophisticated in recent years...BUT the same thing can be observed for any package I checked, e.g.,
| Code: | Sat Apr 9 11:56:00 2016 >>> sys-libs/glibc-2.22-r4
merge time: 6 minutes and 20 seconds.
Thu Jun 29 17:27:37 2023 >>> sys-libs/glibc-2.37-r3
merge time: 23 minutes and 42 seconds. |
or
| Code: | Thu Mar 10 21:35:49 2016 >>> x11-base/xorg-server-1.17.4
merge time: 54 seconds.
Fri Jun 23 22:34:48 2023 >>> x11-base/xorg-server-21.1.8
merge time: 2 minutes and 7 seconds. |
I't all good but some years ago I was kind of hoping that in the future packages would compile in almost no time as we switch to more powerful CPUs
I guess the reason is not only every piece of software getting more and more complex but also compilers getting slower due to more features and/or checks. Any takes on this?
Cheers. |
Yes. Software is getting more and more sophisticated. Including compilers. And that's going to go that way for a while. For a least until there's development in that area.
And yes, newer CPU's compile faster and in general work faster and in fact CPU's have been developing more or less in accordance to Moor's law which is quadratic by the way.
And if people were aware of at least some of the theory surrounding programming languages they would not be surprised by that.
And yes, as your CPU ages it'll be less prepared to face today's challenges, isn't that natural?
Best Regards,
Georgi |
|
| Back to top |
|
|
dmpogo
Advocate
Joined: 02 Sep 2004
Posts: 3267
Location: Canada
|
| Posted: Fri Aug 18, 2023 8:34 pm Post subject: |
|
|
| logrusx wrote: | [
And yes, as your CPU ages it'll be less prepared to face today's challenges, isn't that natural?
Best Regards,
Georgi |
The point is that people always consider today's challenges such that 'today's' CPUs are barely enough |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA
|
| Posted: Sat Aug 19, 2023 1:27 am Post subject: |
|
|
I still don't like sloppy redundant code.
Consider some pseudocode like this:
| Code: | define SUS_ARGUMENT -1
sub code (var argument)
if(argument <0) return SUS_ARGUMENT
return popular_function(argument)+do_stuff_with(argument)
endsub
sub popular_function (var suspicious_argument)
if (suspicious_argument < 0) return SUS_ARGUMENT;
do_stuff_with(suspicious_argument)
return suspicious_argument
endsub
sub do_stuff_with(var argument)
if (argument < 0) return SUS_ARGUMENT;
work_with_private(argument)
return(argument)
endsub
sub work_with_private(var argument)
if (argument < 0) return SUS_ARGUMENT;
else return argument+1 # or whatever
endsub
main (argc, argv)
if(argc>2) and (code(argv[2]) == SUS_ARGUMENT) then print "Bad arg2"
if(argc>1) and (popular_function(argv[1]) == SUS_ARGUMENT) then print "bad arg1"
|
while this is a contrived, sloppy example in itself and bounds checking is good to have, how many times does it need to be done? Will gcc optimize out the redundant code? If the compiler can't get rid of the redundant checks, do we need a faster CPU to run these redundant bounds checks? Could this have been written better with fewer bounds checks?
While in mega builds/flat main code it could do analysis, but what about code in libraries?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Mon Aug 21, 2023 11:21 am Post subject: |
|
|
| eccerr0r wrote: | I still don't like sloppy redundant code.
Consider some pseudocode like this:
| Code: | define SUS_ARGUMENT -1
sub code (var argument)
if(argument <0) return SUS_ARGUMENT
return popular_function(argument)+do_stuff_with(argument)
endsub
sub popular_function (var suspicious_argument)
if (suspicious_argument < 0) return SUS_ARGUMENT;
do_stuff_with(suspicious_argument)
return suspicious_argument
endsub
sub do_stuff_with(var argument)
if (argument < 0) return SUS_ARGUMENT;
work_with_private(argument)
return(argument)
endsub
sub work_with_private(var argument)
if (argument < 0) return SUS_ARGUMENT;
else return argument+1 # or whatever
endsub
main (argc, argv)
if(argc>2) and (code(argv[2]) == SUS_ARGUMENT) then print "Bad arg2"
if(argc>1) and (popular_function(argv[1]) == SUS_ARGUMENT) then print "bad arg1"
|
while this is a contrived, sloppy example in itself and bounds checking is good to have, how many times does it need to be done? Will gcc optimize out the redundant code? If the compiler can't get rid of the redundant checks, do we need a faster CPU to run these redundant bounds checks? Could this have been written better with fewer bounds checks?
While in mega builds/flat main code it could do analysis, but what about code in libraries? |
I'm afraid this is somewhat of a strawman. As you say, it's contrived. I see no reason to believe this is at all representative of the issues we've been discussing in this thread. It also conflates bounds checking which the compiler would introduce (which was referred to earlier and has different properties, e.g. being elided when sizes are known) vs code written by the developer. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA
|
| Posted: Mon Aug 21, 2023 12:55 pm Post subject: |
|
|
One thing that is clear, any bounds checking is completely overhead - it does not contribute to performance for both software runtime, and is more code to compile during build time.
I'm specifically indicating code written by developers whether it is gcc or some other code -- a reason why code is getting bigger is because they forget bounds checking is occurring at multiple places... over and over again. You can't deny that people forget whether code still is there or not, plus the code may not have been written by the same person.
Whether or not the compiler is realizing there is redundant code and removing the extra developer written bounds checking is another problem, which adds to compiler complexity if it does exist, and also whether or not it's sane to do so because the bounds check may be needed for other reasons.
The bounds check problem to be fixed by the compiler/build time I suppose is similar to a halting problem and is yet another NP complexity problem, whether or not it's done yet to improve runtime is another question... I venture not, at least in all cases...yet...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Mon Aug 21, 2023 1:06 pm Post subject: |
|
|
Not sure I get the fixation on this. Even if there's some applications where it's the case, most complaints about build time are where software is written in a language which does a lot of heavy lifting at compile time. C++ and Rust being the main ones.
As for runtime: some examples would be welcome where profiling has showed that this is a recurring problem.
Anyway, for the specific example you give, compilers can propagate range information. They can also backpropate. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA
|
| Posted: Mon Aug 21, 2023 4:08 pm Post subject: |
|
|
Mainly because this will never get detected. This is correct by design, there's no error in compiling as is... and nobody will notice there's a problem.
It's a few cycles here and there and it adds up, and it's worse for interpreted languages.
I'm not sure if this will even show up in profiling, as it's just death by a thousand papercuts.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
|
Gentoo Chat
