Big increase in compile times for new versions of software

[mv]

I used ccache in the past as I wanted to speedup kde updates. It didn't really help, mostly because one of the central includes in kde was version.h, and it changed with every release.

Yes, if version.h is included and changed, you will not get a speedup. However, many recompilations do not involve a modification of version.h. For instance, gentoo revisiion bumps never do. Recompilation due to changed dependencies (which occurs meanwhile more and more frequently due to := dependencies) also usually require actual recompilation of only a few files. The same holds for changes in USE-flags - for the latter your mileage may vary.
For example, my average time for emerge of libreoffice is much below 3 hours, although a full compilation takes 20 hours or more.

[schorsch_76]

I have noticed large increases in compile times for some software. Below is the output of genlop -t for webkit-gtk. Is this to be expected with the increasing complexity of software or is it due to system age/performance (this is fairly old single-processor AMD system).

make-4 apparently slows things down a lot. (the new one with GUILE built-in; cos what every shellscripter wants is a LISP..

Do you have any sources for that statement that it "make with GUILE" is slower than the old make?

[Dr Croubie]

I just picked a few huge packages at random for comparison, and there's really much of a muchness between them.
Firstly gcc 4.8.4 seems a lot faster than 4.8.3, with 4.7.3-r1 in between:

Code: Select all

# genlop -t sys-devel/gcc
 * sys-devel/gcc

     Sun Aug 10 13:00:55 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 21 seconds.

     Sun Aug 10 15:20:11 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 29 seconds.

     Sun Aug 10 23:40:20 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 32 seconds.

     Mon Aug 11 09:20:45 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 20 seconds.

     Tue Aug 12 01:52:02 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 58 seconds.

     Tue Aug 12 12:47:03 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 28 minutes and 46 seconds.

     Wed Aug 13 01:14:52 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 32 minutes and 19 seconds.

     Wed Aug 13 13:52:50 2014 >>> sys-devel/gcc-4.7.3-r1
       merge time: 32 minutes and 31 seconds.

     Wed Nov  5 08:51:02 2014 >>> sys-devel/gcc-4.8.3
       merge time: 35 minutes and 57 seconds.

     Wed Nov  5 22:37:03 2014 >>> sys-devel/gcc-4.8.3
       merge time: 37 minutes and 33 seconds.

     Fri Nov  7 05:09:23 2014 >>> sys-devel/gcc-4.8.3
       merge time: 38 minutes and 31 seconds.

     Sun Dec 14 17:20:29 2014 >>> sys-devel/gcc-4.8.3
       merge time: 36 minutes and 38 seconds.

     Sun Apr 12 17:37:10 2015 >>> sys-devel/gcc-4.8.4
       merge time: 26 minutes and 51 seconds.

     Mon Apr 13 01:11:32 2015 >>> sys-devel/gcc-4.8.4
       merge time: 26 minutes and 41 seconds.

     Tue Apr 14 08:21:46 2015 >>> sys-devel/gcc-4.8.4
       merge time: 26 minutes and 52 seconds.

     Sat Jun  6 22:32:37 2015 >>> sys-devel/gcc-4.8.4
       merge time: 26 minutes and 33 seconds

I don't have the ages of data going back years, but my webkit-gtk isn't that slow:

Code: Select all

# genlop -t webkit-gtk
 * net-libs/webkit-gtk

     Mon Mar 23 13:43:37 2015 >>> net-libs/webkit-gtk-2.4.8
       merge time: 1 hour, 7 minutes and 55 seconds.

     Mon Mar 23 14:37:08 2015 >>> net-libs/webkit-gtk-2.4.8-r200
       merge time: 53 minutes and 31 seconds.

     Tue Apr 14 10:52:29 2015 >>> net-libs/webkit-gtk-2.4.8-r200
       merge time: 48 minutes and 43 seconds.

     Tue Apr 14 11:54:20 2015 >>> net-libs/webkit-gtk-2.4.8
       merge time: 1 hour, 1 minute and 51 seconds.

     Thu May 28 21:00:09 2015 >>> net-libs/webkit-gtk-2.4.8
       merge time: 59 minutes and 26 seconds.

     Thu May 28 21:47:16 2015 >>> net-libs/webkit-gtk-2.4.8-r200
       merge time: 47 minutes and 7 seconds.

Indeed, 2.4.8-r200 is about 10 mins / 15% faster than 2.4.8.

Everyone's favourite bloat, libreoffice:

Code: Select all

# genlop -t libreoffice
 * app-office/libreoffice

     Mon Aug 18 22:34:07 2014 >>> app-office/libreoffice-4.2.5.2
       merge time: 1 hour, 36 minutes and 54 seconds.

     Mon Oct  6 11:21:14 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 1 hour, 52 minutes and 7 seconds.

     Wed Nov  5 11:35:07 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 1 hour, 51 minutes and 5 seconds.

     Thu Nov  6 07:33:36 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 2 hours, 6 minutes and 55 seconds.

     Fri Nov  7 11:09:27 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 2 hours, 12 minutes and 32 seconds.

     Sat Nov 22 18:36:36 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 2 hours, 21 minutes and 3 seconds.

     Sun Dec 14 23:54:47 2014 >>> app-office/libreoffice-4.2.6.3
       merge time: 2 hours, 23 minutes and 32 seconds.

     Mon Jan  5 16:03:32 2015 >>> app-office/libreoffice-4.2.8.2
       merge time: 1 hour, 37 minutes and 37 seconds.

     Sun Jan 11 14:32:14 2015 >>> app-office/libreoffice-4.2.8.2
       merge time: 1 hour, 38 minutes and 22 seconds.

     Tue Feb 17 11:13:12 2015 >>> app-office/libreoffice-4.3.5.2
       merge time: 1 hour, 31 minutes and 13 seconds.

     Mon Mar 23 11:48:21 2015 >>> app-office/libreoffice-4.3.5.2
       merge time: 1 hour, 30 minutes and 31 seconds.

     Mon Apr 13 11:41:23 2015 >>> app-office/libreoffice-4.4.1.2
       merge time: 1 hour, 48 minutes and 52 seconds.

     Tue Apr 14 07:36:41 2015 >>> app-office/libreoffice-4.4.1.2
       merge time: 1 hour, 47 minutes and 54 seconds.

     Thu May 28 19:57:45 2015 >>> app-office/libreoffice-4.4.3.2
       merge time: 1 hour, 48 minutes and 37 seconds.

The latest gcc 4.8.4 came in at the same time as office 4.3.5.2 up to 4.4.1.2, so can't really tell what caused the extra 18 mins. But the latest 4.4.3.2 is about as fast/slow (depending on how you look at it)

If anything, things have gotten faster for me:

Code: Select all

# genlop -t xorg-server
 * x11-base/xorg-server

     Wed Aug 13 07:00:40 2014 >>> x11-base/xorg-server-1.15.0
       merge time: 3 minutes and 40 seconds.

     Wed Aug 13 12:55:44 2014 >>> x11-base/xorg-server-1.15.0
       merge time: 3 minutes and 41 seconds.

     Thu Nov  6 03:55:11 2014 >>> x11-base/xorg-server-1.15.0
       merge time: 3 minutes and 51 seconds.

     Fri Nov  7 02:46:45 2014 >>> x11-base/xorg-server-1.15.0
       merge time: 3 minutes and 50 seconds.

     Sun Dec 14 15:17:44 2014 >>> x11-base/xorg-server-1.15.0
       merge time: 3 minutes and 12 seconds.

     Sun Jan 11 12:19:19 2015 >>> x11-base/xorg-server-1.15.2-r1
       merge time: 3 minutes and 6 seconds.

     Tue Feb 24 18:32:42 2015 >>> x11-base/xorg-server-1.16.4
       merge time: 2 minutes and 53 seconds.

     Tue Apr 14 09:13:54 2015 >>> x11-base/xorg-server-1.16.4
       merge time: 2 minutes and 50 seconds.

     Tue May 26 14:59:05 2015 >>> x11-base/xorg-server-1.17.1-r1
       merge time: 2 minutes and 41 seconds.

     Sat Jun  6 20:12:40 2015 >>> x11-base/xorg-server-1.17.1-r1
       merge time: 2 minutes and 43 seconds.

xorg-server is nearly 25% quicker than it used to be.

Even the bloatiness of firefox has gotten better lately, it seems (although I only keep it around for sites that block tor).

Code: Select all

# genlop -t firefox
 * www-client/firefox

     Thu Aug 14 20:18:32 2014 >>> www-client/firefox-24.7.0
       merge time: 40 minutes and 8 seconds.

     Sat Sep  6 09:05:39 2014 >>> www-client/firefox-24.8.0
       merge time: 41 minutes and 37 seconds.

     Mon Oct  6 21:18:14 2014 >>> www-client/firefox-24.8.0
       merge time: 44 minutes and 8 seconds.

     Thu Nov  6 03:43:02 2014 >>> www-client/firefox-24.8.0
       merge time: 50 minutes and 1 second.

     Fri Nov  7 07:20:18 2014 >>> www-client/firefox-24.8.0
       merge time: 51 minutes and 31 seconds.

     Sun Dec 14 19:49:39 2014 >>> www-client/firefox-31.3.0
       merge time: 44 minutes and 2 seconds.

     Tue Feb 17 09:41:59 2015 >>> www-client/firefox-31.4.0
       merge time: 28 minutes and 10 seconds.

     Tue Mar  3 09:03:49 2015 >>> www-client/firefox-31.5.0
       merge time: 28 minutes and 16 seconds.

     Mon Mar 23 12:16:17 2015 >>> www-client/firefox-31.5.0
       merge time: 27 minutes and 56 seconds.

     Mon Apr 13 09:06:08 2015 >>> www-client/firefox-31.5.3
       merge time: 26 minutes and 18 seconds.

     Tue Apr 14 09:41:04 2015 >>> www-client/firefox-31.5.3
       merge time: 26 minutes and 12 seconds.

     Tue Apr 28 08:45:35 2015 >>> www-client/firefox-31.6.0
       merge time: 27 minutes and 53 seconds.

     Wed May 20 13:18:08 2015 >>> www-client/firefox-31.7.0
       merge time: 1 hour, 55 minutes and 48 seconds.

     Thu May 28 17:16:06 2015 >>> www-client/firefox-31.7.0
       merge time: 26 minutes and 20 seconds.

Not sure what happened on May 20, does genlop take into account download times? That would have been a time I was shaped to 256kbps for going over my quota.

For all of the above, same Phenom II 1055T 6x 2.8GHz. Same 20GB DDR3-1666. Same SSD.

As for ccache, there's a reason it's disabled by default. Any time I've saved by using it has been negated and then some by recompiling and frigging around if a build breaks, trying to fix the problem, which invariably has gotten 'cached' so the only way to fix it is to disable ccache and try again. Happened so many times I've given up enabling it, not worth the headaches.

[krinn]

Just to let you dream on firefox 3

Code: Select all

     Thu Aug  5 03:18:40 2010 >>> www-client/firefox-3.6.8-r1
       merge time: 1 minute and 9 seconds.

     Wed Sep 15 03:24:39 2010 >>> www-client/firefox-3.6.9
       merge time: 58 seconds.

who said bloated???

[mv]

As for ccache, there's a reason it's disabled by default.

The reason is: One developer who is not active anymore used his blog to spread a lot of FUD against ccache (most claims from his blog [concerning ccache] are plainly false) and used his influence to remove the previous default.

Any time I've saved by using it has been negated

You are certainly wrong. Even if only every 10th emerge of libreoffice is captured "fully" by ccache - until the few seconds per emerge which you possibly loose by using ccache add up to one hour... do the easy math yourself.

frigging around if a build breaks, trying to fix the problem, which invariably has gotten 'cached' so the only way to fix it is to disable ccache and try again.

You cannot be speaking about ccache: In order to get troubles from a cached file due to a broken build, one of the following must have happened:

1. The ccache proccess was hard killed (with -9 or kernel OOM killer), and moreover, in a very unfortunate moment. A "normal" compilation error (including compiler segfault) cannot cause such problems.
2. You ran out of disk space.
3. Your disk is not reliable.
4. You ran into the very unlikely situation of a hash collision (chances are 1:10^40 or so per file).
5. ccache has a bug.

Indeed, the cached result is only used if you use exactly the same command line with the same gcc executable with the same source file content and the same include-file content, and ccache works hard to avoid false positives of any of this.

I know of only one case where really ccache is the cause of trouble: The config-file of some nvidia-drivers breaks if used with ccache. I did never investigate whether the reason is 1, 4, or 5 or something which I forgot in this list (although I cannot imagine something I forgot). However, this particular config-file does a lot of nasty things - I would not be surprised if it kills the compilation process to get some timing results or uses a tricky command line which triggers a bug of ccache.

[krinn]

The reason is: One developer who is not active anymore used his blog to spread a lot of FUD against ccache (most claims from his blog [concerning ccache] are plainly false) and used his influence to remove the previous default.

I have no remembering of ccache use as default in gentoo since i use it
One of us is getting old and memory is not as good as before it seems.

For this or that reason, you just cannot deny ccache can lead to compile trouble, that's really not a problem for a user that understand what is going on, that's a problem for many users that don't even know or understand the portage error message (yes the one that keep telling them: if you need support, emerge --info and build.log)

So really, for this kind of users, they should just not use ccache as they will be unable to handle it.
It's a bit the same for distcc, but trouble with distcc are easy to catch up in ebuild and unlike ccache, reproducible. But a "basic user" using distcc failing to build something will not even have the idea to FEATURES="-distcc" emerge one_that_fail
See my point mv? It doesn't matter if ccache is good or bad, it is bad for some users, always.

[mv]

I have no remembering of ccache use as default in gentoo since i use it ;)

Maybe it was "only" a strong recommendation in the installation handbook to enable it which was removed - in practice, it does not make much difference to a default, since everybody who installed by handbook had enabled it.

So really, for this kind of users, they should just not use ccache as they will be unable to handle it.

Even for this kind of users, it should work: The OOM killer and out-of-space scenario - which could be the only reasons of trouble for this type of users - is rare enough, especially nowadays.

[steveL]

make-4 apparently slows things down a lot. (the new one with GUILE built-in; cos what every shellscripter wants is a LISP..;)

Do you have any sources for that statement that it "make with GUILE" is slower than the old make?

Well it was a statement about make-4, but only anecdotal, from someone I trust:

Code: Select all

<UberPinguin> Yesterday I learned that compiling Android with make >= 4.0 results in a 6.5x increase in build time.
<UberPinguin> 130 minutes with make-4.1, 20 minutes with make-3.82
<UberPinguin> Most of the build w/ make-4.x hardly touches CPU. Disk I/O doesn't seem to make much of a difference
<UberPinguin> There are a few parts that actually do thread well, and max all of the cores, but they're few, far between, and don't last long
<UberPinguin> In contrast, make-3.82 leverages all of the cores a majority of the time

[steveL]

I never understood why GNU people are so crazy about LISP. It was a horrible idea to use it for an editor, it is an even worse idea for a make system. I fail to see any rational reasons for this choice.

I think there's some underlying "vision" of guile as the GNU scripting language. Personally, when someone starts talking about "vision" wrt computing, I start to put away the scissors.

Mostly though, it feels like my old joke about gmake being "a horrible way to write LISP" is coming back to haunt me. ;)

GNU in general is "crazy about LISP" because rms was a LISPer; thus back-end gcc RTL is effectively another crufty way to write LISP.

[Roman_Gruber]

I wished devs would do something about ram usage

core 2 t9500 wiht 4gb ram and separate gpu nvidia 9800m gts

somethings tend to take too long or take too much ram in past two years because average dev has probably 8-16gb installed and therefore does not care for low ram

[hans1024]

I never understood why GNU people are so crazy about LISP. It was a horrible idea to use it for an editor, it is an even worse idea for a make system. I fail to see any rational reasons for this choice.

I think there's some underlying "vision" of guile as the GNU scripting language. Personally, when someone starts talking about "vision" wrt computing, I start to put away the scissors.

Mostly though, it feels like my old joke about gmake being "a horrible way to write LISP" is coming back to haunt me.

GNU in general is "crazy about LISP" because rms was a LISPer; thus back-end gcc RTL is effectively another crufty way to write LISP.

Well, it's not just because of RMS, there are actually pretty good reasons some GNU hackers love Lisp or Scheme. You can easily bend Lisps to suit the problem you are solving, whether it's high level things like math, sets and proofs, or lowlevel stuff like RTL. Lisp expressions are also easy to parse and transform (wich might be one of the reasons for usage in gcc machine definitions).

Of course I don't think that *everyone* should write *everything* in Lisp (and I'm sure most GNU hackers agree with me), people are different and some problems are more imperative by nature and many times it's better to be more low-level. But from my experience most programmers just have this irrational fear of parentheses and once they actually learn some Lisp, they usually like it

Back to the topic:
My frustration with bloated software is getting worse every year. Web browsers are becoming the new operating systems - well, they actually compile even slower than the whole operating system. I'm sure this is how Mozilla developers work: https://xkcd.com/303/

Can the code size be getting bigger and bigger forever, without any upper limit or negative consequences? I guess it can't.

I have some hope for projects like Rust and Servo.

[mv]

You can easily bend Lisps to suit the problem you are solving, whether it's high level things like math, sets and proofs, or lowlevel stuff like RTL. Lisp expressions are also easy to parse and transform (wich might be one of the reasons for usage in gcc machine definitions).

I cannot really judge how it is suited for RTL and how much it e.g. differs from java or go byte code, but I can imagine that it was not the best choice, either. For higher level stuff, Lisp is just a nightmare: There is almost nothing for which it is the appropriate language. Perhaps some very basic symbol manipulation - stuff which is actually completely unneeded in an editor GUI or make type programming.

most programmers just have this irrational fear of parentheses and once they actually learn some Lisp, they usually like it :-D

I had spent a few months to learn this thing to turn emacs into a useful editor for me. A few (X)Emacs packages arised (you can see them in mv_emacs on the mv overlay). I was almost biased in favor of it when I learnt it - because I thought, they will have a reason when they made this choice for an editor - but the more I learnt it the more I hated it: Lisp is such a poor language which is missing everything a programmer needs.

1. The difference between "functional" and "imperational" programming is just a philosophical excuse and actually not a feature of the language, since you can simply use both in any language (e.g. C++ and also Lisp); only that Lisp is so poor that one of it is much harder and needs regularly to insert the "dummy" command progn (instead of providing a sane syntax for this task). That also Lisp fans suffers from this can be seen by the fact that later on "macros" were added - something which by the concept should actually be part of functional programming - to allow at least partially writing a real program.
2. Except for the "basic" data types (numbers, characters, strings) only 1 data type (plus a function type which is handled in an exceptional manner - actually in an exceptionally poor manner). Everything which the programmer regularly needs (hash, arrays, more complex types) must be emulated manually using this poor list type.
3. A very poor concept of variable scoping. Nothing comparable to module/class/whatever local variables, static variables, "really" local variables. In fact, there is not even a sane concept of a module etc which is why emacs is lacking a sane package concept up to now despite many efforts and hacks of the emacs hackers to improve the situation somewhat.
4. The syntax is a nightmare: Although actually things like let, when, progn, ... have a very different meaning, they all have they same syntax - apparently only to make it easier for the machine to read and harder to make it for a human to read. A program which cannot be understood by looking at it, because a human is not able to count the number of braces, but only be read with the aid of an editor which counts the braces for you and highlights matches: Something is inherently broken with such type of language - at least, if it is meant to be read and written by a human.
5. The quoting is a major nightmare in Lisp. This is a nuisance particularly in Emacs. If you ever tried to find out how to actually define a special key in Emacs (the subtle difference between multibyte strings, character lists, strings with a additional bits added, etc.) you know what I mean. This might be a particular problem of Emacs and its poor attempts to add utf8 as an afterthought. (However, the problem existed already long before utf8. It is really hard to believe that the author of an editor did not spend considerable time before the implementation about the internal and external representation of symbols and how to fetch key events: The outcome is hack after hack - one transformation table added after the other, since the previous solution always turned out to be insufficient to handle basic stuff.)

[hans1024]

For higher level stuff, Lisp is just a nightmare: There is almost nothing for which it is the appropriate language. Perhaps some very basic symbol manipulation - stuff which is actually completely unneeded in an editor GUI or make type programming.

I'm sorry, but my experience has been the opposite. I should probably clarify that most of my experience is with Scheme - a well standardised dialect with many useful SRFIs ("libraries").

The difference between "functional" and "imperational" programming is just a philosophical excuse and actually not a feature of the language, since you can simply use both in any language (e.g. C++ and also Lisp); only that Lisp is so poor that one of it is much harder and needs regularly to insert the "dummy" command progn (instead of providing a sane syntax for this task). That also Lisp fans suffers from this can be seen by the fact that later on "macros" were added - something which by the concept should actually be part of functional programming - to allow at least partially writing a real program.

I don't see how macros were "added", I think they indeed are a key part of functional programming and one of great strenghts of Lisps.

Everything which the programmer regularly needs (hash, arrays, more complex types) must be emulated manually using this poor list type.

There are hash tables and arrays (vectors) in both Scheme and Common Lisp.

A very poor concept of variable scoping. Nothing comparable to module/class/whatever local variables, static variables, "really" local variables. In fact, there is not even a sane concept of a module etc which is why emacs is lacking a sane package concept up to now despite many efforts and hacks of the emacs hackers to improve the situation somewhat.

Again, in both Common Lisp and Scheme, there are packages/modules (with their variables), "static" variables, lexical scope. I do not have experience with Emacs lisp, it may be pretty bad dialect. I know they use dynamic scope, wich is a bad choice in my opinion.

The syntax is a nightmare: Although actually things like let, when, progn, ... have a very different meaning, they all have they same syntax - apparently only to make it easier for the machine to read and harder to make it for a human to read.

I find it quite easy to read well-written Lisp code and I find it even more easier to write Lisp code - and I consider the simple syntax being one of the benefits when doing so.

A program which cannot be understood by looking at it, because a human is not able to count the number of braces, but only be read with the aid of an editor which counts the braces for you and highlights matches: Something is inherently broken with such type of language - at least, if it is meant to be read and written by a human.

I'm sorry, but this is nonsense. You don't need to count any braces to read a decently formated Lisp code. And non-formated code is as hard to read as is non-formated C code.

The quoting is a major nightmare in Lisp. This is a nuisance particularly in Emacs. If you ever tried to find out how to actually define a special key in Emacs (the subtle difference between multibyte strings, character lists, strings with a additional bits added, etc.) you know what I mean. This might be a particular problem of Emacs and its poor attempts to add utf8 as an afterthought. (However, the problem existed already long before utf8. It is really hard to believe that the author of an editor did not spend considerable time before the implementation about the internal and external representation of symbols and how to fetch key events: The outcome is hack after hack - one transformation table added after the other, since the previous solution always turned out to be insufficient to handle basic stuff.)

Well, this is probably where the problem is. I don't use Emacs, and have almost no experience with it. It is quite possible that Emacs Lisp is a really bad dialect, because I haven't encountered any of the problems you are describing. There's probably a good reason some devs wanted to port Emacs to Guile.

[steveL]

Well, it's not just because of RMS, there are actually pretty good reasons some GNU hackers love Lisp or Scheme.

This is taking it in another direction; God I hate language arguments.

But from my experience most programmers just have this irrational fear of parentheses and once they actually learn some Lisp, they usually like it

That's not been my experience at all.

I'm all for functional programming, but I don't have any irrational attachment to anything, and I agree with mv, that the parentheses-above-all approach of LISP-descendants was indeed designed to make it easy for the machine, above all else.
It's more accurate to say it was designed to make it easy for anyone to write a parser.

IOW it's essentially laziness; we can't be bothered to write a decent parser, and in theoretical terms it's all the same, so let's not bother.
Originally this was a "thought experiment"; when it turned out to be "all the same", an idea became overplayed, as so often.

In any event, the thing that swung it, was a bunch of old-school LISPers' advice to "just use indentation to format (that's what we do)", which kinda gives the lie to the whole thing.

Personally I think there's a bit of confusion wrt different aspects, like mathematics and a functional declaration, computational "purity" and actually getting anything done ("side-effects are the point"), and precisely what the S in CSP means.

Disclaimer: I love (Standard) ML, and rather like Scheme, though don't code in it, and yes I have a copy of SICP within 2 feet of me.
Yes, macros are an essential part of LISPs.

@mv Considering LISPs, perhaps you can see what I mean about bracketing forming contexts, where close must match open, which I think it's fair to say is what's annoying you with it; too much bracket-counting instead of simplicity for a human.
Yes, that's precisely my problem with redundant bracing in sh; it reminds me of FP, in terms of headache caused.

[hans1024]

IOW it's essentially laziness; we can't be bothered to write a decent parser, and in theoretical terms it's all the same, so let's not bother.

It's not just about parsers. It's an important aspect of Lisps that code is essentially data (lists), and thus can be easily manipulated as any other lists. So yeah that's a kind of laziness too, but that's what programming languages are for - making something easier.

Personally I think there's a bit of confusion wrt different aspects, like mathematics and a functional declaration, computational "purity" and actually getting anything done ("side-effects are the point"), and precisely what the S in CSP means.

As I said I don't think that everything should be written in Lisp/Scheme. It really depends on the particular problem you are solving.

[mv]

I don't see how macros were "added", I think they indeed are a key part of functional programming and one of great strenghts of Lisps.

Macros are always a hack to get around shortcomings of a language. You can see this very well in in C, and it is not so much different in Lisp. Of course, in some sense one can also say that lisp is nothing else than a macro language, only that in "non-macros" the expansion is not postponed. And as always in macro languages, lisp suffers from the inherent problem that you have to think a lot which macros are expanded and when, and you have to jump hoops to make it expand in the way you need.

There are hash tables and arrays (vectors) in both Scheme and Common Lisp.

Yes, but all very poor. Mainly, you cannot have complex types like structs/classes without "emulating" them (using "magic" ordering in lists or similar things instead of a "sane" access by names). The same problem holds for function calling, where the order is essentially the only way to distinguish arguments.

Again, in both Common Lisp and Scheme, there are packages/modules (with their variables), "static" variables, lexical scope. I do not have experience with Emacs lisp, it may be pretty bad dialect. I know they use dynamic scope, wich is a bad choice in my opinion.

This might change things a bit. I know only emacs lisp and had thought that this dynamical scoping (which is the only way in emacs lisp to get some local variables - anything else is global unless perhaps there are some ugly tricks to manipulate the symbol hash directly; I am not such an expert who knows such implementation details) is how all lisps dialects are. Emacs introduced buffer-local variables, but obviously this is a hack in the absence of an appropriate concept of scoping variables.

I find it quite easy to read well-written Lisp code and I find it even more easier to write Lisp code. [...] You don't need to count any braces to read a decently formated Lisp code. And non-formated code is as hard to read as is non-formated C code.

It is relatively easy to write but almost impossible to read if written somebody else (or by yourself many years ago): Is the text after the complex bulk of opening and closing braces now the 7th optional argument of a complicated function or is this function already finished, and it is the second part of the preceding "(or"; or does it belong to the preceeding "(progn" or the embracing "(let" or yet another embracing "(let", or does it belong to something you do not remember at all anymore due to the many iterations? If the text would have been "properly" intended you were at a tab level of 20 or so, and thus would not be able to see anything, either.
Probably, the code should not have been written in such a way that you are forced to be at tab level 20 for a "sane" indentation. But every code I have seen is written in such a way - and I would be surprised if all emacs lisp programmers are very poor: In fact, it is the language itself which forces this style of iterated iteration of iteration - it is the core of functional programming.
Usually, I observe this problem for code which would actually be written very easily in an imperative language - in fact, some code should by its nature just be imperative or object oriented - but Lisps does not allow to "escape" to any other style of programming. This is IMHO the biggest shortcoming of Lisp.

[mv]

@mv Considering LISPs, perhaps you can see what I mean about bracketing forming contexts, where close must match open, which I think it's fair to say is what's annoying you with it; too much bracket-counting instead of simplicity for a human.
Yes, that's precisely my problem with redundant bracing in sh; it reminds me of FP, in terms of headache caused.

IMHO, these things are not comparable. In Shell, it is very exceptional that you have iterated ${...}; more than 2 is an extreme exception (and BTW except for the most inner ones in certain situations, the braces are not redundant, anyway). In Lisp, we are speaking about 20-30 iterated braces.

[mv]

It's not just about parsers. It's an important aspect of Lisps that code is essentially data (lists), and thus can be easily manipulated as any other lists.

Such type of self-modifying code is also nothing but an ugly hack. I had to use it to "wrap around" (in emacs Lisp it is called "advice") certain functions (based on their content) to implement a "block" concept in Emacs Lisp, because all functions operating on a mark have to operate on a block instead. Unsurprisingly, it is not possible to do this reliable, and many manual exceptions had to be added, changing with every version of emacs and with any added mode... it was and remains an ugly hack, in the lack of a possibility to do anthing "cleanly" in Lisp.

[hans1024]

Macros are always a hack to get around shortcomings of a language.

Well, if you think of macros this way it's no surprise you don't like Lisp

Of course, in some sense one can also say that lisp is nothing else than a macro language, only that in "non-macros" the expansion is not postponed. And as always in macro languages, lisp suffers from the inherent problem that you have to think a lot which macros are expanded and when, and you have to jump hoops to make it expand in the way you need.

I don't think you have to think that much with well-designed macros.

There are hash tables and arrays (vectors) in both Scheme and Common Lisp.

Yes, but all very poor.

I don't see how are they poor.

Mainly, you cannot have complex types like structs/classes without "emulating" them (using "magic" ordering in lists or similar things instead of a "sane" access by names). The same problem holds for function calling, where the order is essentially the only way to distinguish arguments.

Common Lisp Object System, or Guile's GOOPS or other alternatives provide objects/structs with access to values by name.

This might change things a bit. I know only emacs lisp and had thought that this dynamical scoping (which is the only way in emacs lisp to get some local variables - anything else is global unless perhaps there are some ugly tricks to manipulate the symbol hash directly; I am not such an expert who knows such implementation details) is how all lisps dialects are. Emacs introduced buffer-local variables, but obviously this is a hack in the absence of an appropriate concept of scoping variables.

Yes, Emacs lisp sounds really weird. Most Lisps use lexical scope.

Is the text after the complex bulk of opening and closing braces now the 7th optional argument of a complicated function or is this function already finished, and it is the second part of the preceding "(or"; or does it belong to the preceeding "(progn" or the embracing "(let" or yet another embracing "(let", or does it belong to something you do not remember at all anymore due to the many iterations? If the text would have been "properly" intended you were at a tab level of 20 or so, and thus would not be able to see anything, either. Probably, the code should not have been written in such a way that you are forced to be at tab level 20 for a "sane" indentation. But every code I have seen is written in such a way - and I would be surprised if all emacs lisp programmers are very poor: In fact, it is the language itself which forces this style of iterated iteration of iteration - it is the core of functional programming.

I think you should write small functions and compose them. But even if you don't, I think it should be clear from indentation, where that code belongs to.

Usually, I observe this problem for code which would actually be written very easily in an imperative language - in fact, some code should by its nature just be imperative or object oriented - but Lisps does not allow to "escape" to any other style of programming. This is IMHO the biggest shortcoming of Lisp.

I think using above mentioned object systems, you can be both very object-oriented and very imperative. Most of the time you don't even need to write "progn". But if the whole thing you are doing is imperative by nature, then Lisp might not be the best tool to use.

[hans1024]

It's not just about parsers. It's an important aspect of Lisps that code is essentially data (lists), and thus can be easily manipulated as any other lists.

Such type of self-modifying code is also nothing but an ugly hack.

It's usualy not self-modyfiing code but code modyfing other code. I don't know why it should necessarily be an ugly hack.

[steveL]

IOW it's essentially laziness; we can't be bothered to write a decent parser, and in theoretical terms it's all the same, so let's not bother.

It's not just about parsers. It's an important aspect of Lisps that code is essentially data (lists), and thus can be easily manipulated as any other lists. So yeah that's a kind of laziness too, but that's what programming languages are for - making something easier.

Yes, but my point was that by focussing solely on parentheses (which was done in the name of a thought experiment) it does not make my job easier, but harder.
I'd rather the implementation language had a bit more to it.

Personally I think there's a bit of confusion wrt different aspects, like mathematics and a functional declaration, computational "purity" and actually getting anything done ("side-effects are the point"), and precisely what the S in CSP means.

As I said I don't think that everything should be written in Lisp/Scheme. It really depends on the particular problem you are solving.

Sure; and I'm fine with a LISP in the compilation/evaluation phase, certainly as a metaphor. Just not the rest.
It's easy enough to program functionally in an imperative language; the other way round is usually a pita, and usually comes with associated arguments with people who haven't really grokked what CSP stands for.

Even if you use foo! as a pattern, you still have to write with all-parentheses.
Or you go down the rabbit-hole of writing your parser in a LISP, even though the language it reads isn't, and you're in the same place, as are your users ultimately. Most likely you'd be advising them to escape to your LISP, as you don't think you need anything else, since "theory is practice".

FWIW, I agree that other languages' macro systems are no way near as powerful as the LISPs have enjoyed for decades.
C++ appears to have spent 30 years reinventing one badly, for a start.

[mv]

I think you should write small functions and compose them. But even if you don't, I think it should be clear from indentation, where that code belongs to.

If you look at emacs lisp code, the former rarely happens - I guess, partially also because people do not want to litter the valuable global function name space; every name can lead to possible collisions and problems.
And it is easy to understand how such code arises: You have a complicated function which needs as one parameter e.g. a point position. To calculate the point you must perhaps temporarily switch the buffer, so you have to put it in a save-excursion; then you realize that also local variables are needed for the calculation, so another level is introduced. Thus, without anything special you have already needed 4 or 5 levels, and all should be indented. If the whole thing then happens in an if-construct, and the result has to be stored in local variables, and if the whole if-construct is in a function which also uses local variables, you are already at an indentation level where optical indentation does not work anymore in any reasonable way - unless you have a 200 character wide display. And from a logical point of view, this code is really far from being complex: I am speaking about a task which in an imperative language is one or two lines of trivial assignments and function calling.

[steveL]

@mv Considering LISPs, perhaps you can see what I mean about bracketing forming contexts, where close must match open, which I think it's fair to say is what's annoying you with it; too much bracket-counting instead of simplicity for a human.
Yes, that's precisely my problem with redundant bracing in sh; it reminds me of FP, in terms of headache caused.

IMHO, these things are not comparable. In Shell, it is very exceptional that you have iterated ${...}; more than 2 is an extreme exception (and BTW except for the most inner ones in certain situations, the braces are not redundant, anyway). In Lisp, we are speaking about 20-30 iterated braces.

The point is not how many levels deep it goes, but that you MUST keep track of the bracketing (in any code language), which is why you find LISP annoying (since there's so much to keep track of, and it's annoying to have to count the brackets or rely on your editor to highlight-match them.)

I have the same with sh, since I am keeping track of everything in a function, or block.
Mentally opening a context is jarring when one finds the whole thing was unneeded at the end of it; by only bracing where needed, it doesn't jar, since every braced expansion has a purpose.
Usually some sort of PE manipulation, very occasionally because it's in a string and followed directly by an alphanumeric or underscore.
Practically never because it's a numbered arg above 9 (which really does imply YDIW, in the vast majority of cases.)

Redundant bracing simply looks amateurish, because it wastes coder headspace, which anyone who writes shell on a more than semi-amateur basis, will find irritating for that precise reason.
I find it as (exactly (as (annoying (as most) LISP)(in precisely (the (same)) way))).)

[mv]

@steveL: Counting to 1 does not waste any head space. What wastes head space is thinking whether you are in a special case where perhaps the syntax would allow a symbol from being omitted. The real problem is that you wasted your head space unnecessary by pushing those rules in your unconsciousness which actually are completely superfluous since you could just have refrained from learning and using them from the very beginning.

[hans1024]

Yes, but my point was that by focussing solely on parentheses (which was done in the name of a thought experiment) it does not make my job easier, but harder.

Well yeah, it's a tradeoff.

you are already at an indentation level where optical indentation does not work anymore in any reasonable way - unless you have a 200 character wide display

That sounds bad, most Lisp/Scheme code i worked with had 80 characters maximum width.