| View previous topic :: View next topic |
| Author |
Message |
MarekSieradzki
n00b
Joined: 05 Oct 2010
Posts: 18
|
 Posted: Sun Oct 20, 2013 8:24 pm Post subject: |
 |
|
I find it weird that some people whine when they see hugely integrated stuff like systemd but don't do that when Portage behaves the same way. I'm not a Python expert but last time I looked at emerge I couldn't find any piece to split off it and reimplement as faster library/program.
Emerge is not a compiler but it has some similarities. It performs lexing, parsing and does some processing of things extracted during parsing. In lots of compilers you see that complicated constructs are translated into simpler ones. One language that has a lot of constructs is Haskell. All complicated syntactic constructs are compiled down to simplified Core language. Definition of this simplified and abstract language is heavily scrutinized.
As far as I know Portage is a blob. You can either reimplement all of it or kill yourself. I don't find that inviting for people that want to improve specific bits of Portage. |
|
| Back to top |
|
 |
TomWij
Retired Dev
Joined: 04 Jul 2012
Posts: 1553
|
| Posted: Sun Oct 20, 2013 8:42 pm Post subject: |
|
|
| MarekSieradzki wrote: | I find it weird that some people whine when they see hugely integrated stuff like systemd but don't do that when Portage behaves the same way. I'm not a Python expert but last time I looked at emerge I couldn't find any piece to split off it and reimplement as faster library/program.
Emerge is not a compiler but it has some similarities. It performs lexing, parsing and does some processing of things extracted during parsing. In lots of compilers you see that complicated constructs are translated into simpler ones. One language that has a lot of constructs is Haskell. All complicated syntactic constructs are compiled down to simplified Core language. Definition of this simplified and abstract language is heavily scrutinized.
As far as I know Portage is a blob. You can either re-implement all of it or kill yourself. I don't find that inviting for people that want to improve specific bits of Portage. |
Thank you very much for this view on it; it is an interesting thought that some techniques used to optimize compilers could possible be reused on Portage, that's an interesting field to further investigate.
Pursuing it as one big blob is what I have previously done wrong as profiling hasn't made anything stood out yet for me; why is that so, because if you take a look at those call tree images I shared previously you are forced to kind of take a top down look at it because it is a graph which starts with a root at the top.
I have since talked with Neil Cahill whom has contributed the patch for the reg exp improvement and have learned from that conversation that a bottom up approach will yield more in the short run (that is, start by the leaves), which is actually the thing we are looking after here; because we don't want to re-implement it for as long as we can and don't have to. Since the leaves correspond to smaller parts of the code, (eg. the inefficient regular expression that was patched) they are more easily to deal with; and if multiple smaller parts are added up, you gain a lot of lost time. Will look into this further tomorrow...
P.S.: Another interesting thing to note is that a single package is being emerged as opposed to the whole dependency tree, it lies more focus on the resolution of the dependency tree and less focus on adding all the @world entries.
| MarekSieradzki wrote: | | I'm not a Python expert but last time I looked at emerge I couldn't find any piece to split off it and reimplement as faster library/program. |
pkgcore does this to some extent, there is the separate snakeoil library and some files that were implemented in C and possibly other pieces I haven't come across; maybe we could look into whether it is possible to merge or replicate that work in Portage. |
|
| Back to top |
|
|
ulenrich
Veteran
Joined: 10 Oct 2010
Posts: 1480
|
| Posted: Mon Oct 21, 2013 12:09 am Post subject: |
|
|
If I had the choice
a) portage from 1 year ago: performance but it is an incomplete package manager
b) portage-2.2.7
I vote for todays portage and its completeness! Certainly the main bug of portage now is: Performance 
I just had a new "workaround" idea: From years ago experience with Debians I know they got some performance to the aptitude tool by giving the solver initial hints. What could such a hint be in the world of Gentoo source ebuild tree, for example:
If I know I want to upgrade the whole of Kde, could I hint portage in a performace boosting way by just removing the old ebuild versions (by deleting the files or using an aufs technique) ?
I mean: If I temporarily provide to portage an artificially "reduced" portage tree ?
Could be that such a preprocessing of providing a reduced tree is too costly, or has weird results, because of false premises, but just as a Gedankenexperiment ... |
|
| Back to top |
|
|
saellaven
l33t
Joined: 23 Jul 2006
Posts: 646
|
| Posted: Tue Oct 22, 2013 11:22 pm Post subject: |
|
|
| MarekSieradzki wrote: | | I find it weird that some people whine when they see hugely integrated stuff like systemd but don't do that when Portage behaves the same way. |
Last time I checked, portage hasn't reimplemented half of the software on my system due to NIH syndrome and the desire to be the one tool to rule everything on my system. I also don't remember the last time I had to fundamentally alter portions of my system to make it happy in the hopes that it doesn't break functionality that has worked for decades.
Personally, I'd love to see portage broken down into more maintainable and reusable bits.
But here's the difference... portage didn't just come along and declare everything on my system obsolete for not doing it the way portage wants to. Gentoo essentially is portage (and it's variants) and wouldn't exist without it. systemd takes decades of functional utilities that are well understood and in the vast majority of cases "just work" and re-implements them in the interests of Red Hat's desire to "own" Linux by forcing everyone into their playground, using their control of things like systemd and GNOME as pressure to do so.
When the other distros have rolled over, as Gentoo is starting to do, and given in because it's just too hard to diverge from Red Hat's large upstream, what's the point in even having other distros? If I wanted to run Red Hat, I'd run Red Hat and if I wanted a source version of Red Hat to compile myself, there a source rpms available.
_________________
Ryzen 3700X, Asus Prime X570-Pro, 64 GB DDR4 3200, GeForce GTX 1660 Super
openrc-0.17, ~vanilla-sources, ~nvidia-drivers, ~gcc |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Wed Oct 23, 2013 9:52 am Post subject: |
|
|
| MarekSieradzki wrote: | | As far as I know Portage is a blob. You can either reimplement all of it or kill yourself. I don't find that inviting for people that want to improve specific bits of Portage. |
It is (well, used to be, not sure about current state) possible to reimplement specific parts, but not without modifying the original source as it was never designed to be modular (it originally started as a wrapper for ebuild.sh). When I added the elog and package set subsystems I tried to make them somewhat modular so people could easily extend them with custom modules, but allowing people to actually reimplement the core subsystems would have required a very different (and more complicated) approach.
The key problem for improving specific pieces is to actually locate them in the first place. With the refactoring a while ago, lazy loading of modules, implementation of job handling and other fancy changes this has become quite a task compared to the old days with only a dozen (but sometimes huge) sourcefiles. Other problems include things being coupled to close to avoid interface bloat, too complex data structures in many cases and simply way too many features implemented that require special casing. Mind I'm not trying to blame anyone here, I'm just as guilty of that as everyone who has worked on portage.
As for breaking complex constructs down to simpler ones, that requires to come up with the basic building blocks first, and the dependency syntax isn't actually that complex to start with (even with the increased complexity from recent EAPIs). The problems in portage dependency processing are of semantic nature, e.g. something as simple as choosing the "correct" item from an or-dependency is actually quite difficult due to what different people consider "logical". |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 450
|
| Posted: Thu Oct 24, 2013 5:35 am Post subject: |
|
|
I'm just reading the EAPI documentation to understand portage and the ebuild better and the changes in EAPI 5. Mostly changed as i see: sub-slots. As [1] says
| Quote: |
The sub-slot is used to represent cases in which an upgrade to a new version of a package with a different sub-slot may require dependent packages to be rebuilt. When the sub-slot part is omitted from the SLOT definition, the package is considered to have an implicit sub-slot which is equal to the regular slot.
|
I'm no native english speaker. Does this mean, that the sub-slots are basically used for
| Code: | | emerge @preserved-rebuild |
?
Is this sub-slot change in EAPI5 the reason why the performance has been decreased or is this not directly dependant to that sub-slot change?
Bye
schorsch
[1] http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-660007.2 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Thu Oct 24, 2013 5:00 pm Post subject: |
|
|
| schorsch_76 wrote: | Does this mean, that the sub-slots are basically used for
| Code: | | emerge @preserved-rebuild |
|
sub-slots are not exactly the same as @preserved-rebuild, but nevertheless they are some partial replacement for it (though they are not meant to be a complete replacement).
For instance if you have a package B which depends on "A:=" then, when B is installed, portage stores the slot+subslot (at the time of emerging package B) of package A in the database. Now if (sub)slot of A changes (e.g. due to an update) the dependency "A:=" of package B is no longer satisfied, and so package B will be forced to be reemerged, too.
| Quote: | | Is this sub-slot change in EAPI5 the reason why the performance has been decreased |
Depends how you see it: The mechanism itself is probably not a dramatic slowdown, but it leads implicitly to a "much richer" dependency graph (mistakenly usually called "tree"), i.e. with much more edges, and so algorithms working on the graph, trying to resolve loops etc, take much more time. |
|
| Back to top |
|
|
xaviermiller
Bodhisattva
Joined: 23 Jul 2004
Posts: 8708
Location: ~Brussels - Belgique
|
| Posted: Thu Oct 24, 2013 7:26 pm Post subject: |
|
|
Just an idea : would it be possible to disable by parameter some smart feature and have a faster, but with some blocks that should be solved as 2 years ago ?
_________________
Kind regards,
Xavier Miller |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 450
|
| Posted: Thu Oct 24, 2013 8:11 pm Post subject: |
|
|
| mv wrote: | | For instance if you have a package B which depends on "A:=" then, when B is installed, portage stores the slot+subslot (at the time of emerging package B) of package A in the database. Now if (sub)slot of A changes (e.g. due to an update) the dependency "A:=" of package B is no longer satisfied, and so package B will be forced to be reemerged, too. |
Ok, but as far as i understand it, an update of package A:= would lead to a new version and B needs to reemerge too.
Example:
gcc-4.7 is in slot 4.7. Why would i need a sub-slots 4.7/abc 4.7/def? A slot is a solution to keep multiple versions of the same package on the maschine. Is a sub-slot maybe a special version like 4.7/debug 4.7/release 4.7/test? |
|
| Back to top |
|
|
TomWij
Retired Dev
Joined: 04 Jul 2012
Posts: 1553
|
| Posted: Thu Oct 24, 2013 8:19 pm Post subject: |
|
|
| XavierMiller wrote: | | Just an idea : would it be possible to disable by parameter some smart feature and have a faster, but with some blocks that should be solved as 2 years ago ? |
Lowering the backtrack value should reach that. Also see my earlier example regarding the slot operator rebuilds:
| Quote: | | There are options (--ignore-built-slot-operator-deps=y and --rebuild-if-new-slot=n) that can be played with to get rid of that 33 seconds; but if you do so, you are moving that time to other places where you might possibly have to spend a lot more time on it. I don't think it is really worth disabling it... |
|
|
| Back to top |
|
|
pums974
Tux's lil' helper
Joined: 27 Feb 2010
Posts: 78
|
| Posted: Thu Oct 24, 2013 8:33 pm Post subject: |
|
|
Hi,
| XavierMiller wrote: | | The mechanism itself is probably not a dramatic slowdown, but it leads implicitly to a "much richer" dependency graph (mistakenly usually called "tree"), i.e. with much more edges, and so algorithms working on the graph, trying to resolve loops etc, take much more time. |
| mv wrote: | | Well, dependency resolution is an NP hard problem (just a guess, maybe it is even really exponential) |
I'm not an expert but do we really need to know the << "much richer" dependency graph >> all the time ?
Can't we use two depedency graph ?
One without subslots (the old dependency graph) which seems to be sufficient to compute what change in the installed tree have to be done (updates, new packages, removals).
On with subslots but without non-installed packages which seems to be needed to compute which packages must be reemerged to satisfied sub-slots changes.
In that way couldn't we make the global computation a two step thing with much lighter graph and thus have a global computationnal gain ? |
|
| Back to top |
|
|
den_RDC
Apprentice
Joined: 25 Aug 2002
Posts: 166
Location: beercountry, Belgium;)
|
| Posted: Thu Oct 24, 2013 8:59 pm Post subject: |
|
|
For reference, emerge -uDNavt world takes 6m10s on my machine, a Core i7 860 with tons of ram and a fast SSD. Because i have a small dozen of slot conflicts to get resolved the need to wait 6-7 minutes each time i hope to fix something is killing the upgrade proces. I spent the last 2 weeks trying to get it going.
Might be my own fault, maybe I shouldn't keep an 8-9 year old gentoo install and start a fresh one ...
_________________
Fan of the "Survivor Warriors of the Evil Empire of Bloody Destruction and Bloody Darkness" |
|
| Back to top |
|
|
pigeon768
l33t
Joined: 02 Jan 2006
Posts: 683
|
| Posted: Thu Oct 24, 2013 10:19 pm Post subject: |
|
|
| Code: | ~ $ time emerge -uNDpv @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
real 0m23.178s
user 0m22.884s
sys 0m0.304s
~ $ emerge --version
Portage 2.2.7 (default/linux/amd64/13.0, gcc-4.8.1, glibc-2.17, 3.11.6-gentoo x86_64)
~ $ head /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz
stepping : 9
microcode : 0x12
cpu MHz : 1534.000
cache size : 3072 KB
physical id : 0 |
I'm using the sqlite auxdbmodule thingie. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Oct 25, 2013 6:18 am Post subject: |
|
|
I read analysis of the portage code-base, some of it precise and detailed and in genone's case authoritative as well. However we've been here before: eventually the realisation comes that that knowledge, of the current implementation and its problems, should be used to design a portage-2 from the ground-up. And that's exactly what pkgcore is, by the lead portage developer before zmedico, who handed it over to him, and whose code is indeed imported into portage.
What I'm saying is, I wish Gentoo would call time on the portage codebase, put it into maintenance mode, and focus on pkgcore instead. The work's all been done, and it runs incredibly quickly in my experience; to the extent that about 5 years ago we added an "smerge" mode to update which uses the pkgcore resolver (ie pmerge) to get the list, and portage to build them (this was in the days before --jobs.) This was specifically to make maintenance of update a lot easier as well, since you tend to have to re-run the resolve a lot when you're hacking your front-end code to bits.
Further, most of the stuff people are interested in, such as eclass functionality actually happens in bash, and the package manager has little to do with it, except where new metadata which affects the dependency calculation, such as subslot-operators (which are used as a poor-man's LIB_DEPEND in the cases which most affect users) is introduced. |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 450
|
|
| Back to top |
|
|
LoTeK
Apprentice
Joined: 26 Jul 2012
Posts: 270
|
| Posted: Fri Oct 25, 2013 9:51 am Post subject: |
|
|
Hi,
I also noticed that portage has become terribly slow. At first I thought it is because of my encrypted harddrive (could this be a factor too?), but now I see that many people have this problem. While reading this thread I've realized how bad my knowledge of portage (and therefore the whole gentoo system) is. 
So I can't say many useful things, but I'm with Xavier on this and it's absolutely unacceptable! I'm in a very annoying situation, because I kind of hate portage, but there seems to be no alternative. I even searched for other distros, but Gentoo is still by far the best distro for me. afaik no other system has for example USE flags.
from https://forums.gentoo.org/viewtopic-t-946474-start-0.html
| Genone wrote: | | LoTeK wrote: | I thought that there are only /etc/portage/make.conf(global) and /etc/portage/package.use (local, with higher precedence)...
what are those other uses? |
Disclaimer: I've been out of touch with Gentoo development for a couple years, so the following info may could be incorrect to some degree.
Originally (when I started using Gentoo around 2002) you had to configure USE flags by a) selecting a profile (make.defaults), b) adjust your standard flags via make.conf and c) could override those via the USE environment variable. make.globals could have technically been used, but to my knowledge it never contained a USE setting. I actually forgot about auto-use settings in those days adding a fourth location, but that's a topic you better don't ask about (everyone is happy to not have that anymore).
As the package base grew there was a need to set USE flags (and other stuff) per package in a consistent way, so 2003/2004 /etc/portage/package.use was added. That was still simple enough to understand, even though the implementation was a bit messy.
Later on support for package.use was also added for profiles, you got default USE settings in ebuilds directly, forced overrides via package.use.mask/force, the whole USE_EXPAND system, certain specific flags being triggered by portage internally, the mess called USE-dependencies, the other mess called REQUIRED_USE, proper per-package environment overrides (I think, not sure on that) supporting USE as well and probably a couple more I forgot or don't know about. Oh, and then you can adjust the priority of some of these locations via USE_ORDER if it still exists.
As I said before, there are good reasons for all of these extensions, but they make the topic pretty complicated to understand completely, as you easily get lost in specific details. |
The idea of USE flags is very simple and it can't be that it grows into some bloated, unmanageable monster. Likewise with portage. I mean, I prefer KISS, even when I have to spend some time to solve problems. When I have to do things on my own, I'll learn something, but when portage does everything I won't.
Since gentoo sees itself as a system that hides nothing and worships choice, I can't understand why everything is going toward "user friendlyness".
The complexity of reverse dependency checking makes sense, but why did it worked just a few month ago without this lack of performance? I doubt that the portage tree is very very much more complex than it was 4 month ago.
And I have to ask once again why portage is written in python? since portage is the core of gentoo, why aren't more developers involved with fixing this problem? why not rewrite it in C?
sorry if some of my questions are stupid because I'm too uninformed, but I have to ask them!
_________________
"I want to see gamma rays! I want to hear X-rays! Do you see the absurdity of what I am? I can't even express these things properly because I have to conceptualize complex ideas in this stupid limiting spoken language!" |
|
| Back to top |
|
|
ulenrich
Veteran
Joined: 10 Oct 2010
Posts: 1480
|
| Posted: Fri Oct 25, 2013 10:24 am Post subject: |
|
|
| steveL wrote: | | What I'm saying is, I wish Gentoo would call time on the portage codebase, put it into maintenance mode, and focus on pkgcore instead. The work's all been done, and it runs incredibly quickly in my experience; to the extent that about 5 years ago we added an "smerge" mode to update which uses the pkgcore resolver (ie pmerge) to get the list, and portage to build them (this was in the days before --jobs.) This was specifically to make maintenance of update a lot easier as well, since you tend to have to re-run the resolve a lot when you're hacking your front-end code to bits. |
I think this the way to go. But I doubt it is easy because of too many dirty workaround hacks landed into portage itself. Really clean up code of portage first! That what you mean with maintenance mode?
I doubt the main developer zmedico will do: He knows portage code in and out, he quickly has new workarounds for new problems at hand. But fresh eyes on the code are needed to see where things have to be cleanly seperated.
The dependency tree will exponentially grow in the future. But portage is on its limits to handle this today. The founding ground of a new infrastructure has to be build.
Why not breaking the problem horizontally:
- "layman -S" as the users command to update the tree(s)
- caching of the portage tree on disk
- system update
- atom handling
If you want competition and a more effective development of such tools, you need
the PackageManagerSpecification as stable ground. And
work on the "vertical integration" problem (yet another):
- the strict specification of the layers
- a competition of strict solvers as heart
- eclasses as special case management
- above all of that the helpers providing "artificial intelligence" to the user |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Oct 25, 2013 7:56 pm Post subject: |
|
|
| ulenrich wrote: | | steveL wrote: | | What I'm saying is, I wish Gentoo would call time on the portage codebase, put it into maintenance mode, and focus on pkgcore instead. The work's all been done, and it runs incredibly quickly in my experience. |
I think this the way to go. But I doubt it is easy because of too many dirty workaround hacks landed into portage itself. Really clean up code of portage first! That what you mean with maintenance mode? |
What "dirty workaround hacks" do you mean?
Not exactly: maintenance mode simply means "stop adding new features to portage" and only allow bug-fixes.
| Quote: | | I doubt the main developer zmedico will do: He knows portage code in and out, he quickly has new workarounds for new problems at hand. But fresh eyes on the code are needed to see where things have to be cleanly seperated. |
We've had several sets of fresh eyes on this; jstubbs, ferringb, zmedico and genone to name the ones familiar to me.
And zmedico said to me he'd rewrite the whole thing if he could as well, shortly before he went off-line (he hasn't been on IRC in the last few weeks.) For all I know he's doing exactly that: though I doubt it; imo he's taking a break, since he hasn't had one in several years, but that is pure speculation.
| Quote: | | The dependency tree will exponentially grow in the future. But portage is on its limits to handle this today. The founding ground of a new infrastructure has to be build. |
Yes and that's called pkgcore. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9523
Location: beyond the rim
|
| Posted: Mon Oct 28, 2013 11:16 am Post subject: |
|
|
| LoTeK wrote: | | Since gentoo sees itself as a system that hides nothing and worships choice, I can't understand why everything is going toward "user friendlyness". |
Well, there are a lot of factors playing into that, from changed user expectations, changes in upstream technologies up to different attitude with new developers/leadership. Like 10 years ago nobody cared about having multiple python versions in parallel, ruby wasn't known mainstream, java was only available as a blob from Sun and so on. Developers always have to balance development cost, maintenance cost and support cost against each other, if you get the same complaints again and again eventually you're going to do something about it. And often things look good on paper/developer systems, but don't work as expected in user practice, and rolling back isn't always a viable option.
| Quote: | | And I have to ask once again why portage is written in python? since portage is the core of gentoo, why aren't more developers involved with fixing this problem? why not rewrite it in C? |
For a start:
a) Rewrite in C isn't necessarily faster/better
b) You'd have to reimplement a LOT of stuff (or add a ton of dependencies) that is available for free with python. This includes basic stuff like memory management, core datatypes like lists/maps/strings (or even unlimited integers) and goes up to high level features like XML handling and transparent compression.
c) Development and bugfixing would be much slower (more code to write for the same result, always recompile to test changes, no introspection facilities, ....)
And simply "more developers" working on the same issue are rarely a solution and short-term may actually be harmful. See above about "locating the problems". |
|
| Back to top |
|
|
TomWij
Retired Dev
Joined: 04 Jul 2012
Posts: 1553
|
| Posted: Mon Oct 28, 2013 11:36 am Post subject: |
|
|
| Genone wrote: | | a) Rewrite in C isn't necessarily faster/better |
With the focus on code of high performance; it can very well be, it is not uncommon to rewrite parts in C / C++ (eg. see pkgcore) to gain additional performance.
| Genone wrote: | | b) You'd have to reimplement a LOT of stuff (or add a ton of dependencies) that is available for free with python. This includes basic stuff like memory management, core datatypes like lists/maps/strings (or even unlimited integers) and goes up to high level features like XML handling and transparent compression. |
While I can't comment about C; most of that is free in C++, and some of those high level features can be implemented in a single file. But well, there isn't much reason to be afraid form a small set of dependencies...
| Genone wrote: | | c) Development and bugfixing would be much slower (more code to write for the same result, always recompile to test changes, no introspection facilities, ....) |
It depends on how you write the code; I have not seen statistics or examples of how rewriting code in another language can lead to more code, it can very well be the opposite.
Incremental recompilation as well as proper development practices makes recompilation easy; it shouldn't matter anyway, because the most time should go into design and the least into compilation.
C++ supports introspection facilities; but yes, C indeed doesn't seem to.
| Genone wrote: | | And simply "more developers" working on the same issue are rarely a solution and short-term may actually be harmful. See above about "locating the problems". |
Exactly, http://en.wikipedia.org/wiki/The_Mythical_Man-Month |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Oct 28, 2013 1:04 pm Post subject: |
|
|
| TomWij wrote: | | While I can't comment about C; most of that is free in C++ |
Parsers are not. For special tasks like XML there are parser libraries, but others have to be implemented (or taken from e.g. eix, as I already mentioned...).
| Quote: | | Genone wrote: | | c) Development and bugfixing would be much slower (more code to write for the same result, always recompile to test changes, no introspection facilities, ....) |
It depends on how you write the code; I have not seen statistics or examples of how rewriting code in another language can lead to more code, it can very well be the opposite. |
With eix, I have seen that it is much more cumbersome to implement some major change as e.g. for a new ABI than it would be in a "more higher level languae" like python: One problem is that the syntax is rather cumbersome (this would be better with things like "auto" from C++11 which eix currently avoids due to compatbibility reason). Another problem is that you have to care much more about memory management; again, C++11 features (like std::*ptr*) might help but then at the cost of some overhead - which is exactly what to want to avoid when speed issues are the main reason why you are switching to C/C++. Anyway, I would not suggest to use anything more lower-level than C++11. Currently, this is a compatiblity issue...
Not so sure what is meant here with "introspection", but of course there is gdb...
| Genone wrote: | | And simply "more developers" working on the same issue are rarely a solution and short-term may actually be harmful. See above about "locating the problems". |
Well, for the first implementation, it would not hurt: Many tasks can be strictly separated (parsing, bash-part, dependency resolving, CLI, could all be implemented first by completely different people). Of course, one should first agree about some basic libraries (like concatenating strings, splitting strings, reading files, etc.; things like you find in eixTk/...) |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 450
|
| Posted: Mon Oct 28, 2013 3:44 pm Post subject: |
|
|
| @mv: Most of this can be replaced by boost. Even for older gcc versions. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon Oct 28, 2013 8:22 pm Post subject: |
|
|
| TomWij wrote: | | Genone wrote: | | c) Development and bugfixing would be much slower (more code to write for the same result, always recompile to test changes, no introspection facilities, ....) |
It depends on how you write the code; I have not seen statistics or examples of how rewriting code in another language can lead to more code, it can very well be the opposite. |
Not when you go from a higher-level language to a lower-level one; unless you use decent libs, as every good C coder is taught to. And then your project may have "less code" in it, but overall there's a lot going on. OFC that happens with python, too: people just don't consider it, since it's part of the language runtime, and it's easy not to notice that eg: the python lxml is actually a wrapper to libxml2 and libxlst (C libraries.) The point is that to reimplement you need to start from the ground-up which is a lot of work, and honestly
Why bother when pkgcore has already shown the way?
| Quote: | | Incremental recompilation as well as proper development practices makes recompilation easy; it shouldn't matter anyway, because the most time should go into design and the least into compilation. |
Wow. Just wow. You are new. Yeah, there's no difference between theory and practice; in theory. Since you like quotes, consider (Edison?) "Genius is 1% inspiration and 99% perspiration". The 1% is the "design", and the other 99% is debugging the fscker. Or as genone put it: "often things look good on paper/developer systems, but don't work as expected in user practice." |
|
| Back to top |
|
|
LoTeK
Apprentice
Joined: 26 Jul 2012
Posts: 270
|
| Posted: Mon Oct 28, 2013 9:04 pm Post subject: |
|
|
| Genone wrote: | | LoTeK wrote: | | Since gentoo sees itself as a system that hides nothing and worships choice, I can't understand why everything is going toward "user friendlyness". |
Well, there are a lot of factors playing into that, from changed user expectations, changes in upstream technologies up to different attitude with new developers/leadership. Like 10 years ago nobody cared about having multiple python versions in parallel, ruby wasn't known mainstream, java was only available as a blob from Sun and so on. Developers always have to balance development cost, maintenance cost and support cost against each other, if you get the same complaints again and again eventually you're going to do something about it. And often things look good on paper/developer systems, but don't work as expected in user practice, and rolling back isn't always a viable option. |
But if the user expectations are "user-friendlyness" then there are an army of linux distributions that give them what they want. I put quotes around the it, because to provide an "easy-to-use-environment" you have to make the system a much more complicated. Therefore people that work with computers (not in a secretary way) or are enthusiasts have a hard time with a complex system under the "easy" hood. Moreover with a simple system you have less cost, maintenance, etc, haven't you? I don't know if I probably missed your point in the thread section I've posted, but when even developers or ex-developers don't understand the system in every detail how can anyone have proper control over the system?
| Genone wrote: | | LoTeK wrote: | | And I have to ask once again why portage is written in python? since portage is the core of gentoo, why aren't more developers involved with fixing this problem? why not rewrite it in C? |
For a start:
a) Rewrite in C isn't necessarily faster/better
b) You'd have to reimplement a LOT of stuff (or add a ton of dependencies) that is available for free with python. This includes basic stuff like memory management, core datatypes like lists/maps/strings (or even unlimited integers) and goes up to high level features like XML handling and transparent compression.
c) Development and bugfixing would be much slower (more code to write for the same result, always recompile to test changes, no introspection facilities, ....)
And simply "more developers" working on the same issue are rarely a solution and short-term may actually be harmful. See above about "locating the problems". |
But since the kernel and almost the whole UNIX environment are written in C, you can't avoid C. But if you have a package manager in python then you have to include a whole lot of code, just for a package manager, haven't you?
Why is it not better to have a clean environment that is as uniform as possible? I don't know what you mean with the "free stuff with python", but isn't the whole runtime, etc in the end written in C? For example when you create an object in a OO language, doesn't the language call "malloc" in the end?
Of course I can imagine the huge amount of work, but at least "in theory" it would be better to go this way, wouldn't it?
afaik are many package managers written in C or C++ (pacman, aptitude, etc), so why should it be much slower to develop? Or are those package managers not comparable to portage, because it's source based?
For the code size: in C you can see the size, it's not hidden from you, but in python, C++ it is. Is this only important for system code?
Since C is that old and mature, aren't there way enough libraries for everything we need with portage?
| mv wrote: | | With eix, I have seen that it is much more cumbersome to implement some major change as e.g. for a new ABI than it would be in a "more higher level languae" like python: |
I can't comment on the implementation, but when you search a package with eix, you have your results in a blink of an eye, while with emerge -s pkg you wait, and wait and wait...
If I have to pick python or C++, I probably would go with python, since C++ is IMO "not C", but also not "high-level OO" like for example java. What is even worse is that the language is very complicated while C is small and simple.
Ok, this would probably fit into another thread about programming languages
_________________
"I want to see gamma rays! I want to hear X-rays! Do you see the absurdity of what I am? I can't even express these things properly because I have to conceptualize complex ideas in this stupid limiting spoken language!" |
|
| Back to top |
|
|
TomWij
Retired Dev
Joined: 04 Jul 2012
Posts: 1553
|
| Posted: Mon Oct 28, 2013 9:41 pm Post subject: |
|
|
| steveL wrote: | | TomWij wrote: | | It depends on how you write the code; I have not seen statistics or examples of how rewriting code in another language can lead to more code, it can very well be the opposite. |
Not when you go from a higher-level language to a lower-level one |
You do have statistics and examples for that?
| steveL wrote: | The point is that to reimplement you need to start from the ground-up which is a lot of work, and honestly
Why bother when pkgcore has already shown the way? |
Because to reimplement you need to start from the ground-up; why bother with pkgcore, joining forces will slow its progress down as per The Mythical Man-Month.
| steveL wrote: | | Quote: | | Incremental recompilation as well as proper development practices makes recompilation easy; it shouldn't matter anyway, because the most time should go into design and the least into compilation. |
Wow. Just wow. You are new. Yeah, there's no difference between theory and practice; in theory. Since you like quotes, consider (Edison?) "Genius is 1% inspiration and 99% perspiration". The 1% is the "design", and the other 99% is debugging the fscker. Or as genone put it: "often things look good on paper/developer systems, but don't work as expected in user practice." |
In practice, good practice does not go without good theory; so, indeed, due to their causation relationship they cannot be one and the same thing if you want a good result. Design in the context of software development is not inspiration, it is rather the perspiration to write down what you or your customers are inspired about; inspirations start with thoughts in your mind that need to be written down into concepts and requirements, which should go through proper review. Your quote doesn't reflect any of that and thus doesn't hold in the context of software development; if you think 1% of the time, then no wonder that you end up 99% of the time with cleaning up the crap that was just the result of quick thoughts. Having to debug the whole time is a red herring; it just means you aren't using proper development practices, because either your design hasn't took it into account or tests that verify the design's behavior are missing. It means you haven't thought about it all too well... |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
