| View previous topic :: View next topic |
| Author |
Message |
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
 Posted: Thu Jan 11, 2007 8:18 am Post subject: |
 |
|
| steveL wrote: | | AFAIC portage should just be written in bash. |
I'm not so sure about that.
Scripting in bash is fun, as long as one artificially limits the file names or inputs which can be processed by the script.
That is, the file names or input strings processed should better not contain any
- space characters
- tab character
- single quotes
- double quotes
- dollar signs
- exclamation points
- greater than / less than characters
- vertical bar symbol ("or")
- ampersand character
- literal backslashes
- question marks
- asterisks
- start with a tilde
- fancy UTF-8 characters
- newline characters
which are all perfectly legal characters in filenames,
or the script is likely to run into problems sooner or later, if the coder is not extremely careful.
Of course, one is free to say "such characters should not be used in filenames", but that won't stop a script from failing if an unaware user still creates such file names by accident.
Depending on what the script does, such failures could be everything from harmless to catastrophic.
In other languages, there are no or at least not that many issues with "special/meta characters" and quoting as in bash/sh.
And then not to forget the problem of all shell languages: The arguments of each command can be seen by anyone using "ps -Af", which is bad fun if passwords or sensitive data have to be processed.
That does not mean bash is bad per se.
But it's not a general purpose language, and will never beat Perl, Python, C, awk or even JAVA for application development.
In my opinion, awk is the least-bloated language capable of actual application development. At least for small applications. But bash isn't. |
|
| Back to top |
|
 |
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Thu Jan 11, 2007 3:00 pm Post subject: |
|
|
Personally, I think a package managment system should be in a hard-coded language (I'm thinking C here), or an easy to use scripting language that can actually do the job, and the devs know well enough to work with (python, perl, ruby). awk and bash, while fun to use and they work for scripts, I think should be left behind for such an effort.
Personally I think making ebuilds in bash is a bad idea (because, well, it requires bash to be installed, which is a wasted shell if you use, say, zsh) since it isn't too hard to setup a syntax in a language to handle such things. If you use a database, then even better.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Thu Jan 11, 2007 3:00 pm Post subject: |
|
|
| Conan wrote: | sourcemages package manager is written in pure bash.
It's kind of silly. |
Works for a starting point if nothing else. Got to give bash that.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
Conan
Guru
Joined: 02 Nov 2004
Posts: 360
|
| Posted: Thu Jan 11, 2007 3:44 pm Post subject: |
|
|
err,
1) You're arguement that it requires bash to be installed doesn't hold weight because portage also requires bash to be installed.
2) Sourcemage's package manager is just as functional as portage, it even does some things that portage can't do currently. I'd say its more than a starting point  |
|
| Back to top |
|
|
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
| Posted: Thu Jan 11, 2007 7:07 pm Post subject: |
|
|
| Dralnu wrote: | | awk and bash, while fun to use and they work for scripts, I think should be left behind for such an effort. |
I agree.
Portage is a rather complex tool (as any capable package management system will be), and this beyond the scope of application well suited for bash, and also awk.
I also suppose much of the difficulties with ebuilds originate from the fact that bash, while providing powerful means of expression, does not and cannot actually enforce the ebuild layout guidelines.
For instance, many ebuilds use USE-flags without specifying them in the "IUSE=" lines, although they are supposed to do so.
Just look at the many "QA"-warnings when emerging ebuilds.
Add that to missing or incorrectly specified dependency information, or look at the mess with /etc/init.d where dependencies are verifiably resolved incorrect in some (although rare) situations.
The Linux Standard Base has a clear and well-defined run-level system. Was it really necessary to change this - in favor to a new, more complicated system which, although more elegant, just does not always work correctly, and is totally incompatible with all the other distributions?
I think, Gentoo and the ebuild system are too complicated in this regard, and should be simplified (ebuild scripting) or unified (LSB/FHS conformance).
On the other hand, most of the time it works.
And even if it does not, it's usually rather simple to fix - if you are a programmer and have an understanding of the programming languages involved. Most of the time, it's just the names of header files which have changed, or some declarations are missing. Such problems are rather easy to fix using the "ebuild compile" command and a text editor.
These are also the reasons why I still stick with Gentoo despite of the above facts - there is no other distro except for LFS (which can hardly be called a "distro") where one can do this.
Combined with USE flags and user-defined CFLAGS, this is raw power for the administrator - it's quite the opposite VISTA tries to do.
I only wish ebuilds and Portage were a bit more stable, forcing the administrator to fewer often has to actually wield those raw powers...
And writing ebuilds in a more structured, easier to be verified way (in some other specification language), would be a good starting point.
I wouldn't even use C for this - some sort of expert system might be even more appropriate. There are many good and free expert systems available, some of them completely written in C. And even a full-fledged, C based XPS will hardly ever require as much space as a full Python installation does.
Expert systems could be made intelligent enough to understand all the problems regarding package interdependencies, USE-flags and update build order resolution. And take actions to resolve those problems.
But it's just dreaming - I know this cannot happen. No-one will convert hundreds of thousands of existing ebuilds into whatever different specification language just to make them more reliable.
So ebuilds will continue to be flawed, admins will continue to be able to resolve most of those matters "by hand", and "normal" users without programming experience... will continue to run away and choose UBUNTU/FEDORA instead.
But hey - what am I fussed about? I am a programmer, Gentoo is the programmers/power users club, and so I should feel right at home anyway...  |
|
| Back to top |
|
|
gringo
Advocate
Joined: 27 Apr 2003
Posts: 3793
|
| Posted: Thu Jan 11, 2007 7:41 pm Post subject: |
|
|
as a side note :
| Quote: | | Hmm - nvidia-drivers/alsa-driver/ndiswrapper all have problems with sandbox violations during emerge on 2.6.19 kernels so I thought I could use this feature and placed this line in /etc/portage/env/media-sound/alsa-driver |
if you use gentoo-sources or 2.6.19.2 its already fixed. More info here
And thanks for sharing your scripts Guenther, very handy
cheers
_________________
Error: Failing not supported by current locale |
|
| Back to top |
|
|
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
| Posted: Thu Jan 11, 2007 8:06 pm Post subject: |
|
|
| gringo wrote: | | if you use gentoo-sources or 2.6.19.2 its already fixed. |
Thanks for the info.
| gringo wrote: | More info here
| Quote: | "Because then when
the compilation runs as root some as versions will delete /dev/null on a error.
This has happened in the past." |
|
ROTFL! |
|
| Back to top |
|
|
swimmer
Veteran
Joined: 15 Jul 2002
Posts: 1330
Location: Netherlands
|
| Posted: Fri Jan 12, 2007 12:16 am Post subject: |
|
|
| gringo wrote: | if you use gentoo-sources or 2.6.19.2 its already fixed. More info here
cheers |
Hmm - I'm using =sys-kernel/gentoo-sources-2.6.19-r3 and still get sandbox violations with those modules:
=media-sound/alsa-driver-1.0.14_rc1
=x11-drivers/nvidia-drivers-1.0.9746
=net-wireless/ndiswrapper-1.34
All that on AMD64 ...
Greetz
swimmer |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Fri Jan 12, 2007 1:40 am Post subject: |
|
|
| Conan wrote: | err,
1) You're arguement that it requires bash to be installed doesn't hold weight because portage also requires bash to be installed. |
Whats your point here? I obviously and missing something here.
| Quote: | | 2) Sourcemage's package manager is just as functional as portage, it even does some things that portage can't do currently. I'd say its more than a starting point |
I still say it would be better to use a diffrent language. Like he said, bash has some fairly hard-set limits, and it, again, requires bash, which is a waste for people who don't use it.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
aidanjt
Veteran
Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland
|
| Posted: Fri Jan 12, 2007 2:21 am Post subject: |
|
|
source based package managers will always have a level of shell script dependency anyway with regards to at least configure.
That said, portage really needs a rewrite to a compiled language since it's now getting too heavy for python to handle efficiently.
I don't mind the ebuilds themselves having a bash dependency but the portage core defiantly needs rewritten. |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Fri Jan 12, 2007 2:34 am Post subject: |
|
|
| AidanJT wrote: | source based package managers will always have a level of shell script dependency anyway with regards to at least configure.
That said, portage really needs a rewrite to a compiled language since it's now getting too heavy for python to handle efficiently.
I don't mind the ebuilds themselves having a bash dependency but the portage core defiantly needs rewritten. |
The problem is none of the devs want to spend that kind of time on it. Python might could handle Portage well, but it would still need a rewrite.
Using a script like bash for something like ebuilds it kind of stupid imho. Hell, I think something like xml would do better, with the best option being a database.
Personally I'd prefer a modular design, with things such as emerge --search, the data-handling system (saving ebuilds, the caches, ect), and all that being entered seperatly, with the core making calls to each so that, say, you wanted to use a specific database (we'll say Oracle here) instead of plain text or MySQL, you could emerge a new data-handler. Problem with converting between them I think would be fairly easy to get around (output things into plain text, then reinput them into the new database system), and it would give the user control over what Portage would use for what, with emerge itself being hard-coded into the core of the system (to remove the chance of breakage).
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
PaulBredbury
Watchman
Joined: 14 Jul 2005
Posts: 7310
|
| Posted: Fri Jan 12, 2007 2:47 am Post subject: |
|
|
| Ebuilds have to be scripts - look at the complexity of e.g. /usr/portage/x11-drivers/nvidia-drivers/nvidia-drivers-1.0.9746.ebuild |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Fri Jan 12, 2007 3:11 am Post subject: |
|
|
| PaulBredbury wrote: | | Ebuilds have to be scripts - look at the complexity of e.g. /usr/portage/x11-drivers/nvidia-drivers/nvidia-drivers-1.0.9746.ebuild |
Still doesn't mean you have to use bash. Its not hard to put a scripting language into the code.
Heck, awk may even work well for that.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
gringo
Advocate
Joined: 27 Apr 2003
Posts: 3793
|
| Posted: Fri Jan 12, 2007 9:01 am Post subject: |
|
|
| Quote: | "Because then when
the compilation runs as root some as versions will delete /dev/null on a error.
This has happened in the past." |
| Quote: | Hmm - I'm using =sys-kernel/gentoo-sources-2.6.19-r3 and still get sandbox violations with those modules:
=media-sound/alsa-driver-1.0.14_rc1
=x11-drivers/nvidia-drivers-1.0.9746
=net-wireless/ndiswrapper-1.34
All that on AMD64 ... |
amd64 here too and no problems. -r4 is available, try out and reopen above bug if it doesnt work.
cheers
_________________
Error: Failing not supported by current locale |
|
| Back to top |
|
|
Element Dave
Tux's lil' helper
Joined: 10 Nov 2006
Posts: 82
|
| Posted: Tue Jan 23, 2007 6:22 am Post subject: |
|
|
| Guenther Brunthaler wrote: | Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.
From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.
Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited. |
Another problem with relying on source code as opposed to proper documentation is that one seldom has any indication whether certain functionality is intended to persist into future versions, If public hooks are documented, there is at least some sense that they won't disappear tomorrow. |
|
| Back to top |
|
|
Element Dave
Tux's lil' helper
Joined: 10 Nov 2006
Posts: 82
|
| Posted: Tue Jan 23, 2007 6:26 am Post subject: |
|
|
| steveL wrote: | | AFAIC portage should just be written in bash. |
I hope you have a good sense of humor. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Tue Jan 23, 2007 3:53 pm Post subject: |
|
|
| Element Dave wrote: | | steveL wrote: | | AFAIC portage should just be written in bash. |
I hope you have a good sense of humor. |
 TBH I can't believe I wrote that! |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Tue Jan 23, 2007 11:49 pm Post subject: |
|
|
| Element Dave wrote: | | Guenther Brunthaler wrote: | Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.
From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.
Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited. |
Another problem with relying on source code as opposed to proper documentation is that one seldom has any indication whether certain functionality is intended to persist into future versions, If public hooks are documented, there is at least some sense that they won't disappear tomorrow. |
If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it".
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
likewhoa
l33t
Joined: 04 Oct 2006
Posts: 778
Location: Brooklyn, New York
|
| Posted: Wed Jan 31, 2007 2:24 am Post subject: |
|
|
thx Guenther Brunthaler this looks like a good feature but it would be best if you had implemented per-package not overall cflags for packages that are gonna be merged with custom cflags.
the whole point of the per-package custom package.{cflags,cxxflags,features,ldflags} is to set unique values for packages not one global set of package.{cflags,cxxflags,features,ldflags} per all. anyways, thanks for the scripts and your effort. I guess this would have been something for me if it handle the variables differently. I will stick with setting files with variables instead of your symbolic link method. you gotta remember that not all packages will merge with one global set compile flags. |
|
| Back to top |
|
|
Element Dave
Tux's lil' helper
Joined: 10 Nov 2006
Posts: 82
|
| Posted: Tue Mar 06, 2007 10:26 pm Post subject: |
|
|
| Dralnu wrote: | | If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it". |
Well, how does one know whether something is a "feature" if it isn't documented? Do you audit the source of every program on your machine to determine what "features" you have available?
Last edited by Element Dave on Tue Mar 06, 2007 11:08 pm; edited 1 time in total |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Tue Mar 06, 2007 10:38 pm Post subject: |
|
|
| Element Dave wrote: | | Dralnu wrote: | | If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it". |
Well, how does one know whether something is "feature" if it isn't documented? Do you audit the source of every program on your machine to determine what "features" you have available? |
Apparently some people do.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
Element Dave
Tux's lil' helper
Joined: 10 Nov 2006
Posts: 82
|
| Posted: Tue Mar 06, 2007 11:25 pm Post subject: |
|
|
| Dralnu wrote: | | Element Dave wrote: | | Dralnu wrote: | | If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it". |
Well, how does one know whether something is "feature" if it isn't documented? Do you audit the source of every program on your machine to determine what "features" you have available? |
Apparently some people do. |
That's unlikely. It certainly isn't an efficient use of time compared to reading documentation. Consider the amount of time needed to write documentation, and contrast that with the collective amount of time spent by the users (which also include developers) of that code that would have been saved had documentation been available. More progress would be made in many areas. |
|
| Back to top |
|
|
Dralnu
Veteran
Joined: 24 May 2006
Posts: 1919
|
| Posted: Wed Mar 07, 2007 2:21 am Post subject: |
|
|
Point is, its a generally stupid idea to make a feature avalible if you don't plan on people using it. This is why we have releases and development trees. Make sure the feature works as best you can, THEN release it. Considering this is over a package manager known to be used by coders and oddballs, you don't put anything past them.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner. |
|
| Back to top |
|
|
kowal
n00b
Joined: 23 Apr 2003
Posts: 40
|
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Aug 05, 2007 6:16 am Post subject: |
|
|
| Guenther Brunthaler wrote: | | steveL wrote: | | AFAIC portage should just be written in bash. |
I'm not so sure about that.
Scripting in bash is fun, as long as one artificially limits the file names or inputs which can be processed by the script.
That is, the file names or input strings processed should better not contain any
- space characters
- tab character
- single quotes
- double quotes
- dollar signs
- exclamation points
- greater than / less than characters
- vertical bar symbol ("or")
- ampersand character
- literal backslashes
- question marks
- asterisks
- start with a tilde
- fancy UTF-8 characters
- newline characters
which are all perfectly legal characters in filenames,
or the script is likely to run into problems sooner or later, if the coder is not extremely careful.
Of course, one is free to say "such characters should not be used in filenames", but that won't stop a script from failing if an unaware user still creates such file names by accident.
Depending on what the script does, such failures could be everything from harmless to catastrophic.
In other languages, there are no or at least not that many issues with "special/meta characters" and quoting as in bash/sh.
And then not to forget the problem of all shell languages: The arguments of each command can be seen by anyone using "ps -Af", which is bad fun if passwords or sensitive data have to be processed.
That does not mean bash is bad per se.
But it's not a general purpose language, and will never beat Perl, Python, C, awk or even JAVA for application development.
In my opinion, awk is the least-bloated language capable of actual application development. At least for small applications. But bash isn't. |
Well I didn't see this post before, but basically you need to hang out in #bash on irc.freenode.org. Here's some links to start:
http://wooledge.org/mywiki/BashGuide | http://www.grymoire.com/Unix/index.html | http://wooledge.org/mywiki/BashFaq | and of course | http://www.shelldorado.com/ ;)
File globbing (eg for i in *) handles all those filenames, the only thing you need to be aware of is !quotes: do cmd "$i"; done
BASH is actually an excellent environment to code in imo. awk is nice too tho :D |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
