Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Alternative to etc-update
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Sat Jan 21, 2006 1:19 pm    Post subject: Alternative to etc-update Reply with quote

I was thinking about this, a way of "automising" etc-update, or dispatch-conf, or cfg-update. None are entirely friendly to new users, and while on the one hand I believe people should learn (they don't make userfriendly chisels - you have to learn how to use it), userfriendliness can be nice sometimes. Anways, the idea is below:

Currently config files are stored in /etc, and when you want to change something you edit that file. Next time you update a program a diff of /etc/current-conf and /etc/new-conf is shown, to highlight the changes which the user must make.

Instead, config files installed by programs could be stored in /etc (as now), but the user's changes stored in an overlay, in /etc/configoverlay, where only the changes are stored. Example below:

/etc/example-conf-file
Code:

ok=green
warning=yellow
error=red


/etc/configoverlay/example-conf-file
Code:

ok=black


The overlay overrides the program's config files, which themselves are unchanged. So in this example the colours used by the system would be:
ok=black
warning=yellow
error=red

Now when the user updates the progam, /etc/example-conf-file is replaced with a new one. Maybe the new file is:
Code:

ok=green
warning=orange
error=red
empty=black

and so they get the new features/colours (empty=black), and keep their old settings (in this case ok=black).

What do people think? Has this been tried already?

Thanks,

Erlend
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Sat Jan 21, 2006 1:49 pm    Post subject: Reply with quote

Yes, that could ease the configuration files update mechanics.

But there are corner cases, too:
  • How about gdm config file? There are several sections defining its behavior for different situations: general config, local login, remote login (to name a few). Those sections can contain similar settings, like greeting text - different for local and remote login. How to know to which section a user provided setting should apply?
  • What to do in a situation when one of said sections is removed from an original config file? (hint: it really happens) How to deal with such "orphaned" user provided settings?
  • What to do if an original configuration file is removed or renamed?
  • What if a config file with the same name exists in different sudirectories of /etc? Of course, now it doesn't happen, but if it happens, then what?
  • Programs in general expect the configuration file to be, uhm, exactly that: one file. How this "overlaying" mechanism should be implemented then? A patched version of unionfs? Do you want to do that? (hint: I don't have the time and enough experience)
  • People that use Linux/Unix/*BSD/whatever are already familiar with editing configuration files. How to deal with their habits and how to convince them that your proposal has real benefit for them?

If you know the answers to the above questions, post them! I'm sure that more corner cases could be found...

There exist some solutions, however. First, IIRC Debian uses a more sophisticated mechanism for automatically applying changes to the config files and preserving the user-made modifications at the same time. You'd have to see it for yourself first to believe me. ;)

Second, people use some kind of version control system (like CVS or Subversion) to store the config files. At the same time they have a complete history of modifications for each file, and they can easily revert to one of the previous versions if they don't like recent changes.

HTH
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Sat Jan 21, 2006 2:13 pm    Post subject: Reply with quote

Quote:
How about gdm config file? There are several sections defining its behavior for different situations: general config, local login, remote login (to name a few). Those sections can contain similar settings, like greeting text - different for local and remote login. How to know to which section a user provided setting should apply?

Hmm, I see what you mean. Either the updater would need gdm configs hard-coded into it, or just not work with gdm.

Quote:
What to do in a situation when one of said sections is removed from an original config file? (hint: it really happens) How to deal with such "orphaned" user provided settings?

Does it matter though? I mean if I added a random line to /etc/conf.d/net.eth0 it could just be ignored surely?

Quote:
What to do if an original configuration file is removed or renamed?

You mean if the user accidently removes /etc/example-conf-file ?

Quote:
What if a config file with the same name exists in different sudirectories of /etc? Of course, now it doesn't happen, but if it happens, then what?

The overlay should mirror the directory structure of /etc. So the overlay for xorg.conf would be in /etc/configoverlay/X11/xorg.conf.

Quote:
Programs in general expect the configuration file to be, uhm, exactly that: one file. How this "overlaying" mechanism should be implemented then? A patched version of unionfs? Do you want to do that? (hint: I don't have the time and enough experience)

A similar technique is used by /etc/modules.d and /etc/profile.d. I think a program parses the files in these subdirectories into one file... So etc-update could take the new program config file /etc/example-conf-file and /etc/configoverlay/example-conf-file and merge the results (look at my thread starter post) to give, for example:
Code:
ok=black
warning=yellow
error=red

This file would have to be stored, perhaps in /etc/finalconfig/example-conf-file, and the programs would read this file (that way individual programs don't need to worry about parsing the config file against the overlay).

Quote:
People that use Linux/Unix/*BSD/whatever are already familiar with editing configuration files. How to deal with their habits and how to convince them that your proposal has real benefit for them?

I'm not trying to spell an end to config files - they are great, human readable, easy to backup, and extremely unlikely to corrupt like a database might. People should edit config files, but it's the repetitive editing/re-editing of the same file I'm refering to... everytime you update you have to run etc-update or similar. This takes time, and could prevent Gentoo from being suitable as an offsite unmanned server.

Erlend
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Sat Jan 21, 2006 3:30 pm    Post subject: Reply with quote

Erlend wrote:
Quote:
What to do in a situation when one of said sections is removed from an original config file? (hint: it really happens) How to deal with such "orphaned" user provided settings?

Does it matter though? I mean if I added a random line to /etc/conf.d/net.eth0 it could just be ignored surely?

To some extent it does matter. What do do with a line from, say, /etc/configoverlay/conf.d/rc that overwrites some setting from /etc/conf.d/rc, when the original line is removed after baselayout update? And what if still applying the "orphaned" setting to the final /etc/finalconfig/conf.d/rc file messes for example the keyboard layout or console encoding? That's still a corner case for me.

Erlend wrote:
Quote:
What to do if an original configuration file is removed or renamed?

You mean if the user accidently removes /etc/example-conf-file ?

No, if the original file like /etc/conf.d/something is removed by emerge after updating a package. What to do with the leftover like /etc/configoverlay/conf.d/something ?

Erlend wrote:
A similar technique is used by /etc/modules.d and /etc/profile.d. I think a program parses the files in these subdirectories into one file... So etc-update could take the new program config file /etc/example-conf-file and /etc/configoverlay/example-conf-file and merge the results
...
This file would have to be stored, perhaps in /etc/finalconfig/example-conf-file, and the programs would read this file (that way individual programs don't need to worry about parsing the config file against the overlay).

Ok, but this would mean that configuration files now exist in a non-standard place (see Linux Filesystem Hierarchy Standard). This also means that all programs should be patched to use /etc/finalconfig instead of /etc - and that means only those that can be re-compiled (what about binary ones?)


Erlend wrote:
People should edit config files, but it's the repetitive editing/re-editing of the same file I'm refering to... everytime you update you have to run etc-update or similar. This takes time, and could prevent Gentoo from being suitable as an offsite unmanned server.


Heh, now we're getting somewhere! Do I understand correctly that you want an automatic system which would let you apply ALL the configuration changes without manual intervention, but preserving your changes at the same time? :mrgreen:

It may sound harsh to you, but have you seen Debian/Ubuntu in action? I support a lab full of Debian machines and all works really well. I can't remember the moment when upgrading the installation made a mess with my changes to the configuration settings. Most of the time only a dialog pops up where it's possible to select/confirm the choices that I've already made... And I run full Gentoo installation at home, on my notebook and on my server (call me a fan of Gentoo). But the simplicity of Debian upgrades is... somewhat I'd like to see in Gentoo. 8O

Note:
Something like this is already coming. In the new (~arch) baselayout there is a Perl-based (think: Debian's dpkg-reconfigure) mechanism for managing Gentoo configuration. It's called eselect. 8) A possible time-frame: Gentoo 2006.0
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Sat Jan 21, 2006 4:06 pm    Post subject: Reply with quote

Quote:
To some extent it does matter. What do do with a line from, say, /etc/configoverlay/conf.d/rc that overwrites some setting from /etc/conf.d/rc, when the original line is removed after baselayout update? And what if still applying the "orphaned" setting to the final /etc/finalconfig/conf.d/rc file messes for example the keyboard layout or console encoding? That's still a corner case for me.

Oh I'm with you now. Hmm, I guess if a setting that was once in /etc/conf.d/rc is removed after an update, the equivalent of etc-update for an overlay system would have to say, "well that options been removed so should i remove it from /etc/configoverlay/conf.d/rc?" It could have just been left out by the developer (but still a valid config option), or it could really have been removed from the program. If it's removed from the program it shouldn't be a problem, but if the developer is recommending the option be removed it's more difficult: who to follow, developer or user? I guess it could ask the user, at least this would happen less often than an individual file being updated.

Quote:
Ok, but this would mean that configuration files now exist in a non-standard place (see Linux Filesystem Hierarchy Standard). This also means that all programs should be patched to use /etc/finalconfig instead of /etc - and that means only those that can be re-compiled (what about binary ones?)

Yeah. Probably the correct way of doing it would be /etc/original-config.d/ /etc/configoverlay.d/ and the final config files in /etc/. Portage creates these config files so it could just place them in /etc/original-config.d/ and there could be an editor such that when the user "modifies" /etc/original-config.d the changes are written to /etc/configoverlay.d/.

Quote:
have you seen Debian/Ubuntu in action?

I haven't, I'd quite like to see ubuntu working. One of my friends is thinking of installing it, so I might have a look.

Where can I find out more about baselayout-1.12?

Thanks,

Erlend
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Sun Jan 22, 2006 2:36 am    Post subject: Reply with quote

Hrmm, this is quiet a problem with Gentoo, one solution may be forgetting about the user making manual changes to files in /etc.
Instead what you might have is /etc/portage/etc-configs/*.xml where the xml file contains user defined config settings for each program, the system could keep a syntax table for understanding how a config file(s) is put together and automatically detect depreciated or invalid settings, when a user makes a change to a config option in the xml file it could then write to the apropiate config file in etc.

Also, when a program is removed from portage, the config(s) in /etc could automatically be removed while keeping the users settings in the /etc/portage/etc-configs/*.xml file should they wish to remerge the program at a later date. I think something like this would be very handy in a server enviroment where the last thing you want is accidently overwritting your httpd.conf, and have some kind of central-point system config tool, and a more sane enviroment. Something like this may even be doable in bash in a modular fashion. ebuilds would be responsible of maintaining their own config pharser module.

This of course could be optional in make.conf, and/or in the portage ebuild.
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Sun Jan 22, 2006 8:50 am    Post subject: Reply with quote

AidanJT wrote:
[some extracts by SirYes]
  • create /etc/portage/etc-configs/*.xml where the xml file contains user defined config settings for each program
  • a syntax table for understanding how a config file(s) is put together and automatically detect depreciated or invalid settings
  • after a change to a config option in the xml, appropiate config file in /etc is updated
  • when a program is removed from portage, the config(s) in /etc could automatically be removed
  • the users settings in the /etc/portage/etc-configs/*.xml could be kept instead (should they wish to remerge the program at a later date)
  • ebuilds would be responsible of maintaining their own config parser module (???)
  • all the above should be optional in make.conf
  • and/or in the portage ebuild (???)

I think something like this would be very handy in a server enviroment where the last thing you want is accidently overwritting your httpd.conf, and have some kind of central-point system config tool, and a more sane enviroment. Something like this may even be doable in bash in a modular fashion.


Now this becomes interesting!

There are some points, however, that should be explained more. Could you please do so?
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Sun Jan 22, 2006 10:33 am    Post subject: Reply with quote

Quote:
a syntax table for understanding how a config file(s) is put together and automatically detect depreciated or invalid settings

How would you automatically detect if something's depreciated?
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Sun Jan 22, 2006 7:40 pm    Post subject: Reply with quote

Erlend wrote:
Quote:
a syntax table for understanding how a config file(s) is put together and automatically detect depreciated or invalid settings

How would you automatically detect if something's depreciated?


Have a list of valid options in the package syntax table, so when it comes to pharsing the *.xml file after an update, if an invalid option is detect, the script would point out the fact to the user.
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Sun Jan 22, 2006 7:53 pm    Post subject: Reply with quote

SirYes wrote:
Now this becomes interesting!

There are some points, however, that should be explained more. Could you please do so?


I assume you wish me to elabourate on the points you have (???). so here it goes

  • ebuilds would be responsible of maintaining their own config parser module (???)

Well the maintainer of apache2 for e.g. would be responsible for their own apache2.mod config pharser, as such when a new software release is added to portage stable and updated on a system, then the config module would automatically pharse /etc/portage/etc-configs/apache2.xml to detect options that may have been deprechiated.

  • and/or in the portage ebuild (???)

well we could have a global USE variable such as "autoconfig", if it's enabled in the portage package, then the software and layout that drives the system would automatically put in place. If a package supporting the system (again say apache2 for e.g.), then the default configs are omited, instead installing the apache2 module config parser and getting the user to configure the software themselves via the system, an apache2.xml template with some typical defaults could be used instead.

I hope this clarifies things somewhat.
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Mon Jan 23, 2006 9:40 am    Post subject: Reply with quote

Quote:
Have a list of valid options in the package syntax table, so when it comes to pharsing the *.xml file after an update, if an invalid option is detect, the script would point out the fact to the user.

Some files, like xorg.conf, have custom options introduced by drivers. In general though, would an unrecognised option do any harm?

Thanks,

Erlend
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Mon Jan 23, 2006 11:06 am    Post subject: Reply with quote

Erlend wrote:
Some files, like xorg.conf, have custom options introduced by drivers. In general though, would an unrecognised option do any harm?

Good point. In such case the mentioned syntax description file could relax the warning requirement - to a different level of control. For example (from the most strict control to the least strict):
  • unknown_options="warn-always"
  • unknown_options="allow-selected" allowed_sections="Device"
  • unknown_options="warn-selected" warn_sections="Files,Modules"
  • unknown_options="warn-never"

And it might use subkeys instead of attributes too.
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Mon Jan 23, 2006 11:21 am    Post subject: Reply with quote

Also, finding out all the configuration options for a file sounds like a lot of work... portage already deals with dependencies, because the developers' sites usually tell you the requirements that isn't too difficult... but it's rare to see an exhaustive list of configuration options for a file. Currently ebuilds are easy to write, I think this would complicate them (look at the number of unofficial ebuilds available in this forum).

You could achieve a similar effect by the following:

/etc/example-conf-file:
Code:

ok=green
warning=yellow
error=red
special=brown


/etc/configs/example-conf-file:
Code:

ok=yellow
warning=blue
special=purple


Later the /etc/example-conf-file is updated to:
Code:

good=green
ok=black
warning=yellow
error=red


This is a problem (albeit a relatively small one, could be worse). Warnings and Oks are both yellow. The problem would be detectable because the developer's config file changes the colour for ok and warning, which are both in the user's override file. The system would have to show this particular example to the user. So if an option the user has set is changed in the original file the user should be (optionally) alerted in a similar way to etc-update: except it will involve less information (single lines, not whole file) and happen less often. special has been removed from the config file, but it doesn't really matter that it's still in the user's config: application should ignore it.
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Wed Jan 25, 2006 5:09 am    Post subject: Reply with quote

It wouldn't nessecerily have to be manditory to support the feature for a package to be added to the offical tree, at least it is there for any maintainers who wish to use it.. the main application would be for server deamons and core system configs where config integridy is absolute paramount for mission-critical servers.
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Wed Jan 25, 2006 9:13 am    Post subject: Reply with quote

Can you explain a bit more about how your parser would actually work please?

Thanks,

Erlend
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Wed Jan 25, 2006 11:17 pm    Post subject: Reply with quote

Ok, I'll throw in a few quick code snippits, bear in mind i actually havn't put 'pen to paper' this is all off the top of my head. lets assume a call to etc-config with $1 = xml-update and $2 = apache2.. where $1 = update would act like the old update style, and $1 = update + $2 = package name would only use the old method on a specific package.

Code:


# /var/portage/etc-config/pharsers
apache2
mysql
php
xorg-x11
etc...
# end of /var/portage/etc-config/pharsers

etc-config {edit,verify,update,xml-update} {apache2,mysql,etc.. whichever packagename}

# /usr/sbin/etc-config main code

for pharser in /var/portage/etc-config/pharsers
do
if [ $pharser = $1 ]; then
source /usr/lib/etc-config/$pharser
fi

# end of etc-config main code

# /usr/lib/etc-config/apache2 pharser
edit_xml() {
{do editing ui code} (might be a way to do this in a generic fashion, again this is just totally rough thoughts.)
}

verify_xml() {
{pharse xml file and check for invalid/deprechated config entries}
return true if valid
return false if invalid
}

update_configs() {
read xml
loop through the options outputing corect format to whatever config files apropiate.
}

case "$action" in
  "edit") edit_xml;;
  "verify") verify_xml;;
  "update") if verify_xml; then
    update_configs
    else; fi;;
esac
# end of apache2 pharser


Again, this is all completely roughshot ideas right off the top of my head, nothing concrete or even remotely tested ideas.
Back to top
View user's profile Send private message
Erlend
Guru
Guru


Joined: 26 Dec 2004
Posts: 493

PostPosted: Fri Jan 27, 2006 10:31 am    Post subject: Reply with quote

Code:
# /usr/lib/etc-config/apache2 pharser
edit_xml() {
{do editing ui code} (might be a way to do this in a generic fashion, again this is just totally rough thoughts.)
}

How would you do this bit? Bearing in mind you might have:
Code:

=== Section 1: Keyboard ===
Driver = kbd
=== Section 2: Mouse ===
Driver = mouse

and you want to safely update this.

Quote:
Something like this is already coming. In the new (~arch) baselayout there is a Perl-based (think: Debian's dpkg-reconfigure) mechanism for managing Gentoo configuration. It's called eselect. 8) A possible time-frame: Gentoo 2006.0

I've looked at eselect, but I don't see anything about being able to keep configs after update?

Erlend
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Sat Jan 28, 2006 6:16 pm    Post subject: Reply with quote

Erlend wrote:
Quote:
Something like this is already coming. In the new (~arch) baselayout there is a Perl-based (think: Debian's dpkg-reconfigure) mechanism for managing Gentoo configuration. It's called eselect. 8) A possible time-frame: Gentoo 2006.0

I've looked at eselect, but I don't see anything about being able to keep configs after update?

Erlend


Well, all what I was saying can (and should) be understood as "eselect ~= dpkg-reconfigure". No more, no less. ;)
_________________
My blog: In search for ultimate programming language


Last edited by SirYes on Sat Jan 28, 2006 6:31 pm; edited 1 time in total
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1118
Location: Rep. of Ireland

PostPosted: Sat Jan 28, 2006 6:27 pm    Post subject: Reply with quote

Erlend wrote:
How would you do this bit? Bearing in mind you might have:
Code:

=== Section 1: Keyboard ===
Driver = kbd
=== Section 2: Mouse ===
Driver = mouse

and you want to safely update this.

I wouldn't imagine there would be a way to do it completely generic, partially perhaps.
Most of it would be implimentation specific to xorg-x11, the parser for xorg-x11 would take care of this, its not terribly difficult, afterall the xorg developers have a method for validating this, why wouldn't an x11 parser be able to do so as well?

edit: no harm done.


Last edited by aidanjt on Sat Jan 28, 2006 6:35 pm; edited 1 time in total
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Sat Jan 28, 2006 6:31 pm    Post subject: Reply with quote

AidanJT wrote:
Why would I call it a pharser? because I paid no attention to spelling at all? because people know what I mean anyway? I don't see how making a big deal about a spelling error is a good contribution to etc-update alternitives.

Easy... No harm intended or done (I hope).
I've edited my post... Maybe you should do that to? :mrgreen:
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
derelict
n00b
n00b


Joined: 18 Mar 2006
Posts: 2
Location: Switzerland

PostPosted: Sat Mar 18, 2006 5:25 pm    Post subject: Reply with quote

Common Configuration Parser might be of interest for a "different" etc-update approach.

Cheers

Marcel
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum