Versioning /etc with subversion is safe?

[Azangod]

Hi there,
the scenario is a production server where there is no tollerance for errors, where every etc-update session is pure fear.

Merging config files sometimes becomes tricky with the endless question "keep the old version ? better grab the new version and apply the personalizations or keep the old in the risk of a new-born option default value would do something nasty?"

My answer is: why not to use a cvs?
I know subversion wich works flawlessy for my work but I'm no guru and I'm wondering if there may be relevant security issues in this approach. well... first of all I don't know how to save permissions and ownerships of files stored.

So, before trying, I'm wondering (and asking) if there are serious drawbacks into doing this.

[OneOfOne]

I personally use git, i have this git alias

Code: Select all

etc-update = !cd /etc && etc-update && git add -A && git ci 

my /etc/.git/info/exclude

Code: Select all

! /X11/xorg.conf                                                                                                                              
! php.ini                                                                                                                                     
*-                                                                                                                                            
*.[oa]                                                                                                                                        
*.default                                                                                                                                     
*.env                                                                                                                                         
*.example                                                                                                                                     
*.lock                                                                                                                                        
*0                                                                                                                                            
*compiz*                                                                                                                                      
*csh*                                                                                                                                         
*~                                                                                                                                            
._*                                                                                                                                           
/X11                                                                                                                                          
/acpi                                                                                                                                         
/adjtime                                                                                                                                      
/bash_completion.d                                                                                                                            
/config-archive                                                                                                                               
/cups/ppd                                                                                                                                     
/eselect                                                                                                                                      
/fonts
/gconf
/gre.d
/group
/gshadow
/gtk-2.0
/hotplug
/ld.*
/login.*
/make.*
/mono
/mtab
/ndiswrapper
/opt
/pango
/php
/protocols
/qt4
/reader.conf.d
/revdep-rebuild
/rmt
/runlevels
/skel
/splash
/ssl
/t1lib
/terminfo
/xdg
/xinet.d
ca-certificates.conf
modprobe.conf
modprobe.d
modules.conf
timezone

hope that helps.

[Azangod]

Thanks, I've heard of many admins using git, but nothing about subversion.
I'm wondering why.
It's because GIT is old & well know or there are some other issues about subversion. Or simply subversion does'nt fit for this particular task?

[depontius]

I believe subversion is older than GIT. I've been considering moving /etc into subversion, but the details of the process and job, along with my lacking understanding of subversion have held me back. At home I have 2 servers and 3 clients. I think I'd like a tree structure using branches, with one root, split into servers and clients, the each of those split into per-system. But I've never really worked like that with subversion - at work I use tags to mark release points, and those tags don't seem to "inherit" changes from the trunk. My impression is that branches are like tags, with the similar inheritance issue, and at that point it alls seems to start looking like enough extra work with enough pitfalls that I never get around to it.

I've learned enough about subversion to make it do what I need to do at work. I could probably stand to learn quite a bit more.

[Azangod]

I did'nt know that subversion is older than git. I've always believed that git was a milestone, the vcs choice of kenerl developers from the stone era.

Maybe i did'nt understood your use of subversion but seems to me that there is something odd.

at work I use tags to mark release points, and those tags don't seem to "inherit" changes from the trunk

I use subversion for software developing. As far as I know there are'nt (in subversion) any built-in concept like 'tags', 'branch' or trunk. For svn a dirctory is simply a directory and branches, tags or trunk makes no difference. These names are a mere naming-convention among developers.
Every time you publish a release (let say for example svn cp repos/project/trunk repos/project/tags/myreleas) you do nothing else but a mere copy of all files.

If you want to start a new branches to try to insert a new feature you will do something like
svn cp repos/project/trunk repos/project/branches/test

At this point who works on the trunk is unaware of the changes made in test and vice-versa. Lets assume that 10 is the number of the revision at the creation of the branch test and that at the revision number 30 the developing of the branch is done and so it's time to merge it into the trunk. At this point it's you who decide to merge, svn takes no action on it's own.
You coul merge test in trunk using something like:
svn merge repos/project/branches/test -r10:30
*typed on a working copy of the trunk*

Doing so svn will try to automatically merge and you'll end up with your working copy modified to accomodate any update applied on test between revisions 10 and 30. If you are lucky svn did all the work quetly, otherwise you have to manually resolve some conflict.

Uhm, I hope this whole soup is at least readable in my ridiculous english... and I hope it had be of any help too!

[depontius]

I did'nt know that subversion is older than git. I've always believed that git was a milestone, the vcs choice of kenerl developers from the stone era.

Git isn't that stone-aged. Some years ago there was a proprietary versioning system called bitkeeper. The owner of bitkeeper struck some sort of deal with the kernel developers, allowing the Linux kernel to use it for free, yet remaining proprietary. The kernel developers rather got to like it, especially compared to what they had before. (Don't remember what that was, if it was anything at all other than patches in Linus' inbox.) Then someone started reverse-engineering some sort of client access, and the bitkeeper people got annoyed. Linus developed git in a feverish Gordon Letwin HPFS weekend, and it's all history.

My version is probably somewhat incorrect, but wikipedia can be friendly, too: http://en.wikipedia.org/wiki/BitKeeper

As for subversion, what you say sounds like the impression I have, just better phrased. I've never used "svn merge", and could clearly stand to learn more.

[Azangod]

Nice story founded on wiki... another piece of code "once granted for free" that after some time becomes commercial-only.
At least I could say I've learned something new today.

It seems I've to take a good look at GIT for my admin issues.. one time or another... if I'm can find the time

About subversion I suggest this guide: http://svnbook.red-bean.com/
The merge process is quite simple when you have understood the logic that lies beneath. I've tryed svn two years ago and since then I've never thought of moving nor trying something else: it simply has everithing needed. At least for what I need it for.
Mybe one of the colest features is its capability of managing folders that permits to have many project as needed in a single repository! No need to manage a repos-per-project.

[depontius]

I use the redbook regularly - for what I need to do at work. Home admin is on a time-available basis, and I just don't have that much to spare.

[Azangod]

I know! Days of 48 hours wont help much in the long run... but sometimes I really would like to have more time

[V-Li]

A distributed VCS like Bazaar, Git, Mercurial or Darcs should be the better choice as you can transfer the whole history easily by just copying the directory.

[Azangod]

A distributed VCS like Bazaar, Git, Mercurial or Darcs should be the better choice as you can transfer the whole history easily by just copying the directory.

What do you mean with history?

With svn I can see the whole story of every file (or directory) since creation no matter what happens: moving, copying into other branches or projects.
Maybe you are speaking about interaction between different repos? I've never tryed that. I live very well having a single phisical repository with all my projects in.

I could say I'm satisfied. I can manage easilly documentation and binary (such images and videos) too.

By the way I've never tryed any other VCS. Surely I'll try GIT (because it seems the most commonn choice among sysadmins) for the only purpose of maintaining /etc directory of my servers. One day... for now I've more urgent matter to take care of

[V-Li]

Inside of /etc you have a .git or .bzr or .whatever directory which contains the whole (!) repository. So copying the /etc directory over to another machine preserves all the history without creating a Subversion repository there, importing and so on.

[Azangod]

Inside of /etc you have a .git or .bzr or .whatever directory which contains the whole (!) repository. So copying the /etc directory over to another machine preserves all the history without creating a Subversion repository there, importing and so on.

Whoa, it's surely handy but over time, it does'nt become too huge? I mean... if every directory brings along the whole history, after fifthy revision (on a photoshop source for example) it may become really really huge and space wasting

[depontius]

Inside of /etc you have a .git or .bzr or .whatever directory which contains the whole (!) repository. So copying the /etc directory over to another machine preserves all the history without creating a Subversion repository there, importing and so on.

Whoa, it's surely handy but over time, it does'nt become too huge? I mean... if every directory brings along the whole history, after fifthy revision (on a photoshop source for example) it may become really really huge and space wasting

Revision systems usually don't store whole files, they store diffs in one form or another. The size growth is much slower, assuming changes are really changes, not wholesale replacements. I'd expect the biggest such volumes to be certificates in /etc/ssl and I believe X tucks some binaries into /etc/X11.

[Azangod]

Revision systems usually don't store whole files, they store diffs in one form or another. The size growth is much slower, assuming changes are really changes, not wholesale replacements. I'd expect the biggest such volumes to be certificates in /etc/ssl and I believe X tucks some binaries into /etc/X11.

Yes I know. My concernings where about binary files managment. Images are considered binary. Not too much sense in storing diff-files. So I was wondering what will become in a directory if there are stored 100 revision of a 100mb photoshop file... it's a lot of space. Thaking along the whole repository in such a case is suicide. I'm sure that developers of these systems have found a moure suitable solution.

Btw i'm happy about svn. Got to try GIT to store my etc files and nothing more. One day I'll try.

[V-Li]

Git packs quite well, although binary files press a big problem here, but that is usually not the target for such a version control system.

[Massimo B.]

Just to point out my idea using mercurial for /etc: [topic=806432]Planning Mercurial for /etc[/topic] ...

[cwr]

I'd be inclined to use Mercurial, (or even Bazaar, which I haven't tried) rather than Subversion.
You're dealing with text files, which a DVCS is at home with, and they all handle merges better
than Subversion. Check the docs on http://mercurial.selenic.com before making a decision;
they give a good overview of VCS and DVCS advantages and disadvantages.

On the whole, avoid Git - it's a good idea for the kernel, but it has a major learning curve.
It took me five minutes to learn enough to start using Mercurial, with I suppose half a
dozen essential commands, and it really is a very powerful and convenient piece of
software.

Will

[Ant P.]

I've found git repositories with a working directory checkout to be smaller than the corresponding SVN checkout alone, even with huge (1.5GB) directory trees of binaries and 5 years of history. Also it doesn't pollute the entire FS tree with hidden files like SVN does.

[rh1]

On the whole, avoid Git - it's a good idea for the kernel, but it has a major learning curve.
It took me five minutes to learn enough to start using Mercurial, with I suppose half a
dozen essential commands, and it really is a very powerful and convenient piece of
software.

Not trying to start another mercurial vs git flame war but it took me 5 minutes to learn enough to start using git too.

[Massimo B.]

Continuing the discussion about mercurial here...
I had long time discussions with mercurial people at #mercurial channel. Summarily what they said is, Mercurial is not SVN and Mercurial is not the right tool.
This is mainly because of the requirements and methods I tried to explain:

I have different Linux machines each having more or less different configuration. Mainly I need to track /etc, but because sometimes I also add usr/share/kde/....kdmrc or even var/lib/portage/world I made / being a mercurial repo (beware of [bug=284673]Bug 284673[/bug]!). Each machine gets its own feature branch. I never merge feature branches. I think about having a seperate feature branch being a template branch with default configurations.
Now what is not the way of thinking in mercurial is what I often need: I would need to branch files directly. Say I have adapted etc/conf.d/local.start on hostA and next time on hostB I would need to merge parts of that file.
Mercurial people say, a revision is for the complete set of files. You can merge branches and revisions of branches, but never single files. Ok there are a lot of Mercurial Extensions, some of them could do the job:

• Convert extension and using a filemap
• Transplant extension

The Mercurial way of merging actually is: Go back to the revision the bug appears, fix it (making a unnamed or even named branch), comit and merge. For our case that would mean: If there is a version of a file that might be distributed among other branches this must be a pre-branch revision and you can branch from that. This model absolutely breaks with what I planned.
Does anybody has any suggestion about that?

Anyway, I'm already using Mercurial on this purpose for a while now. I need to merge a file? No problem, I do

Code: Select all

vimdiff etc/make.conf <(hg cat -r hostB etc/make.conf)

I install a fresh system and need a template to start with, I just do

Code: Select all

hg cat -r templates etc/make.conf > etc/make.conf

Btw. it actually makes no sense to switch to a different branch and update.
One disadvantage of this approach is that hg cat does not really merge. So as I know from other SCM like Perforce (not sure about SVN?) there is no coherent history of a file that is merged from an ancestor. So this approach does not use Mercurial the way it is made for, it just uses Mercurial as a datastore.

Pushing and pulling is done with a central server, using Bitbucket like mentioned in the other thread. Just make sure you are not pushing your wireless keys or stuff.
There is the side effekt, for some people a disadvantage, that all hosts know the complete repository and data of all other hosts. For me this is fine. If the host is offline (some of them are mobile PCs) I still have access to all other hosts configurations. If I'm online again, I can push or pull from the central repository. With this model I would never push or pull between hosts directly.

[Massimo B.]

I've found git repositories ........doesn't pollute the entire FS tree with hidden files like SVN does.

Mercurial neither. Please for further discussion talk about if any other SCM does better fit to the requirements than Mercurial does, like merging a single file with history.
All the other comparisons of hg vs. svn (and git, svk, p4, cvs etc.) are written in lots of other developer places...

[wcg]

Mercurial people say, a revision is for the complete set of files.

This is git's paradigm of a revision, too:
http://en.wikipedia.org/wiki/Git_%28software%29

(A numbered or named revision is not a revision of a file, but rather
a revision of the repository context that includes that file.)