Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Migrating portage tree to git
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
hasufell
Developer
Developer


Joined: 29 Oct 2011
Posts: 247

PostPosted: Mon Sep 01, 2014 12:09 pm    Post subject: Reply with quote

desultory wrote:
or take up the work yourself

I can. I am running several git servers myself. I'd dump the whole old cvs history and start with a new git repo, set up a gitlab platform or even phabricator instead of this broken bugzilla stuff.

So, start by giving me root access or show me a real reason why the migration (of what exactly?) takes years instead of linking me to mailing lists that are either dead or have no significant information.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2374
Location: UK

PostPosted: Mon Sep 01, 2014 1:17 pm    Post subject: Reply with quote

Maybe if Gentoo could get anything done, the distro itself wouldn't have been forked twice...
Back to top
View user's profile Send private message
hasufell
Developer
Developer


Joined: 29 Oct 2011
Posts: 247

PostPosted: Mon Sep 01, 2014 1:26 pm    Post subject: Reply with quote

Ant P. wrote:
Maybe if Gentoo could get anything done, the distro itself wouldn't have been forked twice...

haha, if it was only twice... but good point
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Mon Sep 01, 2014 1:39 pm    Post subject: Reply with quote

hasufell wrote:
desultory wrote:
or take up the work yourself

I can. I am running several git servers myself. I'd dump the whole old cvs history and start with a new git repo, set up a gitlab platform or even phabricator instead of this broken bugzilla stuff.

So, start by giving me root access or show me a real reason why the migration (of what exactly?) takes years instead of linking me to mailing lists that are either dead or have no significant information.


Wow! A gentoo developer with an actually working brain. So rare these days. No sarcasm here, srsly.
Back to top
View user's profile Send private message
onodera-punpun
n00b
n00b


Joined: 10 Jul 2014
Posts: 2

PostPosted: Mon Sep 01, 2014 4:22 pm    Post subject: Reply with quote

a3li wrote:
I think github would be rather angry if we piggybacked all of our syncing on them. Oh, and we have plenty of rsync mirrors all over the world.


This makes it sound like you don't even know what git is, like you don't know the difference between git and github and haven't even looked into to subject, kinda worries me coming from a developer.
Back to top
View user's profile Send private message
a3li
Developer
Developer


Joined: 02 Sep 2008
Posts: 109
Location: 德国

PostPosted: Mon Sep 01, 2014 5:13 pm    Post subject: Reply with quote

onodera-punpun wrote:
a3li wrote:
I think github would be rather angry if we piggybacked all of our syncing on them. Oh, and we have plenty of rsync mirrors all over the world.


This makes it sound like you don't even know what git is, like you don't know the difference between git and github and haven't even looked into to subject, kinda worries me coming from a developer.


I was referring to the "funtoo experience" mentioned in the quoted post, which to the best of my knowledge "relies" on github for all of the syncing. That certainly won't work for Gentoo, and neither would convincing 100+ mirror sponsors to start offering git.
_________________
I am Confuism. Do not bother me.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2374
Location: UK

PostPosted: Mon Sep 01, 2014 6:11 pm    Post subject: Reply with quote

The nice thing about git is that you can use vanilla rsync to update it.
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Mon Sep 01, 2014 8:24 pm    Post subject: Reply with quote

a3li wrote:
onodera-punpun wrote:
a3li wrote:
I think github would be rather angry if we piggybacked all of our syncing on them. Oh, and we have plenty of rsync mirrors all over the world.


This makes it sound like you don't even know what git is, like you don't know the difference between git and github and haven't even looked into to subject, kinda worries me coming from a developer.


I was referring to the "funtoo experience" mentioned in the quoted post, which to the best of my knowledge "relies" on github for all of the syncing. That certainly won't work for Gentoo, and neither would convincing 100+ mirror sponsors to start offering git.


Git works over http, convincing anybody won't be much of a problem.
Back to top
View user's profile Send private message
a3li
Developer
Developer


Joined: 02 Sep 2008
Posts: 109
Location: 德国

PostPosted: Mon Sep 01, 2014 9:18 pm    Post subject: Reply with quote

Heh, keep up the enthusiasm. But, no point in convincing me, I don't care/decide on this. The people who do are on the list that was already mentioned, try posting there. I'm sure they'll be glad to finally have someone tell them what wonderful things git can do.
_________________
I am Confuism. Do not bother me.
Back to top
View user's profile Send private message
onodera-punpun
n00b
n00b


Joined: 10 Jul 2014
Posts: 2

PostPosted: Tue Sep 02, 2014 11:02 am    Post subject: Reply with quote

a3li wrote:
onodera-punpun wrote:
a3li wrote:
I think github would be rather angry if we piggybacked all of our syncing on them. Oh, and we have plenty of rsync mirrors all over the world.


This makes it sound like you don't even know what git is, like you don't know the difference between git and github and haven't even looked into to subject, kinda worries me coming from a developer.


I was referring to the "funtoo experience" mentioned in the quoted post, which to the best of my knowledge "relies" on github for all of the syncing. That certainly won't work for Gentoo, and neither would convincing 100+ mirror sponsors to start offering git.


Ah sorry I missed that I think.
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Tue Sep 02, 2014 1:25 pm    Post subject: Reply with quote

Ok, I've created a public repo, and will try to keep it updated daily for the time being. For those who wanna test-drive it, read the following mini how-to.

1. Back up your old portage tree: mv /usr/portage /usr/portage_old
2. Make sure you have enough space. A shallow clone occupies about 1.5G including 126M in .git directory.
3. git clone --depth 1 https://gitorious.org/gentoo-x86/gentoo-x86.git /usr/portage
4. Run 'egencache -j4 --repo=gentoo --update'. When done the first time, it takes several minutes. Drop the following script to /etc/portage/postsync.d to avoid bothering with this after each sync.
Code:
#!/bin/sh
if [ "$1" = "<INSERT REPO URL HERE>" ]; then
    [ -x /usr/bin/egencache ] && /usr/bin/egencache --repo=gentoo --update
fi
Don't forget to 'chmod +x' it. You should also make changes to /etc/portage/bin/post_sync. It should look like this:
Code:
> cat /etc/portage/bin/post_sync
#!/bin/sh
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

for f in /etc/portage/postsync.d/* ; do
   [ -x "${f}" ] && "${f}" "$@"
done
:
Many thanks to Zac Medico for a hint.
5. Update your gentoo.conf. It should look like this:
Code:
> cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
#sync-type = rsync
#sync-uri = rsync://rsync.gentoo.org/gentoo-portage
sync-type = git
sync-uri = https://gitorious.org/gentoo-x86/gentoo-x86.git

6. ???
7. PROFIT!!

I'm looking forward to any comments or suggestions.
o/

IMPORTANT NOTE:
You must have git version 2.0 or later installed. Version 2.1 would be even better. Otherwise you may have some trouble with updating a shallow clone.

UPDATE #1: Updates postsync script.
UPDATE #2: Another mirror is up and running. URL: git://git.schorsch-tech.de/gentoo-x86/gentoo-x86.git. Thanks to schorsch_76.


Last edited by apathetic on Sat Sep 06, 2014 4:16 pm; edited 3 times in total
Back to top
View user's profile Send private message
schorsch_76
Apprentice
Apprentice


Joined: 19 Jun 2012
Posts: 210

PostPosted: Tue Sep 02, 2014 7:48 pm    Post subject: Reply with quote

Currently i have trouble cloning ....

Code:
error: RPC failed; result=22, HTTP code = 504
fatal: The remote end hung up unexpectedly


Is this server having serious load right now?
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Tue Sep 02, 2014 8:06 pm    Post subject: Reply with quote

schorsch_76 wrote:
Currently i have trouble cloning ....

Code:
error: RPC failed; result=22, HTTP code = 504
fatal: The remote end hung up unexpectedly


Is this server having serious load right now?


Works for me at the moment. AFAIK gitorious sometimes has troubles with IPv6. Try disabling it.
Back to top
View user's profile Send private message
schorsch_76
Apprentice
Apprentice


Joined: 19 Jun 2012
Posts: 210

PostPosted: Tue Sep 02, 2014 8:20 pm    Post subject: Reply with quote

@apathetic: I have an 24 hours online server with 1 GBit uplink. I can offer to sync with you with an cron job. so we can have two servers for the portage tree.
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Tue Sep 02, 2014 8:35 pm    Post subject: Reply with quote

schorsch_76 wrote:
@apathetic: I have an 24 hours online server with 1 GBit uplink. I can offer to sync with you with an cron job. so we can have two servers for the portage tree.


Replied via PM.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4153

PostPosted: Tue Sep 02, 2014 11:10 pm    Post subject: Reply with quote

apathetic wrote:
2. Make sure you have enough space. A shallow clone occupies about 1.5G including 126M in .git directory.

How can this be a shallow clone? This is 3-4 times the space that a "normal" resp. "compressed" portage tree takes.
Just for comparison: With squashfs the total disk requirement is about 50M.

Actually, this unacceptable disk requirement was my main apprehension concering the planned change to git from the very beginning.
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Tue Sep 02, 2014 11:29 pm    Post subject: Reply with quote

mv wrote:
How can this be a shallow clone? This is 3-4 times the space that a "normal" resp. "compressed" portage tree takes.
Just for comparison: With squashfs the total disk requirement is about 50M.


You have probably probably misunderstood me. 1.5G is for the whole unpacked tree. Are you sure squashfs packs that into 50M?
Back to top
View user's profile Send private message
desultory
Administrator
Administrator


Joined: 04 Nov 2005
Posts: 7987

PostPosted: Wed Sep 03, 2014 3:58 am    Post subject: Reply with quote

apathetic wrote:
Yeah, right, and monty python.

http://article.gmane.org/gmane.linux.gentoo.scm-migration/184

It looks dead, Jim.
Try one message later, in that same thread: http://article.gmane.org/gmane.linux.gentoo.scm-migration/185. What it lacks in oft repeated references to British television, it makes up in references to oft ignored actual data.

hasufell wrote:
I can. I am running several git servers myself.
In that case, why not offer your services to the infrastructure team?
hasufell wrote:
I'd dump the whole old cvs history and start with a new git repo, set up a gitlab platform or even phabricator instead of this broken bugzilla stuff.
Just to be clear, your radical new plan for migrating the repository to git is what the infrastructure team has repeatedly stated was their plan? Aside from the gitlab or phabricator instances, that is.
hasufell wrote:
So, start by giving me root access or show me a real reason why the migration (of what exactly?) takes years instead of linking me to mailing lists that are either dead or have no significant information.
Fun fact: I am not a member of the infrastructure team, as such I do not have the authority to grant your request. Bonus fact: given your comments here and elsewhere, I would not grant your request if I had such authority. Furthermore, while I am willing to direct you to information, I will not spoon feed it to you.

apathetic wrote:
Replied via PM.
Just a general point: posting that you have sent a private message is singularly pointless, the user that it was sent to would know anyway while the rest of the readers get no useful information from it.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4153

PostPosted: Wed Sep 03, 2014 7:50 am    Post subject: Reply with quote

apathetic wrote:
You have probably probably misunderstood me. 1.5G is for the whole unpacked tree.

So you probably calculated with a giant inode size. The whole unpacked tree (without any git) with du is shown here as 354M. Apparently, du assumes an inode size of 512 for squashfs which is by far too much, of course: One of the nice things about squashfs is that the unpacked data requires no space at all (and thus no inodes).

What worries me more is the size of the .git repository you mentioned, because this is compressed data and cannot be packed away with squashfs: I would have expected that this should be almost empty for a shallow clone since I expected that essentially it contains only the filenames and their checksums (together with a small overhead), but it is about 2-3 time as large as the current full squashfs file. So even when combining git with squashfs, the space requirement would increase by the factor 4, roughly. I cannot afford this on some of my machines.
Quote:
Are you sure squashfs packs that into 50M?

Yes, roughly. For several years I had a slightly stripped-down version with 43M (with -comp xz), currently my sfs file contains not only the full tree but also some local and public layman overlays including their full git history, and its size is about 77M. So I guess that the full portage tree alone takes sometihng in between.
Even in the worst case of 77M this is only about half of the size of the .git directory (and its usage does not require any unpackaged data...)
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Wed Sep 03, 2014 10:00 am    Post subject: Reply with quote

mv wrote:
So you probably calculated with a giant inode size. The whole unpacked tree (without any git) with du is shown here as 354M. Apparently, du assumes an inode size of 512 for squashfs which is by far too much, of course:

Yeah, thats probably the issue.
mv wrote:
What worries me more is the size of the .git repository you mentioned, because this is compressed data and cannot be packed away with squashfs: I would have expected that this should be almost empty for a shallow clone since I expected that essentially it contains only the filenames and their checksums (together with a small overhead)

No, it actually contains the whole tree compressed. The term shallow in git means that the clone does not have the full commit history.
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Wed Sep 03, 2014 10:43 am    Post subject: Reply with quote

desultory wrote:
apathetic wrote:
Replied via PM.
Just a general point: posting that you have sent a private message is singularly pointless, the user that it was sent to would know anyway while the rest of the readers get no useful information from it.

Just a general point: replying to a thread with the subject you're not generally interested in is singularly pointless.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2374
Location: UK

PostPosted: Wed Sep 03, 2014 1:34 pm    Post subject: Reply with quote

I can't understand why everyone seems averse to working with someone with such a fragrant personality...
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Wed Sep 03, 2014 2:38 pm    Post subject: Reply with quote

Ant P. wrote:
I can't understand why everyone seems averse to working with someone with such a fragrant personality...

Can this be a thread about using git to manage the portage tree and not about somebody's personal issues? Please? Pretty please?
Back to top
View user's profile Send private message
desultory
Administrator
Administrator


Joined: 04 Nov 2005
Posts: 7987

PostPosted: Thu Sep 04, 2014 3:48 am    Post subject: Reply with quote

apathetic wrote:
Just a general point: replying to a thread with the subject you're not generally interested in is singularly pointless.

apathetic wrote:
Can this be a thread about using git to manage the portage tree and not about somebody's personal issues? Please? Pretty please?
One of these quotes is undermined by the other. Still, we can try, can we not?

Have you found the e-mail address issue that steveL alluded to earlier?
Back to top
View user's profile Send private message
apathetic
n00b
n00b


Joined: 28 Aug 2014
Posts: 22

PostPosted: Thu Sep 04, 2014 9:47 am    Post subject: Reply with quote

desultory wrote:
Have you found the e-mail address issue that steveL alluded to earlier?


No. But I'm using different tools than Patrick.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum