| View previous topic :: View next topic |
| Author |
Message |
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
 Posted: Sat Sep 05, 2009 1:53 pm Post subject: [funtoo / git] cleaning of .git |
 |
|
I noticed the free space on my partition holding portage is getting smaller and smaller and after some investigation, I found out it's related to the .git-directory inside. This dir now has a whooping weigth of 310MB (and counting). I know that git stores some - or better, a whole lot - of metadata in there, most of which I'll probably never use, so is there a safe way to do some spring-cleaning in there?
thanks,
ph
PS: Before you ask, diskspace does matter on this box and mounting portage via i.e. nfs is no option here. |
|
| Back to top |
|
 |
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Sat Sep 05, 2009 2:11 pm Post subject: |
|
|
The only way I was able to "spring clean" funtoo git portage is unmounting and reformatting the partition, then untar and git checkout a fresh snapshot.
Otherwise I have found that the funtoo portage is quite sensitive to "fiddling".
The funtoo FAQ "Forking the Portage Tree" may provide some clues though.  |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Sat Sep 05, 2009 2:26 pm Post subject: |
|
|
Erm, yay, that's surely a way, but... one major problem with this is, that the filesystem resides on a cf-card(pda, so no better way) and this would pretty surely trash the disk in "no time".
Thinking about it, how big must .git on kernel.org be, there the people are checking-in/-out 24/7 and I'm only doing it ~twice a weak and for ~3 months now.
Edit, normal portage is no option, at least not as long as it takes ages to bring some stuff from the overlays into portage and I don't want to manage dozens of overlays myself. |
|
| Back to top |
|
|
Naib
Watchman
Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy
|
| Posted: Sat Sep 05, 2009 2:33 pm Post subject: |
|
|
try:
git clean -X && git gc
it should clean it up a bit
_________________
| Quote: | | Removed by Chiitoo |
|
|
| Back to top |
|
|
Earthwings
Bodhisattva
Joined: 14 Apr 2003
Posts: 7753
Location: Germany
|
| Posted: Sat Sep 05, 2009 3:26 pm Post subject: |
|
|
Moved from Other Things Gentoo to Unsupported Software.
_________________
KDE |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Sat Sep 05, 2009 5:31 pm Post subject: |
|
|
| Mh, is there a way to generate the needed temporary data in another place? I've only got ~150MB left on this partition and that's not enough. |
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Sat Sep 05, 2009 6:13 pm Post subject: |
|
|
git-gc looks promising. The manpage describes several options. Let me know how that goes (if you try it).
git-fsck might also help.
You could symlink /usr/portage/distfiles to some other place. |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Sat Sep 05, 2009 6:57 pm Post subject: |
|
|
| Now out to the airport, trying it later. But symlinking distfiles, mh, won't help, nothing in there and git's tmpfiles seem to go to .git/objects/... as default. |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Sat Sep 05, 2009 9:42 pm Post subject: |
|
|
Tried `git gc --aggressive` which reduced the size to ~240MB, imho still pretty heavy. Someone using funtoo can tell me, what their `du -h $PORTDIR/.git` gives?
Edit, did an `cd $portage/.git ; rm -rf objects refs; emerge --sync` and now I'm at 13MB, will see how this one grows in the next few days or if I suffered from a bug before. |
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Wed Sep 09, 2009 11:43 pm Post subject: |
|
|
Wow 268M and that's after running git-gc.
| Code: | # du -h .git
3.0K .git/info
19K .git/logs/refs/remotes/origin
20K .git/logs/refs/remotes
13K .git/logs/refs/heads
34K .git/logs/refs
47K .git/logs
1.0K .git/refs/tags
2.0K .git/refs/remotes/origin
3.0K .git/refs/remotes
1.0K .git/refs/heads
6.0K .git/refs
19K .git/hooks
2.0K .git/objects/info
255M .git/objects/pack
255M .git/objects
268M .git |
Looks like it's the objects/pack/pack-nnn.pack file that's taking 95% of the diskspace. |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Thu Sep 10, 2009 12:00 am Post subject: |
|
|
Yep, annoying. Just setup a new system, /portage is on reiser4, complete /portage is 493MB and .git is 272MB - still looking for good solutions  |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Thu Sep 10, 2009 10:54 am Post subject: |
|
|
| Quote: | | Mh, is there a way to generate the needed temporary data in another place? |
there is a way to mount two dirs via aufs/unionfs so that writes are redirected to one directory, and reads are taken from union of both.
this way you could keep ebuilds in one filesystem and the git metadata on another. and set up some kind of rsync job to move ebuilds and assorted files to first filesystem and keep the git data on another one.
it's just a rough idea, but it definitely is possible. it's a variation on keeping your portage on squashfs with writable /dev/shm on top of it (portage is really small this way - nowadays ~48mb)
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
Naib
Watchman
Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy
|
| Posted: Thu Sep 10, 2009 11:47 am Post subject: |
|
|
| Code: |
git repack -a -d -l |
might be worth a shot
_________________
| Quote: | | Removed by Chiitoo |
|
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Thu Sep 10, 2009 12:43 pm Post subject: |
|
|
| Quote: | | might be worth a shot |
Yay, another 2MB won. :/
yoshi314, thanks, I'm considering this, weekend's coming. |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Wed Sep 23, 2009 4:30 pm Post subject: |
|
|
Woha, just cloned the zen-sources... | Code: | du -hs /usr/src/zen -> 916M
du -hs /usr/src/zen/.git -> 410M |
even worse, this is on Reiser4, again.
Edit, after `git clean -Xf; git repack -a -d -l; git gc --agressive`, which took about one hour total, .git is down to 355M - still sucks. |
|
| Back to top |
|
|
Naib
Watchman
Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy
|
| Posted: Wed Sep 23, 2009 7:26 pm Post subject: |
|
|
| ph030 wrote: | Woha, just cloned the zen-sources... | Code: | du -hs /usr/src/zen -> 916M
du -hs /usr/src/zen/.git -> 410M |
even worse, this is on Reiser4, again.
Edit, after `git clean -Xf; git repack -a -d -l; git gc --agressive`, which took about one hour total, .git is down to 355M - still sucks. |
well the zen-sources are taken straight from the main linux src, check the history
| Code: |
git log $(git rev-list HEAD | tail -n1)
commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2
Author: Linus Torvalds <torvalds@ppc970.osdl.org>
Date: Sat Apr 16 15:20:36 2005 -0700
Linux-2.6.12-rc2
Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.
Let it rip!
|
4years worth of commits,branches,merges,rebases, patches.
do a gitk on the folder (althought gitg is better  and see how much is actually stored
I do see the merit that downstream users probably don't have a use for all this but hey \o/
_________________
| Quote: | | Removed by Chiitoo |
|
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Wed Sep 23, 2009 11:21 pm Post subject: |
|
|
Mh, reading the `git help clone` I found the --depth X option, setting X=1 brings .git down to 134M - way better, yet not percect.
edit, the same "trick" brought down the funtoo-tree from 505M(incl. .git 283M) to 314M(incl. .git 93M).
Now trying what git-gc and repack can do. |
|
| Back to top |
|
|
tcx
n00b
Joined: 23 Mar 2005
Posts: 32
Location: Porto, Portugal
|
| Posted: Tue Oct 06, 2009 2:39 pm Post subject: |
|
|
I was totally unaware of that.
Apparently I haven't been checking my portage tree with enough care.
Has anyone found a better solution? |
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Fri Oct 09, 2009 3:03 am Post subject: |
|
|
| ph030, is git clone --depth 1 affecting performance ? |
|
| Back to top |
|
|
avx
Advocate
Joined: 21 Jun 2004
Posts: 2152
|
| Posted: Fri Oct 09, 2009 3:55 pm Post subject: |
|
|
Good question, haven't `time`d that. Doing it right now with zen-stable, stay tuned.
Edit, didn't use `time`, instead used | Code: | | date > file.txt ; git clone git://zen-sources.org/kernel/zen-stable.git zen && sync && date >> file.txt |
To be sure, I rebooted between the tries.
timediff normal run: ~17 minutes
timediff depth 1 run: ~4.5 minutes
Specs: CPU 1.2GHz Pentium M, 4.200rpm HDD, max. download capacity 1.4MB/s (execpt the checkout it's been idle), filesystem is reiser4 (not the latest version with the fast-sync patches). If needed, request `time`d results. |
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Mon Oct 12, 2009 10:57 pm Post subject: |
|
|
| That is nice improvement if a truncated history is sufficient. Have you tried it with git-portage? |
|
| Back to top |
|
|
|
Unsupported Software
