Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Filesystem for /usr/portage
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
fangorn
Veteran
Veteran


Joined: 31 Jul 2004
Posts: 1886

PostPosted: Mon May 25, 2009 9:59 am    Post subject: Reply with quote

I prefer reiserfs (v3) for size reasons. (support vor 512 byte blocks)

Makes it small enough to place the whole filesystem in RAM if you need to. -> flying cows! :twisted: :twisted:
_________________
Video Encoding scripts collection | Project page
Back to top
View user's profile Send private message
gentoo-dev
Apprentice
Apprentice


Joined: 24 Jan 2006
Posts: 172

PostPosted: Mon May 25, 2009 10:38 am    Post subject: Reply with quote

BitJam wrote:
I've tried several different file systems for /usr/portage over the years. I'm currently using XFS with the smallest blocksize possible. I can typically sync in well under two minutes. If I sync regularly, the time goes down to about one minute.
And you're happy with that? I emerge --sync once in a while and it takes 20-30s. I use a local rsync mirror, so network speed is not an issue in my case.
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10303
Location: Córdoba (Spain)

PostPosted: Mon May 25, 2009 10:52 am    Post subject: Reply with quote

gentoo-dev wrote:
BitJam wrote:
I've tried several different file systems for /usr/portage over the years. I'm currently using XFS with the smallest blocksize possible. I can typically sync in well under two minutes. If I sync regularly, the time goes down to about one minute.
And you're happy with that? I emerge --sync once in a while and it takes 20-30s. I use a local rsync mirror, so network speed is not an issue in my case.


Which is irrelevant to the thread and is even more irrelevant since cron exists. It doesn't really matter if it takes 30s, 2mins or 1 hour, just add the job to cron.daily or whichever one you prefer. If there's a task that can run unattended, that's emerge --sync :wink:
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
djinnZ
Advocate
Advocate


Joined: 02 Nov 2006
Posts: 4831
Location: somewhere in L.O.S.

PostPosted: Mon May 25, 2009 1:32 pm    Post subject: Reply with quote

I agree, the problem with reiser 3 and portage is the performance decrease, but not for the sync operation, at least for the metadata rebuild (called at end of the sync).
And the fragmentation is not determined by the large number of small files (reiser 3 is optimized to host this), is determined by the precence of a large number of small files (<1kb) and a large number of "normal" (10->250 MB is not really a very big filesize) files.
Try to put the source tars and the binaries on another partition and you will not see any slowdown.
And not forget the filesystem than host the metadata cache.

@BitJam : Using a small blocksize will only spare space on disk, never increasing the performance. There is a reason for having a default blocksize of 4kb for every filesystem.
_________________
scita et risus abundant in ore stultorum sed etiam semper severi insani sunt:wink:
mala tempora currunt...mater stultorum semper pregna est :evil:
Murpy'sLaw:If anything can go wrong, it will - O'Toole's Corollary:Murphy was an optimist :wink:
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2478
Location: Silver City, NM

PostPosted: Mon May 25, 2009 5:53 pm    Post subject: Reply with quote

frostschutz wrote:
If you're using XFS you could just use the defrag tool that comes with XFS...

Thanks!

Although I wonder if that tool works as well as brute-force copying. When the portage partition slows down, I don't think the problem is *file* fragmentation since most of the files are smaller than a single block. I think the slowdown is caused when files within a directory become scattered on the disk. Assume each file is contained in one block. Then defragging files gains you nada. But what you do want is that the order of the blocks on the disk is the same as the order in which the files will be read. I believe the copying hack provides this ordering assuming the copying occurs in the same order that Portage uses for searching.

Of course, all of this is rather crazy because it is crazy to try to use any file system as if it were a database. IMO this is a fundamental architectural flaw in Portage. I understand the rationale for why it is done this way: the devs don't want Portage to depend on a database because that would defeat the bootstrap idea of Gentoo. They don't want the boards flooded with "I accidentally unmerged MySQL and now emerge won't work". So then the next suggestion is to have two systems. The current FS based system and also a DB based system that both deal with the same information. People would use the FS system to get started and then switch to the DB based system for better performance. I think there have been many attempts at this but they have never full succeeded because they are hampered by the fact that the current FS based system is "good enough" and no one really want to maintain two parallel systems. Therefore we're stuck with this craziness of pretending the FS is a database. There is a similar craziness in the init system that pretends Bash is a full-fledged high level language. But it's not so they have to keep playing tricks with increasing the size of the Bash environment which then temporarily breaks other programs (such as xargs). But, to paraphrase Kahlil Gibran:
Quote:
People who do not forgive Gentoo her little faults will never enjoy her great virtues.


Anyway, thanks for the tip regarding the XFS defrag tools.
Back to top
View user's profile Send private message
kdorf
n00b
n00b


Joined: 28 May 2009
Posts: 2

PostPosted: Fri May 29, 2009 3:30 pm    Post subject: Reply with quote

yngwin wrote:
Ext4 considered stable??? That's ridiculous. Reiser4 has been around much, much longer and is a lot more stable. So if you want stable and fast, Reiser4 is the way to go. But you're free to use whatever else. :wink:


http://en.wikipedia.org/wiki/Ext4

I just started using Ext4 myself though. I don't tend to jump on the bandwagon of "it's marked stable now, so let's use it!" I've always waited for a few bugs to be ironed out. =) Have to say though, using it on my laptop I seem to get really good performance, though I don't have a separate partition for /usr/portage. I should do that.
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10303
Location: Córdoba (Spain)

PostPosted: Fri May 29, 2009 3:53 pm    Post subject: Reply with quote

[offtopic]

kdorf wrote:
yngwin wrote:
Ext4 considered stable??? That's ridiculous. Reiser4 has been around much, much longer and is a lot more stable. So if you want stable and fast, Reiser4 is the way to go. But you're free to use whatever else. :wink:


http://en.wikipedia.org/wiki/Ext4

I just started using Ext4 myself though. I don't tend to jump on the bandwagon of "it's marked stable now, so let's use it!" I've always waited for a few bugs to be ironed out. =) Have to say though, using it on my laptop I seem to get really good performance, though I don't have a separate partition for /usr/portage. I should do that.


This really depends on who do you ask to.

The "it's been around much longer" argument from yngwin is a double blade weapon. Windows has also been around longer than Linux around, following the same logic it must be a a lot more stable. Maybe it's not the fairest example because to start with, Linux is not even a complete OS, but you get the point. Ext4 has come a very long way in a much shorter time than reiser4, and even if there are bugs (and nasty ones indeed) you must also look at the other side of the coin and agree with me that reiser4 has had equally nasty bugs for a much longer time.

I am not here to argue what's better, it's been long since I studied things like these and anyway I lack the understanding about the reiser4 internals to be able to argue technically about it. There's already a lot of that on the net. I'll just leave it at "it depends on the lenses you look through". Personally, I'd invest my energies in defending something like btrfs, because it's the way to go for the future. In my eyes, reiser4 has little present, and neither of reiser* and ext* are modern fs's oriented to the future, which in my humble opinion lies on btrfs and zfs. Those two are the fs's that extend the fs paradigm into this new century. All the rest are basically a remake of the same old film of a monster chasing teenagers: the blonde girl survives, the black guy dies first (just telling the stereotype, not that I like these movies at all).

[/offtopic]
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
JohnBlbec
Apprentice
Apprentice


Joined: 08 Feb 2003
Posts: 246

PostPosted: Sat May 30, 2009 4:03 pm    Post subject: Reply with quote

i used reiserfs, now i am using ext2 and i am really satisfied.

Code:

for /usr/portage ............. mke2fs -b 1024 -N 200000 /dev/xxx
for /usr/portage/distfiles ... mke2fs -b 4096 -T largefile /dev/yyy
Back to top
View user's profile Send private message
Captain Newbie
Apprentice
Apprentice


Joined: 22 Dec 2006
Posts: 182
Location: Socal

PostPosted: Sun May 31, 2009 6:55 am    Post subject: Reply with quote

yngwin wrote:
Ormaaj wrote:
d2_racing wrote:
Ext3 or reiserFs4, ext4 is still experimental.
ext4 is considered stable - moreso than reiser4 (and faster). If you look at the benchmarks reiser4 pretty consistently bombs in comparison to everything else.

Says who?
Ext4 considered stable??? That's ridiculous. Reiser4 has been around much, much longer and is a lot more stable. So if you want stable and fast, Reiser4 is the way to go. But you're free to use whatever else. :wink:

Only-sort-of-offtopic: If you want your bugs fixed...I can tell you which one is in the mainline kernel tree, and which one isn't. ReiserFS is not all it's cracked up to be and using Reiser4 requires you to run the -mm trees or pull from the -mm tree and write the changes over a mainline kernel. I think there are bad politics at work against reiserfs/reiser4... :/ It's lousy, but politics plays a big role in open source. If we could all stop fighting each other we would've knocked Microsoft clean out of business by now with the energy that has been expended on religious wars.

My Portage tree lives happily on ext2.

Ontopic: I agree with what's been said about btrfs...that is an exciting filesystem. However, the Extended filesystems aren't going anywhere, no matter how much we might throw rocks at them. Ext3 is really quite good and I'm excited about Ext4. Resier is still alright, although some of its features scare me when I consider that I might not be able to get my data off if I hose the tree...
_________________
/* Nobody will ever see this message :-) */
panic("Cannot initialize video hardware\n");
"As much as it pains me, we hope that developers know what they're doing." - wolf31o2
Back to top
View user's profile Send private message
yngwin
Retired Dev
Retired Dev


Joined: 19 Dec 2002
Posts: 4572
Location: Suzhou, China

PostPosted: Sun May 31, 2009 10:46 am    Post subject: Reply with quote

Captain Newbie wrote:
[...] using Reiser4 requires you to run the -mm trees or pull from the -mm tree and write the changes over a mainline kernel.

Not true. There are patches against vanilla linux releases.
_________________
"Those who deny freedom to others deserve it not for themselves." - Abraham Lincoln
Free Culture | Defective by Design | EFF
Back to top
View user's profile Send private message
Captain Newbie
Apprentice
Apprentice


Joined: 22 Dec 2006
Posts: 182
Location: Socal

PostPosted: Sun May 31, 2009 2:02 pm    Post subject: Reply with quote

yngwin wrote:
Captain Newbie wrote:
[...] using Reiser4 requires you to run the -mm trees or pull from the -mm tree and write the changes over a mainline kernel.

Not true. There are patches against vanilla linux releases.

Very well, I stand corrected.
_________________
/* Nobody will ever see this message :-) */
panic("Cannot initialize video hardware\n");
"As much as it pains me, we hope that developers know what they're doing." - wolf31o2
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum