| View previous topic :: View next topic |
| Author |
Message |
r2dtu
Tux's lil' helper
Joined: 23 Aug 2002
Posts: 87
|
 Posted: Sun Oct 14, 2007 12:15 pm Post subject: reiser4 and how to speed up KDE |
 |
|
Reiser4 has one weakness - slow sync. While using KDE (mostly konqueror) on reiser4 you may notice that there is a lot of disk activity...e.g. when opening new tab in konqueror. This is because KDE is sending a sync request when working with temp files. So here is a patch that disables this behavior. I'm using it for a couple of months and see a nice performance boost and much less disk activity. No bad effects noticed yet, so I've decided to share it. This idea is taken from: http://bugs.kde.org/show_bug.cgi?id=110318
| Code: |
--- kdecore/ktempfile.cpp.org 2005-10-10 17:06:02.000000000 +0200
+++ kdecore/ktempfile.cpp 2007-01-30 21:39:50.000000000 +0100
@@ -231,16 +231,6 @@
}
}
- if (mFd >= 0)
- {
- result = FDATASYNC(mFd);
- if (result)
- {
- kdWarning() << "KTempFile: Error trying to sync " << mTmpName << ": " << strerror(errno) << endl;
- mError = errno;
- }
- }
-
return (mError == 0);
} |
|
|
| Back to top |
|
 |
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Sun Oct 14, 2007 1:29 pm Post subject: |
|
|
I didn't test the patch, yet, but I tried an idea from the bug-link you posted. I mounted /home/myuser/.kde/share as tmpfs and the speed difference is very obvious.
Logging out/logging in/konqueror it becomes faaaast. Much unlike kde we used to know. 
I'll play with the patch later tonight. 
Thanks
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
Gentree
Watchman
Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe
|
| Posted: Sun Oct 14, 2007 6:51 pm Post subject: |
|
|
Yes I like that tmpfs idea a bit better than circumventing ALL use of sync. KDE is so wide ranging and had it's fingers in so many pies I think it would be near impossible to say it is a safe move. Despite this being a falgrant abuse of sync I can only think it was done because somewhere there is a need for it.
Could you post details of how you set up ~/.kde/share , I dont use kde so I can't test but I have a friend using Suse who does and on a laptop so this could be very useful.
TIA.
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Sun Oct 14, 2007 8:00 pm Post subject: |
|
|
| Gentree wrote: | Yes I like that tmpfs idea a bit better than circumventing ALL use of sync. KDE is so wide ranging and had it's fingers in so many pies I think it would be near impossible to say it is a safe move. Despite this being a falgrant abuse of sync I can only think it was done because somewhere there is a need for it.
Could you post details of how you set up ~/.kde/share , I dont use kde so I can't test but I have a friend using Suse who does and on a laptop so this could be very useful.
TIA. |
Here is how i set it up:
| /etc/conf.d/local.start: | # /etc/conf.d/local.start
echo Mounting /home/myuser/.kde/share/ to ram
mkdir -p /home/__tmp
mount -t tmpfs tmpfs /home/__tmp
cp /home/myuser/.kde/share/* /home/__tmp/ -Rp
mount -t tmpfs tmpfs /home/myuser/.kde/share/
mv /home/__tmp/* /home/myuser/.kde/share/
umount /home/__tmp |
| /etc/conf.d/local.stop: | # /etc/conf.d/local.stop
echo Unmounting /home/myuser/.kde/share/ from ram
mount -t tmpfs tmpfs /home/__tmp
mv /home/myuser/.kde/share/* /home/__tmp/
umount /home/myuser/.kde/share/
cp /home/__tmp/* /home/myuser/.kde/share/ -Rp
umount /home/__tmp |
You have to change myuser to your user's name.
And when you change these two files, can do: | Code: | | # bash /etc/conf.d/local.start |
to test it.
When you'll reboot the files will be saved. If a power cut happens, then you'll be at the same position as it was before the last boot. (This might be even safer than to loose some of the kde's files)
Here du shows that the tmpfs mounted directory uses 16Mb of RAM. Not much.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
ok
Guru
Joined: 11 Jul 2006
Posts: 390
Location: germany
|
| Posted: Sun Oct 14, 2007 10:13 pm Post subject: |
|
|
This is a good idea, but unfortunately not for me:
| Code: | $ du ~/.kde/share/ -sh
303M /home/oliver/.kde/share/
$ du ~/.kde/share/apps/ -sh
265M /home/oliver/.kde/share/apps/
|
I'm using kmail and a lot of other kde applications.
I don't want to loose my emails if i can not shutdown the computer the normal way. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Mon Oct 15, 2007 12:46 am Post subject: |
|
|
Hmm, what is "the normal way". The script copies the data back to disk during the normal shutdown, but not when you plugh out the cable)
BTW, are there any issues considering the patch in the first post?
Anybody else tried to use it?
And, would it be possible/easy to add it to ebuilds and put it to local repository?
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
Gentree
Watchman
Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe
|
| Posted: Mon Oct 15, 2007 7:54 am Post subject: |
|
|
| ok wrote: | This is a good idea, but unfortunately not for me:
| Code: | $ du ~/.kde/share/ -sh
303M /home/oliver/.kde/share/
$ du ~/.kde/share/apps/ -sh
265M /home/oliver/.kde/share/apps/
|
I'm using kmail and a lot of other kde applications.
I don't want to loose my emails if i can not shutdown the computer the normal way. |
Even if something locks up X , if you can kill X or get a console you can still
| Code: | | bash /etc/conf.d/local.stop |
and flush everything back to disk.
How often does kde completely screw your Gentoo to the point where you have to pull the plug??
@zxy
Thanks for the scripts, saved me some digging.
I think the question with the patch is not only does it work but is it desirable, for the reasons I cited. I think effectively disabling all sync calls in KDE is too wide reaching. One example off the top of my head: how would this affect "safely disconnecting" a USB device? I think it is impossible to anticipate all possible uses of sync and decide whether this is safe. Hence my conclusion that it is not safe.
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
| Back to top |
|
|
r2dtu
Tux's lil' helper
Joined: 23 Aug 2002
Posts: 87
|
| Posted: Mon Oct 15, 2007 4:28 pm Post subject: |
|
|
Please note that the patch modifies only ktempfile. I don't think you'll ever set up your kde to use removable device for .kde/share or .kde/cache- I personally can't think of a reason why a temp file would ever need to be synced. If anyone can then I'll be glad to hear it. Maybe I do brake something with this patch that I'm not aware of  .
If you want to play with tmpfs maybe try to link .kde/cache- directory as well |
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Tue Oct 16, 2007 6:23 pm Post subject: |
|
|
| I think we should ask the KDE developers if this patch is useful/dangerous. I'll send an email to the kde-dev mailinglist. |
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Tue Oct 16, 2007 7:45 pm Post subject: |
|
|
KDE developer:
| Quote: | All that patch does is chop out the sync() function in kdelibs.
There's absolutely no way that this can possibly be a good idea. 
There aren't many valid uses of sync(), but when you need it there's
no other way to get the job done.
The areas where konqi is calling sync() should be audited to determine
if they are necessary (they probably aren't). This would be a good
Junior Job actually. |
In other words, they would gladly accept a patch which would do that. Maybe r2dtu would be into that
Edit:
| Quote: |
I suspect konqi is not calling sync() directly but is using KTempFile which is
calling sync() implicitly. I haven't read any of the code though so I'll go
look at that before discussing further. |
|
|
| Back to top |
|
|
r2dtu
Tux's lil' helper
Joined: 23 Aug 2002
Posts: 87
|
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Wed Oct 17, 2007 6:50 pm Post subject: |
|
|
| Well, there currently is a discussion going on, started by the first email I send after I read about your patch. KConfig does the syncing, and it does it because otherwise config files would get screwed up if you have a crash. Browsing means writing configfiles (history!) so that's the reason. They are looking for a solution, but so far they can't seem to find it. Personally, I don't think syncs are needed, but I'll keep you guys posted about the issue. |
|
| Back to top |
|
|
r2dtu
Tux's lil' helper
Joined: 23 Aug 2002
Posts: 87
|
| Posted: Wed Oct 17, 2007 7:20 pm Post subject: |
|
|
Found it http://marc.info/?t=119255937200009&r=1&w=2 Thanks for getting the professionals interrested. I'll shut up and see what comes out of this.
Anyway I feel like users of reiser4 "atomic filesystem" should be save. Stuff should be either written and safe, or not written at all. Unless this requires use of some not existing API somewhere in libreiser
I personally don't care about browser history, cache, cookies or bookmarks. But I'll think twice about disabling sync in KMail... |
|
| Back to top |
|
|
Gentree
Watchman
Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe
|
| Posted: Wed Oct 17, 2007 7:50 pm Post subject: |
|
|
yeah nice work superstoned you seem to have shone a light on this mirky business. KDE is pretty clunky , now I can see why if they are trying to cater for any time there's a power out by crippling disk caching.
Just what you dont need on a portable with no chance of a power off but a limitted battery. So much for spin down.
Maybe the whole partition should be backing itself up to google via ftp in case the disk drive goes down and some one loses an email 
Well you pays you money and you takes you pick.
I dont use kde but I do use R4 just because the power is not too relaible here. R4 is pretty bullet proof in that context. I use sync command when I need it, once a day before I hit the power switch. 
Well I do do a shutdown -h now sometimes just to play nice. 
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Wed Oct 17, 2007 9:01 pm Post subject: |
|
|
| Gentree wrote: | yeah nice work superstoned you seem to have shone a light on this mirky business. KDE is pretty clunky , now I can see why if they are trying to cater for any time there's a power out by crippling disk caching.
Just what you dont need on a portable with no chance of a power off but a limitted battery. So much for spin down.
Maybe the whole partition should be backing itself up to google via ftp in case the disk drive goes down and some one loses an email |
Well, that would be Online Desktop stuff, saving configuration online somewhere... Actually there are ppl thinking about such things in the KDE camp, we are currently very busy with KDE 4.0 but after that, expect interesting things to happen (as always, of course, hehe) |
|
| Back to top |
|
|
Gentree
Watchman
Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe
|
| Posted: Wed Oct 17, 2007 10:31 pm Post subject: |
|
|
Online Desktop the very idea makes me cringe.
You're obviously pretty close to the hub on this so I'll throw in my two penneth.
We're clearly not just talking about desktop icons here. Were talking bout the whole damn filing cabinet. Anything that requires remote backup is obviously important data and important data if often confidential data.
Most countries have some sort of data protection act that legally requires any person/business storing sensitive data to ensure it is safe from disclosure.
It's hard to see what "online" storage may be used as a destination for such data that would fullfil the legal requirement to protect it.
Certainly not some of the gigabytes of space offered by google and the likes who almost certainly will scan and index it.
I find the whole concept rather scary. Please dont encourage it.
Getting back to the sync abuse issue , it's probably worth noting that attempting to cover some users from power outs on desktop systems is the complete antithesis of what is required on a laptop. The one-size-fits-all approach being suggested in the thread you linked to can never work.
You've done a good job in getting this on the agenda, I hope you can instill some reason into the direction it takes.
Good luck.
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
| Back to top |
|
|
enderandrew
l33t
Joined: 25 Oct 2005
Posts: 731
|
| Posted: Thu Oct 18, 2007 12:52 am Post subject: |
|
|
Again however, I'm no expert but it seems like having a journal, or an atomic file system should eliminate the need for the software to create redundancy here, and slow down performance.
_________________
Nihilism makes me smile. |
|
| Back to top |
|
|
planesight
n00b
Joined: 18 Oct 2007
Posts: 1
Location: Terra
|
| Posted: Thu Oct 18, 2007 4:24 am Post subject: Atomic file systems. |
|
|
| enderandrew wrote: | | Again however, I'm no expert but it seems like having a journal, or an atomic file system should eliminate the need for the software to create redundancy here, and slow down performance. |
Adding a journal doesn't necessarily solve the problem.
Or in other words what happens in this case is that the new file is written out first (to avoid corruption in case there is a power failure or such while overwriting the old file). Then when the data is on disk the new file is renamed over the old one in one step so that the file is always consistent (it either has the old data or the new data, but not a mix).
On file systems like XFS, the rename operation can complete writing over the new metadata before the data for the new file is actually on-disk (i.e. it is still cached in memory). If the power fails then the file name in question points to the new inode but there's no data, or the data is incomplete (since it was never written). This occurs even with journaling since (unless I'm mistaken) journaling only ensures that the metadata is corrected, not the actual file data (because otherwise you'd always be writing out files to disk twice, first to the journal, then finally to its correct location).
The solution for userspace applications is to ensure that the new file is actually on-disk before calling rename(). Since KSaveFile can't know the relative importance of the file it's supposed to be saving it is not feasible to remove the fsync()/fdatasync() call from kdelibs. The actual thread on kde-devel has more discussion for those interested.
But the only solution from a Konqueror-coding standpoint would be to alter the history and bookmarks code to not use KConfig or KSaveFile when saving and instead create the files manually using Qt. (For those interested, history manager code is in kdebase/libkonq. Bookmarks are handled in kdelibs/kio/bookmarks. Both use KSaveFile)
If you're wondering why bookmarks are involved, Konqueror has functionality to keep track of the number of times a bookmarked page has been visited, which then requires updating the bookmarks file.
An easier solution may be to just have the bookmarks and history files in a tmpfs (requires only the $KDEHOME/share/apps/konqueror directory to be in the tmpfs for those low on memory, mine is 384K) |
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Thu Oct 18, 2007 8:30 am Post subject: |
|
|
| Gentree wrote: | Online Desktop the very idea makes me cringe.
You're obviously pretty close to the hub on this so I'll throw in my two penneth.
We're clearly not just talking about desktop icons here. Were talking bout the whole damn filing cabinet. Anything that requires remote backup is obviously important data and important data if often confidential data.
Most countries have some sort of data protection act that legally requires any person/business storing sensitive data to ensure it is safe from disclosure.
It's hard to see what "online" storage may be used as a destination for such data that would fullfil the legal requirement to protect it.
Certainly not some of the gigabytes of space offered by google and the likes who almost certainly will scan and index it.
I find the whole concept rather scary. Please dont encourage it. |
Well, you are right about possible dangers, but hey, a knife is dangerous - yet they sell 'em. Imagine how useful it could be to just log in into any KDE desktop anywhere in the world with your own password, and have your own desktop available... Of course, the idea is to sync this data, not ONLY have it online. It's an option.
Anyway, there is more to this than putting your data online in one huge storage. Much data IS online already - like flickr pictures, Youtube videos, Google docs documents and much more. Having your bookmarks always available, transparantly sharing every file you have with just a mouseclick, being able to search online data from your offline applications - things like that are what Online Desktop stuff is about. And it's not some utopian thing, we have much of it already - we just want to make it easier for application developers etc to do.
Take GHNS - you know it? It's the system you can use to download wallpapers in KDE, or superkaramba applets, and Amarok Extensions, and language packs for Kvoctrain. We've improved it for KDE 4, you can also upload your own stuff now, share it with the world. Give ratings, comments. All in an easy to use, the-same-for-all-apps interface. No more searching on websites etc - it's just there.
Or take marble, or Kstars. Or amarok. These apps download data from wikipedia and other sites for you, like info about songs, cities, geographical data, extra star data, pictures of constellations... The ability to share pictures on flickr with Digikam. That's online desktop stuff. And all these things ARE already in KDE, we just want to elaborate on them, make it more seamless.
| Gentree wrote: | Getting back to the sync abuse issue , it's probably worth noting that attempting to cover some users from power outs on desktop systems is the complete antithesis of what is required on a laptop. The one-size-fits-all approach being suggested in the thread you linked to can never work.
You've done a good job in getting this on the agenda, I hope you can instill some reason into the direction it takes.
Good luck.
|
Well, lets hope so. I do believe there must be better ways of doing this without losing data... |
|
| Back to top |
|
|
funkyou
n00b
Joined: 08 Feb 2007
Posts: 3
|
|
| Back to top |
|
|
musv
Advocate
Joined: 01 Dec 2002
Posts: 3337
Location: de
|
| Posted: Sun Oct 21, 2007 5:26 pm Post subject: |
|
|
| ok wrote: | This is a good idea, but unfortunately not for me:
| Code: | $ du ~/.kde/share/ -sh
303M /home/oliver/.kde/share/
$ du ~/.kde/share/apps/ -sh
265M /home/oliver/.kde/share/apps/
|
I'm using kmail and a lot of other kde applications.
I don't want to loose my emails if i can not shutdown the computer the normal way. |
In my case the most space is needed by the amarok-albumcovers. So I copied the albumcovers (.kde/share/apps/amarok/albumcovers) to ~/.covers and linked that directory to the kde-share dir. Now there are 28 mb in the tmpfs used. That's ok for me. Maybe you can do the same with your email account.
I use reiser4 for more than one and a half year. With some kde apps I began to use one year ago. I almost believed that some kde apps are terribly buggy and unusable but it's only a problem of that permanent syncing. E.g. with Quanta sometime I got crazy. It was simply unusable if you have bigger documents opened and have to wait about 2 seconds to type a php command under 90% cpu last. After caching .kde/share to tmpfs the first tests are impressive.
So thanks a lot for that hint. |
|
| Back to top |
|
|
charlie
n00b
Joined: 21 Oct 2007
Posts: 36
|
| Posted: Wed Oct 24, 2007 3:19 pm Post subject: |
|
|
Thanks for the link. I had to break up that patch though for my build.
Will that change come with KDE 3.5.9 in the future KDE releases? |
|
| Back to top |
|
|
superstoned
Guru
Joined: 17 Dec 2004
Posts: 432
|
| Posted: Wed Oct 24, 2007 5:45 pm Post subject: |
|
|
| charlie wrote: |
Thanks for the link. I had to break up that patch though for my build.
Will that change come with KDE 3.5.9 in the future KDE releases? |
Not unless they can solve the problems around it, and as far as I can tell, there currently really isn't enough manpower for this. First getting KDE 4.0 out, then we'll see...
in case I forget, someone should ping this thread some time after the 4.0 release... |
|
| Back to top |
|
|
Hell-Razor
Guru
Joined: 10 Jun 2004
Posts: 458
|
| Posted: Mon Jun 09, 2014 5:06 pm Post subject: |
|
|
PING!
_________________
Don't ever tell anybody anything. If you do, you start missing everybody. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
|
| Back to top |
|
|
|
Unsupported Software
