View previous topic :: View next topic |
Author |
Message |
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Wed Apr 29, 2015 12:08 am Post subject: |
|
|
0.6.4.1 is highly recommended - especially due to the following commit:
https://github.com/zfsonlinux/zfs/commit/7af63221dc6a1dbd3ce7ebf88136b0169341a4f8
Allocate zfs_znode_cache on the Linux slab wrote: |
The Linux slab, in general, performs better than the SPl slab in cases
where a lot of objects are allocated and fragmentation is likely present.
This patch fixes pathologically bad behavior in cases where the ARC is
filled with mostly metadata and a user program needs to allocate and
dirty enough memory which would require an insignificant amount of the
ARC to be reclaimed.
If zfs_znode_cache is on the SPL slab, the system may spin for a very
long time trying to reclaim sufficient memory. If it is on the Linux
slab, the behavior has been observed to be much more predictible; the
memory is reclaimed more efficiently. |
(stability & responsiveness, reliability under load)
https://github.com/zfsonlinux/zfs/commit/e548a597d0820fecf89137c0aa7add8ad4aa340c
Mark additional functions as PF_FSTRANS wrote: |
Prevent deadlocks by disabling direct reclaim during all NFS, xattr,
ctldir, and super function calls. This is related to 40d06e3. |
+
https://github.com/zfsonlinux/zfs/commit/320e58d5a61fb4264acfb1782b2a337063ae13fa
Extend PF_FSTRANS critical regions wrote: |
Additional testing has shown that the region covered by PF_FSTRANS
needs to be extended to cover the zpl_xattr_security_init() and
init_acl() functions. The zpl_mark_dirty() function can also recurse
and therefore must always be protected. |
the latter 2 out of the three are mostly related to stability & locking (deadlocks, livelocks, etc.),
which can occur under certain workloads or after some runtime
the branch is:
https://github.com/zfsonlinux/zfs/commits/zfs-0.6.4-release
so - no, it's not much ("too") behind
but it addresses some sore points _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
fearedbliss Retired Dev
Joined: 13 Jul 2012 Posts: 4 Location: Brooklyn, NY
|
Posted: Wed Apr 29, 2015 2:47 am Post subject: |
|
|
WWWW wrote: | thank you for the work.
Is the 0.6.4.1 too far behind?
Should I try 0.6.4.1 from bliss overlay?
I went through all the trouble setting up layman and such. Now I can choose official 0.6.4 or non-official 0.6.4.1.
There seems to be important fixes:
Code: |
- Fixed io-spare.sh script for ZED.
- Fixed multiple deadlocks which might occur when reclaiming memory.
- Fixed excessive CPU usage for meta data heavy workloads when reclaiming the ARC.
|
Deadlocking with memory seems like a delicate issue.
thanks again.
ps.: oh packaging, that looks like an easy task, is it possible to help with packaging? How to start helping? |
It is ready when it is ready. Asking/demanding for updates when this is completely a volunteer project won't help anyone. Also ryao shouldn't need to explain himself on why he hasn't pushed the builds.
My overlay is highly volatile btw. Most of the stuff I push there is pretty much for myself (I just make it public), don't blame me if stuff breaks or vanishes.
@kernelOfTruth
I see you are maintaining some zfs srm stuff. I made this a year ago and it automates the process:
https://github.com/fearedbliss/sysresccd-moddat
You can also find the latest isos here (with mirrors): http://xyinn.org/doku.php |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Wed Apr 29, 2015 12:32 pm Post subject: |
|
|
WWWW wrote: | is it possible to help with packaging? How to start helping? |
Start by bumping ebuilds in your local overlay. This simply means copying them to a newer-versioned filename, and then seeing how the build goes, then editing the ebuild to make it work for you.
Then you post it to fellow users, usually via bugzilla; quite often you don't need to edit anything at all and can simply state that copying the old ebuild to the new name works.
That's how everyone else is doing it, and where all those overlays come from.
You should be discussing with fellow users, preferably in IRC or on the forums, especially when you're starting out.
#gentoo-dev-help on IRC: chat.freenode.net and #bash for the implementation language.
For build-system help, eg make, autoconf, automake, cmake et al, the best place is ##workingset though there is a #cmake as well.
Another way to help out easily, is to file stabilisation bugs for packages you follow, provided they fit the criteria outlined. |
|
Back to top |
|
|
ryao Retired Dev
Joined: 27 Feb 2012 Posts: 132
|
Posted: Wed Apr 29, 2015 1:09 pm Post subject: |
|
|
WWWW wrote: | thank you for the work.
Is the 0.6.4.1 too far behind?
Should I try 0.6.4.1 from bliss overlay?
I went through all the trouble setting up layman and such. Now I can choose official 0.6.4 or non-official 0.6.4.1.
There seems to be important fixes:
Code: |
- Fixed io-spare.sh script for ZED.
- Fixed multiple deadlocks which might occur when reclaiming memory.
- Fixed excessive CPU usage for meta data heavy workloads when reclaiming the ARC.
|
Deadlocking with memory seems like a delicate issue.
thanks again.
ps.: oh packaging, that looks like an easy task, is it possible to help with packaging? How to start helping? |
They are useful fixes and they are a continuation of the kmem-rework that I started last summer. However, there are a few things to consider.
- The dependencies for sys-kernel/spl and sys-fs/zfs-kmod were not designed for the interim versioning scheme that upstream decided to use. It was originally just for Fedora/EPEL packaging fixes, but it was expanded recently to include important fixes. If we go with them, I will need to take time to redesign the dependencies to handle it and I don't have that for a few days. This is why I went with 0.6.4 rather than 0.6.4.1.
- There are multiple deadlocks that I am tracking, some of which do not yet have fixes in HEAD and some of them are arguably more severe. My time is limited enough and the deadlock fixes in 0.6.4.1 are rare enough that I would prefer to batch them with the other deadlock fixes.
As for helping out, I regret to say that ZoL packaging is not quite as straightforward as it appears. Doing proper QA on core components such as filesystem drivers to ensure that they work is very important. Consequently, I have developed a few rules for myself when doing ZoL packaging:
- Do proper regression testing to ensure that as many foreseeable problems as possible are avoided, including but not limited to recurrences of past mistakes.
- Never commit after 9pm (EST for me) because errors are more likely at that time.
- Never commit if unsure about QA.
The rationale behind these priorities requires some explanation. I have been doing the ZoL packaging for over 3 years and I have had made my share of mistakes during that time. I have also seen mistakes made by upstream by both myself and other upstream contributors. Each time there is a mistake by myself or by upstream, we both make our own efforts to prevent them from recurring. The best method is the development of automated unit tests that encapsulate knowledge gained from past mistakes, but writing them requires at least 3 times more time than testing by hand requires because you need to be certain that they test what you intended to test and that you design them in such a way that changes in the code do not cause false positives/negatives. The upstream tests are geared primarily toward the environment at LLNL where the project lead is employed as part of a team responsible for the software stack on the Sequoia supercomputer. Distribution-specific concerns (especially backports for key fixes) are not covered quite as well. This requires me to take time to develop my own automated tests, especially for things that are Gentoo-specific. Until I do, I will need to run tests by hand. This has been by far the largest time requirement for the ZoL packaging. While I have made some progress toward that in my spare time, I am not quite there yet. Once we have them, it will become a negligible drain on time and could be done by anyone when the tests are properly automated, but that would also eliminate the circumstances that lead to your offer of assistance.
That said, I would love assistance in developing automated tests. If you are interested in assisting in the development of those, please contact me in #zfsonlinux on the freenode IRC network. That is by far the most efficient way to reach me at the moment. |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Thu Apr 30, 2015 1:14 pm Post subject: |
|
|
FearedBliss wrote: |
It is ready when it is ready. Asking/demanding for updates when this is completely a volunteer project won't help anyone. Also ryao shouldn't need to explain himself on why he hasn't pushed the builds.
My overlay is highly volatile btw. Most of the stuff I push there is pretty much for myself (I just make it public), don't blame me if stuff breaks or vanishes.
|
Acknowledged,
so I hope you don't mind if I forked your repo immediately & uploaded your scripts to have a non-moving target
Thank you very much !
that should definitely make things easier,
I'll take a deeper look in my free time _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
ryao Retired Dev
Joined: 27 Feb 2012 Posts: 132
|
Posted: Fri May 01, 2015 2:20 pm Post subject: |
|
|
kernelOfTruth wrote: | FearedBliss wrote: |
It is ready when it is ready. Asking/demanding for updates when this is completely a volunteer project won't help anyone. Also ryao shouldn't need to explain himself on why he hasn't pushed the builds.
My overlay is highly volatile btw. Most of the stuff I push there is pretty much for myself (I just make it public), don't blame me if stuff breaks or vanishes.
|
Acknowledged,
so I hope you don't mind if I forked your repo immediately & uploaded your scripts to have a non-moving target |
If someone doesn't want others to be able to fork their software, they shouldn't make it OSS. I know of very few people whose idea of OSS is "look, but don't touch". While I am not a lawyer, I believe all software under an OSI-compliant license is licensed in such a way that forking is legal, which includes all of the OpenZFS/ZFSOnLinux code, all Gentoo-specific code, all OSS and "F/OSS" as per the FSF (which I believe to be a subset of OSS). Consequently, how others feel about it does not restrict you from taking it and doing your own things. If necessary, I have access to lawyers who I could ask, but I do not consider this to be a matter that requires confirmation.
Also, while I might make some comments with regard to the need for rigorous testing to minimize the potential for regressions, I do not intend to stop anyone from taking the code and doing what they want with it. I am a strong believer in the idea that we should share code without trying to force our own individual visions of how it would be used onto others. The 9999 ZFS ebuilds in the tree should facilitate that, with the caveat that less QA is done, especially if pointed at a private development branch. If you want to do this, you can put something along the lines of the following into make.conf:
Code: | #spl_LIVE_BRANCH=master
spl_LIVE_REPO='/home/richard/devel/spl'
#zfs_kmod_LIVE_BRANCH=master
zfs_kmod_LIVE_REPO='/home/richard/devel/zfs'
#zfs_LIVE_BRANCH=master
zfs_LIVE_REPO='/home/richard/devel/zfs' |
The 9999 ebuilds will honor those settings. Change the paths/urls to valid git repositories and you can build from any valid repository. You can even specify a branch (or even a precise commit via ${PKGNAME}_LIVE_COMMIT if I recall correctly). I use this myself when developing changes. My workstation's packages are often built in this manner. I used to use epatch_user for testing changes, but this was much more convenient, especially after I combined it with 9p-virtio such that I could build on the host for the guest without having to deal with mounting block devices or synchronization issues:
https://wiki.gentoo.org/wiki/ZFSOnLinux_Development_Guide#Gentoo_with_a_9p-virtio_rootfs
If anyone does do this, keep in mind NOT to fetch code from unencrypted remote repositories. Doing that leaves you open to the insertion of malicious code into the kernel through a MITM attack on the fetch itself. I am not aware of any instances of this attack in the wild, but it is something that merits attention. It is also why I update my local tree used for updates via emerge-webrsync with PGP verification. In the interest of full disclosure, the tree that I use for Gentoo commits is updated via a SSH-encrypted tunnel authenticated by a 2048-bit RSA keypair that was generated with sane entropy. |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sat May 02, 2015 1:16 pm Post subject: |
|
|
ryao wrote: | kernelOfTruth wrote: | FearedBliss wrote: |
It is ready when it is ready. Asking/demanding for updates when this is completely a volunteer project won't help anyone. Also ryao shouldn't need to explain himself on why he hasn't pushed the builds.
My overlay is highly volatile btw. Most of the stuff I push there is pretty much for myself (I just make it public), don't blame me if stuff breaks or vanishes.
|
Acknowledged,
so I hope you don't mind if I forked your repo immediately & uploaded your scripts to have a non-moving target |
If someone doesn't want others to be able to fork their software, they shouldn't make it OSS. I know of very few people whose idea of OSS is "look, but don't touch". While I am not a lawyer, I believe all software under an OSI-compliant license is licensed in such a way that forking is legal, which includes all of the OpenZFS/ZFSOnLinux code, all Gentoo-specific code, all OSS and "F/OSS" as per the FSF (which I believe to be a subset of OSS). Consequently, how others feel about it does not restrict you from taking it and doing your own things. If necessary, I have access to lawyers who I could ask, but I do not consider this to be a matter that requires confirmation.
|
Thank you very much for that !
Yes, that "look, but don't touch" was or still is very prominent in some opensource projects (over at #XDA a few projects can't even be called opensource anymore since some devs add changes and hide their work [e.g. kernels in binary form] behind a paywall and never publishing the source changes) which is sad - since the goal shall be the (public) improvement of the code as a common gain in quality of work.
Also since the code is published under the MPL 2.0 it should be fine - so no need of lawyers in that case
ryao wrote: |
Also, while I might make some comments with regard to the need for rigorous testing to minimize the potential for regressions, I do not intend to stop anyone from taking the code and doing what they want with it. I am a strong believer in the idea that we should share code without trying to force our own individual visions of how it would be used onto others. The 9999 ZFS ebuilds in the tree should facilitate that, with the caveat that less QA is done, especially if pointed at a private development branch. If you want to do this, you can put something along the lines of the following into make.conf:
Code: | #spl_LIVE_BRANCH=master
spl_LIVE_REPO='/home/richard/devel/spl'
#zfs_kmod_LIVE_BRANCH=master
zfs_kmod_LIVE_REPO='/home/richard/devel/zfs'
#zfs_LIVE_BRANCH=master
zfs_LIVE_REPO='/home/richard/devel/zfs' |
The 9999 ebuilds will honor those settings. Change the paths/urls to valid git repositories and you can build from any valid repository. You can even specify a branch (or even a precise commit via ${PKGNAME}_LIVE_COMMIT if I recall correctly). I use this myself when developing changes. My workstation's packages are often built in this manner. I used to use epatch_user for testing changes, but this was much more convenient, especially after I combined it with 9p-virtio such that I could build on the host for the guest without having to deal with mounting block devices or synchronization issues:
https://wiki.gentoo.org/wiki/ZFSOnLinux_Development_Guide#Gentoo_with_a_9p-virtio_rootfs
If anyone does do this, keep in mind NOT to fetch code from unencrypted remote repositories. Doing that leaves you open to the insertion of malicious code into the kernel through a MITM attack on the fetch itself. I am not aware of any instances of this attack in the wild, but it is something that merits attention. It is also why I update my local tree used for updates via emerge-webrsync with PGP verification. In the interest of full disclosure, the tree that I use for Gentoo commits is updated via a SSH-encrypted tunnel authenticated by a 2048-bit RSA keypair that was generated with sane entropy. |
And using and/or modifying your live ebuilds worked wonderfully !
well, obviously I followed a way much too complicated approach if I understood your lines correctly:
I first uploaded the git branch to github and from there let it fetch by portage
does that mean that e.g.
Code: | spl_LIVE_REPO='/home/richard/devel/spl' |
makes you autonomous from github - so that the repo is fetched from your /home ?
didn't know that was possible
Excellent !
ryao wrote: |
If anyone does do this, keep in mind NOT to fetch code from unencrypted remote repositories. Doing that leaves you open to the insertion of malicious code into the kernel through a MITM attack on the fetch itself. I am not aware of any instances of this attack in the wild, but it is something that merits attention. It is also why I update my local tree used for updates via emerge-webrsync with PGP verification. In the interest of full disclosure, the tree that I use for Gentoo commits is updated via a SSH-encrypted tunnel authenticated by a 2048-bit RSA keypair that was generated with sane entropy. |
It appears that I need to take a deeper look into this,
I'm already syncing my portage tree mainly via webrsync but one can never be too cautious _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
ryao Retired Dev
Joined: 27 Feb 2012 Posts: 132
|
Posted: Sun May 03, 2015 8:25 pm Post subject: |
|
|
kernelOfTruth wrote: | ryao wrote: |
Also, while I might make some comments with regard to the need for rigorous testing to minimize the potential for regressions, I do not intend to stop anyone from taking the code and doing what they want with it. I am a strong believer in the idea that we should share code without trying to force our own individual visions of how it would be used onto others. The 9999 ZFS ebuilds in the tree should facilitate that, with the caveat that less QA is done, especially if pointed at a private development branch. If you want to do this, you can put something along the lines of the following into make.conf:
Code: | #spl_LIVE_BRANCH=master
spl_LIVE_REPO='/home/richard/devel/spl'
#zfs_kmod_LIVE_BRANCH=master
zfs_kmod_LIVE_REPO='/home/richard/devel/zfs'
#zfs_LIVE_BRANCH=master
zfs_LIVE_REPO='/home/richard/devel/zfs' |
The 9999 ebuilds will honor those settings. Change the paths/urls to valid git repositories and you can build from any valid repository. You can even specify a branch (or even a precise commit via ${PKGNAME}_LIVE_COMMIT if I recall correctly). I use this myself when developing changes. My workstation's packages are often built in this manner. I used to use epatch_user for testing changes, but this was much more convenient, especially after I combined it with 9p-virtio such that I could build on the host for the guest without having to deal with mounting block devices or synchronization issues:
https://wiki.gentoo.org/wiki/ZFSOnLinux_Development_Guide#Gentoo_with_a_9p-virtio_rootfs
If anyone does do this, keep in mind NOT to fetch code from unencrypted remote repositories. Doing that leaves you open to the insertion of malicious code into the kernel through a MITM attack on the fetch itself. I am not aware of any instances of this attack in the wild, but it is something that merits attention. It is also why I update my local tree used for updates via emerge-webrsync with PGP verification. In the interest of full disclosure, the tree that I use for Gentoo commits is updated via a SSH-encrypted tunnel authenticated by a 2048-bit RSA keypair that was generated with sane entropy. |
And using and/or modifying your live ebuilds worked wonderfully !
well, obviously I followed a way much too complicated approach if I understood your lines correctly:
I first uploaded the git branch to github and from there let it fetch by portage
does that mean that e.g.
Code: | spl_LIVE_REPO='/home/richard/devel/spl' |
makes you autonomous from github - so that the repo is fetched from your /home ?
didn't know that was possible
Excellent ! |
You would want to do something like:
Code: | mkdir ~/devel
git clone https://github.com/zfsonlinux/spl.git ~/devel/spl |
Then do whatever changes you want there, commit them to a branch and tell the ebuild to look there through either portage's make.conf (like I did) or an environment variable. Once you do that, there is no need for a network connection to use the live ebuilds. |
|
Back to top |
|
|
fearedbliss Retired Dev
Joined: 13 Jul 2012 Posts: 4 Location: Brooklyn, NY
|
Posted: Sun May 03, 2015 11:28 pm Post subject: |
|
|
kernelOfTruth wrote: | FearedBliss wrote: |
It is ready when it is ready. Asking/demanding for updates when this is completely a volunteer project won't help anyone. Also ryao shouldn't need to explain himself on why he hasn't pushed the builds.
My overlay is highly volatile btw. Most of the stuff I push there is pretty much for myself (I just make it public), don't blame me if stuff breaks or vanishes.
|
|
I agree with what ryao said about OSS development. I published my stuff because I want people to use it. Feel free to fork and even burn the overlay on cds LOL. |
|
Back to top |
|
|
Manuel.G n00b
Joined: 07 Jan 2015 Posts: 8
|
Posted: Thu May 14, 2015 6:26 pm Post subject: |
|
|
Yamakuzure wrote: | I have just upgraded and have a question about that ewarn: Code: | * Messages for package sys-fs/zfs-kmod-0.6.4:
* This version of ZFSOnLinux includes support for new feature flags
* that are incompatible with ZFSOnLinux 0.6.3 and GRUB2 support for
* /boot with the new feature flags is not yet available.
* Do *NOT* upgrade root pools to use the new feature flags.
* Any new pools will be created with the new feature flags by default
* and will not be compatible with older versions of ZFSOnLinux. To
* create a newpool that is backward compatible, use
* zpool create -o version=28 ...
* Then explicitly enable older features. Note that the LZ4 feature has
* been upgraded to support metadata compression and has not been
* tested against the older GRUB2 code base. GRUB2 support will be
* updated as soon as the GRUB2 developers and Open ZFS community write
* GRUB2 patchese that pass mutual review. |
|
I've got a question regarding the ewarn as well: My complete / is on a ZFS pool, including /boot. That pool was created with 0.6.3-r1. After upgrading to the latest kernel and ZoL 0.6.4, the corresponding initramfs gives me an error when it tries to import my root pool. ("no dataset found")
As far as I understand the warning, it should mount without problems, I just should not upgrade the pool to the latest version before grub2 has been updated for that. Am I misunderstanding and I currently can't upgrade to 0.6.4 if I need to boot from ZFS? I wouldn't mind that, 0.6.3 is running perfectly fine for me, I'm just asking as I'm not 100% sure if I understand the ewarn correctly. If I'm understanding it right, the problem must be somewhere else in my new initramfs/kernel, I guess.
Thanks in advance! |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2284 Location: Adendorf, Germany
|
Posted: Fri May 15, 2015 8:46 am Post subject: |
|
|
Manuel.G wrote: | Yamakuzure wrote: | I have just upgraded and have a question about that ewarn: Code: | * Messages for package sys-fs/zfs-kmod-0.6.4:
* This version of ZFSOnLinux includes support for new feature flags
* that are incompatible with ZFSOnLinux 0.6.3 and GRUB2 support for
* /boot with the new feature flags is not yet available.
* Do *NOT* upgrade root pools to use the new feature flags.
* Any new pools will be created with the new feature flags by default
* and will not be compatible with older versions of ZFSOnLinux. To
* create a newpool that is backward compatible, use
* zpool create -o version=28 ...
* Then explicitly enable older features. Note that the LZ4 feature has
* been upgraded to support metadata compression and has not been
* tested against the older GRUB2 code base. GRUB2 support will be
* updated as soon as the GRUB2 developers and Open ZFS community write
* GRUB2 patchese that pass mutual review. |
|
I've got a question regarding the ewarn as well: My complete / is on a ZFS pool, including /boot. That pool was created with 0.6.3-r1. After upgrading to the latest kernel and ZoL 0.6.4, the corresponding initramfs gives me an error when it tries to import my root pool. ("no dataset found")
As far as I understand the warning, it should mount without problems, I just should not upgrade the pool to the latest version before grub2 has been updated for that. Am I misunderstanding and I currently can't upgrade to 0.6.4 if I need to boot from ZFS? I wouldn't mind that, 0.6.3 is running perfectly fine for me, I'm just asking as I'm not 100% sure if I understand the ewarn correctly. If I'm understanding it right, the problem must be somewhere else in my new initramfs/kernel, I guess.
Thanks in advance! | Weird. As long as you do not upgrade your pool, it is fully compatible with zfs-0.6.3 and remains so. I did upgrade to sys-kernel/genkernel-next-60 before creating my new initramfs, though.
And I can confirm that if your root is on zfs, but your /boot is not, you can happily upgrade.
So just put your /boot on a tiny ext2 partition and you should be fine.
(Here is a blog describing how to shrink zfs pools.) _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
Manuel.G n00b
Joined: 07 Jan 2015 Posts: 8
|
Posted: Fri May 15, 2015 10:05 am Post subject: |
|
|
Thanks for the reply! But shouldn't my current setup, with /boot being in my root pool, still work? The ewarn says "GRUB2 support for /boot with the new feature flags is not yet available", but my /boot is inside an old 0.6.3 pool, so it is not "/boot with the new feature flags". That's why I'm asking – the way the ewarn states it, I think it should just work. I only want to clarify this before starting to debug other things. Should I have misunderstood the ewarn, I'd rather wait for the GRUB2 update than fiddle with my root pool and /boot, as my current setup works just fine. |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2284 Location: Adendorf, Germany
|
Posted: Tue May 19, 2015 6:31 am Post subject: |
|
|
Manuel.G wrote: | Thanks for the reply! But shouldn't my current setup, with /boot being in my root pool, still work? The ewarn says "GRUB2 support for /boot with the new feature flags is not yet available", but my /boot is inside an old 0.6.3 pool, so it is not "/boot with the new feature flags". That's why I'm asking – the way the ewarn states it, I think it should just work. I only want to clarify this before starting to debug other things. Should I have misunderstood the ewarn, I'd rather wait for the GRUB2 update than fiddle with my root pool and /boot, as my current setup works just fine. | That's how I understood the ewarn, too. As long as you do not upgrade the pool, grub should be happily keep booting. _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sat Jun 13, 2015 7:39 pm Post subject: |
|
|
@ryao:
thank you
@FearedBliss:
thanks !
@ryao:
perhaps you or the others can explain where all the memory goes:
I ran a stage4 backup job shortly after the system started up
this saves everything on the root partition [Btrfs]
(excluding /home [ZFS], /usr/portage [Btrfs])
Code: | time (ionice -c3 schedtool -D -e tar -cp / -X /root/scripts/stage4.excl | ionice -c3 schedtool -D -e 7z a -si -t7z -m0=lzma2 -mx=7 -mfb=64 -md=64m -ms=on -mmt=8 /home/system/stage4-amd64-xeon-backup-197.t7z) |
space consumption of / (root)
Size Used Avail Use%
Worth mentioning is that there's also a 20 GB zram drive in RAM which was empty:
Code: | /dev/zram0 20G 191M 20G 1% /var/tmp |
zfs.conf:
Code: | options zfs zfs_arc_meta_limit=6442450944
#(6 GB)
options zfs zfs_arc_min=0x100000000
#(4 GB)
options zfs zfs_arc_max=0x200000000
#(8 GB) |
Do the caches of the ARC + ARC meta add up or is the ARC meta included in the ARC (logically it should, right ?)
The strange thing is:
cat /proc/meminfo
Quote: | Slab: 1292764 kB
SReclaimable: 883696 kB
SUnreclaim: 409068 kB |
cat /proc/spl/kstat/zfs/arcstats
Quote: | p 4 6041380352
c 4 8589934592
c_min 4 4294967296
c_max 4 8589934592
size 4 6401223376
hdr_size 4 25559456
data_size 4 6218892288
meta_size 4 113704448
other_size 4 38928912
|
Total RAM available is 32 GiB
When 75% of the backup job is done:
cat /proc/buddyinfo
Code: | Node 0, zone DMA 2 2 2 0 3 2 0 0 1 1 3
Node 0, zone DMA32 2244 1526 3282 1883 1155 643 411 277 153 79 86
Node 0, zone Normal 17170 4968 18074 14592 8752 4878 3132 1961 1161 680 512 |
7z consumes 4 GB when it's running
When the backup job is finished and manually running compaction:
cat /proc/buddyinfo
Code: | Node 0, zone DMA 2 2 2 0 3 2 0 0 1 1 3
Node 0, zone DMA32 1490 914 347 53 123 74 37 18 7 48 163
Node 0, zone Normal 12498 6803 2723 230 601 416 219 72 21 293 1066 |
Using the nvidia (proprietary) or the nouveau driver makes no difference
right now Btrfs consumes around 650 MB
Quote: | 631257 631256 99% 0.95K 19129 33 612128K btrfs_inode
198441 191889 96% 0.08K 3891 51 15564K btrfs_extent_state |
Running a
Code: | echo 1 > /proc/sys/vm/drop_caches |
doesn't really make much of a difference
Quote: | Slab: 1167000 kB
SReclaimable: 756032 kB
SUnreclaim: 410968 kB |
echo 3 > /proc/sys/vm/drop_caches
regularly leads to a loop, hang, etc. which does NOT finish and also does NOT help (at least the last time I tried - so I'm leaving that out)
Slub merges are enabled (NO slub_nomerge),
using slub_nomerge did not really make a difference in this behavior
All this points to heavy memory fragmentation right ?
Not many high order allocations are possible anymore
Around 2-2.5 GiB of RAM are used by apps (okular, evince, firefox, wine, xfce, konqueror)
Running ZFS with the ABD or the Lock contention fix patchstack did not make any difference
Last thing I could think of is disabling zram entirely (previously I had a swap-partition in RAM),
now only /var/tmp in RAM with zram
and see whether that makes a change - which I doubt
Tl;dr
is it possible that ZFS leads to such grave memory fragmentation ?
how to reverse or limit it ?
The only thing right now to reset it is to export the zpool
There must be a better way
edit:
it seems dropping caches and running compaction, then giving it some time - and dropping caches and running compaction again - helps somewhat:
Quote: | cat /proc/buddyinfo
Node 0, zone DMA 2 2 2 0 3 2 0 0 1 1 3
Node 0, zone DMA32 4850 3202 1724 780 391 277 199 141 108 85 432
Node 0, zone Normal 27725 18747 10470 5225 1982 935 402 143 66 151 4237 |
that should be more or less okay
Anyway - I filed an issue over at ZFS' github issue-tracker _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Fri Jun 19, 2015 1:36 am Post subject: |
|
|
ZFS lower latency settings
Code: | echo 128 > /sys/module/zfs/parameters/zfs_vdev_max_active
# default zfs_vdev_max_active = 1000
echo 3 > /sys/module/zfs/parameters/zfs_vdev_async_write_max_active
# default zfs_vdev_async_write_max_active = 10
echo 5 > /sys/module/zfs/parameters/zfs_vdev_sync_write_max_active
# default zfs_vdev_sync_write_max_active = 10
echo 3 > /sys/module/zfs/parameters/zfs_vdev_async_read_max_active
# default zfs_vdev_async_read_max_active = 3
echo 10 > /sys/module/zfs/parameters/zfs_vdev_sync_read_max_active
# default zfs_vdev_sync_read_max_active = 10
echo 5 > /sys/module/zfs/parameters/zfs_vdev_sync_write_min_active
# default zfs_vdev_sync_write_min_active = 10
echo 1 > /sys/module/zfs/parameters/zfs_vdev_async_read_min_active
# default zfs_vdev_async_read_min_active = 1
echo 5 > /sys/module/zfs/parameters/zfs_vdev_sync_read_min_active
# default zfs_vdev_sync_read_min_active = 10
echo 1 > /sys/module/zfs/parameters/zfs_vdev_async_write_min_active
# default zfs_vdev_async_write_min_active = 1
echo 2 > /sys/module/zfs/parameters/zfs_txg_timeout
# default zfs_txg_timeout = 5 |
With the default settings I'm getting latencies up to 3000 ms+ when running rsync or copying big stuff (measured with latencytop and the CFS cpu scheduler)
With those settings the latencies are down to 200-300 ms (so far)
For giggles you could try out setting all of the above to 1
except zfs_vdev_max_active
which might work even better _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2284 Location: Adendorf, Germany
|
Posted: Fri Jun 19, 2015 7:09 am Post subject: |
|
|
Alright, I have put your commands into a starter file in /etc/local.d. Let's see what it does.
Good find! Thank you very much!
Maybe I will try out bfq again. I have switched to no-op yesterday, because bfq started to make my system unusable again.
Before the latest eudev and udev-init-scripts update, bfq worked like a charm. Of course I had set /sys/module/zfs/parameters/zfs_vdev_scheduler to "bfq", too. But it no longer works whenever I have a Windows 7 VM open, which I must have all day for my companies project managing. We use Inloox, so no chance without a modern MS Outlook.
Whenever the VM is running, clicking anywhere in a regular window outside the VM causes VMware to go up to 300% CPU usage (without actually doing anything) and the system hangs for 3-5 seconds. Even if the click just moves the cursor in a text editor.
With No-Op, all is well. Not as snappy as with bfq, but well enough. _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
MTZ Tux's lil' helper
Joined: 13 Jul 2002 Posts: 90 Location: Germany - near FFM
|
Posted: Sat Jun 27, 2015 6:09 am Post subject: |
|
|
Next version got released:
Quote: |
Version 0.6.4.2 of the zfs-0.6.4-release branch is now available. This
branch receives low-risk changes and important bug fixes made since the
last major release. New packages based on this release should make their
way into downstream package repositories in the near future.
For a list of changes in this release see below or refer to the release page
at https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.2
* Fix panic due to corrupt nvlist when running utilities zfsonlinux/zfs#3335
* Fix hard lockup due to infinite loop in zfs_zget() zfsonlinux/zfs#3349
* Fix panic on unmount due to iput taskq zfsonlinux/zfs#3281
* Improve metadata shrinker performance on pre-3.1 kernels zfsonlinux/zfs#3501
* Linux 4.1 compat: use read_iter() / write_iter()
* Linux 3.12 compat: NUMA-aware per-superblock shrinker
* Fix spurious hung task watchdog stack traces zfsonlinux/zfs#3402
* Fix module loading in zfs import systemd service zfsonlinux/zfs#3440
* Fix intermittent libzfs_init() failure to open /dev/zfs zfsonlinux/zfs#2556
|
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sat Jun 27, 2015 10:19 pm Post subject: |
|
|
If you're tinkering with ARC values - the following hex, byte and value conversion list might be helpful:
Quote: | Here is a list of hex conversions for common sizes : bytes memory pages
20Gb=0x500000000=21474836480 5242880 16Gb=0x400000000=17179869184 4194304 12Gb=0x300000000=12884901888 3145728 10Gb =0x280000000=10737418240 2621440 8Gb =0x200000000=8589934592 2097152 7Gb 0x1C0000000=7516192768 1835008 6Gb =0x180000000 = 6442450944 1572864 5Gb=0x140000000 =5368709120 1310720 4Gb=0x100000000= 4294967296 1048576 3.75Gb=0xF0000000 =4026531840 983040 3Gb=0xC0000000= 3221225472 786432 2.75Gb=0xB0000000= 2952790016 720896 2.5Gb =0xA0000000= 2684354560 655360 2Gb = 0x80000000 =2147483648 524288 1.92GB =0x7B03A000=2063835136 503866 1.75Gb= 0x70000000 =1879048192 458752 1.5Gb=0x60000000= 1610612736 393216 1Gb = 0X40000000 = 1073741824 262144 944Mb =0x3B031000 = 990056448 241713 940Mb=0XEC00000000=985661440 240640 880Mb= 0x37000000=922746880 225280 768Mb= 0X30000000=805606368 196681 ( 3/4 of 1 gb quad) 512Mb=0x20000000 = 536870912 131072 383Mb=0x17F00000=401604608 98048 256Mb=0x10000000 = 268435456 65536 128Mb=0x8000000 = 134217728 32768 80Mb =0x5000000 =83886080 20480 64MB= 0X4000000 = 67108864 16384 16Mb =0x1000000=16777216 4096 8MB=0x800000=8388608 2048 |
origin:
http://support.hp.com/si-en/document/c00936430 _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Sun Jun 28, 2015 5:06 pm Post subject: |
|
|
Blimey KoT that's a load of hassle; just use printf and shell arithmetic (bash has bitwise operators):
Code: | $ printx $((1 << 12))
1000
$ printd $((1 << 12))
4096
$ printd $((1 << 20))
1048576
$ printx $((1 << 20))
100000 | from my ~/.bashrc: Code: | # scripting
alias print="printf '%s\n'"
alias printl="printf '%s\n'"
# handy for code at cli
alias printx="printf '%X\n'"
alias printd="printf '%d\n'" |
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
tnt Veteran
Joined: 27 Feb 2004 Posts: 1222
|
Posted: Wed Jul 29, 2015 12:26 pm Post subject: |
|
|
any info on 0.6.4 -> 0.6.4.2 changes ?
[edit]
I just have tried 0.6.4.2 on gentoo-sources 3.14.47 (which works great with 0.6.3 for a long time) and after boot I had no pools visible.
modules were there:
Code: | zfs 1825152 0
zunicode 315324 1 zfs
zavl 3753 1 zfs
zcommon 30124 1 zfs
znvpair 37112 2 zfs,zcommon
spl 46914 3 zfs,zcommon,znvpair |
and service was started, but no pools.
Code: | /etc/init.d/zfs status
no pools available
no pools available
|
I had to revert to 0.6.3 and after reboot everything was fine.
the only thing that I notices after this short 0.6.4.2 test, is that now in 0.6.3, when I make zpool status, I get:
Code: | status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details. |
any clue?
[/edit] _________________ gentoo user
Last edited by tnt on Wed Jul 29, 2015 5:01 pm; edited 1 time in total |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
mrbassie l33t
Joined: 31 May 2013 Posts: 772 Location: over here
|
Posted: Thu Jul 30, 2015 7:05 am Post subject: |
|
|
tnt wrote: |
Code: | /etc/init.d/zfs status
no pools available
no pools available
|
I had to revert to 0.6.3 and after reboot everything was fine.
[/edit] |
That happened to me when I upgraded from 0.6.2-r4 to 0.6.3. I very cleverly thought 'Hmm, maybe I need to do zpool upgrade' and had to rebuild the pool and restore a stage 4 (/ on zfs) because I then couldn't downgrade.
More recently I went from 0.6.2-r4 to r5 to 0.6.3 to 0.6.4 to .1 then .2. Everything went fine.
Sorry that I don't know enough to help you troubleshoot but at least you're aware you're not the only one who's encountered this. |
|
Back to top |
|
|
tnt Veteran
Joined: 27 Feb 2004 Posts: 1222
|
Posted: Thu Jul 30, 2015 7:31 am Post subject: |
|
|
thx either way, mrbassie.
at least I saw no pools in 0.6.4.2, so I had no chance to "zpool upgrade" and kill my 40TB _________________ gentoo user |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|