View previous topic :: View next topic |
Author |
Message |
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Fri Sep 05, 2014 3:59 pm Post subject: ZFSonLinux, SystemRescueCD up-to-date kernel-modules (4.9.0) |
|
|
Hi guys,
in case you're running a rather "bleeding-edge" state of the ZFS zpools on your system ("feature flags"),
trying to gain access to it in a case of emergency (e.g. system isn't able to run, has been b0rked, filesystem corruption, etc.)
could be quite difficult.
My little new project might be of help to you:
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD
Quote: | target build architecture of the modules was march=x86-64
so it should run on the most hardware, if you run into issues please report, will have to re-compile all stuff, then |
in a nutshell all you need is:
nutshell wrote: | * an usbstick running SystemRescueCD 4.9.0, access to internet
* adding those modules to the usbstick, rebooting
* running depmod
* loading spl, then icp, then zfs
* and you should have access to your pool |
I hope that will be as helpful to you as it is to me
Enjoy ! _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004
Last edited by kernelOfTruth on Sun Nov 06, 2016 9:25 pm; edited 2 times in total |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2280 Location: Adendorf, Germany
|
Posted: Wed Sep 17, 2014 8:45 am Post subject: |
|
|
Is this the same as the Funtoo sysresccd with ZFS or did you add more to it? How do they differ apart from the version used? (Funtoo still uses 4.2.0) _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Wed Sep 17, 2014 3:21 pm Post subject: |
|
|
Nice work, koT: keep it up :-) |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Wed Sep 17, 2014 5:57 pm Post subject: |
|
|
Yamakuzure wrote: | Is this the same as the Funtoo sysresccd with ZFS or did you add more to it? How do they differ apart from the version used? (Funtoo still uses 4.2.0) |
Funtoo's SysRescCD is whole on it's own
in principle yes: unfortunately I couldn't find the bliss-isomaker since the creator totally nuked the repo with it from github's existance , bliss-initramfs 1.8.1 luckily still was on github's history
so it's the up-to-date kernel modules from yesterday - without the integration into the iso
the modules are bleeding edge in contrast to Funtoo's stable (release) modules and contain additional patches
this should make deduplication, usage of the L2ARC and heavy zvol usage more bearable
I've had some issues with zvol-creation and 2484 in the past so YMMV
otherwise I would have probably used an bleeding-edge 3.16 kernel with latest btrfs integration-branch code (which really seems to behave tons more reasonable than in the past)
steveL wrote: | Nice work, koT: keep it up |
thanks mate _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
nitro322 Guru
Joined: 24 Jul 2002 Posts: 594 Location: USA
|
Posted: Fri Oct 03, 2014 3:59 am Post subject: |
|
|
Thanks, kernelOfTruth. I'm in the middle of testing out ZFS for a new home server right now and this was one of my areas of concern. I was able to successfully import/mount, update, and then export/unmount my ZFS filesystem from within syscresccd after adding your kernel modules to the image. Much appreciated. _________________ https://www.legroom.net/ |
|
Back to top |
|
|
e3k Guru
Joined: 01 Oct 2007 Posts: 513 Location: Inner Space
|
Posted: Thu Oct 09, 2014 8:36 pm Post subject: |
|
|
i used the readme provided by systemrescuecd to add your srms but when i run zpool i get a message that zfs.ko was not found. i am building my own rescue cd for the first time do you have more instructions for me than just in the nutshell? _________________
Flux & Contemplation - Portrait of an Artist in Isolation
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Thu Oct 09, 2014 9:45 pm Post subject: |
|
|
nitro322 wrote: | Thanks, kernelOfTruth. I'm in the middle of testing out ZFS for a new home server right now and this was one of my areas of concern. I was able to successfully import/mount, update, and then export/unmount my ZFS filesystem from within syscresccd after adding your kernel modules to the image. Much appreciated. |
My pleasure !
Glad it was of use to you
e3k wrote: | i used the readme provided by systemrescuecd to add your srms but when i run zpool i get a message that zfs.ko was not found. i am building my own rescue cd for the first time do you have more instructions for me than just in the nutshell? |
which readme are you referring to ?
the one in my repo or the following one: http://www.sysresccd.org/Modules ?
anyway - posting the Howto here:
HOWTO wrote: |
-----------------------------------------------------------------------------------------------------------------------
- when using SystemRescueCD 4.3.0 on an USB stick - everything needed to do is
- follow the instructions from http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick
- have the usbstick mounted (mount -t vfat /dev/sdf1 /mnt/usbstick)
- then copy over the zfs-core and zfs-kmod SRM files to the root of the usbstick(zfs-core-3.10.45-std430-amd64.srm,
zfs-core-3.14.9-alt430-amd64.srm, zfs-kmod-3.10.45-std430-amd64.srm, zfs-kmod-3.14.9-alt430-amd64.srm)
- make sure that the usbstick is bootable, otherwise rewrite syslinux to the device (e.g. syslinux /dev/sdf1 && sync)
- ...
- during boot choose the 64bit versions (both default and alternative are supported), pick your keymap configuration
- after boot has finished run 'depmod'
- then a simple 'modprobe zfs' should be enough
- check state and version of zfs with 'zpool upgrade -v'
- Enjoy ! |
hope that helps in some way
if anything's missing don't hesitate to ask and please write what steps you already did or where it fails _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
e3k Guru
Joined: 01 Oct 2007 Posts: 513 Location: Inner Space
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
drescherjm Advocate
Joined: 05 Jun 2004 Posts: 2790 Location: Pittsburgh, PA, USA
|
Posted: Thu Nov 13, 2014 5:57 pm Post subject: |
|
|
@kernelOfTruth
Thanks a lot. This greatly helped me make my own sysrescue iso. I needed that because I could not use your latest release on an old Athlon 64 X2 system and the funtoo version does not support git zfs..
Also I needed an ISO because I use YUMI to make a bootable USB stick with several different ISOs installed.
http://www.pendrivelinux.com/yumi-multiboot-usb-creator/ _________________ John
My gentoo overlay
Instructons for overlay |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Thu Nov 20, 2014 7:58 pm Post subject: |
|
|
Quote: | testing modules with
CFLAGS="-O2 -march=nocona -mtune=generic -pipe"
CXXFLAGS="${CFLAGS}"
spl- & zfs-tree states are from upstream, NOW (11/20/2014; 20.11.2014)
Since lots of patches from the last testing modules have been merged meanwhile into upstream
and since 2129 (the major ARC changes) and the spl patch don't apply cleanly anymore - due to further evaluation work
they're left out for now;
as requested another round of test modules - now also for 3.10.45 kernel
additional SPL-patches:
* NONE this time
additional ZFS-patches:
* 10.10.2014_Illumos-4950 files sometimes can't be removed from a full filesystem_2784
* 14.11.2014_Illumos_5244 zio pipeline callers should explicitly invoke next stage_2828
* 14.11.2014_Store common Linux xattrs as native SA with xattr-sa_2809
* 19.11.2014_Don't read space maps during import for readonly pools_2909
This should make the modules especially helpful during troubleshooting and if you're affected by
one of the bugs in those additional patches
Keep in mind though that without the ARC changes there might be higher chance of possible OOM issues
(if your workflow causes it or has caused it in the past)
Cheers ! |
Commit: https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commit/729895d94074b9be9a90f1a43367b2544fc05101
Branch: https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commits/ZFS-for-SysRescCD-4.3.0-r2-testing02
lots of the additional patches since testing01 have been merged upstream meanwhile,
both the modules for the 3.14* and 3.10* kernel have been rebuilt
https://github.com/zfsonlinux/zfs/pull/2129 is left out this time (lower write & read rates, higher probability of OOM issues if your workflow or setup is prone to trigger it)
besides that it should (hopefully) work without issues
only compile-tested for now - haven't had any time to test them yet (just came home) _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
drescherjm Advocate
Joined: 05 Jun 2004 Posts: 2790 Location: Pittsburgh, PA, USA
|
Posted: Tue Nov 25, 2014 1:17 am Post subject: |
|
|
Sorry. I was out of the country last week so no access to the amd server. It looks like from the git hub feed that you have created amd modules. I'll look at these when I get a chance.. _________________ John
My gentoo overlay
Instructons for overlay |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2280 Location: Adendorf, Germany
|
Posted: Tue Dec 08, 2015 9:28 am Post subject: |
|
|
Do you have any plans to update to zfs-0.6.5 ? _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sat Dec 19, 2015 12:26 am Post subject: |
|
|
@Yamakuzure:
sorry for the delay,
sure - it's WIP right now ,
let's see if all works well with the kernel compilation and other stuff
It would be much help to know if
http://sourceforge.net/p/systemrescuecd/code/ci/master/tree/kernelcfg/config-3.14.12-100.fc19.x86_64
and
http://sourceforge.net/p/systemrescuecd/code/ci/master/tree/kernelcfg/config-4.1.8-100.fc21.x86_64
are the actual used kernel configs of the 4.6.1 release - otherwise I might have to boot into 4.6.1 just to be really sure
edit:
appears like that's unavoidable - important pieces of the config are missing from the sysresccd homepage ?
temporarily posting stuff here since Gmail (used as a personal pastebin and other cache) crashes with an too old webkit-gtk version + midori which is being used in SysRescCD 4.6.1
http://pastebin.com/MN6hfSzS config-4.1.12-alt461-amd64
http://pastebin.com/cYEvTRAq config-3.14.56-std461-amd64
edit2:
Could you please post the output of
I'm getting the following error right now during kernel compilation due to hardened toolchain + kernel config mismatch and/or errors
Code: | Documentation/vDSO/parse_vdso.o: In function `elf_hash':
parse_vdso.c:(.text+0xa4): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_init_from_sysinfo_ehdr':
parse_vdso.c:(.text+0x445): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_match_version':
parse_vdso.c:(.text+0x545): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_sym':
parse_vdso.c:(.text+0x6f2): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_init_from_auxv':
parse_vdso.c:(.text+0x7ad): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
scripts/Makefile.host:100: recipe for target 'Documentation/vDSO/vdso_standalone_test_x86' failed
make[2]: *** [Documentation/vDSO/vdso_standalone_test_x86] Error 1
scripts/Makefile.build:403: recipe for target 'Documentation/vDSO' failed
make[1]: *** [Documentation/vDSO] Error 2
make[1]: *** Waiting for unfinished jobs....
HOSTLD Documentation/mic/mpssd/mpssd
Makefile:926: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 2
|
so I want to be sure to what profile for GCC to switch to
my current output is
Code: | [11] x86_64-pc-linux-gnu-4.8.4 *
[12] x86_64-pc-linux-gnu-4.8.4-hardenednopie
[13] x86_64-pc-linux-gnu-4.8.4-hardenednopiessp
[14] x86_64-pc-linux-gnu-4.8.4-hardenednossp
[15] x86_64-pc-linux-gnu-4.8.4-vanilla
|
Like written on github on the issue-tracker - there exist pre-packaged rescue media by perfinion and Fearedbliss:
I'm currently using the latter, but since there were several issues that got addressed just recently
there's a real incentive to update the ZFS + SPL kernel-modules _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2280 Location: Adendorf, Germany
|
Posted: Mon Dec 28, 2015 9:27 am Post subject: |
|
|
Sorry for the delay, too!
They hid the kernel information in the ChangeLog. Grew me some grey hair to find the information, too. http://www.sysresccd.org/Changes-x86 wrote: |
4.6.1) 2015-10-29:
Standard kernels: Long-Term-Supported linux-3.14.56 (rescue32 + rescue64)
Alternative kernels: Long-Term-Supported linux-4.1.12 (altker32 + altker64)
Updated filesystem tools: GParted-0.24.0, btrfs-progs-4.2.2 | I could find no other place to get information on the alternative kernel version.
I will try out your new version this week and report back how it works for me. I have three zpools with all modern features enabled, so if they import and mount nicely, all should be well, right?
Thank you for your hard work! _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2280 Location: Adendorf, Germany
|
Posted: Tue Dec 29, 2015 7:09 am Post subject: |
|
|
I could import and use all my pools from sysresccd with your zfs enhancements. Everything worked and looked solid.
Great work! Thank you very much! _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
fearedbliss Retired Dev
Joined: 13 Jul 2012 Posts: 4 Location: Brooklyn, NY
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
Posted: Sun Nov 06, 2016 9:28 pm Post subject: |
|
|
v13 release is out
of the updated modules (for 4.9.0)
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commit/4410d7dde405257ce06cd9bb7733dc1202ffa0a7
Changelog wrote: |
v13
----------------------------------------
upgrade to SystemRescueCD 4.9.0
SPL & ZFS upgrade to 0.7.0-rc2 (November 4th 2016)
(ZFS 04.11.2016, SPL 02.11.2016)
ZFS contains one additional commit for allowing
to set ZFS options prior to pool import
-march=x86-64 -mtune=generic is used
binutils 2.26.1 and gcc 4.8.5 (default hardened profile) was used
ZFS is now capable of selecting much faster fletcher4 algorithms,
an additional kernel module is now needed and loaded: icp
updated CHECKSUMS and CHANGES file, added SIZES file.
Added the modified generic.sh script from files/resources/generic.sh and
hooks/zfs.sh
of bliss-initramfs which includes the additional icp kernel module of ZFS
Adding the ebuild message below for backwards compatibility:
Quote: | * This version of ZFSOnLinux includes support for new feature flags
* that are incompatible with previous versions. GRUB2 support for
* /boot with the new feature flags is not yet available.
* Do *NOT* upgrade root pools to use the new feature flags.
* Any new pools will be created with the new feature flags by default
* and will not be compatible with older versions of ZFSOnLinux. To
* create a newpool that is backward compatible wih GRUB2, use
*
* zpool create -d -o feature@async_destroy=enabled
* -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled
* -o feature@spacemap_histogram=enabled
* -o feature@enabled_txg=enabled
* -o feature@extensible_dataset=enabled -o feature@bookmarks=enabled
* ...
*
* GRUB2 support will be updated as soon as either the GRUB2
* developers do a tag or the Gentoo developers find time to backport
* support from GRUB2 HEAD. |
|
only load-tested so far: the modules loaded
Code: | depmod
modprobe spl
modprobe icp
modprobe zfs |
As always: these modules come with a disclaimer (https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/blob/ZFS-for-SysRescCD-4.6.1/DISCLAIMER)
May these modules help you in your backup and rescue efforts to be fruitful !
Enjoy _________________ https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa
Hardcore Gentoo Linux user since 2004 |
|
Back to top |
|
|
|