Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ZFSonLinux, SystemRescueCD up-to-date kernel-modules (4.6.1)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 05, 2014 3:59 pm    Post subject: ZFSonLinux, SystemRescueCD up-to-date kernel-modules (4.6.1) Reply with quote

Hi guys,


in case you're running a rather "bleeding-edge" state of the ZFS zpools on your system ("feature flags"),
trying to gain access to it in a case of emergency (e.g. system isn't able to run, has been b0rked, filesystem corruption, etc.)
could be quite difficult.

My little new project might be of help to you:

https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD

Quote:
target build architecture of the modules was march=core2

so it should run on the most hardware, if you run into issues please report, will have to re-compile all stuff, then



in a nutshell all you need is:

nutshell wrote:
* an usbstick running SystemRescueCD 4.3.0, access to internet
* adding those modules to the usbstick, rebooting
* and you should have access to your pool :)



I hope that will be as helpful to you as it is to me

Enjoy !
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D


Last edited by kernelOfTruth on Sun Dec 20, 2015 10:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Tue Sep 16, 2014 9:45 pm    Post subject: Reply with quote

2nd release is out

of the updated modules (still for 4.3.0)



https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commits/ZFS-for-SysRescCD-4.3.0-r2


Changelog wrote:

still on SystemRescueCD 4.3.0
sys-kernel/spl at f9bde4f74be5e5030571415d9c869400dff3eda6
sys-fs/zfs-kmod & sys-fs/zfs at 2d501583430e54be39c6aed033fc6f841e469f23
added 2484 patchset (zvol rework)
added 2351 patchset (sha256 x86_64 optimization v2; SSSE3, AVX, AVX2)
added 2672 patchset (Illumos #3525 persistent l2arc)
kernel-modules compiled with march=nocona, should work on older 64bit boxes now

_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1882
Location: Bardowick, Germany

PostPosted: Wed Sep 17, 2014 8:45 am    Post subject: Reply with quote

Is this the same as the Funtoo sysresccd with ZFS or did you add more to it? How do they differ apart from the version used? (Funtoo still uses 4.2.0)
_________________
systemd - The biggest fallacies
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4475
Location: The Peanut Gallery

PostPosted: Wed Sep 17, 2014 3:21 pm    Post subject: Reply with quote

Nice work, koT: keep it up :-)
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Wed Sep 17, 2014 5:57 pm    Post subject: Reply with quote

Yamakuzure wrote:
Is this the same as the Funtoo sysresccd with ZFS or did you add more to it? How do they differ apart from the version used? (Funtoo still uses 4.2.0)


Funtoo's SysRescCD is whole on it's own


in principle yes: unfortunately I couldn't find the bliss-isomaker since the creator totally nuked the repo with it from github's existance :( , bliss-initramfs 1.8.1 luckily still was on github's history :o


so it's the up-to-date kernel modules from yesterday - without the integration into the iso

the modules are bleeding edge in contrast to Funtoo's stable (release) modules and contain additional patches

this should make deduplication, usage of the L2ARC and heavy zvol usage more bearable

I've had some issues with zvol-creation and 2484 in the past so YMMV



otherwise I would have probably used an bleeding-edge 3.16 kernel with latest btrfs integration-branch code (which really seems to behave tons more reasonable than in the past)



steveL wrote:
Nice work, koT: keep it up :-)


thanks mate :)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
nitro322
Guru
Guru


Joined: 24 Jul 2002
Posts: 555
Location: USA

PostPosted: Fri Oct 03, 2014 3:59 am    Post subject: Reply with quote

Thanks, kernelOfTruth. I'm in the middle of testing out ZFS for a new home server right now and this was one of my areas of concern. I was able to successfully import/mount, update, and then export/unmount my ZFS filesystem from within syscresccd after adding your kernel modules to the image. Much appreciated.
_________________
http://www.legroom.net/
Back to top
View user's profile Send private message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 264
Location: Slovakia

PostPosted: Thu Oct 09, 2014 8:36 pm    Post subject: Reply with quote

i used the readme provided by systemrescuecd to add your srms but when i run zpool i get a message that zfs.ko was not found. i am building my own rescue cd for the first time do you have more instructions for me than just in the nutshell?
_________________
--_o_--
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Thu Oct 09, 2014 9:45 pm    Post subject: Reply with quote

nitro322 wrote:
Thanks, kernelOfTruth. I'm in the middle of testing out ZFS for a new home server right now and this was one of my areas of concern. I was able to successfully import/mount, update, and then export/unmount my ZFS filesystem from within syscresccd after adding your kernel modules to the image. Much appreciated.


My pleasure !

Glad it was of use to you :D

e3k wrote:
i used the readme provided by systemrescuecd to add your srms but when i run zpool i get a message that zfs.ko was not found. i am building my own rescue cd for the first time do you have more instructions for me than just in the nutshell?


which readme are you referring to ?

the one in my repo or the following one: http://www.sysresccd.org/Modules ?

anyway - posting the Howto here:

HOWTO wrote:

-----------------------------------------------------------------------------------------------------------------------
- when using SystemRescueCD 4.3.0 on an USB stick - everything needed to do is
- follow the instructions from http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick
- have the usbstick mounted (mount -t vfat /dev/sdf1 /mnt/usbstick)
- then copy over the zfs-core and zfs-kmod SRM files to the root of the usbstick(zfs-core-3.10.45-std430-amd64.srm,
zfs-core-3.14.9-alt430-amd64.srm, zfs-kmod-3.10.45-std430-amd64.srm, zfs-kmod-3.14.9-alt430-amd64.srm)
- make sure that the usbstick is bootable, otherwise rewrite syslinux to the device (e.g. syslinux /dev/sdf1 && sync)
- ...
- during boot choose the 64bit versions (both default and alternative are supported), pick your keymap configuration
- after boot has finished run 'depmod'
- then a simple 'modprobe zfs' should be enough
- check state and version of zfs with 'zpool upgrade -v'
- Enjoy !



hope that helps in some way

if anything's missing don't hesitate to ask and please write what steps you already did or where it fails
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 264
Location: Slovakia

PostPosted: Fri Oct 10, 2014 8:31 am    Post subject: Reply with quote

thank you it is working now:
mounting the sysrescuecd.iso -o loop
running the usb_inst.sh from the mount dir root
copying your srms
rebooting
depmod
modprobe zfs
_________________
--_o_--
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Oct 10, 2014 11:46 pm    Post subject: Reply with quote

awesome ! :)


updated forum signature & github to point to latest branch
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2772
Location: Pittsburgh, PA, USA

PostPosted: Thu Nov 13, 2014 5:57 pm    Post subject: Reply with quote

@kernelOfTruth

Thanks a lot. This greatly helped me make my own sysrescue iso. I needed that because I could not use your latest release on an old Athlon 64 X2 system and the funtoo version does not support git zfs..

Also I needed an ISO because I use YUMI to make a bootable USB stick with several different ISOs installed.
http://www.pendrivelinux.com/yumi-multiboot-usb-creator/
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Nov 15, 2014 6:18 pm    Post subject: Reply with quote

@drescherjm:

my pleasure :)


could you please post the CFLAGS/CXXFLAGS you used to get it to work on your CPU ?

for now I compiled the testing-modules for the 3.14* kernel with

Code:
-march=nocona -mtune=generic


but I'm not sure whether it would run on an Athlon 64 X2 (even though it supports SSE3)


edit:

https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commit/ca50865edb6186ed3b1f9b05a1c0964a338b7a73
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Thu Nov 20, 2014 7:58 pm    Post subject: Reply with quote

Quote:
testing modules with

CFLAGS="-O2 -march=nocona -mtune=generic -pipe"
CXXFLAGS="${CFLAGS}"

spl- & zfs-tree states are from upstream, NOW (11/20/2014; 20.11.2014)

Since lots of patches from the last testing modules have been merged meanwhile into upstream
and since 2129 (the major ARC changes) and the spl patch don't apply cleanly anymore - due to further evaluation work
they're left out for now;
as requested another round of test modules - now also for 3.10.45 kernel

additional SPL-patches:
* NONE this time

additional ZFS-patches:
* 10.10.2014_Illumos-4950 files sometimes can't be removed from a full filesystem_2784
* 14.11.2014_Illumos_5244 zio pipeline callers should explicitly invoke next stage_2828
* 14.11.2014_Store common Linux xattrs as native SA with xattr-sa_2809
* 19.11.2014_Don't read space maps during import for readonly pools_2909

This should make the modules especially helpful during troubleshooting and if you're affected by
one of the bugs in those additional patches

Keep in mind though that without the ARC changes there might be higher chance of possible OOM issues
(if your workflow causes it or has caused it in the past)

Cheers !



Commit: https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commit/729895d94074b9be9a90f1a43367b2544fc05101

Branch: https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commits/ZFS-for-SysRescCD-4.3.0-r2-testing02

lots of the additional patches since testing01 have been merged upstream meanwhile,

both the modules for the 3.14* and 3.10* kernel have been rebuilt

https://github.com/zfsonlinux/zfs/pull/2129 is left out this time (lower write & read rates, higher probability of OOM issues if your workflow or setup is prone to trigger it)

besides that it should (hopefully) work without issues

only compile-tested for now - haven't had any time to test them yet (just came home)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2772
Location: Pittsburgh, PA, USA

PostPosted: Tue Nov 25, 2014 1:17 am    Post subject: Reply with quote

Sorry. I was out of the country last week so no access to the amd server. It looks like from the git hub feed that you have created amd modules. I'll look at these when I get a chance..
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1882
Location: Bardowick, Germany

PostPosted: Tue Dec 08, 2015 9:28 am    Post subject: Reply with quote

Do you have any plans to update to zfs-0.6.5 ?
_________________
systemd - The biggest fallacies
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Dec 19, 2015 12:26 am    Post subject: Reply with quote

@Yamakuzure:

sorry for the delay,

sure - it's WIP right now ,

let's see if all works well with the kernel compilation and other stuff


It would be much help to know if

http://sourceforge.net/p/systemrescuecd/code/ci/master/tree/kernelcfg/config-3.14.12-100.fc19.x86_64

and

http://sourceforge.net/p/systemrescuecd/code/ci/master/tree/kernelcfg/config-4.1.8-100.fc21.x86_64

are the actual used kernel configs of the 4.6.1 release - otherwise I might have to boot into 4.6.1 just to be really sure :roll:


edit:

appears like that's unavoidable - important pieces of the config are missing from the sysresccd homepage ?

temporarily posting stuff here since Gmail (used as a personal pastebin and other cache) crashes with an too old webkit-gtk version + midori which is being used in SysRescCD 4.6.1

http://pastebin.com/MN6hfSzS config-4.1.12-alt461-amd64

http://pastebin.com/cYEvTRAq config-3.14.56-std461-amd64


edit2:

Could you please post the output of

Code:
gcc-config -l




I'm getting the following error right now during kernel compilation due to hardened toolchain + kernel config mismatch and/or errors

Code:
Documentation/vDSO/parse_vdso.o: In function `elf_hash':
parse_vdso.c:(.text+0xa4): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_init_from_sysinfo_ehdr':
parse_vdso.c:(.text+0x445): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_match_version':
parse_vdso.c:(.text+0x545): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_sym':
parse_vdso.c:(.text+0x6f2): undefined reference to `__stack_chk_fail'
Documentation/vDSO/parse_vdso.o: In function `vdso_init_from_auxv':
parse_vdso.c:(.text+0x7ad): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
scripts/Makefile.host:100: recipe for target 'Documentation/vDSO/vdso_standalone_test_x86' failed
make[2]: *** [Documentation/vDSO/vdso_standalone_test_x86] Error 1
scripts/Makefile.build:403: recipe for target 'Documentation/vDSO' failed
make[1]: *** [Documentation/vDSO] Error 2
make[1]: *** Waiting for unfinished jobs....
  HOSTLD  Documentation/mic/mpssd/mpssd
Makefile:926: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 2


so I want to be sure to what profile for GCC to switch to

my current output is


Code:
 [11] x86_64-pc-linux-gnu-4.8.4 *
 [12] x86_64-pc-linux-gnu-4.8.4-hardenednopie
 [13] x86_64-pc-linux-gnu-4.8.4-hardenednopiessp
 [14] x86_64-pc-linux-gnu-4.8.4-hardenednossp
 [15] x86_64-pc-linux-gnu-4.8.4-vanilla




Like written on github on the issue-tracker - there exist pre-packaged rescue media by perfinion and Fearedbliss:

Quote:
0.6.5.2-r1 https://perfinion.com/sysresccd/

or 0.6.5.3 https://wiki.gentoo.org/wiki/User:Fearedbliss



I'm currently using the latter, but since there were several issues that got addressed just recently

there's a real incentive to update the ZFS + SPL kernel-modules
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6022
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Dec 19, 2015 4:35 pm    Post subject: Reply with quote

v12 release is out

of the updated modules (for 4.6.1)



https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/commit/3a4a89d4ba0fa32488be10fdccce96d0792d736e


Changelog wrote:

upgrade to SystemRescueCD 4.6.1
SPL & ZFS upgrade to v0.6.5.3+ (ZFS 18.12.2015, SPL 18.12.2015)
Containing significant performance and stability improvements
+ regression fixes and additional commits for enhanced maintenance

sys-fs/zfs* branch upstream state @76d5bf1
+ Illumos 4950, 6292, 6268, 5745 and zfs_object_mutex_size option
sys-kernel/spl branch upstream state @b4ad50a
(both December 18th 2015)

kernel and modules built with x86_64-pc-linux-gnu-4.8.4-vanilla
CFLAGS and CXXFLAGS set to -march=x86-64 -mtune=generic
binutils set to x86_64-pc-linux-gnu-2.23.2 and ld.bfd

added information on kernel build to README, updated the kernel config
in the kernel branches to make kernel build less error-prone

updated README, CHECKSUMS and CHANGES file.



lightly tested , and it worked for me :)

As always: these modules come with a disclaimer (https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/blob/ZFS-for-SysRescCD-4.6.1/DISCLAIMER)


May these modules help you in your backup and rescue efforts to be fruitful ! :D

Enjoy 8)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.6.1
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1882
Location: Bardowick, Germany

PostPosted: Mon Dec 28, 2015 9:27 am    Post subject: Reply with quote

Sorry for the delay, too!

They hid the kernel information in the ChangeLog. Grew me some grey hair to find the information, too.
http://www.sysresccd.org/Changes-x86 wrote:

4.6.1) 2015-10-29:
Standard kernels: Long-Term-Supported linux-3.14.56 (rescue32 + rescue64)
Alternative kernels: Long-Term-Supported linux-4.1.12 (altker32 + altker64)
Updated filesystem tools: GParted-0.24.0, btrfs-progs-4.2.2
I could find no other place to get information on the alternative kernel version. :roll:

I will try out your new version this week and report back how it works for me. I have three zpools with all modern features enabled, so if they import and mount nicely, all should be well, right? ;)

Thank you for your hard work!
_________________
systemd - The biggest fallacies
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1882
Location: Bardowick, Germany

PostPosted: Tue Dec 29, 2015 7:09 am    Post subject: Reply with quote

I could import and use all my pools from sysresccd with your zfs enhancements. Everything worked and looked solid.

Great work! Thank you very much!
_________________
systemd - The biggest fallacies
Back to top
View user's profile Send private message
FearedBliss
n00b
n00b


Joined: 13 Jul 2012
Posts: 4
Location: New York

PostPosted: Sun Apr 17, 2016 12:28 pm    Post subject: Reply with quote

Hey kerneloftruth,

great work keeping things updated :). I still have the scripts on my github though and I've always created new cds. Below are the locations for all of that:

latest iso atm (4.7.0 w/ 0.6.5.6): http://xyinn.org/gentoo/livecd/
archive of older isos: http://xyinn.org/gentoo/archive/livecd/
sysresccd creation scripts: https://github.com/fearedbliss/sysresccd-moddat
bliss-initramfs: https://github.com/fearedbliss/bliss-initramfs

(initramfs does support the ability to include manual pages - you have to switch a flag in pkg/hooks/Zfs.py)

- Jonathan
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum