Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
New kernel (hardened-sources) causes looping reboot?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1490

PostPosted: Sun Aug 19, 2012 3:21 pm    Post subject: New kernel (hardened-sources) causes looping reboot? Reply with quote

Hello All

Just recently upgraded my kernel from 3.2.2-hardened-r1 to 3.4.5-hardened. Here is the weird deal, I don't get a panic or issues of not finding the kernel, but as soon as I pass grub, the box reboots - (BIOS to grub) after trying to load the 3.4.5 image. I've never seen this. If I roll back to 3.2.2-hardened-r1 it's all good again.

Grub:
Code:
default 0
timeout 10
splashimage=(hd0,0)/grub/splash.xpm.gz

title=bzImage-3.4.5-hardened (sda)
root (hd0,0)
kernel (hd0,0)/bzImage-3.4.5-hardened ro root=/dev/md1

title=bzImage-3.4.5-hardened (sdb)
root (hd1,0)
kernel (hd1,0)/bzImage-3.4.5-hardened ro root=/dev/md1


title=bzImage-3.2.2-hardened-r1 (sda)
root (hd0,0)
kernel (hd0,0)/bzImage-3.2.2-hardened-r1 ro root=/dev/md1

title=bzImage-3.2.2-hardened-r1 (sdb)
root (hd1,0)
kernel (hd1,0)/bzImage-3.2.2-hardened-r1 ro root=/dev/md1


Here is what's in /boot
Code:
-rw-r--r--  1 root root 2174832 Apr  7 19:39 bzImage-3.2.2-hardened-r1
-rw-r--r--  1 root root 2201968 Aug 18 23:18 bzImage-3.4.5-hardened


I verified the image... is an image:
Code:
file bzImage-3.4.5-hardened
bzImage-3.4.5-hardened: Linux kernel x86 boot executable bzImage, version 3.4.5-hardened (root@comp) #1 SMP Sat Aug 18 23:01:57 MDT 2, RO-rootFS, swap_dev 0x2, Normal VGA


Here is my /usr
Code:
lrwxrwxrwx  1 root root    21 Aug 18 22:48 linux -> linux-3.4.5-hardened/
drwxr-xr-x 25 root root  1640 Apr  7 19:38 linux-3.2.2-hardened-r1
drwxr-xr-x 25 root root  1640 Aug 18 23:17 linux-3.4.5-hardened


Here is my filesystem (thought /usr was full or something.. but it looks good)
Code:
rootfs          1.5G  257M  1.3G  18% /
/dev/root       1.5G  257M  1.3G  18% /
tmpfs           264M  238k  263M   1% /run
rc-svcdir       1.1M  123k  926k  12% /lib/rc/init.d
udev             11M  4.1k   11M   1% /dev
shm             264M     0  264M   0% /dev/shm
/dev/md2        502M   34M  468M   7% /tmp
/dev/md3        502M   35M  468M   7% /home
/dev/md4         13G  5.4G  6.7G  45% /usr
/dev/md5        235G  213G   23G  91% /var
/dev/md0         96M   47M   45M  52% /boot


Here is my emerge --info output:
Code:
Portage 2.1.11.9 (hardened/linux/x86, gcc-4.5.3, glibc-2.15-r2, 3.2.2-hardened-r1 i686)
=================================================================
System uname: Linux-3.2.2-hardened-r1-i686-Intel-R-_Pentium-R-_D_CPU_3.00GHz-with-gentoo-2.1
Timestamp of tree: Sun, 19 Aug 2012 02:30:01 +0000
app-shells/bash:          4.2_p37
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.5.4-r4, 2.6.8, 2.7.3-r2, 3.1.5, 3.2.3
dev-util/cmake:           2.8.8-r3
dev-util/pkgconfig:       0.27
sys-apps/baselayout:      2.1-r1
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13::<unknown repository>, 2.68
sys-devel/automake:       1.4_p6::<unknown repository>, 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1::<unknown repository>, 1.8.5-r3::<unknown repository>, 1.9.6-r2::<unknown repository>, 1.10.3, 1.11.6
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            3.4.6-r2::<unknown repository>, 4.1.2::<unknown repository>, 4.3.4, 4.4.5, 4.5.3-r2
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.15-r2
Repositories: gentoo
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--autounmask=n"
FCFLAGS="-march=i686 -O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-march=i686 -O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acl acpi acpi4linux apache2 berkdb bzip2 clamav cli cracklib crypt cups cxx dri foomaticdb gd gdbm hardened iconv libclamav modules mudflap mysql ncurses nptl nptlonly oav openmp pam pax_kernel pcre pic ppds pppd pwdb readline samba session ssl tcpd unicode urandom x86 zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON


Any ideas?

Thanks!
hanji
_________________
Server Admin Blog - Uno-Code.com


Last edited by hanj on Sun Aug 19, 2012 7:31 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W

PostPosted: Sun Aug 19, 2012 3:53 pm    Post subject: Reply with quote

hanj,

Check the CPU type you built the kernel for. Early reboot loop problems are often caused by illegal instruction exceptions before the kernel has the exception handler running.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1490

PostPosted: Sun Aug 19, 2012 4:00 pm    Post subject: Reply with quote

NeddySeagoon wrote:
hanj,

Check the CPU type you built the kernel for. Early reboot loop problems are often caused by illegal instruction exceptions before the kernel has the exception handler running.


Thanks for the reply!

This is the processor:
Code:
cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Pentium(R) D CPU 3.00GHz
stepping        : 7
microcode       : 0x3
cpu MHz         : 2992.533
cache size      : 1024 KB


I have the following set in the kernel config:
Code:
Pentium-4/Celeron(P4-based)/Pentium-4 M/older Xeon


Code:
Symbol: X86_MINIMUM_CPU_FAMILY [=5]


This is the same config (after make oldconfig) as 3.2.2-hardened-r1

Thanks!
hanji
_________________
Server Admin Blog - Uno-Code.com
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1490

PostPosted: Sun Aug 19, 2012 4:20 pm    Post subject: Reply with quote

I diff'd the two configs.. maybe this might shed some light. Spidey sense is tinglin on PAX/GRSEC stuff (but that may be way after the problem). Left is old config, right is new config.

Code:
3c3
< # Linux/i386 3.2.2-hardened-r1 Kernel Configuration
---
> # Linux/i386 3.4.5-hardened Kernel Configuration
20d19
< CONFIG_ZONE_DMA=y
24d22
< CONFIG_GENERIC_IOMAP=y
35a34
> CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
42d40
< CONFIG_ARCH_POPULATES_NODE_MAP=y
88d85
< CONFIG_HAVE_SPARSE_IRQ=y
100d96
< # CONFIG_RCU_TRACE is not set
159a156
> CONFIG_OPROFILE_NMI_TIMER=y
177a175,176
> CONFIG_HAVE_CMPXCHG_LOCAL=y
> CONFIG_HAVE_CMPXCHG_DOUBLE=y
198a198,203
> # Partition Types
> #
> # CONFIG_PARTITION_ADVANCED is not set
> CONFIG_MSDOS_PARTITION=y
>
> #
214d218
< CONFIG_INLINE_SPIN_UNLOCK=y
241a246
> CONFIG_ZONE_DMA=y
287,289c292
< CONFIG_CMPXCHG_LOCAL=y
< CONFIG_CMPXCHG_DOUBLE=y
< CONFIG_X86_L1_CACHE_SHIFT=7
---
> CONFIG_X86_L1_CACHE_SHIFT=5
290a294
> # CONFIG_X86_PPRO_FENCE is not set
296d299
< CONFIG_X86_INTEL_USERCOPY=y
304d306
< CONFIG_CPU_SUP_CYRIX_32=y
308d309
< CONFIG_CPU_SUP_UMC_32=y
335a337,338
> # CONFIG_NOHIGHMEM is not set
> # CONFIG_HIGHMEM4G is not set
346a350,351
> CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
> CONFIG_ARCH_DISCARD_MEMBLOCK=y
429a435
> # CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
441a448,449
> # CONFIG_NET5501 is not set
> # CONFIG_GEOS is not set
450a459
> CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
462a472
> # CONFIG_UNIX_DIAG is not set
485a496
> # CONFIG_INET_UDP_DIAG is not set
499a511
> # CONFIG_NETFILTER_NETLINK_ACCT is not set
503a516
> CONFIG_NF_CONNTRACK_PROCFS=y
504a518
> # CONFIG_NF_CONNTRACK_TIMEOUT is not set
516a531
> # CONFIG_NF_CT_NETLINK_TIMEOUT is not set
534a550
> CONFIG_NETFILTER_XT_TARGET_LOG=y
557a574
> CONFIG_NETFILTER_XT_MATCH_ECN=y
568a586
> # CONFIG_NETFILTER_XT_MATCH_NFACCT is not set
595a614
> # CONFIG_IP_NF_MATCH_RPFILTER is not set
599d617
< CONFIG_IP_NF_TARGET_LOG=y
632a651
> # CONFIG_OPENVSWITCH is not set
635a655
> CONFIG_BQL=y
671a692
> # CONFIG_GENERIC_CPU_DEVICES is not set
690a712
> # CONFIG_BLK_DEV_PCIESSD_MTIP32XX is not set
702a725
> # CONFIG_BLK_DEV_NVME is not set
711a735,738
>
> #
> # Misc devices
> #
713d739
< CONFIG_MISC_DEVICES=y
802a829
> # CONFIG_SCSI_UFSHCD is not set
814a842
> # CONFIG_SCSI_ISCI is not set
994a1023
> # CONFIG_NET_CALXEDA_XGMAC is not set
1108a1138
> # CONFIG_AMD_PHY is not set
1149d1178
< # CONFIG_PHONE is not set
1179a1209
> # CONFIG_KEYBOARD_OMAP4 is not set
1197a1228
> # CONFIG_MOUSE_SYNAPTICS_USB is not set
1234a1266,1270
> # KCopy
> #
> CONFIG_KCOPY=m
>
> #
1263a1300
> # CONFIG_TCG_TPM is not set
1266a1304
> # CONFIG_HSI is not set
1354a1393
> # CONFIG_EXYNOS_VIDEO is not set
1358,1362d1396
< # Display device support
< #
< # CONFIG_DISPLAY_SUPPORT is not set
<
< #
1419d1452
< # CONFIG_HID_QUANTA is not set
1420a1454
> # CONFIG_HID_SAITEK is not set
1426a1461
> # CONFIG_HID_TIVO is not set
1431,1433d1465
< CONFIG_USB_SUPPORT=y
< CONFIG_USB_COMMON=m
< CONFIG_USB_ARCH_HAS_HCD=y
1436a1469,1471
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_COMMON=m
> CONFIG_USB_ARCH_HAS_HCD=y
1447d1481
< # CONFIG_USB_DWC3 is not set
1596a1631,1634
>
> #
> # Microsoft Hyper-V guest support
> #
1601a1640
> # CONFIG_APPLE_GMUX is not set
1610a1650,1657
>
> #
> # Remoteproc drivers (EXPERIMENTAL)
> #
>
> #
> # Rpmsg drivers (EXPERIMENTAL)
> #
1628a1676
> CONFIG_DCACHE_WORD_ACCESS=y
1703a1752
> # CONFIG_QNX6FS_FS is not set
1714,1719d1762
<
< #
< # Partition Types
< #
< # CONFIG_PARTITION_ADVANCED is not set
< CONFIG_MSDOS_PARTITION=y
1784d1826
< # CONFIG_SYSCTL_SYSCALL_CHECK is not set
1827a1870
> CONFIG_ARCH_TRACK_EXEC_LIMIT=y
1829,1835c1872,1873
< # CONFIG_GRKERNSEC_LOW is not set
< # CONFIG_GRKERNSEC_MEDIUM is not set
< # CONFIG_GRKERNSEC_HIGH is not set
< # CONFIG_GRKERNSEC_HARDENED_SERVER is not set
< # CONFIG_GRKERNSEC_HARDENED_WORKSTATION is not set
< # CONFIG_GRKERNSEC_HARDENED_VIRTUALIZATION is not set
< CONFIG_GRKERNSEC_CUSTOM=y
---
> # CONFIG_GRKERNSEC_CONFIG_AUTO is not set
> CONFIG_GRKERNSEC_CONFIG_CUSTOM=y
1838c1876,1927
< # Address Space Protection
---
> # Customize Configuration
> #
>
> #
> # PaX
> #
> CONFIG_PAX=y
>
> #
> # PaX Control
> #
> # CONFIG_PAX_SOFTMODE is not set
> CONFIG_PAX_EI_PAX=y
> CONFIG_PAX_PT_PAX_FLAGS=y
> # CONFIG_PAX_XATTR_PAX_FLAGS is not set
> # CONFIG_PAX_NO_ACL_FLAGS is not set
> CONFIG_PAX_HAVE_ACL_FLAGS=y
> # CONFIG_PAX_HOOK_ACL_FLAGS is not set
>
> #
> # Non-executable pages
> #
> CONFIG_PAX_NOEXEC=y
> CONFIG_PAX_PAGEEXEC=y
> CONFIG_PAX_SEGMEXEC=y
> CONFIG_PAX_EMUTRAMP=y
> CONFIG_PAX_MPROTECT=y
> # CONFIG_PAX_MPROTECT_COMPAT is not set
> # CONFIG_PAX_ELFRELOCS is not set
> # CONFIG_PAX_KERNEXEC is not set
> CONFIG_PAX_KERNEXEC_PLUGIN_METHOD=""
>
> #
> # Address Space Layout Randomization
> #
> CONFIG_PAX_ASLR=y
> CONFIG_PAX_RANDKSTACK=y
> CONFIG_PAX_RANDUSTACK=y
> CONFIG_PAX_RANDMMAP=y
>
> #
> # Miscellaneous hardening features
> #
> # CONFIG_PAX_MEMORY_SANITIZE is not set
> # CONFIG_PAX_MEMORY_STACKLEAK is not set
> # CONFIG_PAX_MEMORY_UDEREF is not set
> # CONFIG_PAX_REFCOUNT is not set
> # CONFIG_PAX_USERCOPY is not set
> # CONFIG_PAX_SIZE_OVERFLOW is not set
>
> #
> # Memory Protections
1862d1950
< # CONFIG_GRKERNSEC_PROC_USERGROUP is not set
1864a1953
> # CONFIG_GRKERNSEC_SYMLINKOWN is not set
1917c2006
< # Sysctl support
---
> # Sysctl Support
1926,1972d2014
<
< #
< # PaX
< #
< CONFIG_ARCH_TRACK_EXEC_LIMIT=y
< CONFIG_PAX_ENABLE_PAE=y
< CONFIG_PAX=y
<
< #
< # PaX Control
< #
< # CONFIG_PAX_SOFTMODE is not set
< CONFIG_PAX_EI_PAX=y
< # CONFIG_PAX_PT_PAX_FLAGS is not set
< # CONFIG_PAX_NO_ACL_FLAGS is not set
< CONFIG_PAX_HAVE_ACL_FLAGS=y
< # CONFIG_PAX_HOOK_ACL_FLAGS is not set
<
< #
< # Non-executable pages
< #
< CONFIG_PAX_NOEXEC=y
< CONFIG_PAX_PAGEEXEC=y
< CONFIG_PAX_SEGMEXEC=y
< CONFIG_PAX_EMUTRAMP=y
< CONFIG_PAX_MPROTECT=y
< # CONFIG_PAX_MPROTECT_COMPAT is not set
< # CONFIG_PAX_ELFRELOCS is not set
< # CONFIG_PAX_KERNEXEC is not set
< CONFIG_PAX_KERNEXEC_PLUGIN_METHOD=""
<
< #
< # Address Space Layout Randomization
< #
< CONFIG_PAX_ASLR=y
< CONFIG_PAX_RANDKSTACK=y
< CONFIG_PAX_RANDUSTACK=y
< CONFIG_PAX_RANDMMAP=y
<
< #
< # Miscellaneous hardening features
< #
< # CONFIG_PAX_MEMORY_SANITIZE is not set
< # CONFIG_PAX_MEMORY_STACKLEAK is not set
< # CONFIG_PAX_MEMORY_UDEREF is not set
< # CONFIG_PAX_REFCOUNT is not set
< # CONFIG_PAX_USERCOPY is not set
2022a2065
> # CONFIG_CRYPTO_LRW is not set
2023a2067
> # CONFIG_CRYPTO_XTS is not set
2065a2110
> # CONFIG_CRYPTO_SERPENT_SSE2_586 is not set
2095a2141,2143
> CONFIG_GENERIC_PCI_IOMAP=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_IO=y
2100a2149,2153
> # CONFIG_CRC32_SELFTEST is not set
> CONFIG_CRC32_SLICEBY8=y
> # CONFIG_CRC32_SLICEBY4 is not set
> # CONFIG_CRC32_SARWATE is not set
> # CONFIG_CRC32_BIT is not set
2127a2181
> CONFIG_DQL=y

_________________
Server Admin Blog - Uno-Code.com
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1490

PostPosted: Sun Aug 19, 2012 7:14 pm    Post subject: Reply with quote

When I build the kernel without PAX/GRSEC it gets worse...

Code:
Decompressing Linux... Parsing ELF... done.
Booting the kernel.


Returning back to the original config but removing APM/ACPI stuff and adding some other x86 options.

This is very puzzling!!
hanji

[UPDATE] removing APM/ACPI made no difference. I compared /proc/config.gz with the working (old) config to make sure that what I thought was a good config was actually good. diff showed that they were the same.
_________________
Server Admin Blog - Uno-Code.com
Back to top
View user's profile Send private message
CMoH
n00b
n00b


Joined: 08 Sep 2004
Posts: 27
Location: Timisoara, Romania

PostPosted: Sun Aug 26, 2012 9:05 pm    Post subject: Reply with quote

I am experiencing a different issue. I have gentoo hardened on a headless server. Since moving to hardened-sources-3.4.5 the machine simply powers down from time to time. There is nothing in the log files, and since it powers down there's no system console to look after this event.

For now I'm reverting to 3.4.2-r1 which worked just fine.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum