Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ck-sources : All about choice !
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Tue Mar 05, 2013 5:52 pm    Post subject: Reply with quote

thunderrd wrote:
I think your plan is good, test it for a week or so more, then release a 3.8* to everyone

Hey... we do not get the benefice of stable keywords so... new users will automatically install the last version... hence... I must take greater care.
thunderrd wrote:
TBH, I skipped over the 3.7.x versions on my boxen... [I would think that many others did, as well.]

:) TBH... I would have skipped the 3.7 branch completely and actually waited for 3.7.9 before bumping... with a severe ewarn...
thunderrd wrote:
waiting for the next major update. ...So I will be going directly from 3.6.11 to 3.8.x

Then... you... might well wait for, at least, the 3.10 branch ! 8O

Look... the CONFIG_PREEMPT_RT guys do not look that much interested in 3.7 / 3.8 as well and what I read from Greg-KH does not let me understand that he finds the 3.7 / 3.8 and 3.9 motivating enough for basing a long term support on them... :wink:
_________________
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Tue Mar 05, 2013 8:19 pm    Post subject: Reply with quote

I have had that accounting bug for a long time, it's still present in with 3.8.2 + bfs428
Code:

13935 root       7   0 19416 1388 1012 R 9999  0.0  5124095h top                                                             

but otherwise it's running great.
_________________
http://wojia.be
Back to top
View user's profile Send private message
thunderrd
n00b
n00b


Joined: 20 Aug 2010
Posts: 35

PostPosted: Wed Mar 06, 2013 7:39 am    Post subject: Reply with quote

aCOSwt wrote:

Then... you... might well wait for, at least, the 3.10 branch ! 8O


Don't want to do that. Remember, 3.6.11 is still on bfs-.426

That's why I think that a 3.8x + bfs-428 is important, as the next step.
http://ck.kolivas.org/patches/3.0/3.8/3.8-ck1/patches/
http://ck.kolivas.org/patches/bfs/3.0/3.8/

I'm running 3.8/427 for testing, will patch to 428 soonish, but would like to see 3.8/428 in portage. Then it isn't a problem to wait for the next major revision, as long as the 428 is in there.
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Wed Mar 06, 2013 12:10 pm    Post subject: Reply with quote

xming wrote:
I have had that accounting bug for a long time, it's still present in with 3.8.2 + bfs428
Code:

13935 root       7   0 19416 1388 1012 R 9999  0.0  5124095h top                                                             

but otherwise it's running great.

This is certainly not the bug that ck corrected with bfs427.

I am not able to reproduce this on my ck-sources-3.6.11-r1 / ck-sources-3.4.34

But It seems from ck-hacking that Alfred Chen is experiencing it too.
_________________
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Sat Mar 09, 2013 9:48 am    Post subject: Reply with quote

@ ulenrich

Just curious about your tests with RCU you posted in some other page of ck's hacking blog.
The one with :
<< CONFIG_RCU_FANOUT=32
CONFIG_RCU_FANOUT_LEAF=4 >>

What value did you assign to CONFIG_NR_CPUS ? and how many cores your machine gets at boot time ?

@ xming

Could you please post the values of RCU related CONFIG* as well as CONFIG_NR_CPUS and how many cores your machine gets at boot time ?
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Sat Mar 09, 2013 2:59 pm    Post subject: Reply with quote

@aCOSwt, first of all I appreciate you closely watching. I am a little frustrated about BFS in this moment: I get these timeaccounting overflows again also - but less often than with linux-3.7
aCOSwt wrote:
Just curious about your tests with RCU you posted in some other page of ck's hacking blog.
The one with :
<< CONFIG_RCU_FANOUT=32
CONFIG_RCU_FANOUT_LEAF=4 >>

What value did you assign to CONFIG_NR_CPUS ? and how many cores your machine gets at boot time ?
This was just cluelessness of me. Experiencing RCU settings having an influence how often BFS time accounting fails I just experimented. I have
CONFIG_NR_CPUS=4
also my system has only two cpus:
Code:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     P8700  @ 2.53GHz
stepping        : 10
microcode       : 0xa0b
...

processor       : 1
...

But, if I set up RCU_BOOST priority I have less of these overflows.

If you diff 3.7-sched-bfs-427.patch 3.8-sched-bfs-428.patch
You will see most of Cons work was about time accounting though ...
:(
_________________
fun2gen2
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Sat Mar 09, 2013 3:22 pm    Post subject: Reply with quote

aCOSwt wrote:

@ xming

Could you please post the values of RCU related CONFIG* as well as CONFIG_NR_CPUS and how many cores your machine gets at boot time ?

Code:

# egrep "RCU|CPUS" .config
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=14
CONFIG_RCU_BOOST_DELAY=440
# CONFIG_RCU_NOCB_CPU is not set
CONFIG_NR_CPUS=2
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y

_________________
http://wojia.be
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Sat Mar 09, 2013 5:23 pm    Post subject: Reply with quote

I also recompiled using:
Code:
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=32
CONFIG_RCU_FANOUT_LEAF=2
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=97
CONFIG_RCU_BOOST_DELAY=720
# CONFIG_RCU_NOCB_CPU is not set
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_NR_CPUS=2
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_PROVE_RCU_DELAY is not set
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=32
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
# CONFIG_RCU_TRACE is not set

To flaten the RCU tree a bit might be no obsticle, because I use systemd without consolekit and much less processes !?
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Sat Mar 09, 2013 8:44 pm    Post subject: Reply with quote

ulenrich wrote:
To flaten the RCU tree a bit might be no obsticle

Do you mean you expect to flatten the RCU tree by lowering CONFIG_RCU_FANOUT_LEAF from 4 to 2 ?
With a greater number of CPUs (>64 in this particular case), things would just go the other way round.

With CONFIG_NR_CPUS=2 and CONFIG_RCU_FANOUT=32, it just cannot be more flat than it is already.
as CONFIG_NR_CPUS < CONFIG_RCU_FANOUT * CONFIG_RCU_FANOUT_LEAF, you get a single level tree.
On my core II duo, I set CONFIG_NR_CPUS=2 / CONFIG_RCU_FANOUT=2 / CONFIG_RCU_FANOUT_LEAF=2 and my tree is not more flat than yours...

@ ulenrich & xming : Do you sometimes experience negative idle times ?
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Sat Mar 09, 2013 9:53 pm    Post subject: Reply with quote

I just used the above configured Bfs Linux-3.8.2 kernel and had the same time accounting bugs. But later on.

[edit] Idle times keep positive numbers - mostly over 90 percantage of "top".
Or what number do you refer to?
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 10:53 am    Post subject: Reply with quote

ulenrich wrote:
Idle times keep positive numbers - mostly over 90 percantage of "top".
Or what number do you refer to?

I was referring to /proc/stat, fourth column of the cpu rows.

I think that xming's problem and yours are not related.
This because I think xming is running tickless while you are not and because the figures he announces are just absurd.
I xming's trouble is the one I suspect, then it should also display negative idle times.

EDIT : @ulenrich : of course, you boot your -ck with threadirqs in the boot command line ! (Full credits to PaulBredburry :wink: )
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 1:00 pm    Post subject: Reply with quote

Is this the same as threadirqs?
Code:
CONFIG_IRQ_FORCED_THREADING=y

I ever had this set.
Quote:
because the figures he announces are just absurd

My figures look exactly the same.
A phenomenon to mentions: These "million" time numbers appear when heavy compiling. The first are
ar
lines of the assembler. But when happened afterwards also other programs have it. It looks that bad for me personal that I misstrust my system then and start a non BFS kernel ...
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 1:29 pm    Post subject: Reply with quote

ulenrich wrote:
Is this the same as threadirqs?
Code:
CONFIG_IRQ_FORCED_THREADING=y

It is related.
But, in order for this config setting to be effective at runtime, you must especially activate it at boot time, appending threadirqs to your boot command line.

Then, you will notice that all irqs default to an RT priority of 50.
Regarding this problem of time accounting, you should make sure that no task (including rcu) gets a higher priority than the rtc0 interrupt.
=> either increase the rtc0 interrupt priority or decrease the priority of others.
_________________


Last edited by aCOSwt on Mon Mar 11, 2013 1:49 pm; edited 2 times in total
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 1:44 pm    Post subject: Reply with quote

@aCOSwt, Uuups. I misunderstood as if "IRQ_FORCE_"
.. bloody little naming differences.
And also yes IRQ priority > RCU priority sounds logical ...

[to be edited with results] ...
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 1:54 pm    Post subject: Reply with quote

ulenrich wrote:
@aCOSwt, Uuups. I misunderstood as if "IRQ_FORCE_"

:oops: I have had as well ! :D
ulenrich wrote:
[to be edited with results] ...

With the results, watch also in /proc/pid/status, the number of non voluntary context switches (last line of the file) for each pid matching the process id of the irqs that matter.
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 2:28 pm    Post subject: Reply with quote

Yeah, this top looks different! As you see I changed RCU_BOOST_PRIORITY to 38 which results "-39" in the output:
Code:
top -b -n 1 -u root -o -PR |grep -e migration -e irq -e rcu -e kthread -e ksoft | cat
    8 root      rt   0       0      0      0 S 0.000 0.000   0:00.00 migration/0
   16 root      rt   0       0      0      0 S 0.000 0.000   0:00.00 migration/1
   25 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/9-acpi
   47 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/40-PCIe PME
   48 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/41-PCIe PME
   49 root     -51   0       0      0      0 S 0.000 0.000   0:00.44 irq/42-ahci
   62 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/8-rtc0
  199 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/20-ehci_hcd
  200 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/18-ehci_hcd
  202 root     -51   0       0      0      0 S 0.000 0.000   0:00.08 irq/21-ohci_hcd
  203 root     -51   0       0      0      0 S 0.000 0.000   0:00.03 irq/19-ohci_hcd
  219 root     -51   0       0      0      0 S 0.000 0.000   0:00.05 irq/23-snd_hda_
  423 root     -51   0       0      0      0 S 0.000 0.000   0:00.00 irq/43-eth0
  428 root     -51   0       0      0      0 S 0.000 0.000   0:00.15 irq/23-b43
  792 root     -51   0       0      0      0 S 0.000 0.000   0:00.19 irq/21-nvidia
   10 root     -39   0       0      0      0 S 0.000 0.000   0:00.00 rcub/0
    9 root      -2   0       0      0      0 S 0.000 0.000   0:00.12 rcuc/0
   14 root      -2   0       0      0      0 S 0.000 0.000   0:00.13 rcuc/1
    2 root       1   0       0      0      0 S 0.000 0.000   0:00.00 kthreadd
    3 root       1   0       0      0      0 S 0.000 0.000   0:00.94 ksoftirqd/0
   11 root       1   0       0      0      0 S 0.000 0.000   0:00.07 rcu_preempt
   12 root       1   0       0      0      0 S 0.000 0.000   0:00.00 rcu_bh
   13 root       1   0       0      0      0 S 0.000 0.000   0:00.00 rcu_sched
   15 root       1   0       0      0      0 S 0.000 0.000   0:01.32 ksoftirqd/1
Did the default handling of CONFIG_IRQ_FORCED_THREADING change between the last kernel major releases (I mean you hadn't to specify the boot parameter?) ? I ask because I once hadn't had this time accounting overflow using BFS ...

With the "voluntary" watching you mean the pids of which have time accounting overflows?
(Which I don't have yet - might come along after some hours ...)
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 3:33 pm    Post subject: Reply with quote

ulenrich wrote:
Did the default handling of CONFIG_IRQ_FORCED_THREADING change between the last kernel major releases

No.
ulenrich wrote:
I ask because I once hadn't had this time accounting overflow using BFS

If the changes you are doing have any impact on what you report about time accounting problems then these time accounting problems are more system's load dependent than kernel version dependent.
ulenrich wrote:
With the "voluntary" watching you mean the pids of which have time accounting overflows?

No, I meant particularly the pid of rtc0.

BTW, of course with two cores, you do not "play" with CONFIG_CGROUPS and cpu affinity, do you ?
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 4:40 pm    Post subject: Reply with quote

I am unsure what cgroups would I need using "Systemd", is it different to the namespaces needed by Systemd?
Code:
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
# CONFIG_CGROUP_HUGETLB is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_NETPRIO_CGROUP is not set


And here my irq8/-rtc0 status:
voluntary_ctxt_switches: 3
nonvoluntary_ctxt_switches: 0
_________________
fun2gen2
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 6:06 pm    Post subject: Reply with quote

I have just seen it again
inducing heavy duty into my system (compile kernel + kdelib)
as ever the assembler - see the second "===" line:

= == == 5.1 0:07 RN+ emerge @ 1465s
4.7 0:00 RN+ cc1
4.5 0:00 RN+ cc1
3.4 7:56 Ss+ X
1.9 4:31 Sl kwin
1.0 0:32 Sl firefox
0.6 1:25 Rl yakuake
0.5 0:09 SN+ emerge
= == == 207725234 21114581:29 SN+ as @ 1470s
5.0 0:07 RN+ emerge
4.5 0:00 RN+ cc1
3.5 0:00 RN+ cc1
3.4 7:57 Ss+ X
1.9 4:31 Sl kwin
1.0 0:32 Sl firefox

the status of irq/8-rtc didn't change (see above)!
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 6:56 pm    Post subject: Reply with quote

please post the output of
Code:
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 7:01 pm    Post subject: Reply with quote

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

These above time accounting overflows where then a a thread ksoftirq

I compiled a new BFS kernel having RCU default boost proiority, which I run now.
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Mon Mar 11, 2013 7:18 pm    Post subject: Reply with quote

ulenrich wrote:
These above time accounting overflows where then a a thread ksoftirq

With 2 cores, you should always get 2 ksoftirq daemons.
They are more or less consuming cpu depending on the load of the system because the more load => the more softirqs serviced < softirqs triggered.
Well this is simply confirming your system is under heavy load.
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Mon Mar 11, 2013 7:45 pm    Post subject: Reply with quote

There are two ksoftirqd/0 or 1
I saw the time overflow again at the "as" assembler compile
After compiling - without heavy load - the time accounting overflow switched to one ksoftirqd/n .
I can say it, because I sorted top output by TIME+ sum. The other ksoftirqd entry therefore I didn't see.

I now try without any special RCU boost priority.
But with BFS and boot parameter threadirqs - I will see what happens ...

[edit] Result: without RCU boost priority set high the overflow appears early without any heavy duty jobs ...
_________________
fun2gen2
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Tue Mar 12, 2013 8:52 am    Post subject: Reply with quote

ulenrich wrote:
without RCU boost priority set high the overflow appears early without any heavy duty jobs ...

OK, I just cannot manage to reproduce on a 2 cores. (3.4.34 / 3.6.11-r1) kwin + make -j2 + 4 SCHED_FIFO threads at a higher priority than the one given to the rcu priority boosting.
But I do not use top.

Considering your remark about the ksoftirq daemons, CONFIG_IRQ_TIME_ACCOUNTING is set with the BFS but is not a default setting under a stock Linux kernel.
When you make your non-BFS kernel. Do you specifically set it ? If not then please test under a non-BFS kernel + CONFIG_IRQ_TIME_ACCOUNTING.
( General setup -> CPU/Task time and stats accounting -> Cputime accounting )
_________________
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1136

PostPosted: Tue Mar 12, 2013 10:15 pm    Post subject: Reply with quote

aCOSwt wrote:
Considering your remark about the ksoftirq daemons, CONFIG_IRQ_TIME_ACCOUNTING is set with the BFS but is not a default setting under a stock Linux kernel.
... please test under a non-BFS kernel + CONFIG_IRQ_TIME_ACCOUNTING
What bugs me here:
normally you choose, but the BFS config patch does select both!
Code:
grep ACCOUNTING config-3.8.2-1087.cfs config-3.8.2-1091.bfs
config-3.8.2-1087.cfs:# CONFIG_TICK_CPU_ACCOUNTING is not set
config-3.8.2-1087.cfs:CONFIG_IRQ_TIME_ACCOUNTING=y
config-3.8.2-1087.cfs:CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
config-3.8.2-1091.bfs:CONFIG_TICK_CPU_ACCOUNTING=y
config-3.8.2-1091.bfs:CONFIG_IRQ_TIME_ACCOUNTING=y
config-3.8.2-1091.bfs:CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y


Today I tried for hours to provoke the time accounting overflow using an CFS kernel with CONFIG_IRQ_TIME_ACCOUNTING=y.
Not possible to provoke the overflow without BFS!

Also I had the idea to try the BFS kernel with CONFIG_NO_HZ
as most users do. But no luck, no difference.

@all, who can reproduce these time accounting overflows?
You can use this script "mytop"
Code:
#!/bin/bash
sleepy=3
[ -n "$1" ] && [ "${1%%[1-9]*}" == "" ] && sleepy=$1
export p=1
[ -n "$2" ] && [ "${2%%[0-9]*}" == "" ] && p=$2
echo "linux-$(uname -r -m -p)"
echo "Run  Sleep  Duninterruptible  Tstoped  Xdead  Zombie"
echo "<notNice  Nlow  LioWait  sLead  +foreground  lmultiThr"
echo -e "%CPU   TIME STAT COMMAND"
     ps -e -o pcpu,bsdtime,stat,comm --no-heading --sort -pcpu \
      |head -n $p
while sleep $sleepy ; do
     ps -e -o pcpu,bsdtime,stat,comm --no-heading --sort -pcpu \
      |head -n $p
done

_________________
fun2gen2
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9  Next
Page 3 of 9

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum