| View previous topic :: View next topic |
| Author |
Message |
stahlsau
Guru
Joined: 09 Jan 2004
Posts: 584
Location: WildWestwoods
|
 Posted: Wed Aug 16, 2006 7:28 am Post subject: |
 |
|
| Quote: | Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc
|
It works, and i see no problem with those two enabled. Imho there's no way they could interfere. |
|
| Back to top |
|
 |
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Thu Aug 17, 2006 8:21 pm Post subject: |
|
|
I just installed your script. Good work.
I have a suggestion that would speed thing up even more.
As I saw the files in /etc/conf.d/sofile-list.load are alphabeticaly sorted. As I understand thigs, it would be better to just live files unsorted. I mean, they should stay in the same order as they get loaded.
If a file is near the end of the list it gets loaded twice!
first time when it is realy needed (but not cached yet) and the second time when it wants to be cached, but caching it came too late.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Thu Aug 17, 2006 8:31 pm Post subject: |
|
|
| sorting is required to take out the duplicates. Moreover, its very difficult to tell the order of loading of files from 'lsof' output. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Thu Aug 17, 2006 8:54 pm Post subject: |
|
|
I agree that all entries in the list must be unique.
I think that sorting (uniqueing) algoritm must be different.
The easiest to implement is to check for every entry that should to be added if it is already in the table and add it only if it is not.
I did't program much in bash so I can't help much with this.
I think that even if sorting algoritm is slow it runs only once. But every next time you get better improvement at startup.
Maybe I'll dig into man pages to implement this throwing out non unique entries (not realy sorting).
---EDIT
My digging and surfing results:
| Code: | | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' db.lst |
this takes only non unique lines from file db.lst without sorting them.
If data from every second was cleaned with this and on the end the final file too, the result would be much faster boot. I guess
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Fri Aug 18, 2006 1:35 am Post subject: |
|
|
So here it is. I rewrote parts of the script (I hope the author alows it (no GPL statements there)) 
It makes entries in the loading list unique, but it doesent sort them, so the first needed are preloaded (cached) first, so they can be used imediately from cache.
I changed the file /usr/sbin/sample-init-process
| Code: |
#!/bin/bash
final_db="$1"
# total number of lists to keep in /etc/conf.d/
total_to_keep=4
# total number of samples to make in /forcesampler boot.
# It will take twice this amount of seconds to finish sampling.
total_samples_per_boot=150
# touch empty file if not there
[ ! -f "$final_db" ] && touch "$final_db"
let i=0
while [ "$i" -ne "$total_to_keep" ] ; do
if [ ! -f "$final_db$i" ] ; then
break;
else
j=$((i+1))
if [ "$j" = "$total_to_keep" ] ;then
j=0
fi
if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then
i=$((j))
break;
fi
fi
i=$((i+1))
done
if [ "$i" = "$total_to_keep" ] ;then
i=0
fi
slot="$i"
cp "$final_db" "$final_db$i"
function collect_sample()
{
lsof |awk '{print $9}'|grep \.so| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' > $1
lsof |awk '{print $9}'|grep \.conf| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' >> $1
lsof |awk '{print $9}'|grep /bin/| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' >> $1
lsof |awk '{print $9}'|grep /sbin/| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' >> $1
lsof |awk '{print $9}'|grep ttf| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' >> $1
lsof |awk '{print $9}'|grep /libexec| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' >> $1
}
collect_sample /tmp/init_sample1
i=0
while [ "$i" -ne "$total_samples_per_boot" ] ; do
collect_sample /tmp/init_sample2
cat /tmp/init_sample1 /tmp/init_sample2 | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' > /tmp/init_sample3
mv /tmp/init_sample3 /tmp/init_sample1
sleep 1
i=$((i+1))
done
cat /tmp/init_sample1 "$final_db$slot" | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' > "$final_db"
rm -f /tmp/init_sample[123]
|
I generaly changed the | sort | unique to the line from previous post.
The data collecting took much more processor power (sed) and it took a little longer, so I lowered total_samples_per_boot to 150 (I think you can leave it at 200).
RESULT: startup (boot) is much better.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 3:10 am Post subject: |
|
|
| zxy wrote: | | So here it is. I rewrote parts of the script (I hope the author alows it (no GPL statements there)) |
feel free to edit it as you please. One more thing, you might want to consider is the sampling technique in post 4. It does lsof only once per sample and may be faster overall when combined with your new 'uniquer'. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Fri Aug 18, 2006 2:46 pm Post subject: |
|
|
Here are some boot time measurements without caching and with caching (my uniquing).
I also cached gdm background and icons. Because of many items that must be cached I didn't find booting much faster. But when the machine is booted  the poetry begins. Everything almost "just pops up".
On athlon I use Fluxbox and on Pentium M KDE is installed.
| Code: |
Athlon64+SATA2(Raid0) PentiumM+IDE
no-cache cache no-cache cache
------------------------------------------------------------------
boot 34 35 55 75
Open Office 6.7 4.12 14 4.23
Firefox 3.91 2.4 6.99 3.77
|
I'm going to try the new possibility with only 1 lsof.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
AssociateX
Tux's lil' helper
Joined: 19 Feb 2004
Posts: 134
Location: North Dakota
|
| Posted: Fri Aug 18, 2006 2:51 pm Post subject: |
|
|
I got this from here, how about using :
| Code: | | lsof -F | sed "s/^n//g" | grep -v "^c" | egrep "^/bin|^/lib|^/sbin|^/usr" | sort | uniq |
Will it matter that any of the lsof lines in this howto so far have been producing dir's? Will this actually work:
| Code: | | cat /etc > /dev/null |
Also, how long is:
| Code: | | ps -aef|grep sample-init-process |
supposed to show:
| Quote: | | root 9724 8481 0 09:49 pts/1 00:00:00 grep --colour=auto sample-init-process |
I let this run all night and it's still there.
Also, I checked /etc/conf.d/sofile-list.load* and get this, and this is after sample-init-process running all night:
| Quote: | athlon ~ # wc -l /etc/conf.d/sofile-list.load*
30 /etc/conf.d/sofile-list.load
0 /etc/conf.d/sofile-list.load0
276 /etc/conf.d/sofile-list.load1
0 /etc/conf.d/sofile-list.load2
0 /etc/conf.d/sofile-list.load3
|
Of course that doesn't seem right.
Thanks a ton. I had went out and added a gig of ram to the 512mb that I ready had and really want to play with stuff like this.
_________________
AssociateX
Gentoo rocks! |
|
| Back to top |
|
|
lazarusrat
Guru
Joined: 17 Jul 2002
Posts: 305
Location: Lafayette, IN
|
| Posted: Fri Aug 18, 2006 3:21 pm Post subject: |
|
|
| zxy wrote: |
My digging and surfing results:
| Code: | | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' db.lst |
this takes only non unique lines from file db.lst without sorting them.
If data from every second was cleaned with this and on the end the final file too, the result would be much faster boot. I guess |
This sed script has rapidly diminishing speed returns depending on the size of the file. Taking samples from a random log file I had lying around, at 100 lines:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt
real 0m0.056s
user 0m0.054s
sys 0m0.002s
lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt
real 0m0.003s
user 0m0.002s
sys 0m0.002s |
At 200 lines:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt
real 0m0.161s
user 0m0.160s
sys 0m0.001s
lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt
real 0m0.003s
user 0m0.000s
sys 0m0.003s |
And at 1000 lines:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' bar.txt > sed.txt
real 0m7.739s
user 0m7.460s
sys 0m0.280s
lazarus@massacre ~ $ time sort bar.txt | uniq > su.txt
real 0m0.005s
user 0m0.001s
sys 0m0.004s |
Sort and uniq are not exactly slow (and certainly a lot more readable ).
_________________
obpiper: pipe menu generator for openbox
obtheme: pipe menu to switch openbox themes |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 3:44 pm Post subject: |
|
|
| AssociateX wrote: |
Also, how long is:
| Code: | | ps -aef|grep sample-init-process |
supposed to show:
| Quote: | | root 9724 8481 0 09:49 pts/1 00:00:00 grep --colour=auto sample-init-process |
I let this run all night and it's still there.
|
use | Code: | | ps -aef|grep -v grep|grep sample-init-process |
instead. You are just seeing the process which is grepping. The actual sample-init-process is long gone. |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 3:57 pm Post subject: |
|
|
| lazarusrat wrote: |
And at 1000 lines:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' bar.txt > sed.txt
real 0m7.739s
user 0m7.460s
sys 0m0.280s
lazarus@massacre ~ $ time sort bar.txt | uniq > su.txt
real 0m0.005s
user 0m0.001s
sys 0m0.004s |
Sort and uniq are not exactly slow (and certainly a lot more readable ). |
this can be a problem because sample interval should be definitely smaller than 8 seconds. Otherwise you won't catch many useful files and hence the results that you see (wherein cache boot is longer). I think the another way would be to do this 'sedding' only on the final output and/or do 'sedding' in the while loop every 3 or so iterations. Sample collector phase can be slow overall, but not when its taking the samples. |
|
| Back to top |
|
|
lazarusrat
Guru
Joined: 17 Jul 2002
Posts: 305
Location: Lafayette, IN
|
| Posted: Fri Aug 18, 2006 4:11 pm Post subject: |
|
|
| devsk wrote: | | lazarusrat wrote: |
And at 1000 lines:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' bar.txt > sed.txt
real 0m7.739s
user 0m7.460s
sys 0m0.280s
lazarus@massacre ~ $ time sort bar.txt | uniq > su.txt
real 0m0.005s
user 0m0.001s
sys 0m0.004s |
Sort and uniq are not exactly slow (and certainly a lot more readable ). |
this can be a problem because sample interval should be definitely smaller than 8 seconds. Otherwise you won't catch many useful files and hence the results that you see (wherein cache boot is longer). I think the another way would be to do this 'sedding' only on the final output and/or do 'sedding' in the while loop every 3 or so iterations. Sample collector phase can be slow overall, but not when its taking the samples. |
Well, these times should probably be taken with a grain of salt. The tests were done with a text file with massively varying line lengths and contents, probably nothing close to the format of what's actually being uniqued by these scripts. Is it likely that what's being uniqued will have 1000 lines or more?
Edit: Even still, using a different file (ls /usr/lib /lib > foo.txt) with 1300 lines, sort | uniq is faster than the sed script:
| Code: | lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt
real 0m0.807s
user 0m0.802s
sys 0m0.006s
lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt
real 0m0.003s
user 0m0.000s
sys 0m0.005s |
_________________
obpiper: pipe menu generator for openbox
obtheme: pipe menu to switch openbox themes |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 4:22 pm Post subject: |
|
|
| the idea was to not sort at all because he wanted to preserve the order of loading of files to be same as the order that lsof found when sampling. And he has a point. But what do we pay for it? If useful results are lost while uniquing, then no point. |
|
| Back to top |
|
|
AssociateX
Tux's lil' helper
Joined: 19 Feb 2004
Posts: 134
Location: North Dakota
|
| Posted: Fri Aug 18, 2006 4:38 pm Post subject: |
|
|
| devsk wrote: | use | Code: | | ps -aef|grep -v grep|grep sample-init-process |
instead. You are just seeing the process which is grepping. The actual sample-init-process is long gone. |
Ah man.....dang it for not seeing that. Thank you.
_________________
AssociateX
Gentoo rocks! |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Fri Aug 18, 2006 4:49 pm Post subject: |
|
|
I'm just experimenting with uniquing only at the end. I think it will sample much better.
Though the final sed-ing takes time (athlon64 - 7 min and counting)
--- EDIT
sed-ing lasted for 10 mins ... going to reboot now... 
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
lazarusrat
Guru
Joined: 17 Jul 2002
Posts: 305
Location: Lafayette, IN
|
| Posted: Fri Aug 18, 2006 5:08 pm Post subject: |
|
|
Ah, I misunderstood. Here is a little awk script that is much faster than the sed script and does the same thing (stored in file ~/neek for this example):
| Code: | #!/bin/sh
/usr/bin/awk '{
if ($0 in stored_lines)
x=1
else
print
stored_lines[$0]=1
}' $1
|
| Code: | lazarus@massacre ~ $ wc -l foo.txt
41207 foo.txt
lazarus@massacre ~ $ ls -l foo.txt
-rw------- 1 lazarus users 2987383 Aug 18 09:56 foo.txt
lazarus@massacre ~ $ time ~/neek foo.txt > awk.txt
real 0m0.083s
user 0m0.068s
sys 0m0.014s
lazarus@massacre ~ $ |
If I weren't so rusty with awk I would attempt to get that into a single script file that could just be made executable. 
Edit: Just made it a shell script that invokes awk instead of a flat awk script.
_________________
obpiper: pipe menu generator for openbox
obtheme: pipe menu to switch openbox themes |
|
| Back to top |
|
|
AssociateX
Tux's lil' helper
Joined: 19 Feb 2004
Posts: 134
Location: North Dakota
|
| Posted: Fri Aug 18, 2006 5:23 pm Post subject: |
|
|
OK, I have this now:
| Code: | athlon ~ # wc -l /etc/conf.d/sofile-list.load
554 /etc/conf.d/sofile-list.load |
Yippy!
Although my system stops during boot at:
| Quote: | | Starting syslog-ng ... |
But it continues if I hit the space-bar. That is minor but would be nice to know whats causing it.
Also, is it true that I need to open the apps that I want to be cached (if that's the term) before this gets done:
| Code: | | while true; do ps -aef|grep -v grep|grep sample-init-process; sleep 1; echo done; done |
Or is this supposed to update at some point else also?
Running this with the 554 entries:
| Code: | | cat `cat /etc/conf.d/sofile-list.load` > /dev/null |
gives back:
| Quote: | cat: /home/me/.config/menus: Is a directory
cat: /home/me/.kde/share/apps/kconf_update: Is a directory
cat: /tmp/gconfd-me/lock/0t1155919871ut751148u1000p13434r1702137469k3213382076: No such file or directory
cat: /tmp/kde-me/konsoleHrUTmb.tmp: No such file or directory
cat: /tmp/kde-me/konsoleYxlTMa.tmp: No such file or directory
cat: /tmp/kde-me/konsoleisvloa.tmp: No such file or directory
cat: /usr/kde/3.5/share/apps/kconf_update: Is a directory
cat: /usr/kde/3.5/share/services/kresources: Is a directory
cat: /usr/kde/3.5/share/services/kresources/kabc: Is a directory
cat: /usr/kde/3.5/share/services/kresources/kcal: Is a directory
cat: /usr/share/apps/kconf_update: Is a directory |
Will that matter?
_________________
AssociateX
Gentoo rocks! |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 5:35 pm Post subject: |
|
|
| AssociateX wrote: |
| Quote: |
cat: /usr/share/apps/kconf_update: Is a directory |
Will that matter? |
nope. Ca't cat a directory. that's exactly why the script does 2>&1.... |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 5:36 pm Post subject: |
|
|
| lazarusrat wrote: |
lazarus@massacre ~ $ time ~/neek foo.txt > awk.txt
real 0m0.083s
user 0m0.068s
sys 0m0.014s
lazarus@massacre ~ $[/code]
|
that's fast.
| lazarusrat wrote: |
Edit: Just made it a shell script that invokes awk instead of a flat awk script. |
luckily, we need it as a script and not as an executable. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Fri Aug 18, 2006 6:09 pm Post subject: |
|
|
I just included the awk script lazarusrat gave (thank you very much)
It's quick.
I have put sleep to 0.05 and total_samples_per_boot to 800. So the sampling is accurate. Maybe I will even throw sleep out.
Results are cool.
I think that it is goot to include login managers icons and background in right place in the list of files too, because while computer caches for example firefox or OO it is blockig access to this images and login manager is waiting for them.
What do you think, which files should be in the list too?
-----------------------
P.S. I turned of chrootkit, it loads a ton of files. Maybe now I can put it back.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Fri Aug 18, 2006 6:57 pm Post subject: |
|
|
| I just modified the original post with the discussions so far. |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Fri Aug 18, 2006 7:32 pm Post subject: |
|
|
ROYALE 
Very good!
--- EDIT ---
You should add
| Code: | | rm /usr/bin/uniquer |
to the uninstall instructions.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
AssociateX
Tux's lil' helper
Joined: 19 Feb 2004
Posts: 134
Location: North Dakota
|
| Posted: Sat Aug 19, 2006 3:58 am Post subject: |
|
|
I think I have it working so I thought I would stop back by with some numbers.
I changed the samples from 200 to 400 since you went from sleep 1 to .5 just so I could get enough stuff open in time.
I opened:
kinfocenter
kcontrol
Enemy Territory
Enemy Territory True Combat
The Gimp
Amsn
Mozilla
Mozilla -mail
gFTP
konqueror
xchat
grip
k3b
xmms
oowriter2
kwrite
ksensors
konsole
lshw
top
ls
I checked the word count when sample-init-process was done:
| Quote: | athlon ~ # wc -l /etc/conf.d/sofile-list.load*
1097 /etc/conf.d/sofile-list.load
0 /etc/conf.d/sofile-list.load0
511 /etc/conf.d/sofile-list.load1
761 /etc/conf.d/sofile-list.load2 |
That's about double what I had before... which is fine for me, I want this fast.
On the reboot I checked the memory:
| Quote: | me@athlon ~ $ top | grep -i mem
Mem: 1555020k total, 1120000k used, 435020k free, 15808k buffers |
I can now start and close mozilla this fast:
| Quote: | me@athlon ~ $ time mozilla
No running windows found
real 0m2.190s |
And for OpenOffice Writer to open then close:
| Quote: | me@athlon ~ $ time oowriter2
real 0m3.977s
user 0m2.684s
sys 0m0.172s
|
The machine is:
product: A7N8X-X
vendor: ASUSTeK Computer INC.
product: AMD Athlon(tm) XP 2800+
*-memory 1536MB
Thank you devsk for giving my extra gig of ram something to do. This is fun.
_________________
AssociateX
Gentoo rocks! |
|
| Back to top |
|
|
zxy
Veteran
Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer
|
| Posted: Sun Aug 20, 2006 1:15 pm Post subject: |
|
|
Check this post out.
https://forums.gentoo.org/viewtopic-t-463204-start-0-postdays-0-postorder-asc-highlight-.html
It's a cool defragmenter, though I didn't run it yet, but it looks very promising. It defragments in a very cool way, so it should be a fantastic combination of caching boot and defragmenting the files we need for startup. Then the booting should be done in minimal time.
I can't wait to try the combination. Will post the results.
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
| Back to top |
|
|
wpoely86
n00b
Joined: 01 Aug 2004
Posts: 59
Location: Belgium
|
| Posted: Sun Aug 20, 2006 1:43 pm Post subject: |
|
|
I have a problem with the script: it dies and i don't know why.
I have added echo lines to the script and so i found out it does run and gets into
the sampler loop. But then it dies after +/- 20 loops on the rule "sleep 0.5"
I don't get any error message or so. The moment the script dies is more or less at the same
time with the init scripts end. Anyone got a clue why it dies?
If i run it while the pc is already fully booted, it doesn't die.
_________________
If something is so complicated that you can't explain it in 10 seconds, then
it's probably not worth knowing anyway. -- Calvin |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
