Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Ps3 - External SWAP using nfs + ramdisk how to
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo on PPC
View previous topic :: View next topic  
Author Message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Wed Dec 03, 2008 8:33 pm    Post subject: Ps3 - External SWAP using nfs + ramdisk how to Reply with quote

I know I will probably get told that the performance of this isn't that great, and questioned why I might want to do something like this, but here goes.

When you use the PS3 Drive for swap, the PS3 constantly has to read/write to the HD that it is also mounting for the filesystem. I was just playing around with this today, got it working, and compiling about 20 things at once to try and crash it.

So far it's been stable.

***MAKE SURE YOU HAVE NFS SUPPORT emerged on the PS3!!!

I turned off my native HD swap with the swapoff command, and likewise, to test this disabled the fbswap. I have the loop device driver compiled into my kernel. If you want, you can make it a module, also I have the PREEMPT Kernel compiled 2.26.4-ps3 or something like that.


Code:
sudo mke2fs -b 1024 -vm0 /dev/ram0 1040380



(**1040380 =~ 1.1 gig (so we don't go over this size later, when creating the swapfile the PS3 will use.)
-b option gives you 1024 size blocks, otherwise you get a messed up error due to some other issue. The 1024000 was my first attempt at this, but you want to actually have it set as 524288 * 2 = 1048576, (Someone correct me if I'm wrong.) Also you will need to make sure this is dividable by 4. (Check the 4kb thing in google if you want to try and figure out why.)

Also etx3 is useless, you don't want to journal something that erases at reboot.)

These commands mount the ramdisk at /mnt/ramdisk, and create a swapfile that we are going to share via NFS:

(assuming that you are root!!!)

On Ramdisk Server:
Code:

mkdir /mnt/ramdisk
mount /dev/ram0 /mnt/ramdisk
dd if=/dev/zero of=/mnt/ramdisk/swapfile bs=1024 count=1040380


Edit /etc/exports to reflect this data, also done on the Server, and the ip reflects my PS3, you should change this to whatever your network uses.:
Code:
/mnt/ramdisk 192.168.0.2(rw,no_root_squash,async)


Then mount on the PS3 via this command:
Code:
mount -t nfs 192.168.0.1:/mnt/ramdisk /mnt/swap


Use the swapfile we created with the dd command earlier, but do this on the PS3, make sure you have loopfs support. (modprobe loop, if loopfs is not setup in your kernel)

Code:
losetup /dev/loop0 /mnt/swap/swapfile

(this is using loop0, but you can check what /dev/loop* has listed)
Code:

mkswap /dev/loop0
swapon /dev/loop0


After all this, you will have an external, ram based, nfs swap disk. I have about 1 gig, but I was going to try it with a 4 gig machine at work. :twisted:

swapon -s output:
Filename Type Size Used Priority
/dev/loop/0 partition 1033180 235564 10

vmstat output:
Every 10.0s: vmstat 5 3 Wed Dec 3 20:14:32 2008

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 229708 57584 3268 59156 225 127 490 391 435 555 33 36 24 7
6 0 229696 30768 3484 60660 24 0 275 145 180 599 67 32 0 0
3 0 229680 13056 3552 61816 25 0 105 49 115 626 66 34 0 0

Also, the ethernet on the PS3 shows as Gig ether, but I'm not certain it is performing all that well. Could someone suggest some tools for me to test the throughput of the swap speed, and ethernet connection?

8) Currently the NFS Memory System is cross over cabled to the PS3, without a hub or switch.

Tasks: 108 total, 4 running, 103 sleeping, 0 stopped, 1 zombie
Cpu(s): 32.8%us, 30.9%sy, 0.0%ni, 23.2%id, 8.0%wa, 1.3%hi, 3.9%si, 0.0%st
Mem: 228516k total, 226232k used, 2284k free, 772k buffers
Swap: 1033180k total, 448960k used, 584220k free, 24468k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3551 root 20 0 2204 552 432 S 2 0.2 5:36.21 dhcpcd
14064 root 20 0 3080 1376 720 R 2 0.6 0:00.75 scanelf
27788 root 20 0 30856 10m 1964 D 2 4.7 0:27.20 emerge
30 root 15 -5 0 0 0 S 1 0.0 1:53.66 kswapd0
3889 root 0 -20 0 0 0 S 1 0.0 3:29.48 loop0
9008 root 20 0 237m 125m 1272 D 1 56.4 1:37.09 cc1plus
14130 root 20 0 3600 1772 1280 R 1 0.8 0:00.03 top
22298 root 20 0 73092 3924 2412 S 1 1.7 1:16.46 xfce4-systemloa
1 root 20 0 2180 736 708 S 0 0.3 0:00.72 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.47 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.18 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.03 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:01.46 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.14 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.03 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:01.70 events/0
ps3-betabox ~ #

Anyone want to help me figure out how to compile the Cell SDK3.1 from source? :mrgreen:


Last edited by Pliablemammal on Fri Dec 05, 2008 11:51 am; edited 1 time in total
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Thu Dec 04, 2008 12:27 am    Post subject: Update - 1056772 blocksize = 1GB Reply with quote

I noticed I fudged the math a little...

ramdisk=1056772 is what I'm trying out now.
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Fri Dec 05, 2008 10:43 am    Post subject: Update - USB vs GB Ethernet Reply with quote

Been reading up a bit -http://www.lyberty.com/tech/terms/usb.html

Looks like GB Ethernet is pretty damn fast, but USB 3.0 is coming soon. So it's a toss up as to what would be faster for an external swap, but hey this could be something useful in the near future when the RSX is finally unlocked.
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Fri Dec 05, 2008 10:59 am    Post subject: Reply with quote

Doing this on the Memory Server and the PS3 also helped with my hdparm -t times:
Code:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling


I got this info from here:

http://www.speedguide.net/read_articles.php?id=121

This pretty much doubled my i/o for the network, which is still just a crossover cable from a PC to the PS3 via 100MB Ethernet, can someone with true GB Ethernet PLEASE Attempt this and reply with the output of
Code:

hdparm -t /dev/loop0


So come on guys, someone post a reply.

Can you try this on your box? Also when I ran:

Code:
hdparm -t /dev/loop0
/dev/loop0:
 Timing buffered disk reads:   114 MB in 3.00 seconds = 37.99 MB/sec


hdparm -t /dev/ps3da2
/dev/ps3da2:
Timing buffered disk reads: 68 MB in 3.06 seconds = 22.20 MB/sec[/code]

If anyone has the frame buffer driver please post the output of hdparm for me :P

I found that this NFS ramdisk swap was a little less than twice as fast as the PS3 internet HD swap. If the eth port on my laptop was in fact 1Gb I think we'd see better results.
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3753
Location: sleeping in the bathtub

PostPosted: Sat Dec 06, 2008 12:32 am    Post subject: Reply with quote

Sorry I don't actually have a ps3 (poor me), but this thread is still relevant to my interests.

First thing I'd point out, gigabit ethernet may very well be faster in terms of bandwidth, but what about latency, which IMO would be much more important in this case?

Anyways, using nfs for this is just creating additional unnecessary overhead, what you should look into is NBD (Network Block Device) or similar, which allows you to export a raw block device (presumably including a ramdisk) over the network rather than a filesystem.

Also, you know about p3vram, which enables you to use the 256 mb of vfram on the ps3 as a ramdisk, which you could use as swap?
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Sat Dec 06, 2008 7:22 am    Post subject: Reply with quote

Hopeless wrote:
Sorry I don't actually have a ps3 (poor me), but this thread is still relevant to my interests.

First thing I'd point out, gigabit ethernet may very well be faster in terms of bandwidth, but what about latency, which IMO would be much more important in this case?

Anyways, using nfs for this is just creating additional unnecessary overhead, what you should look into is NBD (Network Block Device) or similar, which allows you to export a raw block device (presumably including a ramdisk) over the network rather than a filesystem.

Also, you know about p3vram, which enables you to use the 256 mb of vfram on the ps3 as a ramdisk, which you could use as swap?


Yeah I have seen that and know a few people that use it for their businesses, it would be great but since there is only one eth port on the PS3 I opted to do it this way, it's also a little bit tough getting my laptop which is running Ubuntu to cooperate. :evil:

You know I'll give it a shot and see what I can figure out, I have a long night tonight at work so I'll research it a bit more.

Ideally this technique could work on servers where you're doing a lot of operations with big chunks of data, but it's scary because you could end up losing it is the power goes out - maybe something like Alicebot with the mysql backend, or sendmail with a large amount of users, or simulations. I was going to try it for ripping and converting media, but my friend pointed out that there is still a bottle neck when the data has to be written to the hard disc.

I have access to a machine here today, but I left the PS3 at home. I plan to attempt the hdparm -t to see what happens, maybe I'll know by tomorrow.

The secret to speeding this up is tweaking tcp/ip a little and making sure the nfs buffer settings are correct, I really wish there was a driver out there to mount this before kboot has a chance to start the kernel, sort of like running a external hypervisor, so the system would just see it as real ram, and maybe slave one of the spus for this process.

I slammed the system with tons of tasks and it ran rock solid when just attached to my laptop.

ps3vram was a lot faster, but then again it is still limited to the 256megs, and some programs like a lot more, with the kernel set with PRE EMT it seems to run really fast.
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3753
Location: sleeping in the bathtub

PostPosted: Sat Dec 06, 2008 8:31 am    Post subject: Reply with quote

Maybe you should actually try using swap on both ps3vram and nbd, setting ps3vram to a much higher priority, that would give you the best of both worlds.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Sat Dec 06, 2008 9:09 am    Post subject: Reply with quote

Do you have a link on where to get the ps3vram module working? :oops:

I have a fresh 2008.0 install, and working on getting the kernel tweaked, but I left my PS3 at home tonight so it will just have to wait till the morning.
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Sat Dec 06, 2008 9:49 am    Post subject: Reply with quote

I found this article to be helpful, and I've now got nbd-server on my laptop.

http://www.linux-mag.com/id/7118
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Tue Dec 16, 2008 2:24 pm    Post subject: Reply with quote

after reading a bit about the /spu mount has anyone attempted to do this ramdisk technique, but mounted the /spu to an external nbd or nfs partition?

I'm almost there, I just have a few more things I want to tinker with before I can finish testing the nfs stuff out.
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Tue Dec 23, 2008 1:01 pm    Post subject: Reply with quote

Finally got a chance to test it, on GB ethernet it looks like the rates are the same. Not too sure if this is correct, I will try again in a moment after tweaking some things.
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Tue Dec 23, 2008 2:01 pm    Post subject: PS3 Full Duplex 1000Mb setting Reply with quote

I messed with the eth0 settings to try and force full duplex / 1000Mbs/sec not sure if I was successful, I had to "force" it on on my server.

I grabbed the ethtool and other utils, but the PS3 doesn't want to let me set the card, I guess more HyperVisor Bastardization. Anyone know a fix? :evil:
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
NewBlackDak
Guru
Guru


Joined: 02 Nov 2003
Posts: 512
Location: Utah County, UT

PostPosted: Thu Jan 01, 2009 4:55 am    Post subject: Reply with quote

You might also try adding the following to your mount options:
-o rsize=32768,wsize=32768,udp


udp is faster and less processor intensive for both the client and server on networks that aren't too busy. You should also get higher throughput with the bigger block reads/writes.
_________________
Gentoo systems.
X2 4200+@2.6 - Athy
X2 3600+ - Myth
UltraSparc5 440 - sparcy
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Sat Jan 17, 2009 3:07 pm    Post subject: Reply with quote

I'm still wanting to try this! I have not had a chance since my room mate has been playing Fallout 3 NONSTOP!!!
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Mon Jun 08, 2009 5:50 am    Post subject: Reply with quote

Hey long time since I've posted anything to this...

Just thought about this tonight, been talking to a student at UC Davis.

It looks like I might be able to test this idea further by installing and running Octave on the PS3, does anyone care to comment, or has anyone attempted this since I posted about a year ago?
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Pliablemammal
n00b
n00b


Joined: 02 Oct 2006
Posts: 52
Location: Sacramento

PostPosted: Tue Feb 18, 2014 8:15 am    Post subject: Using zramdisk compression for the fb ram device Reply with quote

Not sure if this has been read for a long while - I got the bug to get Linux back on my PS3 - It's running a weird Debian variant right now, but it will be running Gentoo again very soon.

I'm going to attempt to compile a kernel with zramdisk compression support and see if it works out okay. Thinking there should be a patch for the spe or cellbe processors to handle the compression, but unlikely since Sony F'd everyone with this.
_________________
Linux ps3-betabox 2.6.28-rc7-00948-gc5fd701 #5 SMP PREEMPT Sun Dec 14 18:25:40 PST 2008 ppc64 Cell Broadband Engine, altivec supported GNU/Linux - gcc version 4.4.0 20081212 (experimental) (GCC) Videos
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on PPC All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum