Breathing new life into an Asus P6T: PCI SSD?

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

Hi.

I have the box I put my first-ever Gentoo install on. I'm trying to figure out if I can breathe new life into it or if I should let it run as-is and get new hardware.

This is for a home office. It's for work, not play. The only failures I've had on this box are a bunch of WD green drives, which have made me swear off WD for a few generations of hardware.

Here's what it is:

http://www.asus.com/us/Motherboards/P6T/specifications/
i7 920 (first-gen i7)
12G RAM, in 6x2g sticks, capable of replacing these with 4g sticks=24g
3x PCIe 2.0 x16 (at x16/x16/x4 mode)
1x PCIe x1
2x PCI
6x sata2 (3 Gb/s)
usb 2.0

So here's the deal:

This will probably be a headless server at this point. I need virtualization and reliability with best reasonable speed.

I'm contemplating replacement of the 12g RAM with 24g and replacement of the drives. The existing drives are flaky WD green drives that I've had horrible luck with.

I want an SSD for boot and high speed functions. I also want fake raid1 and maybe a non-raid bulk storage drive. I know how to do all that but I'm not sure how to speed it up. That said, the bulk drive is probably fine as sata2 which I have plenty of room for.

These are the things I'm contemplating with respect to drives:

PCIe SSD. I'm limited to sata2 (3 Gb/s) and even though the cheaper PCIe SSDs are slower than sata3 they're faster than sata2.
Add a Sata3 card.
Add a USB3 card.
Mix and match?

So what I don't really know about is whether this is all worth it. I have a rule I've been following for years about not mixing significant hardware of different generations, but maybe this is not so true anymore.

Thanks in advance.

NeddySeagoon · Posted: Sun Sep 14, 2014 10:38 am Post subject:

1clue,

Depending on the life extension you want, you can get combined SATA3/USB3 cards. One worth having is between 25% and 33% the price of a new motherboard.
Do not get a single lane PCIe slot one, since a single USB3 or single SATA3 port can max out a single PCIe lane.

I have a ASUS U3S6 because it was about the only 4 lane PCIe card I could find.

12G RAM may be enough. You can always increase it later, if your KVMs seem to want it.

I've had problems with the WD Greens too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

for less than 40$ i don't think anyone could beat the prize of sata3 cards (of course if you are fully aware you don't get any raid with it).
oh and i have a P6T too with i950 and 3gbs of ram (but i'm not using amd64 arch)

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

I'm using fakeraid anyway, so a raid-capable card isn't really important.

I was sort of hoping I could get a combo card with a lot of both kinds of ports on it, but it seems that everyone thinks it's fine to just have 2 lanes in use, and give you 4 ports or so. One card has 6 ports (2 external) but you can only use 4 at a time. WTF?!

It seems that sata hardware and usb3 hardware is not that significant, but the pcie 2 slots seem to usually be either 1 lane or lots of lanes. So why not make a card like this with lots of lanes and lots of ports? I don't get it. Stick 8 sata3 ports on the inside, 4 usb3 ports on the outside, and however many lanes that takes up to get good performance. I can see all sorts of applications for this, for starters if it's a workstation you can never have too many USB slots, including a few usb3 slots. If it's a server and you plan on stuffing it full of drives then there are never enough sata ports. Also, anyone planning on extending the life of a box like I'm doing, a one-stop card is a great idea. Or maybe it's just my sense of economy of slots talking?

krinn · Watchman Joined: 02 May 2003 Posts: 7470

That's customer problem :
If you do a 8 disks fakeraid, the 8 disks will be handle by the cpu, and someone will comes to bug you it doesn't work fine (too poor cpu)
If you do a 8 disks fakeraid, the prize won't be that cheap (still cheap but might be too high for "basic" customer).

Now for server :
You just don't care about prize that much, and prize define how many disks will be handle. This time using hardware raid and the cpu to control it is dedicated for that usage (hence the higher prize, but you already know prize isn't a problem anymore as your target isn't basic customer).

You also have the problem such kind of card should be made for pcie with multi-line or high implementation, because of bandwidth limit.
A one line pcie3x can do 8Gbit/s, so 984MB/s compare that to my sas that do ~150MB/s so if i put 8 disks like that, i would eat a bandwidth of 1200MB/s (as its just one drive * number of drive, so 150*8 ). As you see if you really need hardware to REALLY handle 8 disks, your card should be a pcie3x with two lines or the bus will saturate.
for a one line pcie2 you get 500MB/s so 500/150 = at 4 disks you already saturate it. I suppose it might be ok if you use a fakeraid to handle 4 disks bandwidth with a pcie2 card, as your cpu is already too busy handling them all, you cannot really then saturate the port as your datas never can get reach their maximum speed anyway. So even a bit flaw the solution is acceptable because of its prize.

You can easy find plenty 2-4 ports fakeraid sata cards from many manufacturers, but when it comes to 8 or more ports, you're in another category and only few manufacturers are doing that (it comes with high prize and specialized hardware and options, generally sas as it could handle sas or sata).

If you dig it i'm sure you can find a fakeraid/non-raid sata card with 8 ports (it might be ok if you really want the 8 disks support, at least take great care of the bus the card need, it should gave you good answer on expected result).

NeddySeagoon · Posted: Mon Sep 15, 2014 8:35 pm Post subject:

1clue,

Many common SATA plug in card are at most two real ports. Any extra ports are grafted onto the same bandwith via a multiplexer.
Two ports are usually real. Four are 2 way multiplexed. Then the price takes a leap and you suddenly need one PCIe lane per port, which is the technally correct solution.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

@Krinn,

8 ports is just a number I pulled out of the air, I was just thinking of the typical home box re-tasking, which is a file server. And not all the disks would be in one array. If I'm going to stack it full I generally have 1xSSD for the fast stuff, maybe 1 or 2 raid arrays (usually mode 1 or such with 2 disks in it) and a bulk storage device for crap that I don't really care if it dies. And then a slot-load sata bay for backups.

Realistically if I do the fileserver thing I would probably get a commercially built appliance. It's hard to justify the work when you compare price and features.

I was thinking that "the right way" for these cards would be a couple ports per lane, and actually use some of the lanes in the available slots. And a stack of USB3 ports, which are about the same bandwidth.

My typical use case has one or two disks (an array counts as a disk) fired up at a time for big loads and then incidental access to other disks. I don't ever see high loads going to all disks. So FWIW disk IO typically takes a lot less than half the CPU, so I thought doubling it would still give me significant CPU for the basic tasks. Sometimes disk use spikes but as long as it's close to a matched load I'm fine with it.

Also, for this box and what I plan with it, I'm thinking I'll run non-raid for anything that needs to be fast. So that should offload some of the work away from the CPU.

I'm slowly talking myself into this. Some of the slower/bigger drives don't have to be super fast and so they could go on sata2. A smaller card with an SSD and maybe a fast spinner and I don't know what, and a USB3 card don't sound like that much money.

Thanks.

BobWya

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

BobWya,

That's REALLY interesting. You just made me start browsing again, and I'm pretty sure it will take awhile....

Thanks.

BobWya · Posted: Mon Sep 22, 2014 7:15 pm Post subject:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

OK, intensely interested in this X5650. I'm not overclocking here, just FYI. This is for business, but I'm just a bit too needy for hardware to retire it. I need reliability more than speed. I obviously also need speed though, or I wouldn't have started this thread.

<soapbox>
USB3 is not for everyday data storage for me, ever. I've had a lot of arguments with customers and coworkers who seem to think USB is suitable for an enterprise server's permanent storage. Several times I've gone to a customer site to figure out what's wrong with their server and I see a @^$## USB drive from Walmart, probably a WD green, sitting on top of it, and they can't figure out why it doesn't stay up. One flaky USB drive takes down an enterprise server for a half a day, and they still can't figure out why they should put it inside the case. It makes me hopping mad.

USB is what it is. It's for a portable device that I plug in and use to transfer, as in for a backup. I work with a lot of customer data, and the transfer can be over 100GB, sometimes dramatically over. I don't want to spend all day at it.
</soapbox>

Enough of that.

Do you think the stock 920 heat sink is good, or does the server pull come with one? Actually I get an overheating problem with several computers, this one is marginal. It has the stock i7 920 heat sink, I'm contemplating something else.

This box (and most of my home server hardware) has a removable slot-load SATA slot for backups. If I only get 4 ports on my card I don't know that I want to sacrifice one of them for speedy backups. I don't know if the bay is good for it either, probably not.

This box has been a VM host. I was thinking of retiring it from that and go to more sedate service, but maybe with this chip I could keep it around. I wonder how much life is left in the rest of it?

So I'm thinking:

A good Sata3 (600MB/s) card with 4 PCI lanes because that's the best I can seem to find for cheap
A good USB3 card with however many lanes I can get.
I have one more slot for multi-lane PCIEv2, and one more single-lane slot.

So I'm wondering how much speed a PCIE SSD card would help things above the normal sata3 rate, with this new chip? Is it worth considering? I'm thinking not, but would like input from somebody who has already pimped something similar out.

I'd be tempted to make it into a workstation, but I need pcie3x16 for the 6-way card I want.

I could use a 4-way gigabit card on it to bump up networking, but I don't have enough switch bandwidth for it, and I'm not sure how effective it would be anyway.

BobWya · Posted: Tue Sep 23, 2014 6:27 am Post subject:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

@BobWya,

That sounds really cool. I will definitely bump the CPU now.

BobWya · Posted: Thu Sep 25, 2014 2:46 pm Post subject:

BobWya · Posted: Thu Sep 25, 2014 10:18 pm Post subject:

@1clue,

After hours of futzzing about with Google. The best I came up with was:
Google Shopping results for LSI SAS 9207-8i Storage controller- 600 MBps
It's dead simple - really designed for powering multiple non-RAID'd SAS drives via multiple SAS Edge Connectors I guess. (NB LSI also do a cheaper 4-port controller version). The Google Shopping reviews are much more positive than for the shitty Marvel-controller based cards (I've had the misfortune to product test).

The benefit of a high-end but non-RAID card - would probably be not having to wait 2 minutes for your f**king machine to boot!

It ain't cheap - but it appears you get what you pay for... If I buy it - I'm would probably test it out with some Startech latching 50cm SAS-SATA cables this time. My current (Addonics) SAS breakout cables are non-latching at the SATA end (welcome back to 2005...)

All the best,
Bob
_________________
system: G751JT (ASUS-NotebookSKU); processor: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz; memory: 32GiB System Memory; display: GM204M [GeForce GTX 970M]; disk: 2048GB Samsung SSD 850;BD-CMB UJ172 S;1024GB Samsung SSD 850

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

That looks awesome, but it's pcie3. I would definitely do this if it looked like it would work on my old motherboard. I'll also certainly keep it in mind for new hardware.

BobWya · Posted: Thu Sep 25, 2014 11:11 pm Post subject:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

I guess I learned in the earlier days, I got screwed by slot versions on some of the first computers I built.

I guess given that it has 1 lane per port, I expect it will do better than any of the other options I've looked at.

Thanks again.

BobWya · Posted: Fri Sep 26, 2014 10:51 am Post subject:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

BobWya · Posted: Mon Dec 15, 2014 8:59 pm Post subject:

@1clue,

Further delays, to my LSI HBA purchase, due to getting a new QWHD monitor in the Black Friday sales!!

Anyway since my last post I found an old BSD Now Episode where Allan Jude mentioned the very LSI Controller I was looking at!! So I submitted a show question asking how much this particular LSI slowed down the boot process. Since Allan mentioned that he uses this card in a couple of his home rigs. Allan gave the most detailed Podcast answer I've ever heard in my life!! The long and short of it was - this is a good card!!

So I bit the bullet this month and purchased:

Here's a Youtube video I did of the boot process.

I did a few benchmarks (before and after) using a hack of an Arch Wiki script.

The pre-results were...

Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller

BobWya · Posted: Wed Dec 31, 2014 4:54 pm Post subject:

Hi Folks,

Now the problems start to emerge!! trim support is patchy...

The connected SSD(s) needs to support deterministic read after trim / read zero after trim

Oh well... The Samsung Pro line does support this feature the Samsung 830 doesn't...

Bob
_________________
system: G751JT (ASUS-NotebookSKU); processor: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz; memory: 32GiB System Memory; display: GM204M [GeForce GTX 970M]; disk: 2048GB Samsung SSD 850;BD-CMB UJ172 S;1024GB Samsung SSD 850