Developer Forum

Hi All

For my research I need large storage for satellite data files. I am
looking to build storage for about 1TB redundant array and >2TB
volatile array. I have looked around at snap servers or other SAN
solutions. I also have looked at scsi solutions but don't think that
scsi is economicable. What I am proposing is to build 2 arrays of SATA
drives with the following setup:

1 3u supermicro chasis with 15 hotswappable sata bays
2 3ware RAID controller (for ex 9550SX-8LP)
12 WD SATA HDS (400GB)

RAID ARRAY 1: RAID 5, 5 drives total, 1 hot spare == 1.2TB
RAID ARRAY 2: RAID 0 (or none), 7 drives total == 2.8TB

This will all be hooked to dual xeon chips with and internal boot/sys
mount. Will probably use win2k3 server due to software requirements.

The second array is for temporary storage and can crash and we will not
loose much except the need to download again. The first array needs to
be fault tolerant. We envision swapping out drives in array 2 as
larger drives become available.

My questions are:

+ Can anyone propose reasons NOT to do this?
+ Are the better solutions?
+ NOTE: We anticipate this will cost $5k-$6k -- the other solutins are
MUCH more that I have seen

thanks
daben

Nor sure I understand how you're going to connect the server(s) to the
array. Fibre channel could add significantly to the cost. iSCSI could
create a 1Gb/sec bottleneck, file serving as in CIFS would probably be
even slower, but you don't mention performance reqs.

Also, how are you going to back up array 1?

Another point of concern is that recovery from a disk error on array 1
could take hours or days. I would go with raid 6 and with RAID enhanced
SATA drives (WD and Seagate have them).

In article < XXXX@XXXXX.COM >,




Is there a comodity raid product that allows me to expand the array by
poping larger drives in, one at a time and leting the raid recovery
work?

What's the status of Infiniband?

http://www.yqcomputer.com/


--
a d y k e s @ p a n i x . c o m

Don't blame me. I voted for Gore.

This storage is for server that it is connected to .. it would not be
network storage. In other words, all the processintg and data access
will be on the machine. It will not be a file server in the normal
sense. This is kinda a special case of just needing a lotof storage
for local processing.

Back up will be accomplished with a lot of tapes :).

In article < XXXX@XXXXX.COM >,



It's possible you can do what you want with dfs, part of the server
ntfs file system if you are loooking to RAID just to aggregate disks
it might be acceptable.





--
a d y k e s @ p a n i x . c o m

Don't blame me. I voted for Gore.

Hi Daben,

I have created similar servers with similar equipment (3Ware,
Supermicro,Xeon's and Opterons). My preference is Opterons due to
better performance for the money, at least for our ocean models but as
far as the storage goes what you have should work very well. The only
other concern might be the power supply size for that many drives. I
have 11 drives with dual opterons in a Supermicro 933 case and the
triple-redundant 600 watt PS. It works fine.

Good luck,

Steve

Hi Steve

Thanks for your input. Our latest thinking is to go with 2+ servers.
We too have found that the AMD chips are far faster than the Xeon chips
but I have some doubts about the AMD chipsets and their reliability
over long periods of time. Maybe you can comment on this. Therefore
our plan is as follows:

+ Storage server as above with the Xeon chips (we already have them
from a previous purchase).
+ 1U dual AMD Opteron 275 server with 2GB RAM, 1 sys disk

We would anticipate the ability to add more 1U servers to increase our
computing power over time. I don't think that Blade servers or fiber
is right for our applications or budgets.

Comments?

thanks
daben

Infortrend has that. But it is a dangerous method: drive failure may
kill you.

That said these are rather affordable so why not buy two...



Thomas

We have had some for close to two years now and haven't had an issue. I
think if you go with a reputable brand you should be fine. We have been
using Tyan with good results although I am glad that I didn't start
using Opterons a little earlier because there were some problems with
some early Tyan boards. At this point I think things are pretty well
established and safe.

This is similar to what I have done. I started out with a 2 TB storage
server with two 1.6 Ghz Opterons. We use it to run models on too with
good results. It does fine doing both storage and computing. Then we
started adding 1U dual-Opterons and when we needed more storage we
bought a 6.4 TB RAID array and connected it to one of the 1U compute
servers via SCSI. All volumes are NFS mounted to all machines and
performance is acceptable with Gigabit ethernet, even without Jumbo
frames. I originally wanted to set up a SAN with GFS or GPFS instead of
NFS but the FibreChannel hardware was too expensive. As we add compute
servers maybe the NFS performance will become an issue but I'll deal
with that as it comes.

Good luck,

Steve

Be a little careful here on the Opteron vs. Xeon speed comparisons. In
raw compute speeds the Opteron is definately faster. But the Intel
chipsets support a much higher performance in I/O throughput.

So consider your application. If your system has a PCI based RAID
controller, and multiple Gbit or 10gig NICs, then you could be easily
saturating the I/O bandwidth of an Opteron based platform. If you are
doing a OS based RAID, then you probably need all the CPU speed you can
get, so the Opteron will yield the best performance. A hardware based
RAID solution won't need that kind of CPU power, you will be
bottlenecked on the disk subsystem and the CPU will be idle a lot of
the time.

I don't know what your complete appliaction entails, but keep in mind
that CPU speed is always the best thing to optimize.

-Jim

Jim

The appllications to run on the Opteron chips will actually be all
loaded into memory before run. Tun times are to be about 6hrs so the
IO will be a very small portion of that.

We are doing HW based RAID with 3Ware controllers.

daben

In article < XXXX@XXXXX.COM >,



I have seen, for whatever odd reason, a high rate of failure of the SATA
cables. I have also seen backplane failure in these types of boxes.
Ironically, as some will see it, the SATA drives themselves seem to be
quite reliable.


A self-contained controller is probably simpler and presumably less
likely to fail.


Promise VTrak is not expensive and actually seems to be reliable. It
also has decent management software.

(Someone else mentioned RAID 6; that's the one thing the VTrak lacks.)

In article < XXXX@XXXXX.COM >,



Do you by chance have a reference for this?

flux < XXXX@XXXXX.COM > writes:


The experience I've had with Promise cards is that they're mostly
software RAID (a lot happens in the driver). This is under Linux.

I'm not familiar with the VTrak specifically though.

I've heard many good things about 3ware.

--
David Magda <dmagda at ee.ryerson.ca>
Because the innovator has for enemies all those who have done well under
the old conditions, and lukewarm defenders in those who may do well
under the new. -- Niccolo Machiavelli, _The Prince_, Chapter VI

In article < XXXX@XXXXX.COM >,



The VTrak is a completely self-contained hardware RAID.


Ironically, 3ware may be problematic under Linux. I've occasionally
experienced "Out of IOMMU" space errors and such an error generally
spits junk data on the RAID filesystems, which is not good. I don't
know, however, if the source of the problem is the 3ware driver (is it
really "hardware" RAID?), the Linux kernel, the card itself, or
something interaction with the motherboard (Arima HDAMA) or the (Wintec)
memory. But since I don't see this problem on the LSI/mptfusion
card/driver in the same box, I'm inclined to believe it's the 3ware
driver or the card itself.