Salam Alaykum everyone,
Hope the semester is treating you all well, especially with midterms rolling in.
I’m currently deep in the weeds with my Senior Design Project, which involves processing some pretty massive datasets for a machine learning application. We quickly realized that our laptop SSDs weren’t going to cut it, and relying solely on cloud instances was burning through our budget too fast. So, I decided to set up a local storage server for the team to centralize our data.
I managed to salvage some equipment to build a dedicated storage box. To handle the capacity we need, I’ve connected an external disk shelf to the main server using a SAS-SATA PERC H830 controller. On paper, this setup is perfect because the 12Gb/s throughput is necessary for the read speeds we need during training epochs.
However, I’ve run into a bit of a dilemma regarding the configuration, and I know there are a lot of knowledgeable COE and ICS guys here who might have messed with enterprise gear in the labs or for their own projects.
Here is the situation:
Currently, I have the PERC H830 configured in standard Hardware RAID 6 mode. It’s stable, and the battery-backed cache gives me peace of mind in case of a power flicker (we lost data once already, never again). However, I’ve been reading a lot about ZFS and how it handles data integrity much better than traditional hardware RAID, especially for long-term storage.
My struggle is that the PERC card abstracts the physical disks from the OS. I’ve read that if I want to use ZFS, I need to flash the card or configure it to “HBA mode” / “Non-RAID mode” to pass the raw disks through to the OS.
I honestly thought the hardware setup would be the easy part compared to the actual coding, but I’ve spent more time this week reading technical manuals than actually training our models. It reminds me of the headaches in the lower-level hardware labs where the theory is simple, but the implementation is a nightmare.
My question to you guys:
For a project like this where data integrity is key but we also need high throughput, is it worth the hassle to switch to ZFS (software RAID), or should I just trust the PERC H830 hardware RAID and move on?
I’m worried that if the controller card dies, I won’t be able to mount the array on a different system easily, whereas ZFS seems more portable. But I don’t want to sacrifice too much write performance.
Has anyone dealt with this specific tradeoff in a Linux environment?
Thanks in advance!