Two 60-bay NAS units

These two NDS-4600-JD-05 units each have space for 60 3.5" drives with four 6 Gbps SAS ports on each of the two controllers. The plan is to connect two R610s (eventually R620s) to each of them with the DAS units partitioned so that each of the R610s/R620s has 30 disks (well, 15-disks on each of a pair of redundant 6 Gbps SAS lines). There will be a Ceph OSD per disk on each of the 30 disks. Half of the 30 disks will be 8TB and half will be either 3TB or 2TB disks. 

Since Ceph OSDs are RAM hogs - One or two GB per OSD PLUS one GB per TB of space - The R610s will be maxed out with 192GB of RAM each. As disks are replaced with larger ones I'll have to switch out the R610s for R620s which have space for more RAM. 

One thing to note is that they are just thin enough that they slide past the zero U PDUs I have in back of my rack. 

The attached photo shows the two DAS units and two of the four R610s. 

A few notes about the hardware:

  • Racking those things was a pita - I love me some Dell rack mount gear. 
  • The drive trays are crazy flimsy
  • Definitely needs two people to get racked
  • Keep at the bottom of the rack as they'll get crazy heavy
  • The cable arms are important if you ever want to swap out drives without a full shutdown
  • You can select via (I think) the web interface how you want the drives partitioned. For instance you can have all four ports on each controller go to all disks, two ports per controller to sets of 30 disks, or one port per controller to sets of 15 disks.