Ceph: Networking Between Hosts

Hosts in my cluster all have two 10GbE links to a switch bonded with LACP. That gives them an effective link of 20GbE normally. But really, even 5Gbps would be enough for most hosts. The kernel in most boxes combined with drivers and such isn't going to be able to push 20GbE, much less 40GbE. So faster isn't worth it. 

Note: Don't bother with IPoIB. You're better off with 10GbE x2 via LACP. Although if it's a choice between 40Gbps IB and IPoIB or a couple of 1GbE lines, go IPoIB. 

Ceph: What Drive Sizes To Use

Drive Counts

  • 43 2TB
  • 1 2.5TB
  • 23  3TB
  • 42 8TB (In Ceph)
  • 30 8TB (Not In Ceph)

Why

Why these drives? They're the main data drives I've had and used for years. The only ones I'm going to remove soon are a subset of the 2TB drives with high spin times. Some of them have more than 8.5 years of spin time. I'll probably remove any disk with more than 6 years of spin time as a preventive measure. 

Tags

Ceph: osd_memory_target

A couple of months back I changed the value of "osd_memory_target" for all of my OSDs from 4GiB to 1.5GiB. That change has stopped all RAM related issues on my cluster. While I suspect but can't prove a small performance drop, it's well worth it in my case. 

Tags

NDS-4600 - SATA Drive Failures In Linux

One issue I've recently run into with a failed SATA drive in one of my NDS-4600 units is that Linux frequently tries to recover the drive by resetting the bus. This takes out a few other disks in the group with it. The resulting IO timeouts cause problems for my Ceph OSDs using those disks. 

It should be noted that only some types of disk failures cause this. The host bus resets only are done by the Linux kernel in some cases (I think) and I suspect the cause of the other disks errors is said disk. 

Sterling, Va Wires-X Node & Repeater Is Up: 449.375- 110.9hz

The Sterling, VA, USA repeater and WIRES-X node is now up and operational. 

It is a full duplex WIRES-X node, C4FM Repeater, and FM Repeater in FM19ha run by KG4TIH.

This Yaesu DR-2X repeater on 70cm is set for analog and C4FM mode. The node is set to auto join the VA-Sterling (28558) room after an inactivity period of 10 minutes. When in this room or others all C4FM traffic will be sent to the Wires-X room and all signals received via the room will be transmitted via C4FM. The Wires-X features on some Yaesu radios will allow the user to control what room is connected. 

Subscribe to