OS and File System
The Pod boots 64-bit Debian 4 Linux using the JFS file system, with all access to and from the Pod via HTTPS (Figure 5).
Figure 5: Backblaze Pod software architecture
Backblaze went with JFS because they needed something that would support large TB volumes, run on Debian and have a good support community. Debian was chosen because "it is truly free". Backblaze found that some other free open source OSes are more entry-level options into paid offerings.
To create a volume, fdisk is first used to create one partition per drive. Then 15 1.5 TB drives are then combined into a single RAID6 volume using mdadm and finally, formatted with JFS using a 4 KB block size. Formatted (useable) space is about of 87% of the 67 TB of raw hard drive capacity or about 58 TB per Pod.
As noted earlier, all read/write is via HTTPS running in custom Backblaze application layer logic under Apache Tomcat 5.5. HTTPS was chosen over iSCSI, NFS, SQL, Fibre Channel, etc. because Backblaze believes that other technologies don't scale as cheaply, reliably and big, or can be managed as easily as HTTPs.
The design process is always a battle of cost, time and performance, with successful designs achieving the right balance. Perhaps the most important thing to be learned from the Backblaze Pod design is how its creators made the their tradeoffs to achieve their design goals of low-cost, high density and large capacity.
The Pod ended up with two custom components: the case and Apache application logic. This minimized cost and time by leveraging mass produced components and, as Backblaze put it, being able to "stand on the shoulders of giants" and not have to re-invent the wheel.
The other key was in understanding how much performance was enough. Since the Pod is part of a web-based service, it doesn't need to run at blazing speeds. Given the relatively slow Internet uplink speed and application logic that sit between a user's computer and the Pod's drive, Backblaze understood how fast the Pod had to be and designed to that goal. Pushing for 100 MB/s reads and writes would have made no sense when the larger system isn't able to utilize it.
In fact, there may be other cost savings yet to be made. When I asked how they settled on a relatively expensive (and power hungry) Intel E8600 Core 2 Duo as the Pod's CPU, Backblaze confessed that they didn't do extensive testing. Given the tradeoff between saving perhaps $100 on the CPU and spending their time on other aspects of the design, they chose to throw a little money at the processor.
However, now with a completed design, they may look at lower power CPUs (and drives) in a future version of the Pod. Because the savings in power and facilities cooling can really make a difference in ongoing operational cost.