big data on little linux: hard-won lessons managing dozens of servers processing petabytes of data

big data on little linux: hard-won lessons managing dozens of servers processing petabytes of data

By Daniel Sterling (‎HighBit‎) from Raleigh.pm
Date: Wednesday, 22 June 2016 14:00
Duration: 20 minutes
Target audience: Any
Language: English
Tags: bigdata linux

You can find more information on the speaker's site:


I manage a cluster of Linux servers that store and analyze terabytes of genomic data daily. (Total managed storage w/ archive data is well over a petabyte.)

This talk will cover some hard-won lessons learned during the journey, including:
* what hardware to buy (or not buy)
* tragic mistakes with NFS
* oddities of the Linux kernel storage and memory subsystems (and how to address them)
* the future of linux storage (lots of neat stuff on the horizon)

Come by if you're interested in hearing a few big data war stories!


Attended by: Todd Rinaldo (‎toddr‎), Jeff Till, Greg Brennan, Steve Nolte (‎mcsnolte‎), Marcos Laborde, James Lenz (‎Jim‎), James E Keenan (‎kid51‎), gary, Tom Browder (‎tbrowder‎), Tushar Dave, Ruben Amortegui, Matt Creenan, Michael LaGrasta, Michael South (‎msouth‎), Paul Johnson, Greg Cole (‎wingfold‎), Leonard Harris, Gabriel Munoz (‎gabriel‎), Mark Gardner (‎mjgardner‎), Brad Lhotsky, Adam Engle, Rob Schaber, Julian Brown, Ben Rosengart, Michael Hamlin (‎myrrhlin‎), Thomas Stanton (‎tstanton‎), Scott Duff (‎perlpilot‎), Doug Bell (‎preaction‎), Aaron Staves (‎astaves‎), Brad Oaks (‎bradoaks‎), R Geoffrey Avery (‎rGeoffrey‎),