big data on little linux: hard-won lessons managing dozens of servers processing petabytes of data
big data on little linux: hard-won lessons managing dozens of servers processing petabytes of data
By Daniel Sterling (HighBit) from Raleigh.pm
Date: Wednesday, 22 June 2016 14:00
Duration: 20 minutes
Target audience: Any
Language: English
Tags: bigdata linux
You can find more information on the speaker's site:
I manage a cluster of Linux servers that store and analyze terabytes of genomic data daily. (Total managed storage w/ archive data is well over a petabyte.)
This talk will cover some hard-won lessons learned during the journey, including:
* what hardware to buy (or not buy)
* tragic mistakes with NFS
* oddities of the Linux kernel storage and memory subsystems (and how to address them)
* the future of linux storage (lots of neat stuff on the horizon)
Come by if you're interested in hearing a few big data war stories!
Attended by: Todd Rinaldo (toddr), Jeff Till, Greg Brennan, Steve Nolte (mcsnolte), Marcos Laborde, James Lenz (Jim), James E Keenan (kid51), gary, Tom Browder (tbrowder), Tushar Dave, Ruben Amortegui, Matt Creenan, Michael LaGrasta, Michael South (msouth), Paul Johnson, Greg Cole (wingfold), Leonard Harris, Gabriel Munoz (gabriel), Mark Gardner (mjgardner), Brad Lhotsky, Adam Engle, Rob, Julian Brown, Ben Rosengart, Michael Hamlin (myrrhlin), Thomas Stanton (tstanton), Scott Duff (perlpilot), Doug Bell (preaction), Aaron Staves (astaves), Brad Oaks (bradoaks), R Geoffrey Avery (rGeoffrey),