Bigdata & the ‘Lambda’ architecture
I’ve started reading the Manning book “Big data: Principles & best practices of scalable realtime data systems” by Nathan Marz released under Manning’s MEAP (early access) initiative. The book is slated for publication towards end of April.
In searching for it, it was advertised with a scandisk 8GB memory card, which i found interesting!
It really sums up the challenges people working with data are being forced to (at least) think about.
“The data we deal with is diverse. Users create content like blog posts, tweets, social network trails and photos. Servers continuously log messages about what they’re doing. Scientists create detailed measurements of the world around us. The internet, the ultimate source of data, is almost incomprehensibly large.
This astonishing growth in data has profoundly affected businesses Traditional RDBMSs have been pushed to the limit. In an increasing number of cases, these systems are breaking under the pressures of bigdata. 1960s systems and the data mgmt techniques associated with them, have failed to scale.
To tackle the challenges of bigdata, a new breed of technologies have emerged. Many of these technologies have been grouped under the term noSQL. <B> In some ways, these new technologies are more complex than traditional RDBMSs, and in other ways they are simpler </B>. These systems can scale to vastly larger sets of data, but using these technologies requires a fundamentally new set of techniques. <b? they are not one-size-fits-all </b> solutions.
<b> In order to meet the challenges of big data, you must rethink data systems from the ground up </b>. You will discover that some of the most basic ways peoplpe manage data in RDBMSs <b> too complex </b> for big data systems. The ‘simpler’ alternative approach is the new <i> pardigm </i> for bigdata. We, the authors have dubbed this approach <bi> the Lambda Architecture </bi>. You will explore the “bigdata problem” and why a new paradigm for bigdata is needed. You’ll see the perils of some of the traditional techniques for scaling and discover some deep flaws in the traditional way of building data systems. Then, starting from <i> first principles </i> of data systems, you’ll learn a different way to build data systems that avoids the complexity of traditional techniques.