Stuart Hayes: Attempting to build a BigData BI solution on extreme commodity hardware!
This blog was begun to serve as a place to brain-dump of thoughts and interesting things I’ve found on the web, in books, through my own efforts – a permanent, searchable whiteboard, if you like. Something I can search and refer back to when i begin my MSc project write-up.
Please don’t copy & use any notes on here without permission of the author – I try to link to where i sourced the content. I encourage you to support the vendors & authors I blog about (if they meet your requirements).
The content will be loosely-structured and fall into the categories below.
#Mongodb #Database cluster #Sharding
#R #Data #Analytics
#Raspberry Pi
#University of Dundee #MSc Business Intelligence
#ETL
#Reporting, visualisation
#Data models, #schema-free
#noSQL
As part of my MSc in Business Intelligence @ The University of Dundee, I am attempting to build a cheap DB cluster / decent BI solution on cheap/legacy/commodity hardware for around £300 (although I seem to be spending more and more on cables to connect it all up!)
R Pi (x5!)
The topography of the cluster is one really old Dell Dimension desktop running winXP (though I plan to Linux it!), a new Samsung laptop running Win7 and 3 4 5 Raspberry Pis (ModelB, 512MB, 4GB SD) running Raspbian Wheezy (Linux).
Raspian O/S
I may buy a couple more RPIs (if i can figure out how to get them working(!) as they’re about £25 each (plus ethernet cables, SD cards etc)) and see how that improves cluster performance (speed, ACID/BASE possibilities).
My plan is to get some data into MongoDB using Pentaho Data Integration (‘Kettle’).
PDI
MongoDB
Then Figure out how to shard it across the nodes (and the performance implications) and then produce some nice reports/visualisations, drawing data out of MongoDB into a reporting tool (poss Jaspersoft iReport). I would like to use some unstructured documents (bids/tenders/contracts/court proceedings etc) as my dataset and try to boil up some interesting data using map/reduce on top of MongoDB.
I’d really appreciate any help you may be able to give, as this is a steep learning-curve.
Cheers,
Stuart
Stuart Hayes (<3 data) 
Stuart,
Good luck. Sounds like a great project. I have taken the 10gen free online course on MongoDB for programmers, and am now taking the database administrators class.
https://education.10gen.com/dashboard
At the time, I didn’t consider trying it on my Raspberry Pi. Now, however, it seems interesting. The big/little endian problem is worrying, though. Are you compiling from scratch? Or is there a special .deb package I just haven’t found yet?
Anyway, success with your project and MSc.
John
Hey Guys,
Sounds like a great project. I’m doing the same myself with 2 PI. One is running Raspbian, while the other is on Raspmc and getting a bad habit for smoking Gauloises in the home made box for it.
I have Ruby, Rails and sqlite on one and XBMC on the other. They are both starting to talk very well, so hope to get mongoid going next to help with some NFC experiments.
Sorry I can’t be much help right now, but will let you know any progress I make. Good luck.
Hey,
Any luck building MongoDB on a raspberry pi? I’ve seen a couple of attempts at porting it but nothing that’s current. Would be interested to hear how you’re getting on.
Hi MongoPI,
I’m looking for someone who could provide me the all build directory (I don’t want to waste time to buid the all mongodb). Having that directory, I could easily build the .deb package using fpm tool. I use fpm to make node.js packages for ARMv5, ARMv6 and v7 (see: http://www.nodejs-news.com/nodejs-tech/ruby-to-build-nodejs-raspberrypi/).
If you could provide me a tar.gz of you build directory it could be cOOl !
Best regards,
Vincent
Hi,
I built a Raspberry Pi cluster (https://www.cerbelle.net/redmine/projects/raspberry_cluster), it is up and running from an hardware point of view, the software stack is still light, but I wanted to install MongoDB too and I found your blog.
I work at Jaspersoft, I might help you on this side (either on iReport/Jaspersoft studio report design or on Jaspersoft ETL if you give up with Keetle).
Regards