(Some) Sharding options

  1. Shard by time
  2. Shard by a semi-random key
  3. Shard by an evenly distributed key in the data set
  4. Shard by combining a natural and synthetic key

There are pros and cons for each – for example – with option #1 – all inserts (same timestamp) will flow to the same shard. Most reads will tend to cluster on the same shard, assuming the users query the recent data more frequently.

Sharding

MongoDB startup troubleshooting : “unexpected shutdown”, “Address already in use for port”

I’m having another attempt and building out my cluster today. Following Simon the PiMan’s guides

  1. http://www.simonthepiman.com/beginners_guide_find_my_network_settings.php
  2. simonthepiman.com/how_to_setup_your_pi_for_the_internet.php
  3. http://www.simonthepiman.com/how_to_setup_remote_access.php#raspberrypi

I just want to get one or 2 R Pis connected to the network switch and the Dell (acting as master).

I fire up the Dell and call MongoD (which I haven’t used for a couple of weeks), to get a surprising error:

old lock file: /data/db/mongod.lock.  probably means unclean shutdown recommend removing file and running --repair see: http://dochub.mongodb.org/core/repair for more information

The right way to shut down Mongo

I guess I must have done something wrong the last time I had it open. To ensure a clean shut down, use the mongod –shutdown option, your control script, “Control-C” (when running mongod in interactive mode,) or kill $(pidof mongod) or kill -2 $(pidof mongod).

Finding out/remembering where you installed MongoDB

$ sudo find / -type d -name mongo
This will search  from root (/) for directories named ‘mongo’
Reveals I installed everything to /usr/bin

Find out where you installed the directories for $DBPATH, check the filesize for mongod.lock

$ locate /data/db
/data/db- Note to self, create this in the same tree as the Mongo directories next time!

cd /data/db
ls -l
-rwxr-xr-x 1 stuart stuart 0 Jan  6 22:00 mongod.lock

If the mongod.lock file in the data directory specified by dbpath, /data/db by default, is not a zero-byte file, then mongod will refuse to start.

However, as above, you can see that the filesize is 0 bytes for mongod.lock, so, I’m not quite sure what the problem is.

Repairing MongoDB

$ sudo mkdir /data/db/db0
– Mongo attempts to create a new directoryand move the old/repaired lock file. As I seem to have some ongoing permissions problems, i pre-create this folder.

$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0

Feedback from the shell below suugests it has worked…

stuart@debian:/data/db$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0
Sun Jan  6 22:33:30 Mongo DB : starting : pid = 16652 port = 27017 dbpath = /data/db master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Sun Jan  6 22:33:30 finished checking dbs
Sun Jan  6 22:33:30  dbexit:
Sun Jan  6 22:33:30      shutdown: going to close listening sockets…
Sun Jan  6 22:33:30      shutdown: going to flush oplog…
Sun Jan  6 22:33:30      shutdown: going to close sockets…
Sun Jan  6 22:33:30      shutdown: waiting for fs preallocator…
Sun Jan  6 22:33:30      shutdown: closing all files…
Sun Jan  6 22:33:30      closeAllFiles() finished
Sun Jan  6 22:33:30      shutdown: removing fs lock…
Sun Jan  6 22:33:30  dbexit: really exiting now

Or, to wipe your data files without preserving the original files, do not use the –repairpath option, as in the following procedure:

  1. Remove the stale lock file:
    rm /data/db/mongod.lock

    Replace /data/db with your dbpath where your MongoDB instance’s data files reside.

    Warning: After you remove the mongod.lock file you must run the –repair process before using your database.

  2. Start mongod using –repair to read the existing data files.
    mongod --dbpath /data/db --repair

    When this completes, the repaired data files will replace the original data files in the /data/db directory.

  3. Start mongod using the following invocation to point the dbpath at /data/db:
    mongod --dbpath /data/db

Port already in use?!

After repairing Mongo by clearing out the old lock, I try to start it afresh.

stuart@debian:/data/db$ mongod
mongod –help for help and startup options
Mon Jan  7 12:01:20 Mongo DB : starting : pid = 17622 port = 27017 dbpath = /data/db/ master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Mon Jan  7 12:01:20 db version v1.4.4, pdfile version 4.5
Mon Jan  7 12:01:20 git version: nogitversion
Mon Jan  7 12:01:20 sys info: Linux murphy 2.6.32.14-dsa-ia32 #1 SMP Thu May 27 16:19:20 CEST 2010 i686 BOOST_LIB_VERSION=1_42
Mon Jan  7 12:01:20 waiting for connections on port 27017
Mon Jan  7 12:01:20 listen(): bind() failed errno:98 Address already in use for port: 27017
Mon Jan  7 12:01:20 MiniWebServer: bind() failed port:28017 errno:98 Address already in use
Mon Jan  7 12:01:20   addr already in use
Mon Jan  7 12:01:20 warning: web admin interface failed to initialize on port 28017

It seems this is a recognised problem. It seems that if you installed MongoDB the ‘sudo apt-get…’ way then Ubuntu seems to boot a mongo DB on startup (like a Windows service).

Here’s a quick fix here’s to find and kill the Mongo server process and start completely afresh.

Get the process list:

$ ps -eF | grep 'mongo\|PID'

This will return the PID(s) [1776] which can then be used to kill the process and hopefully close the sockets as well.

$ ps -ef | grep ‘mongo\|PID’

UID        PID  PPID  C STIME TTY          TIME CMD
mongodb   1776     1  0 Jan06 ?        00:00:00 /usr/bin/mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --config /etc/mongodb.conf run
stuart   17622 15178  0 12:01 pts/2    00:00:00 mongod
stuart   18021 17854  0 12:10 pts/1    00:00:00 grep mongo\|PID

$ sudo kill-9 1776

Now we’re good to go, I hope, and I can begin the process of networking/connecting!

An intro to sharding ((horizontal) scalability) & replica sets (durability)

Recommended Reading

Seven Databases in Seven Weeks (pp 165-173)
Scaling MongoDB (pp All!)
MongoDB: The Definitive Guide (Chapters 9-10)
MongoDB In Action (Chapters 8-9)

Intro to Sharding & Replication

What makes document DBs unique is their ability to efficiently handle arbitrarily nested, schemaless ‘documents’. What makes MongoDB special among document DBs is its ability to scale across several servers, by replicating (copying data to other servers) or sharding collections (splitting a collection into pieces) and performing queries in parallel. Both promote availability. More background here

Sharding has a cost: if one part of the collection is lost, the whole thing is compromised – what good is querying a database of world country facts if the southern hemisphere is down? Mongo deals with this implicit sharding weakness – via duplication.

Replication improves durability, but consequently generates issues not found in single-source databases. One problem is deciding which node gets promoted when a master node goes down. Mongo deals with this by giving each mongod service a vote, and the one with the freshest data is elected the new master.

The problem with even nodes
The concept of replication is simple enough. You write to one MongoDB server and that data is duplicated to others within the replica set. If one server is unavailable, then one of the others can be promoted and handle requests.
– MongoDB demands an odd number of total nodes in the replica set.

To see why an odd number of nodes is best, consider what might happen with a 4-node replica set. Let’s imagine two of the servers lose connectivity to the other two. One set will have the original master, but since it can’t see a clear majority of the network (2 vs 2) the master demotes. The other set also won’t be able to elect a master as it too can’t communicate with a clear majority of nodes. Both sets become unable to process requests and the system is effectively down.

Quorum

CouchDB, on the other hand allows multiple masters. Unlike Riak, MongoDB always knows the most recent data. Mongo’s concern is strong consistency on writes and a) preventing a multi-master scenario and b) Understanding the primacy of data spread across nodes are both strategies to achieve this.

SHARDING facilitates handling very large datasets and extensibility aka horizontal scaling.

Rather than having a single server holding all the data, sharding splits the data onto other servers – I tend to think about database statistics within RDBMS engines here. Perhaps the first 3 chunks of data on the histogram on the left might go on shard 1; the 4th chunk on shard 2 and chunks 5, 6 & 7 on shard 3.

A shard is one or more servers in a cluster that are responsible for some subset of the data. If we had a cluster containing 1,000,000 documents representing users of a website, on shard might contain information about 200,000 of those users.

To evenly spread data across shards, MongoDB move subsets of the data from shard to shard. It figures out which subset(s) to move where based on a key that you choose. For example, we might choose to split up a collection of users based on the username field. MongoDB used range-based splitting: that is, data is split into chunks of given ranges eg [“a”, “d”)

A shard can consist of many servers. If there is more than one server in a shard, each server has an identical copy of the subset of data. In production, a shard will usually be a replica set.

Sharding requires some kind of library/lookup. Imagine we created a table to store city names alphabetically. We need some way to know that eg cities starting A-D go to server mongo2, P-Z on mongo4 etc. In MongoDB you create a config server. Certain versions(?) of MongoDB make this easier by autosharding, managing the spread of data for you.

(Networking) Success!

With some help from my former colleague and current classmate Pete Griffiths, I managed to remote logon to MongoDB installed on the ‘clear’ Pi (with the 32GB SD Card & successful GitHub installation) via MongoVue installed on the Win7 laptop.

This was a breakthrough as I’m now confident my cluster will be able to see each device – which then allows me to begin spreading data across nodes aka sharding.

Node(s) visible

Node(s) visible

Here you can see a connection to ‘clear’ has been created. Clear has been given a fixed IP address of 10.0.0.2, yellow has been given a static IP address of 10.0.0.3, Red .4 etc etc. MongoVue is running on my laptop, but able to connect to remote devices in the network switch.

You can see the dummy document I entered into a new collection!

Image

Clear is plugged in to the TP-Link switch on port 3 (lit up; laptop in port 1).

I’m just waiting for my other 4 32GB cards to come in the post and then I should be able to install Mongo on the remaining R Pi’s then hook everything together into the network switch.

We also linuxed the old Dell using Unetbootin, removing WinXP which is incompatible with MongoDB. Raspbian is crafted for the puny ARM processor, so I had to install Debian, which I believe(?) is from the same Linux family, so I hope has more or less the same functionality and syntax,

Also created virtual machines on the laptop using Virtual Box. I think I’ll need these to create virtual ‘config servers’ which mongodb requires for coordination. Config servers store all cluster metadata, most importantly, the mapping from chunks to shards:
Config servers maintain the shard metadata in a config database. The config database stores the relationship between chunks and where they reside within a sharded cluster. Without a config database, the mongos instances would be unable to route queries or write operations within the cluster.

– see more here

Divide and conquer