MongoDB startup troubleshooting : “unexpected shutdown”, “Address already in use for port”

I’m having another attempt and building out my cluster today. Following Simon the PiMan’s guides

  1. http://www.simonthepiman.com/beginners_guide_find_my_network_settings.php
  2. simonthepiman.com/how_to_setup_your_pi_for_the_internet.php
  3. http://www.simonthepiman.com/how_to_setup_remote_access.php#raspberrypi

I just want to get one or 2 R Pis connected to the network switch and the Dell (acting as master).

I fire up the Dell and call MongoD (which I haven’t used for a couple of weeks), to get a surprising error:

old lock file: /data/db/mongod.lock.  probably means unclean shutdown recommend removing file and running --repair see: http://dochub.mongodb.org/core/repair for more information

The right way to shut down Mongo

I guess I must have done something wrong the last time I had it open. To ensure a clean shut down, use the mongod –shutdown option, your control script, “Control-C” (when running mongod in interactive mode,) or kill $(pidof mongod) or kill -2 $(pidof mongod).

Finding out/remembering where you installed MongoDB

$ sudo find / -type d -name mongo
This will search  from root (/) for directories named ‘mongo’
Reveals I installed everything to /usr/bin

Find out where you installed the directories for $DBPATH, check the filesize for mongod.lock

$ locate /data/db
/data/db- Note to self, create this in the same tree as the Mongo directories next time!

cd /data/db
ls -l
-rwxr-xr-x 1 stuart stuart 0 Jan  6 22:00 mongod.lock

If the mongod.lock file in the data directory specified by dbpath, /data/db by default, is not a zero-byte file, then mongod will refuse to start.

However, as above, you can see that the filesize is 0 bytes for mongod.lock, so, I’m not quite sure what the problem is.

Repairing MongoDB

$ sudo mkdir /data/db/db0
– Mongo attempts to create a new directoryand move the old/repaired lock file. As I seem to have some ongoing permissions problems, i pre-create this folder.

$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0

Feedback from the shell below suugests it has worked…

stuart@debian:/data/db$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0
Sun Jan  6 22:33:30 Mongo DB : starting : pid = 16652 port = 27017 dbpath = /data/db master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Sun Jan  6 22:33:30 finished checking dbs
Sun Jan  6 22:33:30  dbexit:
Sun Jan  6 22:33:30      shutdown: going to close listening sockets…
Sun Jan  6 22:33:30      shutdown: going to flush oplog…
Sun Jan  6 22:33:30      shutdown: going to close sockets…
Sun Jan  6 22:33:30      shutdown: waiting for fs preallocator…
Sun Jan  6 22:33:30      shutdown: closing all files…
Sun Jan  6 22:33:30      closeAllFiles() finished
Sun Jan  6 22:33:30      shutdown: removing fs lock…
Sun Jan  6 22:33:30  dbexit: really exiting now

Or, to wipe your data files without preserving the original files, do not use the –repairpath option, as in the following procedure:

  1. Remove the stale lock file:
    rm /data/db/mongod.lock

    Replace /data/db with your dbpath where your MongoDB instance’s data files reside.

    Warning: After you remove the mongod.lock file you must run the –repair process before using your database.

  2. Start mongod using –repair to read the existing data files.
    mongod --dbpath /data/db --repair

    When this completes, the repaired data files will replace the original data files in the /data/db directory.

  3. Start mongod using the following invocation to point the dbpath at /data/db:
    mongod --dbpath /data/db

Port already in use?!

After repairing Mongo by clearing out the old lock, I try to start it afresh.

stuart@debian:/data/db$ mongod
mongod –help for help and startup options
Mon Jan  7 12:01:20 Mongo DB : starting : pid = 17622 port = 27017 dbpath = /data/db/ master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Mon Jan  7 12:01:20 db version v1.4.4, pdfile version 4.5
Mon Jan  7 12:01:20 git version: nogitversion
Mon Jan  7 12:01:20 sys info: Linux murphy 2.6.32.14-dsa-ia32 #1 SMP Thu May 27 16:19:20 CEST 2010 i686 BOOST_LIB_VERSION=1_42
Mon Jan  7 12:01:20 waiting for connections on port 27017
Mon Jan  7 12:01:20 listen(): bind() failed errno:98 Address already in use for port: 27017
Mon Jan  7 12:01:20 MiniWebServer: bind() failed port:28017 errno:98 Address already in use
Mon Jan  7 12:01:20   addr already in use
Mon Jan  7 12:01:20 warning: web admin interface failed to initialize on port 28017

It seems this is a recognised problem. It seems that if you installed MongoDB the ‘sudo apt-get…’ way then Ubuntu seems to boot a mongo DB on startup (like a Windows service).

Here’s a quick fix here’s to find and kill the Mongo server process and start completely afresh.

Get the process list:

$ ps -eF | grep 'mongo\|PID'

This will return the PID(s) [1776] which can then be used to kill the process and hopefully close the sockets as well.

$ ps -ef | grep ‘mongo\|PID’

UID        PID  PPID  C STIME TTY          TIME CMD
mongodb   1776     1  0 Jan06 ?        00:00:00 /usr/bin/mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --config /etc/mongodb.conf run
stuart   17622 15178  0 12:01 pts/2    00:00:00 mongod
stuart   18021 17854  0 12:10 pts/1    00:00:00 grep mongo\|PID

$ sudo kill-9 1776

Now we’re good to go, I hope, and I can begin the process of networking/connecting!

Advertisements

Distribution, Replica sets, Master-Slave

I’m just getting to grips with distributed database terminology pp130 in “MongoDB The Definitive Guide”

Master-Slave Replication

Master-slave replication  can be used for

  • Backup
  • Failover
  • Read scaling, and more

Read scaling could be really interesting from a BI consumption point-of-view eg a peak of user traffic to the DW portal at the beginning of the day.

In MongoDB the most basic setup is to start a master node (in my case the Dell 5100)  and add one or more slave nodes (in my case, the Raspberry Pis) Each of the slaves must know the address of the master.

To start the master run: mongod –master

To start a slave run: mongod –slave –source master_address
– where master_address is the address of the master node just started. This is where my previous post on fixing a static IP comes in handy.

First, create a directory for the master to store data in and choose a port (10000)

$ mkdir -p ~/dbs/master
$ ./mongod --dbpath ~/dbs/master --port 10000 --master

Now, set up the slave(s) choosing a different data directory (if on same/virtual machine) and port. For any slave, you also need to specify who the master is

$ mkdir -p ~/dbs/slave
$ ./mongod --dbpath ~/dbs/slave --port 10001 --slave --source localhost:10000

All slaves must be replicated from a master node. It is not possible to replicate from slave to slave.

…As soon as my 32MB SD cards arrive in the post and I install MongoDB on the remaining 4 R PIs I will give this a go!

Replica Sets

replica set is the same as the above but with automatic failover. The biggest difference between a M-S cluster and a replica set is that a replica set does not have a single master.

One is ‘elected’ by the cluster and may change to another node if the then current master becomes uncontactable.

 

There are 3 different types of nodes which can co-exist in a MongoDB cluster

  1. Standard – Stores a complete, full copy of the data being replicated, takes part in the voting when the primary node is being elected and is capable of being the primary node in the cluster
  2. Passive – As above, but will never become the primary node for the set
  3. Arbiter – Participates only in voting. Does not receive any of the data being replicated and cannot become the primary node

Standard and passive nodes are configured using the priority key. A node with priority 0 is passive and will never be selected as primary.

I suppose in my case if I had decreasing SD cards, or a mix of model A (256 MB) and model B (512 MB) Pis, then I could set priorities in decreasing order so the weakest node was never the master and always selected last/lowest priority.

Initializing a set (pp132)