MongoDB startup troubleshooting : “unexpected shutdown”, “Address already in use for port”

I’m having another attempt and building out my cluster today. Following Simon the PiMan’s guides

  1. http://www.simonthepiman.com/beginners_guide_find_my_network_settings.php
  2. simonthepiman.com/how_to_setup_your_pi_for_the_internet.php
  3. http://www.simonthepiman.com/how_to_setup_remote_access.php#raspberrypi

I just want to get one or 2 R Pis connected to the network switch and the Dell (acting as master).

I fire up the Dell and call MongoD (which I haven’t used for a couple of weeks), to get a surprising error:

old lock file: /data/db/mongod.lock.  probably means unclean shutdown recommend removing file and running --repair see: http://dochub.mongodb.org/core/repair for more information

The right way to shut down Mongo

I guess I must have done something wrong the last time I had it open. To ensure a clean shut down, use the mongod –shutdown option, your control script, “Control-C” (when running mongod in interactive mode,) or kill $(pidof mongod) or kill -2 $(pidof mongod).

Finding out/remembering where you installed MongoDB

$ sudo find / -type d -name mongo
This will search  from root (/) for directories named ‘mongo’
Reveals I installed everything to /usr/bin

Find out where you installed the directories for $DBPATH, check the filesize for mongod.lock

$ locate /data/db
/data/db- Note to self, create this in the same tree as the Mongo directories next time!

cd /data/db
ls -l
-rwxr-xr-x 1 stuart stuart 0 Jan  6 22:00 mongod.lock

If the mongod.lock file in the data directory specified by dbpath, /data/db by default, is not a zero-byte file, then mongod will refuse to start.

However, as above, you can see that the filesize is 0 bytes for mongod.lock, so, I’m not quite sure what the problem is.

Repairing MongoDB

$ sudo mkdir /data/db/db0
– Mongo attempts to create a new directoryand move the old/repaired lock file. As I seem to have some ongoing permissions problems, i pre-create this folder.

$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0

Feedback from the shell below suugests it has worked…

stuart@debian:/data/db$ sudo mongod –dbpath /data/db –repair –repairpath /data/db0
Sun Jan  6 22:33:30 Mongo DB : starting : pid = 16652 port = 27017 dbpath = /data/db master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Sun Jan  6 22:33:30 finished checking dbs
Sun Jan  6 22:33:30  dbexit:
Sun Jan  6 22:33:30      shutdown: going to close listening sockets…
Sun Jan  6 22:33:30      shutdown: going to flush oplog…
Sun Jan  6 22:33:30      shutdown: going to close sockets…
Sun Jan  6 22:33:30      shutdown: waiting for fs preallocator…
Sun Jan  6 22:33:30      shutdown: closing all files…
Sun Jan  6 22:33:30      closeAllFiles() finished
Sun Jan  6 22:33:30      shutdown: removing fs lock…
Sun Jan  6 22:33:30  dbexit: really exiting now

Or, to wipe your data files without preserving the original files, do not use the –repairpath option, as in the following procedure:

  1. Remove the stale lock file:
    rm /data/db/mongod.lock

    Replace /data/db with your dbpath where your MongoDB instance’s data files reside.

    Warning: After you remove the mongod.lock file you must run the –repair process before using your database.

  2. Start mongod using –repair to read the existing data files.
    mongod --dbpath /data/db --repair

    When this completes, the repaired data files will replace the original data files in the /data/db directory.

  3. Start mongod using the following invocation to point the dbpath at /data/db:
    mongod --dbpath /data/db

Port already in use?!

After repairing Mongo by clearing out the old lock, I try to start it afresh.

stuart@debian:/data/db$ mongod
mongod –help for help and startup options
Mon Jan  7 12:01:20 Mongo DB : starting : pid = 17622 port = 27017 dbpath = /data/db/ master = 0 slave = 0  32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more

Mon Jan  7 12:01:20 db version v1.4.4, pdfile version 4.5
Mon Jan  7 12:01:20 git version: nogitversion
Mon Jan  7 12:01:20 sys info: Linux murphy 2.6.32.14-dsa-ia32 #1 SMP Thu May 27 16:19:20 CEST 2010 i686 BOOST_LIB_VERSION=1_42
Mon Jan  7 12:01:20 waiting for connections on port 27017
Mon Jan  7 12:01:20 listen(): bind() failed errno:98 Address already in use for port: 27017
Mon Jan  7 12:01:20 MiniWebServer: bind() failed port:28017 errno:98 Address already in use
Mon Jan  7 12:01:20   addr already in use
Mon Jan  7 12:01:20 warning: web admin interface failed to initialize on port 28017

It seems this is a recognised problem. It seems that if you installed MongoDB the ‘sudo apt-get…’ way then Ubuntu seems to boot a mongo DB on startup (like a Windows service).

Here’s a quick fix here’s to find and kill the Mongo server process and start completely afresh.

Get the process list:

$ ps -eF | grep 'mongo\|PID'

This will return the PID(s) [1776] which can then be used to kill the process and hopefully close the sockets as well.

$ ps -ef | grep ‘mongo\|PID’

UID        PID  PPID  C STIME TTY          TIME CMD
mongodb   1776     1  0 Jan06 ?        00:00:00 /usr/bin/mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --config /etc/mongodb.conf run
stuart   17622 15178  0 12:01 pts/2    00:00:00 mongod
stuart   18021 17854  0 12:10 pts/1    00:00:00 grep mongo\|PID

$ sudo kill-9 1776

Now we’re good to go, I hope, and I can begin the process of networking/connecting!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s