Solution Resort

Learn, try and share…

Category: Linux

MySQL in Docker without losing data after rebuild

When it comes down to running database services or anything that has states in it with docker containers, the first question is often “how about my data” after the container is destroyed or rebuilt?

The simple answer is you can use Docker Data Volumes.

After reading a few articles as well as trying it out myself, the easiest and cleanest way I found is to create a Data Container with a volume first, and then tell your MySQL container to use the data volume on that container.

This can be simply done with two commands,

# creates the volume container and mount /var/lib/mysql

docker create -v /var/lib/mysql --name mysql_data_store busybox /bin/true
# start the mysql container and tell it to use the created volume on the mysql_data_store container. Do not use the -d option if you are running it with CoreOS

docker run --volumes-from mysql_data_store --name mysql -e MYSQL_ROOT_PASSWORD=<your long password> -d mysql:latest

Now if you kill and remove the MySQL container and recreate it by mounting the same data volume again, all the data you have won’t be lost because the data volume has not been destroyed. The MYSQL_ROOT_PASSWORD option will be redundant the second time you run the container as the MySQL service has already been installed.

I’ve not tried this on production yet but will do soon on some hobby projects.

More reading:
https://docs.docker.com/engine/userguide/containers/dockervolumes/
https://github.com/docker-library/docs/tree/master/mysql#where-to-store-data

Run Docker and Docker Compose in a Vagrant box

Created https://github.com/richardhe-awin/vagrant-docker recently which provisions a Vagrant VM (Ubuntu trusty) with everything necessary installed to run docker & docker compose.

The main reason I created this is because it gives you an isolated environment to run things without going through the hassle of installing docker & docker compose which can be quite annoying if you are running i.e. Mac OSX and wanted to use your own docker hub repo.

All you need to do is to install Vagrant which could be much easier than installing docker & docker compose depending on the OS you use.

Simply check it out and follow the instructions.

How to: Install a Virtual Apache Hadoop Cluster with Vagrant and Cloudera Manager on a Mac

Feel free to skip some of the steps if you already have certain packages installed

Get Cask
brew install caskroom/cask/brew-cask

Get Vagrant & Vagrant plugins
brew cask install virtualbox
brew cask install vagrant
brew cask install vagrant-manager
vagrant plugin install vagranthostmanager

Install Hadoop
git clone git@github.com:richardhe-awin/vagrant-hadoop-cluster.git
cd vagrant-hadoop-cluster
vagrant up

Configure Cloudera Manager (mostly referenced from http://blog.cloudera.com/blog/2014/06/how-to-install-a-virtual-apache-hadoop-cluster-with-vagrant-and-cloudera-manager/)

  1. Go to http://hadoop-master:7180/ (you might have to wait for a few minutes for the service to boot up before this is available)  and login with admin/admin
  2. Choose to use the Express version and continue
  3. When you are asked to enter the host names, enter hadoop-node1 and hadoop-node2 and click search. You should see the two hosts coming up and confirm.
  4. Keep using the default option until you got to the page asking “Login to all hosts as”. Change this to “Another user” and enter “vagrant” as the username and enter “vagrant” again for the password fields. Click next and it should start installing (this will take a while).
  5. On the “Cluster Setup” page, choose “Custom Services” and select the following: HDFS, Hive, Hue, Impala, Oozie, Solr, Spark, Sqoop2, YARN and ZooKeeper. Click Continue.
  6. On the next page, you can select what services end up on what nodes. Usually Cloudera Manager chooses the best configuration here, but you can change it if you want. For now, click Continue.
  7. On the “Database Setup” page, leave it on “Use Embedded Database.” Click Test Connection (it says it will skip this step) and click Continue.
  8. Click Continue on the “Review Changes” step. Cloudera Manager will now try to configure and start all services.
  9. Done!

 

Mount Shared Folder to Ubuntu if auto-mount doesn’t work

I had some issues with mounting shared folders with Virtual Box because they don’t appear anywhere. I finally figured out that the Shared drive is already there, it’s just not showing anywhere.

Before continuing, making sure you have installed the latest version of Guest Additions and your user is added to the vboxsf group.

First, add a shared folder under Devices -> Shared Folder Settings with the following properties (I am using windows in this example),

Folder Path: C:your-folder
Folder Name: shared-name-on-ubuntu

Go to a folder, e.g. ~/ and run

mkdir ~/shared-name-on-ubuntu

Finally mount it

sudo mount -t vboxsf shared-name-on-ubuntu ~/shared-name-on-ubuntu

And now you should be able to see everything shared within Ubuntu.

 

Ant returns code 137

I had this problem at work when running php unit tests on Jenkins (CI server) and it took the whole team ages to figure out this problem. I thought it’s something worth sharing.

We use Jenkins to run ant which invokes our unit tests and to speed up the process we are running all the unit tests in one single process (this means we are not doing phpunit –process-isolation which runs each test in a separate PHP process. The up side of using this option is it uses a lot less memory, but the down side is it’s significantly slower). The whole process consumes around 500 mb of memory in one single process but one day we started getting the following error,

BUILD FAILED
<http://jenkins/project/build.xml>:245: The following error occurred while executing this line:
/path..to..build/build.xml:66: exec returned: 137

We thought it might be something related to memory but we weren’t sure about it until one day we checked the kernel log (by typing in dmesg) and we get,

Out of memory: Kill process 16394 (phpunit) score 193 or sacrifice child
Killed process 16394, UID 497, (phpunit) total-vm:791308kB, anon-rss:348416kB, file-rss:4kB

And we know it’s memory related for sure, after increased the virtual memory allowance to 4G, it solved the problem.

However if you don’t have the luxury to increase your memory limit, try to use the –process-isolation option which would only use a tiny amount of memory to run all the tests but I found this takes 5-6 times longer to run all tests.

 

Hope this helps.

Debian Zero To Hero Guide (best for PHP developers)

Took me quite a while to pull all these information together. This guide should install most basic packages you will need to get started with PHP Development.

*This was tested with Debian 7.1.0

Before we start, please launch a root terminal to save typing a million sudo commands. Short cut key: Alt + F2, search for terminal and run Sudo Terminal

Java First (Get rid of Open JDK and install Oracle JDK)

Found these from (http://www.webupd8.org/2012/06/how-to-install-oracle-java-7-in-debian.html)

echo “deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main” | tee -a /etc/apt/sources.list
echo “deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main” | tee -a /etc/apt/sources.list
apt-key adv –keyserver keyserver.ubuntu.com –recv-keys EEA14886
apt-get update
apt-get install oracle-java7-installer

#install this package to automatically set the Java Environment variables
apt-get install oracle-java7-set-default

Basic Stuff

This includes Apache2, PHP5 ,Curl for PHP, Ant, PHP Internationalisation, Tidy, MySQL, PHP-MySQL, Gnome system tools

apt-get install apache2 -y
apt-get install php5 -y
apt-get install php5-curl -y
apt-get install ant -y
apt-get install php5-intl -y
apt-get install php5-tidy -y
apt-get install mysql-server mysql-client -y
apt-get install php5-mysql -y
apt-get install ssh -y
apt-get install gnome-system-tools -y

Apache Authentication

a2enmod auth_digest //enable digest authentication module

Image libraries

apt-get install php5-imagick
apt-get install php5-gd

MySQL Optimization for InnoDB

This will significant increase the query time for InnoDB but it’s important to leave this as 1 on a production environment if you wish MySQL to write to the log on every commit. For more information, see http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html

innodb_flush_log_at_trx_commit = 0

Add a user to the sudo group

The registered user doesn’t come under the sudo group by default, if you wish to install packages or run sudo commands with the current user, add the user to the sudo group by typing in,

adduser <username> sudo

You must re-login for this to take effect

Unmanaged Network Issue

You may encounter this issue when setting up multiple network adapters (the direct impact would be internet connection doesn’t work). This is usually down to this file /etc/network/interfaces having instructions telling the network manager that you want to manage eth0 manually.

To fix it, simply go to this file and make sure the following lines are commented out

#allow-hotplug eth0
#iface eth0 inet dhcp

Install Samba

apt-get install libcupsys2 samba samba-common

To share a directory, edit /etc/samba/smb.conf and add the following lines to the bottom of the file,

[home]
comment = debian.www
path = /home/<username>/www
browseable = yes
valid users = @<username>
writable = yes
create mask = 0700
directory mask = 0700

Restart Samba by typing in

service samba restart

More to follow…

Ubuntu Zero To Hero Guide (best for PHP developers)

Took me quite a while to pull all these information together. This guide should install most basic packages you will need to get started with PHP Development.

*This was tested with Ubuntu 13.04 but most things should also work on Ubuntu 12.

Java First

#remove open jdk and install oracle jdk

sudo apt-get purge openjdk*

sudo apt-get autoremove

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java7-installer

 

Basic Stuff

sudo apt-get install apache2 -y

sudo apt-get install php5-curl

sudo apt-get install ant -y

sudo apt-get install php5-intl

sudo apt-get install php5-tidy

sudo apt-get install mysql[*] (type in mysql and it will tell you what version to install)

sudo apt-get install mysql-server -y

sudo apt-get install php5 -y

sudo apt-get install php5-mysql -y

sudo apt-get install ssh -y

sudo apt-get install gnome-system-tools -y

#the PECL_HTTP extension
sudo apt-get install libcurl3-openssl-dev -y
sudo pecl install pecl_http -y
#when this is done, add “extension=http.so” in /etc/php/apache2/php.ini under the “Dynamic Extensions” section

Apache Authentication

sudo a2enmod auth_digest //install digest authentication module

Image libraries

sudo apt-get install php5-imagick
sudo apt-get install php5-gd

#I think it makes sense not to use imagick and gd directly but to use Imagine which is a wrapper for the two available on packagist. It has a much user friendly API which I highly recommend if you want a time saver.
Here is the link to it on packagist https://packagist.org/packages/imagine/imagine

 

MySQL Optimization for InnoDB, (This will significant increase the query time for InnoDB but it’s important to leave this as 1 on a live environment. For more information, see http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html)

innodb_flush_log_at_trx_commit = 0

#To debug slow queries, set the following configuration options

log_queries_not_using_indexes = on
slow_query_log = on
log_output = table

and then look at the ‘slow_query’ table in the ‘mysql’ database.

Install MongoDB

sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 7F0CEB10
echo ‘deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen’ | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install mongodb-10gen

#gui tool
http://robomongo.org/

Samba

If your Ubuntu is installed as a Virtual Machine, it’s very likely you’ll come cross using Samba to share folders so that a mapped drive can be created in your native OS to access them directly.

The easiest way to do this is by right clicking on a folder and select Sharing Options, then tick the Share this folder box and the Allow others to create and delete files in this folder box. Save your changes.

Once this is done, you’ll need to manually create a Samba user to gain access to the shared folder. This can be simply done by,

sudo smbpasswd -a <username>

 

Xdebug

#xdebug

sudo apt-get install php5-dev -y

sudo apt-get install php-pear -y

sudo pecl install xdebug

#then add the following lines into /etc/php5/apache2/php.ini

[XDebug]

zend_extension=”/usr/lib/php5/[*date*]/xdebug.so”

xdebug.remote_enable=1

xdebug.remote_handler=dbgp

xdebug.remote_mode=req

xdebug.remote_host=127.0.0.1

xdebug.remote_port=9000

#finally make sure xdebug shows in phpinfo();

Xdebug – CMD setup

edit /etc/php5/apache2/php.ini and add the following section on the bottom,

[XDebug]

zend_extension=”/usr/lib/php5/[*date*]/xdebug.so”

xdebug.remote_enable=1

xdebug.remote_handler=dbgp

xdebug.remote_mode=req

xdebug.remote_host=127.0.0.1

xdebug.remote_port=9000

I am using Netbeans as my IDE, Type in

export XDEBUG_CONFIG=idekey=netbeans-xdebug on command line and start Netbean debug session to listen on the same key.

Now, simply run phpunit <Test Script Name>.php should start the debug session.

If you need to stop debugging from command line, type in
unset XDEBUG_CONFIG

 

Rabbitvcs (svn client gui on linux)

http://wiki.rabbitvcs.org/wiki/install/ubuntu

#the following line is a fix for utuntu 13.04 (please note that in Ubuntu 13.04, you can only bring up the GUI from command line. E.g. rabbitvcs commit)

sudo ln -s /usr/lib/x86_64-linux-gnu/libpython2.7.so /usr/lib/libpython2.7.so.1.0

 

Firewall GUI tool

search gufw from software manager

 

VHost Setup

First, follow the guide from

https://www.digitalocean.com/community/articles/how-to-set-up-apache-virtual-hosts-on-ubuntu-12-04-lts

Just a quite note, to enable or disable a vhost,

Enable: sudo a2ensite [site-name]

Disable: sudo a2dissite [site-name]

And,

#Enable rewriteengine

a2enmod rewrite

In order to use the .htaccess file, you need to set

AllowOverRide All within the Apache config file

Nodejs

#install node

sudo apt-get install g++ curl libssl_dev apache2-utils git-core
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

If these commands don’t work, check http://slopjong.de/2012/10/31/how-to-install-the-latest-nodejs-in-ubuntu/

Bower (A Package Manager for the web)

sudo apt-get install npm

Then follow https://github.com/bower/bower

 

Virtual Box Mount new Disk (Centos 6)

This is a step to step guide mounting a disk to Centos 6 running on Virtual Box.

  1. Add a new disk under virtualbox setting -> storage.

  2. Set the new disk as Primary Slave

  3. Boot up centos and type in fdisk -l

You should see a disk called Disk /dev/sdb something like ,

Disk /dev/sdb: 12.8 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  1. Run mkfs -t ext3 /dev/sdb will create a partition for the drive

  2. Mount the drive to /mnt/disk-extended by typing the following commands

mkdir /mnt/disk-extended
mount -t ext3 /dev/sdb /mnt/disk-extended

  1. Finally, automount the disk on boot by adding a line to /etc/fstab

LABEL=79d3d2d4  / ext4 defaults,noatime 0 0
LABEL=e21a3e10  /boot ext3 defaults,noatime 0 0
devpts /dev/pts  devpts  gid=5,mode=620   0 0
tmpfs   /dev/shm  tmpfs   defaults      0 0
proc    /proc proc defaults      0 0
sysfs   /sys   sysfs   defaults      0 0
/dev/sdb   /mnt/disk-extended   ext3  defaults   0 0

SSH Public / Private Key Setup

Create Public / Private key pairs

ssh-keygen -t [Type of the key]  -b [Number of bits in the key to create]

e.g. ssh-keygen -t rsa -b 4096

Copy public key to remote server

ssh-copy-id -i [path to public key] [username]@[host]

e.g. ssh-copy-id -i keys/sample_public_key.pub dummyuser@dummyhost.com

 

p.s.

Public key is always associated with a user when it gets copied onto the remote server. The default location storing the keys is usually ~/[user]/.ssh/. All public keys are appended to the ~/[user]/.ssh/authorized_keys file.

 

Useful Linux Commands

Commonly used Linux commands.

//change permissions recursively on Directories:
find . -type d -exec chmod XXX {} ;

//change permissions recursively on Files:
find . -type f -exec chmod XXX {} ;

//extract .tar.gz file:
tar -zxvf [file].tar.gz

//logout
logout

//switch user:
su [username]

//change owner of a directory recursively
chown -Rv [username] [somedir]

//change group of a directory recursively
chgrp -Rv [usergroup] [somedir]

//add a user and add to group
useradd -G [group-name] [username]

//add a group
groupadd [groupname]

//add an existing user to group
usermod -a -G [group name] [username]

//SSH into another server
ssh -i [path/to/private/keyfile] [username]@[ip]

//Find files / folders greater than a size
du -h / | grep ^[0-9.]*G
du -h /usr/ | grep ^[1-9][0-9][0-9][0-9]*M

Powered by WordPress & Theme by Anders Norén