Backup MySql Docker Container

Here is how you can make mysqldump on container that created from mariadb image

docker run -it --link db_1:mysql --rm mariadb sh -c 'exec mysqldump -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD" wordpress' > /backup/wordpress-$(date +\%F).sql

This command does the following:

  1. creates new container from mariadb image
  2. configure a link to your db container (db_1)
  3. run the mysqldump command inside the new container
  4. save the output of mysqldump command to a file
  5. remove the new container

Create mysql replication

Create mysql replication is a simple procedure that usually can be done with the following steps:

  1. enable bin-log on your master
    /etc/my.cnf
    [mysqld]
    # Replication
    server-id = 1
    relay-log = mysql-relay-bin
    log-bin=mysql-bin
  2. create replication user
    mysql
    mysql> CREATE USER 'repl'@'%.mydomain.com' IDENTIFIED BY 'slavepass';
    mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%.mydomain.com';
    
  3. lock your database and write master position
    mysql> FLUSH TABLES WITH READ LOCK;
    mysql> SHOW MASTER STATUS;
  4. take mysql dump of the database
    mysqldump --all-databases --master-data > fulldb.dump
  5. unlock the database
    mysql> UNLOCK TABLES;
  6. prepare mysql slave server
    /etc/my.cnf
    [mysqld]
    server-id=2
    relay-log = mysql-relay-bin
    log-bin=mysql-bin
  7. restore mysql data
    mysql < fulldb.dump
  8. start replication on the slave server with the change master command
    mysql> CHANGE MASTER TO
        ->     MASTER_HOST='master_host_name',
        ->     MASTER_USER='replication_user_name',
        ->     MASTER_PASSWORD='replication_password',
        ->     MASTER_LOG_FILE='recorded_log_file_name',
        ->     MASTER_LOG_POS=recorded_log_position;
    
    mysql> START SLAVE;

but if you have very big database let say 1TB and you can’t except downtime?

If you prepare right you storage or you are using cloud services then you can lock the database for a few seconds take a snapshot and then copy the data from the snapshot.

if you didn’t prepare right mysql storage then you need to use the right flags in mysqldump command.

These are the flags that I used (relevant for transactional DB like InnoDB):

mysqldump --all-databases --master-data=2 --single-transaction --quick | gzip > outputfile.sql.gz
--all-databases - Used to backup all the databases in mysql server
--master-data=2 - Writes binary log name and position in mysql remark to the dump file
--single-trasaction - This is an important flag that send start trasaction to the mysql server and dump the consistent state of the database at the time when start transaction started. this flag let you use the database while the dump is running. The flag is usefull only for transactional tables like InnoDB.
--quick - Used for large tables to retrieve rows from a table one raw at a time instead of retrieving the entire row set and buffer it in memory before writing it.

To me the dump took about a day and then I restore it with the following command:

gunzip -c outputfile.sql.gz | mysql

The restore took me much longer, it was about 4-5 days. If you have other methods to make the dump or restore faster please let me know.

After the restore we need to run the change master command so we need to grub it from the dump file:

zcat all_db.sql.gz | head -n 200 | grep "CHANGE MASTER"

mysql
mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.xxxx', MASTER_LOG_POS=1111133333;
mysql> start slave;

To check the slave status use the following command:

mysql> SHOW SLAVE STATUS\G;

check that Slave_IO and Slave_SQL are running and wait for the Seconds_Behind_Master to decrease to 0 (to me it took ~4 days).

On the new slave server that I created I installed LVM with enough free space for snapshots so next time I can do the following:

  1. lock mysql databases
  2. flush the tables
  3. get master binary log file and position
  4. create LVM snapshots
  5. unlock mysql databases
  6. rsync the data to another server

These steps should take much less time then mysqldump and restore.

During this work I got help from the following links:

  1. mysql docs – http://dev.mysql.com/doc/refman/5.7/en/replication-howto.html
  2. mysql docs – https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_quick
  3. server fault – http://serverfault.com/questions/220322/how-to-setup-mysql-replication-with-minimal-downtime

 

Write processes data to kibana

Short example on how to write the top 10 processes that consume memory

  • Install logstash
  • Create the following configuration
input { 
 exec {
 command => 'ps aux --sort -rss | head -10 | tail -9'
 interval => 60
 type => 'processes_by_mem'
 } 
}

filter {
 split { }
 csv {
 columns => ['USER','PID','%CPU','%MEMORY','VSZ','RSS','TTY','STAT','START','TIME','COMMAND']
 separator => ' '
 }
 mutate {
 convert => { 
 "%MEMORY" => "integer"
 "%CPU" => "integer"
 }
 }
}

output {
 redis {
 host => 'elk-redis1.nyj.taptica.info'
 port => 6379
 key => 'logstash:redis'
 data_type => 'list'
 }
}

Explanation:

  • I use exec input that run the ps command which sort process by memory consumption
  • I use split filter to split the lines from the command (each line is different event)
  • I use the csv filer with a space separator and give for each filed from the command output a name using columns  property
  • I use mutate filter to convert %CPU and %MEMORY to integer so I can sort in kibana by this field
  • I use redis output as a buffer to elasticsearch because I have another logstash process that reads from this redis and forward all data to elasticsearch

Mysql authenticate using login-path

Mysql version: 5.6.6

Today I wanted to write a script that dump DB from remote mysql server, import to staging and run this script every week.

I didn’t want to write mysql credential in the script so I search for alternative and I found the mysql-config-editor .

To use it I needed to create two configuration: production and staging and I did it like that:

mysql_config_editor set --login-path=prod-db --host=proddb.nachum234.com --user=humus --password
mysql_config_editor set --login-path=localhost --user=humus --password

The command above create encrypted file in current user home directory called .mylogin.cnf.

To check the configuration you can use the following command:

mysql_config_editor print --all

To use this config file just use the –login-path in mysql commands.
Examples:

mysqldump --login-path=prod-db db_name > prod_db_name-$(date +%F).sql
mysql --login-path=localhost db_name < prod_db_name-$(date +%F).sql

001. How To Install Apache Solr On Apache Tomcat

Tested On

OS: CentOS 6.4 x86_64
Solr Version: 4.4
Tomcat Version: 6.0.24.0
Hardware: Virtual Box 4.2.12

About

Quote from Solr site:
“Solr is popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world’s largest internet sites”

Install Solr

  • Update your operating system and reboot
yum update -y
reboot
  • Install tomcat and prerequisite packages
yum install wget tomcat6 java-1.6.0-openjdk apr -y
  • Configure tomcat to start at boot
chkconfig tomcat6 on
  • Download and install solr
mkdir /usr/local/src/solr
cd /usr/local/src/solr
wget http://apache.spd.co.il/lucene/solr/4.4.0/solr-4.4.0.tgz
cd /opt/
tar zvf /usr/local/src/solr/solr-4.*.tgz
mv solr-4.* solr
cp solr/dist/solr-4.4.0.war /opt/solr/example/solr/solr.war
chown tomcat:tomcat -R /opt/solr
chown tomcat:tomcat /usr/share/tomcat6
  • Add UTF-8 URI encoding option to apache tomcat connector
vi /etc/tomcat6/server.xml
...
<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               URIEncoding="UTF-8" />
....
  • Create tomcat configuration file for the Solr application
vi /etc/tomcat6/Catalina/localhost/solr.xml
<Context path="/solr" docBase="/opt/solr/example/solr/solr.war" debug="0" crossContext="true">
        <Environment name="solr/home" type="java.lang.String" value="/opt/solr/example/solr" override="true"/>
</Context>
  • Copy Solr extensions files and log4j configuration file to tomcat
cp /opt/solr/example/lib/ext/* /usr/share/java/tomcat6/
cp /opt/solr/example/resources/log4j.properties /usr/share/java/tomcat6/
  • Create velocity log file
touch /usr/share/tomcat6/velocity.log
chown tomcat:tomcat /usr/share/tomcat6/velocity.log
  • Start tomcat daemon
service tomcat6 start
  •  Browse to Solr application using your web browser to check that it works
    http://192.168.10.12:8080/solr/

That’s all. Now you can use your Solr server to index and search new documents

Useful links:

  • http://lucene.apache.org/solr/

005. Configure Graylog2 To Send Alarm Notifications

Tested On

OS: CentOS 6.3 x86_64
Graylog2-Server Version: 0.11.0
Graylog2-web-interface: 0.11.0
Hardware: VMware Player 5.0.1

About

In this guide I will configure graylog2 to send email notification on stream alarms.

Prerequisite

Before using this guide you need a running graylog2 server. You can use this Graylog2 Installation guide to install graylog2 server

Configure Graylog2 Email Notification

  • Configure email transport section in graylog2 server configuration file
vi /etc/graylog2.conf
...
# Email transport
transport_email_enabled = true
transport_email_hostname = mail.example.com
transport_email_port = 587
transport_email_use_auth = true
transport_email_use_tls = true
transport_email_auth_username = [email protected]
transport_email_auth_password = secret
transport_email_subject_prefix = [graylog2]
transport_email_from_email = [email protected]
transport_email_from_name = Graylog2
...
  • Restart graylog2-server
service graylog2-server restart
  • After graylog2-server restart completed you can enter the graylog2-web-interface and see that a new field for email address is now added on the edit and create new user page
  • Go to users and add the email address for each user that you have on your graylog2 server and want to send him email notifications
  • Go to each stream that you have on your graylog2 and configure the alarms setting
  • Check that you are getting email notifications that matches your streams alarms configuration

Thats it, very simple after you know how to do it 🙂

More guides in Graylog2 Category.

Please visit http://www.graylog2.org for more information about Graylog2 configuration and usage.

004. NXlog To Graylog2 Configuration

Tested On

OS: CentOS 6.3 x86_64
Graylog2-Server Version: 0.11.0
Graylog2-web-interface Version: 0.11.0
NXlog Version: nxlog-ce-2.2
Hardware: Virtual Box 4.2.8

About

NXlog is a great tool to use to send your log files to graylog2 server.

In this guide a will show how to install nxlog with a simple configuration that set nxlog to listen for messages from a local file and send them to graylog2 server

Install NXlog

  • Download and install nxlog
  • Download and extract graylog2-radio
cd /usr/local/src
wget http://sourceforge.net/projects/nxlog-ce/files/nxlog-ce-2.3.1027-1.x86_64.rpm
yum localinstall nxlog-ce-*.rpm -y
  • Configure NXlog to send logs to graylog2 server
vi /etc/nxlog.conf
########################################
# Global directives                    #
########################################
User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules                              #
########################################
<Extension gelf>
    Module      xm_gelf
</Extension>

<Input in>
        Module  im_file
        File    "/var/log/messages"
</Input>

<Output out>
    Module      om_udp
    Host        graylog2-server.local
    Port        12201
    OutputType  GELF
</Output>

########################################
# Routes                               #
########################################
<Route r>
    Path        in => out
</Route>
  • Configure NXlog to start at boot and start it
chkconfig nxlog on
service nxlog start

That’s all. Now you can check your new log messages in graylog2 server

More guides in Graylog2 Category

Useful links:

003. Graylog2-Radio Installation

Tested On

OS: CentOS 6.3 x86_64
Graylog2-Server Version: 0.11.0
Graylog2-web-interface: 0.11.0
Graylog2-Radio: 0.10.0
Hardware: Virtual Box 4.2.8

About

Graylog2-Radio is an add-on for Graylog2-Server that help you send your logs  to an AMQP topic exchange.
With Graylog2-Radio you can avoid problems such as full buffers and rejected messages and also you can stop your graylog2-server without losing any message because they are getting written to an AMQP server.
What graylog2-radio does is listen to a tcp/udp port for syslog or gelf messages and  publish them in an AMQP exchange, then graylog2-server can be configured to consume these messages and store them in elasticsearch.

In this guide I will show how to install graylog2 radio, rabbitmq and configure your graylog2-server to consume messages from rabbitmq.

I am going to use one server for all roles but it’s recommended to use different servers.

Prerequisite

Install Graylog2-Radio

  • Install RabbitMQ server using the following guide: RabbitMQ Installation
  • Download and extract graylog2-radio
mkdir /usr/local/src/graylog2
cd /usr/local/src/graylog2
wget http://download.graylog2.org/graylog2-radio/graylog2-radio-1.0.0.tar.gz -O graylog2-radio.tar.gz
tar zxf graylog2-radio.tar.gz
rm -f graylog2-radio.tar.gz
mv graylog2-radio-* /opt/graylog2-radio
cd /opt/graylog2-radio
  • Create configuration files for graylog2-radio
cp graylog2-radio.conf.example /etc/graylog2-radio.conf
cp graylog2-radio-inputs.conf.example /etc/graylog2-radio-inputs.conf
  • Check that AMQP configuration is suitable to your RabbitMQ server
vi  /etc/graylog2-radio.conf
  • Configure graylog2-radio-inputs file to listen to gelf and syslog messages
vi  /etc/graylog2-radio-inputs.conf
udp gelflogs 0.0.0.0 12501
udp systemlogs 0.0.0.0 12502
  • Create graylog2-radio start script
vi /etc/init.d/graylog2-radio
#!/bin/bash
#
# graylog2-radio:   graylog2 AMQP producer
#
# chkconfig: - 98 02
# description:  This daemon start graylog2-radio
#

# Source function library.
. /etc/rc.d/init.d/functions

CMD=$1
NOHUP=`which nohup`

STOP_TIMEOUT=30
BINARY=java
PROG=graylog2-radio

HOME_DIR=/opt/graylog2-radio
LOG_FILE=${HOME_DIR}/log/${PROG}.log
JAR_FILE=graylog2-radio.jar
CONF_FILE=/etc/graylog2-radio.conf
PID_FILE=/var/run/graylog2-radio.pid

start() {
        graylog2_status > /dev/null 2>&1
        if [ ${RETVAL} -eq 3 ]
        then
                echo "Starting ${PROG} ..."
                cd ${HOME_DIR}
                $NOHUP > /dev/null 2>&1 ${BINARY} -jar ${JAR_FILE} -f ${CONF_FILE} -p ${PID_FILE} >> ${LOG_FILE} &
                RETVAL=0
        else
                echo "${PROG} is already running"
        fi
}

stop() {
        echo -n $"Stopping $PROG: "
        killproc -p ${PID_FILE} -d ${STOP_TIMEOUT} ${PROG}
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f ${PID_FILE}
}

graylog2_status() {
        status -p ${PID_FILE} ${PROG}
        RETVAL=$?
}

restart() {
    echo "Restarting ${PROG} ..."
    stop
    start
}

case "$CMD" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    status)
        graylog2_status
        ;;
    *)
        echo "Usage $0 {start|stop|restart|status}"
        RETVAL=1
esac

exit ${RETVAL}
  • Configure graylog2-radio to start at boot and start it
chmod +x /etc/init.d/graylog2-radio
chkconfig --add graylog2-radio 
chkconfig graylog2-radio on 
service graylog2-radio start
  • Configure RabbitMQ credentials in your graylog2 server
vi /etc/graylog2.conf
...
# AMQP
amqp_enabled = true
amqp_host = localhost
amqp_port = 5672
amqp_username = guest
amqp_password = guest
amqp_virtualhost = /
...
  • Restart graylog2-server to apply new AMQP configuration
service graylog2-server restart
  • Browse to your graylog2 web interface and configure on which exchange your graylog2 server should listen for which messages type:
  • In graylog2 web interface go to Setting -> AMQP
  • Add new AMQP configuration for gelf messages: Exchange=messages, Routing Key=gelflogs, Type=GELF
  • Add another AMQP configuration for syslog messages: Exchange=messages, Routing Key=systemlogs, Type=syslog

That’s all. Now you need to configure your servers to send logs to graylog2 server using graylog2-radio listening ports 12501 and 12502.

More guides in Graylog2 Category

Useful links:

001. RabbitMQ Installation

Tested On

OS: CentOS 6.3 x86_64
RabbitMQ-Server Version: 3.0.2
Hardware: Virtual Box 4.2.6

About

RabbitMQ is an opensource software that implement the Advanced Message Queueing Protocol (AMQP).

In this guide I will show how to install RabbitMQ server with a mamangment plugin for easy management.

Install RabbitMQ

  • Install EPEL repository

 

rpm -Uvh http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
  • Download and Install RabbitMQ
cd /usr/local/src
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.0.2/rabbitmq-server-3.0.2-1.noarch.rpm -O rabbitmq-server.noarch.rpm
yum localinstall rabbitmq-server.noarch.rpm -y
  • Install RabbitMQ management plugin
rabbitmq-plugins enable rabbitmq_management
  • Configure RabbitMQ to start at boot and start it
chkconfig rabbitmq-server on
service rabbitmq-server start
  • Browse to http://rabbitmq_server:15672 using guest/guest and start manage your rabbitmq server

For more information about RabbitMQ please visit http://www.rabbitmq.com/.

002. Configure syslog to forward logs to graylog2

Tested On

OS: CentOS 6.3 x86_64
Graylog2-Server Version: 0.10.0
Graylog2-web-interface: 0.10.2
Hardware: VMware Player 5.0.1

About

In this guide I will configure rsyslog to forward logs to graylog2 server

Prerequisite

Before using this guide you need a running graylog2 server. You can use this Graylog2 Installation guide to install graylog2 server

Configure Rsyslog

  • Configure syslog conf file
vi /etc/rsyslog.conf (change graylog2-server to your graylog2 server name or IP)
#### GLOBAL DIRECTIVES ####

...
#*.* @@remote-host:514
*.* @graylog2-server:514
  • Restart syslog
service rsyslog restart

That’s all now you can check your syslog messages in graylog2 web interface.

More guides in Graylog2 Category.

Please visit http://www.graylog2.org for more information about Graylog2 configuration and usage.