Kafka OffsetOutOfRange error

Today I got the following error on our ruby on rails app server: Poseidon::Errors::OffsetOutOfRange

If you get offset out of range error from kafka client you need to reset your kafka consumer  group offset with the following commands:

  • find smallest offset
cd ${KAFKA_HOME}
./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka_server.local:9092 --topic topic_name --time -2

–time -2 – is for getting the smallest offset, for largest offset use -1

  • stop all consumers in consumer group
  • set kafka consumer group offset in zookeeper
cd ${KAFKA_HOME}
./bin/zookeeper-shell.sh zookeeper_server.local
...
Connecting to zk1.nyj.taptica.info
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
...

set /consumers/consumer_group_name/offsets/topic_name/partition_number new_offset

run the set command for each partition offset that you get from the previous get offset command that you ran.

  • check your new configured offset
./bin/kafka-consumer-offset-checker.sh --zookeeper=zookeeper_server.local:2181 --topic=topic_name --group=consumer_group_name
  • start your consumer

All commands I got from this great blog https://metabroadcast.com/blog/resetting-kafka-offsets

sysctl network tuning template

Sometimes you need your linux server to work in a very high performance network environment, so I created a template to start from that contain sysctl variables that can be tuned.

I used this template in my couchbase cluster and I got these values from the following url:
http://www.couchbase.com/connect/agenda/tuning-couchbase-server-os-network-maximum-performance/

sysctl.conf:

# http://www.couchbase.com/connect/agenda/tuning-couchbase-server-os-network-maximum-performance/
#net.core.somaxconn=
#net.ipv4.tcp_max_syn_backlog=
#net.ipv4.tcp_fin_backlog=
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
#net.ipv4.tcp_sack
#net.ipv4.tcp_fack
#net.ipv4.tcp_fin_timeout
#net.ipv4.tcp_tw_reuse
#net.ipv4.tcp_keepalive_intvl
#net.ipv4.tcp_moderate_rcvbuf
#net.ipv4.tcp_window_scaling

# Max listen queue backlog
# make sure to increase nginx backlog as well if changed
net.core.somaxconn = 16384
# Max number of packets that can be queued on interface input
# If kernel is receiving packets faster than can be processed
# this queue increases
net.core.netdev_max_backlog = 16384
# Only retry creating TCP connections twice
# Minimize the time it takes for a connection attempt to fail
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# Timeout closing of TCP connections after 7 seconds
# net.ipv4.tcp_fin_timeout = 7
# Avoid falling back to slow start after a connection goes idle
# keeps our cwnd large with the keep alive connections
net.ipv4.tcp_slow_start_after_idle = 0

PFSense stable site to site configuration

I know it’s not the most secure configuration but its stable and works great for my use case.

  1. Enable make-before-break in ipsec advanced settings
    VPN -> IPSec -> Advanced settings
    Check "Initiate IKEv2 reauthentication with a make-before-break"
  2. Phase 1 configuration:
    Mode: Main
    My Identifier: IP Address
    Encryption: 3DES
    Hash: SHA1
    DH Group: 1
    Lifetime: 86400
    Auth: PSK
  3. Phase 2
    Protocol: ESP
    Encryption: 3DES (others unchecked)
    Hash: SHA1 (MD5 unchecked)
    PFS: off
    Lifetime: 86400
  4. Create this configuration on both pfsense servers

I used the following topic to create this configuration:
https://forum.pfsense.org/index.php?topic=21515.5

Create simple nginx proxy pass configuration

I tried today to install simple nginx reverse proxy configuration to my website.

I searched in google and find few useful manuals that I can use but it was too long for my needs so here is very quick guide (I am pretty sure it can be improved) to setup nginx as a reverse proxy:

  • Install and configure nginx
sudo apt-get install nginx
sudo vi /etc/nginx/conf.d/proxy.conf
server {
 server_name nachum234.com;

 location / {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_pass http://192.168.1.10:8080;
 }
}

If you have a suggestion to improve it please leave a comment

001. Configure NAT On Linux With IPTABLES

Tested On

OS: CentOS 6.2 i386
iptables version: v1.4.7
Hardware: Virtual Machine (VirtualBox 4.1.14)

About

Network Address Translation (NAT) is a technology that translate private addresses to public and vice versa. In this guide I will show how to implement the main types of NAT using linux and iptables.

Network Address Port Translation (NAPT)/Port Address Translation (PAT)

NAPT is the most common type of NAT. This type of NAT on a traditional outbound transaction change the source IP and the source port, and because it change also the source port multiple devices can share the same IP simultaneously.

  • Add the following rule to the NAT table of iptables in order to configure NAPT
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 192.168.10.1
-t nat - Use NAT table of iptables
-A POSTROUTING - Append the rule to POSTROUTING chain on the NAT table
-o interface - Specify on which outgoing interface apply this rule
-j SNAT - Change the source address
--to-source - Source addresses list to change the original source address
  • If you have a dynamic IP use the following rule
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Traditional/Outbound NAT

Traditional NAT share public IP addresses with local devices that use private IP addresses.

Traditional NAT is implemented in iptables like NAPT with multiple source IP addresses.

  • Add the following rule to the NAT table of iptables in order to configure traditional NAT
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 192.168.10.1-192.168.10.20

Bidirectional/Inbound NAT

Bidirectional NAT is used when a device from the outside network needs to initiate a session with server on the inside network.

  • Add the following rule to the NAT table of iptables in order to configure bidirectional NAT
iptables -t nat -A PREROUTING -i eth0 -j DNAT --to-destination 192.168.10.7
    • If the NAT gateway is the only server how can access to the inside server then you may need to enter this rule also to make sure he will send the traffic using the  NAT gateway
iptables -t nat -A POSTROUTING -d 192.168.10.7/32 -j MASQUERADE
-i interface - Name of an interface via a packet was received
--to-destination local_ip_address - IP address of a local server
  • If you want to configure a specific public address use the following rule
iptables -t nat -A PREROUTING -d 46.116.47.237 -j DNAT --to-destination 192.168.10.7
  • If you want to DNAT only a single port use the following
iptables -t nat -A PREROUTING -p tcp -d 46.116.47.237 --dport 80 -j DNAT --to-destination 192.168.10.7