Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved]HaProxy - slow new connections, performance issues
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
sebekk23
n00b
n00b


Joined: 29 Mar 2015
Posts: 34

PostPosted: Mon Jun 22, 2015 4:00 pm    Post subject: [Solved]HaProxy - slow new connections, performance issues Reply with quote

Hi
I recently made 2 nodes with mariadb galera on banana pi and one with haproxy (on Banana Pi smart router - separate machine) and i notice that when i'm connecting through haproxy node, new connections (i mean opening new connections) take a lot of time (much,much more time than direct)

Direct connect from Haproxy(Banana Pi smart router without haproxy) node works very fast but with Haproxy it takes forever :).

Second thing that i want to solve is ... performance.
Sysbench test through direct connect (100 threads, no restriction or limit on connection count) gives me on both nodes
Code:
OLTP test statistics:
    queries performed:
        read:                            21546
        write:                           7695
        other:                           3078
        total:                           32319
    transactions:                        1539   (49.18 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 29241  (934.43 per sec.)
    other operations:                    3078   (98.36 per sec.)

Test execution summary:
    total time:                          31.2928s
    total number of events:              1539
    total time taken by event execution: 3086.7654
    per-request statistics:
         min:                               1087.02ms
         avg:                               2005.70ms
         max:                               3504.86ms
         approx.  95 percentile:            2658.12ms

Threads fairness:
    events (avg/stddev):           15.3900/2.28
    execution time (avg/stddev):   30.8677/0.34


but with haproxy
Code:
OLTP test statistics:
    queries performed:
        read:                            16548
        write:                           5910
        other:                           2365
        total:                           24823
    transactions:                        1180   (35.33 per sec.)
    deadlocks:                           3      (0.09 per sec.)
    read/write requests:                 22458  (672.46 per sec.)
    other operations:                    2365   (70.82 per sec.)

Test execution summary:
    total time:                          33.3966s
    total number of events:              1180
    total time taken by event execution: 3296.7437
    per-request statistics:
         min:                                 57.14ms
         avg:                               2793.85ms
         max:                              20223.17ms
         approx.  95 percentile:           18962.81ms

Threads fairness:
    events (avg/stddev):           11.8000/12.21
    execution time (avg/stddev):   32.9674/0.35


max is 10 times slower through haproxy!!!


Some configuration and information:
1 sysbench machine config (this one connect through haproxy)
- Intel i5, SSD, 8GB or RAM DDR3 (Linux elitebook 3.18.11-gentoo #1 SMP Mon May 18 19:01:30 CEST 2015 x86_64 Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz GenuineIntel GNU/Linux)
command that is invoked on that machine:
Code:
sysbench --test=oltp  --num-threads=100 --max-time=30 --max-requests=0  --mysql-table-engine=InnoDB  --mysql-db=test --mysql-user=root --mysql-password=passs --mysql-host=r1.dom  --mysql-engine-trx=yes --oltp-table-size=2000000 run


2 sysbnech machine (that one with haproxy)
- BanaPi R1 (smart router) with Allwinner A20(2 cores, 1GHz each), SSD, 1GB or RAM DDR3 (Linux r1 3.4.104+ #1 SMP PREEMPT Thu Jan 8 15:40:40 CET 2015 armv7l ARMv7 Processor rev 4 (v7l) sun7i GNU/Linux)
command that is invoked on that machine:
Code:
sysbench --test=oltp  --num-threads=100 --max-time=30 --max-requests=0  --mysql-table-engine=InnoDB  --mysql-db=test --mysql-user=root --mysql-password=passs --mysql-host=node{one or two}.dom  --mysql-engine-trx=yes --oltp-table-size=2000000 run

on this machine i have 2 vlans one for external connection (my lan) and second for internal connections (mariadb nodes)

haproxy config
Code:
global
  # log redirection (syslog)
  #log 127.0.0.1 local0
  #log 127.0.0.1 local1 debug
  log /dev/log local0
  log /dev/log local1 notice
  #apache2
  tune.ssl.default-dh-param 2048
  # maximum of connections for haproxy
  maxconn 2048
  #maxpipes 512
  # chroot for security reasons
  chroot /var/lib/haproxy
  # user/group for haproxy process
  user haproxy
  group haproxy
  #command to use followed hatop -s /var/lib/haproxy/stats
  stats socket /var/lib/haproxy/stats mode 777 level admin
  # act as a daemon
  daemon
defaults
  # use gloval log declaration
  log global
  # default check type
  #mode http
  # only log when closing session
  # only log failed connections
  # retry 3 times before setting node as failed
  # redispatch traffic to other servers
  option tcplog
  option dontlognull
  option redispatch
  # maximum connection for the frontend
  maxconn 1024
  # timeouts
  timeout client 1s
  timeout client-fin 1s
  timeout server 1s
  timeout connect 1s
  retries 3
  timeout queue 1m
  timeout check 20s
# enable web check health interface on port 80
listen haproxy 192.168.2.5:8080
  stats enable
  mode http
  fullconn 1024
  #stats hide-version
  #stats uri /stats
  #stats realm Haproxy\ Statistics
  # set credentials
  stats auth admin:password
# loadbalance on slaves  - we need acces from eth0_101 and eth0_102
frontend MDBG
  #bind 192.168.2.5:3306
  bind 0.0.0.0:3306
  timeout client 57600s
  #timeout client-fin 57600s
  default_backend mariadb-galera
backend mariadb-galera
  fullconn 1024
  #option splice-auto
  timeout server 57600s
  timeout connect 5s
  # use tcp method
  mode tcp
  # round robin mechanism
  #balance roundrobin
  balance leastconn
  # tcp keepalive (pipelining) on both side (clt/srv)
  option tcpka
  # perform http request
  #option httpchk
  # perform mariadb connection with haproxy user
  option mysql-check user haproxy
  # set all read only nodes
  # inter: interval of check in milliseconds
  # check port 9200 inter 2000 rise 3 fall 3
  server vulcanone 172.16.1.21:3306 check inter 2000 rise 3 fall 3 maxconn 512
  server vulcantwo 172.16.1.22:3306 check inter 2000 rise 3 fall 3 maxconn 512

frontend LB
   bind 192.168.2.5:80
   timeout client 57600s
   #timeout client-fin 57600s
   reqadd X-Forwarded-Proto:\ http
   default_backend apache2
frontend LBS
   #bind 0.0.0.0:443
   bind 192.168.2.5:443 ssl crt /etc/haproxy/ssl/localhost.pem
   timeout client 57600s
   timeout client-fin 57600s
   reqadd X-Forwarded-Proto:\ https
   default_backend apache2
backend apache2
  fullconn 1024
  timeout server 57600s
  timeout connect 5s
  redirect scheme https if !{ ssl_fc }
  timeout http-request    20
  timeout http-keep-alive 30
  mode http
  balance roundrobin
  cookie PHPSESSID prefix indirect nocache
  option httplog
  option http-server-close
  option httpclose
  option forwardfor
  #http-request set-header X-Forwarded-Port %[dst_port]
  #http-request add-header X-Forwarded-Proto https if { ssl_fc }
  option httpchk HEAD / HTTP/1.1\r\nHost:localhost
  server markone 172.16.1.11:80 cookie markone check maxconn 512
  server marktwo 172.16.1.12:80 cookie marktwo check maxconn 512
  #server web4-srv 192.168.0.124:80 check backup # backup fail-over Server, If three of the above fails this will be activated.


Ping from Intel machine to haproxy machine:
Code:
h3x0r@elitebook ~ $ ping r1.dom
PING r1.dom (192.168.2.5) 56(84) bytes of data.
64 bytes from r1.dom (192.168.2.5): icmp_seq=1 ttl=64 time=0.335 ms
64 bytes from r1.dom (192.168.2.5): icmp_seq=2 ttl=64 time=0.255 ms
64 bytes from r1.dom (192.168.2.5): icmp_seq=3 ttl=64 time=0.320 ms
64 bytes from r1.dom (192.168.2.5): icmp_seq=4 ttl=64 time=0.194 ms


Ping from haproxy machine to mariadb nodes:
Code:
[h3x0r@r1 ~]$ ping vulcanone.dom
PING vulcanone.dom (172.16.1.21) 56(84) bytes of data.
64 bytes from vulcanone.dom (172.16.1.21): icmp_seq=1 ttl=64 time=0.339 ms
64 bytes from vulcanone.dom (172.16.1.21): icmp_seq=2 ttl=64 time=0.225 ms
^C
--- vulcanone.dom ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.225/0.282/0.339/0.057 ms
[h3x0r@r1 ~]$ ping vulcantwo.dom
PING vulcantwo.dom (172.16.1.22) 56(84) bytes of data.
64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=1 ttl=64 time=0.432 ms
64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=2 ttl=64 time=0.283 ms
64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=3 ttl=64 time=0.329 ms


I have 1Gb network between my machines, i never notice any problems with CPU or RAM or any machine - so machine resources are fine (during heavy load haproxy machine has almost all ram free and CPU around 20-25%, nodes - of course - almost 200% CPU, 50% free ram, intel machine ... :) no load at all).

Is there any way to make haproxy more "transparentable" ? How to reduce this performance issues ?


Last edited by sebekk23 on Sat Jun 27, 2015 7:08 pm; edited 1 time in total
Back to top
View user's profile Send private message
sebekk23
n00b
n00b


Joined: 29 Mar 2015
Posts: 34

PostPosted: Sat Jun 27, 2015 7:08 pm    Post subject: Reply with quote

Ok - i've got it!!

I've eliminated delays in connection time by moving haproxy bind address from external (external VLAN) to internal :)

I think that iptables was an issue (netfilter has been working too long ?) - now haproxy listen on internal IP and iptables doing DNAT to this IP from external one :D

Problem solved ;)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum