[SOLVED] curl with TLSv1, 1.1 and 1.2 support

Message

GhostTyper · Post by **GhostTyper** » Mon Sep 17, 2018 12:00 pm

Hello,

i want curl to use TLSv1 and above. Currently it only supports SSLv2 and SSLv3, which is pretty outdated.

Sadly, I don't know which use-flags I need to use. My current configuration looks like this:

Code: Select all

ws1 ~ # emerge -ptv openssl curl

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild   R    ] net-misc/curl-7.61.0::gentoo  USE="idn ssl -adns -brotli -http2 -ipv6 -kerberos -ldap -metalink -rtmp -samba -ssh -static-libs {-test} -threads" ABI_X86="(64) -32 (-x32)" CURL_SSL="openssl -axtls -gnutls -libressl -mbedtls -nss (-winssl)" 0 KiB
[ebuild   R    ]  dev-libs/openssl-1.0.2o-r3::gentoo  USE="asm bindist sslv3 tls-heartbeat zlib -gmp -kerberos -rfc3779 -sctp -sslv2 -static-libs {-test} -vanilla" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="(sse2)" 0 KiB

Total: 2 packages (2 reinstalls), Size of downloads: 0 KiB

 * IMPORTANT: 17 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

The error I observe is:

Code: Select all

ws1 ~ # curl https://api.twitch.tv/kraken/base
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Christian99 · Post by **Christian99** » Mon Sep 17, 2018 12:38 pm

I think that's the bindist useflag of openssl. I have it disabled and i'm able to curl the address you specified.

curl telss me, that it is using "* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384"
checking

Code: Select all

equery u openssl
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for dev-libs/openssl-1.0.2p:
 U I
 + + abi_x86_32    : 32-bit (x86) libraries
 + + asm           : Support assembly hand optimized crypto functions (i.e. faster run time)
 - - bindist       : Disable/Restrict EC algorithms (as they seem to be patented) -- note: changes the ABI
 + + gmp           : Add support for dev-libs/gmp (GNU MP library)
 + + kerberos      : Add kerberos support
 - - rfc3779       : Enable support for RFC 3779 (X.509 Extensions for IP Addresses and AS Identifiers)
 - - sctp          : Support for Stream Control Transmission Protocol
 - - sslv2         : Support for the old/insecure SSLv2 protocol -- note: not required for TLS/https
 + + sslv3         : Support for the old/insecure SSLv3 protocol -- note: not required for TLS/https
 - - static-libs   : Build static versions of dynamic libraries as well
 - - test          : Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
 + + tls-heartbeat : Enable the Heartbeat Extension in TLS and DTLS
 - - vanilla       : Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes drastically
 + + zlib          : Add support for zlib (de)compression

it seems, that this cipher is disabled by the bindist useflag.

Also note: the sslv2/3 flags for openssl enable those protocols additionaly, tls1.* is enabled (and can't be disabled, afaik) by default for openssl.

GhostTyper · Post by **GhostTyper** » Mon Sep 17, 2018 12:58 pm

Changing the use flags (-bindist) didn't change curls behaviour. I also added bindist for testing only:

Code: Select all

ws1 ~ # emerge -ptv openssl curl

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild   R    ] net-misc/curl-7.61.0::gentoo  USE="idn ssl -adns -brotli -http2 -ipv6 -kerberos -ldap -metalink -rtmp -samba -ssh -static-libs {-test} -threads" ABI_X86="(64) -32 (-x32)" CURL_SSL="openssl -axtls -gnutls -libressl -mbedtls -nss (-winssl)" 0 KiB
[ebuild   R    ]  dev-libs/openssl-1.0.2o-r3::gentoo  USE="asm sslv3 tls-heartbeat zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -static-libs {-test} -vanilla" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="(sse2)" 0 KiB

Total: 2 packages (2 reinstalls), Size of downloads: 0 KiB

 * IMPORTANT: 17 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

ws1 ~ # curl https://api.twitch.tv/kraken/base
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
ws1 ~ # curl --tlsv1.2 https://api.twitch.tv/kraken/base
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
ws1 ~ # curl -v https://api.twitch.tv/kraken/base
*   Trying 104.86.61.204...
* TCP_NODELAY set
* Connected to api.twitch.tv (104.86.61.204) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

It really seems to be a problem with openssl:

Code: Select all

ws1 ~ # openssl s_client -connect api.twitch.tv:https
CONNECTED(00000003)
140436400553536:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:827:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 308 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1537189359
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Christian99 · Post by **Christian99** » Mon Sep 17, 2018 1:53 pm

what did you do to you r openssl?

Also, can you try other addresses with oepnssl s_client?

eg gentoo.org, startpage.com....

GhostTyper · Post by **GhostTyper** » Mon Sep 17, 2018 5:07 pm

I DON'T KNOW.

gentoo.org supports TLSv1.0, 1.1 and 1.2:

Code: Select all

openssl s_client -connect gentoo.org:https
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = default.gentoo.org
verify return:1
---
Certificate chain
 0 s:/CN=default.gentoo.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
[The certificate...]
-----END CERTIFICATE-----
subject=/CN=default.gentoo.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5030 bytes and written 434 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: BE9D919076F5B63C1852B58CF282F3FEBCE3FD5BDEB289B94015BAAD3A1C71E3
    Session-ID-ctx:
    Master-Key: EDDB0251D6329273E175CA7ABC0449AD92CB2BF0BFAAFC71406EDFE06F9600227B9DD2CF3E57FE0D3E6512EE3A56173C
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1537200659
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Code: Select all

ws1 / # openssl s_client -connect startpage.com:https
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL Wildcard, CN = *.startpage.com
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=EssentialSSL Wildcard/CN=*.startpage.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
[The certificate...]
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=EssentialSSL Wildcard/CN=*.startpage.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5014 bytes and written 434 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: FB939A3B3368D67059304986D07579178CAF06C4A643287AC8A031A641ADD1AD
    Session-ID-ctx:
    Master-Key: 8A9969DEEEE6326A0FCDF2191FFEC090C18BFA96F5215EB1B66BF7B0B5903B8C700996FF9D20053B7DC9F0089D1FF1FF
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    [Some Data...]

    Start Time: 1537200778
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

This is quite strange. So, it's not a protocol problem, it must be something else. What could it be? Some misconfiguration of mine? But I don't remember any changes to the openssl configuration. Even twitch.tv is workin':

Code: Select all

ws1 / # openssl s_client -connect twitch.tv:https
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign CloudSSL CA - SHA256 - G3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Fastly, Inc.", CN = twitch.map.fastly.net
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=twitch.map.fastly.net
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
[Certificate...]
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=twitch.map.fastly.net
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3623 bytes and written 434 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 21C61EADAFCBEA0F0999C8ED305709B02B8E7C007931CFD3BF317E6D751D0820
    Session-ID-ctx:
    Master-Key: 37EF39BC9BAFF879F89A8231D858EA4B5898A9AAEDEB03D558688924E99821DDBA6D426404E0A1AC541B5B7EB33175B4
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    [Some Data...]

    Start Time: 1537201217
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

But api.twitch.tv does not work:

Code: Select all

ws1 / # openssl s_client -connect api.twitch.tv:https
CONNECTED(00000003)
139732243668544:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:827:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 308 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1537201329
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

GhostTyper · Post by **GhostTyper** » Mon Sep 17, 2018 5:15 pm

I fear, something happens on the transport way. I tried 3 different ip ranges to just be sure that the service isn't banned from the twitch API.

Christian99 · Post by **Christian99** » Mon Sep 17, 2018 9:00 pm

can you please post output from

Code: Select all

openssl ciphers|tr : \\n

?

GhostTyper · Post by **GhostTyper** » Tue Sep 18, 2018 2:21 am

I could, but it's not necessary. I re-asked this question with my latest findings: viewtopic-p-8261988.html

Thank you for your help, but it isn't a problem on this gentoo installation.

Christian99 · Post by **Christian99** » Tue Sep 18, 2018 8:31 am

I don't think that this is related to firewall or network in general at all. You are able to establish a connection, it's just not a valid tls connection.
For the firewall, to interfere with this, something like deep packet inspection would be needed, what you are probably not doing.

But let's see...

GhostTyper · Post by **GhostTyper** » Tue Sep 18, 2018 4:04 pm

What does sherlock say? "when you have eliminated the impossible, whatever remains, however improbable, must be the truth"

I know that this shouldn't happen. Facts are:

It works without the firewall.
It works after a firewall restart.

And there could be many options up to a bug in the kernel.