How to Decrypt Getmail IMAPS/POP3S sessions?

[miroR]

title: How to Decrypt Getmail IMAPS/POP3S sessions?
---
I have touched on this issue in the sidelines of:

... the area is huge: mailing, Dillo (with some not completely implemented TLS), Lynx ...

I would like to be able to decrypt SSL traffic when I download mail from the mail hub of my provider or hoster (or if some day I learn to host my own mail server for me, that day not expected soon ), just like I can decrypt most of the traffic that my Firefox browses at.

There is surely also Dillo's traffic to decrypt (newbies, see http://www.dillo.org), and also when I connect to either my hoster, or github.com, or Devuan's Gitlab, or anywhere else, because I don't want SSL to preclude seeing anything to me. Not to me , I mean: not in my box nor where I go as free linuxer. That's the FOSS Linux imperative for any power user, isn't it? (And for wannabe power users ike me, surely I have to add, to be more realistic.)

But I'd like to stay now only with one small portion of what is almost completely opaque in my own box and my ways on the internet to me, when the presentation/application layer of the [O]pen [S]ystems nterconnection model, is encrypted with SSL, and that is I'd like to stay now only with:

my fetching of mail with getmail and how to decrypt it.

Regarding this page below, it took me a while to figure out that it was not really much related to my issue at all:

Follow SSL stream using Master-key and Session-ID
https://ask.wireshark.org/questions/422 ... session-id

Even though there is the string "mail" in this time-frozen howto (of historical --year 2011-- interest: that was the first time the Mozilla [N]etwork [S]ecurity [S]ervices library was put to marvelous use, for Wireshark, to the benefit of an ever-increasing-to-be-thereafter number of SSL-decrypting users.

I have only recently, and after having longed and asked[*] for that secret lore for years, finally joined those users in possession of what most of us thought forever obsured for us, and I told the world about it in that Gentoo topic linked at the start of this post.

But, I was saying, even though there is the string "mail" in that time-frozen breakground howto, it is about https, the SSL over HTTP, and it is not about either imaps, the SSL over IMAP, nor is it about pop3s, the SSL over POP3, the two being the most used among the mail protocols in todays world.

And I use both imaps and pop3s to fetch my mail, from two different accounts, respectively. And none of the two is, as of yet, decryptable to me.

My mentioning of that historical page that testifies to the breakthrough in SSL decryption, may (and it may not) show to be pertinent for imaps/pop3s decryption.

It may be so, because my attempts at openssl connectiong to my current hoster's mail bub:

Code: Select all

# openssl s_client -connect pop.t-com.hr:995

and my povider's mail hub:

Code: Select all

# openssl s_client -connect lin16.mojsite.com:993

which two commands [**] I put together and deployed seeing how the command in that page on ask.wireshark.org:

Code: Select all

# openssl s_client -connect mail.google.com:443 -ssl3

is the tool to get the needed Session-ID and Master-Key.

But the above way may not be really pertinent to solving this decryption issue, because it appears to me that my getmail commands (which consist of using a typical configuration for the respective servers; will probably post them on getmail mailing list when I ask for help there, and then give link here), that I issued first, and in another terminal, worked completely separately, probably in a completely different session each, than the respective openssl commands above, because...

[Because] the captured two separate sets of Session-ID and Master-Key, after I put them in my $SSLKEYLOGFILE that Wireshark uses for decrypytion (newbies see: https://wiki.wireshark.org/SSL), didn't get the two respective getmail's sessions decryted at all for me, after I opened the network capture containing those events (and the network I capture with my uncenz, as on https://github.com/uncenz ).

And this attempt at decryption of getmail's imaps/pop3s fetching of mail has after that got even more complex!

Since the whole of this story is, with all of this so far recounted to the kind reader, already pretty complex, allow me to post the next part of the even greater complexity that just made my own tries even more difficult, in the next post.

Also because some of the users will not need to deal with this additional complexity that I will have to deal with to solve this issue. Those users that don't have grsec-hardened kernel will not need that next post.

I however, don't use kernels that are not grsec-hardened, and so, I have to solve the complexity that aggravates this story for me, first.
---
[*] Yes I have asked about how to decrypt the SSL traffic in various places in these years on Gentoo Forums! However, apart from rare advanced users/developers, few people knew the news/the knowhow, at the time of my asking. See:

... when am I going to learn to decrypt and read the encrypted conversations that those surveillors do on my computer when I'm online? ...

or:

... It takes an expert to decrypt what data, or what ploy, or what ever-else, the Schmoog did in these some half a minute that I tried to connect to DuckDuckgo.com ...

[**] The twin command that I actually issued captures automatically the STDOUT for me in a timestamped file:

Code: Select all

# read FAKE; openssl s_client -connect lin16.mojsite.com:993 |& tee openssl_s_client-connect_lin16.mojsite.com_$(date +%y%m%d_%H%M%S) ; read FAKE; openssl s_client -connect pop.t-com.hr:995 |& tee pop.t-com.hr_$(date +%y%m%d_%H%M%S); read FAKE;

And, for completeness here, here are excerpts with just the Session-ID and Master-Key from the files written by the two.

openssl_s_client-connect_lin16.mojsite.com_160131_214845:

Code: Select all

...
    Session-ID: 46AD456B08AB1071DE058572C4CF8A0769FDE5C3E29F70704E2F2215E36CE6FE
...
    Session-ID-ctx: 
    Master-Key: 09EDE667DAA18CB55C0B26BF4CE10DA864661B6BFC5C2486D41F0A4BC769FDCFC26567B143E04939FAF497EC36FFD9AB
...

openssl_s_client-connect_pop.t-com.hr_160131_215000:

Code: Select all

    Session-ID: E0945C0ECEA7FB8AEE82738F8CA131CC5EF8FB1F1537672FCC2E348BE9CA5C5D
...
    Master-Key: 1D432753541E1A5DC0707C2BBED399F1DB1986D6A0CA648AE6A4E2380B59CA126AE62A8CD0F11E695343D414EC219097

I lowecased those, and fit them in in my $SSLKEYLOGFILE.

The lines are, in this case:
CLIENT_RANDOM <the-Session-ID> <the-Master-Key>
.

But, as I already stated, those appear to me to be two completely different sessions that do not help decrypting the two respective getmail sessions. BTW, I keep, and if need be will be able to post more complete content of those two files, later.

[miroR]

My attempt at decryption of getmail's fetching of mail[*], in two separate SSL sessions, one imaps, and the other pop3, gets even more complex after I, modeling my tries after what I saw in these emails (for getmail mailing list subscribers like me), or pages (for anybody else), in this thread:

Problem retrieving mail from Freenet's 'freemail'
https://marc.info/?t=144496930200002&r=1&w=2

where I found in this (and some later/previous? mails/pages):

https://marc.info/?l=getmail&m=144502725016279&w=2

Code: Select all

Could you try to run your getmail like that so we can get full imap trace?

$ python2 -m pdb $(which getmail)
(Pdb) import imaplib
(Pdb) imaplib.Debug = 4
(Pdb) c

how that appears to give complete debugging in Getmail...

And then I decided to try and go that way, as it seemed (and still seems) to me that was/is the right way to go.

But here's the obstacle I stumbled upon. Even though I somehow left out the entire first line of the command, and issued just

Code: Select all

$ python

instead, my obstacle would remain the same if I tried again with the correct command, and it can be seen in the /var/log/messages, which keeps for me pretty complete account of what goes on in my machine thanks to the exec_logging functionality (newbies to grsecurity: enable it in kernel configuration) of grsecurity:

Code: Select all

Feb  1 06:25:45 g0n kernel: [353942.599063] grsec: (miro:U:/) exec of /usr/bin/python2.7 (/usr/bin/python ) by /usr/bin/python2.7[python:27668] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:3163] uid/euid:1000/1000 gid/egid:1000/1000
...
Feb  1 06:26:38 g0n kernel: [353995.006352] grsec: (miro:U:/) denied socket(inet,stream,tcp) by /usr/bin/python2.7[python:27668] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:3163] uid/euid:1000/1000 gid/egid:1000/1000

In other words, IIUC, there is more tweaking of my /etc/grsec/policy to be done first, before I will be able to issue:

Code: Select all

$ python2 -m pdb $(which getmail)

and be permitted to use it.

So while, if someone else is interested in solving for themselves, separately from me, this issue, to get their imaps or pop3s getmail fetching decrypted, they don't need to wait for me to solve my grsecurity gradm policy issue ... But I have to try and seek help for this issue on all the (it's three different places involved here) places involved:

forums.grsecurity.net
Getmail mailing list
Wireshark mailing list

And the forums.grsecurity.net is the first for me, because I don't go online without a grsec-hardened (complemented by iptables also deployed).

The subject of:

Code: Select all

subject	/usr/bin/python2.7

[**]

I have not had up until now. It appears not to be so common, other then maybe for developers I guess. So I'm in new territory... Not so common, because Gentoo is the home of grsecurity-hardening, and python, in which the Portage architecture is rooted in, is all set to work out-of-the-box in grsecurity hardened kernel, IIUC.

First thing next, for me, while this decryption issue is posted here on Gentoo Forums, is, prepare a question about this python permissions issue on Grsecurity Forums.

Pls. remember that I work slowly if you post any questions to me regarding this topic. The work and explanation of the issues deployed in these two posts have already cost me something like one half of one day's time.

Regards!

---
[*]
For the meaning of "the [whole] story" to be true, I have to say that, previous to the opening of pyhon interface, I also tried to add '--trace' to my getmail commands. The suggestion of it can be found in Getmail documentation:

http://pyropus.ca/software/getmail/configuration.html

and also in that thread meantioned in this post it was used:

https://marc.info/?l=getmail&m=144502194114815&w=2

But I did not get any more info on SSL with it. I have the verbose output of those if need be.

[**]
A subquestion in this topic of mine, if anybody will be kind to advise: I see there is python3.4 also available. Time to switch? Links to the difficulties involved? I read the news:

Code: Select all

  [25]     2015-12-16  Python ABIFLAGS rebuild needed

Do it or wait a while longer? ... Hmmh... Maybe try solve my grsec policy first, and switch and rebuild later...

[miroR]

subject /usr/bin/pythonX.X needed in my RBAC policy?
http://forums.grsecurity.net/viewtopic.php?f=5&t=4373

[miroR]

...

Code: Select all

$ python2 -m pdb $(which getmail)
(Pdb) import imaplib
(Pdb) imaplib.Debug = 4
(Pdb) c

how that appears to give complete debugging in Getmail...
....

Code: Select all

Feb  1 06:25:45 g0n kernel: [353942.599063] grsec: (miro:U:/) exec of /usr/bin/python2.7 (/usr/bin/python ) by /usr/bin/python2.7[python:27668] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:3163] uid/euid:1000/1000 gid/egid:1000/1000
...
Feb  1 06:26:38 g0n kernel: [353995.006352] grsec: (miro:U:/) denied socket(inet,stream,tcp) by /usr/bin/python2.7[python:27668] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:3163] uid/euid:1000/1000 gid/egid:1000/1000

I...
The subject of:

Code: Select all

subject	/usr/bin/python2.7

[**]

---
[**]
A subquestion in this topic of mine, if anybody will be kind to advise: I see there is python3.4 also available. Time to switch? Links to the difficulties involved? I read the news:

Code: Select all

  [25]     2015-12-16  Python ABIFLAGS rebuild needed

Do it or wait a while longer? ... Hmmh... Maybe try solve my grsec policy first, and switch and rebuild later...

No, this is not normal, the python command not giving an interface to me!... I remember, when I tried to learn some Python, a few months ago, I was always able to open that interface.

Either this is a bug, or I have done something wrong somewhere in my system.

I don't think anymore, as I just wrote on my new Grsecurity Forums, that I need to go for the grsec learing on the python subject.

What the reason might be, though?

I am thinking about the gcc issues that requires a one day rebuild of a lot of packages on a machine linke mine.

I am talking about the:

Code: Select all

# eselect news read 24
2015-10-22-gcc-5-new-c++11-abi
  Title                     GCC 5 Defaults to the New C++11 ABI
...
For gentoolkit-0.3.1 or higher:
# revdep-rebuild --library 'libstdc++.so.6' -- --exclude gcc
...

(pls. see that entire news item in your own Gentoo)...

Not everything is completely in place, I'm afraid, after I ran that command above... (and after it rebuilt, IIRC around 100 packages)...

Or it could be something else.

What should I do?

In the absense of some clearer ideas, I'm afraid all I can do is try and update and see if my testing Gentoo (I've been on ~amd64 just about since I arrived here in 2008)....

{So} [update] and see if it goes away. The non-expert way...

BTW, the news that I mentioned:

Code: Select all

  [25]     2015-12-16  Python ABIFLAGS rebuild needed 

it appears to me, would apply only after I installed:

Code: Select all

# emerge -p python

These are the packages that would be merged, in order:

Calculating dependencies... done!                        
[ebuild  NS    ] dev-lang/python-3.5.1-r2:3.5/3.5m::gentoo [2.7.11-r2:2.7::gentoo, 3.4.3-r7:3.4/3.4m::gentoo] USE="gdbm hardened ipv6 ncurses readline sqlite ssl threads xml -build -examples -libressl -tk -wininst" 14,495 KiB

Total: 1 package (1 in new slot), Size of downloads: 14,495 KiB

The command from that news:

Code: Select all

emerge -1v $(find /usr/lib*/python3* -name '*cpython-3[3-5].so')

offers to rebuild nothing in my box, where I have python2.7 and python3.4 only, at this time.

And, to be complete, I'll tell what concerns my grsec policy, and my getmail: (The grsec_160201_g0n_00 is a copy of my current /etc/grsec/policy.)

These are installed with 2.7:

Code: Select all

# ls -ltr $(grep python2.7 grsec_160201_g0n_00 | grep subject | awk '{ print $2 }')
-rwxr-xr-x 1 root root 39610 2016-01-17 17:50 /usr/lib64/python-exec/python2.7/getmail
-rwxr-xr-x 1 root root  1185 2016-01-17 20:15 /usr/lib64/python-exec/python2.7/hg
-rwxr-xr-x 1 root root  2603 2016-01-25 14:27 /usr/lib64/python-exec/python2.7/emerge

And:

Code: Select all

# ls -ltr $(grep python2.7 grsec_160201_g0n_00 | grep subject|sed 's/2\.7/3\.4/' | awk '{ print $2 }')
ls: cannot access '/usr/lib64/python-exec/python3.4/getmail': No such file or directory
ls: cannot access '/usr/lib64/python-exec/python3.4/hg': No such file or directory
-rwxr-xr-x 1 root root 2603 2016-01-25 14:27 /usr/lib64/python-exec/python3.4/emerge

some packages, among which my getmail, are installed with 2.7 only.

Aaaarrhh....

[miroR]

I just solved the python interface permission problem:

subject /usr/bin/pythonX.X needed in my RBAC policy?
http://forums.grsecurity.net/viewtopic. ... 006#p16006

and can move on (after a little rest. It wasn't so easy...).