Protect a web mail form against bots

[kgdrenefort]

Hello,

This topic is not Gentoo related. It's more a discussion about the good practices I should use for my actual project.

The TL;DR bit is at the end, if you like.

The project

I am working on a web mail form for a project I'm in. This form is simply allowing users to contact us without going to the other platform (such as Discord).

It is written using PHP8, no framework and rely on some classes such as PHPMailer and PHPDotenv. Written from scratch as an occasion to actually learn PHP.

The form takes the firstname, secondname, zipcode, e-mail (for answers) and a message, you can ask to receive a copy of the e-mail in your own mailbox and you have to validate our rules by checking a box before trying to send it. If everything is good (type of data for each field, length, etc), then the mail is stored for about X minutes, in a .json format, before a cron task send it. This cron task call a .php file that is using PHPMailer to send the e-mail with our SMTP server we pay for. The mails are send only one at a time, I'll give more details about it later on in this topic.

This is a web mail form that'll go public this year if everything is ready for.

My concern here is security.

The server hosting is managed by me at home, on my own computer, on Gentoo with an hardened kernel. I try to take great care of the state and security of this machine. Firewall, keeping a eye on everything, trying to automate the process of keeping it alive, updating it very often and trying as much as I can to run only what's needed to me and my hosted sites. Nothing professional of course.

Now comes the bots, the main subject of this topic.


Old CAPTCHA
Nowadays, CAPTCHA is not effective, at all. From my reading and search, they are better than human (AI or not) to fill them.
Plus, the project which I'm written for the web mail form is around some kind of public with neuro-developement issue, such as ADHD, autism and… Dyslexia. How much a pain this could be to them. And since this is freakin' not effective these days, it's a pure no.

Second generation of CAPTCHA and beyond
There is more modern CATPCHA, allowing to instead of reading some blurry text, to play around with images, select some, etc. But again from my reading, I have no source saved down to share but I need to get into it anyway, they are very good at… stealing your users datas. If that is true, it's a pure no, again.

And on top of that, the best bots (AI driven, probably) seems to easily get beyond this. Making it possible, not even a viable way against this threat.

«I'm not a bot» checkbox
Now comes the actual and more popular solution, such as the checkbox you click and you use to say you are not a bot. They seems smart and pretty much effective because even for AI it's not always easy to reproduce human behavior. But to be honest, I guess some are already doing that pretty much effectively. These solutions comes from companies such as Cloudflare.

I dislike their service because first the idea of having about 30% of the web into the pipes of one company is kind of scary. Specially nowadays, American company, if possible I would pick up an EU services. Not going anti-USA here, just how I feel it theses last years, not bad feeling!

Second is because I want to avoid to make the project give money for something I could maybe handle. Which is still a mystery to me.

There is alternative companies doing such service as I saw, for free or against a monthly fee.

But again, is it really a good idea to give a big part of the security of your project to the hands of peoples you don't know. Even if it's a company you are paying for. I see it as a bonus you could get rid off if you want to, because if our services does not works if the providers behind is down too, it's kind of bad IMHO.

As a bonus layer of security, why not.

What my code do against that and what it does not

First, all input are checked and rejected if they do not apply to my filter. E-mail has his very well documented format, names are never having usually numbers or an underscore, zipcode is always 5 numbers (in France, a settings you can tweak in my code pretty easily)… Etc. Plus, of course, length checking, checking if a field (or several) are empty, etc etc. Very trivial checking as a first layer of security, which is of course in the HTML but mostly in my PHP code, since you could rewrite the client code… HTML side is only for the user experience, the security is in my code.

Second, if a mail gets accepted, it waits before being send. Even if you push 1000 e-mails in a minute into my form, and even if they are all accepted to be send, they would only be send every X minutes, defined in a cron task calling a phpmailer function to retrieve the .json file and convert it as we want to a proper e-mail. Since our ESP is allowing 1440 e-mails out per day, since this form could double the amount of send mails if the user check the box to get a receipt of it, every 5 minutes means you can't use more than 576 e-mails (from this form only) per day. Which is way under our limit ratio.
Plus, this could help a bit avoid to get our mailbox full of spam.

Third, I have a list of disposable mail domains, such as the yopmail's one, that is rejecting on sight any of these disposable mailbox. But the amount I get is far, FAR from being actually effective, I do not rely my security against bot on that. It's another layer that is not very important, in the end, I think.

Fourth, I had an honey pot hidden in the HTML code, but so far it was giving more issue than anything. I still need and want to add one, simply again, as another layer for security.

Other solutions I heard off

As a counter-measure against bot, I saw different kind of way.

I could check how fast the form was fulfilled, and find a correct value that is reasonable for an human.

I could try to reject request made from some type of networks, such as VPN, Tor or even by country… I mean, this form will probably never be reached in a legit way by non-french speaker (in my own case), and other peoples taking my code (oh dang you are crazy) for their own could block country that is not legitimate to them and are usually in their experience giving issue. But even if VPN or Tor are blocked atop of this, I'm pretty sure you could still get some IP to works for you in the desired area if you are really motivated to annoy us.

I could use a list of blacklisted IP, which could before reaching the form would have been blocked in my firewall. Again, IP isn't a real good data to works this kind of issue on, merely you do it as another layer ?

CRFS protection, decreasing the number of way you could try something.

There is way more, which start to lead me to several head-scratching and lost my self into «Where I should even start ? What would be a priority ?».

In conclusion…

What would you recommend to protect a web mail form against bots and many other malicious attempt against this ? While old CAPTCHA are non-effective and not compatible with our public, some more modern seems to wants our datas instead of protecting us, and modern CATPCHA are relying on solution I would like (if possible) to avoid.

Any ideas, experience to share ?

Regards,
GASPARD DE RENEFORT Kévin

[grknight]

I recommend to try Freindly CAPTCHA which works by using the browser to do things to verify if it is a person or not. They are always updating it and based in Europe and GDPR compliant.

[pietinger]

Moved from Gentoo Chat to Networking & Security.

[Banana]

There is no silver bullet and you can not create the a solution for cases you do not have (yet).
First, try to get to know your threat and prepare for that. And only for that. New threats will come and will need future solutions.

Also, do not re-event the wheel. PHPMailer is a good start.

It is written using PHP8, no framework and rely on some classes such as PHPMailer and PHPDotenv. Written from scratch as an occasion to actually learn PHP.

Please have a look at: https://cheatsheetseries.owasp.org/ and especially: https://cheatsheetseries.owasp.org/chea ... Sheet.html

The form takes the firstname, secondname, zipcode, e-mail (for answers) and a message, you can ask to receive a copy of the e-mail in your own mailbox and you have to validate our rules by checking a box before trying to send it. If everything is good (type of data for each field, length, etc), then the mail is stored for about X minutes, in a .json format, before a cron task send it. This cron task call a .php file that is using PHPMailer to send the e-mail with our SMTP server we pay for. The mails are send only one at a time, I'll give more details about it later on in this topic.

File storage and access can be a bigger threat, then storing it in a DB. Also, if the project is getting bigger, handling DB records is easier then a lot of files.
If you may use a DB, have a look at: https://sqlmap.org/

This is a web mail form that'll go public this year if everything is ready for.

More visibilty, the greater the attack surface.

The server hosting is managed by me at home, on my own computer, on Gentoo with an hardened kernel. I try to take great care of the state and security of this machine. Firewall, keeping a eye on everything, trying to automate the process of keeping it alive, updating it very often and trying as much as I can to run only what's needed to me and my hosted sites. Nothing professional of course.

You are dealing with multiple attack vectors. Make sure you can handle them all. Securing a web application is only a small part of runnuing everything on your own.

I can only second the mentioned Friendly captcha solution grknight mentioned. But it will not stop any spam from "valid" users. It does not check the content of the data.

Here are some more tips

- Use a csrf token: https://stackoverflow.com/questions/520 ... es-it-work
- Read about php_value post_max_size and php_value upload_max_filesize
- Take a look at: https://perishablepress.com/7g-firewall/
- Do not trust user input. Validate in the client for usability, but also check at server level.
- Check for what you want. NOT for what you do not want.
- You can also limit the number of posts from that user/cookie (throttling), which can prevent certain forms of automated abuse.
- Do not try to be smart. Attackers are smarter
- Keep it simple. Less code, less bugs, less problems.
- PHP is a good language and can handle this scenario. Use the language you are comfortable with. There is no "the best" programming language.


If you plan to make your code public and want some feedback, just let me know.

[pjp]

The form takes the firstname, secondname, zipcode, e-mail (for answers) and a message, you can ask to receive a copy of the e-mail in your own mailbox and you have to validate our rules by checking a box before trying to send it. If everything is good (type of data for each field, length, etc), then the mail is stored for about X minutes, in a .json format, before a cron task send it. This cron task call a .php file that is using PHPMailer to send the e-mail with our SMTP server we pay for. The mails are send only one at a time, I'll give more details about it later on in this topic.

Where is this from? I didn't notice it in the owasp links, nor PHPMailer's github page (seemingly their primary site?).

[Banana]

The form takes the firstname, secondname, zipcode, e-mail (for answers) and a message, you can ask to receive a copy of the e-mail in your own mailbox and you have to validate our rules by checking a box before trying to send it. If everything is good (type of data for each field, length, etc), then the mail is stored for about X minutes, in a .json format, before a cron task send it. This cron task call a .php file that is using PHPMailer to send the e-mail with our SMTP server we pay for. The mails are send only one at a time, I'll give more details about it later on in this topic.

Where is this from? I didn't notice it in the owasp links, nor PHPMailer's github page (seemingly their primary site?).

I do think you missed something to quote? What you quote is from kgdrenefort

[Zucca]

I've seen some sites using Anubis for bot detection. I don't have any personal experience of using it.
https://anubis.techaro.lol

[pjp]

Where is this from? I didn't notice it in the owasp links, nor PHPMailer's github page (seemingly their primary site?).

I do think you missed something to quote? What you quote is from kgdrenefort

Hah! Indeed i missed that.

[kgdrenefort]

Thank everyone for all your answers.

I recommend to try Freindly CAPTCHA which works by using the browser to do things to verify if it is a person or not. They are always updating it and based in Europe and GDPR compliant.

The monthly fee hurts a bit to be honest, but that is still a very good kind of link I want in my bookmarks, thanks!

Moved from Gentoo Chat to Networking & Security.

Ouch it hurts !

There is no silver bullet and you can not create the a solution for cases you do not have (yet).
First, try to get to know your threat and prepare for that. And only for that. New threats will come and will need future solutions.

I know by experience that crawling bot could easily hit it. A good example is our half-open forum: I did made a mistake for non-registered users, I was sure you could not post. Seems the interface wasn't explaining this to me well enough (can't be me ! :P) and we had a wave of bots attacks. Not much, closed the forum the time to ban that and fix my mistakes.

My current threats for this projects are so far in our knowledge:

- Bots trying to fill for filling our mailbox
- Bots trying to duck off our rates limit of our ESP
- Bots trying to send spam or worst with link inside to our mailbox
- (D)DoS by spamming, I mean if there is PHP behind it could means each request is heavier than a simple GET to index.html. Right ?
- Any kind of CRFS or XSS attack that could lead to bad things, a request trying to get access to root by downing the web server with a buffer overflow… And all of kind of code issue I can't remember or I'm not even aware of…!
- Mostly, possible other project looking alike ours and, for some reasons, motivate them to attack us. That sounds silly, but this is something you can see in game server community, association sometimes…

Also, do not re-event the wheel. PHPMailer is a good start.

It was at first one of my goal but after some search it seems I have not the skill and also this is probably not the best idea. If I'm not good enough to manage the RFC and workflow of mails it is indeed safer to rely on PHPMailer which seems to be very well working in our conditions so far in my tests.

File storage and access can be a bigger threat, then storing it in a DB. Also, if the project is getting bigger, handling DB records is easier then a lot of files.

Literally, for now, I just store .json file as this:

Code: Select all

{
    "firstname": "K\u00e9vin",
    "secondname": "GASPARD DE RENEFORT",
    "email": "xxx@koshie.fr",
    "zipcode": "75012",
    "subject_prefix_list": "demande_renseignement",
    "body": "Ce sujet est valide.\r\n\r\nCe texte est assez long pour pouvoir passer le nombre minimum de caract\u00e8res, car il est assez long, simple, non ?\r\n\r\nCependant, il est \u00e0 not\u00e9 qu'il est int\u00e9ressant de tester ce champ avec plusieurs lignes, s\u00e9par\u00e9 par un retour \u00e0 la ligne.",
    "IP": "192.168.1.254",
    "date_and_time": "2025-05-31_110432",
    "send_copy": false
}

This is an example of accepted e-mail that has already been send.

Then they are moved from one place to another (pending queue to ACCEPTED, REJECTED…).

Since only one mail at a time is send, there is not much to take care of, once they are send they are moved and only the oldest file is taken in account for the next send. I guess if I had 1 million files holding in this directory my scandir() function will die somewhere, of course.

I already asked myself about database, it could come to my project later, but not now.

More visibilty, the greater the attack surface.

And we get more and more popular slowly.

You are dealing with multiple attack vectors. Make sure you can handle them all. Securing a web application is only a small part of runnuing everything on your own.

That's why I'm a bit paranoid.

I can only second the mentioned Friendly captcha solution grknight mentioned. But it will not stop any spam from "valid" users. It does not check the content of the data.

It seems my code handles it pretty well, for now.

Each field has a minimum and maximum allowed length. I took care of what was necessary for names and other kind of datas. Possibly now you can't send non expected data to my form. To ensure that, I have 2088 test cases that are still in works but soon ready to be used to validate the next update of my code.

It test a lot of cases, from each field being invalid (one at a time, two at a time… etc) with length (under min, above max) and characters pattern checking, but also with one or more empty fields, … I try to test **all** possible combinaisons that comes to my mind. That is still under work, but this is a list of my test cases names: https://bpa.st/UACA

Thanks, by copy/pasting this I realize I miss a lot of cases to my tests, which are several non-valid fields at the same times ;) ! More works, yay!

Use a csrf token

That is in my list of things to do for this case.

Read about php_value post_max_size and php_value upload_max_filesize

Did not done that in a while, will do.

Take a look at: https://perishablepress.com/7g-firewall/

Will do thanks.

Check for what you want. NOT for what you do not want.

In my opinion, this is wrong. You only validate a very small amount of things, in my 2088 cases only 16 cases are leading to an actual mail sending.

If you don't check for things you want to avoid, how can you be sure your code reject what it is supposed to ?

You can also limit the number of posts from that user/cookie (throttling), which can prevent certain forms of automated abuse.

This is also in my list.

If you plan to make your code public and want some feedback, just let me know.

The code is already available on my github. If I did not pushed it yet it's because the code base wasn't update for at least 3 months, because these last 3 months were 100% about my tests cases. Once they are ready, this week-end or next week, I would be able to update my code base with a sets of tests.

They took a lot of time to run, these 2088 cost me 3h30 each time I need to run it. I can select a sets of tests of course to test specific cases. But at least, with that I cover more than 2k cases scenario without much effort (since all of it was did).

I've seen some sites using Anubis for bot detection. I don't have any personal experience of using it.

Yes I see that yesterday and very glad to have found it, I will take a look at it because it seems to almost do what could Cloudflare for bots, but self-hosted, as a free software… Looks very cool !

Thanks for your answers everyone.

PS: In a few days or weeks I might come back with the link to my project, updated.

Regards,
GASPARD DE RENEFORT Kévin

[pietinger]

Moved from Gentoo Chat to Networking & Security.

Ouch it hurts !

Why? In ‘Gentoo Chat’ sometimes unimportant topics can be dealt with, while in ‘Networking & Security’ only deals with really important topics. You were careful and ‘just’ threw it into the chat. But it is important and then I have to ‘upgrade’ it.

[Banana]

@pjp I still dont get your question, but it may because I'm sick and the weather is a rollercoaster

In my opinion, this is wrong. You only validate a very small amount of things, in my 2088 cases only 16 cases are leading to an actual mail sending.
If you don't check for things you want to avoid, how can you be sure your code reject what it is supposed to ?

That are a lot of testcases

Example:
If you want to make sure that an input is a number and a number only. You can try to start to check for all the chars you can think of which are NOT part of a number. And you will still miss some.
So do it the other way around. Since the input is a string, replace/remove everything which is a digit. If something is left (string is not empty/null), you are sure that the input has other chars than digits you do not want.

[NeddySeagoon]

kgdrenefort,

wrote:
Check for what you want. NOT for what you do not want.

In my opinion, this is wrong. You only validate a very small amount of things, in my 2088 cases only 16 cases are leading to an actual mail sending.

If you check for what you want/expect, then everything else is rejected.
When you check for what you don't want/expect, its easy to miss something.

[Hu]

Regarding Anubis, I have a small bit of experience using it as an end user. Specifically, I have a zero percent success rate getting into sites that have decided to hide their content behind it, because the one time I unblocked enough scripts to get it to run, it used up far too much CPU time. Most (all?) such sites don't even provide proper contact information to report the problem, so I'm just slowly losing access to things as sites lock up their content behind it. It's very frustrating.

[kgdrenefort]

Regarding Anubis, I have a small bit of experience using it as an end user. Specifically, I have a zero percent success rate getting into sites that have decided to hide their content behind it, because the one time I unblocked enough scripts to get it to run, it used up far too much CPU time. Most (all?) such sites don't even provide proper contact information to report the problem, so I'm just slowly losing access to things as sites lock up their content behind it. It's very frustrating.

That is very interesting, I would have to test that.

If you check for what you want/expect, then everything else is rejected.
When you check for what you don't want/expect, its easy to miss something.

Not if you test everything, I could show you in more details (and share here) my testing scenario, I've hit +8k cases, a lot of it is covered.

And I also test things that is specifically rejected on the code side, allowing me to be sure it does the job I want.

Why? In ‘Gentoo Chat’ sometimes unimportant topics can be dealt with, while in ‘Networking & Security’ only deals with really important topics. You were careful and ‘just’ threw it into the chat. But it is important and then I have to ‘upgrade’ it.

No worry, merely joking.

That are a lot of testcases :-)

Example:
If you want to make sure that an input is a number and a number only. You can try to start to check for all the chars you can think of which are NOT part of a number. And you will still miss some.
So do it the other way around. Since the input is a string, replace/remove everything which is a digit. If something is left (string is not empty/null), you are sure that the input has other chars than digits you do not want.

Indeed, a lot.

Using DataDriver is quite of nice, allowing you to work on every scenario with a single template and then, only the .csv files are edited to guide the testing cases the way you want, again I'll share this a bit later :).

Thanks for your answers.

[NeddySeagoon]

kgdrenefort,

Not if you test everything, ...

I'm confident that you won't do that. It's effectively saying that you are cleverer than all your attackers put together.
That is unlikely.

[kgdrenefort]

kgdrenefort,

Not if you test everything, ...

I'm confident that you won't do that. It's effectively saying that you are cleverer than all your attackers put together.
That is unlikely.

By everything, I mean everything an user should be able to do, or a bot acting like an user, by trying to tweak and find any issue in the testing of one of my field.

My tests won't cover other things that «what a browser could actually do, as a '''user'''», sure.

But I try to cover all these cases nonetheless in hope to reduce as most as possible the risk.