syslog-ng-3.22.1 aborting and restarting repeatedly[solved?]

[eccerr0r]

Anyone seen the behavior where syslog-ng under logging pressure, constantly shuts down and abort()... and then the syslog-ng supervisor restarts it every few seconds?

[cboldt]

I've not seen the stop/start behavior you describe, that is under load, but 3.22 deliberate stop (using OpenRC) is buggy.

Later versions work fine for me.

Edit to add - I run an active firewall, and watch sshd attempts to login. Hundreds a day, sometimes at a fast clip, e.g., 20 a minute or more. Never had syslog-ng crash, that I know of, due to load. I'd notice it because logging include PID of the "watch-logs" banning script, and that script restarts every time syslog-ng does, for any reason. Never had the "watch-logs" PID change without deliberate action on my part. Not even under 3.22

[eccerr0r]

Hmm... this is still most curious. Not sure what's going on here.

I did notice that as I tried to stop 3.22 just before starting 3.25.1, I see a lot of segfaults show up with 3.22, which would start to explain some things but not exactly where.

And I'm still under attack... sigh. Why is my machine so important to them?? Getting one ssh attack every 5 seconds on average for the past 60 hours or so.

And just this past 12 days of this year, I've been hit by over 3000 unique machines. This is so disgusting that this many people have gotten their machines compromised... Are people outrightly individually banning 3000 unique hosts?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[cboldt]

I get the same sort of action, and it came out of the clear blue. The mass attack was a real world stress test of the log-watch and ban script, revealing several significant weaknesses (flaws) in the script logic and methods.

I'd a history with an average of somewhere between 50 and 200 banned IP ranges at a time, from about January 2018 until February 2019. Then, in one day, the number went to 4000 banned IP ranges (ban uses CIDR /24). iptables logs attempts to hit my sshd port (not 22, but not an uncommon choice either, 2222), and the log was scrolling pretty quickly. Ban script tolerates 6 hits each half hour, or 3 failed password. At some point the script imposes a fire rule that rejects everything from the offending IP (range). In that one day, a CIDR range was banned, on average, each 20 seconds.

Somebodies have cranked up the script kiddie nets, and once an IP address is ID'd as having an open sshd port, the news is spread around. Interesting selection of failed usernames too.

I've considered using the portknocker for more than whitelisting myself from away. Meaning, I've considered closing the sshd port to all except those let in by portknocker.

Edit to add: Yes, the script literally bans thousands of "individual" (CIDR /24) hosts. They are all in an ipset. Bans have a life of weeks, then are lifted. No way could that be done manually.

[eccerr0r]

Yeah once again I can't take port knocking (security should not be inconvenience), unless this problem becomes so bad it's eating a large portion of my limited link bandwidth. In that case I may have to use an HTTP based port knocking - more like simply submitting a HTTP request of some sort to open the ssh port, and then close it after a few minutes. Sure someone can sniff it, but not the whole world...

---

oh crap. You know what, the skriptkiddiez nmapped my machine...
sigh.
why me.

---

and you know what else is stupid on my part? Lots of stupid on my part?

-- I haven't checked logs since forever due to logrotate. This has been happening for several months.
-- I noticed that I had a huge increase in attacks in the past 6 months (!)
-- My logfiles have increased from 1MB compressed per month to over 15MB compressed in the past year, most due to these ssh attacks.
-- So far this month (only half way through) I've gotten 133MB uncompressed logs:

133MB total
123MB without the syslog-ng problems
24MB without my logging of known bad hosts/subnets trying to connect again.
0.5MB without sendmail stuff, mostly from people trying to relay spam through my machine.
0.05MB without xinetd logs, most of which are libwrap denied machines logs
0.02MB -- remainder of stuff that I really should look at if there are problems with my machine unrelated to security.

sigh. And this is a HOME machine, not a company machine...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[cboldt]

Yep. Same for me. Home machine. I have 2222 and 2223 to sshd, separate machines, router directed the requests. 2223 was mostly quiet "forever," then got lively a few weeks ago. Lots of common IP intrusion between the two targets, I checked that over the weekend just for kicks. Fun to track the dictionary login names too.

Like you, I have no concern about breach of ssh. I do wonder if the scriptkiddiez would "skip it" if those ports were closed.