Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
syslog-ng-3.22.1 aborting and restarting repeatedly[solved?]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7446
Location: almost Mile High in the USA

PostPosted: Sun Jan 12, 2020 11:31 am    Post subject: syslog-ng-3.22.1 aborting and restarting repeatedly[solved?] Reply with quote

Anyone seen the behavior where syslog-ng under logging pressure, constantly shuts down and abort()... and then the syslog-ng supervisor restarts it every few seconds?

Code:
Jan 12 04:28:17 host1 syslog-ng[18118]: syslog-ng starting up; version='3.22.1'
Jan 12 04:28:17 host1 sshd[18108]: Received disconnect from 178.33.67.12 port 34914:11: Bye Bye [preauth]
Jan 12 04:28:17 host1 sshd[18108]: Disconnected from invalid user admin 178.33.67.12 port 34914 [preauth]
Jan 12 04:28:18 host1 syslog-ng[18122]: syslog-ng starting up; version='3.22.1'
Jan 12 04:28:18 host1 sshd[18114]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.31.166  user=root
Jan 12 04:28:19 host1 syslog-ng[18127]: syslog-ng starting up; version='3.22.1'
Jan 12 04:28:20 host1 sshd[18114]: Failed password for root from 222.186.31.166 port 62744 ssh2


Trying the unstable versions to see if it's any better, but not sure if I'll fall asleep before finishing, it's late and I'm under attack from the botnet...

---
EDIT
---
Looks like though re-emerging 3.22.1 did not do anything, emerging 3.25.1 appears to have solved the behavior. Something weird going on here, any ideas?

The botnet is still hitting me hard *sigh* getting tons of unique hosts hitting...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
cboldt
l33t
l33t


Joined: 24 Aug 2005
Posts: 884

PostPosted: Sun Jan 12, 2020 1:11 pm    Post subject: Reply with quote

I've not seen the stop/start behavior you describe, that is under load, but 3.22 deliberate stop (using OpenRC) is buggy.

Later versions work fine for me.

Edit to add - I run an active firewall, and watch sshd attempts to login. Hundreds a day, sometimes at a fast clip, e.g., 20 a minute or more. Never had syslog-ng crash, that I know of, due to load. I'd notice it because logging include PID of the "watch-logs" banning script, and that script restarts every time syslog-ng does, for any reason. Never had the "watch-logs" PID change without deliberate action on my part. Not even under 3.22
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7446
Location: almost Mile High in the USA

PostPosted: Sun Jan 12, 2020 6:23 pm    Post subject: Reply with quote

Hmm... this is still most curious. Not sure what's going on here.

I did notice that as I tried to stop 3.22 just before starting 3.25.1, I see a lot of segfaults show up with 3.22, which would start to explain some things but not exactly where.

And I'm still under attack... sigh. Why is my machine so important to them?? Getting one ssh attack every 5 seconds on average for the past 60 hours or so.

And just this past 12 days of this year, I've been hit by over 3000 unique machines. This is so disgusting that this many people have gotten their machines compromised... Are people outrightly individually banning 3000 unique hosts?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
cboldt
l33t
l33t


Joined: 24 Aug 2005
Posts: 884

PostPosted: Sun Jan 12, 2020 10:16 pm    Post subject: Reply with quote

I get the same sort of action, and it came out of the clear blue. The mass attack was a real world stress test of the log-watch and ban script, revealing several significant weaknesses (flaws) in the script logic and methods.

I'd a history with an average of somewhere between 50 and 200 banned IP ranges at a time, from about January 2018 until February 2019. Then, in one day, the number went to 4000 banned IP ranges (ban uses CIDR /24). iptables logs attempts to hit my sshd port (not 22, but not an uncommon choice either, 2222), and the log was scrolling pretty quickly. Ban script tolerates 6 hits each half hour, or 3 failed password. At some point the script imposes a fire rule that rejects everything from the offending IP (range). In that one day, a CIDR range was banned, on average, each 20 seconds.

Somebodies have cranked up the script kiddie nets, and once an IP address is ID'd as having an open sshd port, the news is spread around. Interesting selection of failed usernames too.

I've considered using the portknocker for more than whitelisting myself from away. Meaning, I've considered closing the sshd port to all except those let in by portknocker.

Edit to add: Yes, the script literally bans thousands of "individual" (CIDR /24) hosts. They are all in an ipset. Bans have a life of weeks, then are lifted. No way could that be done manually.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7446
Location: almost Mile High in the USA

PostPosted: Sun Jan 12, 2020 10:49 pm    Post subject: Reply with quote

Yeah once again I can't take port knocking (security should not be inconvenience), unless this problem becomes so bad it's eating a large portion of my limited link bandwidth. In that case I may have to use an HTTP based port knocking - more like simply submitting a HTTP request of some sort to open the ssh port, and then close it after a few minutes. Sure someone can sniff it, but not the whole world...

---

oh crap. You know what, the skriptkiddiez nmapped my machine...
sigh.
why me.

---

and you know what else is stupid on my part? Lots of stupid on my part?

-- I haven't checked logs since forever due to logrotate. This has been happening for several months.
-- I noticed that I had a huge increase in attacks in the past 6 months (!)
-- My logfiles have increased from 1MB compressed per month to over 15MB compressed in the past year, most due to these ssh attacks.
-- So far this month (only half way through) I've gotten 133MB uncompressed logs:

133MB total
123MB without the syslog-ng problems
24MB without my logging of known bad hosts/subnets trying to connect again.
0.5MB without sendmail stuff, mostly from people trying to relay spam through my machine.
0.05MB without xinetd logs, most of which are libwrap denied machines logs
0.02MB -- remainder of stuff that I really should look at if there are problems with my machine unrelated to security.

sigh. And this is a HOME machine, not a company machine...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
cboldt
l33t
l33t


Joined: 24 Aug 2005
Posts: 884

PostPosted: Tue Jan 14, 2020 1:10 pm    Post subject: Reply with quote

Yep. Same for me. Home machine. I have 2222 and 2223 to sshd, separate machines, router directed the requests. 2223 was mostly quiet "forever," then got lively a few weeks ago. Lots of common IP intrusion between the two targets, I checked that over the weekend just for kicks. Fun to track the dictionary login names too.

Like you, I have no concern about breach of ssh. I do wonder if the scriptkiddiez would "skip it" if those ports were closed.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum