Folks, we had a break-in on this Web server. I think it's cleaned up now, but if you have created a "visitor" account (I think very few people have) it would be wise to change your password just to be on the safe side. Discussion below.
The intruders got in through a hole in the timthumb.php script used for making thumbnails of images. That is shipped as part of the PivotX Web log software, but it's an independently-maintained open source package not created by the PivotX maintainers. It was configured to allow reading user-specified remote URLs - which I had thought I had turned off, but that setting was apparently restored to an insecure default during a previous security-related upgrade of PivotX - and it was validating the remote URLs in an incorrect way, apparently because the author of the code just plain doesn't understand regular expressions. This hole in timthumb.php was a problem for many WordPress installations, and the timthumb maintainers released a patch for it, but even the latest version still does URL validation in an incorrect (though harder to exploit) way.
I'm pretty sure that this was a general attack aimed at all vulnerable sites, not in any way specific to Ansuz (and probably not even aimed at PivotX sites - it was mostly against the more popular WordPress), and it doesn't look to me like the attackers bothered looking for interesting secrets such as passwords that might have been stored on my server. I think they just wanted a bot for a botnet. Very few secrets are stored here, anyway; and PivotX's own password database, at least, is stored in hashed form so even if someone did get it they'd still have some more work to do before they'd actually get anything useful out of it. But I'm still recommending password changes just in case.
I'm disappointed with PivotX. This is now the second time in the last year that I've had to do an unscheduled upgrade in panic mode because of a critical security issue, and I've now had one successful intrusion as well. (The previous one was only a threat - many other sites were compromised but mine wasn't.) The PivotX maintainers' response to my recent email was that I should upgrade more often, but that's not really a good response. Substituting even more unscheduled panic-mode upgrades for my current experience would not be a big enough improvement; the upgrade I didn't do promptly that might have spared me this particular intrusion was not announced as being security-critical when it was released; and I found and fixed a couple more security holes in default configurations of the latest version of PivotX while I was installing it yesterday. Even the latest version of timthumb.php is not trustworthy, so "fixing" that part means disabling it entirely and either doing without the features it provides or else hacking in a replacement. So it looks like continuing to use PivotX is going to mean I'm back to the situation I was in before I started using PivotX, where I'm still obligated to audit all the code myself and keep fixing it on a regular basis, and that's one of the big things I was hoping to avoid when I switched away from my custom-written CMS. If I must become a maintainer of the CMS code in order to have a trustworthy CMS, then maybe I'd be better off if it were my own code.
I'm thankful - and this is a good day for it - that Tom from Baremetal, my hosting provider, was on the job. The intrusion-detection systems noticed the problem and he let me know about it before the attackers could do much damage.
I'm also thankful that this happened now, and not while I was in Japan. The system was vulnerable, some significant number of people knew about the hole, and it appears from the logs that we may even have received a few unsuccessful attempts, throughout the time I was away. Fixing it from my laptop over an hotel Internet connection would not have been fun - and that's one of several reasons why "Oh, you're obligated to install EVERY PivotX upgrade IMMEDIATELY when it's announced, whether it has anything obviously to do with security or not!" is not really a workable line to take. It's important that I should be able to leave the code untouched for months at a time without that being an unacceptable security risk.
ETA: another email from the PivotX people includes the line "I can't help you with you." in response to my complaint that because of undocumented default-insecure configuration settings, I have to audit each upgrade myself before installing it. Maybe that sounds less rude in German than it does in English.