|
 |
|
The Debian administration team and security experts are finally able to pinpoint the method used to break-in into four project machines. However, the person who did this has not yet been uncovered. The package archives were not altered by the intruder. The Debian administration and security teams have checked these archives (security, us, non-us) quite early on in the investigation and re-installation process. That's why the project was able to open up the security archive again and confirm that the stable update (3.0r2) wasn't compromised. If the project had anticipated to get compromised at the same time the stable update was implemented, the involved people would have postponed it. However, the updated packages were already installed in the stable archive and mirror servers at the time the break-ins were discovered, so it wasn't possible to hold it back anymore. Several methods based on different control data were used to verify the packages and to ensure that the archives weren't altered by the attacker:
TimelineBelow is the timeline of discovery and recovery of the compromised machines. All times are in UTC. Some times are only estimates since our conversation did not contain exact timestamps. Sep 28 01:33 Linus Torvalds releases 2.6.0-test6 with do_brk() fix DiscoveryOn the evening (GMT) of Thursday, November 20th, the admin team noticed several kernel oopses on master. Since that system was running without problems for a long time, the system was about to be taken into maintenance for deeper investigation of potential hardware problems. However, at the same time, a second machine, murphy, was experiencing exactly the same problems, which made the admins suspicious. Also, klecker, murphy and gluck have "Advanced Intrusion Detection Environment" (package aide) installed to monitor filesystem changes and at around the same time it started warning that /sbin/init had been replaced and that the mtime and ctime values for /usr/lib/locale/en_US had changed. Further investigation revealed the cause for both these problems to be the SucKIT root-kit. It includes password sniffing and detection evasion capabilities (i.e. tools to hide processes and files) which are installed directly into the kernel, which in turn caused the oopses that were noticed. Detailed Attack AnalysisOn Wednesday, November 19th, at approximately 5pm GMT, a sniffed password was used to log into an unprivileged developer account on the host klecker (.debian.org). The attacker then retrieved the source code through HTTP for an (at that time) unknown local kernel exploit and gained root permissions via this exploit. Afterwards, the SucKIT root-kit was installed. The same account and password data were then used to log into the machine master, to gain root permissions with the same exploit and also to install the SucKIT root-kit. The attacker then tried to get access to the host murphy with the same account. This failed because murphy is a restricted machine and its only purpose is to act as list server to which only a small subset of developers can log into. Since the initial login attempt didn't work the person used his root access on master to access an administrative account which was used for backup purposes and gained access to murphy as well. The SucKIT root-kit was installed on this host as well. On the next day the attacker used a password sniffed on master to log into gluck, get root there and also install the SucKIT root-kit. The forensic analysis revealed exact dates and times when the program /sbin/init was overwritten and the root-kit installed. The analysts also discovered the executable file which was used to gain root access on the machines, which was protected and obfuscated with Burneye. Upon unwrapping and disassembling the exploit, security experts discovered which kernel bug was utilised. An integer overflow in the brk system call was exploited to overwrite kernel memory (change page protection bits). By doing so the attacker gained full control about the kernel memory space and was able to alter any value in memory. Even though this kernel bug was discovered in September by Andrew Morton and already fixed in recent pre-release kernels since October, its security implication wasn't considered that severe. Hence, no security advisories were issued by any vendor. However, after it was discovered to be used as a local root exploit the Common Vulnerabilities and Exposures project has assigned CAN-2003-0961 to this problem. It is fixed in Linux 2.4.23 which was released last weekend and in the Debian advisory DSA 403. Linux 2.2.x is not vulnerable to this exploit because boundary checking is done before. It is also believed that Sparc and PA-RISC kernels are not vulnerable since user and kernel addresses are stored in different address spaces on these architectures. Please understand that we cannot give away the used exploit to random people who we don't know. So please don't ask us about it. RecoveryAfter the machines were shut down, images of the compromised hard disks were created and stored on a separate machine. They were distributed to the people doing the forensic analysis. The three machines in the US (master, murphy, gluck) were reinstalled afterwards and their services re-instated one by one after investigation by the relevant service admin. On klecker, however, this was postponed for a scheduled maintenance so the security archive could be brought online again sooner than the other services. At that time we also didn't have console access to klecker, so recovery had to be done remotely. After a disk-image was made via serial console login to a local machine on a firewalled network connection, the root-kit was removed, the kernel exchanged and hardened, binaries double-checked and the security archive verified against several different external sources. This machine will be re-installed in the next few weeks. As a security precaution all developer accounts were disabled in LDAP and SSH keys removed on the more important machines, so that no more machines could be compromised. This, however, effectively disabled just about any public Debian work that involved uploading files and accessing the CVS repositories. All passwords used on quantz (i.e. all Alioth, arch and subversion passwords) have been invalidated as well. All SSH authorized keys have been removed as well. Please use the lost password system to receive a new password at: https://alioth.debian.org/account/lostpw.php When all services are running again and the machines are sufficiently secured, LDAP will be reset so that developers can create a new password again (<http://db.debian.org/password.html>). It can't currently be predicted when this will happen, though. Upon recovery SSH was re-installed on the compromised machines. Hence, there are new RSA host keys and key fingerprints for these hosts. The keys will be included in LDAP as soon as they are created and can be taken from <http://db.debian.org/machines.cgi>. Consequences!! Renew your passwords! !! Since passwords were sniffed on the compromised hosts, any outgoing connection that involved a password is to be considered compromised as well, i.e. the password should be considered known to the attacker. It should therefore be changed immediately. Additionally, if somebody had access to a Debian machine and was using the same password or passphrase on other machines or keys we strongly advise to change the password or passphrase respectively as soon as possible. If an SSH key was generated or stored on one of these machines and was used to log into other machines (i.e. by installing it in .ssh/authorized_keys), it should be removed as well. The secret GnuPG/PGP keys which were found on debian.org machines were also removed from the Debian keyrings and thus deactivated. Developers who are worried about their own machines should at least run chkrootkit and watch its output. Matt Taggert maintains a backport of the current version for woody at the following address: deb http://lackof.org/taggart/debian woody/chkrootkit main Additionally, a detailed list of precaution issues is provided by Wichert Akkerman and Matt Taggart at: http://www.wiggy.net/debian/developer-securing/ SucKIT Root-KitSucKIT is a root-kit presented in Phrack issue 58, article 0x07 ("Linux on-the-fly kernel patching without LKM", by sd & devik). This is a fully working root-kit that is loaded through /dev/kmem, i.e. it does not need a kernel with support for loadable kernel modules. It provides a password protected remote access connect-back shell initiated by a spoofed packet (bypassing most firewall configurations), and can hide processes, files and connections. Usually, SucKIT is launched as /sbin/init at system bootup, forks to install itself into the kernel, start up a backdoor, and launches a copy of the original "init" binary from the parent (with pid 1). Any subsequent executions of /sbin/init are redirected to the original init. TESO's Burneye ProtectionBurneye is a means of obfuscating ELF binaries on the UNIX platform presented in Phrack issue 58, article 0x05 ("Armouring the ELF: Binary encryption on the UNIX platform", by grugq & scut). Using tools like TESO's Burneye, an attacker can alter an executable program to encrypt its true purpose, hiding it from firewall filters, intrusion detection systems, anti-virus software and the prying eyes of investigators. Thanks
Related Stories:
 
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
|
|
|
|
| All times are recorded in UTC. Linux is a trademark of Linus Torvalds. Powered by Linux, Apache and PHP 
Jupitermedia Corporation has two divisions: Jupiterimages and JupiterOnlineMedia
Legal Notices, Licensing, Reprints, Permissions, Privacy Policy.
Whitepapers and eBooks
Webcasts
Downloads and eKits
Tutorials and Demos
|
