Sniffing in the Sun: History of a Disaster

February 1994

The Warning

On February 3rd, 1994, Computer Emergency Response Team issued the following warning. What followed was a flurry of activity by the media, system administrators and users. CERT came out with a software "patch" to detect the sniffer which had brought about this panic. Rumours and accusations flew: How accurate was the warning? What's going on? Is the Internet under attack? Users are still suffering the consequences of the sniffer attack, now, some 6 months after the announcement by CERT. This article profiles the people, players and code involved.

For most people, the story begins here, with this: the "official warning":

                         February 3, 1994
                 Ongoing Network Monitoring Attacks
-----------------------------------------------------------------
In the past week, CERT has observed a dramatic increase in reports of
intruders monitoring network traffic.  Systems of some service providers
have been compromised, and all systems that offer remote access through
rlogin, telnet, and FTP are at risk.  Intruders have already captured
access information for tens of thousands of systems across the Internet.
The current attacks involve a network monitoring tool that uses the
promiscuous mode of a specific network interface, /dev/nit, to capture
host and user authentication information on all newly opened FTP, telnet,
and rlogin sessions.
In the short-term, CERT recommends that all users on sites that offer
remote access change passwords on any network-accessed account.  In
addition, all sites having systems that support the /dev/nit interface
should disable this feature if it is not used and attempt to prevent
unauthorized access if the feature is necessary. A procedure for
accomplishing this is described in Section III.B.2 below.  Systems known
to support the interface are SunOS 4.x (Sun3 and Sun4 architectures) and
Solbourne systems; there may be others. Sun Solaris systems do not support
the /dev/nit interface. If you have a system other than Sun or Solbourne,
contact your vendor to find if this interface is supported.
While the current attack is specific to /dev/nit, the short-term
workaround does not constitute a solution.  The best long-term solution
currently available for this attack is to reduce or eliminate the
transmission of reusable passwords in clear-text over the network.

But the story really doesn't begin with the "OFFICIAL WARNING".

Sexy sites. Hackers love sexy sites. Well known users, famous/interesting people. Security experts. Ambience. This makes for the sexy site. Hacker bait. Systems like The Well and Panix are "sexy sites", full of desireable users, with desireable information.

According to an unidentified participant in the Panix hack, there were "many user and security experts such as Shabbir Shafdar, Phiber Optik, etc. The size of the system and users from Sun, Inc., EFF, HP, and SGI contributed to its magnetism."

Spiders, crackers, system hackers--pick your term. Some of the individuals who made use of the sunsniffer -are- malicious. Case in point, a group using the name of "Posse". Security experts say some members of this group are based in Philadelphia and Phoenix. Those in the know say this group is well known in the hacker community as malicious manipulators who take down systems for the thrill of watching them crumble. According to information published in the Wall Street Journal, this group is under investigation for their alleged break-ins and compromise of multiple systems.

Yet, our sources for the information we're bringing you today show no tinge of malice or ill-intent. In fact, they have attempted to bring these flaws to the attention of security "professionals". for a long time, well before CERT made this announcement. The reaction on the part of many security experts has been less than appreciative. Whatever the reasons for this, the ones paying the largest price are the users. Users who have lost their privacy, been compromised, and in many cases are still not even aware of it!

Who is "The Posse"?

Chris Goggans, former hacker turned security consultant, had this to say about the break-ins: (quote used with permission, from the firewalls mailing list)

"The biggest perpetrators of the recent break ins (recent meaning the last year or so) have been a group of miscreants who are oftimes referred to as "The Posse. ".

They, and their friends, are located in Pennsylvania, New York/New Jersey, Ohio, Arizona, and Florida.

One of the PA residents, and the FL person, involved in the breakins has parted ways with the two main people involved due to in-fighting among their little group. The New York/New Jersey parties are not as actively involved in the hacking, but perform needed social engineering and phone related tricks for the group in exchange for other favors. The main antagonists are both in their late teens...a PA data entry clerk, and an OH hotel desk clerk.

Their main method of attack involves getting root on a site then monitoring incoming and outgoing traffic using ethernet sniffers (on suns since they are too pathetic to port their swiped esniff.c program to run on ultrix or other variants) and capturing all tcp activity. They then use this information to get in other hosts and start over.

They have programs that allow them to get ypmaps from remote (ypsnarf.c); to nfs mount damn near anything; to get root using sendmail, rdist, the mult bug, and others.

They have patches to allow them the ability to place backdoors in login and in.telnetd, and to run other shells to let them jump over firewalls. They have utilities to remove themselves from wtmp, utmp, pacct, ps, and netstat. Unless you have a tcp-wrapper going, you probably wont notice them.

I would estimate that about 25% of the American Internet is compromised. This is predominantly university traffic but since these are the people behind breakins at The Well, CNS, Panix, NSFNet, BarrNet, Sun, and others, its pretty safe to assume that they have a lot of fun addresses to play with.

Although they have amassed a HUGE amount of hosts through their sniffing, it is unclear as to what they want with the hosts. The predominant motive appears to be the ability to get on IRC anonymously and send ICMP floods to servers and annoy people. They also play games impersonating people on netnews and mail, they fake hacking attempts in order to try to frame people, they play phone games and prank people over and over or otherwise disconnect or disrupt service, they get credit information or billing records to spread around, etc. (As I said before, its pretty pathetic)

The real crime here is that the authorities know precisely who is involved, and it persists. One individual was even involved with the MOD busts a few years back and is no longer a minor. Perhaps this time his father won't be able to intervene.

They really don't seem to care what happens to them, and they know full well that the authorities have been questioning people about them, yet they persist. Obviously the illusion of power on the net is far more desirable than their petty real lives."

How did this happen?

According to an article published in Secure Computing, "Hackers are all different. Some are very methodical, working their way down a checklist. Some never get beyond looking for suid programs lying around. Some only use one hole, relying on it to the exclusion of everything else." How did the sunsniffer get into the Panix machine, where it could begin its task of finding your passwords, and other information?

1. Portmapping (Portmapping is a simple function used to see what is running on what ports).
2. Open access to UUCP: addition of the "+ +". to the account, enabling remote shell access via a command such as rsh site.com -l uucp /bin/chs -bif. (If an NFS server has an /etc/exports file containing an "-access=" string longer than 256 bytes, this file system is exported to world.

   (This is a well known bug in rpc.mountd. This was now patched, so the uucp bug is used.)

3. Root is obtained to enable snooping of any outgoing sessions. (to enable reading of data from any /dev/nit device and access all system files)

"When I got in, I noticed that most and trivial security holes were plugged they fixed rdist, all the known vulnerabilities in kernel, expreserve, loadmodule, LD_* environment variables, however I paid attention to some suid script in /usr/local/something dir, SUID scripts owned by root is a biggest hole I have ever seen. I have been laughing for some time then all I did was ln -s /usr/local/something/script ./-i and then I executed ./-i, and it forked /bin/sh -i (bourne shell in an interactive mode) as root. so was I...

Boring? :)

I looked at their syslogd.conf, inetd.conf files, and there was nothing suspicious, they didn't log anything, so I wasn't worried. also I looked at all the running processes, and again, I haven't found anything unusual, or unknown...

I looked in /usr/spool/mail..reading phiber optik's mail was fun :) [I still have his mbox :-)] then I have scanned all the .rhosts files, to see what they got, and only then I was looking at files of some users..

I noticed that they run a shadowing password scheme, and I couldn't find online source of /bin/login to patch it, hence I grabbed the source of crypt.o routine of the shared libraries, recompiled the shared libraries (It was fun 'coz they ran some dummy resolving modules in it) replaced crypt.o , fixed ctime/atime/mtime of the file, and bingo. the backdoor was planted.

Well, it was the shared libraries backdoor, as I mentioned. Not too many crackers used it by that time, and even CERT did not recommend to system admins to check integrity of the shared libraries, aside of that not all the admins know what shared libraries do after all :) Shared libraries are not included in the list of all critical files ( no suid bit..:-)

I decided to run a sniffer on Panix only in 2 months after I planted the backdoor :) lame admins eh? 2 months! I noticed there was a site called news.panix.com with only a few users, and it wasn't overloaded with tons of processes as panix.com (their nfs server and main site ) also, I noticed that the site was pretty quiet, the admins almost never log in, and etc. and the site was on the same data-segment of the router as panix.com, hence it was pretty nice to see all the outcoming sessions from panix.com :) the sniffer was there, and I run it for about 4 (!!) months. totally I got only 90-100 MB of logs, not a lot ..I expected more. I was logging in once a week, and picking up all the logs :) I am sending you 1.6MB of logs. hope you find them funny :) ".

(The logs contained information which appeared to be legitimate. and funny. and now, deleted.)

How widespread is the problem? CERT announced a "dramatic increase" in monitoring. Increase? How much is tolerable? If one system is known to be monitored, is that acceptable? How many does it take before it becomes worthy of issuing a patch or warning? Commercial service providers are seen by their users as secure places to transact business. The information superhighway is being built on the premise that the ground under these roads is stable. If it isn't, shouldn't those travelling the roads, negotiating the turns and using the bridges be informed they are subject to collapse at any time? Not only are thousands of 'users' analogous to people driving without a license being turned loose on the information tollroad, other users are expecting the roads to be 'policed' and 'safe'. We know they are not. CERT knows they are not.

Why now? Why release this information now and not a year ago when it was abundantly clear to anyone with any insight into the hacking scene that this was going on? If the Panix public system had not publicly announced their compromise, would it have taken yet longer for this to be brought to the attention of the people? You've seen above how simple the break-in was to accomplish. Some folks think Panix was wrong for going public. Sweep it under the rug, they say..with the rest of the dirt...

In the matter of this particular "wide spread" compromise, however, there is an interesting twist. CERT issued a detector for the sniffer; however, there is already a modified version of the sniffer which can easily evade this detection. And, the hackers have come up with a detector for their modified version of the sniffer. It's part of what seems to be an ongoing contest. Holes are found, patched, and new ones are found. Who's to blame? Some would say CERT is to blame, for being slow in announcing the fixes. But, is this fair? CERT is only made up of people, and they certainly aren't magicians. They can't know every hole in ever OS, and its not their fault the holes are there, or that they are discovered. Some would say system designers. But, can we expect them to know every possible way their design may be exploited? If they waited til every possibility in the world was known, they would never release a single system. Hackers? They find the holes. Some say they are irresponsible for even looking for them. Users? They don't insist the systems they use are up to date patching holes; for the most part, they don't even know the holes exist! We don't know who to blame, but we do know a little about the sunsniffer.

The Meat What is this sniffer that is getting all of the recent attention?

sunsniffer.c: the idea:

The device /dev/nit is the network interface tap. It can be configured to read/write from/to different interfaces. The program sunsniffer.c configures it to read from some ethernet interface like le0, le1, etc. Roughly speaking this utility opens /dev/nit in READ_ONLY mode and reads each incoming packet from the chosen ethernet interface. Several headers (layers) have to be stripped on different levels as well, as not all packets may be of interest to the sniffer owner. He's only looking for logins via ftp, telnet, or rlogin. To find them, the sniffer opens /dev/nit for reading. An infinite loop is set up to read incoming packets and strip the headers from them. The 'meat' of the data is written to a log file.

Basically /dev/nit is a device where from one can read *ALL* travelling packets on the specified interface (in this case some ethernet interface). These packets hold things like your login name. Your password. Your confidential information.

One can read ALL travelling packets on the local ethernet medium (cable) only if the device is put into promiscuous mode, at least in this version of the program. A modified version of the sniffer, which allows reading of packets only destined to the local machine (the machine the sniffer is running on), does not require promiscuous mode. And, if someone uses this modified version (which is easily enough configurable for any beginner ueberhacker), they can evade detection by the CERT fix. In fact, a fix for this modification has been written by Beavis and Butthead, and circulated via the computer underground. Uhh huh..huh....

Setting the interface into promiscuous mode is the most malicious act of the sunsniffer, because in such a mode one can extract the confidential info from packets not even related to the machine he is on - *all* packets travelling on the local ethernet cable. What if there is a backbone router on the same cable? The ueberhacker would be able to retrieve information for some totally remote sites, which use a machine connected to the same ethernet cable only for routing purposes.

Sniffing is not a new technique. It's been around for ages. There are sniffers which log all of the keystrokes on DOS based machines. There are Mac based sniffers. There is even rumour of sniffer programs for the Amiga. However, unlike with most other sniffing programs, physical access to the machine is not required. The tool used to accomplish this 'sniffing' is not new. Its been around for at least a year, maybe longer. CERT's "fix" failed to mention that sniffing is not limited to UNIX machines. CERT's initial announcement failed to mention that the removal of /dev/nit will not remove the threat of sniffers. In fact, removal of /dev/nit may well cause other problems, but in the land where passwords are unencrypted and holes are unpatched though solutions DO exist, what are a few 'problems'. It gets much better. The sunsniffer.c program was found on several public FTP sites. Since the time we found them, they have been removed. The question is, how many other people found them before they were removed?

An Ongoing Problem

Why go to all this trouble? Here is what one of the WELL hackers had to say about it:

" After a while, one system begins to seem like another. There just seems to be no point sometimes. But then you pick up a magazine and see about a groovy new software package that is coming out, or a new online service with influential people on, and you start getting the urge. Hacking has been so romanticized by the media that you can't HELP but get excited sometimes.

I deleted almost all of it except a few logs and billy idols email. It really was kinda boring. not exactly bedtime reading. I think people had this idea that all sorts of sinister things were being done with the information. The truth is, it may have, but not by us. We can't speak for the others who broke into the systems after us (or before us. it had been penetrated about 7 months prior, from what we can tell from administrator email). I know peoples private mail was captured, deleted and passed out. To do this isn't just wrong. Its idiotic. The main reason we didn't do anything with it is because there was really nothing interesting on there...or at least not anything interesting to us. Some think it was a big thing. On the contrary, it was not. It was another system with another problem with bored teenagers looking around. No nazi's. No terrorists. No prying government. "

Germany's ZDF television informed the public on Feb. 5th, that hackers had invaded a network which was in place to protect US Armed forces even in the case of a nuclear war. Users scurried to change their passwords, as news of this hack spread to radio and television, and some newspapers. Was this information accurate? According to some sources, this information came from two agencies, German press agency, and Agence France Press. The following text, taken from a Usenet comp.risks newsgroup (and translated by the poster, if we understand correctly), gives the rundown:

"Washington, February 5 (AFP) - Computer pirates have cracked the largest computer network in the world. Totally 20 million users on 'Internet' should receive new passwords, told the emergency committee installed by US ministry of defence. Internet is used by universities, government agencies, enterprises and private persons. The network was established in times of the Cold War to serve US Armed Forces also in case of Atomic War as 'invulnerable' information network. The hackers, so far unidenti- fied, succeeded according to the emergency committee to read data from ten thousands of systems on 'Internet'. They succeeded by using a program named 'Trojan Horse' which allows legal access to Internet central com- puter but then does not go any further. "

There have been some reports indicating it is not known if the hackers left 'trojans or viruses' loose on the machines. This sort of hysteria can lead to nothing good. We've seen the effects of mass hysteria during the Michelangelo "crisis". True, there was a problem. No one can deny viruses are a problem. All one has to do is a bit of investigation into the socio-economic costs of viruses to see just how much a problem they really are. But, just as happened when the doomsayers heralded Michelangelo as a coming apocalypse, we are seeing the ripples of desensitization regarding the sunsniffer incident: "Ignore the warnings, it's just more media hype. " But is it? There remain a lot of unanswered questions. Who's in control here? We can answer that question with this one:

" Who Will Save Us from Ourselves? "

You control the horizontal...you control the vertical.....we now return control of the Internet to you. Take it or be taken.

CERT CONTACT INFORMATION: CERT advisories may be obtained using anonymous file-transfer protocol (FTP) to reach cert.org. The advisories are in the /pub/cert_advisories directory. If you prefer to use the postal service, you can contact CERT at: CERT Coordination Center, Software Engineering Institute, Carnegie Mellon University, Pittsburgh 15213-3890. CERT's electronic-mail address is [email protected].

UPDATE-- Postscript: Even as this article was about to be published, word (and evidence) reached us of continued attacks on 'trusted' Internet hosts.

The administrators of rahul.net, a San Jose CA-based Internet service provider, posted a message to several security-related Usenet newsgroups stating

"On July 6, at slightly after 2 am local time (PDT, 7 hours west of UTC), an intruder installed a TCP/IP-sniffing daemon on one of the machines at a2i communications (domain rahul.net). The sniffer was discovered and disabled on the evening of the same day, about 18 hours later. During this time, the daemon collected data including passwords."

Of course, not all sniffer incidents are so public. Through the usual channels (pgp'd file sent from an anon remailer), we were given yet another look at the results of our friend the sniffer.

What we received was a file that appeared to be the logfile generated by the now-infamous sniffer. The machine belongs to a respected university, and has among its users a well-known security expert/author with close ties to CERT (the administrators of the machine have been contacted to inform them of the breach). What follows is a piece of that log. (Note: names have been crossed out or changed to prevent embarrassment & lawsuits.) Using logical device ie0 [/dev/nit] Output to stdout. Log started at => Fri May 13 20:15:20 [pid 24591]

-- TCP/IP LOG -- TM: Fri May 13 20:18:59 --
 PATH: xxxx1.yyy.zzzzzz.edu(966) => aaan.yyy.zzzzzz.edu(rlogin)
 STAT: Fri May 13 20:21:17, 192 pkts, 128 bytes [DATA LIMIT]
 DATA: :bogus:
     : mike
     : adm5/9600
     : 8o4enYzO
     : f mike@fred
     : 
     : chf(127)(127)(127)(127)(127)rlogin mark
     : f mark(127)(127)(127)(127)(127)qo(127)(127)(127)quota
     : (127)passwd
     : 8o4enYzO
     : 8o4enYzO
     : 8o4enYzO
     : rlog
-- 
 

What do we do, who do we turn to, when even the watchmen are being watched? It could be that it's time to start thinking about new approaches to problems which have been around a long time, and which are not going to go away.

About the Authors

Sarah Gordon's work in various areas of IT Security can be found profiled in various publications including the New York Times, Computer Security Journal and Virus Bulletin. She is a frequent speaker at such diverse conferences as those sponsored by NSA/NIST/NCSC and DEFCON. Recently appointed to the Wildlist Board of Directors, she is actively involved in the development of anti-virus software test criteria and methods. She may be reached as [email protected]