Capturing VMware Traffic and Viewing with CloudShark

Here where we make CloudShark, we have a pile of dev op and admin tools we use every day to make our lives easier: Jenkins, Capybara, and Nagios are some absolutely wonderful additions to our environment this year. One of the most powerful tools we use here is a combination of VMware Workstation and the powerful Vagrant API interface. With Vagrant, we can test every permutation of CloudShark via a barrage of automated testing.

Along the way, we had to learn some of the lower level interfaces of these tools. We became aware of a vmnet-sniffer command that comes with VMware Workstation and VMware Fusion, which we use on our OS X workstations for development.

The Big Picture

You have this VMware environment. It has virtual machines. Maybe just one, but probably more than one, and maybe way too many. And every day you’re logging into random virtual machines, trying to keep your SSH keys all in order, or worse, deploying LDAP client configurations just so you can log in for five minutes. Maybe you’re doing support services on an in-house thin client cluster, or you’re debugging some web server on some regional node. Hunting down where a VM is hiding is not always quick and easy without some copious planning.

Why go through all this trouble? You already have a tool to skip all this: vmnet-sniffer. Log into your VMware host system, capture the traffic to all your virtual machines at once, and then sort it out later (if you ask us, CloudShark is a great way to view such a capture file).

How It Works

This vmnet-sniffer doesn’t have much documentation available. Fortunately there are not many options, so it’s a quick explanation. On OS X with VMware Fusion:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-sniffer -e -w my_capture.pcap vmnet8

On Linux with VMware Workstation:

sudo /usr/bin/vmnet-sniffer -e -w my_capture.pcap /dev/vmnet8

With either of these commands, you just press control-c when you want to stop capturing.

You might be on vmnet1 if you are using host-only networks. Just give both a shot if you aren’t sure.

Note how the Linux command requires you to specify the full device path, but OS X does not. In both commands, the argument order is imperative! Otherwise you will probably get a nice text report but no pcap file.

Once you have generated the my_capture.pcap file, you can upload it to a CloudShark system using whatever is the most convenient. The API Upload is the most versatile, since you can integrate it with any custom tool you already have.

Or, you could put your CloudShark API token into your .bash_profile and upload the file with a little bash function. Put this into your ~/.bash_profile file and then open a new terminal:

CS_TOKEN=8d9e1eace0dbb71e8fdebc0d15810272
CLOUDSHARK_ROOT=http://your-cloudshark-hostname
CLOUDSHARK_UPLOAD=$CLOUDSHARK_ROOT/api/v1/$CS_TOKEN/upload

function cloudshark_upload () {
  JSONID=$(curl -s -F "file=@$1" -F "additional_tags=upload_by_$USER" $CLOUDSHARK_UPLOAD | python -mjson.tool | grep id)
  if [ "$JSONID" != "" ]; then
    ID=$(echo $JSONID | sed 's/:/ /1' | awk -F" " '{ print $2 }'| sed 's/\"//g')
    echo "$CLOUDSHARK_ROOT/captures/$ID"
  else
    echo "Could not upload capture to CloudShark"
  fi
}

Now every time you load a terminal, you’ll have the cloudshark_upload command available:

# sudo /usr/bin/vmnet-sniffer -e -w my_capture.pcap /dev/vmnet8 
# cloudshark_upload my_capture.pcap
Command returns: http://your-cloudshark-hostname/captures/44562ef7bb73

It would be even easier to just export a directory with NFS from the CloudShark system and then configure an Auto Importer for it. Let’s pretend you have /captures mounted on your system, which is configured as an Auto Import directory already:

sudo /usr/bin/vmnet-sniffer -e -w /captures/my_capture.pcap /dev/vmnet8

These are some fun ways to get your pcap file imported. You can always just drag it from your Desktop onto the CloudShark webpage, as well.

How to Use It

Since your traffic on the vmnet8 or vmnet1 interface is relayed via a private address of your host system, you’re going to see a lot of traffic to and from this address. For example, on our settings, we have 172.16.254.1 set as the NAT gateway for our virtual machines. (Remember that for a bridged VMware environment, you don’t need vmnet-sniffer. You just use tcpdump or your regular sniffer of choice, on the regular eth0/eth1/etc interface). You’ll also be able to see all of the traffic the virtual machines are sending to each other, and this is where the vmnet-sniffer tool comes in really handy. You have a total scope of this private network from the top down. It’s straight-forward to apply CloudShark filters to zoom in on a specific area of interest. Some examples:

  • You want to isolate HTTP traffic that is being sent through a host level load balancer to a random virtual machine
  • Or you want to determine where a specific DNS query is getting picked up by some fault-tolerant DNS VM
  • You want to develop a new low level network management packet protocol without causing danger to your corporate network (or your job)

There are plenty of scenarios we can invent where this environment is particularly useful. It’s an exercise to you, the VMware user, to come up with yet another brilliant network model best solved by virtual network isolation. If you have your own, email us! We’re geeks at heart and always love a great networking anecdote to share.