Search

Searching CloudShark via the API is one of the most powerful features. It is also one of the more complex methods in the API. With API search, you can build tools and automation around your CloudShark repository.

Quick Example

The API uses the GET HTTP method. If you’re using curl on the command line, you may need to specify that method using the -X option.

curl -XGET "https://www.cloudshark.org/api/v1/<token>/search?search[tags][]=Boston"

Introduction to Nested Query Strings

Nested Query Strings are a way of translating complex data structures into a string that can be sent along with an HTTP requests. They make use of a lot of square-brackets and can represent key-value pairs, hashes, and arrays all inside of a single flat string.

Examples

A key-value pair is very simple. foo=22 is translated directly. The variable ‘foo’ is assigned to the value 22. Of course, when using these in an HTTP request, items should be encoded to avoid special characters and spaces.

Arrays

An array is translated into a series of parameters. Consider the following:

myArray = [1,2,3]

This is translated into 3 parameters that have the same name and have square brackets attached:

myArray[]=1&myArray[]=2&myArray[]=3

When the server receives this, it builds up an Array holding all 3 values.

Hashes

Hashes can store a number of key-value pairs within one variable. In Ruby, these look like this:

myHash = { 'one' => 1, 'two' => 2, 'three' => 3 }

Translated into our Nested Query String it looks like this:

myHash[one]=1&myHash[two]=2&myHash[three]=3

Complex Structures

These structures can get complex when you start building them up and combining them. Consider a Hash that has elements that are either simple values, or arrays! Once again in Ruby:

myHash = {
  'myArray' => [1,2,3],
  'myValue' => 4
}

This is translated into the following:

myHash[myArray][]=1&myHash[myArray][]=2&myHash[myArray][]=3&myHash[myValue]=4

These URLs can become really difficult to read when the square brackets are encoded as %5B%5D. It may be easier to write them out with [ and ], and let whatever HTTP client you are using do the conversion for you.

Existing Libraries

Luckily, you don’t need to do all this translation yourself. Programming languages can help with the translation, and there are probably packages out there for your favorite.

Ruby

Ruby is our favorite language, and CloudShark is written mostly in Ruby. Internally, it uses the Rack::Utils module to parse the parameter string sent to it. There are two methods for translating these strings to and from objects.

require 'rack'

# convert to nested query from a Hash of key-value pairs
post_data = Rack::Utils.build_nested_query(myHash)

# parse a query back to a Hash
params = Rack::Utils.parse_nested_query(myString)

Search Call Structure

There are two parameters that need to be provided in order for searches to take place on the appliance. One is the search parameter, and the other is called matchers.

The search object is a key-value Hash that maps individual fields to what value they should be matched with. The format of each one can be specific, so be sure to match them up with the documentation below. They can be a single value, or in some cases, an Array of values.

Matchers

For single-value fields like num_packets or duration etc. you need to also specify how to match the value. Are we looking for less than that number, more? Exactly that amount? By also specifying a collection of matchers for each field you’re searching for, you can tell CloudShark exactly how to search.

Matchers must be one of the following three strings:

  • .eq for exact matches
  • .lt for matching values less-than whats given
  • .gt for matching values greater-that the value

Example Search Structure

To give a quick example of a search, let’s consider asking for a list back of all the capture files on the appliance that have more than 1000 packets and are tagged with ‘noc-A’. Here’s the hash that would need to be constructed:

query = {
  :search => {
     :num_packets => 1000,
     :tags => ['noc-A'],
     
     :matchers => {
        :num_packets => ".gt"
     }
  }
}

params = Rack::Utils.build_nested_query(query)
#=> "search[num_packets]=1000&search[tags][]=noc-A&search[matchers][num_packets]=.gt"

Paging and Sorting

You can store hundreds of thousands, maybe millions, of capture files within CloudShark. There has to be a way to return only the ones you’re looking for, but not all at once, and in the order that makes sense to you.

CloudShark’s main index is composed of individual pages. Search works the same way. You can specify a limit of the number of results per-page to return. You can pick which page you would like returned for the current query, and you can affect the sort_order of the captures returned.

Example Pages

In addition to the search parameter, you can also send limit and page. All searches are limited to a maximum page-size of 90 results, but you can make that number smaller if you need. The default is 50 results per-page.

To return 25 results at a time, and look at the 2nd page of all the captures tagged with “Boston”, you can use the following query:

search[tags][]=Boston&limit=25&page=2

note that the parameters limit and page live on their own, and aren’t part of the search[] parameter

Sorting

Capture results can be sorted by any of the fields that can be searched for. A direction for the sort should also be specified. The sort_order field is part of the search[] structure. Allowed sort fields are:

  • created_at (upload time)
  • user
  • filename
  • size
  • num_packets
  • duration
  • data_byte_rate
  • data_bit_rate
  • avg_packet_size
  • avg_packet_rate
  • start_time (capture start)
  • end_time (capture end)
  • type
  • encapsulation

Sort direction is decided by either .asc ascending low-to-high, or .desc descending high-to-low.

For example to search for captures tagged with “Boston” and order them by the number of packets, highest-to-lowest:

search[tags][]=Boston&search[sort_order]=num_packets.desc

Searchable Fields

Filename

  • Parameter: filename

This will perform a partial name match across the whole capture repository. If the specified value for filename is found in any captures, they will be included in the result set.

Example

search[filename]=sipcall

User

  • Parameter: user
  • Type: Array

The user field is an Array that will return the capture files owned by each user specified. Even if only specifying a single user, you must still use the Array notation as shown in the example.

Example

search[user][]=admin&search[user][]=guest

Group

  • Parameter: group
  • Type: Array

The group field is an Array that will return the capture files owned by each group specified. Even if only specifying a single group, you must still use the Array notation as shown in the example.

If you want to search for capture files that do not belong to any groups, simple add a term that equals nothing: search[group][]=

Example

search[group][]=Admin&search[group][]=Group-B

Public Captures

  • Parameter: ispublic

Searches for capture files that are visible without needing to log-in.

Example

search[ispublic]=true

Group-Writable

  • Parameter: groupwrite

Useful in combination with the Group search, this allows you to find files that have given their group ownership write access to the file.

Example

search[groupwrite]=true

Comments and Annotations

  • Parameter: annotations
  • Parameter: hascomments

Setting either of these will return those captures that have either packet annotations, or file-wide comments saved on them.

To search the contents of individual comments, the comments parameter can be used directly:

  • Parameter: comments

Example

search[annotations]=true&search[hascomments]=true&search[comments]=bug+129

Tags

  • Parameter: tags
  • Type: Array
  • Matchers: ‘tags’

Tags are a very special search parameter. If no matcher is specified only the captures that have ALL the specified tags.

The tags matcher can take the values AND, or OR. If the AND matcher is specified then only captures with all of the specified tags will be returned. If the OR matcher is specified then captures with any of the specified tags will be returned.

Example

search[tags][]=orange&search[tags][]=blue&search[matchers][tags]=and

Upload Date and Capture Date

  • Parameters: date and cap_date
  • Type: Date or Ranges

Searching by Date is very powerful. CloudShark can search either by the date the file was uploaded, or the date that the actual network capture was taken.

Date ranges need to be in the format mm/dd/yyyy-mm/dd/yyyy to specify a Range of dates, or just a single mm/dd/yyyy date for a single date. The example below searches for a capture uploaded on October 11th, that had been captured either on October 10th or 11th.

Example

search[date]=10/11/2013&search[cap_date]=10/10/2013-10/11/2013

Upload Time and Capture Time

  • Parameters: up_time_start, up_time_end and cap_time_start, cap_time_end
  • Type: Times

Similar to how searching by Date works for both upload date and capture date, searching by time lets you narrow down when a file was uploaded during the day, or by when the capture file was taken. You must specify both a start and an end.

Times must be in the format h:mm AM/PM

Example

search[cap_time_start]=5:30+AM&search[cap_time_end]=6:30+AM

Encapsulation

  • Parameter: encapsulation
  • Type: Array

This will find captures that have any of the specified encapsulations. If you are only searching for one, you still need to use the correct notation.

Example

search[encapsulation][]=Ethernet

Type

  • Parameter: file_type
  • Type: Array

Searches for captures that match any item on the list of file types. This can be useful for finding files of a certain kind. The type must match exactly for the file to be returned.

Example

search[file_type][]=Wireshark+-+pcapng

File/Data Size

  • Parameters: size or data_size
  • Matchers: .eq, .lt, or .gt
  • Type: Number of bytes

Searches either the File’s size or the PCAP’s reported data_size field. You must also supply a matchers entry for the same field you are searching for. The size parameter is specified in bytes.

Example

search[size]=2000&search[matchers][size]=.gt

Duration

  • Parameter: duration
  • Matchers: .eq, .lt, or .gt
  • Type: Length of time in seconds

Searches for files that match then given duration in seconds, according to the matcher specified.

Example

search[duration]=300&search[matchers][duration]=.lt

Number of Packets

  • Parameter: num_packets
  • Matchers: .eq, .lt, or .gt
  • Type: Integer number of packets

Search for files with the given number of packets according to the specified matcher.

Example

search[num_packets]=1000&search[matchers][num_packets]=.eq

Packet/Byte/Bit Rate

  • Parameters: avg_packet_rate, data_bit_rate, data_byte_rate
  • Matchers: .eq, .lt, or .gt
  • Type: Integer number of packets/bits/bytes per second

Example

search[data_bit_rate]=1000&search[matchers][data_bit_rate]=.gt

Average Packet Size

  • Parameters: avg_packet_size
  • Matchers: .eq, .lt, or .gt
  • Type: Numeric average packet size in bytes

Searches the reported average packet size of individual captures.

Example

search[avg_packet_size]=512&search[matchers][avg_packet_size]=.lt

SHA-1 Hash

  • Parameters: sha1_hash

Searches for capture files with the specified SHA-1 hash

Example

 search[sha1_hash]=d75ab989ee0f76e868a41bd30fa61d14336610c4