Documentation‎ > ‎

NetLogger Calipers API

Overview

Although system-level statistics can provide an overview of the load on disks and network interface cards (NICs), they do not provide as much insight into the pauses related to buffering by one transfer among many being handled by the server. These pauses expose the relative performance of disk and NIC I/O interfaces and therefore can provide valuable information about bottlenecks in the host filesystem. To address this gap, we created the NetLogger Calipers API.

NetLogger Calipers is based on our earlier work on embedded log summarization. It is a library written in C, though the concept can of course be extended to any language. Each caliper records statistics for a single stream of activities, where the “arms” of the calipers are the timestamps produced by calls to the NetLogger begin() and end(value) functions. Summary statistics are computed continuously for any number of concurrent begin/end pairs. In the case of data transfers, the activities of interest are read and write system calls, and the value of interest is the number of bytes read or written. The NetLogger Calipers API reports the summarized results at arbitrary time periods, in either the native NetLogger format or as perfSONAR data blocks (e.g. using BSON).

Summary statistics

NetLogger Calipers tracks summary statistics for both the duration between the start and end timestamps, the numeric value provided by the user, and the ratio between these. The available statistics are:
  • count: Number of begin/end pairs
  • sum: Sum of values, sum of duration, sum of (value / duration)
  • mean: Mean of values, mean of duration, mean of (value / duration)
  • min: Minimum value, minimum duration, minimum (value / duration)
  • max: Maximum value, maximum duration, maximum (value / duration)
  • standard deviation: Stdev of value, duration and (value/duration)

API Documentation

The NetLogger Calipers C API is documented in the NetLogger C API section.

Example
#include <netlogger_calipers.h>

/* Create caliper */
caliper = netlogger_calipers_new();
/* Use caliper in loop */
for(;;) {
    netlogger_calipers_begin(caliper);
    value = do_something();
    netlogger_calipers_end(caliper, value);
     /* Note: caller decides when to report */
    if (report_now(caliper)) {
        /* Calculate summary values */
        netlogger_calipers_calc(caliper);
        report(caliper);
        /* Skip _clear() for cumulative summaries */
        netlogger_calipers_clear(caliper);
    }
}

Comments