Using REST sinks for traffic data retrieval

REST sinks were introduced in the general article as programming elements that allow your application to query the server for traffic data using its REST API. In this article, the concept is explored further.

As trajectories pass through operators, they change the operators’ internal values like statistical information about speeds and accelerations. These internal values are called attributes. Currently, all operators have the same kinds of attributes, but in the future, some new attributes will be relevant only for some types of operators. You'll find information about different attributes and their associated sink types in the next article.

The typical procedure for your application would be to request an array of all defined sinks and then periodically request data from sinks that are currently of interest to it. The following sections illustrate these steps using HTTP and JSON requests.

Getting a list of all sinks

Method: GET
URI:/cubes/{cube_id}/analytics/{analytic_id}/sinks
Query string parameters:

timestamp_format—can only be empty or have the "iso8601" string value. If it's not set, all timestamps in the response body will be in the milliseconds since epoch format. If the "iso8601" value is set, all timestamps will be in the ISO 8601 format, specifically in the format yyyy-MM-ddTHH:mm:ss.zzzZ, where
- yyyy is the year as a four-digit number
- MM is the month as a number with a leading zero (01 to 12)
- dd is the day of the month as a number with a leading zero (01 to 31)
- T is a literal separator for the time part of the DateTime string
- HH is the hour with a leading zero (00 to 23)
- mm is the minute with a leading zero (00 to 59)
- ss is the whole second, with a leading zero (00 to 59)
- zzz is the fractional part of the second, to millisecond precision, including trailing zeroes where applicable (000 to 999)
- Z is a literal character denoting the UTC timezone

Keep in mind that using the ISO 6801 format is computationally demanding for the FLOW block and may delay the response by a few milliseconds depending on the number of required timestamp conversions.

Example response body:

{
  "sequence_number": "36",
  "timestamp": "1619012692635",
  "sinks": [
    {
      "id": 0,
      "name": "No. of Speeding Vehicles",
      "output_value_type": "value",
      "history": {
        "enabled": true,
        "policy": "fixed_interval",
        "interval": "1000"
      },
      "history_capacity": 10000,
      "history_records_count": 69,
      "history_start_timestamp": "1619012690366"
    },
    {
      "id": 1,
      "name": "Speed Statistical Sink - North Gate",
      "output_value_type": "statistical_value",
      "history": {
        "enabled": false
      },
      "history_capacity": 10000,
      "history_records_count": 0,
      "history_start_timestamp": "0",
    },
    {
      "id": 2,
      "name": "custom_heatmap",
      "output_value_type": "heatmap",
      "history": {
        "enabled": false
      }
    }
  ]
}

Comments:

sequence_number refers to the current analytic. You can look at it as the analytic’s version—editing the analytic increments the number. Your application needs to send this number in all sink output requests. Obtaining the current sequence number is explained in the article about the basics of FLOW's REST API. Note that the sequence number will always be a string.
timestamp is the timestamp of the frame from which the data in the response have been evaluated. The frame (and its timestamp) is provided by the FLOW NODE. If e.g. the sink's analytic would be disabled, this timestamp won't be updated.
The cubes and analytics part of the URI is explained in the article about the basics of FLOW's REST API.
For details about individual sink types and their properties in this output, see their dedicated sections.
history describes whether history is being saved by the sink and if yes, what policy is being used. History policies are described in a later article. Possible values are off, on_value_change, on_time_change, and fixed_interval. If the policy is set to fixed_interval, then interval contains the interval length in milliseconds.
history_start_timestamp refers to the origin time of the oldest snapshot of this sink stored in the BLOCK. A zero or an empty string means that there are no saved snapshots yet. See the Getting the history of sink values section for more details.

Getting outputs of selected sinks

Method: POST
URI:/cubes/{cube_id}/analytics/{analytic_id}/sinks/data
Query string parameters:

timestamp_format—see Getting a list of all sinks for details

Example request body:

{
  "sequence_number": "36",
  "sinks": [
    {
      "id": 0
    },
    {
      "id": 1,
    }
  ]
}

Example response body:

{
  "sequence_number": "36",
  "settings_sequence_number": "0",
  "timestamp": "1619088804432",
  "sinks": [
    {
      "id": 0,
      "name": "Speed - Value",
      "output_value_type": "value",
      "data_end_timestamp": "1619088804432",
      "data_start_timestamp": "1619088624709",
      "data": {
        "data_validity": "ok",
        "evaluation_validity":"ok",
        "object_count": 32,
        "value": 2
      }
    },    
    {
      "id": 1,
      "name": "Speed - Statistical value",
      "output_value_type": "statistical_value",
      "data_end_timestamp": "1619088804432",
      "data_start_timestamp": "1619088624709",
      "data": {
        "average": 11.363175912117441,
        "data_validity": "ok",
        "evaluation_validity":"ok",
        "maximum": 20.562898635864258,
        "median": 11.363175912117441,
        "minimum": 8.132431983947754,
        "object_count": 32,
        "size": 1
      }
    }
  ] 
}

Comments:

The server compares the received sequence_number to the analytic's current sequence number. If it's equal, it sends a standard response. If not, it returns an Error object with the outdated_sequence_number reason. In that case, your application should request a new list of sinks and save the current sequence number. You must provide the sequence number as a string.
settings_sequence_number is an artifact and can be ignored.
For identifying sinks, only their ID is relevant. The user-defined name is intended mainly for easier response parsing. The ID is persistent as long as the sink exists. However, the ID of a deleted sink can be reused for a new sink. It is recommended to reestablish mapping between the name of the sink and its ID when the sequence number changes.
This example illustrates a situation where only outputs of sinks 0 and 1 are needed. You may ignore sinks where the output is not needed at the moment.
It might take a long time for the block to respond to a heavy data sink (e.g. the trajectory view sink) data request. Due to this, all heavy data sink data requests must be sent separately—without requests for any other sink data in the same message. See the Overview of sink types for a list of light data and heavy data sinks.
If the sinks array in the request is empty or missing, the block sends outputs of all light data sinks.
The sink's attributes data_start_timestamp and data_end_timestamp denote the time interval from which the sink data originates. More specifically, they denote the time interval in which the trajectories from which the sink output has been calculated have been closed. Trajectory closure is explained in our article Working with time in FLOW.
data_validity is an enumeration string referring to the rate at which available trajectories' time of existence intersects with the snapshot's time interval or time block. Possible values are:
- ok—the data fully intersects with the snapshot's time interval.
- cropped—the data partially intersects with the snapshot's time interval. The data_start_timestamp and/or data_end_timestamp have been modified to match the trajectories' actual time of existence.
- invalid—there is no intersection of the trajectories' time of existence with the snapshot's time interval.
evaluation_validity is an enumeration string describing whether there's a problem with an expression associated with a widget. This doesn't concern any sink types because no sink's output can be controlled by expressions—this property is present because of widgets, as sink and widget output has the same structure.
- ok—there is no problem with expression evaluation. This should always be the case with sinks.
- invalid—there's a syntactic or semantic problem (e.g. circular dependency on other expression widget) with the expression definition.

Getting the history of sink values

For certain types of sinks, the FLOW block saves snapshots of their output. For more details, refer to the article on sink history. Each sink type has a different limit of snapshots. If the limit is exceeded, the oldest snapshot is replaced with the most recent one. You can find out the timestamp of the oldest snapshot from the history_start_timestamp property while getting the list of all sinks, or from the request below.

See Overview of sink types for a list of sinks types that support snapshots.

Method: POST
URI:/cubes/{cube_id}/analytics/{analytic_id}/sinks/history
Query string parameters:

timestamp_format—see "Getting a list of all sinks" for details. The request body is also expected to contain the timestamps in the format set by this parameter.

Example request body:

{  
    "sequence_number": "36",  
    "sinks": [    
        {      
            "id": 0,
            "start_timestamp": "1619106105644",
            "end_timestamp": "1619106108644"        
        }
    ]
}

Example response body:

{
  "sequence_number": "36",
  "settings_sequence_number": "0",
  "timestamp": "1619106834779",
  "sinks": [
    {
      "id": 0,
      "name": "Speed - Value",
      "output_type": "sink",
      "output_value_type": "value",
      "history": {
        "enabled": true,
        "interval": "1000",
        "policy": "fixed_interval"
      },
      "history_capacity": 10000,
      "history_records_count": 656,
      "history_start_timestamp": "1619106105644",
      "snapshots": [
        {
          "data": {
            "data_validity": "ok",
            "evaluation_validity":"ok",
            "object_count": 3,
            "value": 3
          },
          "data_end_timestamp": "1619106633167",
          "data_start_timestamp": "1619106103875",
          "timestamp": "1619106633167"
        },
        {
          "data": {
            "data_validity": "ok",
            "evaluation_validity":"ok",
            "object_count": 3,
            "value": 3
          },
          "data_end_timestamp": "1619106634168",
          "data_start_timestamp": "1619106103875",
          "timestamp": "1619106634168"
        },
        {
          "data": {
            "data_validity": "ok",
            "evaluation_validity":"ok",
            "object_count": 4,
            "value": 4
          },
          "data_end_timestamp": "1619106635169",
          "data_start_timestamp": "1619106103875",
          "timestamp": "1619106635169"
        }
      ]
    }
  ]
}

Comments:

You can query for multiple sink histories in the same request. The sinks property is an array.
The request's start_timestamp and end_timestamp denote the time interval into which the response snapshots' timestamps must belong. This refers to the timestamp property inside the elements of the snapshots array.
The timestamp property inside the elements of the snapshots array mostly contains the same value as the element's data_end_timestamp. If, however, the sink's Time mode is set to Time Blocks, then the timestamp property contains the same value as the element's data_start_timestamp.
data_start_timestamp, data_end_timestamp attributes are explained in the section Getting outputs of selected sinks.
settings_sequence_number is an artifact and can be ignored.

What next?

See each sink type in detail, or read about UDP sinks for a subscription-based variant of sinks.

Need more help or have some questions? Contact us!