This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Consumer API

The outward facing API which consumers will use to access data from BitBroker

1: Catalog API
2: Entity API
3: Time Series API

The Consumer API is the main outward facing API for the BitBroker system. It is the API used by consumers to access data from the system, but always in the context of a data sharing policy.

All Consumer API calls happen in the context of a data sharing policy. The policy defines a data segment which you are permitted access to. There are no operations within the Consumer API which allow you to “break out” of this defined data segment.

Before you use this API, you should become familiar with the general, system-wide API conventions - which are used across all three BitBroker API sets. This covers topics such as API architecture, authorization, error reporting and handling, etc.

Paging Lists

It is possible that record lists returned from APIs may run to a great many records. There is a hard limit to how many records will be returned in a single call to any Consumer API.

The hard limit to how many records will be returned in a single call to the Consumer API is 250

It is possible to page access to record lists by using a set of URL query parameters, as follows:

Attribute	Necessity	Description
`limit`	optional	An integer number of data records, between 1 and 250
`offset`	optional	A positive integer index of the earlier desired record

If the paging parameters are incorrect, the API will respond with a standard validation error containing details of the violation.

If the specified offset is greater than the count of records available, the API will return an empty list. Using limit and offset you can page your way through a long list, getting the entire list a page at a time.

Timeseries data has separate paging semantics, which is more attuned to it’s time-value pair data points.

Rate and Quota Limits

All Consumer API calls happen within the context of a data sharing policy. Amongst other things, policy defines a rate and quota limit on calls you can make with the Consumer API. These should have been communicated to you by the coordinator user who gave you your consumer authorization token.

Rate - the maximum calls-per-second that you are allowed to make (e.g. 250)
Quota - the total calls you can make in a given period (e.g. 86,400 per day)

If you breach a rate or quota limit then calls to any part of the Consumer API will respond as follows:

HTTP/1.1 429 Too Many Requests

This response will persist until the breach has expired.

Legal Notices

All Consumer API calls happen within the context of a data sharing policy. Amongst other things, policy defines the legal context under which data access is permitted. A legal notice will be present at the end of every entity instance record returned by any part of the Consumer API.

These legal notices are in the form of a JSON array of objects, each with three attributes:

Attribute	Presence	Description
`type`	always	The type of the legal notice
`text`	always	The description of the legal notice
`link`	always	A link for more information about the legal notice

Examples of types of legal notices which may be present are:

Type	Description
`attribution`	How the data should be attributed within your application
`contact`	Who to contact about the data and it’s use
`license`	The licenses under which this data sharing operates
`note`	Ad hoc notes about the data and/or it’s use
`source`	Information about the source or origination of the data
`terms`	The terms and conditions of use for the data

Please respect the legal basis under which the data is being shared. Coordinator users have the power to rescind consumer authorization tokens and hence cut-off access to data for users who breach the legal context.

It is possible that the coordinator user who gave you your consumer authorization token will have more information about the legal basis of use of the data. They may require you to perform additional legal steps in order to be given a consumer authorization token.

Accessing “Staged” Records

It is important to understand that data connectors might be in a live or staged state. That is, their contribution might be approved for the live catalog, or might be being held back into a staging space only.

When staged, any records they contributed will not be visible in any part of the Consumer API. This concept is outlined in more detail in the key concepts documentation.

When developing a new connector, it is often useful to be able to see data from it alongside other public data. There is a mechanism available in the Consumer API which allows data connectors to see how their records will look alongside other existing public records.

In order to achieve this, connectors authors can send in a list of connector IDs in a private HTTP header attribute to any part of the Consumer API. Only connector authors will be aware of their connector ids. They can send up to 16 connectors in a single request header. The API will then include in responses, records contributed by those connectors as if they were live.

The header they should use for this is x-bbk-connectors. This should be a string value which is a comma separated list of connector IDs without any spaces.

1 - Catalog API

The catalog API enabling search and discovery services for entity instances

All the entity instances being managed by BitBroker are recorded with the catalog. This Catalog API is used to search this index and to discover entity instances which are needed by applications. The Catalog API returns a list of entity instances, which are then accessible via the Entity API.

In this section, we will explore the capabilities of the Catalog API in depth.

A quick way to get going with building your own applications is to adapt the example apps which use this and the other Consumer APIs.

All Consumer API calls happen within the context of a data sharing policy. The policy defines a data segment which you are allowed to access. Any queries which you make using the Catalog API can only operate within the data segment you are permitted access to.

In our sample calls, we use the standard server name and port format outlined for local installations and demos. If you have an alternative API host base URL, then enter it in the box below to update all the sample calls on this page :

Your API Host Base URL

The Consumer API can be run in production or development mode; each requiring a different header. If you are running a local installation of BitBroker, then you can switch the sample calls on this page to development mode by activating the toggle below. If you are unsure, leave this toggle off.

Apply development mode

All API calls in BitBroker require authorization. The sample calls below contain a placeholder string where you should insert your consumer API authorization token. This authorization token should have been provided to you by the coordinator user who administers the BitBroker instance. If you already have a token, enter it in the box below to update all the sample calls on this page:

Your Consumer API Authorization Token

When installing BitBroker locally, authorization is bypassed. In this scenario, you need to specify the policy you are using in a development header. Enter your desired policy in the box below to update all the sample calls on this page:

Your Current Policy

Querying the Catalog

You can query the catalog by issuing an HTTP/GET to the /catalog end-point.

curl http://bbk-consumer:8003/v1/catalog \
     --header "x-bbk-auth-token: your-header-goes-here"

This will return an empty JSON array.

The catalog API will return a list of entity instances which match the submitted query string. The query string is submitted by adding a q URL parameter to the call. Submitting no query string, results in an empty array (as opposed to all items).

Lists retrieved from this API will be returned in pages

For example, using the query string:

q={"name" : "India"}

Would return the following JSON array:

[
    {
        "id": "917d0311c687e5ffb28c91a9ea57cd3a306890d0",
        "url": "http://bbk-consumer:8003/v1/entity/country/917d0311c687e5ffb28c91a9ea57cd3a306890d0",
        "type": "country",
        "name": "India",
        "legal": []
    }
]

Here we have one matching entity instance. The API returns Entity API links to matching instances.

In the examples here we show the query strings in readable JSON format, but when actual calls are made, the query string is required to be url encoded.

Querying Options

Here, we will go through each available query option one-by-one, giving an example of each.

All query strings are url encoded JSON objects. The query string syntax used is loosely based on the query language used within Mongo DB.

Implicit Equality

This query format is a shorthand for using the $eq operator

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name": "India" }'

Equals / Not Equals

The $eq and $ne operators work for a range of data types such as integers, floats and strings.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.capital": { "$eq": "London" } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.capital": { "$ne": "Paris" } }'

Less Than (or Equal to)

The $lt and $lte operators work for a range of data types such as integers, floats and strings.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.population": { "$lt": 100000 } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.population": { "$lte": 100000 } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name": { "$lt": "China"} }'

Greater Than (or Equal to)

The $gt and $gte operators work for a range of data types such as integers, floats and strings.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.population": { "$gt": 100000 } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.population": { "$gte": 100000 } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name": { "$gt": "Mexico"} }'

In / Not In

The $in and $nin operators are for searching with array parameters.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name": {
         "$in": ["United Kingdom", "India", "France"]
     } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name": {
         "$nin": ["United Kingdom", "India", "France"]
     } }'

Contains

The $contains operator is for searching within array values.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.languages": { "$contains": [ "Japanese" ] } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.languages": { "$contains": [ "Japanese", "English" ] } }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "entity.location.coordinates": { "$contains": [ -3.435973, 55.378051 ] } }'

Logical Operators

The $and, $or and $nor operators can be combined in any combination.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "$and": [
         { "entity.continent": "Europe" },
         { "entity.population": { "$gt": 50000000 } }
     ] }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "$or": [
         { "entity.continent": "Europe" },
         { "entity.population": { "$gt": 50000000 } }
     ] }'

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "$nor": [
         { "name": "United Kingdom" },
         { "entity.population": { "$gt": 50000000 } }
     ] }'

Regular Expressions

The $regex operator allows for the use of regular expressions to query any string attribute.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "name":
         { "$regex": "United .*", "$options": "i" }
     }'

Geo Spatial Queries

The $near operator is used to find entity instances close to a GeoJSON geometry. The $min and $max parameters are specified in meters. You must specify either one of $min or $max or both together. The $geometry attribute must be present.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={
         "entity.location": {
             "$near": {
                 "$max": 750000,
                 "$min": 0,
                 "$geometry": {
                     "type": "Point",
                     "coordinates": [
                         -3.435973,
                         55.378051
                     ]
                 }
             }
         }
     }'

The $within operator is used to find entity instances inside a closed GeoJSON geometry. The $geometry attribute must be present.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={
        "entity.location": {
          "$within": {
            "$geometry": {
              "type": "Polygon",
              "coordinates": [
                [
                  [ -12.386341, 59.062341 ],
                  [ -12.386341, 49.952269 ],
                  [   2.500282, 49.952269 ],
                  [   2.500282, 59.062341 ],
                  [ -12.386341, 59.062341 ]
                ]
              ]
            }
          }
        }
      }'

Querying for Timeseries

You can also search for entity instances which have timeseries associated with them, by using catalog query calls. For example:

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "timeseries.population.value": "people" }'

This call will find all entity instances which have a timeseries called population with values of people.

curl http://bbk-consumer:8003/v1/catalog \
     --get \
     --header "x-bbk-auth-token: your-header-goes-here" \
     --data-urlencode 'q={ "timeseries.noise.unit": "decibel" }'

This call will find all entity instances which have a timeseries called noise with a unit of decibel.

2 - Entity API

The entity API enabling direct access to entity types and entity instances

The Entity API is used to retrieve information about entity instances which are present within the catalog. You can use this API to get a list of such entity instances or to get details of one particular entity instance. Calls to the Entity API are most often a result of a query to the Catalog API.

In this section, we will explore the capabilities of the Entity API in depth.

A quick way to get going with building your own applications is to adapt the example apps which use this and the other Consumer APIs.

All Consumer API calls happen within the context of a data sharing policy. The policy defines a data segment which you are allowed to access. Any queries which you make using the Entity API can only operate within the data segment you are permitted access to.

Apply development mode

Entity Types Lists

You can query for a list of known entity types by issuing an HTTP/GET to the /entity end-point.

curl http://bbk-consumer:8003/v1/entity \
     --header "x-bbk-auth-token: your-header-goes-here"

This will return a JSON array as follows:

[
    {
        "id": "country",
        "url": "http://bbk-consumer:8003/v1/entity/country",
        "name": "Countries",
        "description": "All the countries in the world as defined by the UN"
    },
    {
        "id": "heritage-site",
        "url": "http://bbk-consumer:8003/v1/entity/heritage-site",
        "name": "World Heritage Sites",
        "description": "A landmark or area with legal protection by an international convention administered by UNESCO"
    }
    // other entity types here
]

Each entity type, which you are permitted to see, will be returned within this array.

Lists retrieved from this API will be returned in pages

Entity Instance Lists

You can query for a list of entity instances of a given entity type by issuing an HTTP/GET to the /entity/:type end-point.

curl http://bbk-consumer:8003/v1/entity/country \
     --header "x-bbk-auth-token: your-header-goes-here"

This will return an empty JSON array as follows:

[
    {
        "id": "d38e86a5591b1f6e562040b9189556ff2d190ea7",
        "url": "http://bbk-consumer:8003/v1/entity/country/d38e86a5591b1f6e562040b9189556ff2d190ea7",
        "type": "country",
        "name": "Andorra",
        "legal": []
    },
    {
        "id": "34c3ab32774042098ddc0ffa9878e4a1a60b33c0",
        "url": "http://bbk-consumer:8003/v1/entity/country/34c3ab32774042098ddc0ffa9878e4a1a60b33c0",
        "type": "country",
        "name": "United Arab Emirates",
        "legal": []
    },
    {
        "id": "8c52885171d12b5cda6c77e2b9e9d52ed6bfe867",
        "url": "http://bbk-consumer:8003/v1/entity/country/8c52885171d12b5cda6c77e2b9e9d52ed6bfe867",
        "type": "country",
        "name": "Afghanistan",
        "legal": []
    },
    // other entity instances here
]

Each entity instance, which you are permitted to see, will be returned within this array.

Lists retrieved from this API will be returned in pages

Entity Instance Details

You can get the details of a particular entity instance by issuing an HTTP/GET to the /entity/:type/:id end-point.

curl http://bbk-consumer:8003/v1/entity/country/34c3ab32774042098ddc0ffa9878e4a1a60b33c0 \
     --header "x-bbk-auth-token: your-header-goes-here"

This will return a JSON object as follows:

{
    "id": "34c3ab32774042098ddc0ffa9878e4a1a60b33c0",
    "url": "http://bbk-consumer:8003/v1/entity/country/34c3ab32774042098ddc0ffa9878e4a1a60b33c0",
    "type": "country",
    "name": "United Arab Emirates",
    "entity": {
        "area": 83600,
        "code": "AE",
        "link": "https://en.wikipedia.org/wiki/United_Arab_Emirates",
        "capital": "Abu Dhabi",
        "currency": {
            "code": "AED",
            "name": "Arab Emirates Dirham"
        },
        "location": {
            "type": "Point",
            "coordinates": [
                53.847818,
                23.424076
            ]
        },
        "continent": "Asia",
        "population": 9682088,
        "calling_code": 971,
    },
    "instance": {
        "independence": 1971
    },
    "timeseries": {
        "population": {
            "unit": "x1000",
            "value": "people",
            "period": "P1Y",
            "url": "http://bbk-consumer:8003/v1/entity/country/34c3ab32774042098ddc0ffa9878e4a1a60b33c0/timeseries/population"
        }
    },
    "legal": []
}

You can only get details of entity instances which you are permitted to see.

3 - Time Series API

The time series API enabling access to time-value pair datasets within entity instances

The Timeseries API is used to retrieve timeseries data points from entity types present within the catalog. Not all entity types have timeseries associated with them.

In this section, we will explore the capabilities of the Timeseries API in depth.

A quick way to get going with building your own applications is to adapt the example apps which use this and the other Consumer APIs.

All Consumer API calls happen within the context of a data sharing policy. The policy defines a data segment which you are allowed to access. Any queries which you make using the Timeseries API can only operate within the data segment you are permitted access to.

In the Catalog API documentation, you can find examples of how to search for entity instances which have timeseries associated with them.

Apply development mode

Getting Timeseries Data

You can query for a list timeseries data by issuing an HTTP/GET to the /entity/:type/:id/timeseries/:tsid end-point.

Timeseries are always housed within a parent entity type and each has a unique ID on that entity type. Hence, you will need to know the entity type ID (type), the entity instance ID (id) and timeseries ID (tsid), in order to get access to such data points.

curl http://bbk-consumer:8003/v1/entity/country/34c3ab32774042098ddc0ffa9878e4a1a60b33c0/timeseries/population \
     --header "x-bbk-auth-token: your-header-goes-here"

This will return a JSON array as follows:

[
    {
        "from": 1960,
        "value": 89608
    },
    {
        "from": 1961,
        "value": 97727
    },
    {
        "from": 1962,
        "value": 108774
    },
    {
        "from": 1963,
        "value": 121574
    },
    // other timeseries data points here
]

In this response, the attribute you will see is as follows:

Attribute	Presence	Description
`from`	always	An ISO 8601 formatted date
`to`	sometimes	When present, an ISO 8601 formatted date
`value`	always	A valid JSON data type or object

The exact nature of the value attribute will depend upon the context of the timeseries you are using. It might be as simple as an integer, or as complex as an object.

The timeseries will always be returned sorted on the from attribute. Taking the first item of a returned array, should always represent the latest data point.

Paging Timeseries

It is possible that timeseries may run to a great many data points. There is a hard limit to how many data points will be returned in a single call to this API.

The hard limit to how many data points will be returned in a single call to this API is 500

It is possible to page access to timeseries data points by using a set of URL query parameters, as follows:

Attribute	Necessity	Description
`start`	optional	An ISO 8601 formatted date
`end`	optional	An ISO 8601 formatted date
`duration`	optional	An integer number of seconds, greater than 0
`limit`	optional	An integer number of data points, between 1 and 500

Here are additional considerations for when using these paging parameters:

If you specify and end, you must also specify a start
If you specify and end, it must be after the start
If you specify and duration, you must also specify a start
You cannot specify an end and a duration

If the paging parameters are incorrect, the API will respond with a standard validation error containing details of the violation.