Getting Started

• 1: Demos and Examples
• 2: Installing Using Kubernetes
• 3: Installing Locally
• 4: Configuring BitBroker

1 - Demos and Examples

Here you will find information about a range of BitBroker demo applications and connectors to help you understand what BitBroker is and how it can operate in complex data sharing scenarios.

Most importantly, it will help you get started building your own applications which use BitBroker data or your own data connectors to contribute data into the system.

Demo Applications

We have a number of example applications, which allow users to explore policy based access to data via the Consumer API.

Data Explorer

This application allows you to explore the entire Consumer API by directly trying out a number of interactive scenarios. It has a set of example data and polices already pre-installed and running.

You can explore using the Catalog API to try different, complex catalog queries. You can see how the results of these queries differ in the light of different policies - which you can switch between simply in the application.

Once you have executed a query and obtain entity instance records, you can use the Entity API to browse the whole list and inspect the details of individual entities.

Finally, for country data, you can also see the Timeseries API in actions and integrated with a charting library.

Mapping Sample

This application allows you to explore the Consumer API via the medium of a mapping application. It has a set of example data and polices already pre-installed and running. The geographical attributes within the example data are used to populate a map view of the data records.

You can explore how the application outputs are changed in the light of different policies - which you can switch between simply in the application.

Demo Data Connectors

Here we provide a range of data connectors to help you understand what they are and how to build your own. Indeed, it is hoped that you can simply modify one of these data connectors to achieve your own data submission aims.

We currently have two types of example connector:

• File based - dataset loaded directly from a file
• RDBMS - data drawn from a relational database

All implementations upload data to the BitBroker catalog. They also fetch and return third-party data in their entity webhooks for the example country dataset, and both support time series data for the country dataset.

NodeJS » File Based

This example connector is implemented in NodeJS and uses a simple file as its source data store. Key characteristics of this connector are:

• Initial dataset loaded from a file over HTTP
• Supports for a range of file types: JSON, Microsoft Excel, CSV, etc
• Implements catalog updating via sessions
• Implements all webhook end-points

NodeJS » RDBMS

This example connector is implemented in NodeJS and uses a PostgreSQL database as its source data store. Key characteristics of this connector are:

• Initial dataset loaded via SQL calls on a relational database
• Implements catalog updating via sessions
• Implements all webhook end-points

2 - Installing Using Kubernetes

There are several ways in which you can install BitBroker, depending on what you are trying to achieve.

Are you trying to deploy to a public cloud or to a local environment? Are you wanting to use BitBroker in a production environment, or as a local sandbox, perhaps you are developing data connectors or even trying to enhance BitBroker itself?

In this section, we will cover in detail and step-by-step, all the different ways in which you can install a BitBroker instance using Kubernetes, to help you achieve your goals.

Bootstrapping Fresh Installations

For all Kubernetes installations, you should be aware of the ways in which fresh installations perform certain bootstrap operations.

Bootstrap User

Every fresh installation of BitBroker comes with one preinstalled user (uid: 1). This user is automatically created when the system is bought-up for the first time.

As we will explain below, this user is linked to the authorization of API calls. The user is also important for the up-coming web portal BitBroker control interface.

Bootstrap Coordinator Token

All interactions with BitBroker stem, ultimately, from interactions with the Coordinator API. This is the main administrative API for the whole system. In order to use this API, you need an access token.

Using this API, new users can be created and then promoted to have coordinator status. This results in the production of a new coordinator access token for them. But this act of promotion itself, requires permission. So how can we get started with this circular scenario?

Whenever a fresh system is installed using Kubernetes, a special bootstrap coordinator authorization token is produced. This token is valid for use with the Coordinator API. You can use this token to get going with the process of then creating your own users and giving them coordinator status.

The detailed install steps below, will contain more information about how to extract and use this bootstrap token.

Cloud Kubernetes Installation

In the section, we will explore how to you can use our pre-prepared Helm charts to install a complete BitBroker instance into your cloud of choice. These charts will be downloaded directly from Docker Hub.

Prerequisites

Start with a clean machine, with no remnants of previous BitBroker installations. Please ensure you have the following software installed, configured and operational:

• Kubernetes command line tools
• Helm command line tools
• cURL command line tool (used for testing installs only)

Create a brand new directory to act as a workspace for your installation:

mkdir bbk
cd bbk

Make sure that your current Kubernetes context is pointing at your cloud of choice:

kubectl config get-contexts

Installation

First, let’s prepare the context we want to install:

helm repo add bit-broker https://bit-broker.github.io/charts
JWKS=$(docker run bbkr/auth-service:latest npm run --silent create-jwks)
kubectl apply -f https://app.getambassador.io/yaml/emissary/2.2.2/emissary-crds.yaml

Next, let’s extract and save the default chart values:

helm show values bit-broker/bit-broker > values.yaml

Next update values.yaml with your required values:

Now we are ready to run the cloud installation with the selected values:

helm install --values values.yaml \
             --create-namespace bit-broker \
             --set bbk-auth-service.JWKS=$JWKS \
             --namespace bit-broker \
               bit-broker/bit-broker

This step takes a few moments to complete. After it has finished, you will see a series of notes which discuss some key points about your installation. The sections on JWKS, Auth Service and Rate Service are for advanced use cases, and you can ignore these for now.

It can take a few moments for the system to come into existence and for it to complete its initialization steps. You can test that the system is up-and-ready, by the using this command:

if [ $(curl --max-time 5 --write-out '%{http_code}' --silent --head --output /dev/null https://your-cloud-host/coordinator/v1) == "401" ]; then echo "Ready"; else echo "Not Ready"; fi;

This will output Not Ready until all the servers are up, after which it will output Ready. Keep trying this command until it signals it’s OK to proceed.

Installing a DNS Alias

If you want to add an alias to this installation into your DNS record, then you need to first get the service load balancer address:

kubectl get svc --no-headers -o custom-columns=":status.loadBalancer.ingress[0].hostname" --selector=app.kubernetes.io/name=bbk-emissary-ingress -n bit-broker | head -1

Then you can use this to add an ALIAS into your DNS record. Depending on the domain and the registrar, the procedure and naming terminology will be different. Here is an example procedure for AWS’s Route 53.

Bootstrap Coordinator Token

A key thing to note in the results output, is the section which says: “Here is how to get the Coordinator token”. Extracting and recording this token is a vital step to proceed with the install. This is the bootstrap coordinator token, which we outlined earlier in this document.

As the results output states, you can get hold of this bootstrap coordinator token as follows:

kubectl exec $(kubectl get pods --no-headers -o custom-columns=":metadata.name" --selector=app=bit-broker-bbk-auth-service -n bit-broker | head -1) -n bit-broker -c auth-service -- npm run sign-token coordinator $(kubectl get secret --namespace bit-broker bit-broker-bbk-admin-jti -o jsonpath="{.data.ADMIN_JTI}" | base64 --decode)

This is a long command and it will take a few seconds to complete. It will output the token, which will be in the format of a long string. Copy this token and store in a secure location. Be careful if sharing this token, as it has full rights to the entire Coordinator API.

Testing Your Installation

If everything worked as expected, the BitBroker API servers will be up-and-running in your cloud waiting for calls. You can test this by using the sample call below:

curl https://your-cloud-host/coordinator/v1 \
     --header "x-bbk-auth-token: bootstrap-token-goes-here"

The base end-points of all the three API servers respond with a small announcement:

{
    "now": "2022-06-16T10:44:53.970Z",
    "name": "bit-broker coordinator service",
    "base": "https://your-cloud-host/coordinator/v1",
    "status": "production"
}

Like all BitBroker API end-points, these require a working authorization to be in place. Hence, this announcement can be used for testing or verification purposes.

Uninstallation

If you want to completely uninstall this instance of BitBroker, you should follow these steps:

helm uninstall bit-broker -n bit-broker
kubectl delete -f https://app.getambassador.io/yaml/emissary/2.2.2/emissary-crds.yaml
kubectl delete namespace bit-broker
helm repo remove bit-broker
docker ps -a | grep "bbkr/" | awk '{print $1}' | xargs docker rm
rm values_local.*

This will remove all the key elements which were created as part of the installation. If you want to go further and clear out all the images which were downloaded too, use the following:

docker images -a | grep "bbkr/" | awk '{print $3}' | uniq | xargs docker rmi

Local Kubernetes Installation

In the section, we will explore how to you can use our pre-prepared Helm charts to install a complete BitBroker instance on your local machine. These charts will be downloaded directly from Docker Hub.

Prerequisites

Start with a clean machine, with no remnants of previous BitBroker installations. Please ensure you have the following software installed, configured and operational:

• Kubernetes command line tools
• Helm command line tools
• Docker Desktop
• cURL command line tool (used for testing installs only)

Create a brand new directory to act as a workspace for your installation:

mkdir bbk
cd bbk

Make sure that your current Kubernetes context is docker-desktop:

kubectl config get-contexts
kubectl config use-context docker-desktop

Installation

First, let’s prepare the context we want to install:

helm repo add bit-broker https://bit-broker.github.io/charts
JWKS=$(docker run bbkr/auth-service:latest npm run --silent create-jwks)
kubectl apply -f https://app.getambassador.io/yaml/emissary/2.2.2/emissary-crds.yaml

Our pre-prepared Helm charts are, by default, configured for production, cloud environments. However, here we want to install to localhost only. So we need to make some modifications to the default chart to cater for this:

helm show values bit-broker/bit-broker > values_local.yaml
sed -e 's/gateway[ ]*:/gateway: {}/g' \
    -e 's/acmeProvider/# acmeProvider/g' \
    -e 's/certificateIssuer/# certificateIssuer/g' \
    -e 's/#[ ]*host:[ ]*"https:\/\/bit-broker.io"/host: "http:\/\/localhost"/g' \
    -i '_old' \
    values_local.yaml

Now we are ready to run the local installation:

helm install --values values_local.yaml \
             --create-namespace bit-broker \
             --set bbk-auth-service.JWKS=$JWKS \
             --namespace bit-broker \
               bit-broker/bit-broker

This step takes a few moments to complete. After it has finished, you will see a series of notes which discuss some key points about your installation. The sections on JWKS, Auth Service and Rate Service are for advanced use cases, and you can ignore these for now.

It can take a few moments for the system to come into existence and for it to complete its initialization steps. You can test that the system is up-and-ready, by the using this command:

if [ $(curl --max-time 5 --write-out '%{http_code}' --silent --head --output /dev/null http://localhost/coordinator/v1) == "401" ]; then echo "Ready"; else echo "Not Ready"; fi;

This will output Not Ready until all the servers are up, after which it will output Ready. Keep trying this command until it signals it’s OK to proceed.

Bootstrap Coordinator Token

A key thing to note in the results output, is the section which says: “Here is how to get the Coordinator token”. Extracting and recording this token is a vital step to proceed with the install. This is the bootstrap coordinator token, which we outlined earlier in this document.

As the results output states, you can get hold of this bootstrap coordinator token as follows:

kubectl exec $(kubectl get pods --no-headers -o custom-columns=":metadata.name" --selector=app=bit-broker-bbk-auth-service -n bit-broker | head -1) -n bit-broker -c auth-service -- npm run sign-token coordinator $(kubectl get secret --namespace bit-broker bit-broker-bbk-admin-jti -o jsonpath="{.data.ADMIN_JTI}" | base64 --decode)

This is a long command and it will take a few seconds to complete. It will output the token, which will be in the format of a long string. Copy this token and store in a secure location. Be careful if sharing this token, as it has full rights to the entire Coordinator API.

Testing Your Installation

If everything worked as expected, the BitBroker API servers will be up-and-running on localhost waiting for calls. You can test this by using the sample call below:

curl http://localhost/coordinator/v1 \
     --header "x-bbk-auth-token: bootstrap-token-goes-here"

The base end-points of all the three API servers respond with a small announcement:

{
    "now": "2022-06-16T10:44:53.970Z",
    "name": "bit-broker coordinator service",
    "base": "http://localhost/coordinator/v1",
    "status": "production"
}

Like all BitBroker API end-points, these require a working authorization to be in place. Hence, this announcement can be used for testing or verification purposes.

Uninstallation

If you want to completely uninstall this instance of BitBroker, you should follow these steps:

helm uninstall bit-broker -n bit-broker
kubectl delete -f https://app.getambassador.io/yaml/emissary/2.2.2/emissary-crds.yaml
kubectl delete namespace bit-broker
helm repo remove bit-broker
docker ps -a | grep "bbkr/" | awk '{print $1}' | xargs docker rm
rm values_local.*

This will remove all the key elements which were created as part of the installation. If you want to go further and clear out all the images which were downloaded too, use the following:

docker images -a | grep "bbkr/" | awk '{print $3}' | uniq | xargs docker rmi

3 - Installing Locally

There are several ways in which you can install BitBroker, depending on what you are trying to achieve.

Local installations using Docker Compose or even direct on to your physical machine, can be a useful option for some use cases. For example, where you are developing data connectors or even trying to enhance BitBroker itself.

In this section, we will cover in detail and step-by-step, all the different ways in which you can install a BitBroker instance locally, to help you achieve your goals.

Server Naming and Ports

For consistency across the system, in local and development mode we use a set of standard logical server name and port for addressing the three principle API services.

This helps with readability and removes ambiguity, since some APIs share resource names. Also, it reduces confusion, if you start multiple API servers on the same physical machine.

Logical Server Names

We use a convention of assigning each API service a logical server name as follows:

You will find these names used across all the documentation and in the sample code. This is merely a convention; you do not need to use these names in your code. You can, instead, use your cloud URLs or even base IP addresses.

If you choose to stick to this convention, you will need to map these name to their ultimate end-points inside your system hosts file. Here is an example, mapping the standard logical server names to localhost.

127.0.0.1 bbk-coordinator
127.0.0.1 bbk-contributor
127.0.0.1 bbk-consumer

Server Ports

Each API service listens on a distinct, non-clashing port. Unless you configure it otherwise, even the docker images are designed to start each service on its designated port.

This port mapping makes it simple and unambiguous to bring up multiple (or indeed all) of these API services on the same physical machine, without them interfering with each other. The assigned service ports are as follows:

Development Only Headers

When installing BitBroker locally, authorization is bypassed. In this scenario, you do not need to supply authorization tokens to any API.

However, when using the Consumer API, you do still need to specify the policy you are using. This is so that BitBroker is aware of the data segment which is in-play for consumer calls. This is achieved by specifying a development header value.

Rather than the authorization header:

x-bbk-auth-token: your-token-will-go-here

You instead use the development header, as follows:

x-bbk-audience: your-policy-id-will-go-here

Failure to specify the development header on Consumer API calls when in local mode, will lead to those requests being rejected.

Bootstrap User

Every fresh installation of BitBroker comes with one preinstalled user (uid: 1). This user is automatically created when the system is bought-up for the first time.

As we will explain below, this user is linked to the authorization of API calls. The user is also important for the up-coming web portal BitBroker control interface.

Docker Compose Local Installation

In the section we will explore how to you can use our pre-prepared Docker Compose files to install a BitBroker instance on your local machine.

Prerequisites

Start with a clean machine, with no remnants of previous BitBroker installations. Please ensure you have the following software installed, configured and operational:

• Docker command line tools
• cURL command line tool (used for testing installs only)

Create a brand new directory to act as a workspace for your installation:

mkdir bbk
cd bbk

Installation

Let’s start by cloning the main BitBroker engine from its GitHub repository:

git clone https://github.com/bit-broker/bit-broker.git

This will have created a bit-broker directory and you should move into it:

cd bit-broker

First, let’s start by preparing our instance’s environment file. For a standard local install, we can simply copy the existing one which came with the repository:

cp .env.example .env

Now we can use our docker compose scripts to install and launch our local instance:

docker-compose -f ./development/docker-compose/docker-compose.yml up

At this point, your BitBroker installation is up and running on your local machine. You can test it by running the steps below.

If you are going to develop on BitBroker, either for building data connectors or for contributing to the core system, then this mode of install can provide a fast and low friction installation technique.

Testing Your Installation

If everything worked as expected, the BitBroker API servers will be up-and-running on localhost waiting for calls. You can test this by using the sample call below. In this local mode, you don’t need to specify any authorization tokens.

curl http://bbk-coordinator:8001/v1

If you have not applied the standard server name and port format, then you should use http://localhost:8001 here as your API host base URL. The base end-points of all the three API servers respond with a small announcement:

{
    "now": "2022-06-16T10:44:53.970Z",
    "name": "bit-broker coordinator service",
    "base": "http://localhost/coordinator/v1",
    "status": "production"
}

Uninstallation

If you want to completely uninstall this instance of BitBroker, you can follow these steps, from the top-level bit-broker folder:

docker-compose -f ./development/docker-compose/docker-compose.yml down
cd ..
rm -rm bit-broker

This will delete the Git cloned folder created earlier.

Direct Installation on a Physical Machine

In the section, we will explore how to you can install a BitBroker instance directly onto your local machine.

Prerequisites

Start with a clean machine, with no remnants of previous BitBroker installations. Please ensure you have the following software installed, configured and operational:

• NodeJS
• PostgreSQL
• Redis
• cURL command line tool (used for testing installs only)

Create a brand new directory to act as a workspace for your installation:

mkdir bbk
cd bbk

Installation

Let’s start by cloning the main BitBroker engine from its GitHub repository:

git clone https://github.com/bit-broker/bit-broker.git

This will have created a bit-broker directory and you should move into it:

cd bit-broker

For development purposes, BitBroker has a handy shell script called bbk.sh which can be used to manage the system in local install mode. You can use this to prepare your new clone:

./development/scripts/bbk.sh unpack

The unpack step, makes sure that all the dependent node packages needed to operate the system are downloaded and ready. It also creates a .env file automatically, by using the settings in the .env.example file.

Now you can start BitBroker by simply using:

./development/scripts/bbk.sh reset

The reset command given the following output:

  » no services running
  » wiping the database...
  » starting services...

  ┌────── services running ──────┐
  │                              │
  │   90808 » bbk-consumer       │
  │   90812 » bbk-rate-limit     │
  │   90813 » bbk-auth-service   │
  │   90816 » bbk-contributor    │
  │   90817 » bbk-coordinator    │
  │                              │
  └──────────────────────────────┘

At this point, your BitBroker installation is up and running on your local machine. You can test it by running the steps below. But first, let’s just see what other commands the bbk.sh script supports:

./development/scripts/bbk.sh <command>

command:

  unpack → prepares a fresh git clone
  start  → starts bbk services
  stop   → stops bbk services
  status → show bbk service status
  logs   → tails all bbk services logs
  db     → start a sql session
  wipe   → resets the bbk database
  drop   → drops the bbk database
  bounce → stop » start
  reset  → stop » wipe » start
  clean  → stop » drop

If you are going to develop on BitBroker, either for building data connectors or for contributing to the core system, then this mode of install can provide a fast and low friction installation technique.

Testing Your Installation

If everything worked as expected, the BitBroker API servers will be up-and-running on localhost waiting for calls. You can test this by using the sample call below. In this local mode, you don’t need to specify any authorization tokens.

curl http://bbk-coordinator:8001/v1

If you have not applied the standard server name and port format, then you should use http://localhost:8001 here as your API host base URL. The base end-points of all the three API servers respond with a small announcement:

{
    "now": "2022-06-16T10:44:53.970Z",
    "name": "bit-broker coordinator service",
    "base": "http://localhost/coordinator/v1",
    "status": "production"
}

Uninstallation

If you want to completely uninstall this instance of BitBroker, you can follow these steps, from the top-level bit-broker folder:

./development/scripts/bbk.sh clean
cd ..
rm -rm bit-broker

This will delete the Git cloned folder created earlier.

4 - Configuring BitBroker

In the section, we cover how you can configure your BitBroker instance to align to your specific needs.

Environment Settings

In this section, we outline the details of our installation environment file. This file is called .env and resides at the root of the BitBroker file layout.

There is no master record for this in the repository, however there is a file called .env.example, which contains all the most common parameters. You can activate this common set by simply copying this file:

cd bit-broker
cp .env.example .env

Here are all the settings in .env which can be modified:

You will also see some parameters starting with TESTS_ in the .env.example. These are reserved parameters used by the Mocha test suite. You can ignore these values unless you are developing the core system itself.

Advanced Use Cases

In this section, we outline the details of some more advanced use cases.