GCP IoT Core

In this tutorial, we will deploy a simple open-source application that collects metrics and send them to AWS IoT Core for further processing. All code for this blog post can be found at:

https://github.com/synpse-hq/metrics-nats-example-app - Sample metrics application https://github.com/synpse-hq/gcp-iot-core-example - GCP IoT Core example

Technologies used

1. Synpse - manage devices and deploy applications to them

2. NATs - a lightweight message broker that can run on-prem

3. Google IoT Core - message broker between all devices and GCP

Steps:

1. Create GCP IoT core

2. Configure data flow to forward results into Google storage account

3. Create GCP device for Synpse

4. Demo Synpse application from 3 microservices -Metrics demo, Nats messaging, AWS IoT python forwarder containers

GCP IoT Core

GCP Core works by creating a "device registry". So this is one of the first steps we have to do.

Configure GCP IoT Core create registry

Create pubsub topic for registry

gcloud pubsub topics create synpse-events

Create GCP IoT registry:

gcloud iot registries create synpse-registry --region=us-central1 --enable-http-config --enable-mqtt-config --state-pubsub-topic projects/{project_id}/topics/synpse-events

Create storage event

gsutil mb -l us-central1 -b on gs://synpse-events

Create dataflow job

gcloud dataflow jobs run ps-to-avro-synpse-events --gcs-location gs://dataflow-templates-us-central1/latest/Cloud_PubSub_to_Avro --region us-central1 --staging-location gs://synpse-events/temp --parameters inputTopic=projects/iot-hub-xxxxxx/topics/synpse-events,outputDirectory=gs://synpse-events/events,avroTempDirectory=gs://synpse-events/avro-temp

Data flow would look as bellow:

DataflowCreate a device

gcloud iot devices create synpse --region=us-central1 --registry=synpse-registry

Generate certificate for our device

openssl ecparam -genkey -name prime256v1 -noout -out ec_private.pem
openssl ec -in ec_private.pem -pubout -out ec_public.pem

Add public key to GCP IoT Core

gcloud iot devices credentials create --region=us-central1 --registry=synpse-registry --device=synpse --path=ec_public.pem --type=es256

Download google root CA

curl  https://pki.goog/roots.pem -o roots.pem

(Optional) Test application

python gateway/gcp.py --device_id synpse --private_key_file ./ec_private.pem --cloud_region=us-central1 --registry_id synpse-registry --project_id iot-hub-326815 --algorithm ES256 --message_type state

Deploy an application

Deploy Synpse application. Modify application yaml with your thing endpoint.

Create certificate secret

synpse secret create gcp-cert -f ec_private.pem
synpse secret create gcp-root -f roots.pem

Deploy the application:

synpse deploy -f synpse-gcp-example.yaml

where synpse-gcp-example.yaml is

name: GCP-IoT-Hub
description: Google Cloud IoT Core Synpse example
scheduling:
  type: Conditional
  selectors:
    gcp: iot
spec:
  containers:
    - name: nats
      image: nats
      restartPolicy: {}
    - name: metrics
      image: quay.io/synpse/metrics-nats-example-app
      restartPolicy: {}
    - name: gcp-iot
      image: quay.io/synpse/gcp-iot-hub-example
      command: /server/gcp.py
      args:
        - --device_id=synpse
        - --private_key_file=/server/ec_private.pem
        - --cloud_region=us-central1
        - --registry_id=synpse-registry 
        - --project_id=iot-hub-326815 
        - --algorithm=ES256 
        - --message_type=state
        - --ca_certs=/server/roots.pem
      secrets:
      - name: gcp-cert
        filepath: /server/ec_private.pem
      - name: gcp-root
        filepath: /server/roots.pem
      env:
        - name: NATS_HOSTNAME
          value: nats
      restartPolicy: {}

Once running, you should see application running and data coming into the GCP storage account blob.