Google Cloud Platform (GCP) Cloud Pub/Sub Product Overview
Google Cloud Platform (GCP)
A global, multi-tenanted, managed, distributed, real-time messaging service
Use with Dataflow in order to get exactly-once, ordered processing
Concepts
Topic
Subscription
Publisher
Subscriber - can be many subscribers on the same subscription
IoT Data Ingestion Challenges
Handle data streaming from various processes or devices
Distributing event notifications for multiple systems
Scale to handle volume
Reliable delivery of messages, and without duplicates
Features
At-least-once delivery
Protocol
Subscriber ACKs each message
Message is resent if subscriber takes longer than the 'ackDeadline' setting to respond - so subscriber COULD get a DUPLICATE message
A subscriber can extend the deadline per message - to the reduce chance of missing the deadline and getting a duplicate message
No provisioning, auto-everything
Open APIs - all libraries are just doing REST calls
Global
Topic/subscription namespaces are globally unique
Messages are stored in the region closet to the publisher (and duplicated in multiple availability zones)
A subscription collates topic messages stored in different regions
Subscribers can be anywhere
End-to-end encryption
Messages are durable and persist for 7 days
Unordered message delivery
Push (webhook, low-latency) and pull (for large number of subscribers) delivery flows
Fast - order of hundreds of milliseconds
Client libraries in Java, Python, C#, Ruby, PHP, Node.js, and auto-generated in 10 gRPC languages
Fan In - multiple subscribers listening to a single topic
Fan Out - multiple publishers publishing to a single topic
How To:
Topics
Create a topic: gcloud pubsub topics create mytopicname
Create a topic and publish a message: gcloud pubsub topics create mytopicname "my message"
Create a topic in Python: from google.cloud import pubsub ; publisher = pubsub.PublisherClient() ; event_type = publisher.topic_path(project, "mytopicname") ; publisher.create_topic(event_type) ; publisher.publish("mytopicname", b'my message', extra='extra attribute', myid="1234")
Batch messages in a single request in Python: with publisher.batch() as batch: ; batch.publish(payload1) ; batch.publish(payload2, extra='attribute', myid="1234)
Subscriptions
gcloud pubsub subscriptions create --topic mytopicname mysubname
gcloud pubsub subsriptions pull --auto-ack mysubname
sub = topic.subscription(mysubname) ; sub.create() ; results = sub.pull(return_immediately=True) ; if results: ; subscription.acknowledge([ack_id for ack_id, message in results])
gcloud pubsub subscriptions delete mysubname