Sun. Feb 28th, 2021

Guide through Kafka Connector, on how to migrate your streaming to Snowflake

Companies need to work on a non-disruptive migration that allows to simplify and ease the process of real-time ingestions, snowflake has easy integration with Apache Kafka.

Customers using the publish/subscribe Kafka platform can now push data directly into the snowflake tables using Kafka Connector with a minimal effort.

A simple path would be one table for each topic, with messages containing one row each, the table will contain two semi-structured columns ( Called VARIANT in snowflake ) which are RECORD_METADATA and RECORD_CONTENT, the content could be either JSON or Avro.

Tables can be automatically created by the Kafka connector, or if the user has a predefined table, the other columns apart from those two mentioned must be nullable.

A small schema below to illustrate the flow post-migration

Kafka Connector for Snowflake

How to install Kafka connector :

By Amine Hallam

Author & blogger on the latest data analytics technologies: Hadoop, Snowflake, Cloud ( AWS ), ETL .. etc

Leave a Reply

Your email address will not be published. Required fields are marked *