Guide through Kafka Connector, on how to migrate your streaming to Snowflake

Companies need to work on a non-disruptive migration that allows to simplify and ease the process of real-time ingestions, snowflake has easy integration with Apache Kafka.

Customers using the publish/subscribe Kafka platform can now push data directly into the snowflake tables using Kafka Connector with a minimal effort.

A simple path would be one table for each topic, with messages containing one row each, the table will contain two semi-structured columns ( Called VARIANT in snowflake ) which are RECORD_METADATA and RECORD_CONTENT, the content could be either JSON or Avro.

Tables can be automatically created by the Kafka connector, or if the user has a predefined table, the other columns apart from those two mentioned must be nullable.

A small schema below to illustrate the flow post-migration

https://docs.snowflake.com/en/user-guide/kafka-connector.html

How to install Kafka connector :

https://docs.snowflake.com/en/user-guide/kafka-connector-install.html