We are very proud to announce that we have recently published a Snowflake Native App on the Snowflake Marketplace. In this article we will introduce our data transformation framework, TiPS, and summarise our journey converting it into a Native App and achieving a listing on the Snowflake Marketplace.
Snowflake originally emerged as a revolutionary cloud data warehouse before evolving into ‘The Data Cloud’ in recognition of the six additional workloads supported (see diagram below). One of the capabilities that underpins all workload is the Snowflake Marketplace and now, in addition to buying curated datasets, the marketplace supports the purchase of apps that can interact with the data you hold in Snowflake.
At launch (public preview in June 2023) there were 23 Snowflake Native Apps. As of 30th August 2023, this number has increased to 31 (a complete list can be found here), providing various capabilities such as:
TiPS is a metadata driven data transformation framework which can be used to build and execute data pipelines. As TiPS relies only on first class database objects, there is almost no learning curve and so engineers already proficient in SQL can focus on writing transformation logic instead of spending their valuable time learning yet another tool.
TiPS is a metadata driven data transformation framework which can be used to build and execute data pipelines.
Once your data source is created in the database (generally a view encapsulating appropriate business logic) and a target is in place (generally a table), a step of your data pipeline is defined and TiPS will take care of dynamically generating all the DML statements needed to execute that step and move the data across.
TiPS was originally built for a client around two years ago, where their specific requirements precluded other data transformation options. Its capability has evolved in the intervening period and today it supports all of their Snowflake data transformation needs. We’ve since used TiPS with other clients and have observed that it works particularly well as an ‘accelerator’ in a Proof of Concept to minimise the number of new vendors or technologies the client needs to get to grips with and/or be reviewed by their InfoSec team.
TiPS works particularly well as an ‘accelerator’ in a Proof of Concept to minimise the number of new vendors or technologies.
Some of the salient features of TiPS include:
For more detailed information about TiPS including a guide to getting started and our road-map for the future, see our documentation.
Accessing TiPS as a Snowflake Native App means you benefit from the stringent and secure process Snowflake has put in place to review all apps before they are accepted onto the marketplace. This includes checks such as:
Having passed the above checks, we are delighted that TiPS is now approved and listed in the marketplace, meaning all you need to do is click a button to install it in your Snowflake account. Installation also includes a sample pipeline that can be executed immediately, which checks installation has been successful and shows TiPS in action. Since everything is hosted on Snowflake, you can rest assured that it is all running in your trusted secure environment and your data is not moving anywhere.
Snowflake Native Apps is currently in public preview and still evolving. There are some database capabilities not yet supported but that will hopefully be available in future releases, some that may never be supported (due to security), and some process improvements that would be useful. Our key observations include:
Native Apps are a very exciting new capability in Snowflake. It has been an interesting journey to get to grips with them and we’ve learned a lot along the way. Aside from Snowflake’s documentation (which is also evolving), there isn’t much information in the public domain right now. However, we would encourage you not to be put off by this as the support offered by the Snowflake Marketplace team is fantastic.
All you need to do is click a button to install it in your Snowflake account.
If you’d like to know more about TiPS, discuss your own ideas for a Snowflake Native App, or talk about anything related to Snowflake and Data Engineering, please get in touch.
Established in 2006, Projective Group is a leading Financial Services change specialist. With deep expertise across practices in Data, Payments, Transformation and Risk & Compliance.
We are recognised within the industry as a complete solutions provider, partnering with clients in Financial Services to provide resolutions that are both holistic and pragmatic. We have evolved to become a trusted partner for companies that want to thrive and prosper in an ever-changing Financial Services landscape.