In this article, Steve Jenkings, our Data Engineering Lead, gives a high-level introduction to some of the new features in Snowflake and explains how they may enable a secure, governed, and scalable technical architecture that can support all operational activity and reduce the need for many other technology vendors.
Snowflake started life as the first data warehouse written specifically for cloud. The initial architectural decision to separate storage and compute underpinned many ground-breaking capabilities that were built upon exceptional out-of-the-box query performance and consideration for security and governance as a first party capability.
The offering has since expanded considerably and the Snowflake Data Cloud now also encompasses features such as the marketplace, data clean rooms, SnowPark, Snowflake Native Apps, and support for unstructured data.
At Projective Group, we have delivered Snowflake projects for our clients in a variety of areas including FRTB compliance, Finance Product Control and quant analytical platforms. So with a host of new features recently released or coming soon, how many more use cases can Snowflake cater to? Let’s find out.
Snowflake is currently optimised for the execution of analytic queries (e.g. what was the total sales value by quarter by country in the last year). Hybrid tables introduces a new row format storage model designed to best optimise the execution of transactional queries (e.g. update the status to ‘expired’ on the trade with ID ‘12345’). In a nutshell, Snowflake is moving to support both database and data warehouse workloads.
Hybrid tables introduces a new row format storage model designed to best optimise the execution of transactional queries.
FS Use Cases: Trade capture and trade lifecycle management, or near-real-time analysis of data such as fraudulent transaction detection, could now be undertaken in Snowflake. Further, the need for data integration tooling (i.e. to move data from a source system into Snowflake) is negated or minimised as more data is fully mastered in one single platform. Data transformation tooling will likely remain as a significant component in the delivery of a typical Snowflake project to create and populate a model suitable for a specific use case, however even this could be reduced with the new table type.
Our Observations: Hybrid tables have just been released into public preview in limited regions and on limited clouds. Although performance will no doubt improve over time, it’s unlikely in the first instance that very highly demanding transactional use cases can be supported. Additionally, support for zero-copy clone has not been extended to hybrid tables at this point, limiting the ability to leverage modern data ops practices against this new table type.
Streamlit is a simple scripting language that supports the quick creation of interactive, user-friendly web applications running against data stored in Snowflake. With a short learning curve, Streamlit will be accessible to many people; not just experienced engineers. Our very own data transformation framework, TiPS, which is fully open sourced, is now available as a Snowflake native app which includes a UI built in Streamlit (see this page for full details).
Streamlit is a simple scripting language that supports the quick creation of interactive, user-friendly web applications running against data stored in Snowflake.
FS Use Cases: Existing web apps for trade capture and trade lifecycle management, and many more too, could be now be built in Streamlit (underpinned by hybrid tables). Streamlit also excels in the provision of interactive dashboards encompassing machine learning models, potentially replacing many existing dashboarding tools.
Our Observations: At this point in time Streamlit isn’t quite rich enough to replace a sophisticated trade capture tool, and state management in Streamlit UIs can also be a challenge. However, the breadth of capability in Streamlit has increased very quickly in the time we’ve been using it and with each release more sophisticated UI support is available, meaning more existing use cases become viable.
Snowpark Container Services provides a fully managed capability that supports the deployment, management, and scaling of containerised workloads. This means both new and existing business logic (in any programming language that can be packaged as a container image, e.g. C/C++, Node.js, Python, R and more) can now run on ‘compute pools’ in Snowflake. These can be specified as either CPU or GPU hardware and are automatically provisioned and managed by Snowflake on the cloud provider of your choice. As containers run co-located with your data, your data never leaves Snowflake, thus security and governance are simplified considerably and execution performance promises to be exceptional.
As containers run co-located with your data, your data never leaves Snowflake, thus security and governance are simplified considerably and execution performance promises to be exceptional.
FS Use Cases: Existing applications can run in Snowflake with minimal change; an overnight compute batch using complex and computationally intensive pricing and risk models, such as Monte Carlo, for example. In this case it’s harder to think of a business process that would not be impacted by this capability than the reverse.
Our Observations: It’s a little early for us to add any informed observations on this capability, but stay tuned for a dedicated blog on our findings in the future.
Cortex is a fully managed service supporting data analysis and the creation of AI applications within Snowflake. Although Snowpark Container Services can also support AI & ML capabilities, creating these requires significant engineering skills. By contrast, Cortex provides built-in capabilities that make AI & ML accessible to people without such expertise in this area.
Cortex provides built-in capabilities that make AI & ML accessible to people without such expertise in this area.
FS Use Cases: There are some well-established AI & ML use cases in Financial Services, such as fraudulent transaction detection or loan default prediction, as well as some newer use cases emerging, especially around LLMs and Generative AI, such as executing natural language queries against both structured and unstructured data. Cortex promises to support many of these use cases whilst reducing the need for Data Science expertise and speeding up the delivery of models into your production estate.
Our Observations: Cortex is in private preview and we have limited exposure to this capability at this point. One general observation is that an organisation needs access to significant volumes of up-to-date data that is also governed and checked automatically to ensure quality. If your organisation isn’t at this point already, then now is a great time to put these foundations in place, or your AI & ML agenda could very quickly become blocked.
For Financial Services organisations looking to create a competitive advantage with their technology architecture, we believe that the combination of existing and new Snowflake capabilities could be a game-changer. By leveraging the entire Snowflake stack, banks and insurers can reduce, or even ultimately eliminate, their usage of many other technology products. In doing so, they can streamline their operations to be cost-effective whilst maintaining or even increasing their operational speed, agility, security, and control.
The combination of existing and new Snowflake capabilities could be a game-changer.
Modern FinTech companies, without the legacy technical debt of the majority of their longer established counterparts, have a distinct advantage here. They are already cloud centric (no mainframes!) so their existing architectures are a step closer to the one presented here. Plus, their Agile approach means they can deliver controlled change very rapidly.
Although longer established organisations retain significant market advantages, this is being slowly eroded. When combined with the operational efficiencies derived from technology simplification and increased rate of change, we could be moving closer towards something of a tipping point unless significant initiatives to modernise are undertaken everywhere.
At Projective Group, our experienced engineers are highly skilled across the whole Snowflake stack and continually invest time learning and experimenting with new capabilities, such as those highlighted in this article. Plus, we were an early adopter of Snowflake Native Apps, having converted our own data transformation framework, TiPS, into a native app that is now available on the Snowflake Marketplace.
If you’d like to find out more about how our industry leading expertise can support your plans to use Snowflake, optimise your existing usage of Snowflake, or help at any stage in your Snowflake journey, please get in touch with us here.
Established in 2006, Projective Group is a leading Financial Services change specialist.
We are recognised within the industry as a complete solutions provider, partnering with clients in Financial Services to provide resolutions that are both holistic and pragmatic. We have evolved to become a trusted partner for companies that want to thrive and prosper in an ever-changing Financial Services landscape.
