Why would anyone use Snowflake as a backend for Django?

Thank you very much. Thank you. It's Friday, 4 p. m. We are still here. Um that's amazing. It's been an amazing three days I stopped doing web development many years ago so to be back to my roots with Django people and showing off what I've been doing on the data world. Since I left the web development, I joined Google. I was KT's teammate. I spoke around the world about BigQuery and two years later I left Google and I became a developer advocate for Snowflake, which is a real company, not an insult. And and yeah, uh the question I w I asked myself a couple months ago uh here in the data world is why would anyone use snowflake as a back

end for Django? And we're going to review this. Um I'm glad that the next talk here will be about uh getting corporate uh sponsorship for open source development because that's part of this story so it's not only friday at 4 p. m later there's more to go deeper into this um When I was thinking about the audience here, I'm thinking about three different audiences as a developer advocate. One it's you, DjangoCon. I want to introduce you to Snowflake that you can use for big data projects. But also I want your feedback. I want you to help me improve this presentation and my talking points when I talk to Snowflake customers. Because There's a lot of Snowflake customers that

want Snowflake and Django. And I want to tell the story to other Snowflake customers. And at the same time, my audience are Snowflake employees as a developer advocate I work inside the company and I have to convince some people that Django is still super cool, it's still super interesting, and that Snowflake, the company, should sponsor uh more open source development. To start with uh the my initial question is what how how do how why would people want to combine Django and Snowflake? Django is famously known as a project to build websites, or LTP as we all call it on the database world, transactional. And Snowflake is a database built for data analytics, all

up. Let's analyze a lot of data, let's not have any indexes, let's just allow people to analyze a lot of data. And these two worlds In my mind, a couple of months ago don't mix very well. And I'm going to use some people reactions as the only reactions I had uh a couple of months ago because yes transactional data and data for analyzing are two different worlds. Um Django Uh supports uh you know Postgres, Maria DB, MySQL, Oracle, SQLite, those are the standard databases. These are not analytical databases. So when I went to Twitter and I talked back in February about Snowflake as a backend

for Django, uh Ben Stancil reply was because you can. It's not really something you should do. And other people talked about data gravity. All your data is in a snowflake. So let's use snowflake because and Django because the data is there. And people are like, meh, I don't know about that. In this day and age it's super easy to move data around. Uh reverse ETL, you get your data into your post out of Snowflake, into Postgres, etc. And again Ben is like this feel more so like experimentation, someone just pushing the edges, this is not meaningful, uh smart people do stupid stuff And this is the part of the talk when I tell the not Django audiences what Django is. So these are my

I go to Wikipedia, I tell people that don't know that Django is a web framework, that it has been powered on Instagram since 2011. It was powering Clubhouse while Clubhouse was alive. And what you might not know much is about Snowflake which is a database in the cloud, software as a service, nothing to install, you just pay for what you use. And it's built to analyze data and it's built from scratch for the cloud and that made it different than many other things that uh started as a database not in the cloud and then like for example You might know Redshift. Redshift is like a

fork of Postgres and then they made a cloud service out of it, but it's not cloud native. Snofflake has been built for many workloads, data engineering, data leg, data warehouse, data science, data applications, data sharing. Um I was at a previous conference where I gave this talk the last time and the w someone at the booth was like, hey, we already tried Redshift as a backend for Django. It didn't work. So people have been trying this combination of OLAP on LTP. It doesn't work. And this is something important on the Snowflake architecture, how it works, how it's built for the cloud On the first layer you have the database storage that we use

uh cloud storage, Google Cloud Storage or S3 and basically that's infinite And then you have virtual warehouses that can query that data uh in parallel. So you can have one virtual warehouse dedicated to your all your analyst and you can have a virtual warehouse dedicated just to Django and in that way you can split workloads and you can scale and you instead of thinking about sharding you just create more compute units for dedicated workloads. So you don't have the problem of if you put everything together in one database, then things stop working. No. You just ask for more compute power, it works. And yes, uh Ben is still fighting with me uh because yeah

apps need tons of transactional queries versus few batch ones, apps read and write. Apps have lots of consistency and concurrency issues, this will not work. And in case you don't know Ben, he's the chief analytics officer and the founder of Mode. So he understands data. This is how I used to think. On the other side, who wants Snowflake as a backend for Django? I'm going to talk about Scott Scott Fott. He's a lead product manager, data science as an engineering for a company. I cannot say the name. He didn't give me permission, but He gave me permission to say that it's a big entertainment company with theme parks in Orlando and in Paris and in other places. You figure out. Ben Ryan, I don't know who knows Tim Graham.

He says hi. I'm going to talk about him too. And thousands of people around the world want this. When I say thousands, I just go to Stack Overflow. Django to Snowflake Connection running on Real queries from four years ago, 2,000 views. Similar question from two years and ten months ago, 3,000 views. People are asking this question. It might be a silly question, but they are asking it. And people started creating these connectors. Price move created a really Publish it a read-only database backend for Django. Uh Ben Ryan, he also created one, he wrote the zero docs for it But there is a uh package ready to be installed so and this one since March this year had more than a thousand installs.

And then Scott Fod from this big corporation came to the Django Bailey list and he's like, we wrote this Snowflake DB backend at this. Mouse Corporation and they wanted to release it. They never released it, but he started that conversation back in January 2021. And that's when Tim Graham replies and he's like, hey, I wrote the backend for Cochroat, I wrote the backend for CloudSpanner, maybe I can do something here. This is where he's offering to develop something in exchange for money. That's a corporate sponsored open source that I love If you don't know Tim, he was a Django fellow from 2014-2019. Cockroach sponsored him to build the Django

adapter. He also googled it and he does a lot of open source work sponsored by corporations. And yeah, he comes back December 2021, he released it. It's great. A company called Cedar Team created the connector and we are all very happy because someone that really understands uh Django created it. It's It's open, these are his notes, how he developed the the backend and uh finally we have this on GitHub available for everyone And since March it has at least five thousand installs, which means people want it, people are using it, we have corporations paying for its development, we have Team Graham developing it.

This is just to give you a little data in context. Um in the last month Snowflake had a thousand installs, while the most popular third-party connector that I found this SQL Server, also Cobroach, Redshifter 's a lot of people using it, YugaByte, Spanner that, um Tim also created and TDB has very few, but you can see that there is demand there. And of course, Tim while working on this he made fixes for the Snowflake connector, which is another cool thing about uh open source. I'm not uh how I'm doing on time because I have a demo but it probably doesn't make sense to show you this demo. I'll just show you the screenshots. You know Django

Uh one factor here is that team created it for Django 3. 2. Uh we are still waiting for uh someone to sponsor the 4. 0 version. And I'm waiting I really want to say Snowflake is sponsoring it, but these are the challenges while developing open source. Uh Snowflake wants to sponsor it, but getting the money out, working through the machine is always a little harder. Hopefully I can announce that soon. But yes, uh for people that don't know Django, you start a project, uh on the Snowflake side you create a user, a database. Of course you should do this with Terraform really, but I'm showing you the basics, then the connection configuration. And then you have Django

working, you can run the first migrations, and it's all working and it's cool. And you can start creating your own apps This is the these are the tables on the Snowflake side. And then I made my first example with the Stack Overflow. I have a whole copy of Stack Overflow all the questions, all the answers, and I wanted to create an app to show it off working at that scale without indexes. And it works. So I created my three tables. answers, questions and the tag of each question. And this is the what I love about Django the ORM just having um Managing false databases, having the ability to define foreign keys, and then the ORM

making my life super easy while I'm developing a web app. And it works. This is a view. This is a template. You all know that part, but I love your feedback on how I introduce this to other people that are not familiar, and I can show them the power of using Django with Snowflake and it runs. You have the server, it's working. This is my simple example. These are the queries running on Snowflake. So for example, looking at all the questions for Stack Overflow Um with those filterings takes three seconds, answers, blah blah blah. It works. And it took me no time to get this working. I didn't have to think about indexes. I can have a data app in a very short time. And these are okay the queries

running. And this is where I want to show you a little bit of the source code, the notes that team wrote while developing this. So of course when creating a new connector, there are the connections of the conventions of Snowflake , the surprise that Snowflake doesn't support indexes. JSON field is not supported not because Snowflake doesn't support JSON, there is a variant type, but again that there's part of the uh development challenges of how would we adapt the variant to JSON and he did Snowflake doesn't check constraints. It's built for big data. It's not built for a transactional And of course, how do we deal with subqueries?

Little issues and limitations that he had to deal with. Um so going back to the question, why would anyone use a snowflake as a bucket for Django? Scott uh has the best answer Uh yes, maybe Snowflake is not as fast as Postgres and Barilla when doing transactional work, but If you are working with big data and if you want to work with Django, uh sometimes the best answer is a snowflake. Um this is my favorite quote He was put on a project that used the angle and postgres. And they had jobs that were failing because Postgres wasn't performant enough. They tried to make password faster, let's get a bigger material, but the performance wasn't there

because it didn't scan queries They did the proof of concept, they moved they deleted the Postgres database, they moved everything to Snowflake. All the data was already in Snowflake, so they didn't need to move it anymore, and things went way faster. And this is the real value. Sometimes we super optimize things like by creating copies, going to Postgres, and sometimes it's like, no, just leave it in leave it in a snowflake with big data, it might scale way more. Um this is the part where I want to show you a little bit about the future of data apps, how is Snowflake thinking of how these things are evolving because Snowflake is doing big investments in this part that can have a lot of um consequences,

nice things that can come when we try to merge Snowflake and Django. One is I don't know how many people know Streamlit is another uh Python web framework which works in ve with components. And it's pretty magic how it does things and Snowflake bought this open source Python project like for 800 million dollars and it's a way of showing the commitment that Snowflake has to We want to work with the open source community, we want to work with the Python community, we want to people to build data apps and facilitate this Um I I I wrote a quick demo of working with Streamlit this, I don't know, I've got I could run it for you now, but

Just to show you what's happening here, I have three selectors to look at Stack Overflow tags. You can choose a tag, you can choose what you want to see, you can choose other tags you want to compare with And it's super easy to just write these data apps. You say you get the data from SQ you write the SQL query, you get the data out. And then you just ask for the component. Show this as a table, show this as a selector, give me the result. And with no views, no templates, you just get like an interactive web app, which is pretty magic. But it's different than Django and part of my work now at Snowflake has people asking me why should we work with Django if we have Streamlit already? Streamlit has all this magic.

Well with Django you have less magic, you need to develop more of your templates and views. You have way more flexibility and power and a huge community of people that are very experts on Django and There is space for both worlds and there's also space to combine both. Another announcement that Snowflake did this year, that which is huge Is that there's a new workload for a new way to store data in Snowflake which is transactional, which has indexes, and you have tables living right next to each other So all of the problems that I described earlier that Snowflake would not deal with indexes or with quick transactions

Now we're changing that and we're all build building that inside the data warehouse. And basically the basic element here is the ability to create a hybrid table. And the hybrid table has indexes, it has constraints, and it works way faster when you need to do a lot of updates, inserts, etc. And another great innovation on the Snowflake world is that Snowflake now runs Python inside the data warehouse That means I can write Python code, Snowflake understands it, I can create um Python UDFs that can I can use inside my SQL queries of my tables. I can move a lot of logic inside Snowflake. And I can

use uh the expertise that Python developers have. So instead of writing everything on the front end layer You can move code to the database, you can move logic to the database and write it all on Python. Just as an example here, this is a UDF that does call Profit, a forecasting library. At first it looks like a SQL UDF. I'm just saying it's a Python UDF. So the code inside will be in Python. And I want to use Profit, uh which is pretty hard to install normally in uh When you try to install it by yourself, but Snowflake is providing all of these packages already installed. Anaconda is the one developing all of these, bringing all these packages into Snowflake.

Which makes uh my ability to just go and write some Python code and logic uh pretty cool. And with that just coming back to Ben uh with all these announcements uh He looks at this and he basically posts this blog post that says hey Snowflake could be on the cap of changing what a database is, what data apps are, how they get built and solved, and what we can do with both of them They could be building a platform that stops the industry once more and leads to another explosion of ideas and products. So It's really cool to be able to change bands' minds and change people's minds of what you could do with Snowflake, with Python, with data

apps. uh enable all of these use cases and hopefully you can use this in your work hopefully uh It it this talk has been useful in case you ever need uh you have big data problems that Snowflake could solve if you could be end up working in one of these projects, if you could uh participate in this ecosystem, hopefully I'm I'm able to help In summary, uh people want Jungle and Snowflake, it works well, it it's better than many of the alternatives in the right context. I love corporate sponsor open source. Hopefully we get more of that. Hopefully I can soon announce that Snowflake is sponsoring the Django project and the Python Foundation, etc.

Um and yes, uh just choose the database that meets your requirement best There's all these new abilities and if you want to try Snowflake of course it's free, there's a free trial, no credit card, so you can go and put all of this together. Thank you very much. I'm Felipe Hoffa. I'm on Twitter. I'm Stack Overflow. Thanks a lot. Yay.