Postgres Performance: From Slow to Pro with Elizabeth Christensen

This video features Elizabeth Garrett Christensen at DjangoCon US 2023 in Durham, North Carolina, USA.

Postgres Performance: From Slow to Pro with Elizabeth Christensen
0:43:06
Published November 22, 2023
957 views

At some point, every application is limited by the database. You don’t have to be a Postgres expert to get started with a few key performance improvements. This gentle introduction is meant for folks who’ve never ventured into their database before, or those who have been turning knobs blindly. I’ll present how Postgres uses memory. Then, I’ll connect that to how you can monitor, tune, and optimize queries. You’ll be ready to take on the challenge as your application grows.

This talk was presented at: https://2023.djangocon.us/talks/postgres-performance-from-slow-to-pro/

LINKS:
Follow Elizabeth Christensen 👇
On Twitter: https://twitter.com/sqlliz

Follow DjangCon US 👇
https://fosstodon.org/@djangocon
https://twitter.com/djangocon

Follow DEFNA 👇
https://www.defna.org/

Video production by the presenter and DjangoCon US 2023 volunteers.

Summary

Elizabeth Christensen explains how to improve PostgreSQL performance in Django applications by understanding memory, disk I/O, connections, caching, and query execution. She recommends tuning shared buffers and per-connection memory, monitoring cache-hit ratios, IOPS, bloat, locks, and long-running queries, and using tools such as `pg_stat_activity`, `pg_stat_statements`, `EXPLAIN`, Django Debug Toolbar, Django Silk, and PgBouncer. She argues that practical performance work starts with measurement: identify the slowest and most frequent queries, then address them with appropriate indexes, better data modeling, efficient ORM usage, and sensible connection management.

Key takeaways

  • Configure shared buffers, working memory, maintenance memory, and effective cache size instead of relying on PostgreSQL defaults.
  • Use cache-hit ratios, IOPS, temporary-file logging, table-bloat checks, and statement timeouts to identify database problems.
  • Enable `pg_stat_statements` and use `EXPLAIN (ANALYZE, BUFFERS)` to find slow, frequent, CPU-heavy, or disk-intensive queries.
  • Add carefully chosen indexes, including multicolumn and partial indexes, while remembering that indexes consume storage and slow writes.
  • Prevent excessive connections with appropriate limits and consider PgBouncer as applications scale.
  • Avoid ORM N+1 queries and separate frequently updated data from relatively static data to reduce bloat and unnecessary work.

Summarised automatically from the transcript.

Chapters

  1. 0:00 Introduction and Performance Roadmap Overview of PostgreSQL performance, costs, application speed, and the main topics covered.
  2. 4:12 PostgreSQL Data Flow and Caching How application connections, shared buffers, disk storage, reads, writes, and I/O fit together.
  3. 6:28 Memory Configuration Tuning shared buffers, work memory, maintenance memory, effective cache size, and cache-hit ratios.
  4. 11:58 CPU, IOPS, and System Resources Monitoring CPU and disk activity, limiting runaway statements, and using resource usage to diagnose memory issues.
  5. 14:20 Table Bloat and Vacuum How dead rows accumulate, how autovacuum helps, and when table bloat requires attention.
  6. 16:42 PostgreSQL Version Upgrades Why newer PostgreSQL releases improve performance and why supported versions matter.
  7. 17:28 Connections and PgBouncer Managing connection limits and memory usage, and using PgBouncer to pool connections as applications scale.
  8. 20:38 Application Performance Diagnostics Using Django tools, application logs, and response-time targets to locate slow database interactions.
  9. 24:34 PostgreSQL Logging Configuring slow-query, temporary-file, lock, and structured logging without overwhelming log storage.
  10. 27:39 pg_stat_statements and EXPLAIN Finding frequently or expensively executed queries and analyzing their execution plans, scans, buffers, and I/O.
  11. 33:52 Indexing Strategies Choosing indexes, evaluating multicolumn and partial indexes, testing hypothetical indexes, and removing unused ones.
  12. 38:26 Data Modeling and N+1 Queries Reducing update-related bloat through data modeling and preventing ORM-generated N+1 query patterns.
  13. 41:31 Performance Checklist Final recommendations for statement timeouts, query monitoring, indexes, memory, connections, PgBouncer, and upgrades.

Transcript

6,739 words · auto-generated Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:22

Good morning. I'm so excited to be here. Um we're going to talk about Postgres performance and we're going to talk about getting started. So going from the very, very beginning to kind of getting up to speed. I am Elizabeth Christensen. I love Postgres. I work at a Postgres company called Crunchy Data. I'll be happy to talk to you about that if you want to know more. I'm a volunteer for the United States Postgres organization and I have been at this booth out here and I do some other volunteer work for them and I host a meetup for Postgres in Kansas Kansas City. I am also from Lawrence, Kansas, like a lot of people here. If you're in the area, we would love to have you. And I'm on various social media platforms

1:07

as SQL Liz. Um, so even though the title of my talk is From Slow to Pro, the Postgres isn't actually slow. You know, really if if you're getting started with a database with Postgres or any of the databases that work with the Django ORM, you're you're gonna be just fine. You you know hook up your stuff Use your ORM, stuff goes in the database, and things just work. And as a sort of overarching idea, Postgres is very stable. It doesn't lose your data. It 's got tons of options for data types and different things that you want to do with your application. It's an overall great database for running with your front end.

1:52

applications. So when you're talking about performance, right, you are building some kind of application, you need to sort of do some things with your database. Um and you're probably gonna be concerned about cost, right? Um compute costs money, um databases are notoriously memory intensive. And so what we kind of want to do when we're talking about Postgres performance is really maximize what you're getting out of the database itself in terms of how much you're spending on the database We want to keep things really fast for our users, right? Because you guys are application developers and if stuff is slow, it looks like your application is slow. So we never want that to be the case.

2:39

And we want to kind of optimize as we go. So a little bit of a roadmap today for what we're going to talk about. We're going to talk about how data gets in and out of Postgres. Some memory settings that you'll want to look at, some other machine things like CPU and IOPS. We'll talk about connection usage and then we'll talk about some query performance. So Writing a performance talk is really challenging, right? Because there's two big parts to um to database performance, right? One is the machine itself. And so how Postgres is doing things, what's going on inside the database, and the other piece is how your application is interacting with the database. I will spend a lot more time on the database stuff itself

3:27

because that's a little bit easier to talk about to a general audience And I'll give you some tips about how to find individual performance issues inside your application. But generally once you're past the Everything is good with the database. There's a lot of performance stuff that goes into the application side. So I'll just touch on that. And if you're wondering, um Well but what about my hosting? I'm on Amazon RDS. Like how does this gonna work? So in general, um the stuff I'm gonna talk about today is pretty applicable to anybody hosting Postgres anywhere If you're using a managed service like a Crunchy Bridge or an Amazon RDS, you might have some of your configuration menus

4:12

in a different place. And you may have some of the memory settings and some of this stuff pre-configured for you. It's a good idea to know what's going on under the hood. So even if some of it's done for you, we'll we'll cover some of that. All right, so at the sort of start, I just kind of want to go over all of the layers of data when you're when you're using an application and a database and where lots and lots of caching happens. So if you're you're getting started in performance, you're going to be talking a lot about caching data and having data at the ready. So at the very high level, you've got, you know, your your web Stuff.

4:57

You've got people coming in from the outside world, and then you've got your application layer, and then underneath that you've got your database layer. And inside your database layer, you may or may not have a connection pooler, and we'll talk about that in a little bit. And then the individual pieces that actually connect to the database are called. They're called a client backend. So when you're digging through the Postgres docs, that's what it's called. And that individual connection that is coming from your application to read or write data. is going into Postgres and into the shared buffer cache. So if you're writing data to a database, it's going to go through the shared buffers and down

5:43

into the physical disk storage. Right, because Postgres is stored on disk, it's a stateful storage, and at the end it has to be stored on disk. If you're reading data Some of your frequently accessed data will be in shared buffers, which means you don't have to go to the underlying disk to go get the data. And so when you're talking about a database and you're talking about IOPS and input-output, every write that you write to Postgres is going to write to the underlying disk. and reads that are not in shared buffers are going to read from the disk. So when you are kind of setting up your memory and your database memory for performance

6:28

you want to minimize maximize what's in shared buffers, right? You want the data that your application is querying to be in that buffer cache so that when you your application asks for it, it's right there. And then you kind of want to know that if you're using IOPS, and we'll get into some of these specifics later. The more IOPS that you use and the more stuff that you read from disk, the slower your database is going to be and the slower your query is going to be for your users. Um so the first kind of big picture idea for getting sort of your your memory configured right is working on these shared buffers. So in general you can do about a quarter of the memory that you have on your machine for shared buffers.

7:17

I've got a few sample sample numbers in here for just like a really small kind of standard production machine that's like, let's say it's eight gigs. You know, it's just a couple cores. It's not a big huge machine. Um, but you can run a decent production size machine and Postgres on eight gigs of memory. And so that standard Postgres memory setting of 128 megs is not going to do it. You're going to want to get yourself some more shared buffers. And the way that you can check shared buffers in Postgres is with the cash-hit ratio. So Postgres stores when a query come in when a query comes in, whether or not that query data was part of the cache

8:03

or whether or not it was missed and it had to read from the underlying disk. And so that's a query that you can just ask Postgres, what is my cash hit ratio? And Postgres will tell you You're gonna want a number in the high 90s, you're doing really well. If you're lower than that, you're not doing great Something to keep in mind here is that if this is new data, if you added a lot of data, if you restarted recently, this is gonna be different. So this is Cash hit ratio for all the data is the same, you haven't loaded anything big recently, you'll want to be in that high 90s. So each individual connection to Postgres, so those individual client backends

8:50

Also connect to Postgres and each of those pieces, each of those connections use their own memory. And so Postgres will default to four megs for each of those connections And then typically you will want to make that a little bit bigger, right? It used to be you could get a really easy number for work memory by just knowing how big your machine is, like it's 8 gigs and I'm going to use a quarter of my memory for working memory and I've got 100 connections It's going to be about 20 megs. Um Postgres, all the modern versions of Postgres now have parallel queries, so a single shared backend

9:35

can use a little bit more than it's allocated. And so some of this math gets a little complicated. So when you're kind of doing this, you play around In general, for people on like an 8 gigabyte machine, we found that, you know, 16, 18, 20, somewhere in that range is probably good. Definitely the default is too small for a production Postgres use case. Um so if you um have your Postgres connections coming in, right? And you have that working memory and the each connection is allocated a certain amount of memory to go and get the data that it needs from shared buffers or from the disk If that connection doesn't have enough memory, it will spill

10:22

that query and that workload to a temporary file. And so when we'll talk about this a little bit later when we talk about logging, but if you're trying to figure out am I am I good with working memory or do I not have enough, your your sort of clue to figuring that out is the Postgres temporary file and how many of those you're generating. There's also a maintenance work memory setting inside Postgres, and this is for maintenance tasks. like building indexes and then the Postgres vacuum. The default is probably too small for any production. use case. In general, you can allocate 3 , 5% of your RAM and you can do a little math to figure out what's a good number for your

11:11

machine. And then you can also tell Postgres what your effective cache size is. This is a setting and you'll just tell it, you know, sort of as you've allocated that 25% to shared buffers, you'll tell it the other 75%. percent of that system memory. Um let's see here. And then if you're kind of wondering, um this seems really complicated. I don't want to have to do all this math myself. Um there is a developer out of the Ukraine who wrote a really nice tool called PGTune. And you can just put in like your the size of your database and kind of what you're doing with it, and he'll kind of give you some recommended settings.

11:58

I'm gonna move on and talk a little bit more about some other things other than memory happening inside your Postgres machine. So you'll probably have some kind of third-party tool. There's lots of different things out there, open source and not open source, that will monitor things like CPU and your system load. If you want to know what's going on in Postgres and what's using CPU, you can use the PGSTAT activity table. And if you need to stop transactions or or do something with those things, you can get the ID from there. When you're kind of peeking at CPU and thinking about things that might just take your database and run away with it,

12:44

one of those things are long-running statements. Postgres has a way to set a statement timeout. These are something that you can set by role. So it's a good idea to, you know, tell your app, you know, give your application role a statement timeout of a minute or two minutes and that way if you have some weird code update that starts generating a bunch of long running stuff it doesn't just completely take your whole database out And then you can, you know, obviously set a different timeout for yourself or for like a reporting application or something like that. Um, IOPS is another thing that is a good idea to keep an eye on when you're kind of peeking at your

13:29

your memory settings and kind of how everything is going on. IOPS is a great indicator of I don't have enough working memory. I don't have enough shared buffers. Postgres is using the disk all the time to read and write queries. So you should be seeing an IOP spike if you're, you know, loading data every night. Like you have an ETL tool that loads data every night. Obviously that's gonna use a lot of IOPs because Because it's writing a bunch of stuff to disk. But what you don't want to see is a huge, huge IOPS use all the time as you're using your application and writing queries, right? Because we want a lot of that data to be in those shared buffers. So this is kind of another sort of clue to get your memory kind of set up the way that you want it.

14:20

And another thing that is important to talk about with Postgres performance is table bloat. There's a lot of good like full hour-long talks on table bloat. And if you're kind of dealing with any of these really large tables that are frequently updated. There's some great information out there. So just at a super, super high level to not like spend a whole hour talking about table bloat The way that Postgres updates rows in a table is that it will update a row in a table, but because Postgres allows multiple versions to uh to be happening at a time, right? Because it allows many queries to be running in the database, it will keep that row around

15:09

if something else is using it. So if you update one field in a row, Postgres leaves the old sort of dead row around. And then later Postgres will come in and it will vacuum up that dead row. So what ends up happening is if you have a high um lots and lots of changes happening in the database, you will get lots and lots of dead rows. And that's called table bloat. And so that's sort of part of the the overall size of your disk, right? Because those are rows stored on disk. That are taking up disk space. There's a great query on the Postgres wiki to help you measure your table bloat, and then some of the managed tools will do this for you too.

15:54

If you're you know, under 50% table bloat, that's fine. You're doing a good job. Once you start getting above 50%, you want to keep an eye on it and and maybe figure out what's going on and and look at your application and and whether or not you kind of need to do some other things with vacuum. Um in general, Postgres will auto-vacuu and and most of that stuff is fine. So you in general you don't need to do a ton with it until you get to a A pretty active database. And the other thing just kind of for your underlying Postgres machine is that the Postgres version that you're on really matters in terms of performance. This is a a a little benchmark that the Enterprise DB company

16:42

did about Postgres versions. Um and every version of Postgres that comes out has a lot of things sort of under the hood, right? They're not, you know, features that you're gonna read about on Hacker News, but they're, you know, underlying things that make queries faster, that make the underlying kind of database parts work. Postgres 16 just came out a couple weeks ago, and that'll have its first 16. 1 come out probably early next year. Um Postgres 17 is open for commits and you should be running a version of Postgres 13, 14, 15 or newer. 12 is getting end of life and um

17:28

Twel and then versions older than that are are kind of out of support. And let's talk a little bit about connections, right? So the individual pieces that come into the database. Are going to affect how your database operates. So Postgres knows it has a max number of connections that it will allow into the database. So you can run out of connections. The key there is to make sure that if you have, you know, however your application server is connecting to the database, you haven't over-committed connections and you've, you know, got four web servers and they each have 50 connections to the database and you left your default at 100, that means half the time the connections are going to be refused.

18:20

So you need to keep an eye on your max connections and then look at what you're using. This does this defaults to 100. We have seen like recently I've seen a few people just say like, well, I'll just set it to like 3,000 and it'll be fine and then I'll never run out of connections. That's not a good idea because those connections use memory. Um and so if you have a connection that was made and it's idle, it's still taking up whatever you gave it as that working memory. Um and now your the connections the c that come in don't have any memory. So you can count the number of active connections that you have in Postgres. You can just query that from the database directly with this query. Um

19:06

you can ask Postgres what queries are using connections. Um and then it's a good idea once you get past um a couple hundred connections and you start getting into the point where you're sort of scaling your application, right? You've got several hundred users and and you've got things kind of happening to think about using um a connection pooler So if you let individual connections come into Postgres , those will open up a bunch of individual connections and then those will stay idle. And so you'll have a lot of

19:51

idle connections taking up memory and you'll have a lot of open things going on. And so a connection pooler will just sort of pool those resources right and it'll sort of manage keeping, you know, letting things have open connections to the database, but not necessarily using tons of database resources to keep those connections. going. And so PG Bouncer is like the sort of de facto best use case connection pooler for Postgres. There's several other um connection poolers out there. I haven't tested a bunch of them. Um we have a small Django application that we use where I work inside Crunchy that does some Some tools for our sales um

20:38

teams. And we, I don't know, I think we probably have like a couple hundred people that use it. Um and things were a little bit slow and we were kind of not not loving the performance for the end users and we put PG Bouncer in front of it and and it's been awesome. Um, so I don't I haven't seen any big downsides to adding that to even a pretty small application. Okay, so let's um sort of transition now. I've talked a lot about a bunch of the machine stuff um and talk a little bit more about Looking for issues with inside the application in the database, right? And so there's a couple different places that we're gonna talk about where you can find stuff.

21:27

Um You know, there's application logs. Like Django has a lot of tools and, you know, perf things and lots of ways to kind of get at what's happening inside Django. Postgres has a very robust logging system and we'll talk about that. And then we'll talk about a couple of the query tools. That will help you kind of get some data out of what's going on with your Postgres queries. And those are the PG statements and the explain plans. Um one thing I thought would be helpful if you're giving a talk about performance and queries is to just sort of set a bar about what is fast and slow. So I think This is obviously dependent on your application

22:13

and you know the standard engineering um answer is it depends. Um but In general, if you're writing a quick query to your database, it should give you information in a millisecond. If you're not getting data that fast. There's probably an issue. For your other sort of bigger queries where you're getting a set of data, you're going to be wanting to get things in the hundreds of milliseconds, right? So if you're reading explain plans or looking through your tools. you know, hundreds of milliseconds good, thousands of milliseconds generally bad. And if you're, you know, up above, if you're in the couple seconds range, up above five seconds That's going to be super obvious to your

22:59

end users, and that's a query that you either need to work on, you know, in the database or from the application perspective. You can turn on debugging inside the Django tool if you want to just watch what's happening as your app runs. And then it will kind of show you the SQL that's happening under the hood. The Django debug toolbar does something kind of similar, except it does it in a toolbar. I just kind of tested this in a in the like tutorial application um to see how that would work and it just kind of spits out the the raw SQL queries for you and gives you some information about how those work.

23:47

There's a Django Silk tool out there that I did a little bit of testing with that will give you even more query information. Um and it will go by page and and kind of collect stuff up for you. I found a nice tool on GitHub when I was oh I'm must not have a slide for that. If you're looking for like N plus one queries, I'll show you a slide for that later. And when you're sort of digging into Postgres, sort of beyond the Django layer, which you'll you'll probably need to do if you want to get really into any kind of query performance tuning. You'll definitely want to take a look at the Postgres

24:34

logs. So if you're self-hosting and you can get to your Postgres logs, you can start digging through them. Most people are going to be shipping logs to some kind of third party that's going to parse those and do something with them. There's a lot of APM tools out there. I know Scout is here and a lot of our clients have had great luck with that tool. There's an open source tool that come if you want to stay sort of in the open source Postgres only world called PG Badger that will let you ingest logs and it kind of does a whole bunch of stuff with your logs to kind of help you with it So you'll have to set up a few things in Postgres in terms of logging. And I think out of the box it'll just log

25:20

errors. So if you want Postgres to log, you know, table modifications or everything that's happening, you'll have to tell it to do that with the log statement. You can also tell Postgres how long of an operation to log. So in general, if you start logging everything that Postgres is doing. It's going to be really noisy and you're going to fill up whatever it is that you're storing logs on if you're paying somebody to store them or if you're storing them yourself. In general, you can pick a pick a speed at which you want to log something. One second is a good idea since you know anything under a second

26:07

It's probably, you know, running and then you're kind of picking stuff higher, but you can decide what kind of works for your application. You can log the temp files that are created in Postgres , and this is a really good idea. And you can decide the size of temp file to set. A lot of people just set this at what they set their working memory to be. That way you're logging um, you know, temp files that are a little bit bigger and um you're not sort of logging a ton of stuff. But this is a good indicator and and something you can keep an eye on if you're trying to work on your memory settings that we talked about earlier. And then you can tell Postgres what you want your logs to look like.

26:54

You can set up a little, you know, prefixer, and then that way they'll come out in the way that you want. And you can also um log um locks. Um so Postgres um will lock tables for certain operations There's a this is kind of like vacuum where table locking is one of those topics where there's tons of information about it out there and lots of good talks and blogs. Generally, if you're seeing table locks and something's waiting on a lock, it means the thing that happened before is locking it. But if you're logging in table locks and you're trying to figure out what's happening,

27:39

this is a good thing to know so that you know This query wasn't slow because the query is slow. It's slow because it's waiting on a lock and something happened before it. Um and another tool that Postgres has for sort of doing like query, I call it query logging, but it's sort of a A way that Postgres keeps track of what's happening with your queries is the PG Stat statements extension. So this is part of Postgres, it'll come with Postgres, but you have to load it in and say that you want to use it because it does use a little bit of of memory. And then it collects statistics on

28:26

all the queries that come into the database until you reset it, which you You can do with a reset command. Um and PG stat statements like in the sort of application developer world is definitely like the the tool to make sure you're using. So if you If you walk away from this talk with one thing that you're not doing today, turn on PG stat statements and start seeing what's going on under the hood with your queries So you can just once you set up PG statements, you can just look at Postgres and you can just query Postgres for information about your queries. So you can just say like, okay, I want to know, you know, my ten longest running queries, just give those to me

29:13

and and Postgres will spit that out for you. You can look at what queries run the most often. So if you're sort of picking away at what to work on performance-wise and you think, okay, I'm gonna, you know, I'm gonna pick. a query to work on. I want to make my really slowest queries work. And then anything that my users and my application use all the time. I want to work on that too. That's a good place to start. You can also look at what queries use CPU and sort of how what they're doing with the actual machine if that's a concern to you and that's something you want to work on.

30:01

And then once you start getting into PG stat statements and seeing what is happening, what queries are coming in. You're probably going to want to know more about that specific query. And so the way that Postgres does this is with what's called in the explain plans. And so basically explain is just something that you run inside Postgres. You run an explain plan and then you put the query after it. And then Postgres gives you a bunch of information about the query itself. And explain analyze will do the same thing and it'll give you the execution time, so how long it take to run the query. So under the hood, you know, Postgres does tons of work

30:47

when your query from Django application comes in, Postgres does all of this stuff to kind of parse it out, generate the fastest path, make a query plan, execute that, and then return it to you. Um so if you change the data. that you're working with, you're gonna get a different explain plan. So one of the sort of little tricks if you're working on trying to make your queries faster. You'll have to do this on a copy of your actual production data. You can't really do this with a task data set, right? You can't do it with seed data and you can't do it on a different version of Postgres. Um so you'll have to kind of keep things all the same if you want to do some some explain

31:32

plans and kind of experiment with things. You can also run the explain tool with buffers. And so that will tell you things like what kind of scan it did, how long it took. Here's like a little breakdown of kind of what an explain plan looks like. And so it'll tell you, okay, I did this kind of scan on this table, lots of different information about you know loops and what's going on. And then If it used shared buffers to get that information , and then if it had to read from disk, it'll tell you that. And then it'll also tell you how much time it spent reading I. O. So If you're kind of starting to get into

32:19

putting this all together, right? Trying to make sure that your queries are running in shared buffers, minimize the amount of I. O. Um, this is kind of how you're gonna put that together, right? You're gonna get the queries out of PG stat statements, you're gonna run some explained plans and sort of figure out what's happening under the hood When you're looking at the scan types in explain , you'll want to look for, and especially if later you get into some kind of performance things like building indexes. You can check the explain plan to make sure Postgres is actually using your index. And then you can also see like if Postgres is using joins and what type of join it's using in that query. AutoExplain

33:06

will um log explain plans for you. So it'll send kind of those plans that I showed you into your logs and you can set a minimum duration so you can auto-explain every query that's longer than two seconds. Um, our support team told me to not even put the slide in the presentation because most people um just run themselves out of um log space with auto explain. It's super noisy and um it uses a ton of memory because it it has to calculate so much to get you that explain plan. So if you do it, do it carefully and Maybe on a test database or something.

33:52

All right. So I've kind of talked a little bit about how to get all of the the query stuff, how to kind of find some of the problem areas, how to sort of start taking a look at what's inside the queries that are problem areas And I kind of want to give you just a few ideas about things that you can do to kind of improve the queries and improve some of the operations inside those. database commands. So the first one is indexing. I'm I'm sure a lot of you have heard of this, but if you haven't started adding any indexes , They're super helpful in terms of performance. Postgres has a lot of different kinds of indexes where

34:39

it'll group information together, you know, based on different ideas. and kinds of data that you have. So if you have just, you know, text data, stuff like that, you'll be fine with a B tree index. If you've got lots of date range and time series data, you'll want a brand index. There's some nice spatial data indexes like the GIST and the new SPGIST. And then if you're using a lot of the newer Postgres and the JSON functionality, you can use a gen index for that. And then for sort of application developers, multi-column indexes can be super, super

35:24

helpful. You can kind of combine things that you query together all the time, you know, like names or you know, things that are obviously always coming together. Um, if you've just dipped your toe into indexing. Multi-column indexing is definitely the next step in terms of getting things moving really fast. And partial indexing is also really helpful, especially since you know if you're working on um web development data you probably have a lot of data fields that are null or don't have data that's relevant and you're never gonna pull them out in a query If you write a partial index, that helps Postgres know, you know, just skip over that data. You don't need to

36:10

have that as readily available. I did some little kind of messing around with indexing performance just to kind of show you. I have a little table of weather data, and if I run, you'll see it runs that sequence scan. And I just pulled that in you know 30 or so milliseconds. If I create just a plane B tree index on that weather data, run the exact same query, it'll run a an index scan and it'll return that in three milliseconds. So just your simple B tree index is going to be pulling data so much faster from the underlying data set. Um, indexing can

36:55

be um kind of complicated, especially if you're working with a production database. And you've got a really big data set, creating an index takes a lot of time. The indexes themselves are stored on disk , so they take up disk space. And then every write that comes in obviously has to be written to the index. So if you have a really, really write heavy you know, database you want to kind of be careful with with what you're doing with indexes. Um Postgres has an extension for hypothetical indexing. um which some of our clients use, especially if the creation of an index is really time consuming and it's gonna take, you know, like it's gonna take a couple hours to build this index

37:41

You can create a hypothetical index and then you can ask Postgres for that explain plan on that hypothetical index and Postgres will give you an explain plan on an index that doesn't actually exist. And this is kind of a cool way to do it. Really helpful if you have big data sets and you know kind of you know an index is going to take a long time to run. And then the sort of other thing about indexes is that you don't want too many indexes. So like I talked about, um indexes have to be stored on disk and every write needs needs to get into the index as well so that it can be pulled out when you query it again. You can just ask Postgres with a query

38:26

what indexes are not being used. And Postgres will tell you your unused indexes and you can drop those and kind of decide what you want to do from there. Another sort of idea to takeaway too is just sort of your data modeling. I don't obviously I I can't give you an hour-long talk on data modeling. But a couple tips that are helpful is just keeping your tables small. Don't create tables where you're updating tons of data at a time. Um so, you know, an example of this that that our CTO likes to talk about is um, you know, if you have a bunch of user contact data, right?

39:12

And you're you've got their address and their email address and tons of information about this person. Um and but you also keep a a timestamp every time they log into your application, which is like 45 times a day. Don't store that timestamp in the same place that you store their contact information because like we were talking about with, you know, the table bloat, every Every time you update data, Postgres keeps a dead row around. Um, and then just, you know, you're kind of adding to a lot of amplification with with different things there. So if you can kind of keep your really, really, really frequently updated data separate from other data, it's a lot better for the underlying performance of the data itself.

39:57

And another thing to sort of keep in the back of your mind are N plus one queries. And this is like primarily something that you'll get straight from the ORM when you're writing the models to kind of decide what you want and then you, you know, get a batch of data and then you want to print something related to it So you'll kind of essentially what happens with an N plus one query is that you'll write one query and then you'll ask for something else and that generates, you know, hundreds or thousands of individual queries instead of writing it as a batch of queries. There's a nice project on GitHub that I found called Django Perfinu.

40:43

where um he has a little sample like template site that you can just spin up a little Django project um and then you start clicking and you generate um n plus one queries and so it'll kind of show you that in in the Django Silk if you want to sort of see what an N plus one query looks like without without finding it in your own in your own application. So what it'll look like is you'll have, you know, that query, that initial query at the bottom, and then you'll have, you know, hundreds of individual, the exact same query running right after it. Some of the APM tools also have a lot of this N plus one stuff built into it now. I know that the Scout tool, I've seen people use that. And then there's probably other ways that people are kind of solving this problem.

41:31

And I think in the the Django world, um, you know, you've got some tools inside your ORM to kind of make sure that you're selecting the related data or or pre-fetching it first so that you're send, you know, you're querying the whole batch of data from Postgres instead of querying one piece at a time. And those are really well documented in the Django docs. Okay, so I'm gonna just leave you with my final tips for Postgres performance. Stop runaway queries and set a statement timeout, especially for things that shouldn't have long running queries Set up PG stat statements and know what your slowest queries are and if they're important, try to fix them or add indexes or do things to make them faster.

42:22

Add indexes for your most frequent queries. Check your cash hit ratio Tune your memory and add memory if you need to. Keep an eye on that. Make sure that your connections have enough memory and make sure that you have enough connections. And if you're kind of scaling up, take a look at that PG bouncer tool. Stay on top of your Postgres versions. And that is it. Are there any questions? There you come

Questions this talk answers

How much memory should I allocate to PostgreSQL shared_buffers?

A useful starting point is about a quarter of the machine’s memory for shared_buffers. On an 8 GB production machine, the default 128 MB is generally too small.

Discussed at 6:28

What should my PostgreSQL cache hit ratio be?

PostgreSQL’s cache hit ratio shows how often requested data was already in cache rather than read from disk. You generally want it in the high 90s, while remembering that recent restarts or newly loaded data can temporarily lower it.

Discussed at 8:03

How do I tell whether PostgreSQL needs more memory?

High, persistent I/O can indicate that PostgreSQL does not have enough shared buffers or working memory and is reading from or writing to disk too often. Temporary files generated by queries are another useful clue that work memory is insufficient.

Discussed at 13:29

What is PostgreSQL table bloat, and when should I worry about it?

When PostgreSQL updates rows, it can retain old row versions until vacuum removes them; those dead rows accumulate as table bloat. Less than 50% bloat is generally fine, but above 50% merits investigation, especially for frequently updated large tables.

Discussed at 14:09

How can I prevent PostgreSQL from running out of connections?

Check max_connections against the number of connections your application servers can open, rather than simply setting it to a very high value because each connection consumes memory. As an application grows, a connection pooler such as PgBouncer can reuse database connections and reduce idle connection overhead.

Discussed at 17:28

What is the best way to find my slowest PostgreSQL queries?

Enable the pg_stat_statements extension and query it for the longest-running, most frequently executed, or highest-CPU queries. This gives you a practical list of queries to investigate and optimize.

Discussed at 28:26

How do I use PostgreSQL EXPLAIN to diagnose a slow query?

Run EXPLAIN or EXPLAIN ANALYZE with the query to see its plan and execution time, and use EXPLAIN with buffers to inspect scans, shared-buffer usage, disk reads, and I/O time. Test with production-like data and the same PostgreSQL version because the plan depends on the data and environment.

Discussed at 30:01

Which PostgreSQL indexes can make queries faster?

A basic B-tree index is suitable for many text and ordinary lookup queries, while BRIN can help with large date or time-series data, GiST/SP-GiST with spatial data, and GIN with JSON. Multi-column and partial indexes can also help when queries repeatedly use the same column combinations or ignore many null or irrelevant rows.

Discussed at 34:39

What causes N+1 queries in Django, and how do I fix them?

An N+1 problem occurs when one initial query is followed by hundreds or thousands of individual queries for related data. Use Django’s related-object tools, such as selecting or prefetching related data, so the related records are fetched in batches instead.

Discussed at 40:57

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos by Elizabeth Garrett Christensen

More videos from DjangoCon US