http://www.louisemeta.com/blog/ Recent content in Blogs on Louise Grandjonc, Python/SQL developer but also a human Hugo -- gohugo.io en-US Mon, 01 Jul 2019 10:20:22 -0700 http://www.louisemeta.com/blog/indexes-gin-algorithms/ Mon, 01 Jul 2019 10:20:22 -0700 http://www.louisemeta.com/blog/indexes-gin-algorithms/ Introduction In the previous article, I introduced the internal data structure of a postgres GIN index. Now it’s time to talk about searching and maintaining them. Searching in a GIN index When postgres uses the GIN index, you’ll notice in the EXPLAIN that it’s always doing a bitmap heap scan. It means that the search algorithm returns a map of tids (pointers to rows) in physical memory order. Considering the way the data is stored inside the posting lists and trees, it makes a lot of sense to use this method. http://www.louisemeta.com/blog/indexes-gin/ Mon, 01 Jul 2019 10:20:21 -0700 http://www.louisemeta.com/blog/indexes-gin/ Introduction GIN (Generalized Inverted Index) indexes are often used to index arrays, jsonb, and tsvector (for fulltext search) columns. When it comes to array, they are, for example, used to verify if an array contains an other array or elements (the <@ operator). In the documentation you can see the full list of operators. But the question that I want to answer in this article is: “why should we use GIN indexes for this data types and operators ? http://www.louisemeta.com/blog/indexes-btree-algorithms/ Thu, 06 Sep 2018 10:20:21 -0700 http://www.louisemeta.com/blog/indexes-btree-algorithms/ Introduction In the previous article, I introduced the internal data structure of a postgres BTree index. Now it’s time to talk about searching and maintaining them. I think that understanding this can help know when using a BTree makes sense and when it wouldn’t be very efficient. Searching in a BTree Initializing scankeys In order to search in a BTree, postgres first uses the query scan to define scankeys. If possible, redundants keys in your query are eliminated to keep only the tightest bounds. http://www.louisemeta.com/blog/indexes-btree/ Wed, 05 Sep 2018 10:20:21 -0700 http://www.louisemeta.com/blog/indexes-btree/ Introduction BTree is the index type used by default on CREATE INDEX. As I said in the introduction to indexes article, BTrees are also created automatically for primary key and unique constraints. Before I start going into the internal structure of BTree, I’d like to talk to you about pageinspect_inspector, a python library to generate a graphical representation of what’s going on inside PostgreSQL’s BTrees. You pass your index to it on a command line and it generates an HTML allowing you to explore pages and items of your index. http://www.louisemeta.com/blog/intro-to-indexes/ Tue, 04 Sep 2018 10:20:21 -0700 http://www.louisemeta.com/blog/intro-to-indexes/ Introduction When I understand precisely what’s happening in my database, it helps me make smarter choices. It’s why previous articles focused on EXPLAIN and the algorithms used. Because to me, it was the best way to understand why things were not working as expected. And I believe that when it comes to indexes, it’s the same. Of course you could use your ORM to add an index on a column you picked kind of randomly, but there’s a smarter way… First because ORMs create BTrees and if you’re using PostgreSQL there are other types that can be very handy. http://www.louisemeta.com/blog/indexes-gist/ Fri, 29 Jun 2018 10:20:21 -0700 http://www.louisemeta.com/blog/indexes-gist/ Introduction The GIST index is a balanced tree, but that’s basically the only thing that it has in common with a BTree. One of its particularity is that it can be extended to any type of data as long as a few functions implemented. I will mention these function in this article, and if you want to go further, you can go read the GiST indexing Project website But for now we are going to focus on a few usecases, and not list all the extensions that great people have made for GiST :) http://www.louisemeta.com/blog/explain-4/ Tue, 27 Mar 2018 14:39:21 -0700 http://www.louisemeta.com/blog/explain-4/ Introduction Here we are, the last article of this serie. This one will cover the algorithms used to sort rows, and why OFFSET should be banned from your applications. Previously on the EXPLAIN show You learned that it all starts with logs You met the amazing pg_stat_statements You got to know a bit more about the life of a query, how EXPLAIN can be used and what are costs and actual time in this episode You understood (I hope! http://www.louisemeta.com/blog/explain-3/ Mon, 26 Mar 2018 14:39:21 -0700 http://www.louisemeta.com/blog/explain-3/ Introduction If you haven’t read the previous articles, I really encourage to start from the beginning, EXPLAIN can seem a bit complicated if you don’t have the basics. So don’t hesitate to take a look at the first and second article on the subject. Moreover reading them will give you the data model and a bit more information on the examples used here ! But if you have read them, welcome back ! http://www.louisemeta.com/blog/explain-2/ Sun, 25 Mar 2018 19:39:21 -0700 http://www.louisemeta.com/blog/explain-2/ Introduction If you don’t know EXPLAIN and are starting with this article, I really encourage you to start from the beginning, to understand better what a query plan is and how to use EXPLAIN. The previous article covers that. In this article, I’m going to try to clarify the different types of scans used to filter tables and retrieve the exptected rows. I’m here taking the example of a letter delivery system. http://www.louisemeta.com/blog/explain/ Sat, 24 Mar 2018 16:39:21 -0700 http://www.louisemeta.com/blog/explain/ Introduction If you didn’t read the articles on logs and pg_stat_statements this article is based on a talk that I did at the pgdayParis and here are the slides. You can watch the Djangocon Europe 2017 version, it’s more django developer oriented, which can be interesting if you’re using django’s ORM. And here is the video from the PGdayParis. If you want to test the queries, here is the github project for the talk. http://www.louisemeta.com/blog/pg-stat-statements/ Fri, 23 Mar 2018 17:39:21 -0700 http://www.louisemeta.com/blog/pg-stat-statements/ Introduction As a reminder, if you ended up on this page by looking up on bing pg_stat_statements, first I’m impressed, you get results with bing ? But also, this article is based on a talk that I did at the pgdayParis 2018 and here are the slides What is pg_stat_statements ? pg_stat_statement is a postgreSQL extension, once you enable it, it tracks statistics on the queries executed by a server. http://www.louisemeta.com/blog/developers-and-logs/ Wed, 21 Mar 2018 18:39:21 -0700 http://www.louisemeta.com/blog/developers-and-logs/ Introduction A few years back, when I was an intern, the CTO asked me if I checked how many queries were executed on the page, turns out there were over a hundred… I realized that I was missing some basics : Like a lot of web developers I didn’t think of looking into my logs to know what my ORM was doing Once I knew, I didn’t understand why some queries were slow After that I became a bit obsessed with performances and that’s the story of why I submitted a talk at the django con 2017, and then at the pgdayParis 2018. http://www.louisemeta.com/blog/indexes/ Mon, 01 Jan 0001 00:00:00 +0000 http://www.louisemeta.com/blog/indexes/ What is an index ? An index is a B-Tree. Each leave of the tree contains a tuple (value, pointer to the row). Multi column indexes Different types of indexes and their use case http://www.louisemeta.com/blog/select-all/ Mon, 01 Jan 0001 00:00:00 +0000 http://www.louisemeta.com/blog/select-all/