tom/forkjo
tom/forkjo
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
forkjo/models
History
Chl 544cbc6f01 Optimization of labels handling in issue_search (#4228) 
This PR optimizes the SQL query and de-duplicate the labels' ids when generating the query string, on the issue page.

<hr/>

### Background

Some time ago, BingBot and some other crawlers have been putting my instance on its knees with requests containing a lot of label ids, like this one :

```
[07/Aug/2023:11:28:37 +0200] "GET /Dolibarr/sendrecurringinvoicebymail/issues?q=&type=all&sort=&state=closed&labels=1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c2%2c10%2c2%2c1%2c1%2c10%2c10%2c7%2c6%2c10%2c10%2c3%2c2%2c1%2c5%2c10%2c1%2c6%2c2%2c7%2c3%2c7%2c6%2c10%2c1%2c10%2c1%2c1%2c7%2c7%2c1%2c1%2c1%2c1%2c10%2c10%2c1%2c2%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c2%2c1%2c12%2c6%2c6%2c10&milestone=0&project=-1&poster=0 HTTP/1.1" 499 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
```

Since each of the label ids implies a join, it grows exponentially expensive for the database engine (at least on PostgreSQL but SQLite suffers a little too).

Thus, this PR proposes two enhancements:

* rewrite the database query to use only one squashed condition,
* deduplicate the label ids when generating the URL.

### Performance comparison

Here are some timings on Postgresql-backed, Forgejo 7.0.4 instances :
```sh
$ time curl -s -o /dev/null "http://localhost:3000/toto/tata/issues?q=&type=all&sort=&labels=19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25&state=open&milestone=0&project=0&assignee=0&poster=0"

real    0m10,491s
user    0m0,017s
sys     0m0,008s
```
...and with the patch:
```sh
$ time curl -s -o /dev/null "http://localhost:3000/toto/tata/issues?q=&type=all&sort=&labels=19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25&state=open&milestone=0&project=0&assignee=0&poster=0"

real    0m0,094s
user    0m0,012s
sys     0m0,013s
```

### Annex

This issue was originally proposed to [Gitea](https://github.com/go-gitea/gitea/pull/26460) but didn't get much attention, and I switched to Forgejo in the meantime :)

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4228
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Chl <chl@xlii.si>
Co-committed-by: Chl <chl@xlii.si>
2024-06-28 05:11:57 +00:00
..
actions Drop IDOrderDesc for listing Actions task and always order by id DESC (#31150) 2024-06-02 16:26:54 +02:00
activities Rename project board -> column to make the UI less confusing (#30170) 2024-06-02 09:42:39 +02:00
admin Next round of db.DefaultContext refactor (#27089) 2023-09-16 14:39:12 +00:00
asymkey Enable more revive linter rules (#30608) 2024-04-28 15:39:00 +02:00
auth Add codespell support and fix a good number of typos with its help (#3270) 2024-05-09 13:49:37 +00:00
avatars Replace 10 more gt- classes with tw- (#29945) 2024-03-26 19:04:27 +01:00
db [Refactor] Unify repo search order by logic (#30876) 2024-06-16 13:42:58 +02:00
dbfs Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
fixtures [FEAT] Adds x-mode-only anchor styles to display images based if the theme is light or dark mode. (#3985) 2024-06-09 10:50:53 +00:00
forgefed enhance test & fix reviews 2024-05-14 08:24:31 +02:00
forgejo/semver [SEMVER] store SemVer in ForgejoSemVer after a database upgrade 2024-02-05 14:44:33 +01:00
forgejo_migrations fix review 2024-05-29 18:31:06 +02:00
git Add tag protection via rest api #17862 (#31295) 2024-06-16 13:42:59 +02:00
issues Optimization of labels handling in issue_search (#4228) 2024-06-28 05:11:57 +00:00
migrations Fix the possible migration failure on 286 with postgres 16 (#31209) 2024-06-02 16:26:54 +02:00
organization Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-05-05 08:38:16 +01:00
packages Fix package list performance (#30520) 2024-04-21 16:28:16 +02:00
perm Users with read permission of pull requests can be assigned too (#27263) 2024-02-26 22:30:27 +01:00
project Rename project board -> column to make the UI less confusing (#30170) 2024-06-02 09:42:39 +02:00
pull [GITEA] GetScheduledMergeByPullID may involve a system user 2024-02-05 16:09:41 +01:00
repo Rename repo_model.SearchOrderByMap to repo_model.OrderByMap (#31359) 2024-06-16 13:42:59 +02:00
secret Make runs-on support variable expression (#29468) 2024-03-11 23:36:59 +07:00
shared/types Refactor locale&string&template related code (#29165) 2024-02-16 15:20:52 +01:00
system Refactor deletion (#28610) 2023-12-25 21:25:29 +01:00
unit Rename project board -> column to make the UI less confusing (#30170) 2024-06-02 09:42:39 +02:00
unittest [CHORE] Remove Microsoft SQL Server Support 2024-04-05 23:37:36 +02:00
user Performance improvements for pull request list API (#30490) 2024-06-02 16:26:54 +02:00
webhook webhook: sourcehut_builds: ask for access_token instead of authorization_header 2024-04-16 08:59:55 +02:00
error.go Add merge style fast-forward-only (#28954) 2024-02-14 17:19:19 +01:00
fixture_generation.go Replace more db.DefaultContext (#27628) 2023-10-15 17:46:06 +02:00
fixture_test.go Replace more db.DefaultContext (#27628) 2023-10-15 17:46:06 +02:00
main_test.go make writing main test easier (#27270) 2023-09-28 01:38:53 +00:00
org.go Refactor deletion (#28610) 2023-12-25 21:25:29 +01:00
org_team.go Remove GetByBean method because sometimes it's danger when query condition parameter is zero and also introduce new generic methods (#28220) 2023-12-07 15:27:36 +08:00
org_team_test.go Reduce usage of db.DefaultContext (#27073) 2023-09-14 17:09:32 +00:00
org_test.go Replace more db.DefaultContext (#27628) 2023-10-15 17:46:06 +02:00
repo.go Refactor deletion (#28610) 2023-12-25 21:25:29 +01:00
repo_test.go Penultimate round of db.DefaultContext refactor (#27414) 2023-10-11 04:24:07 +00:00
repo_transfer.go chore: use errors.New to replace fmt.Errorf with no parameters will much better (#30621) 2024-04-28 15:39:00 +02:00
repo_transfer_test.go [MODERATION] User blocking 2024-02-05 15:56:45 +01:00