tom/forkjo
tom/forkjo
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Merge pull request 'tests: Improve in-repo documentation' (#5235) from fnetx/testing-docs into forgejo

Browse source 
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5235
Reviewed-by: thefox <thefox@noreply.codeberg.org>
This commit is contained in:
Otto 2024-09-09 12:48:14 +00:00
commit b2ea43b25b
5 changed files with 190 additions and 137 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.forgejo/workflows/e2e.yml
View file

@ -8,6 +8,9 @@ on:
- .forgejo/workflows/e2e.yml
- tests/e2e/**
- web_src/js/**
- web_src/css/form.css
- templates/webhook/shared-settings.tmpl
- templates/org/team/new.tmpl
jobs:
test-e2e:

185
tests/e2e/README.md
View file

@ -1,42 +1,93 @@
# End to end tests
E2e tests largely follow the same syntax as [integration tests](../integration).
Whereas integration tests are intended to mock and stress the back-end, server-side code, e2e tests the interface between front-end and back-end, as well as visual regressions with both assertions and visual comparisons.
They can be run with make commands for the appropriate backends, namely:
```shell
make test-sqlite
make test-pgsql
make test-mysql
```
Thank you for your effort to provide good software tests for Forgejo.
Please also read the general testing instructions in the
[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/)
and make sure to also check the
[Playwright documentation](https://playwright.dev/docs/intro)
for further information.
This file is meant to provide specific information for the integration tests
as well as some tips and tricks you should know.
Feel free to extend this file with more instructions if you feel like you have something to share!
## How to run the tests?
Before running any tests, please ensure you perform a clean frontend build:
Make sure to perform a clean front-end build before running tests:
```
make clean frontend
```
## Install playwright system dependencies
Whenever you modify frontend code (i.e. JavaScript and CSS files),
you need to create a new frontend build.
For tests that require interactive Git repos,
you also need to ensure a Forgejo binary is ready to be used by Git hooks.
For this, you additionally need to run
~~~
make TAGS="sqlite sqlite_unlock_notify" backend
~~~
### Install dependencies
Browsertesting is performed by playwright.
You need certain system libraries and playwright will download required browsers.
Playwright takes care of this when you run:
```
npx playwright install-deps
```
## Interactive testing
> **Note**
> On some operating systems, the installation of missing libraries can complicate testing certain browsers.
> It is often not necessary to test with all browsers locally.
> Choosing either Firefox or Chromium is fine.
You can make use of Playwright's integrated UI mode to run individual tests,
get feedback and visually trace what your browser is doing.
To do so, launch the debugserver using:
### Run all tests
If you want to run the full test suite, you can use
```
make test-e2e-sqlite
```
### Interactive testing
We recommend that you use interactive testing for the development.
After you performed the required builds,
you should use one shell to start the debugserver (and leave it running):
```
make test-e2e-debugserver
```
Then launch the Playwright UI:
It allows you to explore the test data in your local browser,
and playwright to perform tests on it.
> **Note**
> The modifications persist while the debugserver is running.
> If you modified things, it might be useful to restart it to get back to a fresh state.
> While writing playwright tests, you either
> need to ensure they are resilient against repeated runs
> (e.g. when only creating new content),
> or that they restore the initial state for the next browser run.
#### With the playwright UI:
Playwright ships with an integrated UI mode which allows you to
run individual tests and to debug them by seeing detailed traces of what playwright does.
Launch it with:
```
npx playwright test --ui
```
You can also run individual tests while the debugserver using:
#### Running individual tests
```
npx playwright test actions.test.e2e.js:9
@ -46,18 +97,29 @@ First, specify the complete test filename,
and after the colon you can put the linenumber where the test is defined.
## Run all tests via local act_runner
#### With VSCodium or VSCode
To debug a test, you can also use "Playwright Test" for
[VScodium](https://open-vsx.org/extension/ms-playwright/playwright)
or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
### Run all tests via local act_runner
If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/),
you can use it to run the test jobs:
```
act_runner exec -W ./.github/workflows/pull-e2e-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
forgejo-runner exec -W .forgejo/workflows/e2e.yml --event=pull_request
```
## Run sqlite e2e tests
Start tests
```
make test-e2e-sqlite
```
### Run e2e tests with another database
This approach is not currently used,
neither in the CI/CD nor by core contributors on their lcoal machines.
It is still documented for the sake of completeness:
You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want.
## Run MySQL e2e tests
Setup a MySQL database inside docker
```
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
@ -68,7 +130,6 @@ Start tests based on the database container
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql
```
## Run pgsql e2e tests
Setup a pgsql database inside docker
```
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
@ -78,11 +139,12 @@ Start tests based on the database container
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql
```
## Running individual tests
### Running individual tests
Example command to run `example.test.e2e.js` test file:
_Note: unlike integration tests, this filtering is at the file level, not function_
> **Note**
> Unlike integration tests, this filtering is at the file level, not function
For SQLite:
@ -90,13 +152,11 @@ For SQLite:
make test-e2e-sqlite#example
```
For PostgreSQL databases(replace `mysql` to `pgsql`):
### Visual testing
```
TEST_MYSQL_HOST=localhost:1433 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=sa TEST_MYSQL_PASSWORD=MwantsaSecurePassword1 make test-e2e-mysql#example
```
## Visual testing
> **Warning**
> This is not currently used by most Forgejo contributors.
> Your help to improve the situation and allow for visual testing is appreciated.
Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following:
- Check out `main`, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1` to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes.
@ -106,14 +166,55 @@ VISUAL_TEST=1 will create screenshots in tests/e2e/test-snapshots. The test will
ACCEPT_VISUAL=1 will overwrite the snapshot images with new images.
## With VSCodium or VSCode
To debug a test, you can use "Playwright Test" for
[VScodium](https://open-vsx.org/extension/ms-playwright/playwright)
or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
Before doing that you will need to manually start a Forgejo instance and populate it
with data from `models/fixtures` by running:
## Tips and tricks
```sh
make TAGS='sqlite sqlite_unlock_notify' 'test-e2e-debugserver'
```
If you know noteworthy tests that can act as an inspiration for new tests,
please add some details here.
### Run tests very selectively
Browser testing can take some time.
If you want to iterate fast,
save your time and only run very selected tests.
Use only one browser.
### Skip Safari if it doesn't work
Many contributors have issues getting Safari (webkit)
and especially Safari Mobile to work.
At the top of your test function, you can use:
~~~javascript
test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.');
~~~
### Don't forget the formatting.
When writing tests without modifying other frontend code,
it is easy to forget that the JavaScript test files also need formatting.
Run `make lint-frontend-fix`.
### Define new repos
Take a look at `declare_repos_test.go` to see how to add your repositories.
Feel free to improve the logic used there if you need more advanced functionality
(it is a simplified version of the code used in the integration tests).
### Accessibility testing
If you can, perform automated accessibility testing using
[AxeCore](https://github.com/dequelabs/axe-core-npm/blob/develop/packages/playwright/README.md).
Take a look at `shared/forms.js` and some other places for inspiration.
### List related files coverage
If you think your playwright tests covers an important aspect of some template, CSS or backend files,
consider adding the paths to `.forgejo/workflows/e2e.yml` in the path filter.
It ensures that future modifications to this file will be tested as well.
Currently, we do not run the e2e tests on all changes.

10
tests/e2e/declare_repos_test.go
View file

@ -21,15 +21,19 @@ import (
"github.com/stretchr/testify/require"
)
// first entry represents filename
// the following entries define the full file content over time
type FileChanges [][]string
// put your Git repo declarations in here
// feel free to amend the helper function below or use the raw variant directly
func DeclareGitRepos(t *testing.T) func() {
var cleanupFunctions []func()
cleanupFunctions = append(cleanupFunctions, newRepo(t, 2, "diff-test", FileChanges{
cleanupFunctions := []func(){
newRepo(t, 2, "diff-test", FileChanges{
{"testfile", "hello", "hallo", "hola", "native", "ubuntu-latest", "- runs-on: ubuntu-latest", "- runs-on: debian-latest"},
}))
}),
// add your repo declarations here
}
return func() {
for _, cleanup := range cleanupFunctions {

50
tests/integration/README.md
View file

@ -1,5 +1,23 @@
# Integration tests
Thank you for your effort to provide good software tests for Forgejo.
Please also read the general testing instructions in the
[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/).
This file is meant to provide specific information for the integration tests
as well as some tips and tricks you should know.
Feel free to extend this file with more instructions if you feel like you have something to share!
## How to run the tests?
Before running any tests, please ensure you perform a clean build:
```
make clean build
```
Integration tests can be run with make commands for the
appropriate backends, namely:
```shell
@ -8,40 +26,39 @@ make test-pgsql
make test-mysql
```
Make sure to perform a clean build before running tests:
```
make clean build
```
## Run tests via local act_runner
### Run tests via local forgejo runner
### Run all jobs
If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/),
you can use it to run the test jobs:
#### Run all jobs
```
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request
```
Warning: This file defines many jobs, so it will be resource-intensive and therefore not recommended.
### Run single job
#### Run single job
```SHELL
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request -j <job_name>
```
You can list all job names via:
```SHELL
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request -l
```
## Run sqlite integration tests
### Run sqlite integration tests
Start tests
```
make test-sqlite
```
## Run MySQL integration tests
### Run MySQL integration tests
Setup a MySQL database inside docker
```
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
@ -52,7 +69,7 @@ Start tests based on the database container
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
```
## Run pgsql integration tests
### Run pgsql integration tests
Setup a pgsql database inside docker
```
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
@ -62,7 +79,7 @@ Start tests based on the database container
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
```
## Running individual tests
### Running individual tests
Example command to run GPG test:
@ -99,3 +116,8 @@ SLOW_FLUSH = 5S ; 5s is the default value
```bash
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite
```
## Tips and tricks
If you know noteworthy tests that can act as an inspiration for new tests,
please add some details here.

77
tests/integration/README_ZH.md
View file

@ -1,77 +0,0 @@
# 关于集成测试
使用如下 make 命令可以运行指定的集成测试：
```shell
make test-mysql
make test-pgsql
make test-sqlite
```
在执行集成测试命令前请确保清理了之前的构建环境，清理命令如下：
```
make clean build
```
## 如何在本地 act_runner 上运行测试
### 运行所有任务
```
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
```
警告:由于在此文件中定义了许多任务，因此此操作将花费太多的CPU和内存来运行。所以不建议这样做。
### 运行单个任务
```SHELL
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
```
您可以通过以下方式列出所有任务名称:
```SHELL
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
```
## 如何使用 sqlite 数据库进行集成测试
使用该命令执行集成测试
```
make test-sqlite
```
## 如何使用 mysql 数据库进行集成测试
首先在docker容器里部署一个 mysql 数据库
```
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:8 #(just ctrl-c to stop db and clean the container)
```
之后便可以基于这个数据库进行集成测试
```
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
```
## 如何使用 pgsql 数据库进行集成测试
同上，首先在 docker 容器里部署一个 pgsql 数据库
```
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:14 #(just ctrl-c to stop db and clean the container)
```
之后便可以基于这个数据库进行集成测试
```
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
```
## 如何进行自定义的集成测试
下面的示例展示了怎样在集成测试中只进行 GPG 测试：
sqlite 数据库:
```
make test-sqlite#GPG
```
其它数据库 (用 PGSQL 取代 MYSQL）:
```
TEST_MYSQL_HOST=localhost:1433 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=sa TEST_MYSQL_PASSWORD=MwantsaSecurePassword1 make test-mysql#GPG
```