###
### Automated Github PR tester
### (GSOC 2021?)
###
### 2021-02-06

Summary: 
Getting a tinderbox to test our Github pull requests, that the contributions 
build. Print or link all build output to Github PR. This'd work on-demand, 
either by manually launching it from a shell, or even better, with Github 
hooks. Using Github hooks I could imagine there to be "initiate build" label
which then launches the process on a tinderbox, and update it to "build passed"
or "build failed" when done. These tags are browsable in Github.

(Although I'd like a shell/vnc access for runtime testing)


Rationale:
Most of the time when testing contributions go to build-testing them. It's an 
important part that can't be skipped. We are constantly drifting behind on 
contributions offered -> contributions merged meter. It's also very 
demotivating for the contributor and the reviewer when a PR that has been 
sitting for weeks is unbuildable.
(you could argue it's on the contributor's side, but we can't really require 
 all of our contributors to have empty stage3 test chroots)

This'd surely allow the review to focus more on quality, and for us to answer
faster on contributions towards us.

On-demand so that we aren't getting trolled by someone pushing 100 chromium 
PRs, wasting either AWS or infra-hosted CPU cycles. There's also security 
issues to take into account, we wouldn't want to autobuild something with 
  SRC_URI="https://underwatertrollsociety.com/~leeth4x0r/mymalwarebomb.zip"

I'm sure in the future some better logic can be used to predict this, but for 
now I'm not aware of such. AI perhaps, one day? Nevertheless on-demand allows
us to check the ebuild, etc before launching build-test to minimize wasted CPU
usage.


Technical details:
 - find a machine that hosts this setup, 
 - set up Github hooks, 
 - use containers, virtualization.


Implementation idea àla juippis:

1: Set up host:
 - stable amd64, 
 - unprivileged lxd containers, 
 - chron jobs to update the *base* container image 1-4 times a day, always 
   against latest state of ::gentoo repo (non-broken).

2: Set up containers / whatever virtualization was decided to be used:
 - ~amd64, 
 - binpkgs are acceptable and likely preferable, but the base container needs 
   to be updated using '--binpkg-changed-deps=y' switch,
 - different settings for emerging dependencies and for testing the *changed*
   ebuilds, 
   - can easily be done with https://github.com/slashbeast/pkg-testing-tools , 
   - use https://dev.gentoo.org/~juippis/configs/test.conf for testing, 
   - 

3: Set up Github:
 - ?


4: Integration:
 - do lxd snapshot per PR,
   - allow interaction with that instance through shell should it be required, 
   - it will most likely take a lot of storage size,
 - have scripts that downloads the Github PR patch, identify what packages 
   were touched, and launch testing specifics for those listed below, 
 - it should identify what packages have been updated and test *all* of those, 
 - testing should happen with multiple USE flag combinations, 
   - pkg-testing-tool?
   - tatt?
   - (but even basic with default USE flags is fine)
 - testing should ideally be also done for at least some of the rdeps, 
   - tatt?
 - the full build.log report needs to be available in the Github PR,
   - directly in Github,
   - via web link, 
 - delete the snapshot image once PR gets merged.


Making it a GSOC project:
 - I (juippis) can mentor,
 - I can provide hosting / tinderbox during GSOC while building the system, 

 - Required skills from a student: 
  - containers, virtualization, 
  - most likely setting up an interactable web server, 
  - Github hooks.
  - (I can assist with containers, setting up web server, general Gentoo stuff 
     but not with Github hooks)

 - should be doable in 3 months.