buildbot/phase2, branch v2

phase2: fix just another byte/string inconsistencies

2022-03-26T09:18:10Z

File "/phase2/master.cfg", line 203, in parse_feed_entry if parts[0].startswith("src-git"): builtins.TypeError: startswith first arg must be bytes or a tuple of bytes, not str Fixes: f3fcdbec877e ("phase2: fix obsolete assumption about feed methods") Signed-off-by: Petr Štetiar

phase2: handle missing change sources

2022-02-23T17:46:41Z

In commit f3fcdbec877e ("phase2: fix obsolete assumption about feed methods") we've fixed stuck buildmaster for master/phase2 builds, but the issue wasn't obvious at first sight and some digging was necessary. So lets make it obvious if/when it happens next time. Signed-off-by: Petr Štetiar

phase2: fix obsolete assumption about feed methods

2022-02-23T17:23:43Z

In commit 7fae1e5677e9 ("feeds: use git-src-full to allow Git versioning") there was `src-git` method of shallow clone replaced with `src-git-full` and this has caused issue with GitPoller configuration as it only assumed, that only `src-git` method is going to be used for feed updates. Fix currently stuck buildbot master by allowing usage of both Git update methods. Signed-off-by: Petr Štetiar

phase1,phase2: improve round robin builds

2021-07-21T19:56:13Z

There seems to be some issue with database updates, where database updates are asynchronous and thus the status update of the previous build (complete_at DB column) might take little bit longer, then preparation of the next/current build. This is issue during job prioritization as this might result in the build of the same builder twice in a row, for example: 2021-06-22 02:42:54+0000 [-] : build finished 2021-06-22 02:42:55+0000 [-] prioritizeBuilders: mipsel_24kc complete_at: 2021-06-21 20:17:14+00:00 2021-06-22 03:14:30+0000 [-] prioritizeBuilders: mipsel_24kc complete_at: 2021-06-22 02:42:54+00:00 Build finishes at 02:42:54, scheduler then asks for next build at 02:42:55, but the build still has the old complete_at timestamp 2021-06-21 20:17:14 instead of the correct one 2021-06-22 02:42:54, thus scheduling the build of oldest mipsel_24kc builder one more time. This is so far very promising workaround attempt which checks latest builder complete_at in builds table which seems to be updated faster, thus using greater complete_at value seems to work for now. References: https://github.com/buildbot/buildbot/issues/4592#issuecomment-801163587 Signed-off-by: Petr Štetiar

phase1,phase2: add file space usage reporting

2021-03-25T08:54:49Z

Add `du` step so it's clear how much space was used by the build. Signed-off-by: Petr Štetiar

phase1, phase2: exclude temporary source files from sourceupload step

2021-03-25T08:51:21Z

When the dl/ directory is shared with several workers, there can be temporary "*.hash" and "*.dl" files being created by other workers. These files should be excluded from the sourceupload step, otherwise it can cause a failure when rsync tries to read them. See for example: https://buildbot.openwrt.org/master/packages/#/builders/14/builds/18/steps/39/logs/stdio Signed-off-by: Baptiste Jonglez

phase1, phase2: don't mark the build failed if a non-critical step fails

2021-03-25T08:50:42Z

Some steps are informative or optional (haltOnFailure=False), so it makes no sense to mark the whole build as failed if such a step fails. Depending on the case, we either mark the build as "warning" (warnOnFailure=True) or as successful (warnOnFailure=False). Signed-off-by: Baptiste Jonglez

phase1,phase2: fix reconfig command by creating twistd.pid

2021-03-19T20:54:30Z

Currently it's not possible to reconfig buildmaster as twistd.pid is missing so fix it by creating one manually. Signed-off-by: Petr Štetiar

phase1,phase2: implement round robin builds

2021-03-17T08:33:29Z

Gather newest complete and not skipped build timestamps, reverse sort them and use that as builder priority, so the newest built targets are at the bottom. Signed-off-by: Petr Štetiar

phase1,phase2: shuffle order of builds

2021-03-15T21:47:41Z

It seems that once there are new commits, buildbot restarts building all targets in alphabetical order. The current logic pretty much disables the building of the alphabetically last targets, so until there is proper solution just workaround it by randomly shuffling the builders list. References: https://github.com/buildbot/buildbot/issues/5193 Signed-off-by: Petr Štetiar