discussion: clarify Go support policy for secondary ports

[ianlancetaylor]

This is now a proposal at #53383.

---

This discussion is intended to become a proposal after considering comments.

proposal: clarify Go support policy for secondary ports

Background

Go supports a number of different GOOS/GOARCH targets. We've defined a policy for adding new ports, described at https://go.dev/wiki/PortingPolicy.

Ports are divided into first class ports and secondary ports. The current first class ports are:

• darwin/amd64
• darwin/arm64
• linux/386
• linux/amd64
• linux/arm
• linux/arm64
• windows/386
• windows/amd64

The current secondary ports are:

• aix/ppc64
• dragonfly/amd64
• freebsd/386
• freebsd/amd64
• freebsd/arm
• freebsd/arm64
• illumos/amd64
• linux/ppc64
• linux/ppc64le
• linux/mips
• linux/mipsle
• linux/mips64
• linux/mips64le
• linux/riscv64
• linux/s390x
• android/386
• android/amd64
• android/arm
• android/arm64
• ios/arm64
• ios/amd64
• js/wasm
• netbsd/386
• netbsd/amd64
• netbsd/arm
• netbsd/arm64
• openbsd/386
• openbsd/amd64
• openbsd/arm
• openbsd/arm64
• openbsd/mips64
• plan9/386
• plan9/amd64
• plan9/arm
• solaris/amd64
• windows/arm
• windows/arm64

The core Go team maintains the first class ports. It is less clear how the secondary ports are handled.

The existing porting policy says:

• Every secondary port must have a maintainer who will make required updates in a timely manner.
• Every secondary port must have a builder, and a person who maintains that builder.
• If the builder for a port is failing for more than two weeks, a new maintainer is needed.
• If a builder fails for more than four weeks or is failing at the time of a release freeze, and a new maintainer cannot be found, the port will be removed from the tree.

However, in practice we do not follow those rules.

• It has not been clear who the secondary port maintainers are, and there was no mechanism for adding or removing maintainers; this was recently improved by creating the @golang/port-maintainers GitHub team and subteams.
• When a change requires an update to the port, it is the person making the change who takes responsibility for updating the port, not the port maintainer.
• Maintainer responsibilities are in general unclear.
• Several secondary port builders are maintained by the Go team.
• We do not remove ports from the tree if they are failing for four weeks (we do remove ports if the OS is no longer supported, such as darwin/386).

The effect is that the work required to maintain secondary ports falls on people who are not familiar with those ports. This was not the goal of the porting policy, and it tends to slow down overall development of the core Go systems and discourages the adoption of new secondary ports.

We propose both loosening and tightening the current porting policy to address these concerns.

Proposal

• The Go team will stop publishing binaries for secondary ports.
  • The Go team currently publishes binaries for the following secondary ports, which we would stop doing if we adopt this part of the proposal:
    • freebsd/386
    • freebsd/amd64
    • linux/ppc64le
    • linux/s390x
    • windows/arm64
  • We encourage port maintainers to publish them instead; we can link to pages from https://go.dev/dl as appropriate.
  • Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.
• Require at least two maintainers for each secondary port.
  • If maintainers resign or become unreachable and can't be replaced, we mark the port as broken; see below.
  • This means that for some existing secondary ports we will have to find additional maintainers. As I write this I think that additional maintainers are needed for android, dragonfly, illumos, ios, js/wasm, openbsd, and solaris.
• In general developers will not be required to make their code work on every secondary port.
  • They are encouraged to do so if it is straightforward.
  • If not, they are encouraged to notify port maintainers about pending requirements as soon as feasible, and to work with the port maintainers to get the ports working again.
  • That said, ultimately the port maintainers are responsible for keeping their ports working.
• Notify maintainers when a port breaks.
  • By sending an e-mail to maintainers and golang-dev.
    • Perhaps also open an issue.
  • We can do this preemptively if we know that some change is going to break a secondary port.
  • Ideally this is an automated system, but for now it may be manual.
    • For example, perhaps the system that creates https://build.golang.org/ can send e-mails about all failures to a newly created list. Maintainers can filter that list for the ports that they care about.
    • Currently maintainers can track build dashboard triage log #52653 to see problems.
• Clarify maintainer responsibilities on the PortingPolicy wiki page.
  • A GOOS maintainer is responsible for any _GOOS files, any files with just that build tag, and blocks of code guarded by if runtime.GOOS == "mygoos".
  • Similarly for a GOARCH maintainer.
  • If technically possible, GOOS/GOARCH maintainers who are not already approvers should be given the right to +2 changes to files for which they are responsible.
    • These CLs will still require a +1 review for style and sanity but not for detailed correctness; we will defer to the maintainers for that.
    • Note that with two maintainers one of them can +2 changes by the other.
• We will introduce a new concept: the broken port.
  • We should not assume that maintainers will respond within 2 or 4 weeks; people are busy.
  • If a port stops working, including the case where a builder stops working, we can decide to mark the port as broken.
  • In general, a port can be considered broken if its builder has failed multiple times in a development cycle with a failure mode that does not occur for first class ports, and that failure mode is not believed to have been fixed or suppressed by a change in either a Go repository or the builder's configuration
  • Any approver can declare that a port is broken.
  • The list of broken ports will be maintained in cmd/dist and will appear in the release notes if broken at release time.
  • Attempting to build a broken port will fail unless make.bash or go tool dist is invoked with a new option -force.
  • If a port is broken in release 1.N, then the core Go team can choose to remove the port from release 1.N+1.
    • This is not obligatory and will depend on whether anybody is willing and able to maintain the port going forward.
  • We currently have a list of incomplete ports in cmd/dist/build.go; these should probably be treated as broken ports.
    • Currently the only entry on that list is linux/sparc64.
• The default set of trybots will change to only cover first class ports.
  • The https://build.golang.org/ page will continue to show all ports.

Discussion

This is not intended to be a big change to the current process. However, it is intended to be a change. It is intended to take some of the porting load off of the core Go team, while making it easier for port maintainers to make changes. It is intended to make it easier to add new ports to the tree.

In the long run it would be good to support out of tree ports. However, that requires a bunch of technical work, and there is no design for it.

[mengzhuo]

1. How can a port maintainer upload binary to go.dev/dl? Others try to download the manifest.
2. Could the build farm alert the maintainer that the first CL breaks builder?
3. Could we setup a page like "port status" as a reference in go.dev? Since users might not trust "secondary ports".

---

Port status (WIKI?) includes:

1. maintainers and their sponsor
2. builder status
3. feature completeness compares to first class
4. performance reports like: go1 bench
5. Github open bugs/issues status

[bcmills]

Could the build farm alert the maintainer that the first CL breaks builder?

At the moment that process is manual. However, you can expect an issue to be filed including a mention of the appropriate subteam(s) of @‌golang/port-maintainers.

[changkun]

In general developers will not be required to make their code work on every secondary port.

It is totally understandable for the Go team to reduce their workload and speed up their development process. However, if the "developers" refer to the developers on the Go team, I would argue this proposal may transfer too many responsibilities to the secondary port maintainers and requires too much from the secondary port maintainers.

Clarify maintainer responsibilities on the PortingPolicy wiki page.

The responsibility of maintaining a port is not as simple as what is proposed here. Based on some recent stories from @mengzhuo regarding the porting experience of regabi, maintaining a port requires a port maintainer to understand the current development process, plan, and get familiar with the latest design requirements. However, these tasks are generally not easy and strictly bound to the Go team's schedule, and maintainers may only have little impact on them. Most port maintainers are doing the porting work as a hobby in their spare time. Understand the status is already a lot of investments. This is incomparable to the Go team, which is hired dedicatedly for the development, whereas the daily progress of a port maintainer is not tremendous. Furthermore, port maintainers have to comply with and resolve the conflicts in the Go team's development process. With the current high review requirements, if the first-class port grows too fast, "encourage but not required" may prevent other ports stop to working because the work that has been done in a secondary port may simply conflict with the latest tree.

ultimately the port maintainers are responsible for keeping their ports working.

Therefore, this goal may be too strong. I think a possible relaxation maybe:

In general, developers on the Go team should make their code work on a secondary port if a corresponding builder is online and functioning. If a builder is offline or broken for an entire development or freeze cycle (3 months), then it is not required to make their code work on that port, and the port may be removed in the future. There may be exceptions: Developers should communicate with the current port maintainers and find consensus regarding if it is reasonable for a chance to work for that port (such as skipping a test on a secondary port builder).

[bcmills]

Most port maintainers are doing the porting work as a hobby in their spare time. Understand the status is already a lot of investments.

Indeed. That's partly why we're thinking in terms of 2 or more maintainers per port, and part of why a port broken in release 1.N would not be removed until at least release 1.(N+1) — that gives the maintainers a full release cycle to catch up in the event of a serious problem, and with more than one maintainer it is more likely that at least one maintainer will have the time to look into a regression in any given release cycle.

On the other hand, if we were to require Go developers to always make their code work on secondary ports, then those secondary ports would no longer be “secondary”: the main difference between a “first-class” port and a “secondary” one is whether “all contributors” are responsible for fixing them. Removing that distinction would mean that we would not be able to accept nearly so many secondary ports in the Go repo to begin with, because the Go team would have to budget for the full cost (including the cost to other contributors) of maintaining those ports vs. the benefit of having them in the tree.

[changkun]

that would substantially raise the effort required to contribute even non-port-specific changes

I agree a lot with the vision of not requiring always work for every secondary port. However, in practice, we do not know well enough whether a change is "port-specific". For example, CL 256917 is specifically for darwin/arm64. Without further notice, it implicitly breaks ios/arm64. If developers are informed that only responsible for their ports. This type of break may be amplified over time. We might end up with different implementations on different ports. Therefore I think "best-effort" port responsibility may be better than "encouraged" port policy in this context.

[rhysh]

I'd like to share an experience report, as a contributor who ran into related tests failing on a secondary port.

if a corresponding builder is online and functioning

My own experience has been that "builder is online and functioning" isn't enough to make sure my code works well on a port. I had a hard time with the linux/riscv64 port last fall, with a typical delay of 3 hours between when I asked for trybots/slowbots to when a builder for that port would get to the part of all.bash that tested my changes. The relevant tests didn't fail 100% of the time, which made it even harder to see if I was fixing the problem or not.

Gomote might have helped, but new access requests for that are not currently being granted (since May 2021). The only way I saw to get a tight feedback loop (which I saw as a necessary part of building trust in my code on the first-class ports) was by buying a RISCV64 single-board computer of my own. I didn't find RISCV64 machines available for rent from a cloud provider, and it took about two weeks for the SBC to arrive in the part of the world where I live. But after that, I was able to get my code working well on the port without too much trouble.

It looks like the Gomote security work is moving forward, so maybe that's a good path for Go contributors to get fast feedback on whether a broken secondary port is giving them early warning about a subtle bug that can affect multiple platforms. But without it, I think that "builder is online and functioning, is only available to run all.bash, and gives results after three hours" is too low a bar.

[bsiegert]

As the maintainer of the netbsd-arm and netbsd-arm64 builder, the way it currently works is fairly stressful. I maintain the machines, which are kind of fickle as they are ARM SBCs. But then I also get a near-daily stream of issue CCs on test failures.

As a contributor in my spare time (I am a Googler but not on the Go team), this creates a lot of stress for me, since there is always an implied threat in these messages. The threat is "if YOU don't fix this, or get someone else to fix this, we'll remove the port you are actively using from the tree". This is not a hypothetical concern! @bcmills literally filed issue #50451 to propose dropping the netbsd-arm64 port -- thankfully this was declined eventually.

Running builders and maintaining support is thankless enough as it is. I fear that by making requirements more stringent, you are just going to lose the contributors that keep platforms alive.

[bcmills]

@bsiegert, I'm sorry that the way it currently works ends up causing stress.

FWIW, it's been quite stressful for me too, which is part of why I've been spending a large chunk of my time over the past year fixing platform-independent bugs that happen to surface on these builders — for example, just yesterday I noticed #53160 on one of the -bsiegert builders, triaged it down to a platform-independent issue in the test, and mailed and merged a fix.

But it's a lot more work (for me and others) to triage and fix the platform-independent bugs when there are also platform-specific bugs adding noise to the process, and it really doesn't feel great to do the legwork to collect details on a platform-specific bug (as in #50138 (comment)) only to have that information go apparently-unused because nobody has enough time to actively maintain the port.

As I see it, part of the point of the proposed changes is to alleviate the stress on both sides.

• The two-maintainer requirement should help to distribute load over multiple maintainers. (If you-specifically don't have the bandwidth to fix an issue, there ought to be someone else who can help!)
• The process to mark a port as broken (and guidance about what that means) should reduce the need to ping maintainers about failure modes that nobody is actively working on, and provide a clearer timeline for when a fix is expected — namely, a full 6-month release cycle after the port is marked broken — before anyone even considers removing the port.
• The explicit policy of notifying maintainers when a port breaks should help to prevent platform-independent bugs from being glossed over just because they happen to reproduce more readily on secondary builders. (If someone from the Go project is triaging the failures enough to determine who to notify, it's more likely that they'll notify an appropriate package maintainer instead of just ignoring the failure altogether.)

[FiloSottile]

How does the broken port state interact with the freeze? If a port is broken at the beginning of the freeze, is a freeze exception going to be always approved for unbreaking it, or is it going to be broken in that release?

[bcmills]

I would suggest that it depends on how invasive the fixes are, and at what point in the freeze the work begins.

If the fix only affects source code specific to the broken port, then no freeze exception should be needed in the first place. (The port is already broken, so there is little risk to the release.)

If the fix involves modest cross-platform changes but the work is started fairly early in the freeze, then a freeze exception should pretty much always be granted.

If the fix involves more invasive cross-platform changes, or the work is started late in the freeze (such that it wouldn't be finished by the first RC), then a freeze exception should generally be declined (but may be decided on a case-by-case basis).

[laboger]

In general developers will not be required to make their code work on every secondary port.

• They are encouraged to do so if it is straightforward.

• If not, they are encouraged to notify port maintainers about pending requirements as soon as feasible, and to work with the port maintainers to get the ports working again.

• That said, ultimately the port maintainers are responsible for keeping their ports working.

In the past it has been common to enable new features under the control of GOEXPERIMENT or something similar, allowing time for secondary ports to do their work after the primaries have completed and are merged. Hopefully any major new features that require significant work to enable on secondary ports can continue to be done in this way where possible.

The whole concept of broken ports is very concerning. This means it is possible for someone to make a change that breaks a secondary port right before the freeze, which could take significant time to fix. Then it would be labeled as a "broken port" and removed from the release (not sure what that exactly means -- somehow disabled completely? code completely removed? Sounds quite drastic.)

If a port stops working, including the case where a builder stops working, we can decide to mark the port as broken.

The ppc64le/ppc64 builders are maintained by the Go team and we don't have access to those machines. There have been several cases where there have been failures on the builders that we can't reproduce on systems here which has made it hard to resolve those issues. Are there any plans to change the maintainers of the builders for some cases or will they continue to be maintained by the Go team?

As far as your comments concerning publishing binaries of secondary ports, if that is changed hopefully we could get enough lead time on this so we could set that up, especially if the plan is to do this for Go 1.19. Will the old binaries also be removed as well?

[bcmills]

This means it is possible for someone to make a change that breaks a secondary port right before the freeze, which could take significant time to fix.

If we expect the fix to take significant time, I would expect the developer making the breaking change to discuss the plan ahead of time to avoid that kind of situation.

Changes that are that disruptive (and that difficult to fix) should be landing at the beginning of the cycle (see the early-in-cycle label), not the end, and I think you could reasonably make a case for rolling back a change for one release cycle if it breaks an otherwise-healthy port without advance discussion in a way that requires a lot of work to fix.

(That said, most of the changes that break ports are also not that difficult to fix for someone with knowledge of the platform and a day or two to debug.)

[bcmills]

The flip-side of that is: I don't think we should roll back a late change merely due to breaking a port, especially if nobody has done the work to triage how much work it will be to fix. For an actively-maintained port, we should be able to count on at least a day or two of debugging from the port maintainers in between the start of the freeze and the RC if the need arises.

[rsc]

The ppc64le/ppc64 builders are maintained by the Go team and we don't have access to those machines. There have been several cases where there have been failures on the builders that we can't reproduce on systems here which has made it hard to resolve those issues. Are there any plans to change the maintainers of the builders for some cases or will they continue to be maintained by the Go team?

Hi @laboger! I think if IBM wanted to run the PowerPC64 machines hosted at OSU instead of the Go team, we would not object to that, but I'm also not sure we need to go that far. We can get you and anyone else at IBM at least ssh access to the machines for debugging those kinds of machine-specific problems. I'll mail you off list to find out who we should include. Thanks!

[bcmills]

• Notify maintainers when a port breaks.
  • By sending an e-mail to maintainers and golang-dev.
    • Perhaps also open an issue.

I suggest that we reverse the priority of those — prefer to open a GitHub issue over sending an email.

A GitHub issue makes the problem easier to search (for someone searching duplicates for a bug they've encountered, triaging a test failure, or looking for a fix to contribute) and makes the relevant owners easier to identify (because GitHub will autocomplete the relevant @golang subteam), and in general we expect that Go maintainers will be active in GitHub issue discussions anyway.

[bcmills]

(I do think it's worth sending an email to maintainers and golang-dev before we mark a port as broken, though, in case the GitHub issue notifications have been dropped or overlooked.)

[laboger]

I asked for feedback from our testers and internal users and was told this is a common way to retrieve the binary:

curl -sSfL -o- https://golang.org/dl/go$GO_VERSION.$GOOS-$GOARCH.tar.gz | tar xfz - -C <target_dir>

[laboger]

I want to get back to this part of the proposal:

The Go team currently publishes binaries for the following secondary ports, which we would stop doing if we adopt this part of the proposal:
-freebsd/386
-freebsd/amd64
-linux/ppc64le
-linux/s390x
-windows/arm64
We encourage port maintainers to publish them instead; we can link to pages from https://go.dev/dl as appropriate.
Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.

As I posted above, a common use in scripts and Dockerfiles is to pull the binary directly from the golang.org/dl location. As I read this, golang.org/dl would now contain a link to the location where the secondary ports would host their binaries so existing curl commands like above would no longer work. Other comments in this issue make it sound like we could provide you with the binary to post as before. Can this be clarified?

Also in the wording it says "if we adopt this part of the proposal" which sounds like this is not for sure.

In any case, we would hope to have some lead time if this is something we have to do. Also does this mean you will remove all the old binaries as well?

[ianlancetaylor]

That is a good question. The general goal here is to draw a line between binary releases that the Go team will build and ones that the team will not build. Right now that line seems a bit random, and honestly I'm not sure how we decided it. It seems simpler to just draw the line around the first class ports. That has the advantage of clarifying that the Go release team isn't required to maintain the builders for the other ports with binary releases.

That said, there is a separate issue which is where other binary releases will live. I did not actually intend for golang.org/dl/RELEASE to forward to a binary release hosted elsewhere. It's important that the binary releases be trusted, exactly because people do treat them as a relatively trusted resource. And that's not necessarily consistent with forwarding to some other host. What I was thinking was more like a link on dl.golang.org saying "look here for releases for GOOS/GOARCH".

That said, a different approach might be to continue to host binary releases for some secondary ports, but to expect secondary port maintainers to build and update those releases. I don't actually know myself where those builds are stored today, so I don't know whether that is reasonable.

If that is feasible, and we did that, would that address your concern? Thanks.

[laboger]

We should be able to take over building the release binaries, and it would be great if they could be found at the same location as now to avoid breaking users who have scripts or Dockerfiles that reference the existing location.

Thanks for the clarification.

[ianlancetaylor]

I got some pushback on that internally, because it would mean that golang.org/dl would be hosting hard-to-verify binaries and as such go against the new reviewing rules. But it may be possible for the release team to simply cross-compile release builds for arbitrary targets, with any cgo elements filled in on the user's machine with the user's C compiler.

[bcmills]

I've updated x/build/dashboard with KnownIssues for all of the builders I've been omitting from triage due to flakiness.

For reference, those builders are:

• freebsd-arm-paulzhol
• linux-amd64-unified
• netbsd-386-9_0
• netbsd-amd64-9_0
• netbsd-arm-bsiegert
• netbsd-arm64-bsiegert
• openbsd-arm64-jsing [edit: this one is probably fixed now]
• plan9-386
• plan9-386-0intro
• plan9-amd64-0intro
• plan9-arm
• solaris-amd64-oraclerel
• windows-arm64-11

At this time, I would consider freebsd/arm, openbsd/arm64, all of plan9, [edit: all of netbsd,] and solaris/amd64 to be broken, as they don't have reliable builders.

I would also consider windows/arm64 to be broken: although we have one builder that seems to be passing reliably (windows-arm64-10), that builder is currently skipping a test that shows signs of severe file corruption (see #50706).

(windows-amd64-unified is a builder for an active-but-incomplete GOEXPERIMENT, and thus not a broken port.)

[bcmills]

Is there a way to have a builder that doesn't report status back to the go build dashboard?

Generally we don't triage builders marked with known-issues, so IMO it would be fine to add a builder at HEAD and just leave it marked with an issue ~forever.

(I would only consider the port overall to be broken if all of its builders are broken or unreliable — having one builder intentionally unstable seems fine as long as there is another builder working reliably.)

If not, I can figure out a way to run the tests out of band, perhaps with a cron entry.

FWIW, running src/all.rc in a loop at head with GO_BUILDER_NAME set in the environment ought to at least give a reasonable approximation, although some of the batching and timeout behaviors may differ slightly from the real buildlet.

[0intro]

@oridb I think it would be useful if you could run a plan9/amd64 builder running the 9front pc64 kernel on your side. It would provide more coverage, since the current plan9/amd64 builder is running Bell Labs' 9k kernel.

[0intro]

@bcmills I think it would be a good idea to try to run the plan9/386 builder on GCE again, if you are interested. I can provide an updated Plan 9 image.

[bcmills]

Running Plan 9 builders on GCE seems like a good idea to me — that would at least eliminate the builder hardware as a variable.

I'm not very involved with the builder deployment at this point; maybe start a discussion on #29801 with @golang/release to see what's viable?

[jaredpar]

windows-amd64-unified is a builder for an active-but-incomplete GOEXPERIMENT, and thus not a broken port

Is there a reason this builder isn't listed in https://farmer.golang.org/builders?

[laboger]

Hi @laboger! I think if IBM wanted to run the PowerPC64 machines hosted at OSU instead of the Go team, we would not object to that, but I'm also not sure we need to go that far. We can get you and anyone else at IBM at least ssh access to the machines for debugging those kinds of machine-specific problems. I'll mail you off list to find out who we should include. Thanks!

If we could get on to try and reproduce the problems that would be sufficient.

[ianlancetaylor]

I've opened a proposal issue for this at #53383. Closing this discussion.