Actions Runner fails on IPv6 only host

[alsutton]

Describe the bug
On an IPv6 only host a self-hosted action runner fails to register itself as online due to a failed network request to a URL with an IPv4 only address.

Context: Many hosting companies are now charging for IPv4 addresses (most recently AWS), so having IPv6-only self-hosted runners is desirable from an operational cost perspective.

To Reproduce

1. Configure a self-hosted action-runner on an IPv4/v6 host.
2. Reconfigure the host to be IPv6 only.
3. Try to start the action-runner.
4. Runner fails to register and reports error in the logs (below)

Expected behavior
I expected the runner to start and register as online.

Runner Version and Platform

Version of your runner?
2.312.0

OS of the machine running the runner? OSX/Windows/Linux/...
Linux (Debian 12)

What's not working?

Start-up and registration flow requires IPv4 to connect to a URL (pipelinesghubeus4.actions.githubusercontent.com)

nslookup for host indicates no IPv6 address;

$ nslookup pipelinesghubeus4.actions.githubusercontent.com
Server:  UnKnown
Address:  fdc9:993e:6df:0:3e9e:c7ff:fe92:2ee8

Non-authoritative answer:
Name:    pipelinesghubeus4.actions.githubusercontent.com
Address:  20.232.252.48

Job Log Output

Runner_20240208-075819-utc.log, potential token redacted;

[2024-02-08 07:59:02Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2024-02-08 07:59:02Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 11.01 seconds.
[2024-02-08 07:59:11Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.97 seconds.
[2024-02-08 07:59:12Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.91 seconds.
[2024-02-08 07:59:13Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 12.829 seconds.
[2024-02-08 07:59:24Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 17.574 seconds.
[2024-02-08 07:59:25Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 18.057 seconds.
[2024-02-08 07:59:25Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The operation will be retried in 18.337 seconds.
[2024-02-08 07:59:41Z ERR  GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus4.actions.githubusercontent.com/{{REDACTED}}/_apis/connectionData?connectOptions=0&lastChangeId=1500696579&lastChangeId64=1500696579 failed (Socket Error: NetworkUnreachable). The maximum number of attempts has been reached.
[2024-02-08 07:59:41Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2024-02-08 07:59:41Z INFO RunnerServer] Catch exception during connect. 3 attempt left.
[2024-02-08 07:59:41Z ERR  RunnerServer] System.Net.Http.HttpRequestException: Network is unreachable (pipelinesghubeus4.actions.githubusercontent.com:443)
 ---> System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32
lastChangeId, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2024-02-08 07:59:41Z ERR  RunnerServer] #####################################################
[2024-02-08 07:59:41Z ERR  RunnerServer] System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2024-02-08 07:59:41Z INFO GitHubActionsService] Starting operation Location.GetConnectionData

Runner and Worker's Diagnostic Logs

Logs given above

[alsutton]

Possible duplicate of #2840. Is there any news on progress?

[DracoBlue]

I am interested in this, too.

[undergroundwires]

Here's my workaround (open-source and documented) that I hope that can help you too:

• privacy.sexy | force-ipv4: GitHub action to use for GitHub workflows.
• privacy.sexy | force-ipv4.sh.: Script for general use.

After days of research and trial/error, this is how I got this working:

1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd

[KeithYeh]

We faced the same issue when migrating our self-hosted runners to IPv6-only VPC on AWS.
BTW, their CloudFormationInit endpoint does not support IPv6 either.

[esantora]

The runner needs a configurable switch to use IPv4 only in dual-stack systems until GitHub actually supports IPv6.

[touhidurrr]

Still facing this issue.

[guidograzioli]

I have been experiencing this problem pretty rarely but in the last few weeks it started to happen ten times a day.

[basvandijk]

Note that our CI has to run tests that require IPv6 so we have to patch the runner (dfinity@3b9c15c) to not create a local docker network and instead use --network host (essentially using pod's network namespace that has IPv6 configured):

Ideally this is just a runner's configuration option so we don't have to maintain our own runner fork.

[marko-k0]

Note that our CI has to run tests that require IPv6 so we have to patch the runner (dfinity@3b9c15c) to not create a local docker network and instead use --network host (essentially using pod's network namespace that has IPv6 configured):

Ideally this is just a runner's configuration option so we don't have to maintain our own runner fork.

Correct. Changes proposed in #3622, would allow us to use --network option.