Unable to connect to establish a TCP connection (ECONNREFUSED)

[JeffreyArt1]

Version

v17.0.1

Platform

Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

I'm trying to connect to a micro-service with this piece of code:

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket.connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', e => console.log(e));
};

connect('localhost', 5098);

this prints out:

Error: connect ECONNREFUSED ::1:5098
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 5098
}

How often does it reproduce? Is there a required condition?

Since installed node 17

What is the expected behavior?

No errors.

What do you see instead?

Error: connect ECONNREFUSED ::1:5098
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 5098
}

Additional information

As far a i tested (v14.18.1 and v16.12.0), this only happens in this version of node.

[nsainaney]

We have been having the same issue with node 17 and it appears to be a breaking change with how DNS.lookup works. With node 16, the lookup would return a IPv4 address but with node 17, it returns an IPv6 address which will break most REST clients that hardcode URLS like http://localhost:4040/api if the upstream server only binds to the IPv4 address (e.g. server.listen('127.0.0.1'...) etc..

sudo n latest
   installed : v17.0.1 (with npm 8.1.0)
╭─[prompt redacted]›
╰─$ node
Welcome to Node.js v17.0.1.
Type ".help" for more information.
> require('dns').lookup('localhost', null, (e,a,f) => console.log(e,a,f))
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'localhost',
  oncomplete: [Function: onlookup]
}
> null ::1 6
.exitit
╭─[prompt redacted]›
╰─$ sudo n 16.2.0
   installed : v16.2.0 (with npm 7.13.0)
╭─[prompt redacted]›
╰─$ node
Welcome to Node.js v16.2.0.
Type ".help" for more information.
> require('dns').lookup('localhost', null, (e,a,f) => console.log(e,a,f))
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'localhost',
  oncomplete: [Function: onlookup]
}
> null 127.0.0.1 4

Notice the results are null ::1 6 vs null 127.0.0.1 4 with null options

[richardlau]

For reference the change was #39987 -- Node.js no longer re-sorts results of IP address lookups and returns them as-is (i.e. it no longer ignores how your OS has been configured). You can change the behaviour via the verbatim option to dns.lookup() or set the --dns-result-order command line option to change the default.

[nsainaney]

Thanks, @richardlau. How is one supposed to fix this if they are not the author of the server/client. Libraries like axios do not let you pass in the verbatim option so I suppose we all have to use the command line option until packages catch up.

[nsainaney]

Update to my comment above. When dealing with 3rd party libraries, we can use dns.setDefaultResultOrder('ipv4first') and have to be careful to use this in child_processes as well

[wibobm]

v17.0.1 and v17.1.0 have same issue with remote debugging

Error in debuggerConnector: connect ECONNREFUSED ::1:49629
Error: connect ECONNREFUSED ::1:49629
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)

[treysis]

@richardlau The better option would of course be to change localhost to 127.0.0.1 when connecting to legacy localhost.

@nsainaney If you cannot fix server/client, the solution would be to stay on the support NodeJS version, i.e. <17, until the authors of server/client add compatibility for Node v17.

[nsainaney]

Thanks @treysis. There are enough workarounds that this isn't an issue for us.

[treysis]

@nsainaney Yes, I'm just mentioning it because I consider it to be the cleanest way. If you set the setDefaultResultOrder option you might forget about that in the future and will limit IPv6 capability of your applications.

[nsainaney]

That's a good point. Thank you

[lydell]

I run this:

require("http").get("http://localhost:3004/", res => console.log(res.statusCode))

• In Node.js 16 it prints a status code.
• In Node.js 17 I get ECONNREFUSED.

Where can I find the OS configuration that Node.js now honors…

• …on macOS?
• …on Linux?

Sorry if it feels like a stupid question or like a question that belongs somewhere else, but I think more people than me don’t know this DNS stuff very well and will wonder why things “aren’t working”.

[lexicalunit]

I run this:

require("http").get("http://localhost:3004/", res => console.log(res.statusCode))

You have to change this to:

const dns = require('dns');
dns.setDefaultResultOrder('ipv4first');
require("http").get("http://localhost:3004/", res => console.log(res.statusCode))

Which should work in both Node v16 and v17, as well as on macOS and Linux.

[lydell]

@lexicalunit Yes, I read that in the various threads, but that’s not what I’m asking for.

[treysis]

It's difficult to find because bc iirc it depends on sooo many parameters: IPv6 connectivity available? Glibc? Another libc implementation? Systemd?

Instead of changing the dns sorting order globally for a package, it's advised to use 127.0.0.1 when addressing localhost though.

[lydell]

I don’t quite buy that explanation. If visiting http://localhost:3004 in a browser works and curl http://localhost:3004 works, but require("http").get("http://localhost:3004/", res => console.log(res.statusCode)) fails then it’s gonna look like Node.js is broken. If the explanation is “Well actually, Node.js is correct because it respects hidden OS configuration, just trust me” that doesn’t feel very reassuring?

Note that I’m looking for the default configuration on macOS for example. I bet many developers don’t change anything (knowingly). I’ve tried to find more info myself, but I don’t know what to search for.

[treysis]

Browsers and curl support happy eyeballs, so they try both IPv4 and IPv6. Node has no support (yet?) for happy eyeballs.

DNS lookup in general is documented well. However, localhost is a special case as it's not resolved by the DNS server but before DNS is even asked. It seems to also depend on macOS version. Even for Linux it's not well because again it depends on the library that is being used. So you'd have to look up what the library is doing, and it's also different from distro to distro. E.g.:
NixOS/nixpkgs#19148

Glibc for example also apparently violates some RFC and in some special case maps ::1 to 127.0.0.1, see:
https://sudonull.com/post/60933-Resolve-IP-Addresses-on-Linux-An-Intuitive-and-Detailed-Description

For the time being: don't use localhost but instead use 127.0.0.1.

[lydell]

Thank you!

I think I understand this better now. I’ll summarize what I (think I have) learned in case it helps others.

When I do require("http").get("http://localhost:3004/"), Node.js needs to turn localhost into an IP address. The default /etc/hosts on macOS starts with:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1	localhost

So if you try to look up localhost in there, you will get two answers: 127.0.0.1 and ::1. So I’ve asked the computer to connect to “localhost”, but now there’s ambiguity.

Node.js 16 and older arbitrarily implemented code that favored ipv4 (127.0.0.1).

Node.js 17 seems to follow the principle of least astonishment, and take the first result from what was returned by the OS when doing the lookup (seems to be getaddrinfo(3) via https://nodejs.org/api/dns.html#dnslookup). I don’t know exactly how that order is defined.

To test, I temporarily commented out ::1 localhost from /etc/hosts. Then Node.js 17 manages to make the request to my HTTP server only listening on ipv4 (0.0.0.0), as 127.0.0.1 is the only result returned. (Reordering the lines in /etc/hosts did not make a difference).

• Node.js 17 no longer has an “astonishing” default, which seems good.
• But people will accidentally have relied on the previous behavior, so there will be a transition period of fixing things.
• One can use 127.0.0.1 to explicitly say “ipv4 localhost”.
• Check the server you connect to: Does it serve on 0.0.0.0 (ipv4) or :: (ipv6) or both?
• There’s a thing called Happy Eyeballs which means connecting to both ipv4 and ipv6 in parallel, which Node.js doesn’t have, but explains why other tools/languages can connect. Update 2023-01-27: Node.js now has Happy Eyeballs: net: add autoSelectFamily option #44731. Released in Node.js 20.

[treysis]

Yes. What is still surprising to me: why does Node sometimes seem to resolve localhost differently depending on if opening a listening socket and on opening a connect socket. At least on a local system it should be the same, no? I couldn't figure it out when introducing the PR last year that stopped reordering the results.

• Yes, in general it's good.
• That's why it wasn't backported to 16. You should expect changes with a new major version. But it was also introduced because more and more people started having problems that have to rely on IPv6. There were ambitions to introduce it much much earlier, probably as early as Node 12 (iirc), but it required a larger effort because the automated tests were already crafted with too much IPv4 in mind and because IPv6 connectivity of users back then wasn't always too reliable.
• You could also specify family: 4. but 127.0.0.1 is well known and even IPv6-only systems will still use 127.0.0.1 for many more years, so it seems reasonable enough to just keep it for now for simplicity.
• Unfortunately there's no default listening socket for dualstack on localhost. You can easily listen on :: and typically it should mean both 0.0.0.0 and ::, but you can't do that for localhost. You'd need to open a socket on 127.0.0.1 and a second one on ::1.
• Unfortunately Happy Eyeballs is not as trivial as it sounds. But it's something that would also help in general DNS round robin situations.

[javierfuentesm]

@javierfuentesm Upgrade to node 20 where this has been resolved.

that worked! thank you!

[angelorc]

@javierfuentesm Upgrade to node 20 where this has been resolved.

Which version? I'm running on v20.5.1 and I still get the error

[treysis]

I still get the error

Which error exactly and what do you do?

[Rakeshbora007]

NEXT_PUBLIC_API_URL=http://127.0.0.1:3000

it still not working formw

[tahmidrahman-dsi]

Could anyone find this problem in Node v18.19.0 with Ubuntu 20.04.6?

[treysis]

NEXT_PUBLIC_API_URL=http://127.0.0.1:3000

it still not working formw

Which Node version? Did you try localhost instead of 127.0.0.1?

[treysis]

Could anyone find this problem in Node v18.19.0

Use Node 20!

[1-Rishav]

Make sure that your password and port number of database should be correct because this "ERROR" take place when your database credentials are wrong.

[mattdesl]

Is the code in the OP supposed to work on node 20?

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket
    .connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', (e) => console.log(e));
};

connect('localhost', 5098);

I get the same error in node v20.11.1 and v21.6.2:

AggregateError [ECONNREFUSED]: 
    at internalConnectMultiple (node:net:1116:18)
    at afterConnectMultiple (node:net:1680:7) {
  code: 'ECONNREFUSED',
  [errors]: [
    Error: connect ECONNREFUSED 127.0.0.1:5098
        at createConnectionError (node:net:1643:14)
        at afterConnectMultiple (node:net:1673:16) {
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 5098
    },
    Error: connect ECONNREFUSED ::1:5098
        at createConnectionError (node:net:1643:14)
        at afterConnectMultiple (node:net:1673:16) {
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '::1',
      port: 5098
    }
  ]
}

This also seems related to the error I'm getting from this demo app:
p2panda/mushroom-app-tutorial#7

[lydell]

@mattdesl In your error message, you can see how there are two errors: Node tried both connecting via IPv4 127.0.0.1:5098, and via IPv6 ::1:5098, which is the Happy Eyeballs thing added in #44731.

I get that error too, because I have nothing running on port 5098. But when I start a server listening on 5098, OP’s code runs without errors:

require('http').createServer().listen(5098);

// If you do this instead, the below code fails on Node.js 16 but works on later Node.js versions (unsure exactly which releases):
// require('http').createServer().listen(5098, '::1');

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket
    .connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', (e) => console.log(e));
};

connect('localhost', 5098);

[undergroundwires]

Here's my workaround (open-source and documented) that I hope that can help you too:

• privacy.sexy | force-ipv4: GitHub action to use for GitHub workflows.
• privacy.sexy | force-ipv4.sh.: Script for general use.

After days of research and trial/error, this is how I got this working:

1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd