Resolving `localhost` hostname

[przlada]

When resolving the hostname, only the first IP address is forwarded. While this is ok, it can pose issues on systems with IPv6 enabled. Specifically, localhost is resolved to ::1 when loopback addresses are permitted. This can lead to connection errors with services that bind their sockets to IPv4 addresses, such as 127.0.0.1 or 0.0.0.0.

[NyanKiyoshi]

Update surrounding this issue (if a reader is looking for mitigations, jump to the bottom of this reply):

Context

getaddrinfo() will always return IPv6 addresses first. Ordering is dictated by RFC 8305 section 3.1 (Happy Eyeballs v2).

requests-hardened's IP filtering feature (SSRF filter), currently only tries the first IP address. i.e., it only tries to connect to ::1, and aborts if the connection couldn't be established (in contrast to urllib3's behavior which first tries ::1 then tries 127.0.0.1).

The behavior is currently expected because we need to pin the IP address and we can't override the way urllib3 resolves IP addresses.

Potential Solutions

What we considered:

1. • Cycling through IP addresses: we could "knock" first to see if something is listening on the address (e.g., try to connect to [::1]:1234, and cycle through the IP address list until we find something that's online.
   • Issue: it would degrade the performance of the library (due to having to connect twice to the remote host) which isn't acceptable.

2. • Putting in place a retry loop: we could try to cycle through IP addresses by sending an HTTP request to the next IP address if the HTTP connection failed to be established by urllib3. e.g., something like this:

      for addr_res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
          af, socktype, proto, _canonname, socket_address = addr_res
          do_filter_ip(socket_address)
          
          try:
              return do_send_request(...)
          except ...:
               continue  # try the next IP address

   • Issue: this could be slow, hard to maintain, and would make it difficult to respect the Happy Eyeballs RFC.

3. • Change the logic for localhost: we could override our behavior for localhost, e.g. prefer IPv4 over IPv6.
   • Issues:
     • It complicates the code for one very specific use-case, and we do not wish to violate the Happy Eyeball RFC.
     • It may also cause to break other things, e.g. an application that only listens to the first IP address returned by getaddrinfo (thus ::1) instead of listening to both IPv6 and IPv4.

Outcome

Soon, we will be opening a discussion in urllib3 to try to explore more potential solutions:

• Maybe there is another way of handling this.
• Maybe we could add the capability to override create_connection() in urllib3

Mitigations

For users facing these issues, here is what you can do to mitigate them:

• Make your application listen to the IPv6 address, or both IPv6 and IPv4. Typically, if you tell your server to listen on localhost (instead of 127.0.0.1), it will listen to both ::1 and 127.0.0.1.
• Use http://127.0.0.1 with the request-hardened library instead of http://localhost

[reneleonhardt]

urllib3 doesn't allow discussions, have you opened an issue?
That code block is 10 years old, I guess it could be modernized 😄

[NyanKiyoshi]

Hi @reneleonhardt! I forgot to open a discussion about the issue on urllib3's project; I flagged it and will be doing it in the coming days

Thanks for the bump!

[NyanKiyoshi]

A ticket has been opened to discuss our idea and to discuss potential alternatives: urllib3/urllib3#3675