Skip to content

Docker socket "broken pipe" failures: runner v2.329.0 + Docker v29.0.1 in Kubernetes #4117

@varadajadhav

Description

@varadajadhav

Describe the bug
We are experiencing intermittent Docker socket communication failures across multiple self-hosted GitHub Actions runner scale sets running in Kubernetes with Docker-in-Docker (dind).

Issue started immediately after upgrading from runner v2.328.0 → v2.329.0

The Problem:

  1. Upgraded runner from v2.328.0 to v2.329.0 around late October 2025
  2. Errors began appearing on October 31, 2025 (shortly after upgrade)
  3. No issues with v2.328.0 - runners were stable before the upgrade
  4. Random workflow failures with "broken pipe" errors when the runner tries to attach to container exec stdout streams
  5. 5-10% of jobs fail randomly during peak hours

Error Pattern:
The dind container logs show socket write failures when trying to communicate with /var/run/docker.sock. All scale sets experience errors simultaneously during spike events, suggesting this is more than just resource exhaustion.

Runner Version and Platform
Runner:
Version: v2.329.0
Deployment Method: actions-runner-controller (gha-runner-scale-set)
Controller Version: 0.12.1
Chart Version: 0.12.1

Kubernetes:
Platform: Amazon EKS
EKS Version: 1.33
EKS AMI: 1.33.5-20251023 (October 2025 release)

Docker (in dind container):
Docker Engine: 29.0.1 (built November 14, 2025)
API version: 1.52
containerd: v2.1.5
runc: 1.3.3
Image: docker:dind (latest tag from Docker Hub)

Runner Logs
Error Messages - Primary Error (from dind container stderr): Error running exec <container_id> in container: exec attach failed:
error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe

Secondary Error: http2: server: error reading preface from client @: read unix /var/run/docker.sock->@: read: connection reset by peer

Sample Logs from dind Container
2025-11-14T07:01:11.063364452Z level=error msg="Error running exec 2bdbf255470b5a38 in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe"
2025-11-14T07:04:31.631887068Z level=error msg="Error running exec 06cffe9160ae9 in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe"
2025-11-14T07:04:34.408Z level=warning msg="ShouldRestart failed, container will not be restarted" daemonShuttingDown=false error="" execDuration=3m10.311722009s exitStatus="{1 2025-11-14 07:04:34.390005475 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
2025-11-14T07:04:54.908Z http2: server: error reading preface from client @: read unix /var/run/docker.sock->@: read: connection reset by peer

Runner and Worker's Diagnostic Logs
No errors appear in the runner container logs. All errors are only visible in the dind (Docker-in-Docker) container stderr logs. The runner container shows normal operation.

Questions

  1. What changed in v2.329.0 related to Docker socket handling?
  2. Is there a known regression in v2.329.0 with dind setups?
  3. Are there recommended socket configurations for v2.329.0 with dind?
  4. Could Docker v29.0.1 (Nov 14, 2025) + runner v2.329.0 combination be problematic?

We would appreciate guidance on!

Thank you

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions