Issues with Nomad services.

[coltonshipley]

So I'm trying out this with my homelab. I have my UDM Pro setup with a BGP config. All that seems to be working. I have GoCast running as a job on my ingress nodes. When I assign a VIP, it does get attached to loopback as intended and pings start to work. Howerver, pings/udp/dns are the only things I'm able to get to work. Anything that is tcp doesn't seem to work. I have a sample whoami job setup as follows:

job "whoami" {
  constraint {
    attribute = "${meta.nodetype}"
    value = "ingress"
  }
  datacenters = ["dc1"]
  type        = "service"

  group "whoami" {
    network {
      port "http" {}
    }

    service {
      name = "whoami"
      port = "http"

      tags = [
        "enable_gocast",
        "gocast_vip=192.168.200.101/32",
        "gocast_monitor=consul",
        "gocast_nat=tcp:${NOMAD_PORT_http}:${NOMAD_PORT_http}",
      ]

      check {
        type     = "http"
        path     = "/"
        interval = "2s"
        timeout  = "2s"
      }
    }

    task "server" {
      env {
        WHOAMI_PORT_NUMBER    = "${NOMAD_PORT_http}"
      }

      driver = "docker"

      config {
        image = "traefik/whoami"
        ports = ["http"]
      }
    }
  }
}

However, when I try to hit my vip in this case 192.168.200.101:23722 I can see traffic come in with a tcp dump but it never seems to send anything back to the browser.

The actual goal here is to use this with traefik to create an ha instance of my ingress loadbalancer.

I'm so close and have tried just about everything I can think of. Has anyone successfully had this working with Nomad to help?

[coltonshipley]

Also, another thing I've noticed that should maybe be it's own issue....

If you update a running job and change the gocast tags. Like the nat tags, GoCast never updates the nat rules because 'the app already exists' It seems you have to kill gocast and start it up again and let everything re-register with the updated tag data. Part of the polling process should check the tags and make sure they are the same and update if they have changed.

[Vaelatern]

Have you tried setting the provider in your service block to consul? It looks like you are registering a nomad service not a consul service.

[coltonshipley]

Have you tried setting the provider in your service block to consul? It looks like you are registering a nomad service not a consul service.

The default is consul. The service and tags are being registered in consul. GoCast is indeed attaching the vip on the tag to the lo interface and creating the nat rules as far as I can tell. Looking at the tcp dump I can see requests from my desktop to the vip. It just seems to not make it into the container. This doesn't change with the container in host networking mode either.

[coltonshipley]

So, upon further investigation it seems that the request is never making it to the container. So the nat rules are created on the host (from what I can tell), the host is seeing the traffic on the eth0 interface after it's forwarded to the physical ip address. But it is not making it to the container after that.

[Vaelatern]

Here is my gocast group on my system:

        group "gocast" {
                network {
                        mode = "host"
                        port "gocast" {}
		}

                task "gocast" {
                        driver = "docker"
                        config {
                                network_mode = "host"
                                image = "docker.io/mayuresh82/gocast:${var.gocast_tag}"
                                cap_add = ["net_admin", "net_raw"]
                                args = ["-config=/local/gocast.yml"]
                        }
                        resources {
                                cpu = 100
                                memory = 32
                        }
                        template {
                                destination = "local/gocast.yml"
                                data = <<-EOF
                                        ${file("./gocast_config.yml")}
                                        EOF
                        }
                }
        }

[coltonshipley]

Okay, so it works. My issue before seems the be an asymmetrical routing issue across vlans. When I put the vip in the same network all works well! This is awesome. Good work. I like this way better than a keepalived solution.

I'll post again on the other issue. When I updated the vip the gocast registration did change. I'll try updating the nat tags and see if that also triggers an update. I think I wasn't waiting long enough before.

[Vaelatern]

Can you share more of your setup for future explorers?