Example nomad job

[vacquah]

Hi there - I have read through the blog post and everything here but still not clear to me exactly how the nomad job should be set up - or the overall integration with a nomad / consul cluster. I'd like to test this to enable external access to services running in a nomad cluster, but not sure how to how to do that. Providing an example nomad job here and pointers as to how to integrate this with, for instance, bgp on pfsense will be great!

For the nomad job below, I get this error message:

Failed to create container configuration for image "mayuresh82/gocast:latest" ("sha256..."): driver does not allow the following capabilities: net_admin

Nomad Job: ( my volumes setup is wrong but not sure what it should be )

job "gocast" {
  datacenters = ["dc1"] 
  type        = "system"   

  update {
    stagger      = "30s"
    max_parallel = 2
  }  

  group "gocast" {

    network {
      port "tcp" {
        static  = 9080
      }
    }

    task "gocast-container" {
      driver = "docker"

      config {
        image = "mayuresh82/gocast:latest"
        ports = ["tcp"]
        force_pull = true
        network_mode = "host"
        privileged   = true
        cap_add      = ["net_admin"]

        args = [
          "--config=local/config.yml",
        ]
      }

      resources {
        cpu    = 500   # CPU resources (in MHz)
        memory = 256   # Memory resources (in MB)
      }

      service {
        provider = "consul"
      }

      template {
        change_mode   = "signal" 
        change_signal = "SIGHUP" 
        destination   = "local/config.yml" 
        data = <<EOH
---
agent:
  listen_addr: :9080
  monitor_interval: 10s
  cleanup_timer: 15m
  consul_addr: http://localhost:8500/v1
  consul_query_interval: 1m

bgp:
  local_as: 65000
  remote_as: 64999
  communities:
    - 64999:65000
  origin: igp

EOH
      }
    }
  }
}

And for an MQTT server I have running on the nomad cluster I have:

enable_gocast=true
gocast_vip=10.1.1.1/32
gocast_monitor=consul

Now do I need to do anything on my pfsense router?

[philipcristiano]

net_admin needs to be allowed by the Nomad installation.

For my Nomad client config I have

plugin "docker" {
    config {
       ...
        allow_caps = [
          "audit_write",
          "chown",
          "dac_override",
          "fowner",
          "fsetid",
          "kill",
          "mknod",
          "net_admin",
          "net_bind_service",
          "net_raw",
          "setfcap",
          "setgid",
          "setpcap",
          "setuid",
          "sys_chroot"
       ]
    }
}

I believe setting allow_caps overwrites the Nomad defaults, to the others are mostly the defaults in Nomad that needed to be readded.

Separately, if you pass in the arg "-config=/local/config.yaml" for the Nomad job you can avoid the volumes portion of the task config.

[vacquah]

@philipcristiano I managed to deploy the nomad job using the info above, but it fails on a raspberrypi. I have a cluster with two client nodes on amd64 machines and a third client on a raspberrypi. The two amd64 machines install successfully whilst the pi fails when deploying as a system job.

I assume this wont work on a raspberrypi ?

[philipcristiano]

The current tags are only for linux/amd64 so for raspi you would need to find/build a multi-arch image (to use the same image for the job) or separate images with system architecture constraints.

You can open an issue for more architecture builds, maybe after #18/#19 with automated Docker releases it would be easier.

[vacquah]

This is a bit over my head ( building multi arch images ). But will take a stab at it. I will open an issue for that too. Any tips on how to set things up on the pfsense side of things ( using their frr package ?)

[philipcristiano]

@vacquah I added linux/arm64 for #18, if you're up for running arbitrary code on your systems....

https://hub.docker.com/layers/philipcristiano/gocast/sha-0bb6fae/images/sha256-15e34326a25a6423ad96db46f498828a00b80e6f2d5696e030dbe52e70fd720e?context=explore is the image I built from this PR (in my user-fork where I have credentials to push to Docker Hub).

I'd recommend taking that branch and building/pushing on your own vs trusting my image.

I'm also not sure if a linux/arm64 will work on raspis, I think depends on the board version and OS combo.

Any tips on how to set things up on the pfsense side of things ( using their frr package ?)

I don't run pfsense so not sure what the setup on that side would look like.

[mayuresh82]

thanks for the help @philipcristiano . Ill look into setting up GHA for automatic builds on release again. Docker hub has been a bit of a pain to deal with in the past.