OpenTelemetry: support AWS X-Ray Propagator

[universam1]

Is your feature request related to a problem? Please describe.
OTEL support arrived #3627 (thank you @AlexanderYastrebov !) but misses support for AWS X-Ray Propagator.
The OTEL_PROPAGATORS=xray is currently ignored, the ALB's X-Amzn-Trace-Id headers are not extracted by Skipper, so Traces effectively "start" at Skipper level, not at ALB.

Describe the solution you would like
Adding support for https://pkg.go.dev/go.opentelemetry.io/contrib/propagators/aws/xray propagator

Would you like to work on it?
Yes - with guidance

[szuecs]

I would say just try to add it

[AlexanderYastrebov]

Hi @universam1, thanks for opening an issue. Please note that OpenTelemetry support was just added and we'll likely iron out some details down the line.

I would say just try to add it

I think it should be as easy as #3664

[universam1]

@AlexanderYastrebov I thought so, but this is 🚀 amazing fast!

[AlexanderYastrebov]

Hm, I tried to test it and discovered that it is actually registered here https://github.com/open-telemetry/opentelemetry-go-contrib/blob/613f80e3ffd5cb8de28fcce63fa7dfdf9645bc10/propagators/autoprop/registry.go#L38-L39

[AlexanderYastrebov]

Skipper won't start with an unsupported propagator (e.g. foo):

...
[APP]FATA[0000] failed to setup OpenTelemetry: unknown propagator: foo

and it starts with xray so I think something else is missing.

[AlexanderYastrebov]

The OTEL_PROPAGATORS=xray is currently ignored,

Could you provide a flag value (and env variables for that matter) of how you init OTEL?
Just in case, even if you want to use env variables to configure, you still need to tell Skipper to use OTEL via:

skipper -opentelemetry={}

see https://github.com/zalando/skipper/blob/master/docs/operation/operation.md#opentelemetry

[universam1]

discovered that it is actually registered here

OK thank you for this insight, I missed that!

skipper -opentelemetry={}

see https://github.com/zalando/skipper/blob/master/docs/operation/operation.md#opentelemetry

@AlexanderYastrebov There is a typo in the documentation, it is actually with a hyphen:

skipper -open-telemetry={}

Could you provide a flag value (and env variables for that matter) of how you init OTEL?

        - name: OTEL_TRACES_EXPORTER
          value: "otlp"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "grpc"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://coroot-coroot.coroot.svc.cluster.local:4317"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=skipper-ingress"
        - name: OTEL_PROPAGATORS
          value: "tracecontext,baggage,xray"

So the question remains, why do the traces effectively "start" at Skipper level, not at ALB? I have traced that the X-Amzn-Trace-Id is present.

[universam1]

Found two issues:

1. the X-ray trace ID is not propagated, but a random fake one is generated:

Skipper without the -open-telemetry flag forwards the original X-Amzn-Trace-Id header from ALB:

x-amzn-trace-id: Root=1-68ef9936-5f4df7cb590e654d2f659b2c
x-amzn-trace-id: Root=1-68ef9933-777af7c96b4ea44b68a0dca6
x-amzn-trace-id: Root=1-68ef992d-214de1ac64b26cc315325ee1

Note the second part is the Unix Epoch time that increments.

Skipper with the -open-telemetry flag creates a fake X-Amzn-Trace-Id header:

"traceparent": "00-e3a50be903046347cccf28c10f4ed825-7f385bb4dc11beb0-01",
"x-amzn-trace-id": "Root=1-e3a50be9-03046347cccf28c10f4ed825;Parent=7f385bb4dc11beb0;Sampled=1",

Note the format of header is completelly different. And the timestamp is something random.

This x-ray trace id is not forwarded to OTEL at all.

It sounds like that Skipper generates a trace ID regardless of the xray support.

[AlexanderYastrebov]

There is a typo in the documentation, it is actually with a hyphen

Thanks, created #3667