Slow performance whilst tracing

[jamescrowley]

We enabled OpenCensus in Google Cloud in our PHP 7 application - and experimented initially with tracing every request. This consistently added 1 second of latency to our loading time (on an empty page that did nothing else). While we recognise the recommendation to enable sampling, 1 seconds seems excessive to the point we wouldn't want to enable even in sampling mode.

Is this expected performance or is something strange occurring here?

[nenad]

Definitely not expected. In our applications, there isn't any recognizable latency introduced with this package. Have you tried profiling to see what exactly is causing this slowness?

[jamescrowley]

@nenad As it only happens in GAE, rather than locally, I've been unable to profile or trace to understand what's going on.

We switched:

Tracer::start($exporter, [
    'sampler' => new AlwaysSampleSampler()
]);

(which added around 1 second per request) to

Tracer::start($exporter, [
    'sampler' => new NeverSampleSampler();
]);

which you'd think would then be much faster but not trace... except we still get traces in the app engine environment... even with NeverSampleSampler configured, but now without the delay.

Happy to poke more if you can point me in the right direction.

[nenad]

I don’t have my computer at the moment to dig deeper, but could it be the exporter? Try setting a NoopExporter just to try and eliminate out this doubt.

[basvanbeek]

Which exporter are you using? My suspicion is that establishing the connection to the stats / trace backend is taking too long. This connection happens regardless of span sampling.

Have you tried the Daemon Client exporter together with Go Daemon? This is available in master (has not been released yet) when Go Daemon is deployed together with the PHP server you will not see these slowdowns as unix sockets are used for transferring trace and stats data to the Daemon. If compiling in the extension from master you will even see more speed up as transfer of trace and stats data is taken outside of the request path and handled in a separate long lived thread when using the Daemon exporter.

[jamescrowley]

@nenad I'm using the StackdriverExporter right now. I'll see if I can find a window to test this without disrupting our staging environment too much.

@basvanbeek I can't deploy the Go Daemon unfortunately as we're in the app engine environment.