feat: add support for otel metrics (method 3)

[FrenchGithubUser]

No description provided.

[FrenchGithubUser]

It appears that quite a few things are done one way with prometheus_client and another way with otel. Some of my findings:

• gauge_variable.set_function(fn()) with prometheus_client has the equivalent in otel:

meter.create_observable_gauge(callbacks=[
            lambda options: [
                Observation(
                    fn(),
                    attributes,
                )
            ]
        ],)

This also means that we have to create the gauge where we set the callbacks, we can't set them later

• no direct equivalent for:

        with backfill_processing_after_timer.labels(
            **{SERVER_NAME_LABEL: self.server_name}
        ).time():
          await long_process()

this seems to be a decent equivalent:

        start = time.perf_counter()
        await long_process()
        backfill_processing_after_timer.record(
            time.perf_counter() - start, {SERVER_NAME_LABEL: self.server_name}
        )

however, if long_process fails, the rest of the function might not be executed

• metrics seem to be write-only, so this isn't really possible anymore:

    def _get_sample_with_name(self, metric: ObservableGauge, name: str) -> float:
        """For a prometheus metric get the value of the sample that has a
        matching "name" label.
        """
        for sample in next(iter(metric.collect())).samples:
            if sample.labels.get("name") == name:
                return sample.value

one way I tried doing it (not sure if it is properly functional)

    def _get_sample_with_name(self, metric: ObservableGauge, name: str) -> float:
        """For a prometheus metric get the value of the sample that has a
        matching "name" label.
        """
        for metric_family in REGISTRY.collect():
            for sample in metric_family.samples:
                if sample.labels.get("name") == name and sample.name == metric.name:
                    return sample.value

• no equivalent for track_in_progress():

        with self._number_in_flight_metric.track_inprogress():
            return await make_deferred_yieldable(d)

becomes something like:

        self._number_in_flight_metric.add(
            1, {"name": self._name, SERVER_NAME_LABEL: self.server_name}
        )
        res = await make_deferred_yieldable(d)
        self._number_in_flight_metric.add(
            -1, {"name": self._name, SERVER_NAME_LABEL: self.server_name}
        )
        return res

again, the last add() might not be called if the awaited function fails