feat(otlp): support metrics#257
Open
lukaslihotzki-f wants to merge 2 commits into
Open
Conversation
lukaslihotzki-f
force-pushed
the
ll/otlp-metrics
branch
2 times, most recently
from
9bb5402 to
9535398
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an OTLP metrics backend for Synapse by routing most metric instrument creation through a new backend-switching wrapper (synapse.metrics.instruments). When SYNAPSE_METRICS_BACKEND=otlp is set, metrics are exported via OpenTelemetry OTLP and Synapse avoids exposing the Prometheus /metrics endpoint.
Changes:
- Add a backend selector (
synapse.metrics.instruments) and an OTLP-backed implementation ofCounter/Gauge/Histogram(synapse.metrics._otel). - Switch many modules from
prometheus_clientimports to backend-agnostic imports fromsynapse.metrics.instruments. - Adjust app startup to skip Prometheus listener/resource when OTLP is enabled, and add an OTLP flush hook on shutdown; add a Poetry extra for OpenTelemetry metrics deps.
Reviewed changes
Copilot reviewed 54 out of 55 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| synapse/util/ratelimitutils.py | Swap Counter import to backend-agnostic instruments. |
| synapse/util/metrics.py | Swap Counter/CollectorRegistry imports to backend-agnostic instruments. |
| synapse/util/caches/deferred_cache.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/util/caches/init.py | Swap REGISTRY/Gauge imports to backend-agnostic instruments. |
| synapse/util/batching_queue.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/storage/databases/main/events.py | Swap Counter import to backend-agnostic instruments. |
| synapse/storage/databases/main/events_worker.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/storage/databases/main/event_federation.py | Swap Counter/Gauge imports to backend-agnostic instruments. |
| synapse/storage/database.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/storage/controllers/persist_events.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/state/init.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/rest/client/room.py | Swap Histogram import to backend-agnostic instruments. |
| synapse/replication/tcp/resource.py | Swap Counter import to backend-agnostic instruments. |
| synapse/replication/tcp/protocol.py | Swap Counter import to backend-agnostic instruments. |
| synapse/replication/tcp/handler.py | Swap Counter import to backend-agnostic instruments. |
| synapse/replication/tcp/external_cache.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/replication/http/_base.py | Swap Counter/Gauge imports to backend-agnostic instruments. |
| synapse/push/pusherpool.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/push/mailer.py | Swap Counter import to backend-agnostic instruments. |
| synapse/push/httppusher.py | Swap Counter import to backend-agnostic instruments. |
| synapse/push/bulk_push_rule_evaluator.py | Swap Counter import to backend-agnostic instruments. |
| synapse/notifier.py | Swap Counter import to backend-agnostic instruments. |
| synapse/metrics/jemalloc.py | Swap REGISTRY import to backend-agnostic instruments. |
| synapse/metrics/instruments.py | New: backend selector/re-export layer based on SYNAPSE_METRICS_BACKEND. |
| synapse/metrics/common_usage_metrics.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/metrics/background_process_metrics.py | Swap Counter/Gauge/REGISTRY imports to backend-agnostic instruments. |
| synapse/metrics/_reactor_metrics.py | Swap Histogram/REGISTRY imports to backend-agnostic instruments. |
| synapse/metrics/_otel.py | New: OTLP-backed instrument implementations + process/GC metrics + hook mechanism. |
| synapse/metrics/_gc.py | Swap Gauge/Histogram/REGISTRY imports to backend-agnostic instruments. |
| synapse/metrics/init.py | Use backend-agnostic instrument imports; keep Prometheus exposition utilities. |
| synapse/http/request_metrics.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/http/matrixfederationclient.py | Swap Counter import to backend-agnostic instruments. |
| synapse/http/client.py | Swap Counter import to backend-agnostic instruments. |
| synapse/handlers/sync.py | Swap Counter import to backend-agnostic instruments. |
| synapse/handlers/sliding_sync/init.py | Swap Histogram import to backend-agnostic instruments. |
| synapse/handlers/register.py | Swap Counter import to backend-agnostic instruments. |
| synapse/handlers/presence.py | Swap Counter import to backend-agnostic instruments. |
| synapse/handlers/federation.py | Swap Histogram import to backend-agnostic instruments. |
| synapse/handlers/federation_event.py | Swap Counter/Histogram imports to backend-agnostic instruments. |
| synapse/handlers/auth.py | Swap Counter import to backend-agnostic instruments. |
| synapse/handlers/appservice.py | Swap Counter import to backend-agnostic instruments. |
| synapse/federation/sender/transaction_manager.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/federation/sender/per_destination_queue.py | Swap Counter import to backend-agnostic instruments. |
| synapse/federation/sender/init.py | Swap Counter import to backend-agnostic instruments. |
| synapse/federation/federation_server.py | Swap Counter/Gauge/Histogram imports to backend-agnostic instruments. |
| synapse/federation/federation_client.py | Swap Counter import to backend-agnostic instruments. |
| synapse/config/metrics.py | Add config/env validation plumbing for selecting the metrics backend. |
| synapse/appservice/api.py | Swap Counter import to backend-agnostic instruments. |
| synapse/app/phone_stats_home.py | Swap Gauge import to backend-agnostic instruments. |
| synapse/app/homeserver.py | Skip exposing /metrics resource when OTLP backend is enabled. |
| synapse/app/generic_worker.py | Skip exposing /metrics resource when OTLP backend is enabled. |
| synapse/app/_base.py | Make Prometheus metrics listener a no-op in OTLP mode; flush OTLP on shutdown. |
| synapse/api/auth/init.py | Swap Histogram import to backend-agnostic instruments. |
| pyproject.toml | Add opentelemetry-metrics extra. |
| poetry.lock | Wire new extra into dependency markers/extras. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| from prometheus_client import Gauge | ||
|
|
||
| from twisted.internet import defer |
Comment on lines
+57
to
73
| env_backend = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
| yaml_backend = config.get("metrics_backend", "prometheus").lower() | ||
| if yaml_backend not in ("prometheus", "otlp"): | ||
| raise ConfigError( | ||
| "Invalid metrics_backend %r (must be 'prometheus' or 'otlp')" | ||
| % yaml_backend | ||
| ) | ||
| if yaml_backend == "otlp" and env_backend != "otlp": | ||
| raise ConfigError( | ||
| "metrics_backend is set to 'otlp' in config but the " | ||
| "SYNAPSE_METRICS_BACKEND env var is %r. The env var must also " | ||
| "be set to 'otlp' because metrics are initialised at import " | ||
| "time (before config is read)." % env_backend | ||
| ) | ||
| self.metrics_backend: str = env_backend | ||
|
|
||
| self.report_stats = config.get("report_stats", None) |
Comment on lines
+581
to
+587
|
|
||
| _meter.create_observable_gauge( | ||
| "python_gc_counts", | ||
| callbacks=[_observe_gc_counts], | ||
| description="GC object counts per generation.", | ||
| ) | ||
|
|
Comment on lines
+29
to
+41
| METRICS_BACKEND = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
|
|
||
| if METRICS_BACKEND == "otlp": | ||
| try: | ||
| from synapse.metrics._otel import ( | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) # noqa: F401 | ||
| except ImportError: | ||
| raise ImportError( |
lukaslihotzki-f
force-pushed
the
ll/otlp-metrics
branch
from
9535398 to
72a8381
Compare
lukaslihotzki-f
force-pushed
the
ll/otlp-metrics
branch
from
72a8381 to
0183c13
Compare
Comment on lines
+183
to
+193
| def _callback(options: Any) -> Sequence[Observation]: | ||
| _run_pre_collect_hooks() | ||
| obs: list[Observation] = [] | ||
| for key, value in list(self._values.items()): | ||
| obs.append(Observation(value, dict(key))) | ||
| for attrs, fn in list(self._functions.values()): | ||
| try: | ||
| obs.append(Observation(fn(), attrs)) | ||
| except Exception: | ||
| pass | ||
| return obs |
Comment on lines
+201
to
+206
| def labels(self, *args: str, **kwargs: str) -> _GaugeChild: | ||
| if args: | ||
| attrs = dict(zip(self._labelnames, args)) | ||
| else: | ||
| attrs = kwargs | ||
| key = tuple(sorted(attrs.items())) |
Comment on lines
+606
to
+611
| class Registry: | ||
| def register(self, other): | ||
| def _drain(): | ||
| for _ in other.collect(): | ||
| pass | ||
|
|
Comment on lines
+54
to
+71
| # The metrics backend is authoritative from the env var because metric | ||
| # objects are created at import time (before config is read). The yaml | ||
| # value is accepted for documentation / validation purposes. | ||
| env_backend = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
| yaml_backend = config.get("metrics_backend", "prometheus").lower() | ||
| if yaml_backend not in ("prometheus", "otlp"): | ||
| raise ConfigError( | ||
| "Invalid metrics_backend %r (must be 'prometheus' or 'otlp')" | ||
| % yaml_backend | ||
| ) | ||
| if yaml_backend == "otlp" and env_backend != "otlp": | ||
| raise ConfigError( | ||
| "metrics_backend is set to 'otlp' in config but the " | ||
| "SYNAPSE_METRICS_BACKEND env var is %r. The env var must also " | ||
| "be set to 'otlp' because metrics are initialised at import " | ||
| "time (before config is read)." % env_backend | ||
| ) | ||
| self.metrics_backend: str = env_backend |
Comment on lines
+31
to
+45
| if METRICS_BACKEND == "otlp": | ||
| try: | ||
| from synapse.metrics._otel import ( | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) # noqa: F401 | ||
| except ImportError: | ||
| raise ImportError( | ||
| "SYNAPSE_METRICS_BACKEND is set to 'otlp' but the required " | ||
| "OpenTelemetry packages are not installed. " | ||
| "Install them with: pip install matrix-synapse[opentelemetry-metrics]" | ||
| ) |
Comment on lines
+14
to
+53
| """ | ||
| Re-exports ``Counter``, ``Gauge`` and ``Histogram`` from the appropriate | ||
| backend. | ||
|
|
||
| When the environment variable ``SYNAPSE_METRICS_BACKEND`` is set to ``otlp``, | ||
| the classes come from :mod:`synapse.metrics._otel` and measurements are | ||
| exported via OTLP (configured through the standard ``OTEL_*`` environment | ||
| variables). | ||
|
|
||
| Otherwise the classes are the stock ``prometheus_client`` implementations and | ||
| metrics are exposed on the Prometheus scrape endpoint as usual. | ||
| """ | ||
|
|
||
| import os | ||
|
|
||
| METRICS_BACKEND = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
|
|
||
| if METRICS_BACKEND == "otlp": | ||
| try: | ||
| from synapse.metrics._otel import ( | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) # noqa: F401 | ||
| except ImportError: | ||
| raise ImportError( | ||
| "SYNAPSE_METRICS_BACKEND is set to 'otlp' but the required " | ||
| "OpenTelemetry packages are not installed. " | ||
| "Install them with: pip install matrix-synapse[opentelemetry-metrics]" | ||
| ) | ||
| else: | ||
| from prometheus_client import ( # noqa: F401 | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) |
Member
|
Thanks! We'll take a look at this. Could you please run |
|
|
||
| Internally this creates an OTel *ObservableGauge* whose callback returns | ||
| the most recently stored values. This is a natural fit because Synapse | ||
| gauges are written to sporadically (e.g. from hooks) and read periodically. |
Member
There was a problem hiding this comment.
Suggested change
| gauges are written to sporadically (e.g. from hooks) and read periodically. | |
| gauges are written sporadically (e.g. from hooks) and read periodically. |
| # Global OTel meter – created eagerly so that module-level metric definitions | ||
| # (the common pattern in Synapse) can use it immediately. | ||
|
|
||
| _resource = Resource.create(attributes={"service.name": "synapse", "server_name": os.environ.get('SERVER_NAME', '')}) |
Comment on lines
+118
to
+134
| def set(self, value: float) -> None: | ||
| self._parent._values[self._key] = float(value) | ||
|
|
||
| def inc(self, amount: float = 1) -> None: | ||
| k = self._key | ||
| vals = self._parent._values | ||
| vals[k] = vals.get(k, 0.0) + float(amount) | ||
|
|
||
| def dec(self, amount: float = 1) -> None: | ||
| k = self._key | ||
| vals = self._parent._values | ||
| vals[k] = vals.get(k, 0.0) - float(amount) | ||
|
|
||
| def set_function(self, fn: Callable[[], float]) -> None: | ||
| self._parent._functions[self._key] = (self._attrs, fn) | ||
| self._parent._values.pop(self._key, None) | ||
|
|
Comment on lines
+183
to
+193
| def _callback(options: Any) -> Sequence[Observation]: | ||
| _run_pre_collect_hooks() | ||
| obs: list[Observation] = [] | ||
| for key, value in list(self._values.items()): | ||
| obs.append(Observation(value, dict(key))) | ||
| for attrs, fn in list(self._functions.values()): | ||
| try: | ||
| obs.append(Observation(fn(), attrs)) | ||
| except Exception: | ||
| pass | ||
| return obs |
Comment on lines
+201
to
+206
| def labels(self, *args: str, **kwargs: str) -> _GaugeChild: | ||
| if args: | ||
| attrs = dict(zip(self._labelnames, args)) | ||
| else: | ||
| attrs = kwargs | ||
| key = tuple(sorted(attrs.items())) |
Comment on lines
+31
to
+45
| if METRICS_BACKEND == "otlp": | ||
| try: | ||
| from synapse.metrics._otel import ( | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) # noqa: F401 | ||
| except ImportError: | ||
| raise ImportError( | ||
| "SYNAPSE_METRICS_BACKEND is set to 'otlp' but the required " | ||
| "OpenTelemetry packages are not installed. " | ||
| "Install them with: pip install matrix-synapse[opentelemetry-metrics]" | ||
| ) |
Comment on lines
+57
to
+71
| env_backend = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
| yaml_backend = config.get("metrics_backend", "prometheus").lower() | ||
| if yaml_backend not in ("prometheus", "otlp"): | ||
| raise ConfigError( | ||
| "Invalid metrics_backend %r (must be 'prometheus' or 'otlp')" | ||
| % yaml_backend | ||
| ) | ||
| if yaml_backend == "otlp" and env_backend != "otlp": | ||
| raise ConfigError( | ||
| "metrics_backend is set to 'otlp' in config but the " | ||
| "SYNAPSE_METRICS_BACKEND env var is %r. The env var must also " | ||
| "be set to 'otlp' because metrics are initialised at import " | ||
| "time (before config is read)." % env_backend | ||
| ) | ||
| self.metrics_backend: str = env_backend |
Comment on lines
+60
to
+66
| from synapse.metrics.instruments import ( | ||
| REGISTRY, | ||
| CollectorRegistry, | ||
| Counter, | ||
| Gauge, | ||
| Histogram, | ||
| ) |
Comment on lines
+55
to
+70
| # objects are created at import time (before config is read). The yaml | ||
| # value is accepted for documentation / validation purposes. | ||
| env_backend = os.environ.get("SYNAPSE_METRICS_BACKEND", "prometheus").lower() | ||
| yaml_backend = config.get("metrics_backend", "prometheus").lower() | ||
| if yaml_backend not in ("prometheus", "otlp"): | ||
| raise ConfigError( | ||
| "Invalid metrics_backend %r (must be 'prometheus' or 'otlp')" | ||
| % yaml_backend | ||
| ) | ||
| if yaml_backend == "otlp" and env_backend != "otlp": | ||
| raise ConfigError( | ||
| "metrics_backend is set to 'otlp' in config but the " | ||
| "SYNAPSE_METRICS_BACKEND env var is %r. The env var must also " | ||
| "be set to 'otlp' because metrics are initialised at import " | ||
| "time (before config is read)." % env_backend | ||
| ) |
lukaslihotzki-f
force-pushed
the
ll/otlp-metrics
branch
from
8ce417a to
9b23a1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds an option to send the prometheus metrics over OTLP instead.
Pull Request Checklist