Skip to content

perf(postgres): Optimize table#175

Open
devsjc wants to merge 2 commits into
mainfrom
devsjc/optimize-storage
Open

perf(postgres): Optimize table#175
devsjc wants to merge 2 commits into
mainfrom
devsjc/optimize-storage

Conversation

@devsjc

@devsjc devsjc commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Modifies the schema of the predicted generation values table for optimal storage. Reduces table size by about 40%.

Contribution Checklist

  • Have you followed the Open Climate Fix Contribution Guidelines?
  • Have you referenced the Issue this PR addresses, where applicable?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added a summary of the changes?
  • Have you written new tests for your changes, where applicable?
  • Have you successfully run make lint with your changes locally?
  • Have you successfully run make test with your changes locally?

Warning

PRs may be closed if all the above boxes are not checked.

@github-actions

Copy link
Copy Markdown

Benchmark Results

Benchmark results
?   	github.com/openclimatefix/data-platform/cmd	[no test files]
?   	github.com/openclimatefix/data-platform/internal/gen/ocf/dp	[no test files]
?   	github.com/openclimatefix/data-platform/internal/interceptors	[no test files]
PASS
ok  	github.com/openclimatefix/data-platform/internal/server/dummy	0.005s
{"level":"debug","time":"2026-06-16T12:18:39Z","message":"Completed migrations"}
goos: linux
goarch: amd64
pkg: github.com/openclimatefix/data-platform/internal/server/postgres
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkPostgresClient/small/GetForecastAsTimeseries-4         	      43	  26461048 ns/op
BenchmarkPostgresClient/small/GetForecastAtTimestamp-4          	     147	   7739708 ns/op
BenchmarkPostgresClient/small/GetObservationsAsTimeseries-4     	     852	   1359624 ns/op
BenchmarkPostgresClient/small/CreateForecast-4                  	      92	  12620555 ns/op
BenchmarkPostgresClient/small/StreamForecastData-4              	      13	  87458785 ns/op
PASS
ok  	github.com/openclimatefix/data-platform/internal/server/postgres	71.435s
?   	github.com/openclimatefix/data-platform/internal/server/postgres/gen	[no test files]
Benchmark vs base branch
goos: linux
goarch: amd64
pkg: github.com/openclimatefix/data-platform/internal/server/postgres
cpu: AMD EPYC 7763 64-Core Processor                
                                                   │ bench-devsjc-optimize-storage.txt │
                                                   │              sec/op               │
PostgresClient/small/GetForecastAsTimeseries-4                            26.46m ± ∞ ¹
PostgresClient/small/GetForecastAtTimestamp-4                             7.740m ± ∞ ¹
PostgresClient/small/GetObservationsAsTimeseries-4                        1.360m ± ∞ ¹
PostgresClient/small/CreateForecast-4                                     12.62m ± ∞ ¹
PostgresClient/small/StreamForecastData-4                                 87.46m ± ∞ ¹
geomean                                                                   12.52m
¹ need >= 6 samples for confidence interval at level 0.95

cpu: AMD EPYC 9V74 80-Core Processor                
                                                   │ bench-main.txt │
                                                   │     sec/op     │
PostgresClient/small/GetForecastAsTimeseries-4         18.67m ± ∞ ¹
PostgresClient/small/GetForecastAtTimestamp-4          4.202m ± ∞ ¹
PostgresClient/small/GetObservationsAsTimeseries-4     1.205m ± ∞ ¹
PostgresClient/small/CreateForecast-4                  11.74m ± ∞ ¹
PostgresClient/small/StreamForecastData-4              102.4m ± ∞ ¹
geomean                                                10.26m
¹ need >= 6 samples for confidence interval at level 0.95

This was a field that as of yet wasn't being used. In order to make sure
this very large table doesn't take up more storage space than it needs
to, I'm removing this dynamic column. Any value-related metadata can
live in the forecast metadata, I'm pretty comfident.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant