Skip to content

Add premium geo dimension columns to usage events #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lohanidamodar merged 1 commit into from
Jul 8, 2026
Merged

Add premium geo dimension columns to usage events #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,25 @@
# Changelog

## Unreleased — resourceType rename + queued time + ip dim + gauge fill
## 0.10.0 — premium geo dimensions

### Added

- Nine event-only premium geo dimensions in `Metric::EVENT_COLUMNS`
and `Metric::getEventSchema()`: `city`, `continentCode`,
`subdivisions`, `isp`, `autonomousSystemNumber`,
`autonomousSystemOrganization`, `connectionType`,
`connectionUsageType`, `connectionOrganization`. All are optional
strings. In ClickHouse the lower-cardinality dims (`continentCode`,
`subdivisions`, `connectionType`, `connectionUsageType`,
`autonomousSystemNumber`) are `LowCardinality(Nullable(String))`;
the high-cardinality dims (`city`, `isp`,
`autonomousSystemOrganization`, `connectionOrganization`) are plain
`Nullable(String)`. Existing tables auto-materialize the columns on
`setup()` via the `ADD COLUMN IF NOT EXISTS` path. Gauges are
unchanged; these columns are not added to the primary key or
indexes.

## 0.9.0 — resourceType rename + queued time + ip dim + gauge fill

### Breaking

Expand Down
7 changes: 7 additions & 0 deletions src/Usage/Adapter/ClickHouse.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1148,6 +1148,10 @@ private function getColumnType(string $id, string $type = 'event'): string
'clientEngine', 'clientEngineVersion',
'deviceName', 'deviceBrand', 'deviceModel',
'hostname', 'ip',
// premium geo (lower-cardinality only; city/isp/AS org/connection org
// are high-cardinality and intentionally fall through to Nullable(String))
'continentCode', 'subdivisions', 'connectionType',
'connectionUsageType', 'autonomousSystemNumber',
];

if (in_array($id, $lowCardinality, true)) {
Expand Down Expand Up @@ -1188,6 +1192,9 @@ private function getColumnCodec(string $id): string
'resourceId', 'resourceInternalId',
'teamId', 'teamInternalId',
'osVersion', 'clientVersion', 'clientEngineVersion', 'deviceModel',
'city', 'continentCode', 'subdivisions', 'isp',
'autonomousSystemNumber', 'autonomousSystemOrganization',
'connectionType', 'connectionUsageType', 'connectionOrganization',
];

if (in_array($id, $zstdColumns, true)) {
Expand Down
17 changes: 17 additions & 0 deletions src/Usage/Metric.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ class Metric extends ArrayObject
'service', 'resourceType', 'resourceId', 'resourceInternalId',
'teamId', 'teamInternalId',
'country', 'region', 'hostname', 'ip',
// premium geo
'city', 'continentCode', 'subdivisions',
'isp', 'autonomousSystemNumber', 'autonomousSystemOrganization',
'connectionType', 'connectionUsageType', 'connectionOrganization',
'osCode', 'osName', 'osVersion',
'clientType', 'clientCode', 'clientName', 'clientVersion',
'clientEngine', 'clientEngineVersion',
Expand Down Expand Up @@ -82,6 +86,9 @@ class Metric extends ArrayObject
* - resourceType / resourceId / resourceInternalId: resource identity
* - teamId / teamInternalId: owning team identity
* - country / region / hostname / ip: geographic + caller origin
* - city / continentCode / subdivisions: premium geo location fields
* - isp / autonomousSystemNumber / autonomousSystemOrganization: premium network origin
* - connectionType / connectionUsageType / connectionOrganization: premium connection intelligence
* - osCode / osName / osVersion: parsed user-agent OS fields
* - clientType / clientCode / clientName / clientVersion: parsed client
* - clientEngine / clientEngineVersion: parsed client engine
Expand Down Expand Up @@ -628,6 +635,16 @@ public static function getEventSchema(): array
$stringColumn('region', 64),
$stringColumn('hostname', 255),
$stringColumn('ip', 45),
// premium geo
$stringColumn('city', 256),
$stringColumn('continentCode', 2),
$stringColumn('subdivisions', 256),
$stringColumn('isp', 256),
$stringColumn('autonomousSystemNumber', 255),

@greptile-apps greptile-apps Bot Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 autonomousSystemNumber is defined with size 255 while every other new premium geo string column in the same block uses 256. Given that the other fields in this group (city, subdivisions, isp, etc.) all use 256, this looks like an accidental mismatch rather than a deliberate choice — ASNs are short strings, so the difference has no practical impact, but the inconsistency is worth correcting for schema coherence.

Suggested change
$stringColumn('autonomousSystemNumber', 255),
$stringColumn('autonomousSystemNumber', 256),
Prompt To Fix With AI 
This is a comment left during a code review.
Path: src/Usage/Metric.php
Line: 643

Comment:
`autonomousSystemNumber` is defined with `size` 255 while every other new premium geo string column in the same block uses 256. Given that the other fields in this group (`city`, `subdivisions`, `isp`, etc.) all use 256, this looks like an accidental mismatch rather than a deliberate choice — ASNs are short strings, so the difference has no practical impact, but the inconsistency is worth correcting for schema coherence.

```suggestion
            $stringColumn('autonomousSystemNumber', 256),
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

$stringColumn('autonomousSystemOrganization', 256),
$stringColumn('connectionType', 256),
$stringColumn('connectionUsageType', 256),
$stringColumn('connectionOrganization', 256),
$stringColumn('osCode', 256),
$stringColumn('osName', 256),
$stringColumn('osVersion', 255),
Expand Down
75 changes: 75 additions & 0 deletions tests/Usage/Adapter/ClickHouseColumnTypeTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
<?php

namespace Utopia\Tests\Adapter;

use PHPUnit\Framework\TestCase;
use ReflectionMethod;
use Utopia\Usage\Adapter\ClickHouse as ClickHouseAdapter;

/**
* Pure unit test (no live ClickHouse) for the private column-type mapping.
* The adapter constructor builds a lazy transport client and never connects,
* so getColumnType() can be exercised via reflection to assert the
* LowCardinality vs Nullable(String) decision per dimension column.
*/
class ClickHouseColumnTypeTest extends TestCase
{
private ClickHouseAdapter $adapter;

private ReflectionMethod $getColumnType;

protected function setUp(): void
{
$this->adapter = new ClickHouseAdapter(
'clickhouse',
'default',
'',
8123,
false,
namespace: 'utopia_usage_coltype',
);

$this->getColumnType = new ReflectionMethod(ClickHouseAdapter::class, 'getColumnType');
$this->getColumnType->setAccessible(true);
}

private function columnType(string $id): string
{
/** @var string $type */
$type = $this->getColumnType->invoke($this->adapter, $id, 'event');
return $type;
}

/**
* Lower-cardinality premium geo dims must map to LowCardinality(Nullable(String)).
*/
public function testLowCardinalityPremiumGeoColumns(): void
{
foreach ([
'continentCode', 'subdivisions', 'connectionType',
'connectionUsageType', 'autonomousSystemNumber',
] as $col) {
$this->assertSame(
'LowCardinality(Nullable(String))',
$this->columnType($col),
"{$col} should be LowCardinality(Nullable(String))"
);
}
}

/**
* High-cardinality premium geo dims must fall through to Nullable(String).
*/
public function testHighCardinalityPremiumGeoColumns(): void
{
foreach ([
'city', 'isp', 'autonomousSystemOrganization', 'connectionOrganization',
] as $col) {
$this->assertSame(
'Nullable(String)',
$this->columnType($col),
"{$col} should be plain Nullable(String)"
);
}
}
}
9 changes: 9 additions & 0 deletions tests/Usage/Adapter/ClickHouseSchemaTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ public function testEventsTableCarriesCodecsAndLowCardinality(): void
$this->assertStringContainsString('`teamId` Nullable(String) CODEC(ZSTD(3))', $ddl);
$this->assertStringContainsString('`osVersion` LowCardinality(Nullable(String)) CODEC(ZSTD(3))', $ddl);
$this->assertStringContainsString('`deviceModel` LowCardinality(Nullable(String)) CODEC(ZSTD(3))', $ddl);

// premium geo: lower-cardinality dims are LowCardinality, high-cardinality
// dims fall through to plain Nullable(String).
$this->assertStringContainsString('`continentCode` LowCardinality(Nullable(String)) CODEC(ZSTD(3))', $ddl);
$this->assertStringContainsString('`connectionType` LowCardinality(Nullable(String)) CODEC(ZSTD(3))', $ddl);
$this->assertStringContainsString('`city` Nullable(String) CODEC(ZSTD(3))', $ddl);
$this->assertStringContainsString('`isp` Nullable(String) CODEC(ZSTD(3))', $ddl);
}

public function testEventsTableSwapsBloomForSetOnLowCardinality(): void
Expand Down Expand Up @@ -211,6 +218,8 @@ private function expectedDimAssertions(array $columns, string $type): array
'clientEngine', 'clientEngineVersion',
'deviceName', 'deviceBrand', 'deviceModel',
'hostname', 'ip',
'continentCode', 'subdivisions', 'connectionType',
'connectionUsageType', 'autonomousSystemNumber',
];

$baseKey = ['id', 'metric', 'value', 'time', 'tenant'];
Expand Down
36 changes: 36 additions & 0 deletions tests/Usage/MetricTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -609,6 +609,9 @@ public function testEventColumnsConstant(): void
'service', 'resourceType', 'resourceId', 'resourceInternalId',
'teamId', 'teamInternalId',
'country', 'region', 'hostname', 'ip',
'city', 'continentCode', 'subdivisions',
'isp', 'autonomousSystemNumber', 'autonomousSystemOrganization',
'connectionType', 'connectionUsageType', 'connectionOrganization',
'osCode', 'osName', 'osVersion',
'clientType', 'clientCode', 'clientName', 'clientVersion',
'clientEngine', 'clientEngineVersion',
Expand Down Expand Up @@ -645,6 +648,39 @@ public function testEventSchemaHasAllNewColumns(): void
$this->assertNotContains('tags', $ids, 'tags must be removed');
}

/**
* The premium geo dimensions are event-only string columns that are
* present in EVENT_COLUMNS and the event schema (as non-required
* strings) but never leak into the gauge column set.
*/
public function testPremiumGeoColumnsArePresentAsEventStrings(): void
{
$geo = [
'city', 'continentCode', 'subdivisions',
'isp', 'autonomousSystemNumber', 'autonomousSystemOrganization',
'connectionType', 'connectionUsageType', 'connectionOrganization',
];

$schema = Metric::getEventSchema();
$eventIds = array_column($schema, '$id');
$gaugeIds = array_column(Metric::getGaugeSchema(), '$id');

foreach ($geo as $col) {
$this->assertContains($col, Metric::EVENT_COLUMNS, "EVENT_COLUMNS missing {$col}");
$this->assertContains($col, $eventIds, "Event schema missing {$col}");
$this->assertNotContains($col, Metric::GAUGE_COLUMNS, "{$col} must be event-only");
$this->assertNotContains($col, $gaugeIds, "{$col} must not be in gauge schema");

$matches = array_values(array_filter(
$schema,
static fn (array $attr): bool => $attr['$id'] === $col,
));
$this->assertCount(1, $matches, "{$col} must appear exactly once");
$this->assertSame('string', $matches[0]['type'], "{$col} must be a string column");
$this->assertFalse($matches[0]['required'], "{$col} must be optional");
}
}

/**
* Test that the gauge schema contains the new team and resource id columns.
*/
Expand Down
Loading