Make export usable on large accounts (10k SHOW cap, O(n^2) grant lookup, unqualified resources)

[jacodegroothydrab]

Opening as a draft to demonstrate the problem. This is all Claude generated, and I would recommend to use this PR as inspiration only. The main problem is the 10K limit of SHOW. Beyond that Claude identified an O(n^2) grant lookup. In my case it reduced the import time from > 75 minutes (it never finished) to a few minutes. Because of the slow running it first tried to increase the threads from 4 to 16, that change wasn't needed in the end.

---

On a large account (~15k tables, ~90k grants) snowcap export --all does not complete. Digging into it turned up three independent problems.

1. SHOW ... IN ACCOUNT is capped at 10,000 rows

list_tables, list_views and list_stages all use SHOW ... IN ACCOUNT, which Snowflake caps at 10,000 rows:

090153 (22000): The result set size exceeded the max number of rows(10000)
supported for SHOW statements. on SHOW TABLES IN ACCOUNT

Past that, listing fails outright and takes plan/export with it. They now read SNOWFLAKE.ACCOUNT_USAGE, falling back to SHOW when it is unavailable or errors.

Trade-off, and the bit I would most like your view on: ACCOUNT_USAGE lags live state (commonly up to ~2h), so a very recently created object can be missed. I made it the default on the reasoning that on an account past the cap a slightly stale answer beats no answer at all — but if you would rather it sat behind an explicit flag, or only engaged as a fallback after a 090153, I am glad to rework it.

2. _show_grants_to_role rescanned the whole grant cache per call

It re-filtered the entire cached ACCOUNT_USAGE grant list on every call. A 300-grant profile showed 32.7M str.upper() calls. Now indexed by grantee, once per session.

3. _fetch_grant_to_role scanned a role's grants linearly

It walked the role's full grant list, constructing two ResourceName objects per candidate. Now indexed by (granted_on, privilege, name).

ResourceName can't key a dict directly: __hash__ is hash(str(self)), but __eq__ treats quoted "FOO" and unquoted FOO as equal while str() keeps the quotes — so hash and equality disagree. _grant_name_key normalises to "exact if quoted, upper-cased otherwise", which reproduces __eq__ across all four quoted/unquoted combinations.

Both indexes are invalidated in reset_account_usage_caches alongside the cache they derive from.

Results

Measured on the affected account, 5,000-grant sample, steady state:

	ms/grant	grant phase
before	48.32	~74 min (projected)
after	0.01	~1 s

export --all --exclude=table,view: did not complete → 125 seconds.

Also included

• --threads on export (default raised 4 → 16) and --use-account-usage — neither was reachable from the CLI. Threads help the IO-bound resource types; the grant phase is CPU-bound and GIL-serialised, so only indexing addressed it.
• Separate commit: qualify schema-scoped resources. export omits database/schema from file formats, stages, procedures etc. even though the URN carries them. That makes output ambiguous when two schemas share an object name, and plan/apply reject unqualified resources once a config spans multiple databases (Resource FILE FORMAT 'X' has no schema) — so an --all export of a multi-database account can't be fed back in without hand-editing. Happy to split this into its own PR if you'd prefer.

Not done

No tests added yet — I wanted to check the direction first, especially on the ACCOUNT_USAGE default. Glad to add coverage for the two index helpers and the qualification change if the approach looks right.

[noel]

The ACCOUNT_USAGE approach is the right call here — on large accounts SHOW hits the 10k cap and truncates, so this is the only thing that actually works at scale, and nice job filtering deleted IS NULL / DELETED_ON IS NULL on every query so dropped objects don't get resurrected. A few things before this leaves draft:

1. Bug — quoted/mixed-case database names get silently dropped. In _list_schema_scoped_from_account_usage, user_databases = {str(db) for db in _list_databases(session)} then database not in user_databases. str(ResourceName) renders quoted names with quotes ("MyDb") while ACCOUNT_USAGE returns the bare MyDb, so "MyDb" in {'"MyDb"'} is False and those DBs are excluded. Compare in ResourceName space instead (the SHOW path already does). Uppercase DBs are unaffected, which is why it's easy to miss.
2. Keep ACCOUNT_USAGE as the default, but warn about staleness. Defaulting on is right for export at scale, but ACCOUNT_USAGE has up to ~2h latency, so emit a one-line warning when it's used ("recently created objects may be missing; pass --no-use-account-usage for real-time SHOW"). I'd avoid auto-fallback-on-cap — counting exactly-10k rows to detect truncation is fragile; an explicit default + warning + escape flag is more predictable.
3. Tests. The two index helpers and the DB-qualification change need unit coverage before merge.