Emit changemaker IDs to make permissions easier#274
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the charityNavigator CLI commands to surface which PDC changemakers correspond to Charity Navigator results, making it easier to act on/verify permissions by ID during lookupFromPdc, and it adjusts the updateAll upload loop to iterate over validated nonprofits.
Changes:
- Add
lookupFromPdclogging of changemaker IDs derived from Charity NavigatornonprofitsPublic.edges. - Add an early return when the Charity Navigator GraphQL response contains no
data. - Change
updateAllto iterate overnonprofits(post-validation) instead of rawedges.
Comments suppressed due to low confidence (1)
src/charityNavigator.ts:392
- The async work inside
nonprofits.map(async ...)(and the nestedObject.entries(...).map(async ...)) is never awaited. That means the command can finish before all field posts complete, and any rejections become unhandled. Collect andawaitthe promises (e.g., viaPromise.all) so the CLI only exits after uploads finish and errors propagate.
nonprofits.map(async (e) => {
const changemaker = getChangemakerByEin(e.ein, changemakers);
if (changemaker !== null) {
Object.entries(baseFieldMap).map(async ([cnAttributeName, baseFieldShortCode]) => {
const cnAttribute = e[cnAttributeName as keyof NonprofitPublic];
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const { edges } = charityNavResponse.data.nonprofitsPublic; | ||
| const nonprofits = edges.filter((e): e is NonprofitPublic => isNonprofitPublic(e)); | ||
| const changemakerIds = nonprofits.map( | ||
| (e) => getChangemakerByEin(e.ein, changemakers), | ||
| ).filter((c) => c !== null).map((c) => c.id); |
8ada522 to
1262d46
Compare
bickelj-agent
left a comment
There was a problem hiding this comment.
GLM-5.2: More than two important issues here; the two inline are the most important — unawaited nonprofits.map(async…) in updateAll (writes may not finish, rejections swallowed) and the new !data guard skipping the --output-file write on API errors. Also notable: getCharityNavigatorProfiles fetches only page 1 with a resultSize the query never uses, so the emitted IDs can be silently truncated.
1262d46 to
c90f42d
Compare
bickelj-agent
left a comment
There was a problem hiding this comment.
GLM-5.2: Re-review after force-push. The unawaited-map issue is fixed (now await Promise.all), so I resolved that thread. The guard-before-output-file issue is unchanged and left open. Two new notes below: a concurrency burst in updateAll and possible truncation of the emitted changemaker IDs.
| const fieldBatch = await postChangemakerFieldValueBatch(args.pdcApiBaseUrl, token, { sourceId: source.id, notes: `data-scripts charityNavigator.ts execution ${Date.now()}` }); | ||
| // Last, for each nonprofit, for each field, post the field | ||
| edges.map(async (e) => { | ||
| await Promise.all(nonprofits.map(async (e) => { |
There was a problem hiding this comment.
GLM-5.2: Promise.all here fires every nonprofit's field POSTs at once (and, via the inner Promise.all, all fields per nonprofit), which can burst the PDC API; any single rejection also aborts the batch, leaving it partially posted. Consider bounded concurrency or a sequential loop.
There was a problem hiding this comment.
Bro, you were the one who suggested Promise.all
c90f42d to
e967bf4
Compare
When doing a `lookupFromPdc` command, try to emit the permissions.
e967bf4 to
67ea78d
Compare
|
As imperfect as this is, and the pagination handling needs to be improved, this is still an improvement and should be merged. |
When doing a
lookupFromPdccommand, try to emit the permissions.