v0.16 — idempotent writes (upsert + CSV --upsert)#16
Merged
Conversation
Thin commands over a shared upsertWithDefs/summarizeUpsert lib pair: fetch field defs, build the write body (--field/--body), match by --by (built-in key or searchable custom field), then create or PATCH only the changed fields. Ambiguous matches refuse with exit 65 — never guess. --dry-run previews without writing; table prints a one-line summary, json emits the full action result.
Adds an idempotent CSV mode: with --upsert --match-on <field>, each row is matched on that field's per-row value, then created if absent or PATCHed (changed fields only) if exactly one matches. Ambiguous rows (and empty match values) are collected as per-row failures and exit 1 without aborting the batch — never guessing which record to write. --dry-run looks up and reports created/updated/unchanged counts without writing. Shared bulkUpsertRows lib runs the batch through bulkRun's pacing.
- lookup: reject a non-numeric value for a numeric custom field (exit 65) instead of coercing to NaN — NaN loses every comparison, so a match would silently miss and create / inject a NaN value. - diffBody: set-aware field equality so re-running an unchanged upsert emits no PATCH. emails/phones compare by their value set (case-insensitive, ignoring the primary/label flags the API echoes); label_ids and multi-option custom fields compare order-insensitively. - bulk: preserve each failed item's exit code; CSV --upsert now exits 65 when every row failure is a data-validation error (ambiguous / empty match) and 1 for mixed or transport errors. - import: reject duplicate CSV column headers (exit 65) rather than silently keeping only the last cell. Dismissed: a finding that per-entity v2 /search rejects the 'fields' param — the OpenAPI spec confirms persons/deals/organizations search all accept 'fields' (enum incl. custom_fields) and 'cursor'.
CHANGELOG 0.16.0; README idempotent-writes section; bulk guide gains an upsert (match-or-create) section + CSV --upsert/--match-on; agents page gains the upsert capability paragraph. Regenerated command reference (145 commands) and cli-stats. Version bump to 0.16.0.
Two bugs only live testing on the sandbox surfaced:
- The per-entity /search endpoints cap `limit` at 100 (the API 400s on more,
despite the 500 list cap) — was sending 500, so every lookup errored.
- The search result `item` is a lossy projection: emails/phones come back as
bare strings (not {value} objects) and custom_fields as an unattributed
value array. Verifying against it (and using it as the record to diff)
silently discarded every match, so each upsert created a duplicate.
lookupByField now treats the scoped exact_match search purely as a candidate-id
finder, then fetches each candidate's full record and re-verifies the matched
field against the authoritative shape — which also gives diffBody a correct
record. Verified live: create → unchanged on re-run → updated on change →
exit 65 on a duplicate.
Matching depends on Pipedrive's search index, which is eventually consistent, so upserting the same key twice in quick succession can still create a duplicate before the first write is indexed. Document the gotcha and the mitigation (key on a stable external id; let the index settle).
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.16 — idempotent writes
Match-or-create that never guesses, plus the CSV equivalent.
Added
person upsert/org upsert/deal upsert— match by--by(a built-in key or a searchable custom field), then create if absent or PATCH only the changed fields if exactly one matches. More than one match refuses with exit 65 (searchexact_matchis not a unique key, so every candidate is re-verified against its full record before the count decides).--dry-runpreviews; table prints a one-line summary,--output jsonthe full action result.person import/org import--upsert --match-on <field>— the CSV equivalent: each row matched on its--match-onvalue, then created or PATCHed. Per-row failures (ambiguous / empty match) are collected without aborting; reported as created/updated/unchanged counts. A batch whose failures are all data-validation errors exits 65, else 1.Changed
diffBodycompares emails/phones by value set (case-insensitive, ignoringprimary/label) andlabel_ids/multi-option custom fields order-insensitively, so an unchanged re-run issues no PATCH.Quality
/searchdoes supportfields)./searchcapslimitat 100 (not 500), and the search resultitemis a lossy projection (emails/phones as bare strings, custom_fields unattributed) — so lookup now uses search as a candidate finder and verifies against each candidate's full record. Verified end-to-end: create → unchanged on re-run → updated on change → exit 65 on a duplicate.100% coverage on touched files;
npm run lintclean.