[Tech Scoping] CRON to be created to solve for backup of tickets

[tanishk2907]

The bug of tickets from the community not showing up on our website in cases of old organizations has been identified. Details of the issue can be found [here].

After conducting an RCA, we decided to set up a CRON job to periodically push all available ticket information into our servers at a defined frequency (to be determined). This will serve as a backup mechanism to the existing webhooks and not as a replacement

Key Considerations:

1. CRON frequency: Identify an optimal schedule to balance data freshness with server load.
2. Data consistency: Ensure CRON jobs do not create duplicate entries or overwrite legitimate updates from the webhook.

Success Criteria:

1. Mechanism in place to differentiate tickets marked out of scope/'to be deleted' earlier by PT coming in with new CRON.
2. To Maintain a source-of-truth table to reconcile data discrepancies post CRON processing.
3. Logs to be added for all CRON runs - success and failures

[karntrehan]

Tech flow:

1. Generate JWT - Once
2. Get installations - Once
3. Generate token - (X installations) 300
4. Get Repos - (X installations) 300
5. Get issues with label in each repo - (X installations X Repos) 900
6. Get PRs for each repo - (X installations X Repos) 900
7. Filter PRs from 6 linked to issues in 5
8. Update DB with issues
9. Update DB with PRs
10. Update DB with points to assignees of issues for closed PRs
11. Generate report: Success / Failure: Time taken, orgs found (delta + cumulative), repos found(delta + cumulative), issues found(delta + cumulative), PRs found(delta + cumulative)
12. Send report: via Email, Slack / Discord (to be decided by PT / @tanishk2907)

Gotchas:
Rate limits.

Running: Once a day at 2 am IST

To be added to: https://github.com/Code4GovTech/server repo.

[Shreyash-work-em]

Updates -

• Work's in progress
• No roadblocks here
• Will be ready for staging in the next 2-3 days except the report part.
• As Report platform is yet to be confirmed.

[tanishk2907]

CRON report top be shared via discord (new updates channel to be setup in the C4GT server only visible to the c4gt core and engg team)

Can be shared for every run