🦅 ContentHawk - Add content auditing agentic workflows for SSW Rules

[Calinator444]

(moved from TinaCMS repo https://github.com/tinacms/tinacloud/issues/3305)

CC: @wicksipedia, @adamcogan, @18-th

Description

We are changing the development trajectory for the content auditing GitHub actions. With Agentic Workflows now being available for GitHub implementing a content auditing tool specific to Tina on GitHub marketplace is likely now redundant. Instead it may be preferable to build an agentic workflow that can be used on any Tina content repo by scanning markdowns using cron jobs.

Goals

We need to realign our goals for the content auditing GitHub actions and decide whether the flow will be built using GitHub Actions, GitHub Agentic workflows or a TinaCloud specific implementation. Following this we need to develop the flow and deploy it to SSW Rules for marketing reasons. A video will then be recorded demoing the flow on SSW Rules and sent to SSWAll. Once our implementation on Rules is rolled out and has been thoroughly tested we will need to publish the flow (or at least a readme for setting it up) so that end users can use it on their own TinaCMS projects.

Task

• Draft implementation plan for content auditor using agentic workflows
• Build POC for content auditing agent
• Roll out agentic workflows for SSW Rules

Acceptance criteria

• Plan checked by @wicksipedia, @calumjs, & kosta
• Video recorded & test passed demoing agentic workflow on SSW Rules
• The new agentic flow plan should address the other requirements listed in the PBIs for the main epic

[Calinator444]

I fixed an issue this morning with skipped files not being flagged by the workflow. I am passing skipped issues to the workflow that logs checked issues in a PR using an artifact.

Figure: File check marked as "skipped" by content hawk

[Calinator444]

Also I think the devx for orchestrating the workflow could be improved. Currently the user needs to input a number of parameters to kick off the flow and the inputs are single lines. Maybe just having a trigger that kicks off the workflow when files are added to .github/campaign/<file> would be better?

[Calinator444]

Option A (Recommended)

• Trigger with pushes to .github/ContentHawk/campaigns.
• The label for the PRs/Issues matches the file name (i.e. archive-tfs-rules.md creates a label called archive-rules)
• Other agentic workflows scan the file list using a cron job

example:

### File Scope

{{ Written description of which files need to be checked }} 

### Proposed Changes

{{ Written description of how the content should be ordered (e.g. Archive) }}

### PR Preferences

{{ Written description of how many content checks are done per PR }}

### Sort Order

{{ Written description of how the todo list is organized (e.g. date updated in Git) }}

pros:

• Much easier to start content campaigns for non-developers
• Content checks are initiated in a plain human-readable format

cons:

• Requires an additional PR for to begin the content audit
• Small chance the AI agent might not read file correctly

Option B

Keep agentic workflows as is

pros:

• Doesn't incur additional development time
• Original prompt is already stored in a generated file.

cons:

• Inputting the prompt in the initiating GitHub action is difficult as it is a single line input.

[Calinator444]

As per my conversation with @calumjs & @suiyangqiu We're going with Option A but We'll have a claude skill that generates the TODO file. The events can still be trigger via GitHub action if users don't have claude.

[Calinator444]

I've created a claude skill that can easily be run to kick start the process. It'll create a PR with the right label

[Calinator444]

I found a bug with this thing morning after updating the flow to be a CronJob

ContentHawk is continuing to create PRs to fix the same issues even if a PR is already open

[Calinator444]

I encountered another small problem today. The agent that works on generated issues (Content fixer) works off the items in the TODO folder that have issues against them. However the TODO snapshots automatically get marked as DONE once an issue has been generated for each scoped file, meaning Content Fixer has nothing to work off of.

I will fix this by moving items to "Done" when ContentHawk issues are closed rather than once an issue has been generated for each file.

[Calinator444]

Second issue: Content Fixer sometimes works off of issues that have already been marked as closed. I'll update it so that it only works off of open issues identified in the snapshot file;.

[Calinator444]

@18-th left some feedback for the agentic flow when we were working on it on Friday. We also encountered a breaking issue where the workflow decided that no actionable issues needed to be created, but the workflow could not continue unless that campaign was completed:

Agent 1/content-campaign.md:

• Add a default preference for issue bundling (when no input is provided)
• consider renaming the flow from content-campaign to content-detective
• Auto-generate the slug that binds the workflows together
• If everything is skipped during the bundling phase agent 1 should move the generated campaign to Done immediately

General:

• Add more hooks for moving files to “Done” (currently files are only moved to done when issues are closed)

Tavily:

• Check if we need the search_news tool

I created a PR for adding ContentHawk for SSW Rules with a Readme describing the flow, but was not able to merge due to the issues outlined above.

Figure: readme for content hawk

[Calinator444]

🚀Fixed - I deployed ContentHawk on SSW Rules but the flow failed because of a permissions issue with the Personal Access Token I supplied to run the flow. This is almost certainly because I generated the access token using the SSW Rules service account (sswrules@ssw.com.au) and that account doesn't have a Copilot subscription. I updated the flow to use a PAT generated from my account instead and it worked correctly.

Currently working with @RobJT-wq to resolve this.

[Calinator444]

I did a recording with @bahjatalaadelssw and @MarcusIrmscherSSW on Friday, but the screen share was not captured during the first part of the recording so we will need to re-record.

🛠️ I also fixed the following issues while recording:

• The flow was failing because branches were occasionally being created with duplicate branch names in CI/CD, resulting in PRs being created with no change-set 🧑‍⚖️ Content Judge - Fix flow PRs not being created due to empty change sets SSW.Rules.Content#12383
• The Agentic workflow was failing due to the personal access token being scoped to the GitHub Service account for SSW Rules rather than the SSWConsulting organisation.

Figure: Error during recording due to agentic workflow PAT not being bound to SSWConsulting org