High Level Design Concerns

[sholtomaud]

Having used spec-kit recently for development I'd like to report some concerns. In the main I agree with the design philosophy at a high level. However:

• Firstly, after several hours work, I have a wonderfully specified app, but the implementation is poor, I should not have to specify that the AI should not use deprecated js methods for example, that is just assumed.
• There is no workflow that shows how to go from specifications to debugging code; is the issue an issue in the specifications or an issue in the implementation? I should not have to re-write the specifications to fix a bug, or maybe I do? Maybe I have to re-write the specifications every time the AI incorrectly implements something. Fine-tuning specifications is a nightmare on this point.
• GitHub is not just for tracking features, but also for tracking issues/bugs. some debugging tool needs to be aware of the specifications and track the gap between the spec and the bug. Bugs are tracked against features.
• There is often an as specified vs as implemented gap, which we probably need a KPI around as a good SRE would do.
• I don't want the AI to spend more time re-writing or writing specifications than it does writing code. Why? Because it feels like there is a threshold where more time spent on specs doesn't not equate to better app/code. Indeed there is a $ cost-factor here, where the $ cost of time writing/rewriting specs is costing me more than the $ cost of writing code. This doesn't feel right. If I can get a better implementation from an LLM without using spec-kit at lower $ cost, then Houston has a problem. There needs to be KPIs around this threshold. Not everyone has infinite $ to spend on an LLM writing specs.
• We need to avoid at all costs the "great specs - no MVP" problem.
• If I am to be an orchestrator/PM/System Engineer of many agents, and no longer a developer, I want dashboards on the performance of my agents. And they need to learn how to self improve based on the spec-bug gap. Is the agent that uses a specific system/normal prompt fixing more or less bugs, and or better and writing and implementing specs? Can they learn from each other's prompts, as whatever works.
• We now appear to have two types of SRE: Site Reliability Engineering, and Spec Reliability Engineering and the two need to relate to each other

I'm sure there is more, but for now, if we shift left from Development to Spec Engineering, then there is a burden on the tooling to self-improve, not the developer.

[sholtomaud]

Thinking some more about the "great specs - no MVP" problem, and probably falling on deaf ears, I reckon a Spec Reliability Engineering would need to look at our Coding Agent providers as we would contractors.

In this sense the pain share/gain share contract model would seem to be appropriate rather than a pay per API request and token count model.

The pain share/gain share contract model is a risk and reward mechanism used primarily in target cost contracts and alliance contracting, designed to align the interests of the parties involved by sharing financial outcomes based on project performance relative to a predetermined target cost.

We want to reward great Coding Agents and providers. We don't want our Coding Agent providers to be walking away from a contract without some pain if they get things wrong and go over budget.

This model fosters a collaborative "win-win, lose-lose" mindset where both the principal (or employer) and the contractor share in the financial gains from cost savings or the pain of cost overruns.

I've seen so many development and related projects have cost overruns and poor implementations with contractors/consultants walking away scot-free, and it always comes down to a poorly written contract. If you want a good development, you basically need a good contract writer and contract manager.

Are there golden signals of Spec Reliability Engineering??

SiteRE metrics have the "Four Golden Signals"; latency, traffic, errors, and saturation. What does SpecRE have? Do we need to add cost and budget and pain signals into Coding Agent provider contracts?

[duydduyd]

@sholtomaud Your comment resonated with me! Curious to know what your current workflow with using AI is today. Are you still using spec-kit or any other implementation of spec-driven development?