Skip to content

test: add comprehensive integration tests across modctl#495

Open
aftersnow wants to merge 15 commits intomainfrom
feat/integration-tests
Open

test: add comprehensive integration tests across modctl#495
aftersnow wants to merge 15 commits intomainfrom
feat/integration-tests

Conversation

@aftersnow
Copy link
Copy Markdown
Contributor

Summary

New Files (13 files, 2433 lines)

File Tests Build Tag
test/helpers/mockregistry.go + _test.go 9 self-tests default
test/helpers/tracking.go default
pkg/modelfile/modelfile_integration_test.go 2 default
pkg/backend/pull_integration_test.go 11 default
pkg/backend/push_integration_test.go 8 default
internal/pb/pb_integration_test.go 2 default
cmd/modelfile/generate_integration_test.go 3 default
cmd/cli_integration_test.go 3 default
pkg/backend/pull_slowtest_test.go 4 slowtest
pkg/backend/push_slowtest_test.go 1 slowtest
pkg/modelfile/modelfile_stress_test.go 2 stress
pkg/backend/pull_stress_test.go 2 stress

Test Plan

aftersnow added 15 commits April 7, 2026 22:00
Define 8 test dimensions (~41 scenarios) covering functional
correctness, network errors, resource leaks, concurrency safety,
stress tests, data integrity, graceful shutdown, and idempotency.
Includes shared mock OCI registry infrastructure design.

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
- Known bug tests use reverse assertions (CI stays green)
- Retry-dependent network tests moved to //go:build slowtest
- Remove untestable pull reader/tempdir leak scenarios
- Push leak tests cover both success and error paths
- Deduplicate against existing test coverage
- Add GitHub issue references (#491, #492, #493, #494)
- Add auth-error-retry as known design issue

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
9 tasks covering mock registry infrastructure, resource tracking,
modelfile/pull/push/CLI/pb integration tests, slow tests, and stress
tests. Each task has TDD-style steps with complete code.

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…t coverage)

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…yWorkspace

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add pkg/backend/pull_integration_test.go with 11 integration tests that
exercise the Pull function using MockRegistry (httptest) and mock Storage:

Functional correctness:
- HappyPath: manifest + 2 blobs, verify PushBlob/PushManifest calls
- BlobAlreadyExists: StatBlob returns true, verify PushBlob skipped
- ConcurrentLayers: 5 blobs in parallel, verify all stored

Network errors:
- ContextTimeout: latency + short deadline, verify context error
- PartialResponse: FailAfterNBytes on blob, verify error
- ManifestOK_BlobFails: 500 on specific blob, verify error

Concurrency safety:
- ConcurrentPartialFailure: 2 of 5 blobs fail, errgroup cancels

Data integrity:
- TruncatedBlob: FailAfterNBytes, digest validation catches mismatch
- CorruptedBlob: wrong bytes under correct digest, caught by validateDigest

Graceful shutdown:
- ContextCancelMidDownload: cancel mid-flight, verify error

Idempotency:
- Idempotent: second pull with all blobs existing skips all writes

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…, dedup truncated/partial)

- TestIntegration_Pull_Idempotent: remove unused delta variable and
  misleading comment; clarify that the key invariant is no storage
  writes on second pull (PushBlob/PushManifest not called), not a
  reduction in registry requests.
- TestIntegration_Pull_TruncatedBlob: differentiate from PartialResponse
  by serving same-length wrong-byte content via AddBlobWithDigest instead
  of truncating the stream. This triggers digest-validation failure rather
  than a read error, and asserts the error message contains "digest".
- TestIntegration_Pull_PartialResponse: add RequestCountByPath assertion
  to verify the blob endpoint was contacted at least once.

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add 8 integration tests for the Push workflow covering:
- Functional correctness (happy path, blob-already-exists skip)
- Network errors (manifest push failure via path fault)
- Resource leak documentation (KnownBug reverse assertions for #491)
- Data integrity (byte-level blob verification after push)
- Graceful shutdown (context cancellation mid-upload)
- Idempotency (second push skips existing blobs)

The KnownBug tests use TrackingReadCloser with reverse assertions
(AssertNotClosed) to document that PullBlob ReadClosers are never
closed on either success or error paths. These will fail when #491
is fixed, signaling the assertions should be flipped.

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…ation

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
… (//go:build stress)

Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive integration testing suite for modctl, covering functional correctness, network resilience, concurrency, and stress scenarios. It includes a new mock OCI registry helper and resource tracking utilities. Review feedback identifies a logic error in a 'KnownBug' test that prevents it from failing when fixed, suggests renaming a misleadingly titled test, recommends optimizing the mock registry's performance by reducing mutex contention during I/O, and advises removing a redundant implementation plan file that duplicates the source code.

Comment on lines +348 to +353
if blobTracker.WasClosed() {
// If this branch is reached, the bug may be fixed — flip to AssertClosed.
t.Log("blob tracker was closed — bug #491 may be fixed")
} else {
blobTracker.AssertNotClosed(t)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The if blobTracker.WasClosed() conditional block in TestKnownBug_Push_ReadCloserNotClosed_ErrorPath prevents the test from failing when the bug is fixed. This contradicts the PR's stated strategy for TestKnownBug_ tests, which should use reverse assertions that fail upon a fix to signal that the workaround or documentation is no longer needed. Please remove the conditional and use blobTracker.AssertNotClosed(t) directly.

	blobTracker.AssertNotClosed(t)

// Dimension 6: Data Integrity
// --------------------------------------------------------------------------

func TestIntegration_Pull_TruncatedBlob(t *testing.T) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test name TestIntegration_Pull_TruncatedBlob is misleading because the implementation (lines 312-317) and comments (lines 307-311) describe a scenario where the full response is delivered but with incorrect bytes (corruption). Truncation (partial response) is already tested in TestIntegration_Pull_PartialResponse. Consider renaming this test to TestIntegration_Pull_CorruptedContentSameLength to better reflect its purpose.

Suggested change
func TestIntegration_Pull_TruncatedBlob(t *testing.T) {
func TestIntegration_Pull_CorruptedContentSameLength(t *testing.T) {

return
}
r.mu.Unlock()
w.Header().Set("Location", path)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Holding the global mutex r.mu while performing network I/O (buf.ReadFrom(req.Body)) serializes all registry operations. This creates a bottleneck that can interfere with the effectiveness of concurrent stress tests. It is better to read the request body into a local buffer before acquiring the lock to update the shared state, similar to the pattern used in handleManifest (lines 319-329).

@@ -0,0 +1,2222 @@
# Integration Tests Implementation Plan
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This plan file is extremely large and redundant as it duplicates the entire source code of the PR. Additionally, the code snippets within this plan are outdated and contain bugs (e.g., the mockregistry.go implementation here ignores PATCH data) that are fixed in the actual implementation files. This redundancy increases the maintenance burden and can mislead future developers. Consider removing the code blocks or the entire plan file now that the implementation is complete.

@aftersnow aftersnow requested review from bergwolf and imeoer April 9, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant