test: add comprehensive integration tests across modctl#495
test: add comprehensive integration tests across modctl#495
Conversation
Define 8 test dimensions (~41 scenarios) covering functional correctness, network errors, resource leaks, concurrency safety, stress tests, data integrity, graceful shutdown, and idempotency. Includes shared mock OCI registry infrastructure design. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
- Known bug tests use reverse assertions (CI stays green) - Retry-dependent network tests moved to //go:build slowtest - Remove untestable pull reader/tempdir leak scenarios - Push leak tests cover both success and error paths - Deduplicate against existing test coverage - Add GitHub issue references (#491, #492, #493, #494) - Add auth-error-retry as known design issue Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
9 tasks covering mock registry infrastructure, resource tracking, modelfile/pull/push/CLI/pb integration tests, slow tests, and stress tests. Each task has TDD-style steps with complete code. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…t coverage) Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…yWorkspace Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add pkg/backend/pull_integration_test.go with 11 integration tests that exercise the Pull function using MockRegistry (httptest) and mock Storage: Functional correctness: - HappyPath: manifest + 2 blobs, verify PushBlob/PushManifest calls - BlobAlreadyExists: StatBlob returns true, verify PushBlob skipped - ConcurrentLayers: 5 blobs in parallel, verify all stored Network errors: - ContextTimeout: latency + short deadline, verify context error - PartialResponse: FailAfterNBytes on blob, verify error - ManifestOK_BlobFails: 500 on specific blob, verify error Concurrency safety: - ConcurrentPartialFailure: 2 of 5 blobs fail, errgroup cancels Data integrity: - TruncatedBlob: FailAfterNBytes, digest validation catches mismatch - CorruptedBlob: wrong bytes under correct digest, caught by validateDigest Graceful shutdown: - ContextCancelMidDownload: cancel mid-flight, verify error Idempotency: - Idempotent: second pull with all blobs existing skips all writes Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…, dedup truncated/partial) - TestIntegration_Pull_Idempotent: remove unused delta variable and misleading comment; clarify that the key invariant is no storage writes on second pull (PushBlob/PushManifest not called), not a reduction in registry requests. - TestIntegration_Pull_TruncatedBlob: differentiate from PartialResponse by serving same-length wrong-byte content via AddBlobWithDigest instead of truncating the stream. This triggers digest-validation failure rather than a read error, and asserts the error message contains "digest". - TestIntegration_Pull_PartialResponse: add RequestCountByPath assertion to verify the blob endpoint was contacted at least once. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add 8 integration tests for the Push workflow covering: - Functional correctness (happy path, blob-already-exists skip) - Network errors (manifest push failure via path fault) - Resource leak documentation (KnownBug reverse assertions for #491) - Data integrity (byte-level blob verification after push) - Graceful shutdown (context cancellation mid-upload) - Idempotency (second push skips existing blobs) The KnownBug tests use TrackingReadCloser with reverse assertions (AssertNotClosed) to document that PullBlob ReadClosers are never closed on either success or error paths. These will fail when #491 is fixed, signaling the assertions should be flipped. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…ation Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
… (//go:build stress) Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive integration testing suite for modctl, covering functional correctness, network resilience, concurrency, and stress scenarios. It includes a new mock OCI registry helper and resource tracking utilities. Review feedback identifies a logic error in a 'KnownBug' test that prevents it from failing when fixed, suggests renaming a misleadingly titled test, recommends optimizing the mock registry's performance by reducing mutex contention during I/O, and advises removing a redundant implementation plan file that duplicates the source code.
| if blobTracker.WasClosed() { | ||
| // If this branch is reached, the bug may be fixed — flip to AssertClosed. | ||
| t.Log("blob tracker was closed — bug #491 may be fixed") | ||
| } else { | ||
| blobTracker.AssertNotClosed(t) | ||
| } |
There was a problem hiding this comment.
The if blobTracker.WasClosed() conditional block in TestKnownBug_Push_ReadCloserNotClosed_ErrorPath prevents the test from failing when the bug is fixed. This contradicts the PR's stated strategy for TestKnownBug_ tests, which should use reverse assertions that fail upon a fix to signal that the workaround or documentation is no longer needed. Please remove the conditional and use blobTracker.AssertNotClosed(t) directly.
blobTracker.AssertNotClosed(t)| // Dimension 6: Data Integrity | ||
| // -------------------------------------------------------------------------- | ||
|
|
||
| func TestIntegration_Pull_TruncatedBlob(t *testing.T) { |
There was a problem hiding this comment.
The test name TestIntegration_Pull_TruncatedBlob is misleading because the implementation (lines 312-317) and comments (lines 307-311) describe a scenario where the full response is delivered but with incorrect bytes (corruption). Truncation (partial response) is already tested in TestIntegration_Pull_PartialResponse. Consider renaming this test to TestIntegration_Pull_CorruptedContentSameLength to better reflect its purpose.
| func TestIntegration_Pull_TruncatedBlob(t *testing.T) { | |
| func TestIntegration_Pull_CorruptedContentSameLength(t *testing.T) { |
| return | ||
| } | ||
| r.mu.Unlock() | ||
| w.Header().Set("Location", path) |
There was a problem hiding this comment.
Holding the global mutex r.mu while performing network I/O (buf.ReadFrom(req.Body)) serializes all registry operations. This creates a bottleneck that can interfere with the effectiveness of concurrent stress tests. It is better to read the request body into a local buffer before acquiring the lock to update the shared state, similar to the pattern used in handleManifest (lines 319-329).
| @@ -0,0 +1,2222 @@ | |||
| # Integration Tests Implementation Plan | |||
There was a problem hiding this comment.
This plan file is extremely large and redundant as it duplicates the entire source code of the PR. Additionally, the code snippets within this plan are outdated and contain bugs (e.g., the mockregistry.go implementation here ignores PATCH data) that are fixed in the actual implementation files. This redundancy increases the maintenance burden and can mislead future developers. Consider removing the code blocks or the entire plan file now that the implementation is complete.
Summary
test/helpers/mockregistry.go) with fault injection (latency, partial response, status override, Nth-request failure, per-path faults) for httptest-based integration testingNew Files (13 files, 2433 lines)
test/helpers/mockregistry.go+_test.gotest/helpers/tracking.gopkg/modelfile/modelfile_integration_test.gopkg/backend/pull_integration_test.gopkg/backend/push_integration_test.gointernal/pb/pb_integration_test.gocmd/modelfile/generate_integration_test.gocmd/cli_integration_test.gopkg/backend/pull_slowtest_test.goslowtestpkg/backend/push_slowtest_test.goslowtestpkg/modelfile/modelfile_stress_test.gostresspkg/backend/pull_stress_test.gostressTest Plan
go test ./...— all default tests passgo test ./pkg/backend/ -tags slowtest -timeout 300s— retry-dependent testsgo test ./pkg/modelfile/ ./pkg/backend/ -tags stress -timeout 300s— stress testsgo test ./internal/pb/ -race— confirms bug: disableProgress global variable has data race (no atomic/mutex protection) #493 data race exists