Skip to content

Improve integration and unit tests#1031

Open
Avery-Dunn wants to merge 12 commits into
devfrom
avdunn/test-standardization
Open

Improve integration and unit tests#1031
Avery-Dunn wants to merge 12 commits into
devfrom
avdunn/test-standardization

Conversation

@Avery-Dunn
Copy link
Copy Markdown
Contributor

@Avery-Dunn Avery-Dunn commented Jun 2, 2026

This PR aims to resolve some minor-but-widespread issues in integration and unit tests, update all of our test dependencies and build plugins, re-enable several disabled tests, a refactor the Selenium-based test infrastructure to be much more robust.

All changes generally fall into one of the following categories

None of the changes will impact the released source code in any way, and other than the handful of tests mentioned in 5 that were re-enabled/removed none of the changes will affect code coverage.

1. Code Quality Cleanup

  • Renamed 9 unit test classes from plural *Tests.java to singular *Test.java (e.g., CacheTestsCacheTest, TelemetryTestsTelemetryTest) to follow standard JUnit conventions
  • Fixed typo in MsalOauthAuthorizatonGrantTestMsalOauthAuthorizationGrantTest
  • Fixed typo in ApacheHttpClientAdapter.builGetRequest()buildGetRequest()
  • Fixed ~95 reversed assertEquals(actual, expected) calls across 11 test files to use correct assertEquals(expected, actual) ordering
  • Removed debug System.out.println/print statements from TelemetryTest
  • Removed duplicate readResource() method in AcquireTokenSilentlyTest (now uses shared TestHelper.readResource())
  • Cleaned up unused imports across multiple files

2. Code Coverage Gate

  • Added JaCoCo check goal to pom.xml with haltOnFailure=false (warning-only)
  • Thresholds: 65% line coverage, 50% branch coverage (intentionally generous; to be tightened over time)
  • Runs automatically during mvn verify, providing early warning if coverage regresses

3. Shared Test Helpers

  • Expanded IntegrationTestHelper from ~2 methods to ~12 methods:
    • createPublicApp(), createCca() — standard app builders
    • acquireTokenByRopc(), acquireTokenSilently() (with overloads) — common token acquisition patterns
    • assertAccessTokenNotNull(), assertAccessAndIdTokensNotNull(), assertAccessTokensEqual()/NotEqual(), assertTokensAreEqual()/NotEqual() — assertion helpers
  • Extracted TestTokenCachePersistence from duplicate inner classes in CachePersistenceIT and TokenCacheIT
  • Updated 8+ integration test classes to use shared helpers, eliminating significant duplication

4. Dependency and Plugin Updates

All updated to the latest versions compatible with a Java 8 build JDK:

Dependency/Plugin Old Version New Version
SLF4J 1.6.2 1.7.36
Nimbus JOSE+JWT 1.14.5 1.17.5
Selenium 3.14.0 4.13.0
OkHttp (transitive) 3.14.9 (explicit)
Apache HttpClient (transitive) 4.5.14 (explicit)
JaCoCo 0.8.12 0.8.13
Surefire 2.22.0 3.5.6
Failsafe 2.22.0 3.5.6
maven-compiler-plugin 3.1.0 3.12.0
maven-source-plugin 2.2.1 3.4.0
SpotBugs 3.1.11 4.2.3
maven-javadoc-plugin 3.7.0 3.15.0
build-helper-maven-plugin 1.10 3.6.0
bnd-maven-plugin 5.2.0 6.4.0
maven-jar-plugin 3.1.2 3.11.0
maven-assembly-plugin 2.14.0 2.22.0
RevAPI 0.15.0 0.15.1
revapi-java 0.28.1 0.28.4
maven-site-plugin 3.5.0 3.5.6
maven-project-info-reports 2.5 3.5.0

OkHttp and Apache HttpClient were previously pulled in transitively by Selenium 3 and needed explicit declarations after upgrading to Selenium 4.

5. Selenium Infrastructure Overhaul

Selenium 3 → 4 migration:

  • Updated all WebDriverWait(driver, long) calls to WebDriverWait(driver, Duration) (7 call sites)
  • Updated --headless to --headless=new in ChromeOptions

Diagnostic infrastructure (new):

  • SeleniumDiagnostics — captures screenshots, HTML page source (with sensitive parameter redaction), and browser console logs on test failure, saved to target/selenium-diagnostics/
  • SeleniumTestWatcher — JUnit 5 AfterTestExecutionCallback that automatically triggers diagnostic capture when a Selenium test fails (runs before @AfterEach so the WebDriver is still alive)
  • WebDriverProvider — interface for test classes to expose their WebDriver to extensions
  • Enabled goog:loggingPrefs in ChromeOptions for browser console log capture

Resilient element locators:

  • Added findWithFallback(driver, timeout, By... locators) to SeleniumExtensions — single wait loop that checks multiple locators per poll cycle, logs warnings when fallback locators match
  • Applied fallback locator arrays to AzureADLoginPage, ADFSLoginPage for username, password, and action button elements

New page objects:

  • CIAMLoginPage — handles CIAM-specific multi-step login flow (username → Next → password → Sign in) with correct locators for *.ciamlogin.com pages
  • Updated SeleniumTest.runSeleniumAutomatedLogin() to route CIAM and CIAM-like OIDC authorities to the new page object

Test reliability:

  • Added rerunFailingTestsCount=1 to Failsafe plugin config for transient browser flakiness
  • Re-enabled DeviceCodeFlowADTest and acquireTokenWithAuthorizationCode_CiamCud (both now passing)
  • Fixed scope bug in AcquireTokenInteractiveIT where string literal "TestConstants.USER_READ_SCOPE" was used instead of the constant
  • Fixed DeviceCodeIT using close() instead of quit() for WebDriver cleanup

B2C browser test removal:

  • Removed 3 Selenium-based B2C interactive login tests that were persistently failing due to B2C login page automation issues
  • B2C token acquisition coverage is maintained via 2 passing ROPC-based tests in UsernamePasswordIT (matching MSAL .NET's B2C test strategy, which also only tests B2C via ROPC)
  • Removed B2CLocalLoginPage page object, performLocalLogin(), and 7 now-unused B2C constants

@Avery-Dunn Avery-Dunn requested a review from a team as a code owner June 2, 2026 21:23
Avery-Dunn and others added 6 commits June 4, 2026 07:40
Update test dependencies and build plugins
…emove broken B2C browser tests

- Upgrade Selenium 3.14.0 to 4.13.0 (last Java 8-compatible)
- Add SeleniumDiagnostics: screenshot, HTML dump, browser log capture on failure
- Add SeleniumTestWatcher (AfterTestExecutionCallback) for auto-diagnostics
- Add WebDriverProvider interface for test extension access to WebDriver
- Add CIAMLoginPage page object for CIAM-specific login flows
- Add findWithFallback() helper for resilient element locators
- Add fallback locator arrays to AzureADLoginPage and ADFSLoginPage
- Add failsafe rerunFailingTestsCount=1 for transient browser flakiness
- Enable browser console logging via goog:loggingPrefs
- Fix DeviceCodeIT close() to quit() for proper WebDriver cleanup
- Route CIAM/OIDC authorities to correct page objects in SeleniumTest
- Fix scope bug in AcquireTokenInteractiveIT (string literal vs constant)
- Remove broken B2C Selenium tests (B2C coverage via ROPC tests instead)
- Remove B2CLocalLoginPage and related dead code/constants

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR focuses on improving the quality, reliability, and maintainability of MSAL4J’s unit and integration tests. It standardizes test naming/assertion patterns, consolidates repeated integration-test logic into shared helpers, updates test/build dependencies, and significantly refactors Selenium-based integration test infrastructure (including diagnostics capture on failure).

Changes:

  • Standardize and clean up unit tests (class naming, assertion argument ordering, remove debug output, reduce duplication).
  • Refactor integration tests to use shared helpers and a shared in-memory token cache persistence implementation.
  • Migrate Selenium 3→4 and harden Selenium infrastructure (fallback locators, CIAM page object, diagnostics capture, WebDriver lifecycle fixes), plus add a JaCoCo warning-only coverage gate and update Maven plugins/dependencies.

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/WSTrustRequestTest.java Fix assertion argument ordering in XML escaping test.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/TokenRequestExecutorTest.java Fix assertion argument ordering in token executor tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/TelemetryTest.java Rename test class, remove debug prints, and standardize assertions.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/ServerTelemetryTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/OnBehalfOfTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/MsalOauthAuthorizationGrantTest.java Fix typo in test class name.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/MexParserTest.java Fix assertion argument ordering in Mex parser tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/ManagedIdentityTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/JsonCompatibilityTest.java Rename test class/constructor to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/HttpHeaderTest.java Fix assertion argument ordering across header tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/HelperAndUtilityTest.java Rename test class and adjust assertions in helper/utility tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/DateTimeTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/CacheTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/CacheFormatTest.java Rename test class to singular form.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/AuthorizationRequestUrlParametersTest.java Fix assertion argument ordering and expected values ordering.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/AuthorityTest.java Fix assertion argument ordering across authority tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/AcquireTokenSilentlyTest.java Remove duplicate resource reader and use shared TestHelper.readResource.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/AccountTest.java Fix assertion argument ordering in account/cache tests.
msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/AadInstanceDiscoveryTest.java Fix assertion argument ordering in instance discovery assertions.
msal4j-sdk/src/integrationtest/java/infrastructure/WebDriverProvider.java Add interface to expose WebDriver for JUnit extensions.
msal4j-sdk/src/integrationtest/java/infrastructure/SeleniumTestWatcher.java Add JUnit 5 extension to capture diagnostics on Selenium test failures.
msal4j-sdk/src/integrationtest/java/infrastructure/SeleniumExtensions.java Selenium 4 migration, enable browser logging, add fallback locator helper, route CIAM login.
msal4j-sdk/src/integrationtest/java/infrastructure/SeleniumDiagnostics.java Add diagnostics capture (screenshot, HTML, console logs) with basic sensitive-param redaction.
msal4j-sdk/src/integrationtest/java/infrastructure/pageobjects/CIAMLoginPage.java Add CIAM-specific page object for multi-step CIAM login flow.
msal4j-sdk/src/integrationtest/java/infrastructure/pageobjects/B2CLocalLoginPage.java Remove B2C local login page object (tests removed/strategy changed).
msal4j-sdk/src/integrationtest/java/infrastructure/pageobjects/AzureADLoginPage.java Harden Azure AD page object with fallback locators and Selenium 4 WebDriverWait APIs.
msal4j-sdk/src/integrationtest/java/infrastructure/pageobjects/ADFSLoginPage.java Harden ADFS page object with fallback locators and Selenium 4 WebDriverWait APIs.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/UsernamePasswordIT.java Refactor to use IntegrationTestHelper helpers; remove stray unused builder call.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/TokenCacheIT.java Standardize assertions and use shared TestTokenCachePersistence.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/TestTokenCachePersistence.java Add shared in-memory token cache persistence helper for integration tests.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/TestConstants.java Remove B2C constants no longer needed after B2C Selenium test removal.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/SeleniumTest.java Add SeleniumTestWatcher integration and CIAM/OIDC routing changes; remove B2C Selenium login branch.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/RefreshTokenIT.java Refactor assertions to use IntegrationTestHelper.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/OnBehalfOfIT.java Refactor repeated assertions into IntegrationTestHelper and remove local helper.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/IntegrationTestHelper.java Expand shared builders, acquisition helpers, and assertion utilities for integration tests.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/HttpClientIT.java Use IntegrationTestHelper ROPC/acquisition helpers and shared assertions.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/DeviceCodeIT.java Re-enable device code Selenium test, add diagnostics watcher, and fix driver teardown (quit vs close).
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/ClientCredentialsIT.java Refactor repeated assertions into IntegrationTestHelper.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/CachePersistenceIT.java Use shared TestTokenCachePersistence instead of duplicating persistence logic.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/AzureEnvironmentIT.java Refactor to shared helpers and assertions.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/AuthorizationCodeIT.java Remove B2C interactive browser tests and re-enable CIAM auth code test path.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/ApacheHttpClientAdapter.java Fix method typo (builGetRequest → buildGetRequest).
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/AcquireTokenSilentIT.java Replace local assertion helpers with IntegrationTestHelper assertions.
msal4j-sdk/src/integrationtest/java/com/microsoft/aad/msal4j/AcquireTokenInteractiveIT.java Remove B2C interactive browser tests, re-enable CIAM test, fix scope constant usage.
msal4j-sdk/pom.xml Update test/build dependencies and plugins, add JaCoCo check goal, Selenium 4 deps, and rerunFailingTestsCount.
Comments suppressed due to low confidence (1)

msal4j-sdk/src/test/java/com/microsoft/aad/msal4j/HelperAndUtilityTest.java:81

  • This assertion still has the assertEquals arguments reversed (expected, actual). Since methodReturnedMap is the value under test (returned from AuthorizationRequestUrlParameters.requestParameters()), it should be the "actual" value, and convertedInternalMap should be the "expected" value derived from the internal state.

Comment on lines +53 to 55
assertEquals(XML_ESCAPED_DATA, WSTrustRequest.escapeXMLElementData(DATA_TO_ESCAPE));
assertEquals(WSTrustRequest.escapeXMLElementData(DATA_TO_ESCAPE),
StringEscapeUtils.escapeXml10(DATA_TO_ESCAPE));
Comment on lines +108 to +112
content.append(pageSource);

Path destination = getOutputPath(filePrefix + ".html");
Files.write(destination, content.toString().getBytes("UTF-8"));
LOG.info("Page source saved: {}", destination);
Comment on lines +142 to +146
}

Path destination = getOutputPath(filePrefix + "-console.log");
Files.write(destination, content.toString().getBytes("UTF-8"));
LOG.info("Browser logs saved: {}", destination);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants