Skip to content

164-java-profiling-verify.md

Latest commit

 

History

History
343 lines (268 loc) · 12.5 KB

164-java-profiling-verify.md

File metadata and controls

343 lines (268 loc) · 12.5 KB
author Juan Antonio Breña Moral
version 0.12.0-SNAPSHOT

Java Profiling Workflow / Step 4 / Verify results

Role

You are a Senior software engineer with extensive experience in Java software development

Goal

This cursor rule provides a comprehensive methodology for comparing Java profiling results before and after performance optimizations or refactoring efforts. It serves as the fourth step in the structured profiling workflow, focusing on quantifying the effectiveness of performance improvements and validating that optimizations achieved their intended goals.

The rule establishes a rigorous comparison framework that ensures accurate measurement of performance changes by maintaining consistent testing conditions, measurement techniques, and analysis criteria. It provides systematic approaches for generating post-refactoring profiling data and comparing it against baseline measurements to quantify improvements.

Key capabilities include:

  • Baseline Validation: Ensures proper baseline profiling data collection before implementing changes
  • Controlled Re-testing: Standardized process for generating comparable post-refactoring profiling data under identical conditions
  • Quantitative Comparison: Structured metrics for measuring performance improvements across CPU usage, memory allocation, GC pressure, and threading efficiency
  • Visual Analysis Framework: Systematic approach to comparing flamegraphs side-by-side to identify resolved hotspots and remaining issues
  • Documentation Templates: Standardized formats for documenting comparison results, including before/after metrics, visual evidence, and improvement validation
  • Regression Detection: Methods for identifying unintended performance regressions introduced during optimization efforts
  • Success Validation: Clear criteria for determining whether performance optimization goals were achieved

The rule ensures that performance optimization efforts are properly validated through rigorous before/after comparison, providing quantifiable evidence of improvements and identifying areas that may require additional attention or follow-up optimization work.

Project Organization

The profiling setup uses a clean folder structure with everything contained in the profiler directory:

your-project/
└── profiler/               # All profiling-related files
├── scripts/            # Profiling scripts and tools
│   └── java-profile.sh # Main profiling script
├── results/            # Generated profiling output
│   ├── *.html          # Flamegraph files
│   ├── *.jfr           # JFR recording files
│   ├── *.log           # Garbage Collection Log files
│   └── *.txt           # Thread Dump files
├── docs/               # Analysis documentation
│   ├── profiling-comparison-analysis-YYYYMMDD.md
│   └── profiling-final-results-YYYYMMDD.md
├── current/            # Symlink to current profiler version
└── async-profiler-*/   # Downloaded profiler binaries

Instructions

Step Pre-Refactoring Baseline (Already Done)

Ask the user if they have captured baseline results yet. For this purpose use the script profiler/run-with-profiler.sh to run the application with the right JVM flags for profiling.

Example:

cd profiler
./run-with-profiler.sh --mode (cpu, alloc, wall, lock)

After this task, send load to the application to capture the new behavior afte the refactoring.

Step Post-Refactoring Report Generation (CRITICAL STEP)

Ask the user if they have generated the post-refactoring data from profiling tools yet.

Step 1: Validate Your Refactoring Changes

# Ensure your code changes are applied
git status
git diff HEAD~1  # Review recent changes

Step 2: Generate New Profiling Reports cd profiler/scripts ./java-profile.sh --mode memory --duration 60

Step 3: Verify the reports were generated review the files in the profiler/results directory.

# Check that you have both sets of reports
echo "=== BASELINE REPORTS ==="
ls -la profiler/results/*baseline* || ls -la profiler/results/*before*

echo "=== POST-REFACTORING REPORTS ==="
ls -la profiler/results/*after* || ls -la profiler/results/*$(date +%Y%m%d)*

# Verify reports are not empty
wc -c profiler/results/*.html

Step Performance Metrics Comparison

Memory Analysis Checklist

  • Memory leak detection: Compare heap usage patterns between before/after flamegraphs
  • Allocation patterns: Analyze allocation flamegraphs for reduced object creation
  • GC pressure: Compare garbage collection frequency and duration
  • Peak memory usage: Identify improvements in maximum heap utilization
  • Stack depth: Compare flamegraph complexity (canvas height, levels)

CPU Performance Checklist

  • Hot spots identification: Compare CPU flamegraphs to identify resolved bottlenecks
  • Method execution time: Analyze improvements in critical path performance
  • Thread contention: Look for reduced blocking or improved concurrency

Visual Comparison Process

# Open reports side by side for comparison
# Browser tab 1: Baseline report
open profiler/results/memory-leak-*baseline*.html

# Browser tab 2: After refactoring report
open profiler/results/memory-leak-*after*.html

# Compare:
# - Canvas height (lower = better)
# - Stack depth (fewer levels = better)
# - Hot spot patterns (reduced width = better)

Step Documentation Creation

Note: All documentation files are created with date suffixes (YYYYMMDD format) to enable tracking multiple profiling sessions and maintain historical analysis records.

Step 1: Create Comparison Analysis Document

# Create the comparison analysis document with date suffix
DATE_SUFFIX=$(date +%Y%m%d)
touch profiler/docs/profiling-comparison-analysis-${DATE_SUFFIX}.md

Template for profiler/docs/profiling-comparison-analysis-YYYYMMDD.md:

# Profiling Comparison Analysis - [Date]

## Executive Summary
- **Refactoring Objective**: [What was being fixed]
- **Overall Result**: [Success/Partial/Failed]
- **Key Improvements**: [List main improvements]

## Methodology
- **Baseline Date**: [Timestamp of baseline reports]
- **Post-Refactoring Date**: [Timestamp of after reports]
- **Test Scenarios**: [What endpoints/operations were tested]
- **Duration**: [How long profiling ran]

## Before/After Metrics
| Metric | Before | After | Improvement |
|-----|-----|----|----|
| Stack Depth (levels) | X | Y | Z% |
| Canvas Height (px) | X | Y | Z% |
| Memory Allocations | X | Y | Z% |
| Thread Count | X | Y | Z% |

## Key Findings
### Resolved Issues
- [ ] Issue 1: [Description and evidence]
- [ ] Issue 2: [Description and evidence]

### Remaining Concerns
- [ ] Concern 1: [Description and next steps]

## Visual Evidence
- **Baseline Reports**: `../results/[filename]`
- **After Reports**: `../results/[filename]`
- **Key Differences**: [Describe what to look for in flamegraphs]

## Recommendations
1. [Next step 1]
2. [Next step 2]

Step 2: Create Final Results Summary

# Create the final results summary document with date suffix
DATE_SUFFIX=$(date +%Y%m%d)
touch profiler/docs/profiling-final-results-${DATE_SUFFIX}.md

Template for profiler/docs/profiling-final-results-YYYYMMDD.md:

# Profiling Final Results - [Date]

## Summary
- **Analysis Date**: [Date of analysis]
- **Performance Objective**: [What was being optimized]
- **Status**: [Complete/Ongoing/Failed]

## Key Metrics Summary
| Performance Area | Before | After | Improvement |
|---|---|----|----|
| Memory Usage | [value] | [value] | [%] |
| CPU Usage | [value] | [value] | [%] |
| Response Time | [value] | [value] | [%] |
| Throughput | [value] | [value] | [%] |

## Critical Issues Resolved
1. **[Issue Name]**: [Description and resolution]
2. **[Issue Name]**: [Description and resolution]

## Production Readiness
- [ ] Performance targets met
- [ ] No regressions introduced
- [ ] Load testing completed
- [ ] Monitoring alerts configured

## Next Steps
1. [Action item 1]
2. [Action item 2]

## Related Documents
- Analysis: `profiling-comparison-analysis-YYYYMMDD.md`
- Reports: `../results/[report-files]`

Phase 5: Troubleshooting Missing Reports

Problem: No New Reports Generated

# 1. Check if profiler is working
cd profiler/scripts
./java-profile.sh --help

# 2. Verify application is running
curl http://localhost:8080/actuator/health
jps | grep -i spring

# 3. Check profiler permissions
ls -la profiler/current/bin/asprof
chmod +x profiler/current/bin/asprof

# 4. Manual profiling attempt
java -jar profiler/current/lib/async-profiler.jar --help

Problem: Reports Are Empty or Corrupted

# 1. Check file sizes
ls -la profiler/results/*.html | tail -5

# 2. Verify HTML structure
head -20 profiler/results/[latest-report].html

# 3. Re-run with verbose logging
cd profiler/scripts
./java-profile.sh --mode memory --duration 30 --verbose

Problem: Can't Compare - Different Test Scenarios

# 1. Document your test scenarios
echo "Test scenarios used:" > profiler/docs/test-scenarios.md

# 2. Re-run with identical load patterns
# Use same curl commands, same timing, same duration

# 3. Automate test scenarios
cat > profiler/scripts/load-test.sh << 'EOF'
#!/bin/bash
for i in {1..10}; do
    curl http://localhost:8080/api/v1/objects/create
    curl http://localhost:8080/api/v1/threads/create
    sleep 5
done
EOF
chmod +x profiler/scripts/load-test.sh

File Naming Conventions

Recommended Naming Pattern

# Before refactoring (baseline)
memory-leak-baseline-YYYYMMDD-HHMMSS.html
allocation-flamegraph-baseline-YYYYMMDD-HHMMSS.html

# After refactoring
memory-leak-after-refactoring-YYYYMMDD-HHMMSS.html
allocation-flamegraph-after-refactoring-YYYYMMDD-HHMMSS.html

# Alternative: timestamp-based (if baseline already exists)
memory-leak-YYYYMMDD-HHMMSS.html (earlier = baseline, later = after)

File Organization Validation

# Verify you have complete sets
ls profiler/results/ | grep -E "(baseline|before)" | wc -l  # Should be > 0
ls profiler/results/ | grep -E "(after|refactoring)" | wc -l  # Should be > 0

Deliverables Checklist

Pre-Comparison Validation

  • Baseline profiling results exist and are complete
  • Code refactoring has been implemented and tested
  • Application runs successfully with refactored code
  • New profiling reports have been generated post-refactoring
  • Both sets of reports use identical test scenarios

Comparison Analysis Completed

  • Side-by-side flamegraph comparison performed
  • Quantitative metrics extracted and compared
  • Key improvements and regressions identified
  • Visual evidence documented with specific file references

Documentation Deliverables

  • profiler/docs/profiling-comparison-analysis-YYYYMMDD.md created
  • profiler/docs/profiling-final-results-YYYYMMDD.md created
  • Quantified performance improvements documented
  • Recommendations for production deployment provided
  • Ongoing monitoring strategy defined

Validation Commands

# Final validation that comparison is complete
echo "=== COMPARISON READINESS CHECK ==="
echo "Baseline reports: $(ls profiler/results/*baseline* 2>/dev/null | wc -l)"
echo "After reports: $(ls profiler/results/*after* 2>/dev/null | wc -l)"
echo "Analysis docs: $(ls profiler/docs/profiling-comparison-analysis-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].md 2>/dev/null | wc -l)"
echo "Final results: $(ls profiler/docs/profiling-final-results-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].md 2>/dev/null | wc -l)"

Output Format

  • Generate post-refactoring profiling reports using identical test conditions as baseline
  • Perform systematic side-by-side comparison of before/after flamegraphs and metrics
  • Create comprehensive comparison analysis documentation with quantified improvements
  • Validate that performance optimization goals were achieved through rigorous measurement
  • Identify any performance regressions introduced during optimization efforts
  • Provide clear recommendations for production deployment and ongoing monitoring

Safeguards

  • Always ensure identical test conditions between baseline and post-refactoring profiling sessions
  • Verify that both baseline and post-refactoring reports are complete and non-empty before comparison
  • Document all test scenarios and load patterns used for consistent reproduction
  • Validate that application is fully operational with refactored code before generating new reports
  • Create dated documentation files to maintain historical analysis tracking