fix(dds-submit-slurm): resolve SLURM job submission failures

Fixed a critical bug in the SLURM plugin that caused job submissions
to fail with "No partition specified" error when using lightweight
mode. The issue was due to executable code appearing before #SBATCH
directives in the generated job script, violating SLURM's parsing
requirements.

* Removed lightweight validation code from the job script template.
* Eliminated blank lines between #SBATCH directive placeholders.
* Validation logic is now handled within worker nodes.

This ensures SLURM correctly parses all #SBATCH directives,
including partition specifications and resource requirements.

Author: AnarManafov

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - **DDSWorker.sh**: Fixed inverted logic bug that caused worker package deployment to fail when pre-compiled binaries were present. The script now correctly handles both full packages (with binaries) and lightweight packages (without binaries).
+- **dds-submit-slurm**: Fixed critical bug in lightweight mode where #SBATCH directives were ignored by SLURM scheduler, causing job submission failures with "No partition specified" error. The issue was caused by executable code appearing before #SBATCH directives in the generated job script, violating SLURM's parsing requirements. Fixed by removing the lightweight validation code from the job script template and eliminating blank lines between #SBATCH directive placeholders.
 
 ## [3.15.0] - 2025-10-08
 

ReleaseNotes.md

@@ -14,6 +14,17 @@
 
 ### 🐛 Bug Fixes
 
+#### Critical SLURM Plugin Fix for Lightweight Mode
+
+- **Job Submission Failure**: Fixed critical bug in SLURM plugin that caused job submissions to fail with "No partition specified or system default partition" error when using lightweight mode
+- **Root Cause**: The job script template incorrectly placed executable validation code before #SBATCH directives, violating SLURM's parsing requirements. SLURM stops processing #SBATCH options when it encounters the first executable line, causing all subsequent directives (including `--partition`) to be ignored
+- **Template Bug**: The placeholder `#DDS_LIGHTWEIGHT_VALIDATION` appeared in both a comment line and the code section. The `boost::replace_all()` function replaced both occurrences, breaking the comment syntax and injecting executable code before #SBATCH directives
+- **Resolution**: 
+  - Removed lightweight validation code from the job script template entirely
+  - Eliminated blank lines between #SBATCH directive placeholders
+  - Validation logic moved to worker nodes where it's actually needed (DDSWorker.sh)
+- **Impact**: SLURM now correctly parses all #SBATCH directives including partition specifications, resource requirements, and job options
+
 #### Critical Worker Package Deployment Fix
 
 - **DDSWorker.sh Logic Error**: Fixed inverted logic bug that caused worker package deployment to fail when pre-compiled binaries were present
@@ -24,6 +35,26 @@
 
 ### 🚀 For Users
 
+#### If You Use SLURM with Lightweight Mode
+
+If you experienced SLURM job submission failures with errors like:
+
+```text
+Batch job submission failed: No partition specified or system default partition
+```
+
+This was caused by a critical bug in the SLURM plugin template that placed executable code before #SBATCH directives. SLURM stopped parsing directives when it encountered this code, ignoring your partition specifications and other settings.
+
+**The fix requires rebuilding DDS:**
+
+```bash
+cd /path/to/DDS/build
+make
+make install
+```
+
+After rebuilding, your SLURM submissions with lightweight mode will work correctly, and all #SBATCH directives (including partition, CPU requirements, etc.) will be properly recognized.
+
 #### If You Use Tools API
 
 Before this fix, you had to explicitly set the lightweight flag:

plugins/dds-submit-slurm/src/job.slurm.in

@@ -3,46 +3,16 @@
 #
 # DDS SLURM Job Execution Workflow:
 #
-# PHASE 1: Resource Allocation & Job Scheduling
-# - SLURM scheduler allocates nodes based on #SBATCH directives
-# - Job script is dispatched to a single allocation node (sbatch host)
-#
-# PHASE 2: Sbatch Host Execution (Single Node)
-# - Script executes sequentially on the sbatch host
-# - Lightweight package validation (#DDS_LIGHTWEIGHT_VALIDATION) runs once
-#   * Validates DDS_COMMANDER_BIN_LOCATION environment variable
-#   * Validates DDS_COMMANDER_LIBS_LOCATION environment variable
-#   * Provides immediate failure feedback if prerequisites are missing
-# - Signal handlers are configured to manage process lifecycle
-#
-# PHASE 3: Parallel Task Distribution (Multiple Nodes)
-# - srun command distributes DDS worker tasks across all allocated nodes
-# - Each allocated node receives and executes DDSWorker.sh scout script
-# - Per-node validation occurs within DDSWorker.sh on each srun host:
-#   * Secondary validation of environment variables
-#   * DDS agent initialization and registration
-#   * Worker-specific error handling and logging
-#
-# PHASE 4: Job Lifecycle Management
-# - Parent process waits for all srun tasks to complete
-# - Signal propagation ensures graceful shutdown of distributed tasks
-# - Job artifacts and logs are collected per-node basis
-#
 
 #SBATCH --nodes=%DDS_NMININSTANCES%%DDS_NINSTANCES%
 #SBATCH --no-kill
 
 #SBATCH --job-name=%DDS_SUBMISSION_TAG%
 #SBATCH --chdir=%DDS_JOB_ROOT_WRK_DIR%
-
 #DDS_AGENT_CPU_REQUIREMENT
-
 #DDS_INLINE_CONFIG
-
 #DDS_USER_OPTIONS
 
-#DDS_LIGHTWEIGHT_VALIDATION
-
 # ignore signals
 # continue waiting for child processes by any means
 trap -- '' SIGINT SIGTERM

plugins/dds-submit-slurm/src/main.cpp

@@ -203,51 +203,6 @@ int main(int argc, char* argv[])
                     // Replace %DDS_SUBMISSION_TAG%
                     boost::replace_all(sSrcScript, "%DDS_SUBMISSION_TAG%", _submit.m_submissionTag);
 
-                    // #DDS_LIGHTWEIGHT_VALIDATION
-                    if (isLightweightMode)
-                    {
-                        string lightweightValidation = R"DELIMITER(
-# Early validation for lightweight mode
-echo "Lightweight mode detected. Validating prerequisites..."
-
-# Check DDS_COMMANDER_BIN_LOCATION
-if [[ -z "${DDS_COMMANDER_BIN_LOCATION}" ]]; then
-    echo "ERROR: DDS_COMMANDER_BIN_LOCATION environment variable is not set"
-    echo "Please set it to point to DDS binaries directory (e.g., /opt/dds/bin)"
-    exit 1
-fi
-
-if [[ ! -d "${DDS_COMMANDER_BIN_LOCATION}" ]]; then
-    echo "ERROR: DDS_COMMANDER_BIN_LOCATION points to non-existent directory: ${DDS_COMMANDER_BIN_LOCATION}"
-    exit 1
-fi
-
-if [[ ! -x "${DDS_COMMANDER_BIN_LOCATION}/dds-agent" ]]; then
-    echo "ERROR: Cannot find dds-agent executable in ${DDS_COMMANDER_BIN_LOCATION}"
-    exit 1
-fi
-
-# Check DDS_COMMANDER_LIBS_LOCATION
-if [[ -z "${DDS_COMMANDER_LIBS_LOCATION}" ]]; then
-    echo "ERROR: DDS_COMMANDER_LIBS_LOCATION environment variable is not set"
-    echo "Please set it to point to DDS libraries directory (e.g., /opt/dds/lib)"
-    exit 1
-fi
-
-if [[ ! -d "${DDS_COMMANDER_LIBS_LOCATION}" ]]; then
-    echo "ERROR: DDS_COMMANDER_LIBS_LOCATION points to non-existent directory: ${DDS_COMMANDER_LIBS_LOCATION}"
-    exit 1
-fi
-
-echo "Lightweight mode prerequisites validated successfully"
-)DELIMITER";
-                        boost::replace_all(sSrcScript, "#DDS_LIGHTWEIGHT_VALIDATION", lightweightValidation);
-                    }
-                    else
-                    {
-                        boost::replace_all(sSrcScript, "#DDS_LIGHTWEIGHT_VALIDATION", "");
-                    }
-
                     // Replace %DDS_JOB_ROOT_WRK_DIR%
                     string sSandboxDir(smart_path(CUserDefaults::instance().getWrkPkgDir(submissionId)));
                     fs::path pathJobWrkDir(sSandboxDir);