Merge pull request #11 from amd/development

development -> main merge

Author: alexandraBara

.github/workflows/code_quality_checks.yml

@@ -1,6 +1,9 @@
 # workflow to validate code formatting using pre-commit hooks
 name: Code Quality Check
 
+permissions:
+  contents: read
+
 on: [pull_request]
 
 jobs:

CONTRIBUITING.md

@@ -1,19 +1,19 @@
-# Contribute to Scraper
+# Contribute to Node Scraper
 
 AMD values and encourages contributions to our code and documentation. If you want to contribute
 to our repository, first review the following guidance.
 
 ## Development workflow
 
-Scraper uses GitHub to host code, collaborate, and manage version control. We use pull requests (PRs)
+Node Scraper uses GitHub to host code, collaborate, and manage version control. We use pull requests (PRs)
 for all changes within our repository. We use
-[GitHub issues](https://github.com/<placeholder>/issues) to track known issues, such as
+[GitHub issues](https://github.com/amd/node-scraper/issues) to track known issues, such as
 bugs.
 
 ### Issue tracking
 
 Before filing a new issue, search the
-[existing issues](https://github.com/<placeholder>/issues) to make sure your issue isn't
+[existing issues](https://github.com/amd/node-scraper/issues) to make sure your issue isn't
 already listed.
 
 General issue guidelines:
@@ -33,12 +33,12 @@ General issue guidelines:
 ### Pull requests
 
 When you create a pull request, you should target the default branch.  Our repository uses the
-**develop** branch as the default integration branch.
+**development** branch as the default integration branch.
 
 When creating a PR, use the following process.
 
 * Identify the issue you want to fix
-* Target the default branch (usually the **develop** branch) for integration
+* Target the default branch (usually the **development** branch) for integration
 * Ensure your code builds successfully
 * Do not break existing test cases
 * New functionality is only merged with new unit tests
@@ -52,12 +52,8 @@ When creating a PR, use the following process.
 > By creating a PR, you agree to allow your contribution to be licensed under the
 > terms of the LICENSE.txt file.
 
-### New feature development
-
-Use the [GitHub Discussion forum](https://github.com/<placeholder>/discussions)
-
 ### Documentation
 
-Submit Scraper documentation changes to our
-[documentation](https://github.com/<placeholder>). You must update
+Submit Node Scraper documentation changes to our
+[documentation](https://github.com/amd/node-scraper/development/README.md). You must update
 documentation related to any new feature or API contribution.

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 AMD HPC Application Performance Team
+Copyright (c) 2025 Advanced Micro Devices, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,44 +1,47 @@
 # Node Scraper
-Node Scraper is a tool which performs automated data collection and analysis for the purposes of system debug.
+Node Scraper is a tool which performs automated data collection and analysis for the purposes of
+system debug.
 
 ## Installation
 ### Install From Source
-Node Scraper requires Python 3.10+ for installation. After cloning this repository, call dev-setup.sh script with 'source'. This script creates an editable install of Node Scraper in a python virtual environment and also configures the pre-commit hooks for the project.
+Node Scraper requires Python 3.10+ for installation. After cloning this repository,
+call dev-setup.sh script with 'source'. This script creates an editable install of Node Scraper in
+a python virtual environment and also configures the pre-commit hooks for the project.
 
 ```sh
 source dev-setup.sh
 ```
 
 ## CLI Usage
-The Node Scraper CLI can be used to run Node Scraper plugins on a target system. The following CLI options are available:
+The Node Scraper CLI can be used to run Node Scraper plugins on a target system. The following CLI
+options are available:
 
 ```sh
-usage: node-scraper [-h] [--sys-name STRING] [--sys-location {LOCAL,REMOTE}] [--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}]
-                     [--sys-sku STRING] [--sys-platform STRING] [--plugin-config STRING] [--system-config STRING] [--connection-config STRING]
-                     [--log-path STRING] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
-                     {run-plugins,gen-plugin-config,describe} ...
+usage: node-scraper [-h] [--sys-name STRING] [--sys-location {LOCAL,REMOTE}] [--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}] [--sys-sku STRING] [--sys-platform STRING] [--plugin-configs [STRING ...]]
+                    [--system-config STRING] [--connection-config STRING] [--log-path STRING] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
+                    {run-plugins,describe,gen-plugin-config} ...
 
-Error scraper CLI
+node scraper CLI
 
 positional arguments:
-  {run-plugins,gen-plugin-config,describe}
+  {run-plugins,describe,gen-plugin-config}
                         Subcommands
     run-plugins         Run a series of plugins
-    gen-plugin-config   Generate a config for a plugin or list of plugins
     describe            Display details on a built-in config or plugin
+    gen-plugin-config   Generate a config for a plugin or list of plugins
 
 options:
   -h, --help            show this help message and exit
-  --sys-name STRING     System name (default: MKM-L1-LANDRE53)
+  --sys-name STRING     System name (default: TheraC55)
   --sys-location {LOCAL,REMOTE}
                         Location of target system (default: LOCAL)
   --sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}
                         Specify system interaction level, used to determine the type of actions that plugins can perform (default: INTERACTIVE)
   --sys-sku STRING      Manually specify SKU of system (default: None)
   --sys-platform STRING
                         Specify system platform (default: None)
-  --plugin-config STRING
-                        Path to plugin config json (default: None)
+  --plugin-configs [STRING ...]
+                        built-in config names or paths to plugin config JSONs. Available built-in configs: NodeStatus (default: None)
   --system-config STRING
                         Path to system config json (default: None)
   --connection-config STRING
@@ -49,7 +52,8 @@ options:
 
 ```
 
-The plugins to run can be specified in two ways, using a plugin JSON config file or using the 'run-plugins' sub command. These two options are not mutually exclusive and can be used together.
+The plugins to run can be specified in two ways, using a plugin JSON config file or using the
+'run-plugins' sub command. These two options are not mutually exclusive and can be used together.
 
 ---
 
@@ -80,7 +84,13 @@ node-scraper describe plugin <plugin-name>
 ---
 
 ### Plugin Configs
-A plugin JSON config should follow the structure of the plugin config model defined here. The globals field is a dictionary of global key-value pairs; values in globals will be passed to any plugin that supports the corresponding key. The plugins field should be a dictionary mapping plugin names to sub-dictionaries of plugin arguments. Lastly, the result_collators attribute is used to define result collator classes that will be run on the plugin results. By default, the CLI adds the TableSummary result collator, which prints a summary of each plugin’s results in a tabular format to the console.
+A plugin JSON config should follow the structure of the plugin config model defined here.
+The globals field is a dictionary of global key-value pairs; values in globals will be passed to
+any plugin that supports the corresponding key. The plugins field should be a dictionary mapping
+plugin names to sub-dictionaries of plugin arguments. Lastly, the result_collators attribute is
+used to define result collator classes that will be run on the plugin results. By default, the CLI
+adds the TableSummary result collator, which prints a summary of each plugin’s results in a
+tabular format to the console.
 
 ```json
 {
@@ -101,13 +111,15 @@ A plugin JSON config should follow the structure of the plugin config model defi
 ```
 
 ### 'gen-plugin-config' sub command
-The 'gen-plugin-config' sub command can be used to generate a plugin config JSON file for a plugin or list of plugins that can then be customized. Plugin arguments which have default values will be prepopulated in the JSON file, arguments without default values will have a value of 'null'.
+The 'gen-plugin-config' sub command can be used to generate a plugin config JSON file for a plugin
+or list of plugins that can then be customized. Plugin arguments which have default values will be
+prepopulated in the JSON file, arguments without default values will have a value of 'null'.
 
 #### 'gen-plugin-config' Examples
 
 Generate a config for the DmesgPlugin:
 ```sh
-node-scraper gen-plugin-config DmesgPlugin
+node-scraper gen-plugin-config --plugins DmesgPlugin
 ```
 
 This would produce the following config:
@@ -134,9 +146,12 @@ This would produce the following config:
 ```
 
 ### 'run-plugins' sub command
-The plugins to run and their associated arguments can also be specified directly on the CLI using the 'run-plugins' sub-command. Using this sub-command you can specify a plugin name followed by the arguments for that particular plugin. Multiple plugins can be specified at once.
+The plugins to run and their associated arguments can also be specified directly on the CLI using
+the 'run-plugins' sub-command. Using this sub-command you can specify a plugin name followed by
+the arguments for that particular plugin. Multiple plugins can be specified at once.
 
-You can view the available arguments for a particular plugin by running `node-scraper run-plugins <plugin-name> -h`:
+You can view the available arguments for a particular plugin by running
+`node-scraper run-plugins <plugin-name> -h`:
 ```sh
 usage: node-scraper run-plugins BiosPlugin [-h] [--collection {True,False}] [--analysis {True,False}] [--system-interaction-level STRING]
                                             [--data STRING] [--exp-bios-version [STRING ...]] [--regex-match {True,False}]
@@ -175,3 +190,62 @@ Use plugin configs and 'run-plugins'
 ```sh
 node-scraper run-plugins BiosPlugin
 ```
+
+
+### '--plugin-configs' example
+A plugin config can be used to compare the system data against the config specifications:
+```sh
+node-scraper --plugin-configs plugin_config.json
+```
+Here is an example of a comprehensive plugin config that specifies analyzer args for each plugin:
+```json
+{
+  "global_args": {},
+  "plugins": {
+    "BiosPlugin": {
+      "analysis_args": {
+        "exp_bios_version": "3.5"
+      }
+    },
+    "CmdlinePlugin": {
+      "analysis_args": {
+        "cmdline": "imgurl=test NODE=nodename selinux=0 serial console=ttyS1,115200 console=tty0",
+        "required_cmdline" : "selinux=0"
+      }
+    },
+    "DkmsPlugin": {
+      "analysis_args": {
+        "dkms_status": "amdgpu/6.11",
+        "dkms_version" : "dkms-3.1",
+        "regex_match" : true
+      }
+    },
+    "KernelPlugin": {
+      "analysis_args": {
+        "exp_kernel": "5.11-generic"
+      }
+    },
+    "OsPlugin": {
+      "analysis_args": {
+        "exp_os": "Ubuntu 22.04.2 LTS"
+      }
+    },
+    "PackagePlugin": {
+          "analysis_args": {
+            "exp_package_ver": {
+              "gcc": "11.4.0"
+            },
+            "regex_match": false
+          }
+    },
+    "RocmPlugin": {
+      "analysis_args": {
+        "exp_rocm": "6.5"
+      }
+    }
+  },
+  "result_collators": {},
+  "name": "plugin_config",
+  "desc": "My golden config"
+}
+```

nodescraper/cli/cli.py

@@ -510,7 +510,7 @@ def parse_gen_plugin_config(
         sys.exit(1)
 
 
-def log_system_info(log_path: str, system_info: SystemInfo, logger: logging.Logger):
+def log_system_info(log_path: str | None, system_info: SystemInfo, logger: logging.Logger):
     """dump system info object to json log
 
     Args:
@@ -544,54 +544,61 @@ def main(arg_input: Optional[list[str]] = None):
     config_reg = ConfigRegistry()
     parser, plugin_subparser_map = build_parser(plugin_reg, config_reg)
 
-    top_level_args, plugin_arg_map = process_args(arg_input, list(plugin_subparser_map.keys()))
-
-    parsed_args = parser.parse_args(top_level_args)
-
-    if parsed_args.log_path and parsed_args.subcmd not in ["gen-plugin-config", "describe"]:
-        log_path = os.path.join(
-            parsed_args.log_path,
-            f"scraper_logs_{datetime.datetime.now().strftime('%Y_%m_%d-%I_%M_%S_%p')}",
-        )
-        os.makedirs(log_path)
-    else:
-        log_path = None
-
-    logger = setup_logger(parsed_args.log_level, log_path)
-    if log_path:
-        logger.info("Log path: %s", log_path)
-
-    if parsed_args.subcmd == "describe":
-        parse_describe(parsed_args, plugin_reg, config_reg, logger)
-
-    if parsed_args.subcmd == "gen-plugin-config":
-        parse_gen_plugin_config(parsed_args, plugin_reg, config_reg, logger)
-
-    parsed_plugin_args = {}
-    for plugin, plugin_args in plugin_arg_map.items():
-        try:
-            parsed_plugin_args[plugin] = plugin_subparser_map[plugin][0].parse_args(plugin_args)
-        except Exception:
-            logger.exception("Exception parsing args for plugin: %s", plugin)
+    try:
+        top_level_args, plugin_arg_map = process_args(arg_input, list(plugin_subparser_map.keys()))
 
-    if not parsed_plugin_args and not parsed_args.plugin_configs:
-        logger.info("No plugins config args specified, running default config: %s", DEFAULT_CONFIG)
-        plugin_configs = [DEFAULT_CONFIG]
-    else:
-        plugin_configs = parsed_args.plugin_configs or []
+        parsed_args = parser.parse_args(top_level_args)
 
-    system_info = get_system_info(parsed_args)
-    log_system_info(log_path, system_info, logger)
+        if parsed_args.log_path and parsed_args.subcmd not in ["gen-plugin-config", "describe"]:
+            log_path = os.path.join(
+                parsed_args.log_path,
+                f"scraper_logs_{datetime.datetime.now().strftime('%Y_%m_%d-%I_%M_%S_%p')}",
+            )
+            os.makedirs(log_path)
+        else:
+            log_path = None
+
+        logger = setup_logger(parsed_args.log_level, log_path)
+        if log_path:
+            logger.info("Log path: %s", log_path)
+
+        if parsed_args.subcmd == "describe":
+            parse_describe(parsed_args, plugin_reg, config_reg, logger)
+
+        if parsed_args.subcmd == "gen-plugin-config":
+            parse_gen_plugin_config(parsed_args, plugin_reg, config_reg, logger)
+
+        parsed_plugin_args = {}
+        for plugin, plugin_args in plugin_arg_map.items():
+            try:
+                parsed_plugin_args[plugin] = plugin_subparser_map[plugin][0].parse_args(plugin_args)
+            except Exception as e:
+                logger.error("%s exception parsing args for plugin: %s", str(e), plugin)
+
+        if not parsed_plugin_args and not parsed_args.plugin_configs:
+            logger.info(
+                "No plugins config args specified, running default config: %s", DEFAULT_CONFIG
+            )
+            plugin_configs = [DEFAULT_CONFIG]
+        else:
+            plugin_configs = parsed_args.plugin_configs or []
 
-    plugin_executor = PluginExecutor(
-        logger=logger,
-        plugin_configs=get_plugin_configs(
+        plugin_config_inst_list = get_plugin_configs(
             plugin_config_input=plugin_configs,
             system_interaction_level=parsed_args.sys_interaction_level,
             built_in_configs=config_reg.configs,
             parsed_plugin_args=parsed_plugin_args,
             plugin_subparser_map=plugin_subparser_map,
-        ),
+        )
+
+        system_info = get_system_info(parsed_args)
+        log_system_info(log_path, system_info, logger)
+    except Exception as e:
+        parser.error(str(e))
+
+    plugin_executor = PluginExecutor(
+        logger=logger,
+        plugin_configs=plugin_config_inst_list,
         connections=parsed_args.connection_config,
         system_info=system_info,
         log_path=log_path,

nodescraper/cli/inputargtypes.py

@@ -104,7 +104,7 @@ def process_file_arg(self, file_path: str) -> TModelType:
             return self.model(**data)
         except ValidationError as e:
             raise argparse.ArgumentTypeError(
-                f"Validation errors when processing {file_path}: {e.errors()}"
+                f"Validation errors when processing {file_path}: {e.errors(include_url=False)}"
             ) from e
 
 

nodescraper/interfaces/dataanalyzertask.py

@@ -76,13 +76,12 @@ def wrapper(
                     if not analyze_arg_model:
                         raise ValueError("No model defined for analysis args")
                     args = analyze_arg_model(**args)  # type: ignore
-
                 func(analyzer, data, args)
             except ValidationError as exception:
                 analyzer._log_event(
                     category=EventCategory.RUNTIME,
                     description="Validation error during analysis",
-                    data=get_exception_traceback(exception),
+                    data={"errors": exception.errors(include_url=False)},
                     priority=EventPriority.CRITICAL,
                     console_log=True,
                 )