Merge pull request #103 from amd/development

dev -> main

Author: alexandraBara

.github/workflows/update-plugin-docs.yml

@@ -40,9 +40,16 @@ jobs:
           --package nodescraper.plugins.inband \
           --output docs/PLUGIN_DOC.md
 
+    - name: Clean pre-commit cache
+      run: |
+        rm -rf /tmp/github-actions-home/.cache/pre-commit
+        source venv/bin/activate
+        pre-commit clean || true
+
     - name: Format documentation with pre-commit
       run: |
         source venv/bin/activate
+        pre-commit install-hooks || true
         pre-commit run --files docs/PLUGIN_DOC.md || true
 
     - name: Create Pull Request

docs/PLUGIN_DOC.md

@@ -11,11 +11,12 @@
 | DimmPlugin | sh -c 'dmidecode -t 17 \| tr -s " " \| grep -v "Volatile\\|None\\|Module" \| grep Size' 2>/dev/null<br>dmidecode<br>wmic memorychip get Capacity | - | [DimmDataModel](#DimmDataModel-Model) | [DimmCollector](#Collector-Class-DimmCollector) | - |
 | DkmsPlugin | dkms status<br>dkms --version | **Analyzer Args:**<br>- `dkms_status`: Union[str, list]<br>- `dkms_version`: Union[str, list]<br>- `regex_match`: bool | [DkmsDataModel](#DkmsDataModel-Model) | [DkmsCollector](#Collector-Class-DkmsCollector) | [DkmsAnalyzer](#Data-Analyzer-Class-DkmsAnalyzer) |
 | DmesgPlugin | dmesg --time-format iso -x<br>ls -1 /var/log/dmesg* 2>/dev/null \| grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' \|\| true | **Built-in Regexes:**<br>- Out of memory error: `(?:oom_kill_process.*)\|(?:Out of memory.*)`<br>- I/O Page Fault: `IO_PAGE_FAULT`<br>- Kernel Panic: `\bkernel panic\b.*`<br>- SQ Interrupt: `sq_intr`<br>- SRAM ECC: `sram_ecc.*`<br>- Failed to load driver. IP hardware init error.: `\[amdgpu\]\] \*ERROR\* hw_init of IP block.*`<br>- Failed to load driver. IP software init error.: `\[amdgpu\]\] \*ERROR\* sw_init of IP block.*`<br>- Real Time throttling activated: `sched: RT throttling activated.*`<br>- RCU preempt detected stalls: `rcu_preempt detected stalls.*`<br>- RCU preempt self-detected stall: `rcu_preempt self-detected stall.*`<br>- QCM fence timeout: `qcm fence wait loop timeout.*`<br>- General protection fault: `(?:[\w-]+(?:\[[0-9.]+\])?\s+)?general protectio...`<br>- Segmentation fault: `(?:segfault.*in .*\[)\|(?:[Ss]egmentation [Ff]au...`<br>- Failed to disallow cf state: `amdgpu: Failed to disallow cf state.*`<br>- Failed to terminate tmr: `\*ERROR\* Failed to terminate tmr.*`<br>- Suspend of IP block failed: `\*ERROR\* suspend of IP block <\w+> failed.*`<br>- amdgpu Page Fault: `(amdgpu \w{4}:\w{2}:\w{2}\.\w:\s+amdgpu:\s+\[\S...`<br>- Page Fault: `page fault for address.*`<br>- Fatal error during GPU init: `(?:amdgpu)(.*Fatal error during GPU init)\|(Fata...`<br>- PCIe AER Error: `(?:pcieport )(.*AER: aer_status.*)\|(aer_status.*)`<br>- Failed to read journal file: `Failed to read journal file.*`<br>- Journal file corrupted or uncleanly shut down: `journal corrupted or uncleanly shut down.*`<br>- ACPI BIOS Error: `ACPI BIOS Error`<br>- ACPI Error: `ACPI Error`<br>- Filesystem corrupted!: `EXT4-fs error \(device .*\):`<br>- Error in buffered IO, check filesystem integrity: `(Buffer I\/O error on dev)(?:ice)? (\w+)`<br>- PCIe card no longer present: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`<br>- PCIe Link Down: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`<br>- Mismatched clock configuration between PCIe device and host: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(curren...`<br>- RAS Correctable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- RAS Uncorrectable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- RAS Deferred Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- RAS Corrected PCIe Error: `((?:\[Hardware Error\]:\s+)?event severity: cor...`<br>- GPU Reset: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- GPU reset failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`<br>- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`<br>- MCE Error: `\[Hardware Error\]:.+MC\d+_STATUS.*(?:\n.*){0,5}`<br>- Mode 2 Reset Failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)? (...`<br>- RAS Corrected Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`<br>- SGX Error: `x86/cpu: SGX disabled by BIOS`<br>- GPU Throttled: `amdgpu \w{4}:\w{2}:\w{2}.\w: amdgpu: WARN: GPU ...`<br>- LNet: ko2iblnd has no matching interfaces: `(?:\[[^\]]+\]\s*)?LNetError:.*ko2iblnd:\s*No ma...`<br>- LNet: Error starting up LNI: `(?:\[[^\]]+\]\s*)?LNetError:\s*.*Error\s*-?\d+\...`<br>- Lustre: network initialisation failed: `LustreError:.*ptlrpc_init_portals\(\).*network ...` | [DmesgData](#DmesgData-Model) | [DmesgCollector](#Collector-Class-DmesgCollector) | [DmesgAnalyzer](#Data-Analyzer-Class-DmesgAnalyzer) |
+| FabricsPlugin | ibstat<br>ibv_devinfo<br>ls -l /sys/class/infiniband/*/device/net<br>mst start<br>mst status -v<br>ofed_info -s<br>rdma dev<br>rdma link | - | [FabricsDataModel](#FabricsDataModel-Model) | [FabricsCollector](#Collector-Class-FabricsCollector) | - |
 | JournalPlugin | journalctl --no-pager --system --output=short-iso | - | [JournalData](#JournalData-Model) | [JournalCollector](#Collector-Class-JournalCollector) | - |
 | KernelPlugin | sh -c 'uname -a'<br>wmic os get Version /Value | **Analyzer Args:**<br>- `exp_kernel`: Union[str, list]<br>- `regex_match`: bool | [KernelDataModel](#KernelDataModel-Model) | [KernelCollector](#Collector-Class-KernelCollector) | [KernelAnalyzer](#Data-Analyzer-Class-KernelAnalyzer) |
 | KernelModulePlugin | cat /proc/modules<br>modinfo amdgpu<br>wmic os get Version /Value | **Analyzer Args:**<br>- `kernel_modules`: dict[str, dict]<br>- `regex_filter`: list[str] | [KernelModuleDataModel](#KernelModuleDataModel-Model) | [KernelModuleCollector](#Collector-Class-KernelModuleCollector) | [KernelModuleAnalyzer](#Data-Analyzer-Class-KernelModuleAnalyzer) |
 | MemoryPlugin | free -b<br>lsmem<br>numactl -H<br>wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value | **Analyzer Args:**<br>- `ratio`: float<br>- `memory_threshold`: str | [MemoryDataModel](#MemoryDataModel-Model) | [MemoryCollector](#Collector-Class-MemoryCollector) | [MemoryAnalyzer](#Data-Analyzer-Class-MemoryAnalyzer) |
-| NetworkPlugin | ip addr show<br>sudo ethtool {interface}<br>ip neighbor show<br>ip route show<br>ip rule show | - | [NetworkDataModel](#NetworkDataModel-Model) | [NetworkCollector](#Collector-Class-NetworkCollector) | - |
+| NetworkPlugin | ip addr show<br>ethtool {interface}<br>lldpcli show neighbor<br>lldpctl<br>ip neighbor show<br>niccli --dev {device_num} qos --ets --show<br>niccli --list_devices<br>nicctl show card<br>nicctl show dcqcn<br>nicctl show environment<br>nicctl show pcie ats<br>nicctl show port<br>nicctl show qos<br>nicctl show rdma statistics<br>nicctl show version firmware<br>nicctl show version host-software<br>ip route show<br>ip rule show | - | [NetworkDataModel](#NetworkDataModel-Model) | [NetworkCollector](#Collector-Class-NetworkCollector) | - |
 | NvmePlugin | nvme smart-log {dev}<br>nvme error-log {dev} --log-entries=256<br>nvme id-ctrl {dev}<br>nvme id-ns {dev}{ns}<br>nvme fw-log {dev}<br>nvme self-test-log {dev}<br>nvme get-log {dev} --log-id=6 --log-len=512<br>nvme telemetry-log {dev} --output-file={dev}_{f_name} | - | [NvmeDataModel](#NvmeDataModel-Model) | [NvmeCollector](#Collector-Class-NvmeCollector) | - |
 | OsPlugin | sh -c '( lsb_release -ds \|\| (cat /etc/*release \| grep PRETTY_NAME) \|\| uname -om ) 2>/dev/null \| head -n1'<br>cat /etc/*release \| grep VERSION_ID<br>wmic os get Version /value<br>wmic os get Caption /Value | **Analyzer Args:**<br>- `exp_os`: Union[str, list]<br>- `exact_match`: bool | [OsDataModel](#OsDataModel-Model) | [OsCollector](#Collector-Class-OsCollector) | [OsAnalyzer](#Data-Analyzer-Class-OsAnalyzer) |
 | PackagePlugin | dnf list --installed<br>dpkg-query -W<br>pacman -Q<br>cat /etc/*release<br>wmic product get name,version | **Analyzer Args:**<br>- `exp_package_ver`: Dict[str, Optional[str]]<br>- `regex_match`: bool<br>- `rocm_regex`: Optional[str]<br>- `enable_rocm_regex`: bool | [PackageDataModel](#PackageDataModel-Model) | [PackageCollector](#Collector-Class-PackageCollector) | [PackageAnalyzer](#Data-Analyzer-Class-PackageAnalyzer) |
@@ -224,6 +225,42 @@ DmesgData
 - dmesg --time-format iso -x
 - ls -1 /var/log/dmesg* 2>/dev/null | grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' || true
 
+## Collector Class FabricsCollector
+
+### Description
+
+Collect InfiniBand/RDMA fabrics configuration details
+
+**Bases**: ['InBandDataCollector']
+
+**Link to code**: [fabrics_collector.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/fabrics/fabrics_collector.py)
+
+### Class Variables
+
+- **CMD_IBSTAT**: `ibstat`
+- **CMD_IBV_DEVINFO**: `ibv_devinfo`
+- **CMD_IB_DEV_NETDEVS**: `ls -l /sys/class/infiniband/*/device/net`
+- **CMD_OFED_INFO**: `ofed_info -s`
+- **CMD_MST_START**: `mst start`
+- **CMD_MST_STATUS**: `mst status -v`
+- **CMD_RDMA_DEV**: `rdma dev`
+- **CMD_RDMA_LINK**: `rdma link`
+
+### Provides Data
+
+FabricsDataModel
+
+### Commands
+
+- ibstat
+- ibv_devinfo
+- ls -l /sys/class/infiniband/*/device/net
+- mst start
+- mst status -v
+- ofed_info -s
+- rdma dev
+- rdma link
+
 ## Collector Class JournalCollector
 
 ### Description
@@ -341,7 +378,20 @@ Collect network configuration details using ip command
 - **CMD_ROUTE**: `ip route show`
 - **CMD_RULE**: `ip rule show`
 - **CMD_NEIGHBOR**: `ip neighbor show`
-- **CMD_ETHTOOL_TEMPLATE**: `sudo ethtool {interface}`
+- **CMD_ETHTOOL_TEMPLATE**: `ethtool {interface}`
+- **CMD_LLDPCLI_NEIGHBOR**: `lldpcli show neighbor`
+- **CMD_LLDPCTL**: `lldpctl`
+- **CMD_NICCLI_LISTDEV**: `niccli --list_devices`
+- **CMD_NICCLI_GETQOS_TEMPLATE**: `niccli --dev {device_num} qos --ets --show`
+- **CMD_NICCTL_CARD**: `nicctl show card`
+- **CMD_NICCTL_DCQCN**: `nicctl show dcqcn`
+- **CMD_NICCTL_ENVIRONMENT**: `nicctl show environment`
+- **CMD_NICCTL_PCIE_ATS**: `nicctl show pcie ats`
+- **CMD_NICCTL_PORT**: `nicctl show port`
+- **CMD_NICCTL_QOS**: `nicctl show qos`
+- **CMD_NICCTL_RDMA_STATISTICS**: `nicctl show rdma statistics`
+- **CMD_NICCTL_VERSION_HOST_SOFTWARE**: `nicctl show version host-software`
+- **CMD_NICCTL_VERSION_FIRMWARE**: `nicctl show version firmware`
 
 ### Provides Data
 
@@ -350,8 +400,21 @@ NetworkDataModel
 ### Commands
 
 - ip addr show
-- sudo ethtool {interface}
+- ethtool {interface}
+- lldpcli show neighbor
+- lldpctl
 - ip neighbor show
+- niccli --dev {device_num} qos --ets --show
+- niccli --list_devices
+- nicctl show card
+- nicctl show dcqcn
+- nicctl show environment
+- nicctl show pcie ats
+- nicctl show port
+- nicctl show qos
+- nicctl show rdma statistics
+- nicctl show version firmware
+- nicctl show version host-software
 - ip route show
 - ip rule show
 
@@ -769,6 +832,26 @@ Data model for in band dmesg log
 ### Model annotations and fields
 
 - **dmesg_content**: `str`
+- **skip_log_file**: `bool`
+
+## FabricsDataModel Model
+
+### Description
+
+Complete InfiniBand/RDMA fabrics configuration data
+
+**Link to code**: [fabricsdata.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/fabrics/fabricsdata.py)
+
+**Bases**: ['DataModel']
+
+### Model annotations and fields
+
+- **ibstat_devices**: `List[nodescraper.plugins.inband.fabrics.fabricsdata.IbstatDevice]`
+- **ibv_devices**: `List[nodescraper.plugins.inband.fabrics.fabricsdata.IbvDeviceInfo]`
+- **ibdev_netdev_mappings**: `List[nodescraper.plugins.inband.fabrics.fabricsdata.IbdevNetdevMapping]`
+- **ofed_info**: `Optional[nodescraper.plugins.inband.fabrics.fabricsdata.OfedInfo]`
+- **mst_status**: `Optional[nodescraper.plugins.inband.fabrics.fabricsdata.MstStatus]`
+- **rdma_info**: `Optional[nodescraper.plugins.inband.fabrics.fabricsdata.RdmaInfo]`
 
 ## JournalData Model
 
@@ -840,6 +923,17 @@ Complete network configuration data
 - **rules**: `List[nodescraper.plugins.inband.network.networkdata.RoutingRule]`
 - **neighbors**: `List[nodescraper.plugins.inband.network.networkdata.Neighbor]`
 - **ethtool_info**: `Dict[str, nodescraper.plugins.inband.network.networkdata.EthtoolInfo]`
+- **broadcom_nic_devices**: `List[nodescraper.plugins.inband.network.networkdata.BroadcomNicDevice]`
+- **broadcom_nic_qos**: `Dict[int, nodescraper.plugins.inband.network.networkdata.BroadcomNicQos]`
+- **pensando_nic_cards**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicCard]`
+- **pensando_nic_dcqcn**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicDcqcn]`
+- **pensando_nic_environment**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicEnvironment]`
+- **pensando_nic_pcie_ats**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicPcieAts]`
+- **pensando_nic_ports**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicPort]`
+- **pensando_nic_qos**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicQos]`
+- **pensando_nic_rdma_statistics**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicRdmaStatistics]`
+- **pensando_nic_version_host_software**: `Optional[nodescraper.plugins.inband.network.networkdata.PensandoNicVersionHostSoftware]`
+- **pensando_nic_version_firmware**: `List[nodescraper.plugins.inband.network.networkdata.PensandoNicVersionFirmware]`
 
 ## NvmeDataModel Model
 

docs/node-scraper-external/README.md

@@ -1,7 +1,12 @@
 # node-scraper external plugins (example)
 
 This directory lives at **`/docs/node-scraper-external`** in the `node-scraper` repo and contains
-an example external plugin package you can install in editable mode.
+an example external plugin package that demonstrates how to create plugins for node-scraper.
+
+## Overview
+
+External plugins are discovered by node-scraper via **Python entry points**. This allows plugins
+to be distributed as separate packages and automatically discovered when installed.
 
 ## Installation
 
@@ -12,44 +17,126 @@ cd ~/node-scraper
 source venv/bin/activate
 pip install -e ./docs/node-scraper-external
 ```
-You should see `ext-nodescraper-plugins` installed in editable mode.
 
+This installs `ext-nodescraper-plugins` in editable mode and registers the plugin entry points.
 
-## Verify the external package is importable
+## Verify Plugin Discovery
+
+Check that node-scraper discovered the external plugin:
 
 ```bash
-python - <<'PY'
-import ext_nodescraper_plugins
-print("ext_nodescraper_plugins loaded from:", ext_nodescraper_plugins.__file__)
-PY
+node-scraper run-plugins -h
 ```
 
-## Run external plugins
+You should see `SamplePlugin` listed alongside built-in plugins.
 
-Confirm the CLI sees your external plugin(s):
+## Run the Example Plugin
 
 ```bash
-node-scraper run-plugins -h
 node-scraper run-plugins SamplePlugin
 ```
 
-## Add your own plugins
+## How It Works
+
+### Entry Points
 
-Add new modules under the **`ext_nodescraper_plugins/`** package. Example layout:
+Plugins are registered in `pyproject.toml` using entry points:
+
+```toml
+[project.entry-points."nodescraper.plugins"]
+SamplePlugin = "ext_nodescraper_plugins.sample.sample_plugin:SamplePlugin"
+```
+
+When you install the package, Python registers these entry points in the package metadata.
+Node-scraper automatically discovers and loads plugins from the `nodescraper.plugins` entry point group.
+
+### Plugin Structure
 
 ```
 /docs/node-scraper-external
-├─ pyproject.toml
-└─ ext_nodescraper_plugins/
-   └─ sample/
+├─ pyproject.toml              # Package metadata + entry points
+└─ ext_nodescraper_plugins/    # Plugin package
+   └─ sample/                   # Plugin module
       ├─ __init__.py
-      └─ sample_plugin.py
+      ├─ sample_plugin.py       # Plugin class
+      ├─ sample_collector.py    # Data collector
+      ├─ sample_analyzer.py     # Data analyzer
+      └─ sample_data.py         # Data model
+```
+
+## Creating Your Own External Plugins
+
+### Step 1: Create Package Structure
+
+```bash
+mkdir my-plugin-package
+cd my-plugin-package
+mkdir -p ext_nodescraper_plugins/my_plugin
 ```
 
+### Step 2: Create pyproject.toml
+
+```toml
+[project]
+name = "my-plugin-package"
+version = "0.1.0"
+requires-python = ">=3.10"
+dependencies = ["amd-node-scraper"]
+
+[project.entry-points."nodescraper.plugins"]
+MyPlugin = "ext_nodescraper_plugins.my_plugin:MyPlugin"
+
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
 ```
 
-Re-install (editable mode picks up code changes automatically, but if you add new files you may
-need to re-run):
+### Step 3: Implement Your Plugin
+
+Create `ext_nodescraper_plugins/my_plugin/__init__.py`:
+
+```python
+from nodescraper.base import InBandDataPlugin, InBandDataCollector
+from pydantic import BaseModel
+
+class MyDataModel(BaseModel):
+    """Your data model"""
+    data: dict
+
+class MyCollector(InBandDataCollector[MyDataModel, None]):
+    """Your data collector"""
+    DATA_MODEL = MyDataModel
+
+    def collect_data(self, args=None):
+        # Collection logic
+        return MyDataModel(data={})
+
+class MyPlugin(InBandDataPlugin[MyDataModel, None, None]):
+    """Your plugin"""
+    DATA_MODEL = MyDataModel
+    COLLECTOR = MyCollector
+```
+
+### Step 4: Install and Test
+
 ```bash
 pip install -e .
+node-scraper run-plugins -h  # Should show MyPlugin
+node-scraper run-plugins MyPlugin
 ```
+
+## Adding More Plugins to This Package
+
+To add additional plugins to this example package:
+
+1. **Create a new module** under `ext_nodescraper_plugins/`
+2. **Register the entry point** in `pyproject.toml`:
+   ```toml
+   [project.entry-points."nodescraper.plugins"]
+   SamplePlugin = "ext_nodescraper_plugins.sample.sample_plugin:SamplePlugin"
+   AnotherPlugin = "ext_nodescraper_plugins.another:AnotherPlugin"
+   ```
+3. **Reinstall** to register the new entry point:
+   ```bash
+   pip install -e . --force-reinstall --no-deps
+   ```

docs/node-scraper-external/pyproject.toml

@@ -4,6 +4,9 @@ version = "0.1.0"
 requires-python = ">=3.10"
 dependencies = ["node-scraper"]
 
+[project.entry-points."nodescraper.plugins"]
+SamplePlugin = "ext_nodescraper_plugins.sample.sample_plugin:SamplePlugin"
+
 [build-system]
 requires = ["setuptools", "wheel"]
 build-backend = "setuptools.build_meta"

nodescraper/cli/cli.py

@@ -54,13 +54,6 @@
 from nodescraper.pluginexecutor import PluginExecutor
 from nodescraper.pluginregistry import PluginRegistry
 
-try:
-    import ext_nodescraper_plugins as ext_pkg
-
-    extra_pkgs = [ext_pkg]
-except ImportError:
-    extra_pkgs = []
-
 
 def build_parser(
     plugin_reg: PluginRegistry,
@@ -376,7 +369,7 @@ def main(arg_input: Optional[list[str]] = None):
     if arg_input is None:
         arg_input = sys.argv[1:]
 
-    plugin_reg = PluginRegistry(plugin_pkg=extra_pkgs)
+    plugin_reg = PluginRegistry()
 
     config_reg = ConfigRegistry()
     parser, plugin_subparser_map = build_parser(plugin_reg, config_reg)

nodescraper/interfaces/dataplugin.py

@@ -252,6 +252,9 @@ def analyze(
             )
             return self.analysis_result
 
+        if data:
+            self.data = data
+
         if self.data is None:
             self.analysis_result = TaskResult(
                 task=self.ANALYZER.__name__,
@@ -261,9 +264,6 @@ def analyze(
             )
             return self.analysis_result
 
-        if data:
-            self.data = data
-
         analyzer_task = self.ANALYZER(
             self.system_info,
             logger=self.logger,