Skip to content

Commit f4395f7

Browse files
landrews-amdlandrews-amd
authored
Merge pull request #51 from amd/development
dev -> main merge
2 parents 7611321 + 6081dd1 commit f4395f7

151 files changed

Lines changed: 6923 additions & 851 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/code_quality_checks.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,12 @@ jobs:
1313

1414
steps:
1515
- uses: actions/checkout@v3
16-
- uses: pre-commit/action@v3.0.1
17-
16+
- name: setup environment
17+
run: |
18+
./dev-setup.sh
19+
- name: run pre-commit hooks
20+
run: |
21+
pre-commit run --all-files --show-diff-on-failure --color=always
1822
- name: Print message on failure
1923
if: failure()
2024
run: |

‎.mypy.ini‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
[mypy]
2+
# Global mypy configuration
3+
4+
[mypy-nodescraper.base.regexanalyzer]
5+
ignore_errors = True
6+
7+
[mypy-nodescraper.cli.cli]
8+
ignore_errors = True
9+
10+
[mypy-nodescraper.cli.dynamicparserbuilder]
11+
ignore_errors = True
12+
13+
[mypy-nodescraper.cli.helper]
14+
ignore_errors = True
15+
16+
[mypy-nodescraper.cli.inputargtypes]
17+
ignore_errors = True
18+
19+
[mypy-nodescraper.configbuilder]
20+
ignore_errors = True
21+
22+
[mypy-nodescraper.configregistry]
23+
ignore_errors = True
24+
25+
[mypy-nodescraper.enums.eventpriority]
26+
ignore_errors = True
27+
28+
[mypy-nodescraper.enums.systeminteraction]
29+
ignore_errors = True
30+
31+
[mypy-nodescraper.interfaces.connectionmanager]
32+
ignore_errors = True
33+
34+
[mypy-nodescraper.interfaces.datacollectortask]
35+
ignore_errors = True
36+
37+
[mypy-nodescraper.interfaces.dataplugin]
38+
ignore_errors = True
39+
40+
[mypy-nodescraper.interfaces.task]
41+
ignore_errors = True
42+
43+
[mypy-nodescraper.models.analyzerargs]
44+
ignore_errors = True
45+
46+
[mypy-nodescraper.models.datamodel]
47+
ignore_errors = True
48+
49+
[mypy-nodescraper.pluginexecutor]
50+
ignore_errors = True
51+
52+
[mypy-nodescraper.plugins.inband.bios.analyzer_args]
53+
ignore_errors = True
54+
55+
[mypy-nodescraper.plugins.inband.cmdline.cmdline_analyzer]
56+
ignore_errors = True
57+
58+
[mypy-nodescraper.plugins.inband.dimm.dimm_collector]
59+
ignore_errors = True
60+
61+
[mypy-nodescraper.plugins.inband.dkms.analyzer_args]
62+
ignore_errors = True
63+
64+
[mypy-nodescraper.plugins.inband.dmesg.dmesg_analyzer]
65+
ignore_errors = True
66+
67+
[mypy-nodescraper.plugins.inband.dmesg.dmesgdata]
68+
ignore_errors = True
69+
70+
[mypy-nodescraper.plugins.inband.memory.memory_collector]
71+
ignore_errors = True
72+
73+
[mypy-nodescraper.plugins.inband.os.os_collector]
74+
ignore_errors = True
75+
76+
[mypy-nodescraper.plugins.inband.package.analyzer_args]
77+
ignore_errors = True
78+
79+
[mypy-nodescraper.plugins.inband.process.analyzer_args]
80+
ignore_errors = True
81+
82+
[mypy-nodescraper.plugins.inband.process.process_collector]
83+
ignore_errors = True
84+
85+
[mypy-nodescraper.plugins.inband.rocm.rocm_plugin]
86+
ignore_errors = True
87+
88+
[mypy-nodescraper.taskresulthooks.filesystemloghook]
89+
ignore_errors = True
90+
91+
[mypy-nodescraper.typeutils]
92+
ignore_errors = True
93+
94+
[mypy-nodescraper.utils]
95+
ignore_errors = True
96+
97+
[mypy-test.unit.conftest]
98+
ignore_errors = True
99+
100+
[mypy-test.unit.framework.common.shared_utils]
101+
ignore_errors = True
102+
103+
[mypy-test.unit.framework.test_cli_helper]
104+
ignore_errors = True
105+
106+
[mypy-test.unit.framework.test_dataplugin]
107+
ignore_errors = True
108+
109+
[mypy-test.unit.framework.test_plugin_executor]
110+
ignore_errors = True
111+
112+
[mypy-nodescraper.models.collectorargs]
113+
ignore_errors = True
114+
115+
[mypy-nodescraper.plugins.inband.kernel_module.*]
116+
ignore_errors = True
117+
118+
[mypy-nodescraper.plugins.inband.nvme.nvme_collector]
119+
ignore_errors = True
120+
121+
[mypy-test.unit.framework.test_cli]
122+
ignore_errors = True

‎.pre-commit-config.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,9 @@ repos:
1515
rev: 25.1.0
1616
hooks:
1717
- id: black
18+
- repo: https://github.com/pre-commit/mirrors-mypy
19+
rev: v1.15.0
20+
hooks:
21+
- id: mypy
22+
args: [--install-types, --non-interactive, --explicit-package-bases, --allow-redefinition]
23+
language: system

CONTRIBUITING.md renamed to CONTRIBUTING.md

File renamed without changes.

‎EXTENDING.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Extending Node Scraper
2+
3+
This guide covers how to integrate Nodescraper into another Python tool and how to create and use external plugins.
4+
5+
## Table of Contents
6+
- [nodescraper integration](#nodescraper-integration)
7+
- [external plugins](#external-plugins)
8+
9+
## nodescraper integration
10+
Nodescraper can be integrated inside another Python tool by leveraging its classes and functionality.
11+
See below for a comprehensive example on how to create plugins and run the associated data
12+
collection and analysis.
13+
Sample run command:
14+
```sh
15+
python3 sample.py
16+
```
17+
18+
Sample.py file:
19+
```python
20+
import logging
21+
import sys
22+
from nodescraper.plugins.inband.bios.bios_plugin import BiosPlugin
23+
from nodescraper.plugins.inband.bios.analyzer_args import BiosAnalyzerArgs
24+
from nodescraper.plugins.inband.kernel.kernel_plugin import KernelPlugin
25+
from nodescraper.plugins.inband.kernel.analyzer_args import KernelAnalyzerArgs
26+
from nodescraper.plugins.inband.os.os_plugin import OsPlugin
27+
from nodescraper.plugins.inband.os.analyzer_args import OsAnalyzerArgs
28+
from nodescraper.models.systeminfo import SystemInfo, OSFamily
29+
from nodescraper.enums import EventPriority, SystemLocation
30+
from nodescraper.resultcollators.tablesummary import TableSummary
31+
from nodescraper.connection.inband.inbandmanager import InBandConnectionManager
32+
from nodescraper.connection.inband.sshparams import SSHConnectionParams
33+
from nodescraper.pluginregistry import PluginRegistry
34+
from nodescraper.models.pluginconfig import PluginConfig
35+
from nodescraper.pluginexecutor import PluginExecutor
36+
37+
def main():
38+
39+
#setting up my custom logger
40+
log_level = "INFO"
41+
handlers = [logging.StreamHandler(stream=sys.stdout)]
42+
logging.basicConfig(
43+
force=True,
44+
level=log_level,
45+
format="%(asctime)25s %(levelname)10s %(name)25s | %(message)s",
46+
datefmt="%Y-%m-%d %H:%M:%S %Z",
47+
handlers=handlers,
48+
encoding="utf-8",
49+
)
50+
logging.root.setLevel(logging.INFO)
51+
logging.getLogger("paramiko").setLevel(logging.ERROR)
52+
logger = logging.getLogger("nodescraper")
53+
54+
#setting up system info
55+
system_info = SystemInfo(name="test_host",
56+
platform="X",
57+
os_familty=OSFamily.LINUX,
58+
sku="some_sku")
59+
60+
#initiate plugins
61+
bios_plugin = BiosPlugin(system_info=system_info, logger=logger)
62+
kernel_plugin = KernelPlugin(system_info=system_info, logger=logger)
63+
64+
#launch data collection
65+
_ = bios_plugin.collect()
66+
_ = kernel_plugin.collect()
67+
68+
#launch data analysis
69+
bios_plugin.analyze(analysis_args=BiosAnalyzerArgs(exp_bios_version="XYZ"))
70+
kernel_plugin.analyze(analysis_args=KernelAnalyzerArgs(exp_kernel="ABC"))
71+
72+
#log plugin data models
73+
logger.info(kernel_plugin.data.model_dump())
74+
logger.info(bios_plugin.data.model_dump())
75+
76+
#alternate method
77+
all_res = []
78+
79+
#launch plugin collection & analysis
80+
bios_result = bios_plugin.run(analysis_args={"exp_bios_version":"ABC"})
81+
all_res.append(bios_result)
82+
table_summary = TableSummary()
83+
table_summary.collate_results(all_res, None)
84+
85+
#remote connection
86+
system_info.location=SystemLocation.REMOTE
87+
ssh_params = SSHConnectionParams(hostname="my_system",
88+
port=22,
89+
username="my_username",
90+
key_filename="/home/user/.ssh/ssh_key")
91+
conn_manager = InBandConnectionManager(system_info=system_info, connection_args=ssh_params)
92+
os_plugin = OsPlugin(system_info=system_info, logger=logger, connection_manager=conn_manager)
93+
os_plugin.run(analysis_args=OsAnalyzerArgs(exp_os="DEF"))
94+
95+
#run multiple plugins through a queue
96+
system_info.location=SystemLocation.LOCAL
97+
config_dict = {
98+
"global_args": {
99+
"collection" : 1,
100+
"analysis" : 1
101+
},
102+
"plugins": {
103+
"BiosPlugin": {
104+
"analysis_args": {
105+
"exp_bios_version": "123",
106+
}
107+
},
108+
"KernelPlugin": {
109+
"analysis_args": {
110+
"exp_kernel": "ABC",
111+
}
112+
}
113+
},
114+
"result_collators": {},
115+
"name": "plugin_config",
116+
"desc": "Auto generated config"
117+
}
118+
119+
config1 = PluginConfig(**config_dict)
120+
plugin_executor = PluginExecutor(
121+
logger=logger,
122+
plugin_configs=[config1],
123+
system_info=system_info
124+
)
125+
results = plugin_executor.run_queue()
126+
127+
128+
129+
if __name__ == "__main__":
130+
main()
131+
```
132+
133+
## external plugins
134+
External plugins can be added and installed in the same env as node-scraper plugins. See -> [docs/node-scraper-external/README.md](docs/node-scraper-external/README.md)

0 commit comments

Comments
 (0)