Skip to content

Commit 336937c

Browse files
ElectricalBoymbergenCopilot
authored
ci: rewrite deploy scripts with python3 (#7152)
* rewrite login_and_get_token * add deploy script * syntax * disable async deploy * rewrite with requests * memoize tokens * res deploy script to python * refactor * protect_page script to python * protect script to python * remove dev script to python * adjust main call * update gitignore * update workflows * throw away shellscript versions * disable python cache in GHA * update workflows * bump deploy bot version * kick debugging print * suppress python stdout buffering * use rglob * cleanup * lint * key safety * add python check workflow * adjust workflow * format * safer no change detect * sort paths before looping * sort resources too * adjust workflow * extract wiki deploy function * extract sleep duration to constant * use ruff for styling too * style * oops * adjust workflow trigger * code conciseness Co-authored-by: SyntacticSalt <mail@mbergen.de> * fix login in dev remove script * fix incorrect condition reading * error handling * pick up env file * handle empty lines better * better page existence check * fix indent * restore error printing * exit codes * fix condition * use read().splitlines() * append to protect_errors * Update scripts/deploy_res.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * avoid incorrect overwrite of values Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * lint * handle mw api error more gracefully Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: SyntacticSalt <mail@mbergen.de> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 4e665c2 commit 336937c

24 files changed

Lines changed: 777 additions & 643 deletions

.github/workflows/delete dev env.yml

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ jobs:
2424
with:
2525
fetch-depth: 0
2626

27+
- name: Setup Python
28+
uses: actions/setup-python@v6
29+
with:
30+
python-version: '3.14'
31+
32+
- name: Install Python Dependency
33+
run: pip install requests
34+
2735
- name: Remove Lua Dev Env Modules
2836
env:
2937
WIKI_USER: ${{ secrets.LP_BOTUSER }}
@@ -33,4 +41,5 @@ jobs:
3341
LUA_DEV_ENV_NAME: "dev/${{ github.event.inputs.luadevenv }}"
3442
INCLUDE_COMMONS: ${{ github.event.inputs.includecommons }}
3543
INCLUDE_SUB_ENVS: ${{ github.event.inputs.includesubenvs }}
36-
run: bash scripts/remove_dev.sh
44+
PYTHONUNBUFFERED: 1
45+
run: python3 ./scripts/remove_dev.py

.github/workflows/deploy test.yml

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ jobs:
2727
files: |
2828
lua/wikis/**/*.lua
2929
30+
- name: Setup Python
31+
if: steps.changed-files.outputs.any_changed == 'true'
32+
uses: actions/setup-python@v6
33+
with:
34+
python-version: '3.14'
35+
36+
- name: Install Python Dependency
37+
if: steps.changed-files.outputs.any_changed == 'true'
38+
run: pip install requests
39+
3040
- name: Personal Lua Deploy
3141
if: steps.changed-files.outputs.any_changed == 'true'
3242
env:
@@ -35,4 +45,5 @@ jobs:
3545
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
3646
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
3747
LUA_DEV_ENV_NAME: "/dev/${{ github.event.inputs.luadevenv }}"
38-
run: bash scripts/deploy.sh "${{ steps.changed-files.outputs.all_changed_files }}"
48+
PYTHONUNBUFFERED: 1
49+
run: python3 ./scripts/deploy.py ${{ steps.changed-files.outputs.all_changed_files }}

.github/workflows/deploy.yml

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,16 @@ jobs:
2929
stylesheets/**/*.scss
3030
javascript/**/*.js
3131
32+
- name: Setup Python
33+
if: steps.res-changed-files.outputs.any_changed == 'true' || steps.lua-changed-files.outputs.any_changed == 'true'
34+
uses: actions/setup-python@v6
35+
with:
36+
python-version: '3.14'
37+
38+
- name: Install Python Dependency
39+
if: steps.res-changed-files.outputs.any_changed == 'true' || steps.lua-changed-files.outputs.any_changed == 'true'
40+
run: pip install requests
41+
3242
- name: Resource Deploy
3343
if: steps.res-changed-files.outputs.any_changed == 'true'
3444
env:
@@ -37,7 +47,8 @@ jobs:
3747
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
3848
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
3949
DEPLOY_TRIGGER: ${{ github.event_name }}
40-
run: bash scripts/deploy_res.sh "${{ steps.res-changed-files.outputs.all_changed_files }}"
50+
PYTHONUNBUFFERED: 1
51+
run: python3 ./scripts/deploy_res.py ${{ steps.res-changed-files.outputs.all_changed_files }}
4152

4253
- name: Lua Deploy
4354
if: steps.lua-changed-files.outputs.any_changed == 'true'
@@ -47,7 +58,8 @@ jobs:
4758
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
4859
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
4960
DEPLOY_TRIGGER: ${{ github.event_name }}
50-
run: bash scripts/deploy.sh "${{ steps.lua-changed-files.outputs.all_changed_files }}"
61+
PYTHONUNBUFFERED: 1
62+
run: python3 ./scripts/deploy.py ${{ steps.lua-changed-files.outputs.all_changed_files }}
5163

5264
- name: Lua Protect
5365
if: steps.lua-changed-files.outputs.added_files_count != 0 || steps.lua-changed-files.outputs.renamed_files_count != 0
@@ -56,4 +68,5 @@ jobs:
5668
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
5769
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
5870
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
59-
run: bash scripts/protect.sh "${{ steps.lua-changed-files.outputs.added_files }} ${{ steps.lua-changed-files.outputs.renamed_files }}"
71+
PYTHONUNBUFFERED: 1
72+
run: python3 ./scripts/protect.py ${{ steps.lua-changed-files.outputs.added_files }} ${{ steps.lua-changed-files.outputs.renamed_files }}

.github/workflows/protect new wiki modules.yml

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,20 @@ jobs:
1616
steps:
1717
- uses: actions/checkout@v6
1818

19+
- name: Setup Python
20+
uses: actions/setup-python@v6
21+
with:
22+
python-version: '3.14'
23+
24+
- name: Install Python Dependency
25+
run: pip install requests
26+
1927
- name: Lua Protect
2028
env:
2129
WIKI_USER: ${{ secrets.LP_BOTUSER }}
2230
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
2331
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
2432
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
2533
WIKI_TO_PROTECT: ${{ github.event.inputs.wiki }}
26-
run: bash scripts/protect.sh
34+
PYTHONUNBUFFERED: 1
35+
run: python3 ./scripts/protect.py

.github/workflows/protect new wiki templates.yml

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,20 @@ jobs:
1616
steps:
1717
- uses: actions/checkout@v6
1818

19+
- name: Setup Python
20+
uses: actions/setup-python@v6
21+
with:
22+
python-version: '3.14'
23+
24+
- name: Install Python Dependency
25+
run: pip install requests
26+
1927
- name: Template Protect
2028
env:
2129
WIKI_USER: ${{ secrets.LP_BOTUSER }}
2230
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
2331
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
2432
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
2533
WIKI_TO_PROTECT: ${{ github.event.inputs.wiki }}
26-
run: bash scripts/protect_templates.sh
34+
PYTHONUNBUFFERED: 1
35+
run: python3 ./scripts/protect_templates.py

.github/workflows/pythoncheck.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: Python Code Style
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- '**.py'
7+
- '.github/workflows/*.yml'
8+
workflow_dispatch:
9+
10+
jobs:
11+
python-code-style:
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- name: Checkout
16+
uses: actions/checkout@v6
17+
- name: Run linter
18+
uses: astral-sh/ruff-action@v3
19+
with:
20+
src: scripts/
21+
- name: Check styling
22+
run: ruff format --check --diff scripts/

.github/workflows/scheduled.yml

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,23 @@ jobs:
1515
with:
1616
fetch-depth: 2
1717

18+
- name: Setup Python
19+
uses: actions/setup-python@v6
20+
with:
21+
python-version: '3.14'
22+
23+
- name: Install Python Dependency
24+
run: pip install requests
25+
1826
- name: Resource Deploy
1927
env:
2028
WIKI_USER: ${{ secrets.LP_BOTUSER }}
2129
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
2230
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
2331
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
2432
DEPLOY_TRIGGER: ${{ github.event_name }}
25-
run: bash scripts/deploy_res.sh
33+
PYTHONUNBUFFERED: 1
34+
run: python3 ./scripts/deploy_res.py
2635

2736
- name: Lua Deploy
2837
env:
@@ -31,4 +40,5 @@ jobs:
3140
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
3241
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
3342
DEPLOY_TRIGGER: ${{ github.event_name }}
34-
run: bash scripts/deploy.sh
43+
PYTHONUNBUFFERED: 1
44+
run: python3 ./scripts/deploy.py

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,6 @@
55
/package-lock.json
66
/luacov.report.out
77
/luacov.stats.out
8+
**/__pycache__/
9+
.env
10+
*.ck

.vscode/settings.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@
1212
"css.validate": false,
1313
"less.validate": false,
1414
"scss.validate": false,
15+
"python.envFile": "${workspaceFolder}/.env",
16+
"python.terminal.useEnvFile": true,
1517
"workbench.editor.customLabels.patterns": {
1618
"**/wikis/*/*.lua": "${dirname}: ${filename}",
1719
"**/wikis/*/*/*.lua": "${dirname(-4)}: ${dirname}/${filename}",

scripts/deploy.py

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
import itertools
2+
import os
3+
import pathlib
4+
import re
5+
import sys
6+
7+
from typing import Iterable
8+
9+
import requests
10+
11+
from deploy_util import (
12+
get_git_deploy_reason,
13+
deploy_file_to_wiki,
14+
read_cookie_jar,
15+
read_file_from_path,
16+
write_to_github_summary_file,
17+
)
18+
from login_and_get_token import get_token
19+
20+
HEADER_PATTERN = re.compile(
21+
r"\A---\n" r"-- @Liquipedia\n" r"-- page=(?P<pageName>[^\n]*)\n"
22+
)
23+
24+
25+
def deploy_all_files_for_wiki(
26+
wiki: str, file_paths: Iterable[pathlib.Path], deploy_reason: str
27+
) -> bool:
28+
all_modules_deployed = True
29+
token = get_token(wiki)
30+
with requests.Session() as session:
31+
session.cookies = read_cookie_jar(wiki)
32+
for file_path in file_paths:
33+
print(f"::group::Checking {str(file_path)}")
34+
file_content = read_file_from_path(file_path)
35+
header_match = HEADER_PATTERN.match(file_content)
36+
if not header_match:
37+
print("...skipping - no magic comment found")
38+
write_to_github_summary_file(f"{str(file_path)} skipped")
39+
else:
40+
page = header_match.groupdict()["pageName"] + (
41+
os.getenv("LUA_DEV_ENV_NAME") or ""
42+
)
43+
module_deployed, _ = deploy_file_to_wiki(
44+
session, file_path, file_content, wiki, page, token, deploy_reason
45+
)
46+
all_modules_deployed &= module_deployed
47+
print("::endgroup::")
48+
return all_modules_deployed
49+
50+
51+
def main():
52+
all_modules_deployed = True
53+
lua_files: Iterable[pathlib.Path]
54+
git_deploy_reason: str
55+
if len(sys.argv[1:]) == 0:
56+
lua_files = pathlib.Path("./lua/wikis/").rglob("*.lua")
57+
git_deploy_reason = "Automated Weekly Re-Sync"
58+
else:
59+
lua_files = [pathlib.Path(arg) for arg in sys.argv[1:]]
60+
git_deploy_reason = get_git_deploy_reason()
61+
62+
for wiki, files in itertools.groupby(sorted(lua_files), lambda path: path.parts[2]):
63+
all_modules_deployed &= deploy_all_files_for_wiki(
64+
wiki, list(files), git_deploy_reason
65+
)
66+
67+
if not all_modules_deployed:
68+
print("::warning::Some modules were not deployed!")
69+
sys.exit(1)
70+
71+
72+
if __name__ == "__main__":
73+
main()

0 commit comments

Comments
 (0)