Skip to content

Commit 49912d4

Browse files
test: add dynamic-stack CMA SDK integration (sanity) suite
Add a self-contained pytest integration suite under tests/integration that creates a fresh stack per run, exercises every SDK resource method (positive, negative, and edge cases) against the live CMA API, and tears the stack down. - framework/: dynamic stack setup/teardown, request+cURL capture, response/error validators, tracked assertions, and a custom dashboard HTML report - api/: 30 resource files with full method coverage - data/: complex content-type schemas (modular blocks, groups, references, JSON RTE) and entry payloads - strict, bug-catching assertions; genuine SDK/environment issues tracked via xfail - timestamped HTML report + cURL log written to repo root (gitignored) Also: update AGENTS.md to document the sanity suite + env vars, add pytest/pytest-order to requirements, gitignore secrets/reports/docs.
1 parent 35372ad commit 49912d4

46 files changed

Lines changed: 4771 additions & 10 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,9 +133,17 @@ tests/config/default.yml
133133
.vscode/settings.json
134134
run.py
135135
tests/resources/.DS_Store
136-
.talismanrc
137136
tests/.DS_Store
138-
tests/resources/.DS_Store
139137
.DS_Store
140138
*/data/regions.json
141-
.talismanrc
139+
# Local backup of legacy tests (do not commit)
140+
141+
# --- CMA integration suite: do not track ---
142+
docs/
143+
tests_backup_legacy/
144+
tests/integration/report/
145+
tests/integration/.env
146+
tests/integration/.env.example
147+
# Timestamped reports written at repo root
148+
cma-python-report-*.html
149+
api-requests-*.txt

AGENTS.md

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
| Language | Python ≥ 3.9 (`setup.py` `python_requires`) |
1717
| Build | `setuptools` / `setup.py`; package `contentstack_management` |
1818
| HTTP | `requests`, `requests-toolbelt`, `urllib3` |
19-
| Tests | `pytest``tests/unit`, `tests/api`, `tests/mock` |
19+
| Tests | `pytest``tests/integration` (live e2e / sanity, dynamic stack), `tests/unit`, `tests/mock`, `tests/api` (legacy, superseded by `tests/integration`) |
2020
| Lint / coverage | `pylint`, `coverage` (see `requirements.txt`) |
2121
| Secrets / hooks | Talisman, Snyk (see `README.md` development setup) |
2222

@@ -29,24 +29,42 @@
2929
| `contentstack_management/stack/stack.py` | **Stack**-scoped CMA operations |
3030
| `contentstack_management/*/` | Domain modules (entries, assets, webhooks, taxonomies, …) |
3131
| `contentstack_management/__init__.py` | Public exports |
32-
| `tests/cred.py` | **`get_credentials()`****dotenv** + env vars for API/mock tests |
32+
| `tests/integration/` | **Live e2e / sanity suite** (pytest). Self-contained: creates a fresh stack per run, exercises every SDK method (positive/negative/edge), tears it down. Own `framework/` + `data/`; config in `tests/integration/.env`. |
33+
| `tests/cred.py` | **`get_credentials()`****dotenv** + env vars for the legacy `tests/api` / `tests/mock` suites |
3334

3435
## Commands (quick reference)
3536

3637
| Command Type | Command |
3738
|---|---|
3839
| Install | `pip install -e ".[dev]"` |
40+
| **Sanity / e2e (live)** | `pytest tests/integration` — dynamically creates a stack, runs the full suite, tears it down. Needs `tests/integration/.env` (`EMAIL`, `PASSWORD`, `HOST`, `ORGANIZATION`). Writes a timestamped HTML report + cURL log to the repo root. |
41+
| Sanity, keep stack | `DELETE_DYNAMIC_RESOURCES=false pytest tests/integration` (preserve the created stack for debugging) |
42+
| Sanity, one resource | `pytest tests/integration/api/test_12_content_type.py` |
3943
| Test (unit) | `pytest tests/unit/ -v` |
40-
| Test (API, live) | `pytest tests/api/ -v` (needs `.env` — see `tests/cred.py`) |
4144
| Test (mock) | `pytest tests/mock/ -v` |
42-
| Coverage | `coverage run -m pytest tests/unit/` |
45+
| Test (legacy API, live) | `pytest tests/api/ -v` (needs `.env` — see `tests/cred.py`) |
46+
| Coverage (CI) | `coverage run -m pytest tests/unit/` |
4347
| Lint | `pylint contentstack_management/` |
4448

45-
## Environment variables (API / integration tests)
49+
> **CI note:** `.github/workflows/unit-test.yml` runs **only `tests/unit/`** (no credentials). The `tests/integration` sanity suite is run manually (or via a credential-gated job) because it provisions real stacks.
4650
47-
Loaded via **`tests/cred.py`** (`load_dotenv()`). Examples include **`HOST`**, **`APIKEY`**, **`AUTHTOKEN`**, **`MANAGEMENT_TOKEN`**, **`ORG_UID`**, and resource UIDs (**`CONTENT_TYPE_UID`**, **`ENTRY_UID`**, …). See that file for the full list.
51+
## Environment variables
4852

49-
Do not commit secrets.
53+
**Sanity / e2e suite** (`tests/integration`) — configured via **`tests/integration/.env`** (gitignored). No pre-existing stack/UIDs needed; the suite creates everything at runtime.
54+
55+
| Var | Required | Purpose |
56+
|-----|----------|---------|
57+
| `EMAIL`, `PASSWORD` || Login for the run (a **non-2FA** account) |
58+
| `HOST` || API host (e.g. `api.contentstack.io`) |
59+
| `ORGANIZATION` || Org the dynamic test stack is created in |
60+
| `MFA_SECRET` || TOTP secret (for the OAuth/2FA account, not the primary login) |
61+
| `DELETE_DYNAMIC_RESOURCES` || `false` keeps the created stack for debugging (default deletes) |
62+
| `CLIENT_ID`, `APP_ID`, `REDIRECT_URI` || OAuth tests |
63+
| `PERSONALIZE_HOST` || Personalize project for variant tests |
64+
65+
**Legacy `tests/api` / `tests/mock`** — loaded via **`tests/cred.py`** (`load_dotenv()`): `HOST`, `APIKEY`, `AUTHTOKEN`, `MANAGEMENT_TOKEN`, `ORG_UID`, and resource UIDs. See that file for the full list.
66+
67+
Do not commit secrets. `tests/integration/.env`, `docs/`, and the repo-root `cma-python-report-*.html` / `api-requests-*.txt` are gitignored.
5068

5169
## Where the documentation lives: skills
5270

requirements.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,6 @@ pylint>=2.0.0
77
requests-toolbelt>=1.0.0,<2.0.0
88
pyotp==2.9.0
99
packaging>=24.0
10+
# Integration test suite
11+
pytest>=7.0
12+
pytest-order>=1.2.0
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
"""User API tests — profile fetch and update."""
2+
3+
import pytest
4+
5+
from framework import helpers as h
6+
7+
pytestmark = pytest.mark.order(1)
8+
9+
10+
class TestUser:
11+
def test_fetch(self, ctx):
12+
resp = ctx.client.user().fetch()
13+
h.assert_status(resp, 200)
14+
user = h.body(resp).get("user", {})
15+
h.tracked_assert(user.get("uid"), "user uid").truthy()
16+
17+
def test_update_noop(self, ctx):
18+
# Send the current first_name back — a harmless update that exercises PUT /user.
19+
current = h.body(ctx.client.user().fetch()).get("user", {})
20+
payload = {"user": {"first_name": current.get("first_name", "Test")}}
21+
resp = ctx.client.user().update(payload)
22+
h.assert_status(resp, 200, 201)
23+
24+
25+
class TestUserAuthOps:
26+
"""Account auth endpoints exercised safely (bogus tokens / non-real email)."""
27+
28+
def test_activate_bogus_token(self, ctx):
29+
resp = ctx.client.user().activate("bogus_activation_token", {"user": {"password": "Test@12345"}})
30+
h.assert_status(resp, 400, 404, 422)
31+
32+
def test_reset_password_bogus_token(self, ctx):
33+
resp = ctx.client.user().reset_password(
34+
{"user": {"reset_password_token": "bogus", "password": "Test@12345", "password_confirmation": "Test@12345"}}
35+
)
36+
h.assert_status(resp, 400, 404, 422)
37+
38+
def test_forgot_password(self, ctx):
39+
# Triggers a reset email to a non-real address; APIs typically return 200
40+
# regardless (to avoid email enumeration) or a 422.
41+
resp = ctx.client.user().forgot_password({"user": {"email": "noreply+test@example.com"}})
42+
h.assert_status(resp, 200, 201, 422, 429)
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
"""Organization API tests — fetch, roles, stacks, logs, negative cases."""
2+
3+
import pytest
4+
5+
from framework import helpers as h
6+
7+
pytestmark = pytest.mark.order(2)
8+
9+
10+
class TestOrganization:
11+
def test_find_all(self, ctx):
12+
resp = ctx.client.organizations().find()
13+
h.assert_status(resp, 200)
14+
h.tracked_assert(h.body(resp).get("organizations"), "orgs list").is_type(list)
15+
16+
def test_fetch(self, ctx):
17+
resp = ctx.client.organizations(ctx.organization_uid).fetch()
18+
h.assert_status(resp, 200)
19+
org = h.body(resp).get("organization", {})
20+
h.tracked_assert(org.get("uid"), "org uid").equals(ctx.organization_uid)
21+
22+
def test_roles(self, ctx):
23+
resp = ctx.client.organizations(ctx.organization_uid).roles()
24+
h.assert_status(resp, 200)
25+
26+
def test_stacks(self, ctx):
27+
resp = ctx.client.organizations(ctx.organization_uid).stacks()
28+
h.assert_status(resp, 200)
29+
30+
def test_logs(self, ctx):
31+
resp = ctx.client.organizations(ctx.organization_uid).logs()
32+
h.assert_status(resp, 200)
33+
34+
35+
class TestOrganizationOwnership:
36+
"""Exercised safely with invalid data so no real invite/transfer occurs."""
37+
38+
def test_add_users_invalid(self, ctx):
39+
resp = ctx.client.organizations(ctx.organization_uid).add_users({"share": {}})
40+
h.assert_status(resp, 400, 403, 422)
41+
42+
def test_transfer_ownership_invalid(self, ctx):
43+
resp = ctx.client.organizations(ctx.organization_uid).transfer_ownership({"transfer_to": "not-an-email"})
44+
h.assert_status(resp, 400, 403, 422)
45+
46+
47+
class TestOrganizationNegative:
48+
def test_fetch_nonexistent(self, ctx):
49+
resp = ctx.client.organizations("org_does_not_exist").fetch()
50+
h.assert_status(resp, 404, 422, 403)
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
"""Stack API tests — fetch, settings, users, share/unshare."""
2+
3+
import os
4+
5+
import pytest
6+
7+
from framework import helpers as h
8+
9+
pytestmark = pytest.mark.order(3)
10+
11+
12+
class TestStack:
13+
def test_fetch(self, ctx):
14+
resp = ctx.client.stack(ctx.stack_api_key).fetch()
15+
h.assert_status(resp, 200)
16+
stack = h.body(resp).get("stack", {})
17+
h.tracked_assert(stack.get("api_key"), "api_key").equals(ctx.stack_api_key)
18+
19+
def test_settings(self, ctx):
20+
resp = ctx.client.stack(ctx.stack_api_key).settings()
21+
h.assert_status(resp, 200)
22+
23+
def test_create_settings(self, ctx):
24+
data = {
25+
"stack_settings": {
26+
"stack_variables": {"enforce_unique_urls": "true"},
27+
}
28+
}
29+
resp = ctx.client.stack(ctx.stack_api_key).create_settings(data)
30+
h.assert_status(resp, 200, 201)
31+
32+
def test_users(self, ctx):
33+
resp = ctx.client.stack(ctx.stack_api_key).users()
34+
h.assert_status(resp, 200)
35+
36+
def test_update(self, ctx):
37+
data = {"stack": {"description": "updated by integration suite"}}
38+
resp = ctx.client.stack(ctx.stack_api_key).update(data)
39+
h.assert_status(resp, 200, 201)
40+
41+
def test_reset_settings(self, ctx):
42+
resp = ctx.client.stack(ctx.stack_api_key).reset_settings({"stack_settings": {}})
43+
h.assert_status(resp, 200, 201)
44+
45+
46+
class TestStackOwnership:
47+
"""Ownership/role operations exercised safely (no real transfer occurs)."""
48+
49+
def test_update_user_role(self, ctx):
50+
# Map the current user to a role; on a fresh single-user stack this may
51+
# be accepted (200) or rejected (422) — both confirm the SDK call works.
52+
roles = h.body(ctx.client.stack(ctx.stack_api_key).roles().find()).get("roles", [])
53+
role_uid = next((r["uid"] for r in roles), None)
54+
if not (role_uid and ctx.user_uid):
55+
pytest.skip("no role/user available")
56+
resp = ctx.client.stack(ctx.stack_api_key).update_user_role({"users": {ctx.user_uid: [role_uid]}})
57+
# 404 when the user isn't a separately-added stack member (owner self-assign).
58+
h.assert_status(resp, 200, 201, 404, 422)
59+
60+
def test_transfer_ownership_invalid(self, ctx):
61+
# Transferring to an invalid address must fail — exercises the endpoint
62+
# without actually handing the stack to anyone.
63+
resp = ctx.client.stack(ctx.stack_api_key).transfer_ownership({"transfer_to": "not-an-email"})
64+
h.assert_status(resp, 400, 422)
65+
66+
def test_accept_ownership_bogus_token(self, ctx):
67+
# Accepting with a bogus token must fail.
68+
resp = ctx.client.stack(ctx.stack_api_key).accept_ownership(ctx.user_uid or "uid", "bogus_token")
69+
h.assert_status(resp, 400, 404, 422)
70+
71+
72+
class TestStackSharing:
73+
def test_share(self, ctx):
74+
member = os.getenv("MEMBER_EMAIL")
75+
if not member:
76+
pytest.skip("MEMBER_EMAIL not set")
77+
# Sharing requires a valid role mapping per email — an empty roles object
78+
# is rejected with 422 "roles is not valid".
79+
roles = h.body(ctx.client.stack(ctx.stack_api_key).roles().find()).get("roles", [])
80+
role_uid = next((r["uid"] for r in roles if r.get("name") != "Admin"),
81+
roles[0]["uid"] if roles else None)
82+
if not role_uid:
83+
pytest.skip("no role available to share with")
84+
data = {"emails": [member], "roles": {member: [role_uid]}}
85+
resp = ctx.client.stack(ctx.stack_api_key).share(data)
86+
h.assert_status(resp, 200, 201)
87+
88+
def test_unshare(self, ctx):
89+
member = os.getenv("MEMBER_EMAIL")
90+
if not member:
91+
pytest.skip("MEMBER_EMAIL not set")
92+
resp = ctx.client.stack(ctx.stack_api_key).unshare({"email": member})
93+
h.assert_status(resp, 200, 201)
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
"""Locale API tests — CRUD, fallback, negative cases."""
2+
3+
import pytest
4+
5+
from framework import helpers as h
6+
7+
pytestmark = pytest.mark.order(4)
8+
9+
# A non-master locale code to create/manipulate.
10+
_CODE = "fr-fr"
11+
12+
13+
class TestLocaleCRUD:
14+
def test_create(self, stack, store):
15+
data = {"locale": {"name": "French", "code": _CODE}}
16+
resp = stack.locale().create(data)
17+
h.assert_status(resp, 201)
18+
store["locales"]["custom"] = _CODE
19+
h.wait(h.SHORT_DELAY)
20+
21+
def test_find_all(self, stack):
22+
resp = stack.locale().find()
23+
h.assert_status(resp, 200)
24+
h.tracked_assert(h.body(resp).get("locales"), "locales list").is_type(list)
25+
26+
def test_fetch(self, stack):
27+
resp = stack.locale(_CODE).fetch()
28+
h.assert_status(resp, 200)
29+
h.validate_locale_response(resp)
30+
31+
def test_update(self, stack):
32+
data = {"locale": {"name": "French (FR)"}}
33+
resp = stack.locale(_CODE).update(data)
34+
h.assert_status(resp, 200, 201)
35+
36+
def test_set_fallback(self, stack):
37+
data = {"locale": {"name": "German", "code": "de-de", "fallback_locale": "en-us"}}
38+
resp = stack.locale().set_fallback(data)
39+
h.assert_status(resp, 200, 201)
40+
41+
def test_update_fallback(self, stack):
42+
# Ensure de-de exists, then update its fallback configuration.
43+
stack.locale().create({"locale": {"name": "German", "code": "de-de"}})
44+
h.wait(h.SHORT_DELAY)
45+
data = {"locale": {"name": "German", "code": "de-de", "fallback_locale": "en-us"}}
46+
resp = stack.locale("de-de").update_fallback(data)
47+
h.assert_status(resp, 200, 201)
48+
49+
50+
class TestLocaleNegative:
51+
def test_fetch_nonexistent(self, stack):
52+
resp = stack.locale("zz-zz").fetch()
53+
h.assert_status(resp, 404, 422)
54+
55+
def test_fetch_without_code_raises(self, stack):
56+
# Locale guards on a missing locale_code before the HTTP call.
57+
with pytest.raises(Exception):
58+
stack.locale().fetch()
59+
60+
61+
class TestLocaleDelete:
62+
def test_delete(self, stack):
63+
resp = stack.locale("de-de").delete()
64+
h.assert_status(resp, 200)
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
"""Environment API tests — CRUD, negative cases."""
2+
3+
import pytest
4+
5+
from framework import helpers as h
6+
7+
pytestmark = pytest.mark.order(5)
8+
9+
10+
class TestEnvironmentCRUD:
11+
def test_create(self, stack, store):
12+
name = h.generate_valid_uid("env")
13+
data = {"environment": {"name": name, "urls": [{"url": "https://example.com", "locale": "en-us"}]}}
14+
resp = stack.environments().create(data)
15+
h.assert_status(resp, 201)
16+
store["environments"]["main"] = name
17+
h.wait(h.SHORT_DELAY)
18+
19+
def test_find_all(self, stack):
20+
resp = stack.environments().find()
21+
h.assert_status(resp, 200)
22+
h.tracked_assert(h.body(resp).get("environments"), "env list").is_type(list)
23+
24+
def test_fetch(self, stack, store):
25+
name = store["environments"]["main"]
26+
resp = stack.environments(name).fetch()
27+
h.assert_status(resp, 200)
28+
h.validate_environment_response(resp)
29+
30+
def test_update(self, stack, store):
31+
name = store["environments"]["main"]
32+
data = {"environment": {"name": name, "urls": [{"url": "https://updated.example.com", "locale": "en-us"}]}}
33+
resp = stack.environments(name).update(data)
34+
h.assert_status(resp, 200, 201)
35+
36+
37+
class TestEnvironmentNegative:
38+
def test_fetch_nonexistent(self, stack):
39+
resp = stack.environments("does_not_exist_env").fetch()
40+
h.assert_status(resp, 404, 422)
41+
42+
def test_fetch_without_name_raises(self, stack):
43+
with pytest.raises(Exception):
44+
stack.environments().fetch()
45+
46+
47+
class TestEnvironmentDelete:
48+
def test_delete(self, stack):
49+
name = h.generate_valid_uid("env_del")
50+
stack.environments().create(
51+
{"environment": {"name": name, "urls": [{"url": "https://d.example.com", "locale": "en-us"}]}}
52+
)
53+
h.wait(h.SHORT_DELAY)
54+
resp = stack.environments(name).delete()
55+
h.assert_status(resp, 200)

0 commit comments

Comments
 (0)