improvement(seo): restore explicit AI/search bot allow-list and add l… #16206

Job	Run time
Detect Version	4s
Test and Build / Test and Build	4m 16s
Check Docs Changes	-1s
Build Dev ECR	0s
Build ARM64 (GHCR Only)	0s
Migrate Dev DB	0s
Build AMD64 (./docker/app.Dockerfile, ghcr.io/simstudioai/simstudio, ECR_APP)	3m 50s
Build AMD64 (./docker/db.Dockerfile, ghcr.io/simstudioai/migrations, ECR_MIGRATIONS)	49s
Build AMD64 (./docker/realtime.Dockerfile, ghcr.io/simstudioai/realtime, ECR_REALTIME)	48s
Process Docs	0s
Create GHCR Manifests	0s
Create GitHub Release	0s
	9m 46s

Provide feedback