Skip to content

Commit f888340

Browse files
Merge pull request #67 from microsoft/dev
fix: Merging dev changes to main branch
2 parents deb0a2c + 301fab7 commit f888340

15 files changed

Lines changed: 371 additions & 40 deletions

File tree

docs/LocalDevelopmentSetup.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh
6262
source ~/.bashrc
6363

6464
# Clone and setup
65-
git clone https://github.com/microsoft/Container-Migration-Solution-Acceleratorr.git
65+
git clone https://github.com/microsoft/Container-Migration-Solution-Accelerator.git
6666
cd container-migration-solution-accelerator/src/processor
6767
uv venv .venv
6868
source .venv/bin/activate

docs/QuotaCheck.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ o3:500
1919
### 📌 Default Regions:
2020

2121
```
22-
eastus, uksouth, eastus2, northcentralus, swedencentral, westus, westus2, southcentralus, canadacentral
22+
eastus, uksouth, eastus2, northcentralus, swedencentral, westus, westus2, southcentralus, canadacentral, australiaeast, japaneast
2323
```
2424

2525
### Usage Scenarios:
@@ -37,37 +37,37 @@ eastus, uksouth, eastus2, northcentralus, swedencentral, westus, westus2, southc
3737
✔️ Run without parameters to check default models & regions without verbose logging:
3838

3939
```
40-
./quota_check.sh
40+
./quota_check_params.sh
4141
```
4242

4343
✔️ Enable verbose logging:
4444

4545
```
46-
./quota_check.sh --verbose
46+
./quota_check_params.sh --verbose
4747
```
4848

4949
✔️ Check specific model(s) in default regions:
5050

5151
```
52-
./quota_check.sh --models o3:500
52+
./quota_check_params.sh --models o3:500
5353
```
5454

5555
✔️ Check default models in specific region(s):
5656

5757
```
58-
./quota_check.sh --regions eastus,westus
58+
./quota_check_params.sh --regions eastus,westus
5959
```
6060

6161
✔️ Passing Both models and regions:
6262

6363
```
64-
./quota_check.sh --models o3:500 --regions eastus,westus2
64+
./quota_check_params.sh --models o3:500 --regions eastus,westus2
6565
```
6666

6767
✔️ All parameters combined:
6868

6969
```
70-
./quota_check.sh --models o3:500 --regions eastus,westus --verbose
70+
./quota_check_params.sh --models o3:500 --regions eastus,westus --verbose
7171
```
7272

7373
### **Sample Output**
@@ -85,9 +85,9 @@ The final table lists regions with available quota. You can select any of these
8585
**To check quota for the deployment**
8686

8787
```sh
88-
curl -L -o quota_check.sh "https://raw.githubusercontent.com/microsoft/Container-Migration-Solution-Accelerator/main/scripts/quota_check.sh"
89-
chmod +x quota_check.sh
90-
./quota_check.sh
88+
curl -L -o quota_check_params.sh "https://raw.githubusercontent.com/microsoft/Container-Migration-Solution-Accelerator/main/scripts/quota_check_params.sh"
89+
chmod +x quota_check_params.sh
90+
./quota_check_params.sh
9191
```
9292

9393
- Refer to [Input Formats](#input-formats) for detailed commands.
@@ -100,14 +100,14 @@ The final table lists regions with available quota. You can select any of these
100100
3. Navigate to the `scripts` folder where the script files are located and make the script as executable:
101101
```sh
102102
cd scripts
103-
chmod +x quota_check.sh
103+
chmod +x quota_check_params.sh
104104
```
105105
4. Run the appropriate script based on your requirement:
106106

107107
**To check quota for the deployment**
108108

109109
```sh
110-
./quota_check.sh
110+
./quota_check_params.sh
111111
```
112112

113113
- Refer to [Input Formats](#input-formats) for detailed commands.

infra/main.bicep

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -856,6 +856,14 @@ module appConfiguration 'br/public:avm/res/app-configuration/configuration-store
856856
name: 'APP_LOGGING_LEVEL'
857857
value: 'INFO'
858858
}
859+
{
860+
name: 'AZURE_PACKAGE_LOGGING_LEVEL'
861+
value: 'INFO'
862+
}
863+
{
864+
name: 'AZURE_LOGGING_PACKAGES'
865+
value: ''
866+
}
859867
{
860868
name: 'AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME'
861869
value: ''

infra/main_custom.bicep

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -859,6 +859,14 @@ module appConfiguration 'br/public:avm/res/app-configuration/configuration-store
859859
name: 'APP_LOGGING_LEVEL'
860860
value: 'INFO'
861861
}
862+
{
863+
name: 'AZURE_PACKAGE_LOGGING_LEVEL'
864+
value: 'INFO'
865+
}
866+
{
867+
name: 'AZURE_LOGGING_PACKAGES'
868+
value: ''
869+
}
862870
{
863871
name: 'AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME'
864872
value: ''

scripts/quota_check_params.sh

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
#!/bin/bash
2+
# VERBOSE=false
3+
4+
MODELS=""
5+
REGIONS=""
6+
VERBOSE=false
7+
8+
while [[ $# -gt 0 ]]; do
9+
case "$1" in
10+
--models)
11+
MODELS="$2"
12+
shift 2
13+
;;
14+
--regions)
15+
REGIONS="$2"
16+
shift 2
17+
;;
18+
--verbose)
19+
VERBOSE=true
20+
shift
21+
;;
22+
*)
23+
echo "Unknown option: $1"
24+
exit 1
25+
;;
26+
esac
27+
done
28+
29+
# Fallback to defaults if not provided
30+
[[ -z "$MODELS" ]]
31+
[[ -z "$REGIONS" ]]
32+
33+
echo "Models: $MODELS"
34+
echo "Regions: $REGIONS"
35+
echo "Verbose: $VERBOSE"
36+
37+
for arg in "$@"; do
38+
if [ "$arg" = "--verbose" ]; then
39+
VERBOSE=true
40+
fi
41+
done
42+
43+
log_verbose() {
44+
if [ "$VERBOSE" = true ]; then
45+
echo "$1"
46+
fi
47+
}
48+
49+
# Default Models and Capacities (Comma-separated in "model:capacity" format)
50+
DEFAULT_MODEL_CAPACITY="o3:500"
51+
# Convert the comma-separated string into an array
52+
IFS=',' read -r -a MODEL_CAPACITY_PAIRS <<< "$DEFAULT_MODEL_CAPACITY"
53+
54+
echo "🔄 Fetching available Azure subscriptions..."
55+
SUBSCRIPTIONS=$(az account list --query "[?state=='Enabled'].{Name:name, ID:id}" --output tsv)
56+
SUB_COUNT=$(echo "$SUBSCRIPTIONS" | wc -l)
57+
58+
if [ "$SUB_COUNT" -eq 0 ]; then
59+
echo "❌ ERROR: No active Azure subscriptions found. Please log in using 'az login' and ensure you have an active subscription."
60+
exit 1
61+
elif [ "$SUB_COUNT" -eq 1 ]; then
62+
# If only one subscription, automatically select it
63+
AZURE_SUBSCRIPTION_ID=$(echo "$SUBSCRIPTIONS" | awk '{print $2}')
64+
if [ -z "$AZURE_SUBSCRIPTION_ID" ]; then
65+
echo "❌ ERROR: No active Azure subscriptions found. Please log in using 'az login' and ensure you have an active subscription."
66+
exit 1
67+
fi
68+
echo "✅ Using the only available subscription: $AZURE_SUBSCRIPTION_ID"
69+
else
70+
# If multiple subscriptions exist, prompt the user to choose one
71+
echo "Multiple subscriptions found:"
72+
echo "$SUBSCRIPTIONS" | awk '{print NR")", $1, "-", $2}'
73+
74+
while true; do
75+
echo "Enter the number of the subscription to use:"
76+
read SUB_INDEX
77+
78+
# Validate user input
79+
if [[ "$SUB_INDEX" =~ ^[0-9]+$ ]] && [ "$SUB_INDEX" -ge 1 ] && [ "$SUB_INDEX" -le "$SUB_COUNT" ]; then
80+
AZURE_SUBSCRIPTION_ID=$(echo "$SUBSCRIPTIONS" | awk -v idx="$SUB_INDEX" 'NR==idx {print $2}')
81+
echo "✅ Selected Subscription: $AZURE_SUBSCRIPTION_ID"
82+
break
83+
else
84+
echo "❌ Invalid selection. Please enter a valid number from the list."
85+
fi
86+
done
87+
fi
88+
89+
90+
# Set the selected subscription
91+
az account set --subscription "$AZURE_SUBSCRIPTION_ID"
92+
echo "🎯 Active Subscription: $(az account show --query '[name, id]' --output tsv)"
93+
94+
# Default Regions to check (Comma-separated, now configurable)
95+
DEFAULT_REGIONS="australiaeast,japaneast,eastus,uksouth,eastus2,northcentralus,swedencentral,westus,westus2,southcentralus,canadacentral"
96+
IFS=',' read -r -a DEFAULT_REGION_ARRAY <<< "$DEFAULT_REGIONS"
97+
98+
# Read parameters (if any)
99+
IFS=',' read -r -a USER_PROVIDED_PAIRS <<< "$MODELS"
100+
USER_REGION="$REGIONS"
101+
102+
IS_USER_PROVIDED_PAIRS=false
103+
104+
if [ ${#USER_PROVIDED_PAIRS[@]} -lt 1 ]; then
105+
echo "No parameters provided, using default model-capacity pairs: ${MODEL_CAPACITY_PAIRS[*]}"
106+
else
107+
echo "Using provided model and capacity pairs: ${USER_PROVIDED_PAIRS[*]}"
108+
IS_USER_PROVIDED_PAIRS=true
109+
MODEL_CAPACITY_PAIRS=("${USER_PROVIDED_PAIRS[@]}")
110+
fi
111+
112+
declare -a FINAL_MODEL_NAMES
113+
declare -a FINAL_CAPACITIES
114+
declare -a TABLE_ROWS
115+
116+
for PAIR in "${MODEL_CAPACITY_PAIRS[@]}"; do
117+
MODEL_NAME=$(echo "$PAIR" | cut -d':' -f1 | tr '[:upper:]' '[:lower:]')
118+
CAPACITY=$(echo "$PAIR" | cut -d':' -f2)
119+
120+
if [ -z "$MODEL_NAME" ] || [ -z "$CAPACITY" ]; then
121+
echo "❌ ERROR: Invalid model and capacity pair '$PAIR'. Both model and capacity must be specified."
122+
exit 1
123+
fi
124+
125+
FINAL_MODEL_NAMES+=("$MODEL_NAME")
126+
FINAL_CAPACITIES+=("$CAPACITY")
127+
128+
done
129+
130+
echo "🔄 Using Models: ${FINAL_MODEL_NAMES[*]} with respective Capacities: ${FINAL_CAPACITIES[*]}"
131+
echo "----------------------------------------"
132+
133+
# Check if the user provided a region, if not, use the default regions
134+
if [ -n "$USER_REGION" ]; then
135+
echo "🔍 User provided region: $USER_REGION"
136+
IFS=',' read -r -a REGIONS <<< "$USER_REGION"
137+
else
138+
echo "No region specified, using default regions: ${DEFAULT_REGION_ARRAY[*]}"
139+
REGIONS=("${DEFAULT_REGION_ARRAY[@]}")
140+
APPLY_OR_CONDITION=true
141+
fi
142+
143+
echo "✅ Retrieved Azure regions. Checking availability..."
144+
INDEX=1
145+
146+
VALID_REGIONS=()
147+
for REGION in "${REGIONS[@]}"; do
148+
log_verbose "----------------------------------------"
149+
log_verbose "🔍 Checking region: $REGION"
150+
151+
QUOTA_INFO=$(az cognitiveservices usage list --location "$REGION" --output json | tr '[:upper:]' '[:lower:]')
152+
if [ -z "$QUOTA_INFO" ]; then
153+
log_verbose "⚠️ WARNING: Failed to retrieve quota for region $REGION. Skipping."
154+
continue
155+
fi
156+
157+
TEXT_EMBEDDING_AVAILABLE=false
158+
AT_LEAST_ONE_MODEL_AVAILABLE=false
159+
TEMP_TABLE_ROWS=()
160+
161+
for index in "${!FINAL_MODEL_NAMES[@]}"; do
162+
MODEL_NAME="${FINAL_MODEL_NAMES[$index]}"
163+
REQUIRED_CAPACITY="${FINAL_CAPACITIES[$index]}"
164+
FOUND=false
165+
INSUFFICIENT_QUOTA=false
166+
167+
MODEL_TYPES=("openai.standard.$MODEL_NAME" "openai.globalstandard.$MODEL_NAME")
168+
169+
for MODEL_TYPE in "${MODEL_TYPES[@]}"; do
170+
FOUND=false
171+
INSUFFICIENT_QUOTA=false
172+
log_verbose "🔍 Checking model: $MODEL_NAME with required capacity: $REQUIRED_CAPACITY ($MODEL_TYPE)"
173+
174+
MODEL_INFO=$(echo "$QUOTA_INFO" | awk -v model="\"value\": \"$MODEL_TYPE\"" '
175+
BEGIN { RS="},"; FS="," }
176+
$0 ~ model { print $0 }
177+
')
178+
179+
if [ -z "$MODEL_INFO" ]; then
180+
FOUND=false
181+
log_verbose "⚠️ WARNING: No quota information found for model: $MODEL_NAME in region: $REGION for model type: $MODEL_TYPE."
182+
continue
183+
fi
184+
185+
if [ -n "$MODEL_INFO" ]; then
186+
FOUND=true
187+
CURRENT_VALUE=$(echo "$MODEL_INFO" | awk -F': ' '/"currentvalue"/ {print $2}' | tr -d ',' | tr -d ' ')
188+
LIMIT=$(echo "$MODEL_INFO" | awk -F': ' '/"limit"/ {print $2}' | tr -d ',' | tr -d ' ')
189+
190+
CURRENT_VALUE=${CURRENT_VALUE:-0}
191+
LIMIT=${LIMIT:-0}
192+
193+
CURRENT_VALUE=$(echo "$CURRENT_VALUE" | cut -d'.' -f1)
194+
LIMIT=$(echo "$LIMIT" | cut -d'.' -f1)
195+
196+
AVAILABLE=$((LIMIT - CURRENT_VALUE))
197+
log_verbose "✅ Model: $MODEL_TYPE | Used: $CURRENT_VALUE | Limit: $LIMIT | Available: $AVAILABLE"
198+
199+
if [ "$AVAILABLE" -ge "$REQUIRED_CAPACITY" ]; then
200+
FOUND=true
201+
if [ "$MODEL_NAME" = "text-embedding-ada-002" ]; then
202+
TEXT_EMBEDDING_AVAILABLE=true
203+
fi
204+
AT_LEAST_ONE_MODEL_AVAILABLE=true
205+
TEMP_TABLE_ROWS+=("$(printf "| %-4s | %-20s | %-43s | %-10s | %-10s | %-10s |" "$INDEX" "$REGION" "$MODEL_TYPE" "$LIMIT" "$CURRENT_VALUE" "$AVAILABLE")")
206+
else
207+
INSUFFICIENT_QUOTA=true
208+
fi
209+
fi
210+
211+
if [ "$FOUND" = false ]; then
212+
log_verbose "❌ No models found for model: $MODEL_NAME in region: $REGION (${MODEL_TYPES[*]})"
213+
214+
elif [ "$INSUFFICIENT_QUOTA" = true ]; then
215+
log_verbose "⚠️ Model $MODEL_NAME in region: $REGION has insufficient quota (${MODEL_TYPES[*]})."
216+
fi
217+
done
218+
done
219+
220+
if { [ "$IS_USER_PROVIDED_PAIRS" = true ] && [ "$INSUFFICIENT_QUOTA" = false ] && [ "$FOUND" = true ]; } || { [ "$APPLY_OR_CONDITION" != true ] || [ "$AT_LEAST_ONE_MODEL_AVAILABLE" = true ]; }; then
221+
VALID_REGIONS+=("$REGION")
222+
TABLE_ROWS+=("${TEMP_TABLE_ROWS[@]}")
223+
INDEX=$((INDEX + 1))
224+
elif [ ${#USER_PROVIDED_PAIRS[@]} -eq 0 ]; then
225+
echo "🚫 Skipping $REGION as it does not meet quota requirements."
226+
fi
227+
228+
done
229+
230+
if [ ${#TABLE_ROWS[@]} -eq 0 ]; then
231+
echo "--------------------------------------------------------------------------------------------------------------------"
232+
233+
echo "❌ No regions have sufficient quota for all required models. Please request a quota increase: https://aka.ms/oai/stuquotarequest"
234+
else
235+
echo "---------------------------------------------------------------------------------------------------------------------"
236+
printf "| %-4s | %-20s | %-43s | %-10s | %-10s | %-10s |\n" "No." "Region" "Model Name" "Limit" "Used" "Available"
237+
echo "---------------------------------------------------------------------------------------------------------------------"
238+
for ROW in "${TABLE_ROWS[@]}"; do
239+
echo "$ROW"
240+
done
241+
echo "---------------------------------------------------------------------------------------------------------------------"
242+
echo "➡️ To request a quota increase, visit: https://aka.ms/oai/stuquotarequest"
243+
fi
244+
245+
echo "✅ Script completed."

src/backend-api/src/app/.env.example

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,7 @@ APP_CONFIGURATION_URL=""
33
# APP_LOGGING_ENABLE=true
44
# APP_LOGGING_LEVEL="INFO"
55

6+
# Azure Logging Configuration
7+
# AZURE_PACKAGE_LOGGING_LEVEL="WARNING" # Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
8+
# AZURE_LOGGING_PACKAGES="azure.core.pipeline.policies.http_logging_policy,azure.storage.blob,azure.storage.queue,azure.core,azure.identity,azure.storage,azure.core.pipeline,azure.core.pipeline.policies,azure.core.pipeline.transport,openai,openai._client,httpx,httpcore,semantic_kernel,urllib3,msal"
9+

src/backend-api/src/app/libs/application/application_configuration.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ class Configuration(_configuration_base, KernelBaseSettings):
3333
app_logging_level: str = Field(default="INFO")
3434
app_sample_variable: str = Field(default="Hello World!")
3535

36+
# Azure logging configuration
37+
azure_package_logging_level: str = Field(default="WARNING", alias="AZURE_PACKAGE_LOGGING_LEVEL")
38+
azure_logging_packages: str | None = Field(default=None, alias="AZURE_LOGGING_PACKAGES")
39+
3640
global_llm_service: str | None = "AzureOpenAI"
3741
cosmos_db_process_log_container: str | None = Field(
3842
default=None, env="COSMOS_DB_PROCESS_LOG_CONTAINER"

0 commit comments

Comments
 (0)