Motivation
After updating and re-enabling the daily fetch cronjob to show Coldfront users their monthly allocation charges on 03/19, it was discovered that the Cumulative Daily Charges for Month field did not accurately reflect the total charges for OpenStack allocations. More specifically, some portion of Openstack storage costs may not be reflected, meaning the value for Cumulative Daily Charges for Month is likely deflated compared to the true cost incurred by OpenStack users. This could mean some users may have already exceeded their cost alerts, and are not being notified. The bug can be demonstrated by inspecting the storage invoices since the March Coldfront maintenance (03/10).
Note that this field is only inaccurate for today as of time of writing. It has been accurate up to the last time the daily fetch cronjob was executed (around late February). The cause of this bug was introduced during the March Coldfront maintenance, with the introduction of dynamic resource quotas. Below is a technical explanation of the bug, followed by the proposed solution by @knikolla:
Technical explanation:
Introduction of dynamic resource quotas changed how the storage calculation script (calculate_storage_gb_hours.py) exported storage costs to the storage CSV invoice. Prior to this update, the two storage attributes for Openstack (OpenStack Swift Quota (GiB), OpenStack Volume Quota (GiB)) would have the cost of their usage summed together and written out to CSV as a single row, something that looks like this:
...,<example allocation id>,pi@example.com,...,OpenStack Storage,...,<example storage cost>,...
After the update, the storage costs for each attribute was written in separate rows. Note that both would still have the same SU name of OpenStack Storage. This shared SU name is a component of the bug.
The current logic (the use of the to_dict() function) of the daily usage fetch script (fetch_daily_billable_usage.py) assumes that each allocation has only one row per SU type in the storage invoice (i.e one row for OpenStack Storage, one for OpenShift NESE Storage). This assumption was true until the introduction of dynamic quotas due to the behavior change describe above. Since OpenStack allocations now have two rows with the same SU, one row will "overshadow" the other, leading to a lower number being shown on the Cumulative Daily Charges for Month attribute for OpenStack allocations.
Note that the storage attributes for Openshift have unique SU names, so OpenShift allocations are not affected by this bug
Proposed solution:
To prevent code changes, we can update the SU name for the two OpenStack storage quotas through CLI commands. @jtriley @joachimweyl @Milstein Would like your confirmation on the new names.
The commands would follow this format:
coldfront add_quota_to_resource --resource-name NERC --invoice-name <New SU Name> --display-name "OpenStack Volume Quota (GiB)" --resource-type storage --quota-label <Same as current label> --multiplier 20 --static-quota 0
coldfront add_quota_to_resource --resource-name NERC --invoice-name <New SU Name> --display-name "OpenStack Swift Quota (GiB)" --resource-type storage --quota-label <Same as current label> --multiplier 1 --static-quota 0
Completion Criteria
Cumulative Daily Charges for Month shows the correct costs of Coldfront allocations
Description
Completion dates
Desired - ASAP
Required - ASAP
Motivation
After updating and re-enabling the daily fetch cronjob to show Coldfront users their monthly allocation charges on 03/19, it was discovered that the
Cumulative Daily Charges for Monthfield did not accurately reflect the total charges for OpenStack allocations. More specifically, some portion of Openstack storage costs may not be reflected, meaning the value forCumulative Daily Charges for Monthis likely deflated compared to the true cost incurred by OpenStack users. This could mean some users may have already exceeded their cost alerts, and are not being notified. The bug can be demonstrated by inspecting the storage invoices since the March Coldfront maintenance (03/10).Note that this field is only inaccurate for today as of time of writing. It has been accurate up to the last time the daily fetch cronjob was executed (around late February). The cause of this bug was introduced during the March Coldfront maintenance, with the introduction of dynamic resource quotas. Below is a technical explanation of the bug, followed by the proposed solution by @knikolla:
Technical explanation:
Introduction of dynamic resource quotas changed how the storage calculation script (
calculate_storage_gb_hours.py) exported storage costs to the storage CSV invoice. Prior to this update, the two storage attributes for Openstack (OpenStack Swift Quota (GiB),OpenStack Volume Quota (GiB)) would have the cost of their usage summed together and written out to CSV as a single row, something that looks like this:After the update, the storage costs for each attribute was written in separate rows. Note that both would still have the same SU name of
OpenStack Storage. This shared SU name is a component of the bug.The current logic (the use of the
to_dict()function) of the daily usage fetch script (fetch_daily_billable_usage.py) assumes that each allocation has only one row per SU type in the storage invoice (i.e one row forOpenStack Storage, one forOpenShift NESE Storage). This assumption was true until the introduction of dynamic quotas due to the behavior change describe above. Since OpenStack allocations now have two rows with the same SU, one row will "overshadow" the other, leading to a lower number being shown on theCumulative Daily Charges for Monthattribute for OpenStack allocations.Note that the storage attributes for Openshift have unique SU names, so OpenShift allocations are not affected by this bug
Proposed solution:
To prevent code changes, we can update the SU name for the two OpenStack storage quotas through CLI commands. @jtriley @joachimweyl @Milstein Would like your confirmation on the new names.
The commands would follow this format:
Completion Criteria
Cumulative Daily Charges for Monthshows the correct costs of Coldfront allocationsDescription
Completion dates
Desired - ASAP
Required - ASAP