Add support for CVEListV5 and update the schema parser to handle both CVEListV5 and Vulnrichment

[ziadhany]

• issue: Support CVE JSON 5.0 and Import https://github.com/CVEProject/cvelistV5 data #1015

Logs:

Importing data using cvelistv5_importer_v2
INFO 2026-06-20 14:50:24.445800 UTC Pipeline [CVEListV5ImporterPipeline] starting
INFO 2026-06-20 14:50:24.445968 UTC Step [clone] starting
INFO 2026-06-20 14:50:24.446040 UTC Cloning `git+https://github.com/CVEProject/cvelistV5`
INFO 2026-06-20 16:11:34.224001 UTC Step [clone] completed in 4870 seconds (1.4 hours)
INFO 2026-06-20 16:11:34.233702 UTC Step [collect_and_store_advisories] starting
INFO 2026-06-20 16:11:34.254607 UTC Collecting 2 advisories
...
...
ERROR 2026-06-20 16:22:33.719053 UTC Failed to import advisory: AdvisoryDataV2(advisory_id='CVE-2026-6784', aliases=[], summary='Memory safety bugs present in Firefox 149 and Thunderbird 149. Some of these bugs showed evidence of memory corruption and we presume that with enough effort some of these could have been exploited to run arbitrary code. This vulnerability was fixed in Firefox 150 and Thunderbird 150.', affected_packages=[], references=[ReferenceV2(reference_id='buglist.cgi?bug_id=1536243%2C1745382%2C1851073%2C1893400%2C1963301%2C2001319%2C2002899%2C2012436%2C2014435%2C2016901%2C2019916%2C2020486%2C2020612%2C2020817%2C2021788%2C2022051%2C2022367%2C2022431%2C2023302%2C2023670%2C2024225%2C2024238%2C2024240%2C2024265%2C2024367%2C2024369%2C2024424%2C2024760%2C2025281%2C2025361%2C2025387%2C2025466%2C2025954%2C2025958%2C2026278%2C2026292%2C2026297%2C2026378%2C2027148%2C2027287%2C2027341%2C2027384%2C2027427%2C2027694%2C2027993%2C2028009%2C2028270%2C2028416%2C2028524%2C2029295%2C2029699%2C2029800%2C2029801', reference_type='other', url='https://bugzilla.mozilla.org/buglist.cgi?bug_id=1536243%2C1745382%2C1851073%2C1893400%2C1963301%2C2001319%2C2002899%2C2012436%2C2014435%2C2016901%2C2019916%2C2020486%2C2020612%2C2020817%2C2021788%2C2022051%2C2022367%2C2022431%2C2023302%2C2023670%2C2024225%2C2024238%2C2024240%2C2024265%2C2024367%2C2024369%2C2024424%2C2024760%2C2025281%2C2025361%2C2025387%2C2025466%2C2025954%2C2025958%2C2026278%2C2026292%2C2026297%2C2026378%2C2027148%2C2027287%2C2027341%2C2027384%2C2027427%2C2027694%2C2027993%2C2028009%2C2028270%2C2028416%2C2028524%2C2029295%2C2029699%2C2029800%2C2029801'), ReferenceV2(reference_id='mfsa2026-30', reference_type='other', url='https://www.mozilla.org/security/advisories/mfsa2026-30/'), ReferenceV2(reference_id='mfsa2026-33', reference_type='other', url='https://www.mozilla.org/security/advisories/mfsa2026-33/')], patches=[], date_published=datetime.datetime(2026, 4, 21, 12, 41, 13, 111000, tzinfo=<StaticTzInfo 'UTC'>), weaknesses=[], severities=[VulnerabilitySeverity(system=Cvssv3ScoringSystem(identifier='cvssv3.1', name='CVSSv3.1 Base Score', url='https://www.first.org/cvss/v3-1/', notes='CVSSv3.1 base score and vector'), value='7.5', scoring_elements='CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H', published_at=None, url=None), VulnerabilitySeverity(system=SSVCScoringSystem(identifier='ssvc', name='Stakeholder-Specific Vulnerability Categorization', url='https://www.cisa.gov/stakeholder-specific-vulnerability-categorization-ssvc', notes=''), value='Track', scoring_elements='SSVCv2/E:N/A:N/T:T/P:M/B:A/M:M/D:T/2026-04-22T03:56:17Z/', published_at=None, url=None)], url='https://github.com/CVEProject/cvelistV5/blob/main/cves/2026/6xxx/CVE-2026-6784.json', original_advisory_text='{\n  "dataType": "CVE_RECORD",\n  "dataVersion": "5.2",\n  "cveMetadata": {\n    "cveId": "CVE-2026-6784",\n    "assignerOrgId": "f16b083a-5664-49f3-a51e-8d479e5ed7fe",\n    "state": "PUBLISHED",\n    "assignerShortName": "mozilla",\n    "dateReserved": "2026-04-21T12:41:12.823Z",\n    "datePublished": "2026-04-21T12:41:13.111Z",\n    "dateUpdated": "2026-05-27T17:06:44.028Z"\n  },\n  "containers": {\n    "cna": {\n      "affected": [\n        {\n          "product": "Firefox",\n          "vendor": "Mozilla",\n          "versions": [\n            {\n              "status": "unaffected",\n              "version": "150",\n              "lessThanOrEqual": "*",\n              "versionType": "rpm"\n            }\n          ]\n        },\n        {\n          "product": "Thunderbird",\n          "vendor": "Mozilla",\n          "versions": [\n            {\n              "status": "unaffected",\n              "version": "150",\n              "lessThanOrEqual": "*",\n              "versionType": "rpm"\n            }\n          ]\n        }\n      ],\n      "descriptions": [\n        {\n          "lang": "en",\n          "value": "Memory safety bugs present in Firefox 149 and Thunderbird 149. Some of these bugs showed evidence of memory corruption and we presume that with enough effort some of these could have been exploited to run arbitrary code. This vulnerability was fixed in Firefox 150 and Thunderbird 150.",\n          "supportingMedia": [\n            {\n              "type": "text/html",\n              "base64": false,\n              "value": "Memory safety bugs present in Firefox 149 and Thunderbird 149. Some of these bugs showed evidence of memory corruption and we presume that with enough effort some of these could have been exploited to run arbitrary code. This vulnerability was fixed in Firefox 150 and Thunderbird 150."\n            }\n          ]\n        }\n      ],\n      "title": "Memory safety bugs fixed in Firefox 150 and Thunderbird 150",\n      "references": [\n        {\n          "url": "https://bugzilla.mozilla.org/buglist.cgi?bug_id=1536243%2C1745382%2C1851073%2C1893400%2C1963301%2C2001319%2C2002899%2C2012436%2C2014435%2C2016901%2C2019916%2C2020486%2C2020612%2C2020817%2C2021788%2C2022051%2C2022367%2C2022431%2C2023302%2C2023670%2C2024225%2C2024238%2C2024240%2C2024265%2C2024367%2C2024369%2C2024424%2C2024760%2C2025281%2C2025361%2C2025387%2C2025466%2C2025954%2C2025958%2C2026278%2C2026292%2C2026297%2C2026378%2C2027148%2C2027287%2C2027341%2C2027384%2C2027427%2C2027694%2C2027993%2C2028009%2C2028270%2C2028416%2C2028524%2C2029295%2C2029699%2C2029800%2C2029801",\n          "name": "Memory safety bugs fixed in Firefox 150 and Thunderbird 150"\n        },\n        {\n          "url": "https://www.mozilla.org/security/advisories/mfsa2026-30/"\n        },\n        {\n          "url": "https://www.mozilla.org/security/advisories/mfsa2026-33/"\n        }\n      ],\n      "credits": [\n        {\n          "lang": "en",\n          "value": "Ben Visness, Brian Grinstead, Christian Holler, Dimi Lee, Jens Stutte, Jim Mathies, John Schanck, Jon Coppeard, Karl Tomlinson, Maurice Dauer, Nika Layzell, Randell Jesup, Tom Schuster and the Mozilla Fuzzing Team"\n        }\n      ],\n      "providerMetadata": {\n        "orgId": "f16b083a-5664-49f3-a51e-8d479e5ed7fe",\n        "shortName": "mozilla",\n        "dateUpdated": "2026-05-07T15:12:53.213Z"\n      }\n    },\n    "adp": [\n      {\n        "problemTypes": [\n          {\n            "descriptions": [\n              {\n                "type": "CWE",\n                "cweId": "CWE-125",\n                "lang": "en",\n                "description": "CWE-125 Out-of-bounds Read"\n              }\n            ]\n          },\n          {\n            "descriptions": [\n              {\n                "type": "CWE",\n                "cweId": "CWE-787",\n                "lang": "en",\n                "description": "CWE-787 Out-of-bounds Write"\n              }\n            ]\n          },\n          {\n            "descriptions": [\n              {\n                "type": "CWE",\n                "cweId": "CWE-416",\n                "lang": "en",\n                "description": "CWE-416 Use After Free"\n              }\n            ]\n          }\n        ],\n        "metrics": [\n          {\n            "cvssV3_1": {\n              "scope": "UNCHANGED",\n              "version": "3.1",\n              "baseScore": 7.5,\n              "attackVector": "NETWORK",\n              "baseSeverity": "HIGH",\n              "vectorString": "CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H",\n              "integrityImpact": "HIGH",\n              "userInteraction": "REQUIRED",\n              "attackComplexity": "HIGH",\n              "availabilityImpact": "HIGH",\n              "privilegesRequired": "NONE",\n              "confidentialityImpact": "HIGH"\n            }\n          },\n          {\n            "other": {\n              "type": "ssvc",\n              "content": {\n                "timestamp": "2026-04-22T03:56:17.140808Z",\n                "id": "CVE-2026-6784",\n                "options": [\n                  {\n                    "Exploitation": "none"\n                  },\n                  {\n                    "Automatable": "no"\n                  },\n                  {\n                    "Technical Impact": "total"\n                  }\n                ],\n                "role": "CISA Coordinator",\n                "version": "2.0.3"\n              }\n            }\n          }\n        ],\n        "title": "CISA ADP Vulnrichment",\n        "providerMetadata": {\n          "orgId": "134c704f-9b21-4f2e-91b3-4a467353bcc0",\n          "shortName": "CISA-ADP",\n          "dateUpdated": "2026-05-27T17:06:44.028Z"\n        }\n      }\n    ]\n  }\n}') with error DataError('value too long for type character varying(500)\n'):
Traceback (most recent call last):
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(500)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/vulnerabilities/pipelines/__init__.py", line 341, in collect_and_store_advisories
    if _obj := insert_advisory_v2(
               ~~~~~~~~~~~~~~~~~~^
        advisory=advisory,
        ^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        precedence=self.precedence,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/home/ziad-hany/.pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/vulnerabilities/pipes/advisory.py", line 349, in insert_advisory_v2
    references = get_or_create_advisory_references(references=advisory.references)
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/vulnerabilities/pipes/advisory.py", line 82, in get_or_create_advisory_references
    AdvisoryReference.objects.bulk_create(to_create, ignore_conflicts=True)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/models/query.py", line 825, in bulk_create
    returned_columns = self._batched_insert(
        objs_without_pk,
    ...<4 lines>...
        unique_fields=unique_fields,
    )
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/models/query.py", line 1912, in _batched_insert
    self._insert(
    ~~~~~~~~~~~~^
        item,
        ^^^^^
    ...<4 lines>...
        unique_fields=unique_fields,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/models/query.py", line 1873, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 1882, in execute_sql
    cursor.execute(sql, params)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 122, in execute
    return super().execute(sql, params)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sql, params, many=False, executor=self._execute
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/ziad-hany/PycharmProjects/vulnerablecode/venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
django.db.utils.DataError: value too long for type character varying(500)


INFO 2026-06-20 17:17:03.042258 UTC Successfully collected 359,521 advisories
INFO 2026-06-20 17:17:03.042394 UTC Step [collect_and_store_advisories] completed in 3929 seconds (1.1 hours)
INFO 2026-06-20 17:17:03.042460 UTC Step [clean_downloads] starting
INFO 2026-06-20 17:17:03.042513 UTC Removing cloned repository
INFO 2026-06-20 17:17:08.863958 UTC Step [clean_downloads] completed in 6 seconds
INFO 2026-06-20 17:17:08.864108 UTC Pipeline completed in 8804 seconds (2.4 hours)

Process finished with exit code 0

[ziadhany]

@TG1999 Please have a look and let me know if I need to change anything!