Skip to content

Swallowing Critical GitHub API Errors (KeyError) in github_fetch.py #300

Description

@pranjal2004838

Description

In scripts/1-fetch/github_fetch.py, the script fetches public repository counts for various Creative Commons tools. When parsing the GitHub API search response, the code expects to find "total_count" in the returned JSON object:
count = search_data["total_count"]

However, if the GitHub API returns an error response (such as "API Rate Limit Exceeded" or "Invalid Token" / "Bad credentials"), "total_count" will not exist in the response payload. As a result, the script encounters a KeyError on "total_count", which is caught by:

        except KeyError as e:
            raise shared.QuantifyingException(f"KeyError: {e}", 1)

This swallows the actual error details returned by the GitHub API (e.g., in the "message" field), making debugging extremely difficult without manually outputting the API response payload.

Reproduction

  1. Run python scripts/1-fetch/github_fetch.py using a rate-limited network or an invalid/expired GH_TOKEN.
  2. The GitHub API returns a JSON response lacking the "total_count" key (e.g., containing "message": "API rate limit exceeded..." or "message": "Bad credentials").
  3. The script throws KeyError: 'total_count', which is converted into QuantifyingException("KeyError: 'total_count'", 1).
  4. Observe the script exiting with a generic KeyError: 'total_count' exception, concealing the actual API rate limiting or authentication message.

Expectation

The script should check for GitHub API error messages (like "message" or other error indicators in the JSON response) before attempting to access "total_count", logging the specific error payload from GitHub to provide clarity on why the request failed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions