Skip to content

exemptUserAgents should support substring matching, not just prefix #80

@pavelforever

Description

@pavelforever

Problem

The exemptUserAgents config option only matches user agent prefixes, but most real-world bots (including Yandex) use a user agent string that starts with Mozilla/5.0, not with the bot name itself.

For example, YandexBot user agent looks like:
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Setting exemptUserAgents: YandexBot does nothing because the string starts with Mozilla/5.0, not YandexBot.

Expected behavior

exemptUserAgents should support substring matching (i.e. strings.Contains instead of strings.HasPrefix), so that setting exemptUserAgents: YandexBot would correctly exempt any user agent containing YandexBot.

Current workaround

The only workarounds are:

  1. Use the full prefix: exemptUserAgents: Mozilla/5.0 (compatible; YandexBot — fragile and ugly
  2. Add all Yandex IP ranges to exemptIps — requires maintaining a separate IP list

Suggested fix

Change the matching logic in exemptUserAgents from strings.HasPrefix to strings.Contains (case-insensitive).

This would be backward compatible — existing configs using short prefixes like edge would still work since Contains is a superset of HasPrefix behavior for non-prefix patterns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions