-
Notifications
You must be signed in to change notification settings - Fork 210
fix: Council Fix Pack - March 2026 #1841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
3833956
ce62fa2
a46ffb6
24c50f8
a9f94d4
5dccfdc
9c45fde
add90fc
be071e2
c1c70e6
473c40b
ef6ec89
db52375
1747edf
3aed4cc
36573c3
c5cf578
56dfaa5
8f2823f
04a919c
e2c78af
93fa9f8
3593f0d
ceb5157
f7f1d6c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,57 @@ | ||||||||||||||||||||||||
| from datetime import datetime | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| import requests | ||||||||||||||||||||||||
| from bs4 import BeautifulSoup | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| from uk_bin_collection.uk_bin_collection.common import * | ||||||||||||||||||||||||
| from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| # import the wonderful Beautiful Soup and the URL grabber | ||||||||||||||||||||||||
| class CouncilClass(AbstractGetBinDataClass): | ||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||
| Concrete classes have to implement all abstract operations of the | ||||||||||||||||||||||||
| base class. They can also override some operations with a default | ||||||||||||||||||||||||
| implementation. | ||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| def parse_data(self, page: str, **kwargs) -> dict: | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| user_postcode = kwargs.get("postcode") | ||||||||||||||||||||||||
| check_uprn(user_postcode) | ||||||||||||||||||||||||
| bindata = {"bins": []} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| user_postcode = user_postcode.strip().replace(" ", "") | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
Comment on lines
+20
to
+25
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Validate 🛠️ Suggested change- user_postcode = kwargs.get("postcode")
- check_uprn(user_postcode)
+ user_postcode = kwargs.get("postcode")
+ if not user_postcode:
+ raise ValueError("Postcode is required")📝 Committable suggestion
Suggested change
🧰 Tools🪛 Ruff (0.14.14)[error] 21-21: (F405) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||
| URI = f"https://www.lbhf.gov.uk/bin-recycling-day/results?postcode={user_postcode}" | ||||||||||||||||||||||||
| UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" | ||||||||||||||||||||||||
| session = requests.session() | ||||||||||||||||||||||||
| session.headers.update({"User-Agent": UA}) | ||||||||||||||||||||||||
| # Make the GET request | ||||||||||||||||||||||||
| response = session.get(URI) | ||||||||||||||||||||||||
| response.raise_for_status() | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| soup = BeautifulSoup(response.content, features="html.parser") | ||||||||||||||||||||||||
| results = soup.find("div", {"class": "nearest-search-results"}) | ||||||||||||||||||||||||
| ol = results.find("ol") | ||||||||||||||||||||||||
| bin_collections = ol.find_all("a") | ||||||||||||||||||||||||
| for bin_collection in bin_collections: | ||||||||||||||||||||||||
| collection_day = bin_collection.get_text().split(" - ")[0] | ||||||||||||||||||||||||
| collection_type = bin_collection.get_text().split(" - ")[1] | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
Comment on lines
+34
to
+44
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fail explicitly on unexpected HTML structure or entry format. Right now 🛠️ Suggested change- results = soup.find("div", {"class": "nearest-search-results"})
- ol = results.find("ol")
- bin_collections = ol.find_all("a")
+ results = soup.find("div", {"class": "nearest-search-results"})
+ if results is None:
+ raise ValueError("Could not find nearest search results section.")
+ ol = results.find("ol")
+ if ol is None:
+ raise ValueError("Could not find bin collections list.")
+ bin_collections = ol.find_all("a")
...
- collection_day = bin_collection.get_text().split(" - ")[0]
- collection_type = bin_collection.get_text().split(" - ")[1]
+ parts = bin_collection.get_text().split(" - ")
+ if len(parts) != 2:
+ raise ValueError(
+ f"Unexpected bin collection entry format: {bin_collection.get_text()!r}"
+ )
+ collection_day, collection_type = partsBased on learnings, in uk_bin_collection/**/*.py, prefer explicit failures on unexpected formats with clear exception types and messages. 🤖 Prompt for AI Agents |
||||||||||||||||||||||||
| if days_of_week.get(collection_day) == 0: | ||||||||||||||||||||||||
| collection_day = datetime.now().strftime(date_format) | ||||||||||||||||||||||||
| else: | ||||||||||||||||||||||||
| collection_day = get_next_day_of_week(collection_day) | ||||||||||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| dict_data = { | ||||||||||||||||||||||||
| "type": collection_type, | ||||||||||||||||||||||||
| "collectionDate": collection_day, | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| bindata["bins"].append(dict_data) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| bindata["bins"].sort( | ||||||||||||||||||||||||
| key=lambda x: datetime.strptime(x.get("collectionDate"), "%d/%m/%Y") | ||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| return bindata | ||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -76,14 +76,12 @@ This document is still a work in progress, don't worry if your council isn't lis | |
| - [Chorley](#chorley) | ||
| - [Colchester](#colchester) | ||
| - [Conwy](#conwy) | ||
| - [Copeland](#copeland) | ||
| - [Cornwall](#cornwall) | ||
| - [Cotswold](#cotswold) | ||
| - [Coventry](#coventry) | ||
| - [Crawley](#crawley) | ||
| - [Croydon](#croydon) | ||
| - [Cumberland](#cumberland) | ||
| - [Cumberland](#cumberland) | ||
| - [Dacorum](#dacorum) | ||
| - [Darlington Borough Council](#darlington-borough-council) | ||
| - [Dartford](#dartford) | ||
|
|
@@ -111,6 +109,7 @@ This document is still a work in progress, don't worry if your council isn't lis | |
| - [East Staffordshire](#east-staffordshire) | ||
| - [East Suffolk](#east-suffolk) | ||
| - [Eastleigh](#eastleigh) | ||
| - [Eden District (Westmorland and Furness)](#eden-district-(westmorland-and-furness)) | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| - [City of Edinburgh](#city-of-edinburgh) | ||
| - [Elmbridge](#elmbridge) | ||
| - [Enfield](#enfield) | ||
|
|
@@ -171,7 +170,9 @@ This document is still a work in progress, don't worry if your council isn't lis | |
| - [City of Lincoln](#city-of-lincoln) | ||
| - [Lisburn and Castlereagh](#lisburn-and-castlereagh) | ||
| - [Liverpool](#liverpool) | ||
| - [Camden](#camden) | ||
| - [Ealing](#ealing) | ||
| - [Hammersmith & Fulham](#hammersmith-&-fulham) | ||
| - [Harrow](#harrow) | ||
| - [Havering](#havering) | ||
| - [Hounslow](#hounslow) | ||
|
|
@@ -628,13 +629,14 @@ Note: You will need to use [FindMyAddress](https://www.findmyaddress.co.uk/searc | |
|
|
||
| ### Bexley | ||
| ```commandline | ||
| python collect_data.py BexleyCouncil https://waste.bexley.gov.uk/waste -s -u XXXXXXXX | ||
| python collect_data.py BexleyCouncil https://waste.bexley.gov.uk/waste -s -u XXXXXXXX -w http://HOST:PORT/ | ||
| ``` | ||
| Additional parameters: | ||
| - `-s` - skip get URL | ||
| - `-u` - UPRN | ||
| - `-w` - remote Selenium web driver URL (required for Home Assistant) | ||
|
|
||
| Note: Provide your UPRN. Use [FindMyAddress](https://www.findmyaddress.co.uk/search) to locate it. | ||
| Note: Provide your UPRN. Use [FindMyAddress](https://www.findmyaddress.co.uk/search) to locate it. This parser requires a Selenium webdriver. | ||
|
|
||
| --- | ||
|
|
||
|
|
@@ -1156,17 +1158,6 @@ Note: Use [FindMyAddress](https://www.findmyaddress.co.uk/search) to find your U | |
|
|
||
| --- | ||
|
|
||
| ### Copeland | ||
| ```commandline | ||
| python collect_data.py CopelandBoroughCouncil https://www.copeland.gov.uk -u XXXXXXXX | ||
| ``` | ||
| Additional parameters: | ||
| - `-u` - UPRN | ||
|
|
||
| Note: *****This has now been replaced by Cumberland Council**** | ||
|
|
||
| --- | ||
|
|
||
| ### Cornwall | ||
| ```commandline | ||
| python collect_data.py CornwallCouncil https://www.cornwall.gov.uk/my-area/ -s -u XXXXXXXX | ||
|
|
@@ -1229,18 +1220,6 @@ Note: Pass the house number and postcode in their respective parameters. This pa | |
|
|
||
| --- | ||
|
|
||
| ### Cumberland | ||
| ```commandline | ||
| python collect_data.py CumberlandAllerdaleCouncil https://www.allerdale.gov.uk -p "XXXX XXX" -n XX | ||
| ``` | ||
| Additional parameters: | ||
| - `-p` - postcode | ||
| - `-n` - house number | ||
|
|
||
| Note: Pass the house number and postcode in their respective parameters. | ||
|
|
||
| --- | ||
|
|
||
| ### Cumberland | ||
| ```commandline | ||
| python collect_data.py CumberlandCouncil https://www.cumberland.gov.uk/bins-recycling-and-street-cleaning/waste-collections/bin-collection-schedule -u XXXXXXXX | ||
|
|
@@ -1579,6 +1558,17 @@ Note: Pass the UPRN. You can find it using [FindMyAddress](https://www.findmyadd | |
|
|
||
| --- | ||
|
|
||
| ### Eden District (Westmorland and Furness) | ||
| ```commandline | ||
| python collect_data.py EdenDistrictCouncil https://my.eden.gov.uk/myeden.aspx -u XXXXXXXX | ||
| ``` | ||
| Additional parameters: | ||
| - `-u` - UPRN | ||
|
|
||
| Note: For Eden area addresses within Westmorland and Furness. Provide your UPRN. You can find your UPRN using [FindMyAddress](https://www.findmyaddress.co.uk/search). Note: This returns collection days (e.g., 'Wednesday') rather than specific dates. | ||
|
|
||
| --- | ||
|
|
||
| ### City of Edinburgh | ||
| ```commandline | ||
| python collect_data.py EdinburghCityCouncil https://www.edinburgh.gov.uk -s -p "XXXX XXX" -n XX | ||
|
|
@@ -2314,6 +2304,19 @@ Note: You will need to use [FindMyAddress](https://www.findmyaddress.co.uk/searc | |
|
|
||
| --- | ||
|
|
||
| ### Camden | ||
| ```commandline | ||
| python collect_data.py LondonBoroughCamdenCouncil https://environmentservices.camden.gov.uk/property -s -u XXXXXXXX -p "XXXX XXX" | ||
| ``` | ||
| Additional parameters: | ||
| - `-s` - skip get URL | ||
| - `-u` - UPRN | ||
| - `-p` - postcode | ||
|
|
||
| Note: Pass the property ID as UPRN. Find your property at https://www.camden.gov.uk/check-collection-day then use the property ID from the URL (e.g., https://environmentservices.camden.gov.uk/property/5063139). | ||
|
|
||
| --- | ||
|
|
||
| ### Ealing | ||
| ```commandline | ||
| python collect_data.py LondonBoroughEaling https://www.ealing.gov.uk/site/custom_scripts/WasteCollectionWS/home/FindCollection -s -u XXXXXXXX | ||
|
|
@@ -2326,6 +2329,17 @@ Note: Pass the UPRN. You can find it using [FindMyAddress](https://www.findmyadd | |
|
|
||
| --- | ||
|
|
||
| ### Hammersmith & Fulham | ||
| ```commandline | ||
| python collect_data.py LondonBoroughHammersmithandFulham https://www.lbhf.gov.uk/ -p "XXXX XXX" | ||
| ``` | ||
| Additional parameters: | ||
| - `-p` - postcode | ||
|
|
||
| Note: Pass only the property postcode | ||
|
|
||
| --- | ||
|
|
||
| ### Harrow | ||
| ```commandline | ||
| python collect_data.py LondonBoroughHarrow https://www.harrow.gov.uk -u XXXXXXXX | ||
|
|
@@ -3468,7 +3482,7 @@ Note: Follow the instructions [here](https://www.southlanarkshire.gov.uk/info/20 | |
|
|
||
| ### South Norfolk | ||
| ```commandline | ||
| python collect_data.py SouthNorfolkCouncil https://www.southnorfolkandbroadland.gov.uk/rubbish-recycling/south-norfolk-bin-collection-day-finder -s -u XXXXXXXX | ||
| python collect_data.py SouthNorfolkCouncil https://area.southnorfolkandbroadland.gov.uk/FindAddress -s -u XXXXXXXX | ||
| ``` | ||
| Additional parameters: | ||
| - `-s` - skip get URL | ||
|
|
@@ -4257,12 +4271,13 @@ Note: Provide your UPRN. You can find it using [FindMyAddress](https://www.findm | |
|
|
||
| ### Wirral | ||
| ```commandline | ||
| python collect_data.py WirralCouncil https://www.wirral.gov.uk -u XXXXXXXX | ||
| python collect_data.py WirralCouncil https://www.wirral.gov.uk -p "XXXX XXX" -w http://HOST:PORT/ | ||
| ``` | ||
| Additional parameters: | ||
| - `-u` - UPRN | ||
| - `-p` - postcode | ||
| - `-w` - remote Selenium web driver URL (required for Home Assistant) | ||
|
|
||
| Note: In the `uprn` field, enter your street name and suburb separated by a comma (e.g., 'Vernon Avenue,Seacombe'). | ||
| Note: Pass your postcode and house number. | ||
|
Comment on lines
+4298
to
+4304
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add the missing house-number parameter in the Wirral command. The note says to pass a house number, but the command and parameter list omit 🛠️ Suggested fix-python collect_data.py WirralCouncil https://www.wirral.gov.uk -p "XXXX XXX" -w http://HOST:PORT/
+python collect_data.py WirralCouncil https://www.wirral.gov.uk -p "XXXX XXX" -n XX -w http://HOST:PORT/-Additional parameters:
-- `-p` - postcode
-- `-w` - remote Selenium web driver URL (required for Home Assistant)
+Additional parameters:
+- `-p` - postcode
+- `-n` - house number
+- `-w` - remote Selenium web driver URL (required for Home Assistant)🤖 Prompt for AI Agents |
||
|
|
||
| --- | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add
skip_get_url: trueto avoid a prefetch that can fail before parsing.The parser performs its own request; without
skip_get_url, a failed prefetch to the base URL can block the flow or add unnecessary latency. Consider addingskip_get_url: truesoparse_dataruns directly (the auto-generated wiki entry will then include-s).🛠️ Suggested change
"LondonBoroughHammersmithandFulham": { "postcode": "W12 0BQ", + "skip_get_url": true, "url": "https://www.lbhf.gov.uk/", "wiki_command_url_override": "https://www.lbhf.gov.uk/", "wiki_name": "Hammersmith & Fulham",📝 Committable suggestion
🤖 Prompt for AI Agents