Introduce test and jinja

Author: subdavis

AGENTS.md

@@ -0,0 +1,4 @@
+This is a mise project, and the python virtualenv will always be active here.
+Always read README.md for usage instructions.
+Always run `mise validate` after making changes to the codebase.
+If you are making a large batch of changes, run validation at the end.

README.md

@@ -1,21 +1,27 @@
 # RSS Glue
 
-RSS Glue is a highly extensible, filesystem-based RSS/Atom feed generator and manipulator. Build digests, merge feeds, and use AI tools to make your RSS feed work for you!
+RSS Glue is a highly extensible, filesystem-based RSS/Atom feed generator and manipulator.
+
+* Compose, merge, digest, and filter feeds.
+* Take action on feed content (limited, this isnt an automation platform).
+* Make your own feed pipelines with a little bit of Python.
 
 <img src='./docs/images/glue.webp' width=300 style='border-radius: 10px' />
 
-## Inspiration
+RSS Glue functions as either a static generator and an on-demand feed server.
 
-* [Kill the newsletter](https://kill-the-newsletter.com/)
-* [Rsshub](https://docs.rsshub.app/)
-* [Zapier](https://zapier.com/)
+* `rss-glue watch` - **Static generator mode.** Watches your configuration and regenerates feeds on a schedule.
+* `rss-glue debug` - **On-demand server mode.** Serves feeds and generates them on-demand when requested.
 
-## Features
+## Inputs and Outputs
 
 External Data Sources
 
-* `RssFeed` is a data source using an external RSS Feed.
+* `RssFeed` an external RSS Feed.
 * `RedditFeed` because the standard Reddit RSS feed leaves out too much content and you can get more from the JSON api.
+* `HackerNewsFeed` Hacker News stories (top, new, or best).
+* `InstagramFeed` public Instagram profiles using ScrapeCreator.
+* `FacebookFeed` public Facebook groups and pages using ScrapeCreator.
 
 Meta Data Sources
 
@@ -26,6 +32,8 @@ Meta Data Sources
 
 Outputs
 
+Your feeds generate the following outputs. Each source feed in the configuration file will generate a fixed set of output artifacts.
+
 * `RssOutput` is an output RSS feed.
 * `HTMLOutput` is a very basic single page web feed output.
 * `HTMLIndexOutput` is a meta-output HTML page with a link to all its child outputs. Handy for quick reference and adding feeds to your RSS reader.
@@ -42,12 +50,22 @@ pip install rss-glue
 # Create your configuration file and edit it
 touch config.py
 
-# Then generate your feed
+# Populate your feed
 rss-glue --config config.py update
-# Or start a long-running process
+
+# Start a watcher process to regenerate files on schedule
 rss-glue --config config.py watch
-# Or start the debug web server
+
+# Start a debug web server
 rss-glue --config config.py debug
+
+# Force regeneration of all files
+rss-glue --config config.py generate --force
+
+# Repair the timestamps on the disk database
+# Rss glue depends on manipulating file modification times for caching.
+# Editing files outside of rss-glue can break this.
+rss-glue --config config.py repair
 ```
 
 ### Docker
@@ -96,7 +114,7 @@ global_config.configure(
 # All times and schedules are in UTC
 cron_weekly_on_sunday = "0 5 * * 0"
 
-_outputs = [
+sources = [
     # A weekly digest of the F1 subreddit
     DigestFeed(
         RssFeed(
@@ -115,28 +133,40 @@ _outputs = [
         )
     ),
 ]
+```
 
-# Finally, declare your artifacts
-artifacts = [
-    HTMLIndexOutput(
-        OpmlOutput(RssOutput(*_outputs)),
-        HTMLOutput(*_outputs),
-    )
-]
+### Building your own feeds
+
+This project is intended to be quite extensible. To build a new data source or feed transformer, just fork this project and write a new feed class.
+
+```python
+from rss_glue.feeds.base_feed import BaseFeed
+
+class MyCustomFeed(BaseFeed):
+    def __init__(self, namespace: str, some_parameter: str):
+        super().__init__(namespace)
+        self.some_parameter = some_parameter
+
+    def update(self):
+        for post in []:
+            post_id = 'id'
+            self.cache_set(post_id, value.to_dict())
+        self.set_last_run()
 ```
 
 ## Design philosophy
 
 RSS Glue is a simple python tool to manage feed generation and outputs exclusively to the local filesystem. The files it generates could be exposed with a web server, pushed up to an S3 bucket, or built into a netlify deployment. You don't need to operate a server or have some kind of docker host to run it.
 
-It is not intended to scale beyond a few hundred feeds because, well, you're a human and you can't read all that anyway! It isn't intended to deploy as a multi-user app on the web. It will not get a frontend or a configuation language.
+It is not intended to scale beyond a few hundred feeds because, well, you're a human and you can't read all that anyway! It isn't intended to deploy as a multi-user app on the web. It will get a SPA app or a configuration language.
 
-**Compared with RSSHub**
+This project is a little like a tiny, feed-oriented version of [dagster](https://dagster.io/). Your feed definition will be a list of DAGs.  Each root feed will generate a predefined set of outputs.
 
-RSSHub is cool, but has several problems that RSS Glue tries to solve. The integrations I care about are either broken or don't work very well. Features like merge and digest are impossible under its stateless architecture.
+**Compared with RSSHub**
 
-You can use RSS Glue and RSSHub together though!
+RSSHub does not work very well. If it's meeting your needs, you probably don't need RSS Glue.
 
-**Compared with Zapier**
+**Event-driven vs Polling**
 
-Zapier suffers from trying to be all things to all people, and the configuration hell that such an endeavor always leads to.  It's kinda good at a lot of stuff.
+RSS Glue is a polling-based system. It wakes up on a schedule, fetches feeds, generates outputs, and goes back to sleep. 
+This is because some feeds can be expensive to fetch and process, and it's better to do that work outside the request thread of the downstream RSS consumer. It also makes it easier to trace problems because artifacts are statically generated.

docker-compose.yml

@@ -1,6 +1,19 @@
 version: '3.8'
 services:
 
+  # MongoDB service for caching
+  mongo:
+    image: mongo:7
+    container_name: rssglue_mongo
+    restart: unless-stopped
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: rssglue
+      MONGO_INITDB_ROOT_PASSWORD: changeme
+    volumes:
+      - mongo_data:/data/db
+    ports:
+      - "27017:27017"
+
   # rssglue service generates the static files into the rssglue_static volume
   rssglue:
     image: rssglue:latest
@@ -9,10 +22,14 @@ services:
       dockerfile: Dockerfile
     container_name: rssglue
     restart: unless-stopped
+    environment:
+      MONGO_CONNECTION_STRING: mongodb://rssglue:changeme@mongo:27017/
     volumes:
       - rssglue_static:/opt/rssglue/src/static
       - ./samples/docker-config.py:/opt/rssglue/config.py
     command: --config /opt/rssglue/config.py watch
+    depends_on:
+      - mongo
 
   # Nginx service serves the static files from the rssglue_static volume
   nginx:
@@ -25,4 +42,5 @@ services:
       - "5000:80"
 
 volumes:
-  rssglue_static:
+  rssglue_static:
+  mongo_data:

docs/CacheErrorHandling.md

@@ -0,0 +1,61 @@
+# Cache Error Handling Implementation
+
+## Overview
+
+The cache feed now implements robust error handling for failed media downloads. When a download fails, the system marks it to prevent repeated attempts.
+
+## Implementation Details
+
+### Failed Download Marking
+
+When a media download fails (image or video):
+
+1. **Metadata File Creation**: A `.failed` file is created with the following structure:
+   ```json
+   {
+     "timestamp": 1234567890.0,
+     "error": "HTTP 404: Not Found",
+     "url_hash": "abc123def456"
+   }
+   ```
+
+No empty placeholder file is created - only the `.failed` metadata file is needed.
+
+### Detection Mechanism
+
+The system checks if a download previously failed by simply looking for the existence of a `.failed` file with the expected name pattern.
+
+### Example
+
+For a failed image download:
+- Metadata file: `static/images/my_feed/a1b2c3d4e5f6.jpg.failed` (contains error details)
+- No cache file is created
+
+For a successful download:
+- Cache file: `static/images/my_feed/a1b2c3d4e5f6.jpg` (contains actual image data)
+- No `.failed` file exists
+
+## Benefits
+
+1. **No Repeated Failures**: Once a download fails, it won't be attempted again
+2. **Debugging Information**: The `.failed` file contains error details for troubleshooting
+3. **No External Dependencies**: Uses only standard library features
+4. **Cross-Platform**: Works on macOS, Linux, and Windows
+5. **Minimal Overhead**: Only one small JSON metadata file per failure (no empty placeholder files)
+6. **Efficient**: Only one file check needed (no need to check both file size and metadata)
+
+## Cleanup
+
+To retry failed downloads, simply delete the `.failed` file:
+
+```bash
+# Find all failed downloads
+find static/images -name "*.failed"
+find static/videos -name "*.failed"
+
+# Remove a specific failed download marker to retry
+rm static/images/my_feed/abc123.jpg.failed
+
+# Remove all failed markers to retry everything
+find static -name "*.failed" -delete
+```

docs/FeedLocking.md

@@ -0,0 +1,80 @@
+# Feed Locking Feature
+
+## Overview
+
+The feed locking feature allows you to prevent feeds from being updated automatically. This is useful when a feed requires manual intervention or when you want to temporarily disable updates for a specific feed.
+
+## How It Works
+
+When a feed is locked:
+- A `locked: true` flag is added to the feed's metadata file
+- Updates are skipped during normal `update` and `watch` operations
+- A warning is logged when an update is attempted on a locked feed
+- The lock can be overridden by using `force=True` (via the `--force` flag)
+
+## Usage
+
+### Locking a Feed
+
+```bash
+rss-glue --config config.py lock <namespace>
+```
+
+Example:
+```bash
+rss-glue --config config.py lock reddit_minneapolis
+```
+
+### Unlocking a Feed
+
+```bash
+rss-glue --config config.py unlock <namespace>
+```
+
+Example:
+```bash
+rss-glue --config config.py unlock reddit_minneapolis
+```
+
+### Viewing Lock Status
+
+The lock status is displayed when listing sources:
+
+```bash
+rss-glue --config config.py sources
+```
+
+Output will show 🔒 for locked feeds and 🔓 for unlocked feeds:
+```
+🔒 2025-11-12 10:30:00+00:00 -- Reddit Minneapolis -- RedditFeed#reddit_minneapolis
+🔓 2025-11-12 11:45:00+00:00 -- Hacker News -- HackerNewsFeed#hackernews_best
+```
+
+### Force Updating a Locked Feed
+
+You can still force an update on a locked feed by using the `--force` flag:
+
+```bash
+rss-glue --config config.py update --force --feed <namespace>
+```
+
+Example:
+```bash
+rss-glue --config config.py update --force --feed reddit_minneapolis
+```
+
+## When to Use Feed Locking
+
+1. **Manual Intervention Required**: When a feed needs debugging or manual fixes
+2. **Temporary Disable**: When you want to temporarily stop updates without removing the feed from your configuration
+3. **Rate Limiting**: When a feed source is having issues and you need to prevent repeated failed update attempts
+4. **Development**: When testing other feeds and you want to skip updating certain expensive feeds
+
+## Implementation Details
+
+The lock is implemented at the base feed level, which means:
+- All feed types support locking (RSS, Reddit, Instagram, Merge, Digest, etc.)
+- The lock check happens early in the update process
+- The lock state is persisted in the feed's metadata cache
+- Locked feeds can still be read and generate outputs normally
+- Only the update/refresh operation is blocked

docs/Roadmap.md

@@ -0,0 +1,15 @@
+## Problem: Image download failures cause infinite retries
+
+Images that fail to download will be retried every generate cycle. It's not a huge problem, but it would be better to limit the number of retries per image.
+
+## Problem: Image cache is broken by namespace so images in different feeds are duplicated
+
+Fix the file cache so that images are only stored once, even if they are used in multiple feeds.
+
+## Generate example usages
+
+* Changing the rendering of a feed item
+* Changing the rendering of a specific Instagram Feed Item
+* Appending expensive metadata to a feed item (e.g., fetching article `og` metadata)
+* Building an apprise notification from new feed items
+* Building a digest of a merge of multiple feeds