[LOG] Adding a filter for the weblog by chromoxdor · Pull Request #5484 · letscontrolit/ESPEasy

chromoxdor · 2026-01-23T12:57:49Z

Made a filter. It was easier than I thought :)
Should I make the changes optional/custom?

Filter the log by strings.
Multiple filter are separated by ;.
Exclude phrases with !.
AND operator for linking phrases &.

chromoxdor · 2026-01-23T13:30:43Z

Should i also change the functions in WebStaticData.h? Or leave it to save space?
Also, I was also thinking of extending the filter functionality to the copyClipboard function. It seems this function is not loaded externally and is also not read when the JS file is copied onto the ESP, so here I definitely would need to change the function in WebStaticData.h.

tonhuisman · 2026-01-23T13:39:52Z

Should i also change the functions in WebStaticData.h? Or leave it to save space?

That WebStaticData.h data is only included in the non-LIMIT_BUILD_SIZE builds, AFAICS, so updating the minified version there would be helpful 😉

FR: Can this search (also) handle wildcards or regular expressions?

chromoxdor · 2026-01-23T13:42:57Z

FR: Can this search (also) handle wildcards or regular expressions?

No, I wanted it to be as simple and small as possible, but this can surely be added.
What exactly do we really need?

wildcards

This is basically included since it filters also phrases. At least in my understanding.

chromoxdor · 2026-01-23T14:00:56Z

I might want to add an option to show an x number of lines after a matching line. But not sure how to flag this as an option in the filter field (also not sure if it is really necessary. I am actually quite happy with how it works now).

TD-er · 2026-01-23T14:15:56Z

You could also add a CTRL-F like option to move to the lines and then add an attribute to those lines to highlight the parts where it occurs.
Similar to how many text editors help highlight search results. (and also grep does)

chromoxdor · 2026-01-23T14:42:46Z

lines to highlight the parts where it occurs.

Like so?:

Edit:
BTW, just realizing ACT : vs EVENT: (after ACT there are two spaces)

tonhuisman · 2026-01-23T14:59:31Z

BTW, just realizing ACT : vs EVENT: (after ACT there are two spaces)

The intent is to have the console log nicely aligned, but 5 characters is not enough to hold BME280 f.e. (also an issue with some other plugins) 🫣

chromoxdor · 2026-01-23T15:18:17Z

Ok.. for now we have:
; -> acts as an OR (I used this for convenience instead of |)
& -> acts as an AND (it uses strict order)

I chose this over standard regex so that mere mortal people can use it too. :)

TD-er · 2026-01-23T16:18:45Z

lines to highlight the parts where it occurs.

Like so?:

Edit: BTW, just realizing ACT : vs EVENT: (after ACT there are two spaces)

Looking good :)

TD-er · 2026-01-23T16:22:00Z

The idea of the ACT : and EVENT: was indeed to make it more aligned.
However it does also add more to the build size. Not just the 2 spaces, but also because in some pieces of code this prefix-like string is used in many more log lines.

So that's also one reason why I was thinking about splitting this into something else. The idea was to allow for filtering, but now you already have that part 'solved'... ;)

chromoxdor · 2026-01-23T20:50:16Z

One thing was missing :)

tonhuisman · 2026-01-23T21:12:28Z

Ah, now we're going somewhere 😆 Just waiting for the code changes 😉

chromoxdor · 2026-01-23T21:30:17Z

Just waiting for the code changes 😉

Comes tomorrow. Right now, I have to find a movie. Which takes all my resources :)
And as if that is not enough, I have to watch it after I find it.

TD-er · 2026-01-23T21:34:55Z

And as if that is not enough, I have to watch it after I find it.

If you can't find it, you can watch a YouTube short... and then continue ;)

- Added highlight filter phrases - Added ordered AND operator (&) - Added exclusion (!)

chromoxdor · 2026-01-24T11:41:14Z

Ok, I just committed all the changes. It would be nice if somebody could test it.
I also changed the code in WebStaticData.h but it adds roughly 1.5kbytes. Why not serve this as gzipped array. Maybe someone can assist. because i don't really know where to make the changes for adding the right header.

Also: I am not sure if highlighting is really necessary. If you like it, we keep it.
The copyclipboard button now only copies what is filtered. I think this is a good idea, but if you could think of a reason why not, I will undo.

TD-er · 2026-01-24T14:26:57Z

Well it is hard to add 'context'.
So I think it is good to have copied what you're interested in.

N.B. did you notice you now also get the web log from boot when you access it within 30 sec after boot? (on ESP32, ESP8266 does have a shorter TTL for log entries)

- Added a check to determine if there is already a filter field present; otherwise, add one for older builds.

chromoxdor · 2026-01-25T13:22:18Z

I added a bit of code so that every build can make use of the filter. The script checks first if there is a filter field and if not injects it via the script.
So if you want to test just place the script file fetch_and_parse_log.js onto your device.

TD-er · 2026-01-25T13:29:09Z

Can you also make a commit which removes that one file which is not part of this PR?

tools/pio/pre_custom_esp32.py

chromoxdor · 2026-01-25T17:32:40Z

N.B. did you notice you now also get the web log from boot when you access it within 30 sec after boot? (on ESP32, ESP8266 does have a shorter TTL for log entries)

BTW: This is really cool and helpful! :)

- Fixes a long-existing autoscroll issue where the whole site would scroll instead of just the log content. - reducing the code by simplifying the „browser check" and refining the loop

…into filter-weblog

...while I am at it :)

chromoxdor · 2026-02-05T10:37:18Z

FIY: I think I am done here.... :)

TD-er · 2026-02-12T15:56:07Z

Hmm this breaks the web log.

log:3 Uncaught ReferenceError: ct1 is not defined
    at log:3:3619

Which is:

window.fetch ? ct1.textContent = 'Fetching log entries...' : ct1.textContent = 'Error: This browser is not supported. Please use a modern browser.';

TD-er · 2026-02-12T16:01:41Z

Seems like this line got lost somehow when minimizing the JS?

const ct1 = elId('copyText_1');

TD-er · 2026-02-12T16:24:00Z

Yep, added that line and logs are working again....

chromoxdor · 2026-02-12T16:27:08Z

Yep, added that line and logs are working again....

Ohh…good. I was just starting a new PR with a fix.
Yes, it got lost somehow. Minifying and putting everything in quotation marks was a bit of a pain in the ass...

TD-er · 2026-02-12T16:29:11Z

There is also something else with the minified code...

In file included from src/src/WebServer/SysInfoPage.cpp:36:
src/src/WebServer/../Static/WebStaticData.h:428:47: warning: trigraph '??'' ignored, use '-trigraphs' to enable [-Wtrigraphs]
  428 | "let n=t.Log.SettingsWebLogLevel,r=logLevel[n]??'Undefined';"

Right now it seems to be ignored and I think I can add a flag to explicitly ignore this as we really don't want to use the ?? as escape char in the string.
No idea what the ?? does in JavaScript though....

chromoxdor · 2026-02-12T16:33:22Z

There is also something else with the minified code...

Can be ignored since the compiler is misinterpreting the JS.
Fix would be to break it up: "let n=t.Log.SettingsWebLogLevel,r=logLevel[n]?" "?'Undefined';"

chromoxdor · 2026-02-12T16:34:35Z

You want me to make a new PR with both fixes?

chromoxdor · 2026-02-12T16:36:19Z

No idea what the ?? does in JavaScript though....

"nullish coalescing operator".
https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing

uzi18 · 2026-02-12T16:48:13Z

@TD-er @chromoxdor little off topic, did you know not all log lines are received by weblog?
some lines are on serial log, when at the same time on weblog are lost (tested on esp8266).

chromoxdor · 2026-02-12T17:14:58Z

did you know not all log lines are received by weblog?

Is this an issue that was introduced by this PR?

TD-er · 2026-02-12T17:16:01Z

@TD-er @chromoxdor little off topic, did you know not all log lines are received by weblog? some lines are on serial log, when at the same time on weblog are lost (tested on esp8266).

With latest ESPEasy code?
The only reason I can imagine is that log lines may get thrown out early if memory is low.
But if you have Error or Info log lines and the JSON gets fetched without timeouts each second, then you should not miss lines on the web log.

TD-er · 2026-02-12T17:16:35Z

You want me to make a new PR with both fixes?

Yes please :)

uzi18 · 2026-02-12T19:35:33Z

@TD-er @chromoxdor little off topic, did you know not all log lines are received by weblog? some lines are on serial log, when at the same time on weblog are lost (tested on esp8266).

With latest ESPEasy code? The only reason I can imagine is that log lines may get thrown out early if memory is low. But if you have Error or Info log lines and the JSON gets fetched without timeouts each second, then you should not miss lines on the web log.

@TD-er latest normal 4M1M release from github, need something more to investigate? Just tried PID in rules implementation and surprised about not all variables printed out in weblog (info level). From log: Freemem 18928

TD-er · 2026-02-12T20:40:51Z

Was it a burst of messages?

The current implementation is a list of log messages.
Each message has a timestamp, TTL, a log level and a bitmask whether it has been read by its subscribers. (e.g. serial has 'info' active, weblog 'debug' and syslog 'error', then a new 'debug' log entry will have for only syslog the subscribe bit set, info for serial and syslog and error message for all 3)

And of course the log line itself.

A message will be removed from this list if either all the subscriber bits are cleared or the TTL has expired.
For ESP8266 the TTL is 5 seconds, unless memory is low (and debug might also have a lower TTL)

For the weblog, as soon as a JSON message has been generated, the messages in that JSON are marked as read for the weblog subscriber.
If the web browser for whatever reason does not receive the JSON and retries to fetch it, then the messages are lost for that view.

I have to check the code, but I don't think there is a max. number of messages anymore, only limited by memory usage.

So on ESP32 (with TTL of 30 sec per message), if you connect to the weblog within 30 sec from boot, you will get all messages. In my tests, I managed to fetch >100 kB of JSON which was working fine.

N.B. flash string log entries will take only the pointer to the flash string (+ the header stuff like timestamp, log level, subscriber count, etc.)

N.B.2 Console output is "LogLevelNone", so it is using the same queue system.
Meaning you can also get the result of a command via the web log.

TD-er · 2026-02-12T20:42:18Z

By the way, which "latest release" do you mean?
This latest? https://td-er.nl/ESPEasy/latest/
Or the last official build? (which does not yet have this what I just explained)
Or the last GH Actions builds?

uzi18 · 2026-02-12T20:44:33Z

@TD-er https://github.com/letscontrolit/ESPEasy/releases/tag/mega-20260125 but this problem is also on 2024 builds.

TD-er · 2026-02-12T20:50:14Z

Yep that's not yet using the newer system I made.
This new log system has already been merged to mega branch, but not yet finalized in an official build.

uzi18 · 2026-02-12T20:52:03Z

@TD-er looks like about lets say 10 lines is lost and then you get 20 lines later after that same lines are not shown and 20 lines are ok.
It looks like synchronous because always same lines are not shown.
Will try to check if these data are just now shown on page but in json exists.

TD-er · 2026-02-12T20:53:08Z

The old log system was using separate buffers.
The web log had only 10-ish lines buffer (or 15, not sure anymore)
That's why I introduced the single log queue with different subscribers.

uzi18 · 2026-02-12T20:56:54Z

What am I suppoust to test?

TD-er · 2026-02-12T20:58:35Z

Well unless you are willing to switch to the newer builds, there is not yet much to test.
The newer builds do have the network/wifi rewrite, so make sure to only test on nodes which are accessible via USB as OTA update of ESP8266 is not yet working flawlessly. (working on it)

TD-er · 2026-02-12T21:14:37Z

I just uploaded the last merged build with this log filter to here: https://td-er.nl/ESPEasy/latest/
The all.zip file will download way faster than the build from GH Actions. (or you can use the web flasher, now even from Android ;) )

chromoxdor added 2 commits January 23, 2026 13:51

[LOG] add filter

e60b085

Merge branch 'letscontrolit:mega' into filter-weblog

a4fac00

chromoxdor changed the title ~~[LOG] Adding a Filter for the weblog~~ [LOG] Adding a filter for the weblog Jan 23, 2026

[LOG] refining the filter

6c5ce7e

- Added highlight filter phrases - Added ordered AND operator (&) - Added exclusion (!)

[LOG] revert changes for plugin_sets

c5f06cc

[LOG] add filter field for older builds

5701553

- Added a check to determine if there is already a filter field present; otherwise, add one for older builds.

tonhuisman reviewed Jan 25, 2026

View reviewed changes

tools/pio/pre_custom_esp32.py Show resolved Hide resolved

[LOG] reverting accidents

7c8a5ae

chromoxdor added 3 commits February 3, 2026 10:43

[LOG] autoscroll fix, code reduction

66f6a59

- Fixes a long-existing autoscroll issue where the whole site would scroll instead of just the log content. - reducing the code by simplifying the „browser check" and refining the loop

Merge branch 'mega' into filter-weblog

55a6d25

[LOG] Removed unnecessary code.

f05bd07

chromoxdor added 3 commits February 3, 2026 10:49

Merge branch 'filter-weblog' of https://github.com/chromoxdor/ESPEasy …

730c966

…into filter-weblog

[LOG+DevicePage_JS] refining the device page JS

7f14ce8

...while I am at it :)

[JS] update WebStaticData.h

2c7ce5e

TD-er merged commit f17f364 into letscontrolit:mega Feb 12, 2026

Conversation

chromoxdor commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chromoxdor commented Jan 23, 2026

Uh oh!

tonhuisman commented Jan 23, 2026

Uh oh!

chromoxdor commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chromoxdor commented Jan 23, 2026

Uh oh!

TD-er commented Jan 23, 2026

Uh oh!

chromoxdor commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonhuisman commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chromoxdor commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TD-er commented Jan 23, 2026

Uh oh!

TD-er commented Jan 23, 2026

Uh oh!

chromoxdor commented Jan 23, 2026

Uh oh!

tonhuisman commented Jan 23, 2026

Uh oh!

chromoxdor commented Jan 23, 2026

Uh oh!

TD-er commented Jan 23, 2026

Uh oh!

chromoxdor commented Jan 24, 2026

Uh oh!

TD-er commented Jan 24, 2026

Uh oh!

chromoxdor commented Jan 25, 2026

Uh oh!

TD-er commented Jan 25, 2026

Uh oh!

Uh oh!

chromoxdor commented Jan 25, 2026

Uh oh!

chromoxdor commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

chromoxdor commented Feb 12, 2026

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

chromoxdor commented Feb 12, 2026

Uh oh!

chromoxdor commented Feb 12, 2026

Uh oh!

chromoxdor commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uzi18 commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chromoxdor commented Feb 12, 2026

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

TD-er commented Feb 12, 2026

Uh oh!

uzi18 commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TD-er commented Feb 12, 2026

chromoxdor commented Jan 23, 2026 •

edited

Loading

chromoxdor commented Jan 23, 2026 •

edited

Loading

chromoxdor commented Jan 23, 2026 •

edited

Loading

tonhuisman commented Jan 23, 2026 •

edited

Loading

chromoxdor commented Jan 23, 2026 •

edited

Loading

chromoxdor commented Feb 5, 2026 •

edited

Loading

chromoxdor commented Feb 12, 2026 •

edited

Loading

uzi18 commented Feb 12, 2026 •

edited

Loading

uzi18 commented Feb 12, 2026 •

edited

Loading

uzi18 commented Feb 12, 2026 •

edited

Loading