WebStat analyser, branch mainWebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=main2026-07-09T12:09:34ZInclude type in existingEntities cache2026-07-09T12:09:34ZDan Goodliffedan.goodliffe@octal.co.uk2026-07-09T12:09:34Zurn:sha1:fa277f69b14446b47a3dfa63ed7fc1a8a16ef7f8
Fixes the issues of entities having the same value (hash) but occurring
in different places. This is most often when the client states the
virtual host name as the referrer.
Add a const for the number of entities inserted so far2026-07-08T17:15:13ZDan Goodliffedan.goodliffe@octal.co.uk2026-07-08T17:15:13Zurn:sha1:f8d326a7d6f49a6b6009a7869ce61d782ad9c7b6
Eases updating early tests without the need to change all the later
expected id values.
Fix compilation of test-ingestor with gcc-162026-07-06T18:44:52ZDan Goodliffedan@randomdan.homeip.net2026-07-06T18:44:52Zurn:sha1:1d035cfa7b4b477c11267f15b2e6820466c004ebAdd option --job2026-07-05T17:59:48ZDan Goodliffedan@randomdan.homeip.net2026-07-05T17:59:48Zurn:sha1:4df453867abca0026f2e51f1cf7d927542c5c11c
Allows running with retry-uninsertable or purge-old-logs from the command line.
Update parsed type of duration field2026-07-05T16:12:53ZDan Goodliffedan@randomdan.homeip.net2026-07-05T16:12:53Zurn:sha1:cb47664b464331b88dad62796729fe08f07f30e4
uint32_t doesn't cut it, changed to uint64_t.
Insert uninsirtable lines using the hostnameId recorded in the bad_lines table2026-07-04T19:19:34ZDan Goodliffedan@randomdan.homeip.net2026-07-04T19:19:34Zurn:sha1:870aa90c005f77a73445f5c39f9fd9bbb7604fe8
Includes the same fix from the previous commit which stops caching
entities until the inserts are complete.
Updates storeLogLine to take an insert command with the hostnameId
already bound.
Include extra details when storing unparsable and uninsertable lines.2026-07-04T18:36:30ZDan Goodliffedan@randomdan.homeip.net2026-07-04T11:11:38Zurn:sha1:61e106fe6a15f76b73125682e91d5c17ee875be2
Now includes the hostnameId, timestamp and optionally error message.
Updates idx_entities_retryinsert to account for details not being NULL.
Also fixes the issue where the entities inserted and rollback would
still be recorded in the entity cache if the access log insert failed.
This would have lead to PK violations later on.
Replace use of CAUSEPARSEFAIL in sample strings2026-07-02T21:45:59ZDan Goodliffedan@randomdan.homeip.net2026-07-02T21:45:59Zurn:sha1:409e41d1d97b180233be778a2d23ba0b4328b5c2
Use of that as a http_verb doesn't actually cause parse failure (whether
it should or not is another matter)
So it's replaced with CAUSEINSERTFAIL in one case (it does cause this)
and in the parse failure test the whole line is ripped out.
Fixes to parseEscapedString2026-07-02T14:46:25ZDan Goodliffedan.goodliffe@octal.co.uk2026-07-02T14:46:25Zurn:sha1:fbb5038105e26e07d34184544ea4c70892acef92
Fix the warnings picked up by the latest clang-tidy.
Add test cases lifted from the bad_lines log.
Add missing support for \b in escaping.
Partition the entities table2026-06-30T15:49:25ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-30T15:49:25Zurn:sha1:3cc7a1b1c0bf84ffcb149741c9ca2c60d9f00068
Splits up the entities by type, updates the access_log foreign keys to
reference the specific table, removes now redundant indices which
existed only for processing removal of bad lines.