WebStat analyser/src/sql, branch webstat-0.7.1WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.7.12026-07-04T19:19:34ZInsert uninsirtable lines using the hostnameId recorded in the bad_lines table2026-07-04T19:19:34ZDan Goodliffedan@randomdan.homeip.net2026-07-04T19:19:34Zurn:sha1:870aa90c005f77a73445f5c39f9fd9bbb7604fe8
Includes the same fix from the previous commit which stops caching
entities until the inserts are complete.
Updates storeLogLine to take an insert command with the hostnameId
already bound.
Include extra details when storing unparsable and uninsertable lines.2026-07-04T18:36:30ZDan Goodliffedan@randomdan.homeip.net2026-07-04T11:11:38Zurn:sha1:61e106fe6a15f76b73125682e91d5c17ee875be2
Now includes the hostnameId, timestamp and optionally error message.
Updates idx_entities_retryinsert to account for details not being NULL.
Also fixes the issue where the entities inserted and rollback would
still be recorded in the entity cache if the access log insert failed.
This would have lead to PK violations later on.
Fix uniqueness of entities2026-06-29T08:33:57ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-29T08:33:57Zurn:sha1:26eda4262fa99888e254e9c19b40c04b2a29a75c
Should be unique for value and type, they're not just strings, they can
be different things with the same value... Albeit rarely.
Select uninsertable is for update2026-06-08T22:36:01ZDan Goodliffedan@randomdan.homeip.net2026-06-08T22:36:01Zurn:sha1:856527b65e3c4807cf2bfb9d0e495d9712d56a90
This prevents other instances acting on the same rows in parallel,
potentially duplicating the rows and effort.
Don't order rows when purging2026-06-08T10:27:49ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-08T10:27:49Zurn:sha1:aa8f64a3e140e74a1f8d0074b606dd1f8560ad0a
The need to purge rows in request_time order is not necessary (all rows
will be deleted at some point in the process anyway) and is otherwise
extremely expensive. Removing the order by clause is a roughly 300x
speed up.
Run the retry uninsertable process batched2026-05-18T23:36:50ZDan Goodliffedan@randomdan.homeip.net2026-05-18T23:36:50Zurn:sha1:b298237a297cb70640c85c016b702a35ecafd9b9
Standard sized batch in a transaction, ordered by entity id.
Includes early exit if terminated.
Add job to retry insertion of log lines which had previously failed2026-05-18T19:43:51ZDan Goodliffedan@randomdan.homeip.net2026-05-18T19:43:51Zurn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187
Entities are reparsed and reinserted, removed on success.
Failure to parse updates the entity type to UnparsableLine.
Failure to insert again updates the detail with the reason.
Only call entity insert handler if detail is null2026-05-05T11:24:11ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-05T11:24:11Zurn:sha1:200c3636b600ac2f997316612f1b1321112496cd
Improves handling of entity inserts where the entity already exists and
already has detail; does not call the onInsert handler. This avoids
repeatedly fetching UA detail every time the UA is first seen by a
process.
Add BRIN index to access_log.request_time and improve purge2026-04-23T09:11:13ZDan Goodliffedan.goodliffe@octal.co.uk2026-04-23T09:11:13Zurn:sha1:7c59f2b66284b5b561c5368a0c834640ae0e611e
Purge is now fully request_time based and not hacked around id ranges.
Replace use of crc32 for entity id2026-04-15T11:05:20ZDan Goodliffedan.goodliffe@octal.co.uk2026-04-15T11:03:21Zurn:sha1:5b2166496e5f3ff2c4276e0b5b28f109c70673d5
Entity value is MD5 hashed same as DB unique key, but the id itself is
now taken from the DB primary key which is sequence generated.