WebStat analyser/src/sql, branch main WebStat web site access analyser http://git.randomdan.homeip.net/repo/webstat/atom?h=main 2026-06-08T22:36:01Z Select uninsertable is for update 2026-06-08T22:36:01Z Dan Goodliffe dan@randomdan.homeip.net 2026-06-08T22:36:01Z urn:sha1:856527b65e3c4807cf2bfb9d0e495d9712d56a90 This prevents other instances acting on the same rows in parallel, potentially duplicating the rows and effort. Don't order rows when purging 2026-06-08T10:27:49Z Dan Goodliffe dan.goodliffe@octal.co.uk 2026-06-08T10:27:49Z urn:sha1:aa8f64a3e140e74a1f8d0074b606dd1f8560ad0a The need to purge rows in request_time order is not necessary (all rows will be deleted at some point in the process anyway) and is otherwise extremely expensive. Removing the order by clause is a roughly 300x speed up. Run the retry uninsertable process batched 2026-05-18T23:36:50Z Dan Goodliffe dan@randomdan.homeip.net 2026-05-18T23:36:50Z urn:sha1:b298237a297cb70640c85c016b702a35ecafd9b9 Standard sized batch in a transaction, ordered by entity id. Includes early exit if terminated. Add job to retry insertion of log lines which had previously failed 2026-05-18T19:43:51Z Dan Goodliffe dan@randomdan.homeip.net 2026-05-18T19:43:51Z urn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187 Entities are reparsed and reinserted, removed on success. Failure to parse updates the entity type to UnparsableLine. Failure to insert again updates the detail with the reason. Only call entity insert handler if detail is null 2026-05-05T11:24:11Z Dan Goodliffe dan.goodliffe@octal.co.uk 2026-05-05T11:24:11Z urn:sha1:200c3636b600ac2f997316612f1b1321112496cd Improves handling of entity inserts where the entity already exists and already has detail; does not call the onInsert handler. This avoids repeatedly fetching UA detail every time the UA is first seen by a process. Add BRIN index to access_log.request_time and improve purge 2026-04-23T09:11:13Z Dan Goodliffe dan.goodliffe@octal.co.uk 2026-04-23T09:11:13Z urn:sha1:7c59f2b66284b5b561c5368a0c834640ae0e611e Purge is now fully request_time based and not hacked around id ranges. Replace use of crc32 for entity id 2026-04-15T11:05:20Z Dan Goodliffe dan.goodliffe@octal.co.uk 2026-04-15T11:03:21Z urn:sha1:5b2166496e5f3ff2c4276e0b5b28f109c70673d5 Entity value is MD5 hashed same as DB unique key, but the id itself is now taken from the DB primary key which is sequence generated. Process new field, content-type, in input stream 2026-01-18T01:36:06Z Dan Goodliffe dan@randomdan.homeip.net 2026-01-18T01:36:06Z urn:sha1:04acfa679fd846ac829ded5562093b3766c85154 pg_format schema.sql and sql/*.sql 2026-01-17T20:33:37Z Dan Goodliffe dan@randomdan.homeip.net 2026-01-17T18:33:56Z urn:sha1:9f1dfacb0f236da5f0413ac20e3376388df7d798 No changes. Add job for puring old access log entries from the database 2025-12-20T15:16:25Z Dan Goodliffe dan@randomdan.homeip.net 2025-12-20T15:16:25Z urn:sha1:30a9d45f5322a63c2adf1de7a248ba3a9a0c3903