WebStat analyser/src/ingestor.hpp, branch webstat-0.7.1WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.7.12026-07-09T12:09:34ZInclude type in existingEntities cache2026-07-09T12:09:34ZDan Goodliffedan.goodliffe@octal.co.uk2026-07-09T12:09:34Zurn:sha1:fa277f69b14446b47a3dfa63ed7fc1a8a16ef7f8
Fixes the issues of entities having the same value (hash) but occurring
in different places. This is most often when the client states the
virtual host name as the referrer.
Update parsed type of duration field2026-07-05T16:12:53ZDan Goodliffedan@randomdan.homeip.net2026-07-05T16:12:53Zurn:sha1:cb47664b464331b88dad62796729fe08f07f30e4
uint32_t doesn't cut it, changed to uint64_t.
Insert uninsirtable lines using the hostnameId recorded in the bad_lines table2026-07-04T19:19:34ZDan Goodliffedan@randomdan.homeip.net2026-07-04T19:19:34Zurn:sha1:870aa90c005f77a73445f5c39f9fd9bbb7604fe8
Includes the same fix from the previous commit which stops caching
entities until the inserts are complete.
Updates storeLogLine to take an insert command with the hostnameId
already bound.
Include extra details when storing unparsable and uninsertable lines.2026-07-04T18:36:30ZDan Goodliffedan@randomdan.homeip.net2026-07-04T11:11:38Zurn:sha1:61e106fe6a15f76b73125682e91d5c17ee875be2
Now includes the hostnameId, timestamp and optionally error message.
Updates idx_entities_retryinsert to account for details not being NULL.
Also fixes the issue where the entities inserted and rollback would
still be recorded in the entity cache if the access log insert failed.
This would have lead to PK violations later on.
Increase size field to 64bit and fix scan types2026-06-27T12:05:03ZDan Goodliffedan@randomdan.homeip.net2026-06-27T12:05:03Zurn:sha1:22a85bd4e1f6005455b91bb3edff68d4019526a1
2^31 not big enough for large file downloads, bumped to 64bit.
Replace woolly types with specific uintXX_t in scn call.
Remove the purge time limit2026-06-08T11:31:07ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-08T11:31:07Zurn:sha1:ee9517e452376bc4963aa3a43ca90ebb9d710b85
This is a background job, has been for a while, so we can sanely allow
it to run for as long as is required.
Add job to retry insertion of log lines which had previously failed2026-05-18T19:43:51ZDan Goodliffedan@randomdan.homeip.net2026-05-18T19:43:51Zurn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187
Entities are reparsed and reinserted, removed on success.
Failure to parse updates the entity type to UnparsableLine.
Failure to insert again updates the detail with the reason.
Add ThreadSafeT helper2026-05-17T15:18:08ZDan Goodliffedan@randomdan.homeip.net2026-05-17T13:32:11Zurn:sha1:7b1aeee4565fe0a2eed4a4fa8695b2a5fb671e06
Wraps a templated value and a templated mutex (defaults to shared_mutex)
and provides safe access, locked with either a shared_lock (const value)
or lock_guard (non-const value).
Applies this to existingEntities.
Remove maxBatches2026-05-13T14:01:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:01:30Zurn:sha1:868446ad4d8ba33053b048dd4d82b6d4f5328f07
It's not required; DB insertion occurs only in a background thread now.
Handle completed curl operations in a job2026-05-07T14:41:19ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-07T14:41:19Zurn:sha1:157614349f5f4c2387e0a970a4ca3ddaeaa23446
Removes the need to block the main thread from reading stdin while
performing post curl operation actions, such as updating user agent
details.