WebStat analyser/src/ingestor.cpp, branch webstat-0.7WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.72026-07-05T16:12:53ZUpdate parsed type of duration field2026-07-05T16:12:53ZDan Goodliffedan@randomdan.homeip.net2026-07-05T16:12:53Zurn:sha1:cb47664b464331b88dad62796729fe08f07f30e4
uint32_t doesn't cut it, changed to uint64_t.
Insert uninsirtable lines using the hostnameId recorded in the bad_lines table2026-07-04T19:19:34ZDan Goodliffedan@randomdan.homeip.net2026-07-04T19:19:34Zurn:sha1:870aa90c005f77a73445f5c39f9fd9bbb7604fe8
Includes the same fix from the previous commit which stops caching
entities until the inserts are complete.
Updates storeLogLine to take an insert command with the hostnameId
already bound.
Include extra details when storing unparsable and uninsertable lines.2026-07-04T18:36:30ZDan Goodliffedan@randomdan.homeip.net2026-07-04T11:11:38Zurn:sha1:61e106fe6a15f76b73125682e91d5c17ee875be2
Now includes the hostnameId, timestamp and optionally error message.
Updates idx_entities_retryinsert to account for details not being NULL.
Also fixes the issue where the entities inserted and rollback would
still be recorded in the entity cache if the access log insert failed.
This would have lead to PK violations later on.
Increase size field to 64bit and fix scan types2026-06-27T12:05:03ZDan Goodliffedan@randomdan.homeip.net2026-06-27T12:05:03Zurn:sha1:22a85bd4e1f6005455b91bb3edff68d4019526a1
2^31 not big enough for large file downloads, bumped to 64bit.
Replace woolly types with specific uintXX_t in scn call.
Remove the purge time limit2026-06-08T11:31:07ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-08T11:31:07Zurn:sha1:ee9517e452376bc4963aa3a43ca90ebb9d710b85
This is a background job, has been for a while, so we can sanely allow
it to run for as long as is required.
Switch to std::fstream and std::print for parkLogLines2026-05-20T16:04:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-20T16:03:23Zurn:sha1:bca5ac479de5721e354b70d627fb9654bd94bdeeChange bytesToHexRange into a transform view variable2026-05-20T11:22:27ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-20T11:22:27Zurn:sha1:161f32ebaea5988fe97ad03ad669eca2222e2a54
Simply create the transform constexpr and then apply it to any
collection with the restriction of passing it to a function.
Run the retry uninsertable process batched2026-05-18T23:36:50ZDan Goodliffedan@randomdan.homeip.net2026-05-18T23:36:50Zurn:sha1:b298237a297cb70640c85c016b702a35ecafd9b9
Standard sized batch in a transaction, ordered by entity id.
Includes early exit if terminated.
Add job to retry insertion of log lines which had previously failed2026-05-18T19:43:51ZDan Goodliffedan@randomdan.homeip.net2026-05-18T19:43:51Zurn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187
Entities are reparsed and reinserted, removed on success.
Failure to parse updates the entity type to UnparsableLine.
Failure to insert again updates the detail with the reason.
Extract ENTITY_IDS helper2026-05-17T16:45:53ZDan Goodliffedan@randomdan.homeip.net2026-05-17T16:45:53Zurn:sha1:e8a0fb7b9c61d5603a86b52e0f2144a91fd1d84e
Transform view for getting just the hash and record of a stored entity.