WebStat analyser/src/ingestor.cpp, branch webstat-0.6WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.62026-06-08T11:31:07ZRemove the purge time limit2026-06-08T11:31:07ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-08T11:31:07Zurn:sha1:ee9517e452376bc4963aa3a43ca90ebb9d710b85
This is a background job, has been for a while, so we can sanely allow
it to run for as long as is required.
Switch to std::fstream and std::print for parkLogLines2026-05-20T16:04:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-20T16:03:23Zurn:sha1:bca5ac479de5721e354b70d627fb9654bd94bdeeChange bytesToHexRange into a transform view variable2026-05-20T11:22:27ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-20T11:22:27Zurn:sha1:161f32ebaea5988fe97ad03ad669eca2222e2a54
Simply create the transform constexpr and then apply it to any
collection with the restriction of passing it to a function.
Run the retry uninsertable process batched2026-05-18T23:36:50ZDan Goodliffedan@randomdan.homeip.net2026-05-18T23:36:50Zurn:sha1:b298237a297cb70640c85c016b702a35ecafd9b9
Standard sized batch in a transaction, ordered by entity id.
Includes early exit if terminated.
Add job to retry insertion of log lines which had previously failed2026-05-18T19:43:51ZDan Goodliffedan@randomdan.homeip.net2026-05-18T19:43:51Zurn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187
Entities are reparsed and reinserted, removed on success.
Failure to parse updates the entity type to UnparsableLine.
Failure to insert again updates the detail with the reason.
Extract ENTITY_IDS helper2026-05-17T16:45:53ZDan Goodliffedan@randomdan.homeip.net2026-05-17T16:45:53Zurn:sha1:e8a0fb7b9c61d5603a86b52e0f2144a91fd1d84e
Transform view for getting just the hash and record of a stored entity.
Add ThreadSafeT helper2026-05-17T15:18:08ZDan Goodliffedan@randomdan.homeip.net2026-05-17T13:32:11Zurn:sha1:7b1aeee4565fe0a2eed4a4fa8695b2a5fb671e06
Wraps a templated value and a templated mutex (defaults to shared_mutex)
and provides safe access, locked with either a shared_lock (const value)
or lock_guard (non-const value).
Applies this to existingEntities.
Check terminated flag in jobPurgeOldLogs2026-05-13T14:49:10ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:49:10Zurn:sha1:29f458117184af5b1507cac01b48b41bfbad568a
It's not critical to run to completion during shutdown, we can pick up
where we left off on the next run. This will allow us to bail out
instead of holding up the shutdown process.
Fix checking of background job completeness2026-05-13T14:21:06ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:21:06Zurn:sha1:3e7a1ce999b2223b321573de19cef840021f77f5
OK, this is on me. future::valid does not tell you if the thread has
completed and the result is ready. It tells you if there is some state
you can get() maybe later. Here we replace those checks with a 0s wait
and a test to see if the status is ready.
The exit steps are updated to reflect this. Calling reset will invoke
the future's destructor, which blocks until the thread is joined as
needed.
This should hopefully address the issue where the main thread would
still block if it attempted to runJobAsNeeded for a job that was still
running.
Remove maxBatches2026-05-13T14:01:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:01:30Zurn:sha1:868446ad4d8ba33053b048dd4d82b6d4f5328f07
It's not required; DB insertion occurs only in a background thread now.