WebStat analyser/src/ingestor.hpp, branch webstat-0.6WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.62026-06-08T11:31:07ZRemove the purge time limit2026-06-08T11:31:07ZDan Goodliffedan.goodliffe@octal.co.uk2026-06-08T11:31:07Zurn:sha1:ee9517e452376bc4963aa3a43ca90ebb9d710b85
This is a background job, has been for a while, so we can sanely allow
it to run for as long as is required.
Add job to retry insertion of log lines which had previously failed2026-05-18T19:43:51ZDan Goodliffedan@randomdan.homeip.net2026-05-18T19:43:51Zurn:sha1:e1a6654bd5e284842ffbc3b93bd390f3bad7a187
Entities are reparsed and reinserted, removed on success.
Failure to parse updates the entity type to UnparsableLine.
Failure to insert again updates the detail with the reason.
Add ThreadSafeT helper2026-05-17T15:18:08ZDan Goodliffedan@randomdan.homeip.net2026-05-17T13:32:11Zurn:sha1:7b1aeee4565fe0a2eed4a4fa8695b2a5fb671e06
Wraps a templated value and a templated mutex (defaults to shared_mutex)
and provides safe access, locked with either a shared_lock (const value)
or lock_guard (non-const value).
Applies this to existingEntities.
Remove maxBatches2026-05-13T14:01:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:01:30Zurn:sha1:868446ad4d8ba33053b048dd4d82b6d4f5328f07
It's not required; DB insertion occurs only in a background thread now.
Handle completed curl operations in a job2026-05-07T14:41:19ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-07T14:41:19Zurn:sha1:157614349f5f4c2387e0a970a4ca3ddaeaa23446
Removes the need to block the main thread from reading stdin while
performing post curl operation actions, such as updating user agent
details.
Add support for conditional job execution2026-05-07T14:40:11ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-07T14:40:11Zurn:sha1:cbb2035fd33ba84fea56c7a7223c563b925e8649
Performs a check before launching a job thread, rather than just having
it exit immediately.
Switch to std::map for existingEntities cache2026-05-05T09:00:45ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-05T09:00:45Zurn:sha1:b7346649926b4357b27aa81238eb34cebe39671b
Insertion cost of flat_map too high when map grows large.
Start curl operations from any thread2026-05-02T22:13:08ZDan Goodliffedan@randomdan.homeip.net2026-05-02T22:13:08Zurn:sha1:af422b45c7fb406dae35dc3f811e50d92e854f58
Ingest is now background only, so don't limit where they're started
from. Adds some unfortunate locking around the curl maps.
Ingest log lines in a background thread2026-05-02T17:58:21ZDan Goodliffedan@randomdan.homeip.net2026-05-02T17:58:21Zurn:sha1:4675ab65ea5e807e0d457845a0ca84edcf1262c9
This prevents halting reading input during data insertion.
Rename jobIngestParkedLines to jobReadParkedLines2026-05-01T16:13:29ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-01T16:13:29Zurn:sha1:5c0206f48dc7f90009629d0a74bdc1dd6b4f67ea
Actual ingest is performed by the main process, jobReadParkedLines just
reads the park file and adds it to the queue.