WebStat analyser/src/ingestor.hpp, branch webstat-0.5.1WebStat web site access analyserhttp://git.randomdan.homeip.net/repo/webstat/atom?h=webstat-0.5.12026-05-13T14:01:30ZRemove maxBatches2026-05-13T14:01:30ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-13T14:01:30Zurn:sha1:868446ad4d8ba33053b048dd4d82b6d4f5328f07
It's not required; DB insertion occurs only in a background thread now.
Handle completed curl operations in a job2026-05-07T14:41:19ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-07T14:41:19Zurn:sha1:157614349f5f4c2387e0a970a4ca3ddaeaa23446
Removes the need to block the main thread from reading stdin while
performing post curl operation actions, such as updating user agent
details.
Add support for conditional job execution2026-05-07T14:40:11ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-07T14:40:11Zurn:sha1:cbb2035fd33ba84fea56c7a7223c563b925e8649
Performs a check before launching a job thread, rather than just having
it exit immediately.
Switch to std::map for existingEntities cache2026-05-05T09:00:45ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-05T09:00:45Zurn:sha1:b7346649926b4357b27aa81238eb34cebe39671b
Insertion cost of flat_map too high when map grows large.
Start curl operations from any thread2026-05-02T22:13:08ZDan Goodliffedan@randomdan.homeip.net2026-05-02T22:13:08Zurn:sha1:af422b45c7fb406dae35dc3f811e50d92e854f58
Ingest is now background only, so don't limit where they're started
from. Adds some unfortunate locking around the curl maps.
Ingest log lines in a background thread2026-05-02T17:58:21ZDan Goodliffedan@randomdan.homeip.net2026-05-02T17:58:21Zurn:sha1:4675ab65ea5e807e0d457845a0ca84edcf1262c9
This prevents halting reading input during data insertion.
Rename jobIngestParkedLines to jobReadParkedLines2026-05-01T16:13:29ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-01T16:13:29Zurn:sha1:5c0206f48dc7f90009629d0a74bdc1dd6b4f67ea
Actual ingest is performed by the main process, jobReadParkedLines just
reads the park file and adds it to the queue.
Append unparked lines to queue in finalise function2026-05-01T16:01:56ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-01T15:58:01Zurn:sha1:0e768a00fab74818533458b5b40c84556b2137e0
Fixes issue where queuedLines would be accessed from background thread.
Return a callable from jobs2026-05-01T14:31:02ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-01T14:31:02Zurn:sha1:ed3598ac6f77f62b0c4fe32d4dab05784e5fa51d
Allows safely running finalisation code in the main thread if required.
Explicitly wait for and finalize any running jobs on exit2026-05-01T10:12:39ZDan Goodliffedan.goodliffe@octal.co.uk2026-05-01T10:12:39Zurn:sha1:917487a6de80bf5e81eebcdbbf48bcf9763262fa