summaryrefslogtreecommitdiff
path: root/src/ingestor.hpp
Commit message (Collapse)AuthorAge
* Start curl operations from any threadDan Goodliffe10 days
| | | | | Ingest is now background only, so don't limit where they're started from. Adds some unfortunate locking around the curl maps.
* Ingest log lines in a background threadDan Goodliffe10 days
| | | | This prevents halting reading input during data insertion.
* Rename jobIngestParkedLines to jobReadParkedLinesDan Goodliffe11 days
| | | | | Actual ingest is performed by the main process, jobReadParkedLines just reads the park file and adds it to the queue.
* Append unparked lines to queue in finalise functionDan Goodliffe11 days
| | | | Fixes issue where queuedLines would be accessed from background thread.
* Return a callable from jobsDan Goodliffe11 days
| | | | Allows safely running finalisation code in the main thread if required.
* Explicitly wait for and finalize any running jobs on exitDan Goodliffe11 days
|
* Limit the number lines stored at onceDan Goodliffe11 days
| | | | | | | Limits the number lines inserted per transactions, and the number of transactions before returning to reading input. Prevents long running transactions in the case when queued lines has grown in size.
* Replace use of crc32 for entity idDan Goodliffe2026-04-15
| | | | | Entity value is MD5 hashed same as DB unique key, but the id itself is now taken from the DB primary key which is sequence generated.
* Return path of parked lines log file from parkQueuedLogLinesDan Goodliffe2026-04-10
| | | | Or the last errno on failure.
* Revise stats and add signal handlers to log them and reset themwebstat-0.3Dan Goodliffe2026-03-25
| | | | Also logs them on main loop exit.
* Employ temporary/short files to handle errors reading/writing park logsDan Goodliffe2026-03-22
|
* Add missing -WshadowDan Goodliffe2026-03-22
|
* Add logging :-oDan Goodliffe2026-03-20
| | | | | | | Adds virtual log function, real implementation writes to syslog. Test implementation writes to BOOST_TEST_MESSAGE, perf implementation discards. Replaces existing prints to stderr and adds logs to all key points.
* Insert log entries in batchesDan Goodliffe2026-03-20
| | | | | | | | | | Store log lines in memory until threshold is reach or idle occurs, then insert all the lines in a single transaction. Save points handle the case of insertion errors. On success the queue is cleared. Parked lines also saved in bulk, only necessary if queued lines could not be inserted on shutdown, else the queue simply grows until ability to insert is restored. Importing parked lines just adds them to the queue and the normal process then follows.
* Gracefully handle SIGTERMDan Goodliffe2026-03-19
| | | | | Apache sends SIGTERM to the logger process to it shutdown. Honestly I thought it would just close stdin and I should have checked.
* Use std::future over std::thread for background jobsDan Goodliffe2026-03-17
| | | | | Easier checking if a job has completed [successfully] and reseting state for the next time.
* Don't start new curl operations outside the main threadDan Goodliffe2026-03-17
| | | | | Neither the curl handle, not the operation map is thread safe. This isn't ideal, but it does solve the problem in a safe manor.
* Execute jobs even when processing incoming logsDan Goodliffe2026-03-17
| | | | | Jobs run on background threads now, so we can happily run them even when we're busy.
* Run jobs on a background threadDan Goodliffe2026-03-17
|
* Process new field, content-type, in input streamDan Goodliffe2026-01-18
|
* Add job for puring old access log entries from the databaseDan Goodliffe2025-12-20
|
* Refactor handling of new entity insertDan Goodliffe2025-10-15
| | | | | Replaces weird select with one thing with a function pointer stored in the type definition array.
* Fix premature remembering of saved entity idswebstat-0.2.1Dan Goodliffe2025-10-09
| | | | | | | Don't persist entity ids saved to the DB until the transaction is committed. Prevents the issue where a later DB operation fails, the transaction is rolled back, but we still think the entity has been saved.
* Add parked line import jobwebstat-0.2Dan Goodliffe2025-10-06
| | | | Periodically, on idle, scan for and import previously parked lines.
* Add point to execute scheduled jobs when idleDan Goodliffe2025-10-02
|
* Write log lines to files on errorDan Goodliffe2025-09-30
| | | | | We call this parking, later we can reattempt ingestion after whatever caused the failure has been fixed.
* Create settings structureDan Goodliffe2025-09-24
| | | | | | | Holds all the settings and their defaults for use in program_options and tests. Disables missing-field-initializers in tests because its over sensitive to structures with defaults where you only provide some values specifically.
* Write unparsable lines to the entity tableDan Goodliffe2025-09-23
| | | | Diagnostics and the ability to ingest later.
* Make DB pool protected for access from unit testsDan Goodliffe2025-09-23
|
* Create and perform UA lookup curl op when new user agent is encounteredDan Goodliffe2025-09-13
|
* Perform background curl operations when not processing log inputDan Goodliffe2025-09-13
|
* Always pass API URL to curlGetUserAgentDetailDan Goodliffe2025-09-13
|
* Use curl_multi_poll in main ingestLog loopDan Goodliffe2025-09-10
| | | | Preparation step for having background curl operations.
* Store utsname details in the host's detail fieldDan Goodliffe2025-09-02
|
* Use a DB connection pool rather than a single connectionDan Goodliffe2025-09-01
|
* Insert new entities and log entry in a transactionDan Goodliffe2025-08-25
| | | | If no new entities are required, no transaction is created.
* Merge storeEntities into a single thingDan Goodliffe2025-08-25
| | | | Add visitSum for tuples and makes storeEntities into a single lambda.
* Track existing entities, don't attempt reinsertDan Goodliffe2025-08-25
| | | | | This is an ever growing set, but in theory should be finite, and is only 4bytes per entity.
* Return count of entities passed to the databaseDan Goodliffe2025-08-25
|
* Store log lines in full using entity idsDan Goodliffe2025-08-25
|
* Store the entities for each log lineDan Goodliffe2025-08-25
|
* Provide hostname to IngestorDan Goodliffe2025-08-25
| | | | Store its Entity immediately for later use.
* Add helpers and SQL for storing an entityDan Goodliffe2025-08-25
|
* Pass a DB connection to IngestorDan Goodliffe2025-08-25
|
* Setup moveable/non-copyable special members on IngestorDan Goodliffe2025-08-25
|
* Add helpers for hashing values extracted from log inputDan Goodliffe2025-08-25
| | | | | zlib's crc32 used here, the interface is a bit C and as a result a bit casty, but it'll work.
* Initial commit; basic Apache log parsingDan Goodliffe2025-08-25
generated by cgit v1.3 (git 2.53.0) at 2026-05-12 15:40:25 +0000