Manage log volume

Execution monitoring keeps every run until you delete it. On a busy server that adds up, mostly in log_detail, which holds the log text of every workflow, pipeline, action and transform that produced any.

See what is using the space

SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
WHERE relname IN ('log_object', 'log_detail');

Keep log text for runs only

To keep the drill-through from a run to its log, but stop storing log text per action and transform, open logging/pipeline-log.hpl and logging/workflow-log.hpl in the project and change the filter in the Has log detail transform to also require type IN ('PIPELINE','WORKFLOW').

Re-running install-logging.sh replaces both files with the shipped version. Repeat the change after each upgrade, or install with --no-clobber to keep your copies.

Delete old runs

Deleting from log_object also deletes the matching log_detail rows. For example, to keep 90 days:

DELETE FROM log_object WHERE start_date < now() - interval '90 days';

Schedule it with whatever you already use: a workflow with an SQL action on the logging connection works well and is recorded on the dashboard like any other run. After a large first delete, run VACUUM on both tables so PostgreSQL reuses the space. VACUUM FULL also returns it to the disk, but locks the tables while it runs, so Hop cannot log in the meantime.