Troubleshooting Guide
Overview
This page looks to serve as a resource in case you encounter unusual behavior from flux-accounting and any of its components. The format of the guide follows that of a Frequently Asked Questions (FAQ) page, so if you have a suggestion on a question you'd like to see answered, please consider opening an issue on Github.
My job is held with a flux-accounting dependency. How can I figure out why it's held?
For details on how to interpret flux-accounting dependencies and how they are resolved, please see the Limits page.
The job-usage values reported for some associations and banks seem inconsistent and/or inaccurate. How can I resolve this?
If it seems like association or bank job usage values don't look correct, the
first thing to check is the consistency of the association's userid in
association_table, job_usage_factor_table, and, in periodic decay mode,
job_usage_per_association_table. A mismatch in job_usage_factor_table
can prevent completed jobs from being included in usage updates, since jobs
are found by userid and bank.
Run flux account sync-userids to copy each association's userid into its
existing usage-factor records and periodic usage bins. The command matches
rows by username and bank, and repairs either table even when the other table
already agrees. Usage values, periods, and job timestamps are preserved. For
example, the repair query for periodic bins is:
UPDATE job_usage_per_association_table
SET userid = (
SELECT a.userid FROM association_table a
WHERE a.username = job_usage_per_association_table.username
AND a.bank = job_usage_per_association_table.bank
)
WHERE EXISTS (
SELECT 1 FROM association_table a
WHERE a.username = job_usage_per_association_table.username
AND a.bank = job_usage_per_association_table.bank
AND a.userid != job_usage_per_association_table.userid
);
The usage-factor repair uses the same query with job_usage_factor_table
in place of job_usage_per_association_table. Rows with no matching
association are left unchanged.
Changing a userid with edit-user --userid also updates the userid in the
selected associations' existing usage-factor records and periodic usage bins,
so sync-userids is only needed to repair tables that are already
inconsistent.
Note
This inconsistency between user IDs in the association_table and
job_usage_factor_table was discovered in flux-accounting versions
prior to v0.50.0 and should be fixed in versions v0.50.0 or later.
If you are running flux-accounting v0.50.0 (or, more specifically, adding
associations to the association_table) or later and are still
running into this bug, please open an issue on GitHub.
I am getting a "Database is locked" error when trying to read/write from the flux-accounting DB. How can I resolve this?
SQLite uses file-level locking to ensure data integrity when multiple processes access the database simultaneously. A "Database is locked" error usually occurs when:
write lock contention: Another process holds an exclusive write lock on the database file, preventing the operation from proceeding. SQLite allows multiple concurrent readers, but only one writer at a time.
transaction timeout: A process started a transaction (with BEGIN) but hasn't committed or rolled back yet, keeping the lock active longer than expected.
stale connections: Processes that crashed or were terminated without properly closing their database connections can leave locks in place.
NFS or networked filesystems: SQLite's locking mechanism may not work reliably on some network filesystems, leading to lock conflicts.
While the scripts that update job usage and fair-share values for the
flux-accounting database should be quick, a transaction occurring during this
update could cause the database to enter a locked state. To resolve this, try
restarting the flux-accounting service with
systemctl restart flux-accounting.