[checkmk-commits] 5538 Improved performance when processing a large amount of piggyback data

Module: check_mk Branch: master Commit: 50dbfd5d08f38bed159a001480c2c9b9c3ed83fc URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=50dbfd5d08f38b… Author: Lars Michelsen &lt;lm(a)mathias-kettner.de&gt; Date: Wed Nov 22 15:56:27 2017 +0100 5538 Improved performance when processing a large amount of piggyback data When Check_MK needs to handle a large amount of piggyback data (a lot of piggbacked hosts from a lot of piggyback source hosts, several hundreds to thousands), the performance of Check_MK could decrease during regular monitoring. This was caused by some too expensive house keeping logic that was executed too often. The mechanism has now been changed to work like this: <ul> <li>During regular monitoring now piggyback data is removed anymore from the disk.</li> <li>New piggyback data is written to disk when communicating with the source host.</li> <li>When monitoring piggybacked hosts, the outdated piggyback data available on the disk is filtered.</li> <li>There is a dedicated housekeeping cron job executed sites crontab daily at 00:10 which removes outdated piggyback data. This job is mostly used to free up some tmpfs space, the outated stored data is not read by monitoring anymore.</li> </ul> Change-Id: I22d9610a00e9200ffc69cfb8c9edf4292d613581 --- .werks/5538 | 25 +++ cmk_base/data_sources/__init__.py | 2 +- cmk_base/data_sources/abstract.py | 8 +- cmk_base/data_sources/host_sections.py | 20 +- cmk_base/modes/check_mk.py | 24 ++- cmk_base/piggyback.py | 208 ++++++++++++++++----- .../check_mk/skel/etc/cron.d/cmk_cleanup_piggyback | 2 + 7 files changed, 230 insertions(+), 59 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=50dbfd5d08…