[checkmk-commits] 3531 Reduced Check_MK helper size for certain dists/python versions

Module: check_mk Branch: master Commit: 17ae8d394d39a4a54a129b7035acbedc29553001 URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=17ae8d394d39a4… Author: Andreas Boesl &lt;ab(a)mathias-kettner.de&gt; Date: Tue May 17 10:37:51 2016 +0200 3531 Reduced Check_MK helper size for certain dists/python versions The config of the Check_MK check helper is now written as marshaled python code, instead of plain python text. This reduces the helpers peak resident memory size while parsing the config file. Some distributions (or python versions) tend not to shrink the resident memory of the check helper again, so the check helper size can only grow. By using marshaled config data the initial check helper size can be drastically lower. Actual use case example: System setup: <ul> <li>CentOS6</li> <li>Python 2.6</li> <li>11k Hosts</li> <li>40k Services</li> </ul> The size of the config is roughly 8MB (this highly depends on the configured rules). <table> <tr><td>Marshal config</td><td>320MB</td></tr> <tr><td>Plain text config</td><td>30MB</td></tr> </table> --- .werks/3531 | 34 ++++++++++++++++++++++++++++++++++ ChangeLog | 1 + modules/check_mk.py | 19 ++++++++++++++----- 3 files changed, 49 insertions(+), 5 deletions(-) diff --git a/.werks/3531 b/.werks/3531 new file mode 100644 index 0000000..039d728 --- /dev/null +++ b/.werks/3531 @@ -0,0 +1,34 @@ +Title: Reduced Check_MK helper size for certain dists/python versions +Level: 1 +Component: core +Compatible: compat +Version: 1.2.9i1 +Date: 1463473106 +Class: feature + +The config of the Check_MK check helper is now written as marshaled python code, +instead of plain python text. This reduces the helpers peak resident memory size +while parsing the config file. + + +Some distributions (or python versions) tend not to shrink the resident memory +of the check helper again, so the check helper size can only grow. + + +By using marshaled config data the initial check helper size can be drastically lower. +Actual use case: + +System setup +<ul> +<li>CentOS6</li> +<li>Python 2.6</li> +<li>11k Hosts</li> +<li>40k Services</li> +</ul> +The size of the config is roughly 8MB (this highly depends on the configured rules). + +<table> +<tr><td>Marshal config</td><td>320MB</td></tr> +<tr><td>Plain text config</td><td>30MB</td></tr> +</table> + diff --git a/ChangeLog b/ChangeLog index fcdc6c8..186bf86 100644 --- a/ChangeLog +++ b/ChangeLog @@ -2,6 +2,7 @@ Core & Setup: * 3342 MKP: local/lib and local/bin of sites can now be packed... * 3114 linux and windows agent can now be configured to directly encrypt their output. For real-time updates encryption is now optional (but active per default)... + * 3531 Reduced Check_MK helper size for certain dists/python versions... * 3193 FIX: Fixed fake check results for hosts... * 3214 FIX: Removing SNMP checks for non SNMP hosts and agent based checks for non agent hosts... * 3220 FIX: Check_MK HW/SW Inventory is now always disabled for "No Agent" hosts diff --git a/modules/check_mk.py b/modules/check_mk.py index 2bfae88..f99a5d5 100755 --- a/modules/check_mk.py +++ b/modules/check_mk.py @@ -2421,7 +2421,7 @@ def pack_config(): except: return False - filepath = var_dir + "/core/config.mk" + filepath = var_dir + "/core/helper_config.mk" out = file(filepath + ".new", "w") out.write("#!/usr/bin/python\n" "# encoding: utf-8\n" @@ -2439,7 +2439,19 @@ def pack_config(): out.write("\nif %r in globals():\n del %s\n" % (varname, varname)) out.close() - os.rename(filepath + ".new", filepath) + + import marshal + code = compile(open(filepath + ".new").read(), '<string>', 'exec') + marshal.dump(code, open(filepath + ".compiled", "wb")) + os.rename(filepath + ".compiled", filepath) + # Remove the configs readable version + os.unlink(filepath + ".new") + + +def read_packed_config(): + import marshal + filepath = var_dir + "/core/helper_config.mk" + exec(marshal.load(open(filepath)), globals()) def pack_autochecks(): dstpath = var_dir + "/core/autochecks" @@ -2462,9 +2474,6 @@ def pack_autochecks(): if f not in needed: os.remove(dstpath + "/" + f) -def read_packed_config(): - filepath = var_dir + "/core/config.mk" - execfile(filepath, globals()) #. # .--Man-Pages-----------------------------------------------------------.