Checkmk git commits November 2017

checkmk-commits@lists.checkmk.com

10 participants
325 discussions

Fixed several tests after code and functional cleanups

by Lars Michelsen

Module: check_mk Branch: master Commit: 2515db334ccfa9b372650ca56b30c36d4d5ed047 URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=2515db334ccfa9… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Tue Nov 21 10:42:23 2017 +0100 Fixed several tests after code and functional cleanups Change-Id: Ia5fade7790f75a2cddc6c82130a5542e158927cd --- cmk_base/automations/check_mk.py | 8 ++++++-- cmk_base/discovery.py | 10 +++++++++- cmk_base/inventory.py | 6 +++--- cmk_base/modes/check_mk.py | 11 ++--------- tests/cmk_base/test_automations.py | 7 +++---- tests/cmk_base/test_data_sources.py | 21 ++++++--------------- tests/cmk_base/test_modes.py | 6 +++--- tests/cmk_base/test_rulesets.py | 15 +++++++++++++-- 8 files changed, 45 insertions(+), 39 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=2515db334c…

6 years, 7 months

Improved logging of cache file writing

by Lars Michelsen

Module: check_mk Branch: master Commit: 8a666cf1a394f04c0a74d3074f4a302e84b3dcae URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=8a666cf1a394f0… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Fri Nov 17 12:08:47 2017 +0100 Improved logging of cache file writing Change-Id: I1dcbf452de38ae3bc56969e053fccc3e7ed8ddd2 --- cmk_base/data_sources/abstract.py | 5 +++-- cmk_base/data_sources/snmp.py | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/cmk_base/data_sources/abstract.py b/cmk_base/data_sources/abstract.py index 7597d58..fb3ba09 100644 --- a/cmk_base/data_sources/abstract.py +++ b/cmk_base/data_sources/abstract.py @@ -178,7 +178,7 @@ class DataSource(object): return self._from_cache_file(result) - def _write_cache_file(self, hostname, output): + def _write_cache_file(self, hostname, raw_data): cachefile = self._cache_file_path(hostname) try: @@ -192,8 +192,9 @@ class DataSource(object): except Exception, e: raise MKGeneralException("Cannot create directory %r: %s" % (os.path.dirname(cachefile), e)) + self._logger.verbose("[%s] Write data to cache file %s" % (self.id(), cachefile)) try: - store.save_file(cachefile, self._to_cache_file(output)) + store.save_file(cachefile, self._to_cache_file(raw_data)) except Exception, e: raise MKGeneralException("Cannot write cache file %s: %s" % (cachefile, e)) diff --git a/cmk_base/data_sources/snmp.py b/cmk_base/data_sources/snmp.py index d55da51..9eeef03 100644 --- a/cmk_base/data_sources/snmp.py +++ b/cmk_base/data_sources/snmp.py @@ -102,7 +102,7 @@ class SNMPDataSource(DataSource): def _to_cache_file(self, info): - return repr(info) + return repr(info) + "\n" def set_ignore_check_interval(self, ignore_check_interval):

6 years, 7 months

Moved functions to better places

by Lars Michelsen

Module: check_mk Branch: master Commit: 951e6e5bd0bbd41236079109b164b1fc2d14b30d URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=951e6e5bd0bbd4… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Tue Nov 21 09:41:23 2017 +0100 Moved functions to better places * Now that the data structure is getting clearer the main function for gathering the data of a host get_host_sections() can be moved to the DataSources() object. * Moved get_section_content_for_check() that prepares the section_content for a check/inventory plugin to MultiHostSections() object. Change-Id: I068be0c614c5087200f19b920fad6a5d207088b4 --- cmk_base/checking.py | 10 +- cmk_base/data_sources/__init__.py | 275 +++++++-------------------------- cmk_base/data_sources/host_sections.py | 181 ++++++++++++++++++++++ cmk_base/discovery.py | 8 +- cmk_base/inventory.py | 6 +- 5 files changed, 246 insertions(+), 234 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=951e6e5bd0…

6 years, 7 months

Minor wording / output improvements

by Lars Michelsen

Module: check_mk Branch: master Commit: 4d0bb30884c5b81859102cc9ab68624a1ec9759e URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=4d0bb30884c5b8… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Tue Nov 21 15:38:43 2017 +0100 Minor wording / output improvements Change-Id: Ief084eade837ba64c682e19e6997b828b658749c --- web/htdocs/watolib.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/web/htdocs/watolib.py b/web/htdocs/watolib.py index 20a023e..1d9bc52 100644 --- a/web/htdocs/watolib.py +++ b/web/htdocs/watolib.py @@ -3279,7 +3279,7 @@ class HostTagAttribute(Attribute): if value: return "", title else: - return "", "%s %s" % (_("not"), title) + return "", "%s %s" % (_("Not"), title) for entry in self._taglist: if value == entry[0]: return "", entry[1] and _u(entry[1]) or '' @@ -6066,9 +6066,9 @@ def register_builtin_host_tags(): ("ping", _("only ping this device"), []), ]), ("agent", "%s/%s" % (_("Data sources"), _("Check_MK Agent")), [ - ("cmk-agent", _("Contact either Check_MK Agent or use special agent"), ["tcp"]), - ("all-agents", _("Contact Check_MK agent and all enabled special agents"), ["tcp"]), - ("special-agents", _("Use all enabled special agents"), ["tcp"]), + ("cmk-agent", _("Contact either Check_MK Agent or use datasource program"), ["tcp"]), + ("all-agents", _("Contact Check_MK agent and all enabled datasource programs"), ["tcp"]), + ("special-agents", _("Use all enabled datasource programs"), ["tcp"]), ("no-agent", _("No agent"), []), ], ["!ping"],

6 years, 7 months

Cleaned up discovery code

by Lars Michelsen

Module: check_mk Branch: master Commit: 158be95c82413e8d758ff176975fc8d4eabf35e9 URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=158be95c82413e… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Fri Nov 17 13:48:32 2017 +0100 Cleaned up discovery code * Separated data gathering from doing the discovery * Performance: Reduced duplicate data gathering on service discovery page of WATO * Moved data gathering (including all caching options) to top level * Moved IP address resolution to top level which makes it much clearer in which situations IP addresses are set by caller or not Change-Id: Ied8ee8fa417743047ddd325a4adf83e5fcb79c0e --- cmk_base/discovery.py | 100 +++++++++++++++++++++++++-------------------- cmk_base/ip_lookup.py | 2 +- cmk_base/modes/check_mk.py | 11 ++++- 3 files changed, 65 insertions(+), 48 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=158be95c82…

6 years, 7 months

Updated bug entries #3014

by Lars Michelsen

Module: check_mk Branch: master Commit: bf2d1ea9dd3e0f2617332dda23a460fbd0ea1768 URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=bf2d1ea9dd3e0f… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Fri Nov 17 12:23:38 2017 +0100 Updated bug entries #3014 Change-Id: I6dd1fc936953c041538035fae4b2ab4d8b2ddb93 --- .bugs/3014 | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.bugs/3014 b/.bugs/3014 new file mode 100644 index 0000000..aca49ea --- /dev/null +++ b/.bugs/3014 @@ -0,0 +1,16 @@ +Title: Check cache usage of the HW/SW inventory check +Component: core +State: open +Date: 2017-11-17 12:23:20 +Targetversion: 1.5.0 +Class: bug + +We should check why the inventory check is handled differently than +the discovery check regarding it's cache usage. + +The HW/SW inventory check is always trying to fetch fresh SNMP data. +While the inventory check makes use of the existing data source caches. + +There are SNMP HW/SW inventory plugins that do not exist as checks and +vice versa. We need to care about this when writing the cache in case +we mix these both.

6 years, 7 months

5535 Check_MK hosts can now use multiple agents

by Lars Michelsen

Module: check_mk Branch: master Commit: af6603bba9d8a36e8e265b8c8288067b91c705f9 URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=af6603bba9d8a3… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Tue Nov 21 15:28:21 2017 +0100 5535 Check_MK hosts can now use multiple agents It is now possible to configure multiple Check_MK agents for a single host. With this change, you can now configure e.g. a ESX VCenter to use the ESX special agent together with the regular Check_MK agent installed on the monitored host. All existing hosts remain using their existing configuration after an update. Also new hosts only use a single Check_MK agent using the already existing logic where a) The Check_MK agent was contacted via TCP or b) a configured data source program (special agent or other program invocation) was used. The new feature can be enabled by changing the host attribute (on hosts or folders) "Check_MK Agent" to e.g. "Contact Check_MK agent and all datasource programs". This will make Check_MK use all data sources matching on this host instead of just picking the first matching one. There is also an alternative option "Use all enabled datasources" available which can be used to execute only the data sources matching the host. On the way to this change we have changed server previously existing things: <ul> <li>The host tag group <tt>agent</tt> has been split into multiple tag groups to be more flexible.</li> <li>The tag group <tt>ping</tt> and <tt>snmp</tt> have been added and provide the options which were previously available in the single <tt>agent</tt> attribute.</li> <li>All these tag groups are treated as builtin tag groups defined by Check_MK (can not be modified anymore).</li> <li>Existing configurations of hosts/folders will be translated seamlessly into the new format.</li> <li>During updates your site will only apply the changes above in case you have an unmodified <tt>agent</tt> tag group. In case you have modified it in any way, these changes will not be applied and you won't be able to use the changes introduced with this werk. You will then have to clean up your local changes. Once you delete your local tag group "agent", the builtin one will be used automatically.</li> <li>The Edit host dialog has split up into more independent sections, the new ones are Address and Data sources to better visualize the relation of the different attributes.</li> </ul> Please note: In case you are using the Web-API calls to create or modify hosts or folders while setting attributes we changed with this change, you may have to change your API calls. Change-Id: I54cd61a6ccb662faf456348fbb6f2243c3c21457 --- .werks/5535 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/.werks/5535 b/.werks/5535 new file mode 100644 index 0000000..eaf323b --- /dev/null +++ b/.werks/5535 @@ -0,0 +1,48 @@ +Title: Check_MK hosts can now use multiple agents +Level: 2 +Component: core +Compatible: compat +Edition: cre +Version: 1.5.0i2 +Date: 1511273074 +Class: feature + +It is now possible to configure multiple Check_MK agents for a single host. + +With this change, you can now configure e.g. a ESX VCenter to use the ESX special agent +together with the regular Check_MK agent installed on the monitored host. + +All existing hosts remain using their existing configuration after an update. Also +new hosts only use a single Check_MK agent using the already existing logic where +a) The Check_MK agent was contacted via TCP or b) a configured data source program +(special agent or other program invocation) was used. + +The new feature can be enabled by changing the host attribute (on hosts or folders) +"Check_MK Agent" to e.g. "Contact Check_MK agent and all datasource programs". This +will make Check_MK use all data sources matching on this host instead of just picking +the first matching one. There is also an alternative option "Use all enabled datasources" +available which can be used to execute only the data sources matching the host. + +On the way to this change we have changed server previously existing things: + +<ul> + <li>The host tag group <tt>agent</tt> has been split into multiple tag groups to be + more flexible.</li> + <li>The tag group <tt>ping</tt> and <tt>snmp</tt> have been added and provide the options + which were previously available in the single <tt>agent</tt> attribute.</li> + <li>All these tag groups are treated as builtin tag groups defined by Check_MK + (can not be modified anymore).</li> + <li>Existing configurations of hosts/folders will be translated seamlessly into the new + format.</li> + <li>During updates your site will only apply the changes above in case you have an unmodified + <tt>agent</tt> tag group. In case you have modified it in any way, these changes will not + be applied and you won't be able to use the changes introduced with this werk. You will then + have to clean up your local changes. Once you delete your local tag group "agent", the + builtin one will be used automatically.</li> + <li>The Edit host dialog has split up into more independent sections, the new ones + are Address and Data sources to better visualize the relation of the different + attributes.</li> +</ul> + +Please note: In case you are using the Web-API calls to create or modify hosts or folders +while setting attributes we changed with this change, you may have to change your API calls.

6 years, 7 months

Fix some naming conventions related to agents/checking

by Lars Michelsen

Module: check_mk Branch: master Commit: 5434ba61897cd970688386927a7e2e20fc0353bd URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=5434ba61897cd9… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Mon Nov 20 13:13:17 2017 +0100 Fix some naming conventions related to agents/checking * section_name : The identifier for a section of monitoring data * check_plugin_name : The identifier for a Check_MK check plugin * active_check_name : The identifier of a active check plugin * host_sections : Object managing parsed data received from data sources * section_content : A list of lists (Agent: list of rows splitted by separators) Change-Id: I281deae6bee2885540560b033aed7d42908d5d1f --- cmk_base/automations/check_mk.py | 42 +-- cmk_base/check_api.py | 16 +- cmk_base/check_table.py | 14 +- cmk_base/checking.py | 74 ++--- cmk_base/checks.py | 120 +++---- cmk_base/config.py | 40 +-- cmk_base/core_nagios.py | 34 +- cmk_base/crash_reporting.py | 16 +- cmk_base/data_sources/__init__.py | 148 +++++---- cmk_base/data_sources/abstract.py | 128 ++++---- .../{host_info.py => host_sections.py} | 39 +-- cmk_base/data_sources/programs.py | 2 +- cmk_base/data_sources/snmp.py | 68 ++-- cmk_base/discovery.py | 354 +++++++++++---------- cmk_base/inventory.py | 31 +- cmk_base/inventory_plugins.py | 8 +- cmk_base/item_state.py | 2 +- cmk_base/modes/check_mk.py | 12 +- cmk_base/snmp.py | 26 +- 19 files changed, 594 insertions(+), 580 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=5434ba6189…

6 years, 7 months

Add missing error handling to inventory call

by Lars Michelsen

Module: check_mk Branch: master Commit: cac7ae64f70d26465981c16b36c66f8ce5d8049a URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=cac7ae64f70d26… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Mon Nov 13 09:03:11 2017 +0100 Add missing error handling to inventory call Change-Id: I81a6cc1387350cc2c577524c3311a0bf8a8c122a --- cmk_base/inventory.py | 36 +++++++++----- tests/cmk_base/test_data_sources.py | 96 +++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+), 13 deletions(-) diff --git a/cmk_base/inventory.py b/cmk_base/inventory.py index 7ecdf2b..17344c2 100644 --- a/cmk_base/inventory.py +++ b/cmk_base/inventory.py @@ -83,15 +83,26 @@ def do_inv(hostnames): for hostname in hostnames: try: console.verbose("Doing HW/SW-Inventory for %s..." % hostname) - do_inv_for(hostname) + + if config.is_cluster(hostname): + ipaddress = None + else: + ipaddress = ip_lookup.lookup_ip_address(hostname) + + _do_inv_for(hostname, ipaddress) console.verbose("..OK\n") except Exception, e: - # TODO: handle data_sources.get_data_source_errors_of_host() here if cmk.debug.enabled(): raise + console.verbose("Failed: %s\n" % e) errors.append("Failed to inventorize %s: %s" % (hostname, e)) - cmk_base.utils.cleanup_globals() + finally: + for data_source, exceptions in data_sources.get_data_source_errors_of_host(hostname, ipaddress).items(): + for exc in exceptions: + errors.append("%s" % exc) + + cmk_base.utils.cleanup_globals() if errors: raise MKGeneralException("\n".join(errors)) @@ -105,7 +116,12 @@ def do_inv_check(options, hostname): _inv_fail_status = options.get("inv-fail-status", _inv_fail_status) try: - inv_tree, old_timestamp = do_inv_for(hostname) + if config.is_cluster(hostname): + ipaddress = None + else: + ipaddress = ip_lookup.lookup_ip_address(hostname) + + inv_tree, old_timestamp = _do_inv_for(hostname, ipaddress) num_entries = _count_nodes(g_inv_tree) if not num_entries: console.output("OK - Found no data\n") @@ -142,7 +158,6 @@ def do_inv_check(options, hostname): infotexts.append(infotext) - ipaddress = ip_lookup.lookup_ip_address(hostname) for data_source, exceptions in data_sources.get_data_source_errors_of_host(hostname, ipaddress).items(): for exc in exceptions: infotexts.append("%s" % exc) @@ -158,7 +173,7 @@ def do_inv_check(options, hostname): sys.exit(_inv_fail_status) -def do_inv_for(hostname): +def _do_inv_for(hostname, ipaddress): _initialize_inventory_tree() node = inv_tree("software.applications.check_mk.cluster.") @@ -168,7 +183,7 @@ def do_inv_for(hostname): _do_inv_for_cluster(hostname) else: node["is_cluster"] = False - _do_inv_for_realhost(hostname) + _do_inv_for_realhost(hostname, ipaddress) # Remove empty paths _cleanup_inventory_tree(g_inv_tree) @@ -189,12 +204,7 @@ def _do_inv_for_cluster(hostname): }) -def _do_inv_for_realhost(hostname): - try: - ipaddress = ip_lookup.lookup_ip_address(hostname) - except: - raise MKGeneralException("Cannot resolve hostname '%s'." % hostname) - +def _do_inv_for_realhost(hostname, ipaddress): sources = data_sources.DataSources(hostname) for source in sources.get_data_sources(): diff --git a/tests/cmk_base/test_data_sources.py b/tests/cmk_base/test_data_sources.py new file mode 100644 index 0000000..f980e41 --- /dev/null +++ b/tests/cmk_base/test_data_sources.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python +# These tests verify the behaviour of the Check_MK base methods +# that do the actual checking/discovery/inventory work. Especially +# the default caching and handling of global options affecting the +# caching is checked + +import pytest +from testlib import web, repo_path + +import cmk_base.config as config +import cmk_base.modes + +(a)pytest.fixture(scope="module") +def test_cfg(web, site): + print "Applying default config" + web.add_host("ds-test-host1", attributes={ + "ipaddress": "127.0.0.1", + }) + web.add_host("ds-test-host2", attributes={ + "ipaddress": "127.0.0.1", + }) + + site.write_file("etc/check_mk/conf.d/ds-test-host.mk", + "datasource_programs.append(('cat ~/var/check_mk/agent_output/<HOST>', [], ALL_HOSTS))\n") + + site.makedirs("var/check_mk/agent_output/") + site.write_file("var/check_mk/agent_output/ds-test-host1", + file("%s/tests/data/linux-agent-output" % repo_path()).read()) + site.write_file("var/check_mk/agent_output/ds-test-host2", + file("%s/tests/data/linux-agent-output" % repo_path()).read()) + + web.activate_changes() + + import cmk.debug + cmk.debug.enable() + + config.load() + yield None + + # + # Cleanup code + # + print "Cleaning up test config" + + site.delete_dir("var/check_mk/agent_output") + site.delete_file("etc/check_mk/conf.d/ds-test-host.mk") + + web.delete_host("ds-test-host1") + web.delete_host("ds-test-host2") + + web.activate_changes() + + +# Globale Optionen: +# --cache +# --no-cache +# --no-tcp +# --usewalk +# --force + +# TODO: Use force +# TODO: With hostnames and without +def test_mode_inventory_set_caching(test_cfg, mocker): + use_cachefile_patcher = mocker.patch("cmk_base.data_sources.abstract.DataSource.set_use_cachefile") + use_persisted_sections_patcher = mocker.patch("cmk_base.data_sources.abstract.CheckMKAgentDataSource.use_outdated_persisted_sections") + + # When called without hosts, it uses all hosts and defaults to using the data source cache + cmk_base.modes.check_mk.mode_inventory([], []) + use_cachefile_patcher.assert_called_once_with() + use_persisted_sections_patcher.assert_not_called() + + # When called with an explicit list of hosts the cache is not used by default + cmk_base.modes.check_mk.mode_inventory([], ["ds-test-host1"]) + use_cachefile_patcher.assert_not_called() + use_persisted_sections_patcher.assert_not_called() + + cmk_base.modes.check_mk.mode_inventory(["force"], ["ds-test-host1"]) + use_cachefile_patcher.assert_not_called() + use_persisted_sections_patcher.assert_called() + +# mode_inventory() +# mode_inventory_as_check(options, hostname): +# mode_discover_marked_hosts(): +# mode_check_discovery(*args): +# mode_discover(options, args): +# mode_check(options, args): +# dump host +# +# automation: +# AutomationTryDiscovery +# AutomationDiscovery +# AutomationDiagHost +# +# Keepalive check +# Keepalive discovery +

6 years, 7 months

Wrap info into HostInfo() to be able to transport caching information next to the info iself

by Lars Michelsen

Module: check_mk Branch: master Commit: ff84598319b436a2eda12f037f7e09623bb2f1bd URL: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=ff84598319b436… Author: Lars Michelsen <lm(a)mathias-kettner.de> Date: Fri Nov 10 12:23:29 2017 +0100 Wrap info into HostInfo() to be able to transport caching information next to the info iself Change-Id: I76432adee4f465b2fa9b773424b02a92785f38be --- cmk_base/checking.py | 13 +++++----- cmk_base/data_sources/__init__.py | 36 +++++++++++---------------- cmk_base/data_sources/abstract.py | 51 +++++++++++++++++++++----------------- cmk_base/data_sources/host_info.py | 51 ++++++++++++++++++++++++++++++++++++++ cmk_base/data_sources/snmp.py | 5 ++++ cmk_base/data_sources/tcp.py | 2 +- cmk_base/discovery.py | 16 ++++++------ cmk_base/inventory.py | 4 +-- 8 files changed, 116 insertions(+), 62 deletions(-) Diff: http://git.mathias-kettner.de/git/?p=check_mk.git;a=commitdiff;h=ff84598319…

6 years, 7 months

← Newer
1
...
12
13
14
15
16
17
18
...
33
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Checkmk git commits November 2017