[checkmk-commits] [tribe29/checkmk] 7c1342: Faster IP lookup cache writing

Branch: refs/heads/master Home: https://github.com/tribe29/checkmk Commit: 7c13428c61704cf4eca4c956ec482a1af4b92824 https://github.com/tribe29/checkmk/commit/7c13428c61704cf4eca4c956ec482a1af… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: M cmk_base/ip_lookup.py Log Message: ----------- Faster IP lookup cache writing Storing the file in pretty printed mode takes a lot more time when working with larger cache files. We should really think about changing the default to pretty=False. Change-Id: Ic8a63a2259927f2231888aaf9051d65f2478f9b9 Commit: ee5b44d02075159840cf73db52b7c754d6ac5d06 https://github.com/tribe29/checkmk/commit/ee5b44d02075159840cf73db52b7c754d… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: M cmk_base/ip_lookup.py M tests/unit/cmk_base/test_ip_lookup.py Log Message: ----------- Some more IP cache cleanups * Better name for cache object factory * Extract cache clearing logic * Minor fix: Also clean up the in-memory cache * Add test for clearing logic * Extract host/family enumeration * Add test for host/family enumeration Change-Id: I02232a0467d20d65dc9db1ddca334775a26a0522 Commit: e2259508bb71cfb1391b2509f0dd781fc1bd6cea https://github.com/tribe29/checkmk/commit/e2259508bb71cfb1391b2509f0dd781fc… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: M cmk_base/ip_lookup.py Log Message: ----------- Simplify host / family lookup computation Change-Id: I86debb91216d97f466a5cd165cc99e80c3f8e240 Commit: b4928b21b7508bd2139e9052228bd6ac496adbd7 https://github.com/tribe29/checkmk/commit/b4928b21b7508bd2139e9052228bd6ac4… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: M cmk_base/caching.py M cmk_base/ip_lookup.py M tests/unit/cmk_base/test_ip_lookup.py M tests/unit/conftest.py Log Message: ----------- Improve "Update DNS cache" performance * The IPLookupCache is now managed by a special form of the DictCache which also cares about writing the persisted cache. This is not ideal, but seemed to be the least invasive way. * The cache updates the DNS cache for each individual lookup by default as it did it before. * But this object can now be set to "do not persist" mode to just capture the updates in memory during subsequent lookups like it's done in ip_lookup.update_dns_cache(). * After the lookups are done, the save() method can be called once to perform a single bulk operation The operation is now a 100 times faster (tested with ~2000 hosts and a fast responding DNS server. Made some cleanups on the way and added several tests for the affected functions. Change-Id: Ib927cff7556ae07348e457973a1e0c6fa355a2ef Commit: de7e062cc2f4602e84deecfda39a75ab572b01d3 https://github.com/tribe29/checkmk/commit/de7e062cc2f4602e84deecfda39a75ab5… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: A .werks/10378 Log Message: ----------- 10378 FIX Improve "Update DNS cache" / cmk --update-dns-cache performance The operation was performing a lot of needless operations. The update of several thousands of hosts DNS entries took way too long. The exact time needed depends mostly on the number of hosts, cache size, DNS lookup performance and so on. But even with good responding servers the operation took too long, because the cache was updated in a very inefficient way when calling the "Update DNS cache" action in WATO or "cmk --update-dns-cache" from the command line. The operation is now more than a 100 times faster (tested with ~2000 hosts and a fast responding DNS server). Change-Id: Ie9a2cb589c7861934311043d1e8161a6de4d519e Commit: dd035ae8a94c33648581da3c2a7bf8ef8b981d88 https://github.com/tribe29/checkmk/commit/dd035ae8a94c33648581da3c2a7bf8ef8… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2019-11-04 (Mon, 04 Nov 2019) Changed paths: M cmk/utils/store.py M tests/unit/cmk/utils/test_store.py Log Message: ----------- save_data_to_file: Only pretty print when enabled The pretty formatting costs a lot of time, especially with larger data structures. Most of the treated files are rarely evaluated manually, so the formatting is unnecessary overhead and should only be activated on demand. Change-Id: I5d8cc55f2a3cec45f6450c900a5e4ebccfd11fed Compare: https://github.com/tribe29/checkmk/compare/2e55fa67c4d3...dd035ae8a94c