16 Dec
2013
16 Dec
'13
7:49 a.m.
Module: check_mk
Branch: master
Commit: 5e9c5cc361536f6a5ece27a7150dcbe04141a2c6
URL:
http://git.mathias-kettner.de/git/?p=check_mk.git;a=commit;h=5e9c5cc361536f…
Author: Mathias Kettner <mk(a)mathias-kettner.de>
Date: Mon Dec 16 13:48:56 2013 +0100
FIX Limit the number of lines read from a single logfile
If for some reason you have very large history logfiles with millions of
entries then Livestatus would consume large amounts of memory while parsing
these when you access the <tt>log</tt> table. This could lead to a hanging
system or even a crash.
In order to avoid such situations the number of lines that are read from each
individual file are now limited. The default limit is one million lines. You
can change this default by defining the variable <tt>max_lines_per_logfile/tt>
in your <tt>nagios.cfg</tt>. The following example sets the maximum number
to 20 million lines:
F+:
broker_module=/some/path/livestatus.o max_lines_per_logfile=20000000 ...
F-:
When using the Check_MK Micro Core the configuration can be done
via WATO global setting <i>Maximum number of parsed lines per log file</i>.
---
.bugs/959 | 7 +++++--
.werks/336 | 26 ++++++++++++++++++++++++++
ChangeLog | 1 +
livestatus/src/Logfile.cc | 7 ++++++-
livestatus/src/module.c | 5 +++++
5 files changed, 43 insertions(+), 3 deletions(-)
diff --git a/.bugs/959 b/.bugs/959
index d131cb7..ab501ec 100644
--- a/.bugs/959
+++ b/.bugs/959
@@ -1,9 +1,9 @@
Title: Livestatus/Nagios Core crashes, if nagios.log is too large
Component: livestatus
-State: open
+Class: bug
+State: done
Date: 2013-08-08 11:30:12
Targetversion: 1.2.3i5
-Class: bug
In Benutzung war: Nagios 3.3, Check_MK check_mk-1.2.2p2 sowie
mk-livestatus-1.2.2p2 auf einem Centos 5.5.
@@ -28,3 +28,6 @@ vorhandenen Caching-Optionen.
Wichtig hierbei ist, dem Nutzer den Fehler darzustellen und eventuell
auch einen Eintrag im Logfile zu erzeugen. Der Nutzer sollte dann darauf
hingewiesen werden seine Logging-/Rotations-Einstellungen zu aendern.
+
+2013-12-16 13:43:27: changed state open -> done
+Max Zeilen konfigurierbar, default ist 1000000.
diff --git a/.werks/336 b/.werks/336
new file mode 100644
index 0000000..ecb178a
--- /dev/null
+++ b/.werks/336
@@ -0,0 +1,26 @@
+Title: Limit the number of lines read from a single logfile
+Level: 2
+Component: livestatus
+Class: fix
+State: unknown
+Version: 1.2.5i1
+Date: 1387197814
+Targetversion: future
+
+If for some reason you have very large history logfiles with millions of
+entries then Livestatus would consume large amounts of memory while parsing
+these when you access the <tt>log</tt> table. This could lead to a hanging
+system or even a crash.
+
+In order to avoid such situations the number of lines that are read from each
+individual file are now limited. The default limit is one million lines. You
+can change this default by defining the variable <tt>max_lines_per_logfile/tt>
+in your <tt>nagios.cfg</tt>. The following example sets the maximum number
+to 20 million lines:
+
+F+:
+broker_module=/some/path/livestatus.o max_lines_per_logfile=20000000 ...
+F-:
+
+When using the Check_MK Micro Core the configuration can be done
+via WATO global setting <i>Maximum number of parsed lines per log file</i>.
diff --git a/ChangeLog b/ChangeLog
index 01bca7c..ea881da 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -61,6 +61,7 @@
Livestatus:
* 0335 FIX: Parse state of downtime notification log entries correctly...
+ * 0336 FIX: Limit the number of lines read from a single logfile...
1.2.3i7:
diff --git a/livestatus/src/Logfile.cc b/livestatus/src/Logfile.cc
index b0f84c9..3324554 100644
--- a/livestatus/src/Logfile.cc
+++ b/livestatus/src/Logfile.cc
@@ -38,6 +38,8 @@
extern Core *g_core;
#endif
+extern unsigned long g_max_lines_per_logfile;
+
Logfile::Logfile(const char *path, bool watch)
: _path(strdup(path))
@@ -145,9 +147,12 @@ void Logfile::load(LogCache *logcache, time_t since, time_t until,
unsigned logc
void Logfile::loadRange(FILE *file, unsigned missing_types,
LogCache *logcache, time_t since, time_t until, unsigned logclasses)
{
- //logger(LOG_NOTICE, "Read logfile: %s", this->_path);
while (fgets(_linebuffer, MAX_LOGLINE, file))
{
+ if (_lineno >= g_max_lines_per_logfile) {
+ logger(LG_ERR, "More than %u lines in %s. Ignoring the rest!",
g_max_lines_per_logfile, this->_path);
+ return;
+ }
_lineno++;
if (processLogLine(_lineno, missing_types)) {
logcache->handleNewMessage(this, since, until, logclasses); // memory
management
diff --git a/livestatus/src/module.c b/livestatus/src/module.c
index d1835a7..7d3404d 100644
--- a/livestatus/src/module.c
+++ b/livestatus/src/module.c
@@ -100,6 +100,7 @@ int g_should_terminate = false;
pthread_t g_mainthread_id;
pthread_t *g_clientthread_id;
unsigned long g_max_cached_messages = 500000;
+unsigned long g_max_lines_per_logfile = 1000000; // do never read more than that number
of lines from a logfile
unsigned long g_max_response_size = 100 * 1024 * 1024; // limit answer to 10 MB
int g_thread_running = 0;
int g_thread_pid = 0;
@@ -661,6 +662,10 @@ void livestatus_parse_arguments(const char *args_orig)
g_max_cached_messages = strtoul(right, 0, 10);
logger(LG_INFO, "Setting max number of cached log messages to
%lu", g_max_cached_messages);
}
+ else if (!strcmp(left, "max_lines_per_logfile")) {
+ g_max_lines_per_logfile = strtoul(right, 0, 10);
+ logger(LG_INFO, "Setting max number lines per logfile to %lu",
g_max_lines_per_logfile);
+ }
else if (!strcmp(left, "thread_stack_size")) {
g_thread_stack_size = strtoul(right, 0, 10);
logger(LG_INFO, "Setting size of thread stacks to %lu",
g_thread_stack_size);