rsyslog filling up /var/log puts the system down

Sometimes a process goes wrong and the logs in /var/log grow up so much they ultimately fill up the whole partition.

It happened me once on a server due to a wrong postfix config, and once on a desktop due to a USB printer (not sure exactly what went wrong, all I know is the logs where filled with (hp) did not claim interface 1 before use).

I know the root cause is not the logger but the application. However, I can't help thinking this weak point is a shame. Especially on a desktop, where a printer filling up the whole system partition prevents the user from loading the GUI on the next run (no space on /tmp) which is a total blocker for a non-techy.

  • logrotate is not an answer because it acts daily or even weekly.
  • rsyslog has that max-size,action-on-max-size config option, but it seems non trivial and according to the doc itself, it might break in a future release.
  • Putting /var/log in a dedicated partition, however, would prevent this from happening.

To my knowledge, the separate partition for /var/log is the only solution to this. I've seen this recommended sometimes, but it is not the default in Debian installer, for instance. Should it be?

Is there any other simple way to avoid this? Some way to provide a max size to the /var/log directory, or at least to rsyslog?

Is this issue not frequent enough to justify a protection mechanism that would be activated by default? (I'm particularly thinking of home/desktop installation, for which users are not supposed to be able to deal with this themselves.)

Edit: SystemLogRateLimit

Thanks to Julie Pelletier's answer, I discovered the rate limit mechanism in rsyslog which answers precisely this need and should be activated by default.

It seems, however, that it is not active by default or does not work on two different Debian Jessie machines I'm using.

Should I file a bug at Debian about this?

I have evidence (log excerpt) for both cases.

Note rsyslog docs mention a severity parameter:

There is input rate limiting available, (since 5.7.1) to guard you against the problems of a wild running logging process. If more than SysSock.RateLimit.Interval * SysSock.RateLimit.Burst log messages are emitted from the same process, those messages with SysSock.RateLimit.Severity or lower will be dropped.

I don't know the default value for this one, and I don't know the severity of the messages that filled up my logs.

Replay

rsyslog includes a rate limiting option by default through the imuxsock module.

It defaults to 200 messages per 5 seconds but can easily be changed by setting the following after the module is loaded:

$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 200

The $SystemLogRateLimitInterval is the interval in seconds (which you should increase) and $SystemLogRateLimitBurst is the maximum number of messages allowed by the application during that interval (which you should decrease).

Category: logs Time: 2016-07-30 Views: 0

Related post

iOS development

Android development

Python development

JAVA development

Development language

PHP development

Ruby development

search

Front-end development

Database

development tools

Open Platform

Javascript development

.NET development

cloud computing

server

Copyright (C) avrocks.com, All Rights Reserved.

processed in 2.134 (s). 13 q(s)