I'm working on a project where it is assumed that a few thousand distributed custom devices need to be monitored. So there are pretty simple questions which shall be answered like: Whats the load/memory/disk space of device X now? What was the disk usage lately? And what was the load at the first time the device was online, maybe years back in time? And also more complicated questions like: Which devices did and how long run a custom task? This one would be used for accounting purpose.
One option to solve this is to build a custom monitoring and reporting tool. But this won't be a simple one day hack, so i wanted to ask if anyone knows some open source software which is able to
- add new metrices in a very flexible way like ganglia's gmetric tool
- keep the data forever in some kind of storage to query for simple data points
- use a json api to query datapoints and to be able to generate complex query's via some kind of api
- build graphs like everyone already knows from ganglia or such
I've tried ganglia already, but gmetad seems to dump the collected data as soon as the device is offline. But the flexiblity of gmetric is one key feature since in dont know what might be relevant to monitor in the future.