Golang - Windows and Linux Log collection

Hi,

We have a REST API that’s used to receive log data and provide configurations for what information to log from a host. We would like to develop a Golang application to retrieve logs and basic metrics from a host. This application will run on a client and send logs to the REST API. The client side application is not responsible for parsing logs (no regex needed) just collecting the information and sending it. This will be installed on Windows or Linux as a service and would need to be constantly online.

The functionality is similar to the beats products such as Filebeats, Winlogbeats, and Metricbeats. As these are opensource and written in Go, it is fine to use parts of the applications but we’re not interested in just making a plugin for these to adapt to our system. Unlike the beats products that have many options, we want to focus on just the key functions below.

Retrieve configuration from API:

By doing a REST get and providing the host id, the API will return a configuration that defines which logs to retrieve and which metrics to collect.

Collecting logs from Windows:

The application should query Windows event logs (System, Application, and Security), convert to json format, and then forward to our REST API. The application should keep track so that it does not read in duplicate logs.

Collecting logs from Linux and files in Windows:

The logs are defined in the configuration with a wildcard to support log rotate. Unlike systems like Splunk or beats, we do not need to monitor all files in a directory. We really only need to support monitoring a single file, but support collecting from a rotated file when the log is rotated. Our current method is to monitor a single file, but when a file is smaller than it was when last read, we look to the rotated file to ensure no logs were missed.

Metrics from Linux and Windows:

Though other system collect detailed metrics, we support fairly basic collection of metrics. We do plan to expand these in the future but for now we are just collecting these items.

-Processes with RAM and CPU

-Services with current state

-Uptime

-RAM utilization

-CPU utilization

-Disk utilization

Host Information:

We collect basic information about the host infrequently. This is typically done on an hourly or daily basis.

GUID

-Uptime

-Processor info

-Ram info

-Open ports

-List Installed applications

-Network information (Interfaces, IPs, etc…)

Golang Linux Software Arkitektur

Projekt ID: #19233753

Om projektet