Syslog in Practice: A Guide to Golang Security Product Integration

Building a production-grade log collection service from scratch — TLS certificate generation, high-performance parsing library, SIEM integration, including a complete runnable Golang demo

Introduction: From Basics to Production

at Basics In it, we mastered the syslog protocol format and rsyslog configuration. However, security products in production environments face more severe challenges:

• Encrypted transmission: Logs contain sensitive information, and transmitting them in plaintext poses an extremely high risk.

• High performance: Tens of thousands of logs per second—parsing cannot become the bottleneck.

• Standardization Output format needs to interface with analysis platforms such as SIEM and ELK.

This article will address these issues one by one and deliver Complete, runnable, production-grade code.

• Introduction: From Basics to Production
• 1. Security Architecture Design: Centralized Log Collection Topology
  • 1.1 Typical Deployment Architecture
  • 1.2 Log Pipeline Design Principles for Security Products
• 2. rsyslog receiver configuration: preparing for TLS
  • 2.1 Complete production-grade configuration
  • 2.2 Restart the service and verify
• 3. TLS Certificate Generation: Complete Hands-On Guide
  • 3.1 Generate a self-signed CA
  • 3.2 Certificate File Description
• 4. Golang High-Performance TCP/TLS Receiver
  • 4.1 Introduce a high-performance parsing library
  • 4.2 Complete Production-Grade TLS Receiver
  • 4.3 Operation and Testing
• 5. Golang Sender Implementation
  • 5.1 Standard library syslog example
  • 5.2 Custom TLS Sender (Supports Mutual Authentication)
• 6. Integration with SIEM / Logging Platforms
  • 6.1 JSON Output Format Example
  • 6.2 Integrating with ELK Stack (Filebeat Configuration)
  • 6.3 Integrating with Kafka (Golang Producer)
• 7. Performance Optimization and Production Recommendations
  • 7.1 Performance Benchmark Comparison
  • 7.2 Production Environment Checklist
  • 7.3 Rate Limiting Middleware Example
• 8. Summary
• Reference resources

1. Security Architecture Design: Centralized Log Collection Topology

1.1 Typical Deployment Architecture

1.2 Log Pipeline Design Principles for Security Products

Principle	Description	Implementation approach
Encrypted transmission	Prevent man-in-the-middle eavesdropping	TLS + mutual certificate authentication
Reliable transmission	Avoid log loss	TCP + application layer ACK
Buffer peak shaving	Handle burst traffic	Kafka / Redis buffering
Structured output	Convenient for retrieval and analysis	JSON format + field normalization

2. rsyslog receiver configuration: preparing for TLS

2.1 Complete production-grade configuration

# /etc/rsyslog.conf

# ========== Module Loading ==========
module(load="imtcp")                    # TCP input
module(load="imudp")                    # UDP input (optional)
module(load="imrelp")                   # RELP reliable protocol

# ========== TLS Configuration ==========
global(
    defaultNetstreamDriver="gtls"
    defaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca.pem"
    defaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/server-cert.pem"
    defaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/server-key.pem"
)

# TLS listening (recommended)
input(
    type="imtcp"
    port="5140"
    StreamDriver="gtls"
    StreamDriverMode="1"                # 1=Enable TLS, 0=plaintext
    StreamDriverAuthMode="x509/name"    # Certificate authentication
)

# Plaintext monitoring (compatible with legacy devices; network isolation recommended)
input(type="imtcp" port="514")

# ========== Template: Group into directories by client IP ==========
template(
    name="PerHost"
    type="string"
    string="/var/log/collector/%FROMHOST-IP%/%$YEAR%-%$MONTH%-%$DAY%/%PROGRAMNAME%.log"
)

# ========== Rules ==========
*.*  ?PerHost

2.2 Restart the service and verify

# Verify configuration
sudo rsyslogd -f /etc/rsyslog.conf -N1

# Restart the service
sudo systemctl restart rsyslog

# Check listening ports
sudo ss -tlnp | grep 5140
# Output example: LISTEN 0 25 0.0.0.0:5140 0.0.0.0:* users:(("rsyslogd",pid=1234,fd=18))

3. TLS Certificate Generation: Complete Hands-On Guide

3.1 Generate a self-signed CA

#!/bin/bash
# generate-certs.sh - Generate syslog TLS certificate

# Variable configuration
CA_DIR="./certs"
DAYS=3650
COUNTRY="CN"
STATE="Beijing"
CITY="Beijing"
ORG="SecurityProduct"
ORG_UNIT="LogCollector"

mkdir -p $CA_DIR
cd $CA_DIR

# 1. Generate CA private key and certificate
openssl req -new -x509 -days $DAYS -nodes \
    -newkey rsa:4096 \
    -keyout ca-key.pem \
    -out ca.pem \
    -subj "/C=$COUNTRY/ST=$STATE/L=$CITY/O=$ORG/CN=LogCollector CA"

# 2. Generate the server private key and CSR
openssl req -new -nodes -newkey rsa:4096 \
    -keyout server-key.pem \
    -out server.csr \
    -subj "/C=$COUNTRY/ST=$STATE/L=$CITY/O=$ORG/CN=logcollector.example.com"

# 3. Use the CA to issue a server certificate
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca-key.pem \
    -CAcreateserial -out server-cert.pem -days $DAYS \
    -extfile <(printf "subjectAltName=DNS:localhost,DNS:logcollector.example.com,IP:127.0.0.1")

# 4. Generate a client certificate (for mutual authentication)
openssl req -new -nodes -newkey rsa:4096 \
    -keyout client-key.pem \
    -out client.csr \
    -subj "/C=$COUNTRY/ST=$STATE/L=$CITY/O=$ORG/CN=client"

openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem \
    -CAcreateserial -out client-cert.pem -days $DAYS

# 5. Clean up temporary files
rm *.csr *.srl

# 6. Set permissions
chmod 600 *.key
chmod 644 *.pem

echo "✅ Certificate generation complete!"
ls -la

3.2 Certificate File Description

File	Use	Permission recommendations
ca.pem	CA certificates must be trusted by both the client and the server.	644
ca-key.pem	CA private key, keep it secure.	600
server-cert.pem	Server certificate	644
server-key.pem	server private key	600
client-cert.pem	Client certificate (mutual authentication)	644
client-key.pem	Client private key (mutual authentication)	600

4. Golang High-Performance TCP/TLS Receiver

4.1 Introduce a high-performance parsing library

The standard library has limited syslog parsing capabilities; using a specialized library is recommended:

go get github.com/influxdata/go-syslog/v3
go get github.com/influxdata/go-syslog/v3/rfc5424
go get github.com/influxdata/go-syslog/v3/rfc3164

4.2 Complete Production-Grade TLS Receiver

package main

import (
    "crypto/tls"
    "crypto/x509"
    "encoding/json"
    "flag"
    "fmt"
    "io"
    "log"
    "net"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/influxdata/go-syslog/v3"
    "github.com/influxdata/go-syslog/v3/rfc3164"
    "github.com/influxdata/go-syslog/v3/rfc5424"
)

// Config configuration structure
type Config struct {
    ListenAddr   string
    CertFile     string
    KeyFile      string
    CAFile       string
    EnableTLS    bool
    RequireClientCert bool
    OutputFormat string   // json, raw
    BufferSize   int
}

// LogEntry standardized log entry (for output)
type LogEntry struct {
    Timestamp   string                 `json:"timestamp"`
    Hostname    string                 `json:"hostname"`
    Facility    int                    `json:"facility"`
    Severity    int                    `json:"severity"`
    SeverityStr string                 `json:"severity_str"`
    Tag         string                 `json:"tag,omitempty"`
    Message     string                 `json:"message"`
    Raw         string                 `json:"raw,omitempty"`
    Source      string                 `json:"source"`
    Extra       map[string]interface{} `json:"extra,omitempty"`
}

// Collector Syslog Collector
type Collector struct {
    config     Config
    parser     syslog.Machine
    outputChan chan LogEntry
    stopChan   chan struct{}
}

// NewCollector creates a collector
func NewCollector(config Config) *Collector {
    return &Collector{
        config:     config,
        parser:     syslog.NewParser(rfc3164.NewParser()), // Automatically detect RFC3164/5424
        outputChan: make(chan LogEntry, config.BufferSize),
        stopChan:   make(chan struct{}),
    }
}

// getTLSConfig gets the TLS configuration
func (c *Collector) getTLSConfig() (*tls.Config, error) {
    cert, err := tls.LoadX509KeyPair(c.config.CertFile, c.config.KeyFile)
    if err != nil {
        return nil, fmt.Errorf("load cert failed: %w", err)
    }

    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{cert},
        MinVersion:   tls.VersionTLS12,
    }

    if c.config.RequireClientCert {
        tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert
        caCert, err := os.ReadFile(c.config.CAFile)
        if err != nil {
            return nil, fmt.Errorf("load CA failed: %w", err)
        }
        caPool := x509.NewCertPool()
        if !caPool.AppendCertsFromPEM(caCert) {
            return nil, fmt.Errorf("invalid CA cert")
        }
        tlsConfig.ClientCAs = caPool
    }

    return tlsConfig, nil
}

// parseMessage parses syslog messages
func (c *Collector) parseMessage(raw string, source string) *LogEntry {
    msg, err := c.parser.Parse([]byte(raw))
    if err != nil {
        // Parsing failed, returning original content
        return &LogEntry{
            Timestamp:   time.Now().UTC().Format(time.RFC3339),
            Message:     raw,
            Raw:         raw,
            Source:      source,
            SeverityStr: "unknown",
        }
    }

    entry := &LogEntry{
        Timestamp:   msg.GetTimestamp().Format(time.RFC3339),
        Hostname:    msg.GetHostname(),
        Message:     msg.GetMessage(),
        Raw:         raw,
        Source:      source,
        Extra:       make(map[string]interface{}),
    }

    // Extract Facility and Severity
    if p, ok := msg.GetPriority(); ok {
        entry.Facility = int(p.Facility)
        entry.Severity = int(p.Severity)
        entry.SeverityStr = severityToString(p.Severity)
    }

    // Extract TAG
    if tag, ok := msg.GetTag(); ok {
        entry.Tag = tag
    }

    return entry
}

// severityToString Priority value to string
func severityToString(s uint8) string {
    m := map[uint8]string{
        0: "emerg", 1: "alert", 2: "crit", 3: "err",
        4: "warning", 5: "notice", 6: "info", 7: "debug",
    }
    if str, ok := m[s]; ok {
        return str
    }
    return "unknown"
}

// handleConnection handles a single connection
func (c *Collector) handleConnection(conn net.Conn) {
    defer conn.Close()

    buf := make([]byte, 8192)
    for {
        select {
        case <-c.stopChan:
            return
        default:
        }

        n, err := conn.Read(buf)
        if err != nil {
            if err != io.EOF {
                log.Printf("Read error: %v", err)
            }
            return
        }

        // Process potentially multiple messages (separated by newlines)
        data := buf[:n]
        start := 0
        for i, b := range data {
            if b == '\n' {
                if i > start {
                    raw := string(data[start:i])
                    entry := c.parseMessage(raw, conn.RemoteAddr().String())
                    c.outputChan <- *entry
                }
                start = i + 1
            }
        }
        // Process the last line (may not have a newline character)
        if start < len(data) {
            raw := string(data[start:])
            entry := c.parseMessage(raw, conn.RemoteAddr().String())
            c.outputChan <- *entry
        }
    }
}

// outputProcessor Output Processor (outputs JSON format to stdout)
func (c *Collector) outputProcessor() {
    encoder := json.NewEncoder(os.Stdout)
    for {
        select {
        case <-c.stopChan:
            return
        case entry := <-c.outputChan:
            if c.config.OutputFormat == "json" {
                if err := encoder.Encode(entry); err != nil {
                    log.Printf("JSON encode error: %v", err)
                }
            } else {
                // Original format
                fmt.Printf("[%s] %s: %s\n", entry.Timestamp, entry.Hostname, entry.Message)
            }
        }
    }
}

// Run start collector
func (c *Collector) Run() error {
    // Start output processor
    go c.outputProcessor()

    var ln net.Listener
    var err error

    if c.config.EnableTLS {
        tlsConfig, err := c.getTLSConfig()
        if err != nil {
            return fmt.Errorf("TLS config failed: %w", err)
        }
        ln, err = tls.Listen("tcp", c.config.ListenAddr, tlsConfig)
        log.Printf("🔒 TLS listener started on %s", c.config.ListenAddr)
    } else {
        ln, err = net.Listen("tcp", c.config.ListenAddr)
        log.Printf("📡 Plain TCP listener started on %s", c.config.ListenAddr)
    }
    if err != nil {
        return fmt.Errorf("listen failed: %w", err)
    }
    defer ln.Close()

    // Accept connection
    for {
        select {
        case <-c.stopChan:
            return nil
        default:
        }

        conn, err := ln.Accept()
        if err != nil {
            log.Printf("Accept error: %v", err)
            continue
        }
        go c.handleConnection(conn)
    }
}

// Stop Stop collector
func (c *Collector) Stop() {
    close(c.stopChan)
}

func main() {
    // Command-line arguments
    addr := flag.String("addr", ":5514", "Listen address")
    tlsEnabled := flag.Bool("tls", false, "Enable TLS")
    certFile := flag.String("cert", "certs/server-cert.pem", "TLS certificate file")
    keyFile := flag.String("key", "certs/server-key.pem", "TLS key file")
    caFile := flag.String("ca", "certs/ca.pem", "CA certificate file")
    requireClientCert := flag.Bool("client-auth", false, "Require client certificate")
    outputFormat := flag.String("output", "json", "Output format (json/raw)")
    bufferSize := flag.Int("buffer", 10000, "Output channel buffer size")
    flag.Parse()

    config := Config{
        ListenAddr:        *addr,
        CertFile:          *certFile,
        KeyFile:           *keyFile,
        CAFile:            *caFile,
        EnableTLS:         *tlsEnabled,
        RequireClientCert: *requireClientCert,
        OutputFormat:      *outputFormat,
        BufferSize:        *bufferSize,
    }

    collector := NewCollector(config)

    // Graceful exit
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

    go func() {
        <-sigChan
        log.Println("Shutting down...")
        collector.Stop()
    }()

    if err := collector.Run(); err != nil {
        log.Fatalf("Collector error: %v", err)
    }
}

4.3 Operation and Testing

# Compile
go build -o syslog-collector main.go

# Run in plaintext mode
./syslog-collector -addr ":5514" -output json

# Run in TLS mode
./syslog-collector -tls -addr ":5514" -cert certs/server-cert.pem -key certs/server-key.pem

# Send test logs
echo "<34>Oct 11 22:14:15 mymachine su: 'su root' failed" | nc localhost 5514

5. Golang Sender Implementation

5.1 Standard library syslog example

package main

import (
    "log"
    "log/syslog"
)

func main() {
    // Connect to the local syslog service
    writer, err := syslog.Dial("tcp", "localhost:514", syslog.LOG_INFO|syslog.LOG_AUTH, "myapp")
    if err != nil {
        log.Fatal(err)
    }
    defer writer.Close()

    writer.Alert("Security alert: suspicious login detected")
    writer.Crit("Critical: PAM authentication failure")
    writer.Info("Normal login from 192.168.1.100")
}

5.2 Custom TLS Sender (Supports Mutual Authentication)

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "log"
    "net"
    "os"
    "time"
)

type TLSLogger struct {
    conn     net.Conn
    config   *tls.Config
    address  string
}

// NewTLSLogger creates a TLS log sender
func NewTLSLogger(address, certFile, keyFile, caFile string, requireCert bool) (*TLSLogger, error) {
    config := &tls.Config{
        MinVersion: tls.VersionTLS12,
    }

    // Load client certificate
    if certFile != "" && keyFile != "" {
        cert, err := tls.LoadX509KeyPair(certFile, keyFile)
        if err != nil {
            return nil, fmt.Errorf("load client cert: %w", err)
        }
        config.Certificates = []tls.Certificate{cert}
    }

    // Load CA (server-side certificate verification)
    if caFile != "" {
        caCert, err := os.ReadFile(caFile)
        if err != nil {
            return nil, fmt.Errorf("load CA: %w", err)
        }
        caPool := x509.NewCertPool()
        caPool.AppendCertsFromPEM(caCert)
        config.RootCAs = caPool
    }

    if requireCert {
        config.InsecureSkipVerify = false
    } else {
        config.InsecureSkipVerify = true // Test environment only
    }

    return &TLSLogger{
        config:  config,
        address: address,
    }, nil
}

// Connect establish connection
func (l *TLSLogger) Connect() error {
    conn, err := tls.Dial("tcp", l.address, l.config)
    if err != nil {
        return err
    }
    l.conn = conn
    return nil
}

// Send send syslog message
func (l *TLSLogger) Send(message string) error {
    if l.conn == nil {
        return fmt.Errorf("not connected")
    }
    _, err := l.conn.Write([]byte(message + "\n"))
    return err
}

// Close Close connection
func (l *TLSLogger) Close() error {
    if l.conn != nil {
        return l.conn.Close()
    }
    return nil
}

func main() {
    logger, err := NewTLSLogger("localhost:5140",
        "certs/client-cert.pem",
        "certs/client-key.pem",
        "certs/ca.pem",
        true)
    if err != nil {
        log.Fatal(err)
    }
    defer logger.Close()

    if err := logger.Connect(); err != nil {
        log.Fatal(err)
    }

    // Send test message
    messages := []string{
        "<34>Oct 11 22:14:15 mymachine su: 'su root' failed",
        "<38>Oct 11 22:15:00 mymachine sshd[1234]: Failed password for root",
        "<46>Oct 11 22:16:30 mymachine sudo: user ALL : command not allowed",
    }

    for _, msg := range messages {
        if err := logger.Send(msg); err != nil {
            log.Printf("Send error: %v", err)
        }
        time.Sleep(1 * time.Second)
    }
}

6. Integration with SIEM / Logging Platforms

6.1 JSON Output Format Example

The JSON format output by the collector:

{
  "timestamp": "2024-10-11T22:14:15Z",
  "hostname": "mymachine",
  "facility": 4,
  "severity": 2,
  "severity_str": "crit",
  "tag": "su",
  "message": "'su root' failed for lonvick on /dev/pts/8",
  "source": "192.168.1.100:54321",
  "extra": {}
}

6.2 Integrating with ELK Stack (Filebeat Configuration)

# filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/collector/*/*/*.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "syslog-%{+yyyy.MM.dd}"

setup.template.name: "syslog"
setup.template.pattern: "syslog-*"

6.3 Integrating with Kafka (Golang Producer)

package main

import (
    "encoding/json"
    "github.com/IBM/sarama"
    "log"
)

type KafkaProducer struct {
    producer sarama.SyncProducer
    topic    string
}

func NewKafkaProducer(brokers []string, topic string) (*KafkaProducer, error) {
    config := sarama.NewConfig()
    config.Producer.RequiredAcks = sarama.WaitForAll
    config.Producer.Retry.Max = 5
    config.Producer.Return.Successes = true

    producer, err := sarama.NewSyncProducer(brokers, config)
    if err != nil {
        return nil, err
    }

    return &KafkaProducer{
        producer: producer,
        topic:    topic,
    }, nil
}

func (kp *KafkaProducer) Send(entry LogEntry) error {
    data, err := json.Marshal(entry)
    if err != nil {
        return err
    }

    msg := &sarama.ProducerMessage{
        Topic: kp.topic,
        Value: sarama.ByteEncoder(data),
    }

    _, _, err = kp.producer.SendMessage(msg)
    return err
}

func (kp *KafkaProducer) Close() error {
    return kp.producer.Close()
}

7. Performance Optimization and Production Recommendations

7.1 Performance Benchmark Comparison

Parsing method	QPS	Memory usage	Applicable scenarios
regular expression	~5000	low	Simple scenario
go-syslog library	~20000	Middle	Production Recommendations
Custom state machine	~50000	High	Extreme performance requirements

7.2 Production Environment Checklist

• TLS mutual authentication: Prevent unauthorized clients from sending fake logs

• Current limiting protection Prevent malicious clients from exhausting service resources

• Log rotation: Avoid disk full (use logrotate)

• Monitoring alerts: Prometheus metrics exposure (connection count, parsing error count)

• buffer queue: Use a ring buffer when the output is blocked

7.3 Rate Limiting Middleware Example

type RateLimiter struct {
    limiter *rate.Limiter
}

func NewRateLimiter(r rate.Limit, b int) *RateLimiter {
    return &RateLimiter{
        limiter: rate.NewLimiter(r, b),
    }
}

func (rl *RateLimiter) Middleware(next func(net.Conn)) func(net.Conn) {
    return func(conn net.Conn) {
        if !rl.limiter.Allow() {
            log.Printf("Rate limit exceeded, dropping connection from %s", conn.RemoteAddr())
            conn.Close()
            return
        }
        next(conn)
    }
}

8. Summary

This article completes the full evolution from basics to production:

• ✅TLS Certificate Generation: Self-signed CA + server/client certificates

• ✅Production-grade receiver: Supports TLS mutual authentication, high-performance parsing, and JSON output

• ✅Sender implementation: Standard library + custom TLS sender

• ✅SIEM integration: ELK / Kafka Integration Solution

• ✅Performance optimization: Rate limiting, buffering, and monitoring recommendations

Next article preview: <Syslog Comparison: A Guide to Choosing Between syslog and journald Logs This will delve into the selection decisions for modern Linux dual-logging systems and provide an implementation of a unified log-reading layer in Golang.

Reference resources

• go-syslog documentation

• rsyslog TLS Configuration Guide

• OpenSSL Certificate Generation Manual

Series Articles

• Syslog Basics: Protocol Anatomy and rsyslog Configuration

• Syslog Practical Guide: Golang Security Product Integration Guide

• Syslog Comparison: A Guide to Choosing Between syslog and journald Logs