Metrics

Moleculer has a built-in metrics module that collects a lot of internal Moleculer & process metric values. Moreover, you can easily define your custom metrics. There are several built-in metrics reporters like Console, Prometheus, Datadog, etc.

If you want to use legacy (<= v0.13) metrics use EventLegacy tracing exporter. More info.

Enable metrics & define console reporter

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            "Console"
        ]
    }
};

Options

Name	Type	Default	Description
enabled	Boolean	false	Enable tracing feature.
reporter	Object or Array<Object>	null	Metric reporter configuration. More info
collectProcessMetrics	Boolean		Collect process & OS related metrics. Default: process.env.NODE_ENV !== "test"
collectInterval	Number	5	Collect time period in seconds.
defaultBuckets	Array<Number>		Default bucket values for histograms. Default: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
defaultQuantiles	Array<Number>		Default quantiles for histograms. Default: [0.5, 0.9, 0.95, 0.99, 0.999]
defaultMaxAgeSeconds	Number	60	Max age seconds for quantile calculation.
defaultAgeBuckets	Number	10	Number of buckets for quantile calculation.
defaultAggregator	String	sum	Value aggregator method.

Metrics Reporters

Moleculer has several built-in reporters. All of them have the following options:

Name	Type	Default	Description
includes	String or Array<String>	null	List of metrics to be exported. Default metrics
excludes	String or Array<String>	null	List of metrics to be excluded. Default metrics
metricNamePrefix	String	null	Prefix to be added to metric names
metricNameSuffix	String	null	Suffix to be added to metric names
metricNameFormatter	Function	null	Metric name formatter
labelNameFormatter	Function	null	Label name formatter

Example of metrics options

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "Console",
                options: {
                    includes: ["moleculer.**.total"],
                    excludes: ["moleculer.broker.**","moleculer.request.**"],
                    
                    metricNamePrefix: "mol:", // Original "moleculer.node.type". With prefix: "mol:moleculer.node.type" 
                    metricNameSuffix: ".value", // Original "moleculer.node.type". With prefix: "moleculer.node.type.value"
                
                    metricNameFormatter: name => name.toUpperCase().replace(/[.:]/g, "_"),
                    labelNameFormatter: name => name.toUpperCase().replace(/[.:]/g, "_")
                }
            }
        ]
    }
};

Console

This is a debugging reporter which periodically prints the metrics to the console.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "Console",
                options: {
                    // Printing interval in seconds
                    interval: 5,
                    // Custom logger.
                    logger: null,
                    // Using colors
                    colors: true,
                    // Prints only changed metrics, not the full list.
                    onlyChanges: true
                }
            }
        ]
    }
};

CSV

Comma-Separated Values (CSV) reporter saves changes to a CSV file.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "CSV",
                options: {
                    // Folder of CSV files.
                    folder: "./reports/metrics",
                    // CSV field delimiter
                    delimiter: ",",
                    // CSV row delimiter
                    rowDelimiter: "\n",
                    // Saving mode. 
                    //   - "metric" - save metrics to individual files
                    //   - "label" - save metrics by labels to individual files
                    mode: "metric",
                    // Saved metrics types.
                    types: null,
                    // Saving interval in seconds
                    interval: 5,
                    // Custom filename formatter
                    filenameFormatter: null,
                    // Custom CSV row formatter.
                    rowFormatter: null,
                }
            }
        ]
    }
};

Event

Event reporter sends Moleculer events with metric values.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "Event",
                options: {
                    // Event name
                    eventName: "$metrics.snapshot",
                    // Broadcast or emit
                    broadcast: false,
                    // Event groups
                    groups: null,
                    // Send only changed metrics
                    onlyChanges: false,
                    // Sending interval in seconds
                    interval: 5,
                }
            }
        ]
    }
};

Datadog

Datadog reporter sends metrics to the Datadog server.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "Datadog",
                options: {
                    // Hostname
                    host: "my-host",
                    // Base URL
                    baseUrl: "https://api.datadoghq.eu/api/", // Default is https://api.datadoghq.com/api/
                    // API version
                    apiVersion: "v1",
                    // Server URL path
                    path: "/series",
                    // Datadog API Key
                    apiKey: process.env.DATADOG_API_KEY,
                    // Default labels which are appended to all metrics labels
                    defaultLabels: (registry) => ({
                        namespace: registry.broker.namespace,
                        nodeID: registry.broker.nodeID
                    }),
                    // Sending interval in seconds
                    interval: 10
                }
            }
        ]
    }
};

Prometheus

Prometheus reporter publishes metrics in Prometheus format. The Prometheus server can collect them. Default port is 3030.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "Prometheus",
                options: {
                    // HTTP port
                    port: 3030,
                    // HTTP URL path
                    path: "/metrics",
                    // Default labels which are appended to all metrics labels
                    defaultLabels: registry => ({
                        namespace: registry.broker.namespace,
                        nodeID: registry.broker.nodeID
                    })
                }
            }
        ]
    }
};

StatsD

The StatsD reporter sends metric values to StatsD server via UDP.

// moleculer.config.js
module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            {
                type: "StatsD",
                options: {
                    // Server host
                    host: "localhost",
                    // Server port
                    port: 8125,
                    // Maximum payload size.
                    maxPayloadSize: 1300
                }
            }
        ]
    }
};

Customer Reporter

Custom metrics module can be created. We recommend to copy the source of Console Reporter and implement the init, stop, metricChanged methods.

Create custom metrics

const BaseReporter = require("moleculer").MetricReporters.Base;

class MyMetricsReporter extends BaseReporter {
    init() { /*...*/ }
    stop() { /*...*/ }
    metricChanged() { /*...*/ }
}

Use custom metrics

// moleculer.config.js
const MyMetricsReporter = require("./my-metrics-reporter");

module.exports = {
    metrics: {
        enabled: true,
        reporter: [
            new MyMetricsReporter(),
        ]
    }
};

Supported Metric Types

Counter

A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero. For example, you can use a counter to represent the number of requests served, tasks completed, or errors. It can also provide 1-minute rate.

Counter provides the following methods

increment(labels?: GenericObject, value?: number, timestamp?: number)
set(value: number, labels?: GenericObject, timestamp?: number)

Gauge

A gauge is a metric that represents a single numerical value that can arbitrarily go up and down. Gauges are typically used for measured values like current memory usage, but also “counts” that can go up and down, like the number of concurrent requests. It can also provide 1-minute rate.

Gauge provides the following methods:

increment(labels?: GenericObject, value?: number, timestamp?: number)
decrement(labels?: GenericObject, value?: number, timestamp?: number)
set(value: number, labels?: GenericObject, timestamp?: number)

Histogram

A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values and calculates configurable quantiles over a sliding time window. It can also provide 1-minute rate.

Histogram provides the following methods:

observe(value: number, labels?: GenericObject, timestamp?: number)

Info

An info is a single string or number value like process arguments, hostname or version numbers.

Info provides the following methods:

set(value: any | null, labels?: GenericObject, timestamp?: number)

Built-in Internal Metrics

Process metrics

• process.arguments (info)
• process.pid (info)
• process.ppid (info)
• process.eventloop.lag.min (gauge)
• process.eventloop.lag.avg (gauge)
• process.eventloop.lag.max (gauge)
• process.eventloop.lag.count (gauge)
• process.memory.heap.size.total (gauge)
• process.memory.heap.size.used (gauge)
• process.memory.rss (gauge)
• process.memory.external (gauge)
• process.memory.heap.space.size.total (gauge)
• process.memory.heap.space.size.used (gauge)
• process.memory.heap.space.size.available (gauge)
• process.memory.heap.space.size.physical (gauge)
• process.memory.heap.stat.heap.size.total (gauge)
• process.memory.heap.stat.executable.size.total (gauge)
• process.memory.heap.stat.physical.size.total (gauge)
• process.memory.heap.stat.available.size.total (gauge)
• process.memory.heap.stat.used.heap.size (gauge)
• process.memory.heap.stat.heap.size.limit (gauge)
• process.memory.heap.stat.mallocated.memory (gauge)
• process.memory.heap.stat.peak.mallocated.memory (gauge)
• process.memory.heap.stat.zap.garbage (gauge)
• process.uptime (gauge)
• process.internal.active.handles (gauge)
• process.internal.active.requests (gauge)
• process.versions.node (info)
• process.gc.time (gauge)
• process.gc.total.time (gauge)
• process.gc.executed.total (gauge)

OS metrics

• os.memory.free (gauge)
• os.memory.total (gauge)
• os.memory.used (gauge)
• os.uptime (gauge)
• os.type (info)
• os.release (info)
• os.hostname (info)
• os.arch (info)
• os.platform (info)
• os.user.uid (info)
• os.user.gid (info)
• os.user.username (info)
• os.user.homedir (info)
• os.network.address (info)
• os.network.mac (info)
• os.datetime.unix (gauge)
• os.datetime.iso (info)
• os.datetime.utc (info)
• os.datetime.tz.offset (gauge)
• os.cpu.load.1 (gauge)
• os.cpu.load.5 (gauge)
• os.cpu.load.15 (gauge)
• os.cpu.utilization (gauge)
• os.cpu.user (gauge)
• os.cpu.system (gauge)
• os.cpu.total (gauge)
• os.cpu.info.model (info)
• os.cpu.info.speed (gauge)
• os.cpu.info.times.user (gauge)
• os.cpu.info.times.sys (gauge)

Moleculer metrics

• moleculer.node.type (info)
• moleculer.node.versions.moleculer (info)
• moleculer.node.versions.protocol (info)
• moleculer.broker.namespace (info)
• moleculer.broker.started (gauge)
• moleculer.broker.local.services.total (gauge)
• moleculer.broker.middlewares.total (gauge)
• moleculer.registry.nodes.total (gauge)
• moleculer.registry.nodes.online.total (gauge)
• moleculer.registry.services.total (gauge)
• moleculer.registry.service.endpoints.total (gauge)
• moleculer.registry.actions.total (gauge)
• moleculer.registry.action.endpoints.total (gauge)
• moleculer.registry.events.total (gauge)
• moleculer.registry.event.endpoints.total (gauge)
• moleculer.request.bulkhead.inflight (gauge)
• moleculer.request.bulkhead.queue.size (gauge)
• moleculer.event.bulkhead.inflight (gauge)
• moleculer.event.bulkhead.queue.size (gauge)
• moleculer.event.received.time (histogram)
• moleculer.event.received.error.total(counter)
• moleculer.event.received.active (gauge)
• moleculer.request.timeout.total (counter)
• moleculer.request.retry.attempts.total (counter)
• moleculer.request.fallback.total (counter)
• moleculer.request.total (counter)
• moleculer.request.active (gauge)
• moleculer.request.error.total (counter)
• moleculer.request.time (histogram)
• moleculer.request.levels (counter)
• moleculer.event.emit.total (counter)
• moleculer.event.broadcast.total (counter)
• moleculer.event.broadcast-local.total (counter)
• moleculer.event.received.total (counter)
• moleculer.transit.publish.total (counter)
• moleculer.transit.receive.total (counter)
• moleculer.transit.requests.active (gauge)
• moleculer.transit.streams.send.active (gauge)
• moleculer.transporter.packets.sent.total (counter)
• moleculer.transporter.packets.sent.bytes (counter)
• moleculer.transporter.packets.received.total (counter)
• moleculer.transporter.packets.received.bytes (counter)

Customizing

New metric registration

You can easily create custom metrics.

Create a counter

// posts.service.js
module.exports = {
    name: "posts",

    actions: {
        // Get posts.
        get(ctx) {
            // Increment metric
            this.broker.metrics.increment("posts.get.total", 1);

            return this.posts;
        }
    },

    created() {
        // Register new counter metric
        this.broker.metrics.register({ 
            type: "counter", 
            name: "posts.get.total", 
            description: "Number of requests of posts", 
            unit: "request",
            rate: true // calculate 1-minute rate
        });
    }
};

Create a gauge with labels

// posts.service.js
module.exports = {
    name: "posts",

    actions: {
        // Create a new post
        create(ctx) {
            // Update metrics
            this.broker.metrics.increment("posts.total", { userID: ctx.params.author }, 1);
            return posts;
        },

        // Remove a new post
        remove(ctx) {
            // Update metrics
            this.broker.metrics.decrement("posts.total", { userID: ctx.params.author }, 1);
            return posts;
        },

    },

    created() {
        // Register new gauge metric
        this.broker.metrics.register({ 
            type: "gauge", 
            name: "posts.total", 
            labelNames: ["userID"]
            description: "Number of posts by user",
            unit: "post"
        });
    }
};

Create a histogram with buckets & quantiles

// posts.service.js
module.exports = {
    name: "posts",

    actions: {
        // Create a new post
        create(ctx) {
            // Measure the post creation time
            const timeEnd = this.broker.metrics.timer("posts.creation.time");
            const post = await this.adapter.create(ctx.params);
            const duration = timeEnd();
            
            this.logger.debug("Post created. Elapsed time: ", duration, "ms");
            return post;
        }
    },

    created() {
        // Register new histogram metric
        this.broker.metrics.register({ 
            type: "histogram", 
            name: "posts.creation.time", 
            description: "Post creation time",
            unit: "millisecond",
            linearBuckets: {
                start: 0,
                width: 100,
                count: 10
            },
            quantiles: [0.5, 0.9, 0.95, 0.99],
            maxAgeSeconds: 60,
            ageBuckets: 10
        });
    }
};