Tracing

Moleculer has a built-in tracing module that collects tracing information inside a Moleculer application. Moreover, you can easily create your custom tracing spans. There are several built-in tracing exporter like Zipkin, Jaeger, Datadog, etc.

Enable tracing

// moleculer.config.js
module.exports = {
    tracing: true
};

Enable tracing with options

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: "Console",
        events: true,
        stackTrace: true
    }
};

Options

Name	Type	Default	Description
enabled	Boolean	false	Enable tracing feature.
exporter	Object or Array<Object>	null	Tracing exporter configuration. More info
sampling	Object		Sampling settings. More info
actions	Boolean	true	Tracing the service actions.
events	Boolean	false	Tracing the service events.
errorFields	Array<String>	["name", "message", "code", "type", "data"]	Error object fields which are added into span tags.
stackTrace	Boolean	false	Add stack trace info into span tags in case of error.
tags	Object	null	Add custom span tags to all actions and events spans. More info
defaultTags	Object	null	Default tags. It will be added to all spans.

Sampling

The Moleculer Tracer supports several sampling methods. The determination whether to sample or not is made on the root span and propagated to all child spans. This ensures that a complete trace is always exported regardless of the sample method or the rate selected.

Constant sampling

This sampling method uses a constant sampling rate value from 0 to 1. The 1 means all spans will be sampled, the 0 means none of them.

Samples all spans

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        sampling: {
            rate: 1.0
        }
    }
};

Samples half of all spans

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        sampling: {
            rate: 0.5
        }
    }
};

Rate limiting sampling

This sampling method uses a rate limiter. You can configure how many spans will be sampled in a second.

Samples 2 spans per second

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        sampling: {
            tracesPerSecond: 2
        }
    }
};

Samples 1 span per 10 seconds

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        sampling: {
            tracesPerSecond: 0.1
        }
    }
};

Tracing Exporters

The tracing module supports several exporters, custom tracing spans and integration with instrumentation libraries (like dd-trace).

Console

This is a debugging exporter which prints full local trace to the console.

Console exporter can’t follow remote calls, only locals.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: {
            type: "Console",
            options: {
                // Custom logger
                logger: null,
                // Using colors
                colors: true,
                // Width of row
                width: 100,
                // Gauge width in the row
                gaugeWidth: 40
            }
        }
    }
};

Datadog

Datadog exporter sends tracing data to Datadog server via dd-trace.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: {
            type: "Datadog",
            options: {
                // Datadog Agent URL
                agentUrl: process.env.DD_AGENT_URL || "http://localhost:8126",
                // Environment variable
                env: process.env.DD_ENVIRONMENT || null,
                // Sampling priority. More info: https://docs.datadoghq.com/tracing/guide/trace_sampling_and_storage/?tab=java#sampling-rules
                samplingPriority: "AUTO_KEEP",
                // Default tags. They will be added into all span tags.
                defaultTags: null,
                // Custom Datadog Tracer options. More info: https://datadog.github.io/dd-trace-js/#tracer-settings
                tracerOptions: null,
            }
        }
    }
};

To use this exporter, install the dd-trace module with npm install dd-trace --save command.

Event

Event exporter sends Moleculer events ($tracing.spans) with tracing data.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: {
            type: "Event",
            options: {
                // Name of event
                eventName: "$tracing.spans",
                // Send event when a span started
                sendStartSpan: false,
                // Send event when a span finished
                sendFinishSpan: true,
                // Broadcast or emit event
                broadcast: false,
                // Event groups
                groups: null,
                // Sending time interval in seconds
                interval: 5,
                // Custom span object converter before sending
                spanConverter: null,
                // Default tags. They will be added into all span tags.
                defaultTags: null
            }
        }
    }
};

Event (legacy)

Legacy event exporter sends Moleculer legacy metric events (metrics.trace.span.start & metrics.trace.span.finish) at every request. These events are also used to generate metrics in legacy (<= v0.13) metrics solutions.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: "EventLegacy"
    }
};

Legacy Request Started Payload

The broker emits an metrics.trace.span.start event when a new request is started.
The payload looks like the following:

{
    // Context ID
    id: '4563b09f-04cf-4891-bc2c-f26f80c3f91e',
    // Request ID
    requestID: '6858979d-3298-4a7b-813a-ffb417da822b',
    // Level of call
    level: 1,
    // Start time
    startTime: 1493903164726,
    // Is it a remote call
    remoteCall: false,
    // Called action
    action: {
        name: 'users.get'
    },
    // Called service
    service: {
        name: "users"
    },
    // Params
    params: {
        id: 5
    },
    // Meta
    meta: {},
    // Node ID
    nodeID: "node-1",
    // Caller nodeID if it's requested from a remote node
    callerNodeID: "node-2",
    // Parent context ID if it is a sub-call
    parentID: null
}

Legacy Request Finished Payload

The broker emits an metrics.trace.span.finish event when the call/request is finished.
The payload looks like the following:

{
    // Context ID
    id: '4563b09f-04cf-4891-bc2c-f26f80c3f91e',
    // Request ID
    requestID: '6858979d-3298-4a7b-813a-ffb417da822b',
    // Level of call
    level: 1,
    // Start time
    startTime: 1493903164726,
    // End time
    endTime: 1493903164731.3684,
    // Duration of request
    duration: 5.368304,
    // Is it a remote call
    remoteCall: false,
    // Is it resolved from cache
    fromCache: false,
    // Called action
    action: {
        name: 'users.get'
    },
    // Called service
    service: {
        name: "users"
    },
    // Params
    params: {
        id: 5
    },
    // Meta
    meta: {},   
    // Node ID
    nodeID: "node-1",
    // Caller nodeID if it's a remote call
    callerNodeID: "node-2",
    // Parent context ID if it is a sub-call
    parentID: null,
    // Error if the call returned with error
    error: {
        name: "ValidationError",
        message: "Invalid incoming parameters"
    }
}

Jaeger

Jaeger exporter sends tracing spans information to a Jaeger server.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: {
            type: "Jaeger",
            options: {
                // HTTP Reporter endpoint. If set, HTTP Reporter will be used.
                endpoint: null,
                // UDP Sender host option.
                host: "127.0.0.1",
                // UDP Sender port option.
                port: 6832,
                // Jaeger Sampler configuration.
                sampler: {
                    // Sampler type. More info: https://www.jaegertracing.io/docs/1.14/sampling/#client-sampling-configuration
                    type: "Const",
                    // Sampler specific options.
                    options: {}
                },
                // Additional options for `Jaeger.Tracer`
                tracerOptions: {},
                // Default tags. They will be added into all span tags.
                defaultTags: null
            }
        }
    }
};

To use this exporter, install the jaeger-client module with npm install jaeger-client --save command.

Zipkin

Zipkin exporter sends tracing spans information to a Zipkin server.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: {
            type: "Zipkin",
            options: {
                // Base URL for Zipkin server.
                baseURL: "http://localhost:9411",
                // Sending time interval in seconds.
                interval: 5,
                // Additional payload options.
                payloadOptions: {
                    // Set `debug` property in payload.
                    debug: false,
                    // Set `shared` property in payload.
                    shared: false
                },
                // Default tags. They will be added into all span tags.
                defaultTags: null
            }
        }
    }
};

NewRelic

NewRelic exporter sends tracing spans information in Zipkin v2 format to a NewRelic server.

// moleculer.config.js
{
    tracing: {
        enabled: true,
        events: true,
        exporter: [
            {
                type: 'NewRelic',
                options: {
                    // Base URL for NewRelic server
                    baseURL: 'https://trace-api.newrelic.com',
                    // NewRelic Insert Key
                    insertKey: 'my-secret-key',
                    // Sending time interval in seconds.
                    interval: 5,
                    // Additional payload options.
                    payloadOptions: {
                        // Set `debug` property in payload.
                        debug: false,
                        // Set `shared` property in payload.
                        shared: false,
                    },
                    // Default tags. They will be added into all span tags.
                    defaultTags: null,
                },
            },
        ],
    },    
}

Customer Exporter

Custom tracing module can be created. We recommend to copy the source of Console Exporter and implement the init, stop, spanStarted and spanFinished methods.

Create custom tracing

const TracerBase = require("moleculer").TracerExporters.Base;

class MyTracingExporters extends TracerBase {
    init() { /*...*/ }
    stop() { /*...*/ }
    spanStarted() { /*...*/ }
    spanFinished() { /*...*/ }
}

Use custom tracing

// moleculer.config.js
const MyTracingExporters = require("./my-tracing-exporter");

module.exports = {
    tracing: {
        enabled: true,
        exporter: [
            new MyTracingExporters(),
        ]
    }
};

Multiple exporters

You can define multiple tracing exporters.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: [
            "Console",
            {
                type: "Zipkin",
                options: {
                    baseURL: "http://localhost:9411",
                }
            }
            {
                type: "Jaeger",
                options: {
                    host: "127.0.0.1",
                }
            }
        ]
    }
};

User-defined tracing spans

To add new spans inside an action or event handler, just call the ctx.startSpan and ctx.finishSpan methods.

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        async find(ctx) {
            const span1 = ctx.startSpan("get data from DB", {
                tags: {
                    ...ctx.params
                }
            }); 
            const data = await this.getDataFromDB(ctx.params);
            ctx.finishSpan(span1);

            const span2 = ctx.startSpan("populating");
            const res = await this.populate(data);
            ctx.finishSpan(span2);

            return res;
        }
    }
};

Create span without context

If Context is not available, you can create spans via broker.tracer.

// posts.service.js
module.exports = {
    name: "posts",
    started() {
        // Create a span to measure the initialization
        const span = this.broker.tracer.startSpan("initializing db", {
            tags: {
                dbHost: this.settings.dbHost
            }
        });

        await this.db.connect(this.settings.dbHost);

        // Create a sub-span to measure the creating tables.
        const span2 = span.startSpan("create tables");

        await this.createDatabaseTables();

        // Finish the sub-span.
        span2.finish();

        // Finish the main span.
        span.finish();
    }
};

Connecting spans while using external communication module

It is possible to connect the spans even while communicating via external queue (e.g., moleculer-channels). To do it you just need to pass the parentID and requestID to the handler and then use those IDs to start a custom span.

Connecting spans

module.exports = {
	name: "trace",
	actions: {
		async extractTraces(ctx) {
			// Extract the parentID and the requestID from context
			const { parentID, requestID: traceID } = ctx;

			// Send parentID and traceID as payload via an external queue
			await this.broker.sendToChannel("trace.setSpanID", {
				// Send the IDs in the payload
				parentID,
				traceID,
			});
		},
	},

	// More info about channels here: https://github.com/moleculerjs/moleculer-channels
	channels: {
		"trace.setSpanID"(payload) {
			// Init custom span with the original parentID and requestID
			const span = this.broker.tracer.startSpan("my.span", payload);

			// ... logic goes here

			span.finish(); // Finish the custom span
		},
	},
};

Customizing

Custom Span Names

You can customize the span name of you traces. In this case, you must specify the spanName that must be a static String or a Function.

Creating a custom name for a trace via Function

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                spanName: ctx => `Get a post by ID: ${ctx.params.id}`
            },
            async handler(ctx) {
                // ...
            }
        }
    }
};

Adding Tags from Context

You can customize what context params or meta values are added to span tags.

Default
The default behaviour is that add all properties from ctx.params only.

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                // Add all params without meta
                tags: {
                    params: true,
                    meta: false,
                }
            },
            async handler(ctx) {
                // ...
            }
        }
    }
};

Custom params example

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                tags: {
                    // Add `id` from `ctx.params`
                    params: ["id"],
                    // Add `loggedIn.username` value from `ctx.meta`
                    meta: ["loggedIn.username"],
                    // add tags from the action response.
                    response: ["id", "title"]
                }
            },
            async handler(ctx) {
                // ...
            }
        }
    }
};

Example with custom function
You can define a custom Function to fill the span tags from the Context.

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                tags(ctx, response) {
                    return {
                        params: ctx.params,
                        meta: ctx.meta,
                        custom: {
                            a: 5
                        },
                        response
                    };
                }
            },
            async handler(ctx) {
                // ...
            }
        }
    }
};

Please note, when used with an action the function will be called two times in case of successful execution. First with ctx and the second time with ctx & response as the response of the action call.

Global action and event tags

Custom action and event span tags can be defined using the tags property in the tracer options. These will be applied to all action and event spans unless overridden in the service schema’s action and event definitions. All custom tag types defined above are valid. Any tags defined in the service schema’s action and event definitions will take precendence but the merge of params, meta, and response tag definitions are shallow, meaning that it is possible to do things like define meta tags globally and response tags locally in each service.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        tags: {
            action: {
                // Never add params
                params: false,
                // Add `loggedIn.username` value from `ctx.meta`
                meta: ["loggedIn.username"],
                // Always add the response
                response: true,
            },
            event(ctx) {
                return {
                    params: ctx.params,
                    meta: ctx.meta,
                    // add the caller
                    caller: ctx.caller,
                    custom: {
                        a: 5
                    },
                };
            },            
        }
    }
};

Custom tags defined using the tags property have access to ctx and if used with an action the response. The tags defined in defaultTags must either be a static object or a function that accepts the tracer instance and returns an object. It also has access to the broker instance via the tracer instance but does not have access to ctx.

Example of Event tracing
You can tracing the events, as well. To enable it, set events: true in tracing broker options.

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        events: true
    }
};

safetyTags and Maximum call stack error

In general, sending non-serializable parameters (e.g. http request, socket instance, stream instance, etc.) in ctx.params or ctx.meta is not recommended. If tracing is enabled, the tracer exporter will try to recursively flatten these params (with flattenTags method) which will cause the Maximum call stack error.

To avoid this issue, you can use the safetyTags option in exporter options. If set to true, the exporters remove the cyclic properties before flattening the tags in the spans. This option is available in all built-in exporters.

Performance impact

Please note, this option has a significant impact in performance. For this reason it’s not enabled by default.

Enabling globally the safetyTags

// moleculer.config.js
{
    tracing: {
        exporter: [{
            type: "Zipkin",
            options: {
                safetyTags: true,
                baseURL: "http://127.0.0.1:9411"
            }
        }]
    }
}

To avoid affecting all actions, you can enable this function at action-level. In this case, the remaining actions will be unaffected.
Enabling safetyTags at action-level

broker.createService({
    name: "greeter",
    actions: {
        hello: {
            tracing: {
                safetyTags: true
            },
            handler(ctx) {
                return `Hello!`;
            }
        }
    }
});