django - How to Include a Message Field in Structlog Logs and Best Practices for ElasticSearch Integration - Stack Overflow

I'm working on a Django project where logging is critical, and I'm using structlog to format and manage logs. The plan is to send these logs to ElasticSearch. However, I've encountered an issue: the logs are missing the "message" field, even though I explicitly pass a message in the logger call.

Here’s the log output I currently get:

{
    "code": 200,
    "request": "POST /api/push-notifications/subscribe/",
    "event": "request_finished",
    "ip": "127.0.0.1",
    "request_id": "d0edd77d-d68b-49d8-9d0d-87ee6ff723bf",
    "user_id": "98c78a2d-57f1-4caa-8b2a-8f5c4e295f95",
    "timestamp": "2025-01-21T10:40:43.233334Z",
    "logger": "django_structlog.middlewares.request",
    "level": "info"
}

What I want is to include the "message" field, for example:

{
    "code": 200,
    "request": "POST /api/push-notifications/subscribe/",
    "event": "request_finished",
    "ip": "127.0.0.1",
    "request_id": "d0edd77d-d68b-49d8-9d0d-87ee6ff723bf",
    "user_id": "98c78a2d-57f1-4caa-8b2a-8f5c4e295f95",
    "timestamp": "2025-01-21T10:40:43.233334Z",
    "logger": "django_structlog.middlewares.request",
    "level": "info", 
    "message": "push notification subscribed successfully"
}

Here’s my current setup: settings.py Logger Configuration

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    "formatters": {
        "json_formatter": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.processors.JSONRenderer(),
        },
        "plain_console": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.dev.ConsoleRenderer(),
        },
        "key_value": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.processors.KeyValueRenderer(key_order=['timestamp', 'level', 'event', 'message']),
        },
    },
    'handlers': {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "plain_console",
        },
        "json_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "logs/ft_json.log",
            "formatter": "json_formatter",
            "maxBytes": 1024 * 1024 * 5,
            "backupCount": 3,
        },
        "flat_line_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "logs/flat_line.log",
            "formatter": "key_value",
            "maxBytes": 1024 * 1024 * 5,
            "backupCount": 3,
        },
    },
    "loggers": {
        "django_structlog": {
            "level": "INFO",
            "handlers": ["console", "flat_line_file", "json_file"],
            "propagate": True,
        },
        "ft_log": {
            "level": "INFO",
            "handlers": ["console", "flat_line_file", "json_file"],
            "propagate": False,
        },
    },
}

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.filter_by_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    cache_logger_on_first_use=True,
)

views.py Example:

import structlog
logger = structlog.get_logger(__name__)

def subscribe(request):
    """Subscribes the authenticated user to push notifications."""
    logger.info("push notification subscribed successfully!")

Despite calling logger.info with a message, it doesn’t appear in the logs. How can I ensure that the "message" field is included?

Also, any suggestions or best practices for posting structured logs into ElasticSearch would be greatly appreciated.

I'm working on a Django project where logging is critical, and I'm using structlog to format and manage logs. The plan is to send these logs to ElasticSearch. However, I've encountered an issue: the logs are missing the "message" field, even though I explicitly pass a message in the logger call.

Here’s the log output I currently get:

{
    "code": 200,
    "request": "POST /api/push-notifications/subscribe/",
    "event": "request_finished",
    "ip": "127.0.0.1",
    "request_id": "d0edd77d-d68b-49d8-9d0d-87ee6ff723bf",
    "user_id": "98c78a2d-57f1-4caa-8b2a-8f5c4e295f95",
    "timestamp": "2025-01-21T10:40:43.233334Z",
    "logger": "django_structlog.middlewares.request",
    "level": "info"
}

What I want is to include the "message" field, for example:

{
    "code": 200,
    "request": "POST /api/push-notifications/subscribe/",
    "event": "request_finished",
    "ip": "127.0.0.1",
    "request_id": "d0edd77d-d68b-49d8-9d0d-87ee6ff723bf",
    "user_id": "98c78a2d-57f1-4caa-8b2a-8f5c4e295f95",
    "timestamp": "2025-01-21T10:40:43.233334Z",
    "logger": "django_structlog.middlewares.request",
    "level": "info", 
    "message": "push notification subscribed successfully"
}

Here’s my current setup: settings.py Logger Configuration

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    "formatters": {
        "json_formatter": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.processors.JSONRenderer(),
        },
        "plain_console": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.dev.ConsoleRenderer(),
        },
        "key_value": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": structlog.processors.KeyValueRenderer(key_order=['timestamp', 'level', 'event', 'message']),
        },
    },
    'handlers': {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "plain_console",
        },
        "json_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "logs/ft_json.log",
            "formatter": "json_formatter",
            "maxBytes": 1024 * 1024 * 5,
            "backupCount": 3,
        },
        "flat_line_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "logs/flat_line.log",
            "formatter": "key_value",
            "maxBytes": 1024 * 1024 * 5,
            "backupCount": 3,
        },
    },
    "loggers": {
        "django_structlog": {
            "level": "INFO",
            "handlers": ["console", "flat_line_file", "json_file"],
            "propagate": True,
        },
        "ft_log": {
            "level": "INFO",
            "handlers": ["console", "flat_line_file", "json_file"],
            "propagate": False,
        },
    },
}

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.filter_by_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    cache_logger_on_first_use=True,
)

views.py Example:

import structlog
logger = structlog.get_logger(__name__)

def subscribe(request):
    """Subscribes the authenticated user to push notifications."""
    logger.info("push notification subscribed successfully!")

Despite calling logger.info with a message, it doesn’t appear in the logs. How can I ensure that the "message" field is included?

Also, any suggestions or best practices for posting structured logs into ElasticSearch would be greatly appreciated.

• django
• elasticsearch
• logging
• django-rest-framework
• structlog

Share Improve this question Follow asked Jan 21 at 15:11 Keshav KhanalKeshav Khanal 3133 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

The message field in structlog is the event field. In other words: the log message you're showing is not the log message you're passing "push notification subscribed successfully!".

I don't think your logger's name is "django_structlog" (I suspect this is coming from an example), so I