Overview
The notification and alerting system allows you to:- Configure notification channels for external integrations
- Create alerting monitors that detect security events
- Send notifications to Slack, PagerDuty, Jira, and other services
- Automate incident response workflows
- Customize alert messages and severity levels
Architecture
The system consists of two main components:Notification Channels
Notification channels define the destination and delivery method for alerts. Supported channel types include:- Slack - Send messages to Slack channels
- PagerDuty - Create incidents in PagerDuty
- Jira - Create issues in Jira projects
- Shuffle - Trigger automation workflows
- Webhook - Send HTTP POST requests to custom endpoints
- Email - Send email notifications (requires SMTP configuration)
- Amazon SNS - Publish to Amazon Simple Notification Service topics
- Custom Webhook - Advanced webhook configurations with custom payloads
Alerting Monitors
Monitors continuously evaluate queries against Wazuh data and trigger notifications when conditions are met. The Dashboard health check system can automatically create sample monitors for quick integration testing.Notification Channels
Channel Configuration
Notification channels are configured through the OpenSearch Notifications plugin. The Wazuh Dashboard health check system can automatically create default channels for common integrations.Default Channel IDs
The health check system creates channels with predefined IDs for easy reference:| Channel Type | Default ID |
|---|---|
| Slack | wazuh-slack |
| PagerDuty | wazuh-pagerduty |
| Jira | wazuh-jira |
| Shuffle | wazuh-shuffle |
Creating Notification Channels
Channels can be created through:- Dashboard UI: Navigate to OpenSearch Dashboards > Notifications
- API: Use the Notifications plugin REST API
- Health Check: Automatic creation during Dashboard initialization
Configuring Integrations
Slack Integration
Requirements:- Slack workspace with admin access
- Incoming webhook URL from Slack
-
Create an incoming webhook in Slack:
- Navigate to Slack API > Incoming Webhooks
- Choose a channel for notifications
- Copy the webhook URL
- Configure the notification channel:
Wazuh Slack
Type: Slack
Webhook URL: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
Message Template Example:
PagerDuty Integration
Requirements:- PagerDuty account
- Integration key (routing key) from PagerDuty
-
Create an integration in PagerDuty:
- Navigate to Services > Select Service > Integrations
- Add Integration > Events API V2
- Copy the integration key
- Configure the notification channel:
Wazuh PagerDuty
Type: Custom Webhook
Webhook URL: https://events.pagerduty.com/v2/enqueue
Message Template (JSON):
Jira Integration
Requirements:- Jira instance (Cloud or Server)
- API token or password
- Project key for issue creation
-
Generate API token in Jira:
- Navigate to Account Settings > Security > API Tokens
- Create and copy the API token
- Configure the notification channel:
Wazuh Jira
Type: Custom Webhook
Webhook URL: https://your-domain.atlassian.net/rest/api/3/issue
Authentication: Basic Auth
- Username: Your Jira email
- Password: API token
Shuffle Integration
Requirements:- Shuffle instance (cloud or on-premise)
- Webhook URL from Shuffle workflow
-
Create a webhook trigger in Shuffle:
- Create or edit a workflow
- Add a Webhook trigger
- Copy the webhook URL
- Configure the notification channel:
Wazuh Shuffle
Type: Webhook
Webhook URL: https://shuffler.io/api/v1/hooks/YOUR_WEBHOOK_ID
Message Template:
Alerting Monitors
Monitor Components
An alerting monitor consists of:- Schedule: How frequently the monitor runs
- Data Source: The index pattern to query
- Query: The search query to execute
- Trigger: Conditions that activate the alert
- Actions: Notifications to send when triggered
Sample Monitors
The Dashboard health check system can create sample monitors for testing integrations. These monitors query Wazuh events and trigger notifications based on configurable conditions.Monitor Configuration
Monitor Schedule:Trigger Configuration
Condition Script:1- Critical2- High3- Medium4- Low5- Informational
Action Configuration
Subject Template:Creating Monitors via API
Use the OpenSearch Alerting API to create monitors programmatically: Endpoint:Message Templates
Message templates use Mustache syntax to include dynamic content from the alert context.Available Context Variables
Monitor Information:{{ctx.monitor.name}}- Monitor name{{ctx.monitor.type}}- Monitor type{{ctx.monitor.enabled}}- Monitor enabled status
{{ctx.trigger.name}}- Trigger name{{ctx.trigger.severity}}- Severity level
{{ctx.results.0.hits.total.value}}- Total matches{{ctx.results.0.hits.hits}}- Result documents (array)
{{ctx.periodStart}}- Query period start time{{ctx.periodEnd}}- Query period end time
Template Examples
Detailed Slack Message:Health Check Integration
The Dashboard health check system can automatically create notification channels and sample monitors during initialization.Automatic Channel Creation
The health check creates default notification channels with placeholder configurations. After initial creation, update the channels with actual credentials:- Navigate to OpenSearch Dashboards > Notifications
- Edit each notification channel
- Update webhook URLs, API keys, and other credentials
- Test the notification to verify configuration
Automatic Monitor Creation
Sample monitors are created only if the corresponding notification channels exist. This prevents monitors from being created without valid destinations. Sample Monitors Created:- Sample: Slack
- Sample: PagerDuty
- Sample: Jira
- Sample: Shuffle
Best Practices
Monitor Design
- Specific Queries: Create focused queries that target specific event types
- Appropriate Thresholds: Set thresholds that reduce false positives
- Reasonable Frequency: Balance detection speed with system load
- Clear Naming: Use descriptive names that explain the monitor’s purpose
- Document Monitors: Include descriptions explaining trigger conditions
Notification Management
- Rate Limiting: Implement notification throttling to prevent alert fatigue
- Severity-Based Routing: Send critical alerts to different channels than informational alerts
- Message Clarity: Include enough context in messages for actionable responses
- Test Notifications: Verify notifications are delivered before enabling production monitors
- Secure Credentials: Protect webhook URLs and API tokens
Performance Considerations
- Limit Query Scope: Use time-based filters to reduce data volume
- Optimize Schedules: Avoid running many monitors simultaneously
- Monitor Count: Balance monitoring coverage with system resources
- Result Size: Limit the number of documents returned in queries
Troubleshooting
Notifications Not Delivered
Solutions:- Verify the notification channel configuration is correct
- Test the webhook URL independently
- Check authentication credentials
- Review OpenSearch logs for delivery errors
- Verify network connectivity to external services
Monitor Not Triggering
Solutions:- Verify the query returns results when run manually
- Check the trigger condition script syntax
- Review monitor schedule and last execution time
- Ensure the monitor is enabled
- Check index patterns match actual indices
Invalid Message Templates
Solutions:- Verify Mustache syntax is correct
- Check that referenced context variables exist
- Test templates with sample data
- Review logs for template rendering errors