HTTP Header Anomaly-based Advanced Behavioural Bot Detection

Introduction

HTTP headers are integral components of web communication. They enable information exchange between clients (e.g., web browsers) and servers, helping the server process requests and the client interpret responses correctly.

However, malicious bots often exploit them. Bad bots can manipulate HTTP headers when sending data from the client to the server. This is done to mimic legitimate user behavior. Understanding and detecting anomalies in these headers is crucial for identifying and mitigating the impact of bad bots on web services.

In this blog, we explain how Radware Bot Manager helps identify bots that try manipulating HTTP Headers.

Understanding HTTP Headers

HTTP headers are key-value pairs of metadata included in HTTP requests and responses, facilitating communication between clients and servers over the World Wide Web using the HTTP protocol.

Some examples of metadata are:

Header Manipulation Techniques

Bots can manipulate HTTP headers in many ways to disguise their identity to bypass security measures. Here are some common techniques:

1. User-Agent Spoofing: Bots often alter the User-Agent header to mimic popular web browsers or legitimate user agents. This helps them avoid detection by systems that rely on User-Agent strings to identify and block bots.
2. Referer Header Manipulation: Bots can fake the Referer header to make it appear as though they are coming from a trusted source.
3. Accept-Language Spoofing: By setting the Accept-Language header to common language preferences, bots can blend in with legitimate traffic and avoid detection.
4. Cookie Handling: Bots may manipulate or ignore cookies to avoid tracking and session management mechanisms. They might also send invalid or expired cookies to disrupt normal operations.
5. Custom Headers: Some bots add custom headers or modify existing ones to pass through security checks.
6. Header Order and Case: Bots can change the order of headers to avoid detection by systems that expect headers in a specific order.

HTTP Header Anomaly-based Bot Detection

Radware Bot Manager uses a multi-layered approach to bot protection as referenced in one of the earlier blogs.

HTTP Header Anomaly protection falls under the behavioral-based detection category within this multi-layered strategy. It helps detect HTTP header anomalies using advanced techniques and technologies designed to identify and mitigate malicious bot traffic. Here are some of the common techniques involved in isolating bad bots using HTTP headers:

1. Standard Header Recognition Radware Bot Manager leverages machine learning algorithms to scrutinize standard header keys in customer applications. When a legitimate source accesses the application, these headers are analyzed. Any deviation, such as a missing mandatory header or the presence of an unusual header, is assumed anomalous. Additionally, Radware Bot Manager also correlates various parameters received in API calls from the application to the server to establish mandatory headers. For example, Bot Manager can determine that a specific version of a Mozilla browser identified from the user-agent parameter should include all Accept headers, such as accept-language, accept-charset, and accept-encoding. Any missing header should be a sign of anomaly.

2. Rare Header Recognition Radware Bot Manager employs machine learning algorithms to identify rare and unlikely headers. Occasionally, headers that are not typically seen in applications appear in HTTP requests from clients. Such anomalies are automatically detected and the requests from those sources are blocked effectively.

3. Malicious Header Repository Radware Bot Manager maintains a repository of malicious HTTP header keys and values, compiled over the years from all protected assets. Any unusual information in HTTP request key-value pairs is verified against this repository of bot signatures to instantly flag as anomalous.
4. Header Order and Case Identification Radware Bot Manager uses machine learning algorithms to identify the sequence and letter case of headers in HTTP request packets. These distinctions can be crucial for detecting header anomalies.

Conclusion

HTTP header anomaly detection is just one of the many advanced techniques employed by Radware Bot Manager to identify and mitigate malicious bot traffic. This capability, along with numerous other detection modules, showcases the comprehensive security arsenal of Radware Bot Manager. Given its robust and multi-faceted approach to bot detection, incorporating an anti-bot solution like Radware Bot Manager into a security portfolio is essential for safeguarding web services against evolving threats.