F5 Logo MyF5
Search
  1. MyF5 Home
  2. BIG-IP Application Security Manager: Implementations
  3. Refining Security Policies with Learning
Manual Chapter : Refining Security Policies with Learning

Applies To:

Show Versions Show Versions

BIG-IP ASM

  • 14.1.3, 14.1.2, 14.1.0
Manual Chapter

Refining Security Policies with Learning

About learning

You can use learning resources to help build a security policy, particularly if you are building a security policy manually. When building a security policy manually, the learning mode is set to Manual, and when building a policy automatically, the learning mode is Automatic.
When you send client traffic through the Application Security Manager (ASM), the learning data provides information on requests or responses that do not comply with the current security policy and have triggered a violation. The reason for triggering a violation can be either an actual attack on the site, or a false positive (typically seen during the process of building a policy).
ASM generates learning suggestions for requests that cause violations and do not pass the security policy checks. The system also suggests adding legitimate entities such as URLs, file types, or parameters that often appear in requests. You can examine the requests that cause learning suggestions, and then use the suggestions to refine the security policy. In some cases, learning suggestions may contain recommendations to relax the security policy. When dealing with learning suggestions, make sure to relax the policy only where false positives occurred, and not in cases where a real attack caused a violation. You can use the violation ratings to help determine how likely a request was caused by an attack.
If you are generating a security policy automatically, ASM handles much of the learning for you, adjusting the security policy based on traffic characteristics. In that case, the learning screens show only the elements that the security policy is in the process of learning, or those which require manual intervention to be resolved.

About learning suggestions

Application Security Manager (ASM) generates learning suggestions for violations if the Learn flag is enabled for the violations on the Learning and Blocking Settings screen. When the system receives a request that triggers a violation, the system updates the Traffic Learning screen with learning suggestions using information from the violating request. From this screen, you can review the learning suggestions to determine whether the request triggered a legitimate security policy violation, or if the violation represents a need to update the security policy.
The system can also generate suggestions based on legitimate activity, such as adding a valid URL or host name to the security policy.
Next to each suggestion, ASM assigns a
learning score
 that measures the strength of the suggestion by showing a percentage that indicates how close the system is to recommending that you accept the suggestion. The learning score is also influenced by the violation rating: the lower the rating of the violations, the higher the score.
If the system is working in automatic learning mode, when the learning score reaches 100%, the system can accept and enforce most of the suggestions. If and when the system enforces the suggestions depends on which learning mode
auto-apply
setting you have chosen. It is possible to limit auto-apply to specific days and hours. You can accept suggestions manually at any time. If you are using manual learning, when the learning score reaches 100% (or before that if you know the suggestions are valid), you need to accept the suggestions manually.
Making decisions about which learning suggestions to accept requires a general understanding of application security, and specific knowledge of the protected application (for example, recognizing valid traffic). For example, you should consider accepting a learning suggestion when you see that it is associated with many requests from many different source IP addresses. As long as they are valid, repeated requests may indicate legitimate traffic behavior that warrants relaxing the security policy.
You can also review the violation rating for requests by selecting the suggestion. Learning suggestions associated with requests having a low average violation rating are more likely to be false positives and can be accepted. But if a request has a high violation rating, the learning suggestion should not be accepted. Instead, it should be cleared because it is most likely indicative of an attack.
The Traffic Learning screen also displays violations for which the system does not generate learning suggestions. Typically, these violations are related to RFC compliance and system resources; the resolution for these violations may be to disable the violation rather than to change the configuration. The system displays these violations along with the learning suggestions to ease the security policy management tasks.

What suggestions look like

This figure shows the Traffic Learning screen with several suggestions on it. As an example, on the left, the suggestion to enforce a cookie is highlighted; the information on the right shows what caused the suggestion. The HTTP violations are listed, and one is selected showing details about the request. The cookie
PHPAUCTION_SESSION
 matched the * wildcard in the Allowed Cookies list in several requests so the suggestion is to add and enforce the cookie. If you accept the suggestion, the cookie is added to the Enforced Cookies list.
Traffic Learning screen with suggestions
Traffic Learning screen

What violations are unlearnable?

Some violations that occur indicate a real problem with a request that cannot be learned. These are called
unlearnable violations
. For example, requests for access from disallowed users, disallowed sessions, and disallowed IP addresses are unlearnable. In addition, the system considers requests that trigger the following HTTP protocol compliance violations to be unlearnable:
  • Bad HTTP version
  • Unparsable request content
  • Null in request
They are considered unlearnable because these violations indicate behavior that is never acceptable, so the security policy will never be changed to allow them. Consequently, the violating requests are not used for automatic or manual learning (even if they include additional violations that could be learned). No learning suggestions are created for requests containing these violations. Also, the violation rating for these transactions is always set to 5 (the highest severity).

Configuring how entities are learned

You can adjust the learning settings for file types, URLs, parameters, cookies, and redirection domains. Learning settings specify when Real Traffic Policy Builder adds, or suggests you add, explicit entities to the security policy.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Learning and Blocking Settings
    .
    The Learning and Blocking Settings screen opens.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. On the right side of the Learning and Blocking Settings screen, select
    Advanced
    .
    The screen displays the advanced configuration details for policy building.
  4. In the Policy Building Settings area, click the entity (
    File Types
    ,
    URLs
    ,
    Parameters
    ,
    Cookies
    , and
    Redirection Protection
    ) to show the settings. Then from the
    Learn New
    entity
     setting, select the option that determines which Learning suggestions the system provides (based on real traffic).
    Option
    Description
    Never (wildcard only)
    Specifies that when false positives occur, the system suggests relaxing the settings of the wildcard. This option results in a security policy that is easy to manage, but is not as strict. If it is running in automatic learning mode, the Policy Builder does not add explicit entities that match a wildcard to the security policy. The wildcard entity remains in the security policy. The Policy Builder changes the attributes of any matched wildcard. If it is running in manual learning mode, Policy Builder suggests changing the attributes of matched wildcard entities, but does not suggest that you add explicit entities that match the wildcard entity.
    Selective
    Applies only to * wildcard entity. When false positives occur, adds an explicit entity with relaxed settings. This option serves as a good balance between security, policy size, and ease of maintenance. If using automatic learning, the policy includes explicit entities that do not match the attributes of the * wildcard, and does not remove the * wildcard. If usingmanuial learning, the system suggests adding explicit entities that match the * wildcard. (This option is not available for redirection protection.)
    Compact
    Applies only to the * wildcard. Specifies that the policy includes the most commonly used entities (while enforcing all others types with a wildcard rule), also provides a pre-populated list of known disallowed file types, and includes top-level URLs such as /abc/*. This option serves as a good balance between
    Selective
     and
    Always
     making a smaller, more compact policy, with fewer suggestions.
    When using Automatic learning, the system adds explicit entities that do not exist in the policy but which match the attributes of the * wildcard. The Policy Builder does not remove the * wildcard file type from the security policy. For Manual learning, the system suggests adding explicit entities that match the * wildcard file type. (This option is not available for cookies or redirection protection.)
    Always
    Creates a comprehensive whitelist policy that includes all web site entities. This option results in a large, more granular configuration with stricter security. If Policy Builder is running, it adds explicit entities that match a wildcard to the security policy. When the security policy is stable, the * wildcard is removed. If Policy Builder is not running, the system suggests adding explicit entities that match the wildcard. (This option is not available for cookies.)
  5. Click
    Save
     to save your settings.
  6. To put the security policy changes into effect immediately, click
    Apply Policy
    .
The security policy now learns new file types, parameters, URLs, cookies, and redirection domains according to the learning settings you specified.

Learning from responses

When learning to build a policy, you can have the system examine responses as well as requests for entities to include in the security policy. This is called
learning from responses
, and the system does this by default in automatic mode. Learning from responses is supported in manual mode but is not enabled by default. You may want to learn from responses because a response might include more information about the web application than is found in the request, or if you want to have the system learn login pages automatically.
You can disable this setting in automatic mode if your application does not need to examine responses for entities to add to the security policy, or if the application does not use dynamic parameters.
This setting applies only to what entities can be learned from the response content, such as URLs and parameters. The system does not learn from violations that occur in responses, such as Data Guard leakage. Learning from violations is enabled by selecting the Learn flag of the respective violation.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Learning and Blocking Settings
    .
    The Learning and Blocking Settings screen opens.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. On the right side of the Learning and Blocking Settings screen, select
    Advanced
    .
    The screen displays the advanced configuration details for policy building.
  4. If you do not want the security policy to include elements found in responses when building the security policy, in the Policy Building Process area, expand
    Options
     and clear the
    Learn from responses
     check box.
    You can also have the system learn only from requests that return specific response codes.
    If the setting is not enabled, the Policy Builder never learns from responses.
  5. Click
    Save
     to save your settings.
  6. To put the security policy changes into effect immediately, click
    Apply Policy
    .
If you disabled the
Learn from responses
 check box, the Policy Builder never adds to the security policy elements found in responses. If the check box is enabled, the Policy Builder adds elements found in valid responses to the security policy (meaning those that do not generate violations).

Learning based on response codes

When using automatic or manual learning, the system learns from legitimate traffic including transactions that return response codes of 1xx, 2xx, and 3xx. These classes of codes are added by default to the policy building settings. You can change which response codes are listed, or add specific response codes, such as those used by the web application you are protecting.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Learning and Blocking Settings
    .
    The Learning and Blocking Settings screen opens.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. On the right side of the Learning and Blocking Settings screen, select
    Advanced
    .
    The screen displays the advanced configuration details for policy building.
  4. In the Policy Building Process area, expand
    Options
    .
  5. In the
    Add
     field following
    HTTP Response Status Codes used to learn traffic
    , type the response code you want to add (for example, add specific codes like
    304
     or a class of codes like
    4xx
    ), then click
    Add
    . Use these formats.
    Response code
    Description
    1xx
    All informational responses (the request was received; continuing to process it). Included by default.
    2xx
    All successful responses (the request was received, understood, accepted, and processed successfully). Included by default.
    3xx
    All redirection (the client needs to take additional action on the request). Included by default.
    4xx
    Server failed to fulfill the response as a result of client syntax or input errors.
    5xx
    All server error responses (the server failed to fulfill a request).
    Specific codes such as
    100
    ,
    306
    ,
    400
    , or
    404
    Refer to your web application or the Hypertext Transfer Protocol -- HTTP/1.1 specification (RFC-2616).
  6. Click
    Save
     to save your settings.
  7. To put the security policy changes into effect immediately, click
    Apply Policy
    .
The Policy Builder extracts information for the security policy from traffic transactions that return the specified HTTP response status codes.

Reviewing learning suggestions

Before you can see learning suggestions on the system, it needs to have had some traffic sent to it.
After you create a security policy and begin sending traffic to the application, the system provides learning suggestions concerning additions to the security policy based on the traffic it sees. For example, you can have users or testers browse the web application. By analyzing the traffic to and from the application, Application Security Manager generates learning suggestions or ways to fine-tune the security policy to better suit the traffic and secure the application.
This task is primarily for building a security policy manually. If you are using the automatic learning mode, this task applies to resolving suggestions that require manual intervention, or for speeding up the enforcement of policy elements.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Traffic Learning
    .
    The Traffic Learning screen opens, and lists suggestions based on traffic patterns and violations that the system has detected.
  2. Take a look at the Traffic Learning screen to get familiar with it.
    With no suggestions selected, the right pane displays sections that facilitate the reviewer decision-making process. These include graphical charts that summarize policy activity, a summary of top violations in
    Reduce Potential False-positive Alerts
    , an enforcement readiness summary and a summary of suggestions to add new entity or delete an obsolete entity.
  3. To change the order in which the suggestions are listed, or refine what is included in the list, use the filters at the top of the column. Click the search icon to see basic and advanced filters.
  4. Review the learning suggestions as follows.
    1. Select a learning suggestion.
      Information is displayed about the action the system will take if you accept the suggestion, and what caused the suggestion.
    2. Select a suggestion to learn more about what caused it by looking at the action, the number of samples it is based on, the violations caused and their violation ratings, and if available, by examining samples of the requests that caused the suggestion.
    3. Select a request to view data about the request on the right, including any violations it generated, the contents of the request itself, and the response (if any).
      By examining the requests that caused a suggestion, you can determine whether it should be accepted.
    4. To add comments about the suggestion and the cause, click the Add Comment icon Add Comment icon to the right of the suggestion commands, and type the comments.
  5. Decide how to respond to the suggestion. You can start with the suggestions that have the highest learning scores, or those which you know to be valid for the application. These are the options.
    Option
    What happens
    Accept Suggestion
    The system modifies the policy by taking the suggested action, such as adding an entity that is legitimate. If the entity that triggered the suggestion can be placed in staging (file types, URLs, parameters, cookies, or redirection domains), clicking
    Accept Suggestion
     displays a second option,
    Accept suggestion
     and enable staging on Matched <<entity>>. Click this option to accept the suggestion and place the matched entity in staging.
    Delete Suggestion
    The system removes the learning suggestion, but the suggestion reoccurs if new requests cause it. The learning score of the suggestion starts over from zero in that case.
    Ignore Suggestion
    The system does not change the policy and stops showing this suggestion on the Traffic Learning screen now and in the future. You can view ignored suggestions by filtering by status ignored.
    If you are working in automatic learning mode, when the learning score reaches 100%, the system can accept most of the suggestions if you selected the
    Learning Mode Auto-apply Policy
    , or you can accept suggestions manually at any time. If you are using manual learning, when the learning score reaches 100% (or before that if you know the suggestions are valid), you need to accept the suggestions manually.
    If you know that a suggestion is valid, you can accept it at any time even before the learning score reaches 100%. The ones that reach 100% have met all the conditions so that they are probably legitimate entities.
  6. To put the security policy changes into effect immediately, click
    Apply Policy
    .
By default, a security policy is put into an enforcement readiness period for seven days. During that time, you can examine learning suggestions and adjust the security policy making sure that users can access the application. The security policy then includes elements unique to your web application.
It is a good idea to periodically review the learning suggestions on the Traffic Learning screen to determine whether the violations are legitimate and caused by an attack, or if they are false positives that indicate a need to update the security policy. Typically, a wide recurrence of violations at some place in the policy (with a low violation rating and a high learning score) indicates that they might be false positives, and hence the policy should be changed so that they will not be triggered anymore. If the violations seem to indicate true attacks (for example, they have a high violation rating), the policy should stay as is, and you can review the violations that it triggered.

Viewing requests that caused learning suggestions

To review requests that are related to learning suggestions, you need to have a security policy that is already handling traffic. If the
Learn
 flag is disabled for a violation, you will not see learning suggestions for that violation. If no violations have occurred, you will only see learning suggestions for adding legitimate entities to the security policy.
Before you process a learning suggestion, it is very helpful to examine the details of sample requests that caused the learning suggestion. By viewing the requests, you can determine whether to accept each one, or not. If the suggestion is based on a violation, you can see whether the violation was caused by an attack, or if it is a false positive.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Traffic Learning
    .
    The Traffic Learning screen opens, and lists suggestions based on traffic patterns and violations that the system has detected.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. In the left column, select the learning suggestion that you want to learn more about.
    Sample requests associated with the suggestions show on the right, with the average violation rating. (Legal requests have no violation rating.)
  4. Select the request that you want to review more closely.
    The request details are displayed on the right, including any violations the request generated, the contents of the request itself, and the response (if any).
  5. Review the information about the request on the General Data tab.
    If the request caused violations, they are listed at the top. Click the down arrow to examine the violation occurrences and description.
    The Request Status and a flag highlight requests considered to be illegal. These are the ones you need to examine most closely.
  6. To examine the request or response itself, click
    Request
     or
    Response
    .
    The actual content of the request or response is displayed in the tab.
  7. On the Traffic Learning screen, continue to review the learning suggestions and associated requests.
When you finish reviewing the requests associated with learning suggestions, you can accept, delete, or ignore the suggestions.

Viewing and allowing ignored suggestions

If the system is not generating learning suggestions that you would expect to see, or when suggestions do not appear consistently, you can view learning suggestions that were previously ignored. You can also change the status of those suggestions so that if the situation reoccurs, the suggestion will be included on the Traffic Learning screen.
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Traffic Learning
    .
    The Traffic Learning screen opens, and lists suggestions based on traffic patterns and violations that the system has detected.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. Above the list of suggestions, click the Open Filter icon.
    The filter popup screen opens.
  4. In the filter popup screen, click
    Advanced Filter
    .
  5. From the
    Status
     list, select
    Ignored
    .
  6. Click
    Apply Filter
    .
    Suggestions that were previously ignored are displayed in the list.
  7. Review the ignored suggestions and decide how to handle them:
    • If the suggestion should continue to be ignored, do nothing. Click the Reset Filter (X) above the list to return to the current suggestions.
    • If you do not want this suggestion to be ignored in the future, click
      Delete Suggestion
      .
    • If you want to implement this previously ignored suggestion, click
      Accept Suggestion
      .
    If you delete the suggestion, it is removed from the ignored suggestions list, and it will not be ignored if the conditions that caused it occur again. If you accept the suggestion, the suggested change is made to the security policy, and it is removed from the list of ignored suggestions.

About enforcement readiness

When you are creating a security policy, you specify an enforcement readiness period that indicates a staging period for entities and attack signatures (typically 7 days). When entities or attack signatures are in staging, the system does not enforce them. Instead, the system posts learning suggestions for staged entities.
When the enforcement readiness period is over and no learning suggestions are added for the staging period duration (the default is 7 days), the file type, URL, parameter, cookie, signature, or redirection domain is considered ready to be enforced. Particularly if you are using manual learning, you can delve into the details to see if you want to enforce these entities in the security policy. From the Enforcement Readiness summary on the Traffic Learning screen, you can enforce selected entities to the security policy, or you can enforce all of the entities and signatures that are ready to be enforced. If you are using automatic learning, you can still enforce entities manually, but the Policy Builder enforces entities according to the learning and blocking settings. So you do not need to enforce entities in the security policy.

Enforcing entities

When you create a security policy and traffic is sent to the web application, the system makes learning suggestions about files types, URLs, parameters, cookies, and redirection domains to add to the security policy. You can review the entities and signatures that are ready to be enforced, and enforce them in the security policy.
This task is primarily for building a security policy using manual learning. If you are using the automatic learning mode, the system cab automatically enforce entities in the security policy, if you selected the
Learning Mode Auto-apply Policy
, when it has processed sufficient traffic and sessions over enough time, from different IP addresses, to determine the legitimacy of the file types, URLs, parameters, cookies, methods, and so on. .
  1. On the Main tab, click
    Security
    Application Security
    Policy Building
    Traffic Learning
    .
    The Enforcement Readiness summary is on the bottom right.
  2. In the
    Current edited security policy
     list near the top of the screen, verify that the security policy shown is the one you want to work on.
  3. Below the charts on the right, review the Enforcement Readiness Summary to see if there are entities that are ready to be enforced. If there are, select the entities you want the security policy to enforce, and click
    Enforce Ready Entities
    .
    If you select this option, you are done with this task. Continue only if you want to enforce selected entities or signatures instead of enforcing all entities ready to be enforced.
  4. In the Enforcement Readiness Summary, check to see if a number appears in the Not Enforced column.
    A number greater than zero indicates that entities of that type are in staging, or have wildcard entities configured so that the security policy learns all explicit entities that match them.
  5. Click the number in the Not Enforced column.
    The allowed file types, URLs, parameters, cookies, signatures, or redirection protection list opens showing the entities that you can enforce.
  6. Select the entities you want the security policy to enforce, and click
    Enforce
    .
The system removes the selected entities or signatures from staging, and enforces them in the security policy. If any of the entities are wildcards that are learning explicit entities, the system deletes the wildcards.

Exploring security policy action items

Even though you are done creating a security policy, Application Security Manager (ASM) might have additional action items it recommends for you to do based on your current system configuration and current security policies.
  1. On the Main tab, click
    Security
    Overview
    Application
    Action Items
    .
    The Action Items screen opens.
  2. Examine the Action Items screen for information about recommended actions that you need to complete.
  3. Review the Suggested Action Items area, which lists system tasks and security policy tasks that the system recommends.
    Examples of system-wide action items include updating attack signatures, restarting the system, and setting up synchronization so that all configuration data from this system is duplicated on another system in a device group. Action items related to the security policy include enforcing entities that are ready to be enforced and applying changes previously made to the security policy.
  4. Click the links in the Suggested Action Items area to go to the screen where you can perform the recommended actions.
  5. In the Quick Links area, click any of the links to gain access to common configuration and reporting screens.
  6. If you are using the Automatic Learning Mode, you can see a summary of the policy elements learned in automatic mode for each security policy on the system.
By looking at the recommended Action Items and system reports, you can find out what additional steps you can take to tighten your security policy.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information