F5 Logo MyF5
Search
  1. MyF5 Home
  2. Knowledge Centers
  3. BIG-IP APM
  4. BIG-IP Access Policy Manager: Secure Web Gateway
  5. URL Categorization
Manual Chapter : URL Categorization

Applies To:

Show Versions Show Versions

BIG-IP APM

  • 12.1.6, 12.1.5, 12.1.4, 12.1.3, 12.1.2, 12.1.1, 12.1.0
Manual Chapter
Table of Contents | << Previous Chapter | Next Chapter >>

URL Categorization

About URL categorization

On a BIG-IP® system with an SWG subscription, URL categorization must be configured. The URL database must be downloaded and a download schedule must be set. Optionally, custom URL categories and filters can be created to extend the standard URL categories and URL filters that are provided.

On a BIG-IP system without an SWG subscription, URL categorization is an option. Standard URL categories and URL filters are not provided. URL filtering can be accomplished with user-defined URL categories and user-defined URL filters.

Overview: Downloading the URL database and updating standard URL filters

Secure Web Gateway (SWG) supplies over 150 URL categories and identifies over 60 million URLs that fit within these categories. In addition, you can create custom categories if needed and add URLs to any category, custom or otherwise. You can also use custom categories to define blacklists and whitelists.

Note: A URL database is available only on a BIG-IP® system with an SWG subscription.

SWG supplies default URL filters as a starting point for your configuration. For example, the URL filter named default blocks the majority of inappropriate web sites. You can use any default filter as a starting point from which to define your own URL filters to reflect your acceptable use policies.

Complete these tasks before you create a per-request policy to categorize and filter URL requests.

Task summary

Use these tasks to download URL categories initially, to refresh them over time, and to specify URL filters that support your use and compliance policy. Before you begin, the BIG-IP® system must be licensed and provisioned to support URL categorization.

Task list

About the Instant Messaging URL category

Note: A predefined Instant Message URL category is available only on a BIG-IP® system with an SWG subscription.

Secure Web Gateway (SWG) supports HTTP and HTTPS-based instant messaging protocols. As a result, when you use the Instant Messaging URL category to block messages, SWG can block messages to ICQ, for example, but cannot block messages from applications that use non-standard ports or tunneling over HTTP, such as, Yahoo Messenger, Skype, Google Talk, and so on.

Similarly, SWG cannot block messages from file-sharing and peer-to-peer protocols that do not use HTTP or HTTPS; most of these protocol types do not use either HTTP or HTTPS.

Downloading and updating URL categories

Note: Database download is available only on a BIG-IP®system with an SWG subscription.
For database downloads to work, you must have configured DNS for the BIG-IP device in the System area of the product. You must also must have configured a default route in the Network area of the product.
If URL database download is available on the BIG-IP system, you must download the URL categories for Secure Web Gateway (SWG) to work. In order for SWG to best protect your network from new threats, schedule regular database downloads to update the existing URL categories with new URLs. Without these updates, SWG uses obsolete security intelligence and as a result, protection of your networks is less effective.
Note: Schedule database downloads to occur during off-peak hours (very little to no user activity), so that users are not impacted. Alternatively, you can initiate database downloads on-demand.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Database Settings > Database Download .
  2. In the Download Settings area from the Downloads list, select Enabled.
    Additional settings display. Download Schedule displays a default schedule for the download.
  3. In the Download Schedule settings, configure a two-hour period in which to start the download.
    Schedule the download to occur during off-peak hours. The default schedule is between one and three A.M.
    Warning: After the download completes, database indexing occurs. It consumes a high amount of CPU for approximately 45 minutes.
  4. Click Update Settings.
  5. To download the database immediately, click Download Now.
    A download occurs only when a newer version becomes available.
    Warning: Database indexing occurs after the download and impacts system performance.
    Warning: The ANTserver service is not available on the BIG-IP system for approximately 300 milliseconds after the database download completes.

Adding custom URL categories to the URL database

Note: A URL database is available only on a BIG-IP® system with a Secure Web Gateway (SWG) subscription.
You can add a custom category to the standard Secure Web Gateway URL categories to specify a list of URLs that you want to block or allow, or for which you want to obtain confirmation from a user before blocking or allowing access.
Note: The URL categories that you add become subcategories of Custom Categories. Custom Categories take precedence over standard categories.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories .
    The URL Categories table displays; Custom Categories displays as the first entry in the table.
  2. Click Create.
    The Category Properties screen displays.
  3. In the Name field, type a unique name for the URL category.
  4. From the Default Action list, retain the default value Block; or, select an alternative: Allow or Confirm.
    If no action has been specified in a filter for this category, the URL Filter agent takes the branch for the default action.
  5. Add, edit, or delete the URLs that are associated with the category by updating the Associated URLs list.
  6. To add URLs to the Associated URLs list:
    1. In the URL field, type a URL.
      You can type a well-formed URL that the system must match exactly or type a URL that includes globbing patterns (wildcards) for the system to match URLs.
    2. Select the Glob Pattern Match check box if you typed any globbing patterns in the URL field.
    3. Click Add.
      The URL displays in the Associated URLs list.
    These are well-formed URLs:
    • https://www.siterequest.com/
    • http://www.siterequest.com:8080/
    • http://www.sitequest.com/docs/siterequest.pdf/
    • http://www.sitequest.com/products/application-guides/
    This URL *siterequest.[!comru] includes globbing patterns that match any URL that includes siterequest, except for siterequest.com or siterequest.ru.
    This URL *://siterequest.com/education/* includes globbing patterns that match any HTTP URL that includes siterequest.com/education, but that do not match any HTTPS URLs if Category Lookup specifies that the input is SNI or CN.Subject.
    Important: For SNI or CN.Subject input, Category Lookup uses scheme:://host for matching, instead of matching the whole URL.
  7. Click Finished.
    The URL Categories screen displays.
  8. To view the newly created URL category, expand Custom Categories.
    The custom URL category displays in the Sub-Category column.
Add or edit a URL filter to specify an action (allow, block, or confirm) for the custom category.

Customizing standard categories from the URL database

You can customize the standard URL categories that Secure Web Gateway (SWG) supplies by adding URLs to them. You might do this after you run SWG for a while, view logs and reports, and determine that you need to make changes.
Note: A URL database is available only on a BIG-IP® system with an SWG subscription.
Note: If you add a URL to a URL category, SWG gives precedence to that categorization and database downloads do not overwrite your changes.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories .
    The URL Categories table displays.
  2. Click the name of any category or subcategory to edit the properties for it.
    To view and select a subcategory, expand categories.
    The Category Properties screen displays. There are many URLs in a given category; however, any URLs that display on the Associated URLs list are entered by the user.
  3. Edit or delete any URLs on the Associated URLs list.
  4. To add URLs to the Associated URLs list:
    1. In the URL field, type a URL.
      You can type a well-formed URL that the system must match exactly or type a URL that includes globbing patterns (wildcards) for the system to match URLs.
    2. Select the Glob Pattern Match check box if you typed any globbing patterns in the URL field.
    3. Click Add.
      The URL displays in the Associated URLs list.
    These are well-formed URLs:
    • https://www.siterequest.com/
    • http://www.siterequest.com:8080/
    • http://www.sitequest.com/docs/siterequest.pdf/
    • http://www.sitequest.com/products/application-guides/
    This URL *siterequest.[!comru] includes globbing patterns that match any URL that includes siterequest, except for siterequest.com or siterequest.ru.
    This URL *://siterequest.com/education/* includes globbing patterns that match any HTTP URL that includes siterequest.com/education, but that do not match any HTTPS URLs if Category Lookup specifies that the input is SNI or CN.Subject.
    Important: For SNI or CN.Subject input, Category Lookup uses scheme:://host for matching, instead of matching the whole URL.
  5. Click Add.
    The URL displays in the Associated URLs list.
  6. Click Update.
    The URL Properties screen refreshes.
  7. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories .
    The URL Categories table displays. The screen displays (recategorized) next to the URL category that you customized.
URLs are added to the URL category that you selected.

Configuring URL filters

You configure a URL filter to specify whether to allow, block, or confirm requests for URLs in URL categories. You can configure multiple URL filters.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Filters .
    You can click the name of any filter to view its settings.
    Note: On a BIG-IP® system with an SWG subscription, default URL filters, such as block-all and basic-security, are available. You cannot delete default URL filters.
    The URL Filters screen displays.
  2. To configure a new URL filter, click one of these options.
    • Create button: Click to start with a URL filter that allows all categories.
    • Copy link: Click for an existing URL filter in the table to start with its settings.
  3. In the Name field, type a unique name for the URL filter.
  4. Click Finished.
    Note: User-defined categories are subcategories of Custom Category.
    The screen redisplays. An Associated Categories table displays. It includes each URL category and the filtering action that is currently assigned to it. The table includes a Sub-Category column.
  5. To block access to particular categories or subcategories, select them and click Block.
    Important: When you select a category, you also select the related subcategories. You can expand the category and clear any subcategory selections.
  6. On a BIG-IP system with an SWG subscription, expand the category Miscellaneous, select Uncategorized, and then click Block.
    Important: It is important to block URLs that SWG cannot categorize.
  7. To allow access to particular categories or subcategories, select them and click Allow.
  8. To indicate that you want a user to confirm that access is work-related or otherwise justified before obtaining access to the URLs in a category, select the categories or subcategories and click Confirm.
To put a URL filter into effect, you must assign it in a per-request policy. A per-request policy runs each time a URL request is made.

Looking up a URL category in the master database

You can look up a URL to determine whether it already exists in the master database and, if it exists, to see which categories include it.
Note: A URL database is available only on a BIG-IP® system with an SWG subscription.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Database Settings > URL Category Lookup .
  2. In the URL field, type the URL that you want to look up.
    Type the complete URL, including the URI scheme.
    Type https://www.google.com; not www.google.com or https://www.google.
  3. Click Search.
    Note: Custom categories are not searched.
    Results display in the URL Category table.
If the URL is not found, you can add it to an existing or a custom category. If the URL is found, you do not need to do anything, but can recategorize it by adding it to another category.

Implementation result

Now you have BIG-IP® Secure Web Gateway (SWG) configured to regularly download updates to URL categories. URL filters are configured and ready to be added to per-request policies.

Configuring logging for the URL database

Configure logging for the URL database so that log messages are published to the destinations, and at the minimum log level, that you specify. (Logging for the URL database occurs at the system level, not the session level, and is controlled using the default-log-setting log setting.)
Note: A URL database is available only on a BIG-IP® system with an SWG subscription.
  1. On the Main tab, click Access Policy > Event Logs > Log Settings .
    A log settings table displays.
  2. From the table, select default-log-setting and click Edit.
    A log settings popup screen displays.
  3. Verify that the Enable access system logs check box is selected.
  4. To configure settings for access system logging, select Access System Logs from the left pane.
    Access System Logs settings display in the right panel.
  5. From the Log Publisher list, select the log publisher of your choice.
    A log publisher specifies one or more logging destinations.
    Important: The BIG-IP® system is not a logging server and has limited capacity for storing, archiving, and analyzing logs. For this reason a dedicated logging server is recommended.
  6. To change the minimum log level, from the Secure Web Gateway list, select a log level.
    Note: Setting the log level to Debug can adversely impact system performance.
    The default log level is Notice. At this level, logging occurs for messages of severity Notice and for messages at all incrementally greater levels of severity.
  7. Click OK.
    The popup screen closes. The table displays.

Viewing a URL database report

You can view URL database log messages in an Access System Logs report if local logging is configured for the URL database.
Important: The BIG-IP® system is not a logging server and has limited capacity for storing, archiving, and analyzing logs. For this reason a dedicated logging server is recommended.
Create a report to view URL database event logs.
Note: A URL database is available only on a BIG-IP® system with an SWG subscription.
  1. On the Main tab, click Access Policy > Event Logs > Access System Logs .
    The Reports Browser displays in the right pane. The Report Parameters popup screen opens and displays a description of the current default report and default time settings.
  2. Click Cancel.
    The Report Parameters popup screen closes.
  3. In the Reports Browser in the General Reports list, select URL DB Messages > Run Report .
    The Report Parameters popup screen displays.
  4. Update the parameters, if necessary, and click Run Report.
    The popup screen closes. The report displays in the Report Browser.
Note: The session ID for a URL database message is 00000000 because URL database downloads occur outside of a client session.

Secure Web Gateway database download log messages

When you deploy Secure Web Gateway (SWG), the database downloads output messages to the log destinations specified in the default-log-setting. This table lists messages that are available only when you enable debug.

Note: Database downloads are possible only on a BIG-IP® system with an SWG subscription.
Debug message Description
Transfer Status 247 The file is transferred successfully to the BIG-IP® system. If you see a Transfer Status other than 247, it might indicate an error.
RTU Type The RTU Type is always 1. If you see an RTU Type other than 1, it might indicate an error.
Expiration Date The BIG-IP system does not use the expiration date in this message. Instead, the BIG-IP system enforces the SWG license and the database download works accordingly.

Overview: Configuring user-defined URL categories and filters

If you want to categorize and filter URL requests from your users, you need to use URL categories and URL filters. If URL categories and filters do not exist on a BIG-IP® system, you can create them.

Complete these tasks before you create a per-request policy that includes items to categorize (URL Category) and filter (URL Filter Assign) URL requests.

Task summary

Creating user-defined URL categories

Create a URL category to specify a group of URLs over which you want to control access.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories .
    The URL Categories table displays. If you have not created any categories, the table is empty.
  2. Click Create.
    The Category Properties screen displays.
  3. In the Name field, type a unique name for the URL category.
  4. From the Default Action list, retain the default value Block; or, select Allow.
    Note: A Confirm Box action in a per-request policy subroutine serves the purpose of enabling appropriate choices in a forward proxy (outbound) configuration. Currently, Access Policy Manager® does not support a similar action for reverse proxy.
  5. Add, edit, or delete the URLs that are associated with the category by updating the Associated URLs list.
  6. To add URLs to the Associated URLs list:
    1. In the URL field, type a URL.
      You can type a well-formed URL that the system must match exactly or type a URL that includes globbing patterns (wildcards) for the system to match URLs.
    2. Select the Glob Pattern Match check box if you typed any globbing patterns in the URL field.
    3. Click Add.
      The URL displays in the Associated URLs list.
    These are well-formed URLs:
    • https://www.siterequest.com/
    • http://www.siterequest.com:8080/
    • http://www.sitequest.com/docs/siterequest.pdf/
    • http://www.sitequest.com/products/application-guides/
    This URL *siterequest.[!comru] includes globbing patterns that match any URL that includes siterequest, except for siterequest.com or siterequest.ru.
    This URL *://siterequest.com/education/* includes globbing patterns that match any HTTP URL that includes siterequest.com/education, but that do not match any HTTPS URLs if Category Lookup specifies that the input is SNI or CN.Subject.
    Important: For SNI or CN.Subject input, Category Lookup uses scheme:://host for matching, instead of matching the whole URL.
  7. Click Finished.
    The URL Categories screen displays.
  8. To view the newly created URL category, expand Custom Categories.
    The custom URL category displays in the Sub-Category column.
Add or edit a URL filter to specify an action (allow or block) for the custom category.

Configuring URL filters

You configure a URL filter to specify whether to allow, block, or confirm requests for URLs in URL categories. You can configure multiple URL filters.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Filters .
    You can click the name of any filter to view its settings.
    Note: On a BIG-IP® system with an SWG subscription, default URL filters, such as block-all and basic-security, are available. You cannot delete default URL filters.
    The URL Filters screen displays.
  2. To configure a new URL filter, click one of these options.
    • Create button: Click to start with a URL filter that allows all categories.
    • Copy link: Click for an existing URL filter in the table to start with its settings.
  3. In the Name field, type a unique name for the URL filter.
  4. Click Finished.
    Note: User-defined categories are subcategories of Custom Category.
    The screen redisplays. An Associated Categories table displays. It includes each URL category and the filtering action that is currently assigned to it. The table includes a Sub-Category column.
  5. To block access to particular categories or subcategories, select them and click Block.
    Important: When you select a category, you also select the related subcategories. You can expand the category and clear any subcategory selections.
  6. On a BIG-IP system with an SWG subscription, expand the category Miscellaneous, select Uncategorized, and then click Block.
    Important: It is important to block URLs that SWG cannot categorize.
  7. To allow access to particular categories or subcategories, select them and click Allow.
  8. To indicate that you want a user to confirm that access is work-related or otherwise justified before obtaining access to the URLs in a category, select the categories or subcategories and click Confirm.
To put a URL filter into effect, you must assign it in a per-request policy. A per-request policy runs each time a URL request is made.
Table of Contents | << Previous Chapter | Next Chapter >>
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5 Networks, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information