F5 Logo MyF5
Search
  1. MyF5 Home
  2. Knowledge Centers
  3. BIG-IP APM
  4. BIG-IP Access Policy Manager: Secure Web Gateway Implementations
  5. URL Categorization
Manual Chapter : URL Categorization

Applies To:

Show Versions Show Versions

BIG-IP APM

  • 11.6.5, 11.6.4, 11.6.3, 11.6.2, 11.6.1
Manual Chapter
Table of Contents | << Previous Chapter | Next Chapter >>

Overview: Updating URL categories and specifying web traffic schemes

With BIG-IP system Secure Web Gateway (SWG), you can create a configuration to protect your Internet network assets and end users from threats and enforce a rightful use and compliance policy for Internet access. Users that access the Internet from the enterprise go through SWG, which allows or blocks access to certain URL categories. When recommended or configured to do so, SWG analyzes the content in the request and the response to determine whether it represents a threat, and to block access if needed.

SWG supplies over 150 URL categories and identifies over 60 million URLs that fit within these categories. In addition, you can create custom categories if needed and add URLs to any category, custom or otherwise. You can also use custom categories to define blacklists and whitelists.

SWG supplies default URL filters as a starting point for your configuration. For example, the URL filter named default blocks the majority of inappropriate websites. You can use any default filter as a starting point from which to define your own URL filters to reflect your acceptable use policies.

When you are done, you have SWG schemes that you can assign to users when they access the Internet.

Note: SWG schemes are assigned in an access policy and apply to the whole session. URL filters are assigned in per-request policies as HTTP and HTTPS requests are made throughout a session.

Task summary

Use these tasks to download URL categories initially, to refresh them over time, and to specify URL filters that support your rightful use and compliance policy. Before you begin, the BIG-IP system must be licensed and provisioned to support URL categorization.

Task list

About the Instant Messaging URL category

Secure Web Gateway (SWG) supports HTTP and HTTPs-based instant messaging protocols. As a result, when you use the Instant Messaging URL category to block messages, SWG can block messages to ICQ, for example, but cannot block messages from applications that use non-standard ports or tunneling over HTTP, such as, Yahoo Messenger, Skype, Google Talk, and so on.

Similarly, SWG cannot block messages from file-sharing and peer-to-peer protocols that do not use HTTP or HTTPs; most such protocols do not use either HTTP or HTTPs.

Downloading and updating URL categories

For database downloads to work, you must have configured DNS for the BIG-IP device in the System area of the product.
You must download the URL categories for Secure Web Gateway (SWG) to work. You schedule regular database downloads to update the existing URL categories with new URLs. SWG can then most efficiently protect your network from new threats. Without these updates, SWG uses obsolete security intelligence and as a result, protection of your networks is less effective.
Note: You must schedule database downloads for a time with very little no user activity so that users are not impacted. Alternatively, you can initiate database downloads on-demand.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Database Settings > Database Download.
  2. In the Download Settings area from the Downloads list, select Enabled. Additional settings display. Download Schedule displays a default schedule for the download.
  3. In the Download Schedule settings, configure a two-hour window in which to start the download. Schedule the download to occur during off-peak hours. The default schedule is between one and three A.M.
    Warning: After the download completes, database indexing occurs. It consumes a high amount of CPU for approximately 45 minutes.
  4. Click Update Settings.
  5. To download the database immediately, click Download Now. A download occurs only when a newer version becomes available.
    Warning: Database indexing occurs after the download and impacts system performance.

Adding custom URL categories

You can add a custom category to the existing Secure Web Gateway URL categories to specify a list of URLs that you want to block or to allow.
Note: The URL categories that you add become subcategories of Custom Categories. Custom Categories take precedence over other categories.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays. Custom Categories displays as the first entry in the table.
  2. Click Create. The Category Properties screen displays.
  3. In the Name field, type a unique name for the URL category.
  4. From the Default Action list, retain the default value Block; or, select the alternative, Allow. If no action has been specified in a filter for this category, the default action is taken.
  5. Add URLs to the Associated URLs list:
    1. In the URL field, type a well-formed URL that ends with a backslash (/). Here are some examples.
      • https://www.siterequest.com/
      • http://www.siterequest.com:8080/
      • http://www.sitequest.com/docs/siterequest.pdf/
      • http://www.sitequest.com/products/application-guides/
    2. To specify that the URL is a prefix to be used for matching multiple URLs, click the Prefix Match check box.
    3. Click Add. The URL displays in the Associated URLs list. If the URL is used for prefix matching, an asterisk is appended to the URL; for example, http://www.sitequest.com/products/application-guides/*.
  6. Add, edit, or delete URLs to make the list.
  7. Click Finished. The URL Categories screen displays.
  8. To view the newly created URL category, expand Custom Categories. The custom URL category displays in the Sub-Category column.
Add or edit a URL filter to specify an action (allow or block) for the custom category.

Looking up the category for a URL

You look up a URL to determine whether it already exists in the master database and, if it exists, to see which categories include it.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Database Settings > URL Category Lookup.
  2. In the URL field, type the URL that you want to look up. Type the complete URL, including the URI scheme. Type https://www.google.com; not www.google.com or https://www.google.
  3. Click Search.
    Note: Custom categories are not searched.
    Results display in the URL Category table.
If the URL is not found, you can add it to an existing or a custom category. If the URL is found, you do not need to do anything, but can recategorize it by adding it to another category.

Customizing preconfigured URL categories

You can customize the URL categories that Secure Web Gateway (SWG) supplies by adding URLs to them. You might do this after you run SWG for a while, view logs and reports, and determine that you need to make changes.
Note: If you add a URL to a URL category, SWG gives precedence to that categorization and database downloads do not overwrite your changes.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays.
  2. Click the name of any category or subcategory to edit the properties for it. To view and select a subcategory, expand categories. The Category Properties screen displays. There are many URLs in a given category; however, any URLs that display on the Associated URLs list are entered by the user.
  3. Edit or delete any URLs on the Associated URLs list.
  4. To add URLs to the Associated URLs list:
    1. In the URL field, type a well-formed URL that ends with a backslash (/). Here are some examples.
      • https://www.siterequest.com/
      • http://www.siterequest.com:8080/
      • http://www.sitequest.com/docs/siterequest.pdf/
      • http://www.sitequest.com/products/application-guides/
    2. To specify that you want to use the URL as a prefix, for matching multiple URLs, select the Prefix Match check box.
    3. Click Add. The URL displays in the Associated URLs list. If the URL is used for prefix matching, an asterisk is appended to the URL; for example, http://www.sitequest.com/products/application-guides/*.
  5. Click Update. The URL Properties screen refreshes.
  6. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays. The screen displays (recategorized) next to the URL category that you customized.
URLs are added to the URL category that you selected. When categorizing these URLs, SWG selects the customized URL category regardless of whether the URL is assigned, by default, to the customized URL category or any other URL category.

Configuring URL filters

You configure a URL filter to specify the URL categories that are allowed and those that are blocked. You can configure multiple URL filters.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Filters. You can click the name of any filter to view its settings.
    Note: Default URL filters, such as block-all and basic-security, are available. You cannot delete default URL filters.
    The URL Filters screen displays.
  2. To configure a new URL filter, click one of these:
    • Create button - Click to start with a URL filter that allows all categories.
    • Copy link - Click this link for an existing URL filter in the table to start with its settings.
    Another screen opens.
  3. In the Name field, type a unique name for the URL filter.
  4. In the Description field, type any descriptive text.
  5. Click Finished. The screen redisplays. An Associated Categories table displays. It includes each URL category and the filtering action that is currently assigned to it. The table includes a Subcategory column.
  6. To view filtering actions that are assigned to subcategories, expand the category or categories by clicking the plus button for the category or in the table heading.
  7. To block access to particular categories or subcategories, select them and click Block.
    Important: When you select a category, you also select the related subcategories. You can expand the category and clear any subcategory selections.
    Note: To block URLs that SWG cannot categorize, expand the category, Miscellaneous, and select Uncategorized.
  8. To allow access to particular categories or subcategories, select them and click Allow.
To use a URL filter, you must assign it in a per-request policy. A per-request policy runs each time a URL request is made.

Configuring Secure Web Gateway schemes

A Secure Web Gateway (SWG) scheme is a required component of an SWG configuration.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Schemes. The Schemes screen displays.
  2. Click Create. The New Scheme screen displays.
  3. In the Name field, type a unique name for the scheme.
  4. Click Finished.
A scheme goes into effect when an access policy assigns it to a user in an SWG explicit forward proxy or transparent forward proxy configuration; this assignment must occur in the access policy.

Implementation result

Now you have BIG-IP Secure Web Gateway (SWG) configured to regularly download updates to URL categories. Schemes are configured and ready to be added to access policies.

Secure Web Gateway database download log messages

When you deploy Secure Web Gateway (SWG), the database downloads output messages to the /var/log/apm file. This table lists messages that are available only when you enable debug.

Debug message Description
Transfer Status 247 The file is transferred successfully to the BIG-IP system. If you see a Transfer Status other than 247, it might indicate an error.
RTU Type The RTU Type is always 1. If you see an RTU Type other than 1, it might indicate an error.
Expiration Date The BIG-IP system does not use the expiration date in this message. Instead, the BIG-IP system enforces the SWG license and the database download works accordingly.
Table of Contents | << Previous Chapter | Next Chapter >>
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5 Networks, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information