F5 Logo MyF5
Search
  1. MyF5 Home
  2. Knowledge Centers
  3. BIG-IP APM
  4. BIG-IP Access Policy Manager: Secure Web Gateway Implementations
  5. URL Categorization
Manual Chapter : URL Categorization

Applies To:

Show Versions Show Versions

BIG-IP APM

  • 11.5.10, 11.5.9, 11.5.8, 11.5.7, 11.5.6, 11.5.5, 11.5.4, 11.5.3, 11.5.2, 11.5.1
Manual Chapter
Table of Contents | << Previous Chapter | Next Chapter >>

Overview: Updating URL categories and specifying web traffic schemes

With BIG-IP® system Secure Web Gateway (SWG), you can create a configuration to protect your Internet network assets and end users from threats and enforce a rightful use and compliance policy for Internet access. Users that access the Internet from the enterprise go through SWG, which allows or blocks access to certain URL categories. When recommended or configured to do so, SWG analyzes the content in the request and the response to determine whether it represents a threat, and to block access if needed.

SWG supplies over 150 URL categories and identifies over 60 million URLs that fit within these categories. In addition, you can create custom categories if needed and add URLs to any category, custom or otherwise. You can also use custom categories to define blacklists and whitelists.

SWG supplies default URL filters as a starting point for your configuration. For example, the URL filter named default blocks the majority of inappropriate websites. You can use any default filter as a starting point from which to define your own URL filters to reflect your acceptable use policies.

When you are done configuring URL filters, you can group them and schedule them into SWG schemes. In an SWG scheme, you select and schedule URL filters so that at any time of day during a week, only one URL filter is actively being enforced. You can configure different schemes for different groups of users. In a scheme, you specify URL filters that you want to apply at specific periods in the day or and on specific days of the week.

When you are done, you have SWG schemes that you can assign to users when they access the Internet.

Task summary

Use these tasks to download URL categories initially, to refresh them over time, and to specify URL filters that support your rightful use and compliance policy. Before you begin, the BIG-IP system must be licensed and provisioned to support URL categorization.

Task list

About the Instant Messaging URL category

Secure Web Gateway (SWG) supports HTTP and HTTPs-based instant messaging protocols. As a result, when you use the Instant Messaging URL category to block messages, SWG can block messages to ICQ, for example, but cannot block messages from applications that use non-standard ports or tunneling over HTTP, such as, Yahoo Messenger, Skype, Google Talk, and so on.

Similarly, SWG cannot block messages from file-sharing and peer-to-peer protocols that do not use HTTP or HTTPs; most such protocols do not use either HTTP or HTTPs.

Downloading and updating URL categories

For database downloads to work, you must have configured DNS for the BIG-IP® device in the System area of the product.
You must download the URL categories for Secure Web Gateway (SWG) to work. You schedule regular database downloads to update the existing URL categories with new URLs. SWG can then most efficiently protect your network from new threats. Without these updates, SWG uses obsolete security intelligence and as a result, protection of your networks is less effective.
Note: You must schedule database downloads for a time with very little no user activity so that users are not impacted. Alternatively, you can initiate database downloads on-demand.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Database Download.
  2. In the Download Settings area from the Downloads list, select Enabled. Additional settings display. Download Schedule displays a default schedule for the download.
  3. In the Download Schedule settings, configure a two-hour window in which to start the download. Schedule the download to occur during off-peak hours. The default schedule is between one and three A.M.
    Warning: After the download completes, database indexing occurs. It consumes a high amount of CPU for approximately 45 minutes.
  4. Click Update Settings.
  5. To download the database immediately, click Download Now. A download occurs only when a newer version becomes available.
    Warning: Database indexing occurs after the download and impacts system performance.

Adding custom URL categories

You can add a custom category to the existing Secure Web Gateway URL categories to specify a list of URLs that you want to block or to allow. You can use a custom category, for example, as a blacklist or as a whitelist.
Note: The URL categories that you add become subcategories of Custom Categories. Custom Categories take precedence over other categories.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays. Custom Categories displays as the first entry in the table.
  2. Click Create. The Category Properties screen displays.
  3. In the Name field, type a unique name for the URL category.
  4. Add URLs to the Associated URLs list:
    1. In the URL field, type a well-formed URL that ends with a backslash (/). Here are some examples.
      • https://www.siterequest.com/
      • http://www.siterequest.com:8080/
      • http://www.sitequest.com/docs/siterequest.pdf/
      • http://www.sitequest.com/products/application-guides/
    2. To specify that the URL is a prefix to be used for matching multiple URLs, click the Prefix Match check box.
    3. Click Add. The URL displays in the Associated URLs list. If the URL is used for prefix matching, an asterisk is appended to the URL; for example, http://www.sitequest.com/products/application-guides/*.
  5. Add, edit, or delete URLs to make the list.
  6. Click Finished. The URL Categories screen displays.
  7. To view the newly created URL category, expand Custom Categories. The custom URL category displays in the Sub-Category column.
Add or edit a URL filter to specify an action (allow or block) for the custom category.

Customizing preconfigured URL categories

You can customize the URL categories that Secure Web Gateway (SWG) supplies by adding URLs to them. You might do this after you run SWG for a while, view logs and reports, and determine that you need to make changes.
Note: If you add a URL to a URL category, SWG gives precedence to that categorization and database downloads do not overwrite your changes.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays.
  2. Click the name of any category or subcategory to edit the properties for it. To view and select a subcategory, expand categories. The Category Properties screen displays. There are many URLs in a given category; however, any URLs that display on the Associated URLs list are entered by the user.
  3. Edit or delete any URLs on the Associated URLs list.
  4. To add URLs to the Associated URLs list:
    1. In the URL field, type a well-formed URL that ends with a backslash (/). Here are some examples.
      • https://www.siterequest.com/
      • http://www.siterequest.com:8080/
      • http://www.sitequest.com/docs/siterequest.pdf/
      • http://www.sitequest.com/products/application-guides/
    2. To specify that you want to use the URL as a prefix, for matching multiple URLs, select the Prefix Match check box.
    3. Click Add. The URL displays in the Associated URLs list. If the URL is used for prefix matching, an asterisk is appended to the URL; for example, http://www.sitequest.com/products/application-guides/*.
  5. Click Update. The URL Properties screen refreshes.
  6. On the Main tab, click Access Policy > Secure Web Gateway > URL Categories. The URL Categories table displays. The screen displays (recategorized) next to the URL category that you customized.
URLs are added to the URL category that you selected. When categorizing these URLs, SWG selects the customized URL category regardless of whether the URL is assigned, by default, to the customized URL category or any other URL category.

Configuring URL filters

You configure a URL filter to specify the URL categories that are allowed and those that are blocked. You can configure multiple URL filters.
  1. On the Main tab, click Access Policy > Secure Web Gateway > URL Filters. You can click the name of any filter to view its settings.
    Note: Default URL filters, such as block-all and basic-security, are available. You cannot delete default URL filters.
    The URL Filters screen displays.
  2. To configure a new URL filter, click one of these:
    • Create button - Click to start with a URL filter that allows all categories.
    • Copy link - Click this link for an existing URL filter in the table to start with its settings.
    Another screen opens.
  3. In the Name field, type a unique name for the URL filter.
  4. In the Description field, type any descriptive text.
  5. Click Finished. The screen redisplays. An Associated Categories table displays. It includes each URL category and the filtering action that is currently assigned to it. The table includes a Subcategory column.
  6. To view filtering actions that are assigned to subcategories, expand the category or categories by clicking the plus button for the category or in the table heading.
  7. To block access to particular categories or subcategories, select them and click Block.
    Important: When you select a category, you also select the related subcategories. You can expand the category and clear any subcategory selections.
    Note: To block URLs that SWG cannot categorize, expand the category, Miscellaneous, and select Uncategorized.
  8. To allow access to particular categories or subcategories, select them and click Allow.
To use a URL filter, you must add it to a scheme.

Configuring Secure Web Gateway schemes

You configure schemes to specify and schedule a group of URL filters that you want to apply to users.
  1. On the Main tab, click Access Policy > Secure Web Gateway > Schemes. The Schemes screen displays.
  2. Click Create. The New Scheme screen displays.
  3. In the Name field, type a unique name for the scheme.
  4. In the Configuration area from the SWG Service Failure Action list, select the filtering action to take in the event that a service failure occurs.
    • Block
    • Allow
    A service failure condition applies when SWG determines that an error occurred while trying to categorize a URL or analyze the response.
  5. From the Content Scanning (Response) list, select whether and when to scan the content. Content scanning inspects the response web page contents for malicious embedded components.
    • None. No content scanning occurs.
    • Recommended. Content scanning occurs only when the system recommends it.
    If you select Recommended, Max Buffer Size and Max Buffer Time fields display.
  6. In the Max Buffer Size field, retain the default value or type another value. This field specifies the maximum amount of response data (in bytes) to collect before sending it for content scanning. The system sends the content for analysis when the buffer reaches this size or when the buffer contains all of the response content. Otherwise, the data is retained in the buffer.
  7. In the Max Buffer Time field, retain the default value or type another value. This field specifies the maximum amount of time (in seconds) to retain data in the buffer. If the time elapses, SWG allows the response through to the client or sends the client a block page based on the URL category and action that SWG determined before the scan started.
  8. From the Default URL Filter list, select the URL filter to apply if no other URL filter is scheduled.
  9. Click Finished. The screen redisplays. The Associated Schedules table displays.
  10. Add schedules to the scheme: Configure schedules that do not overlap one another.
    1. Click Add.
    2. From the URL Filter list, select a URL filter.
    3. For the fields in the Time Range setting, type or select the time to start using this scheme and the time to stop using this scheme. The default time range specifies 24 hours.
    4. For the Days Valid setting, select the days of the week that this schedule is in effect.
    5. The default settings specify all 7 days.
    6. To add another schedule, click Repeat.
    7. Click Finished.
    If there are gaps in the schedule, SWG uses the default filter to enforce the scheme. The Schemes screen displays.
A scheme goes into effect when an access policy assigns it to a user in a Secure Web Gateway (SWG) explicit forward proxy or transparent forward proxy configuration.

Implementation result

Now you have BIG-IP® Secure Web Gateway (SWG) configured to regularly download updates to URL categories. Schemes are configured and ready to be added to access policies.

Secure Web Gateway database download log messages

When you deploy Secure Web Gateway (SWG), the database downloads output messages to the /var/log/apm file. This table lists messages that are available only when you enable debug.

Debug message Description
Transfer Status 247 The file is transferred successfully to the BIG-IP® system. If you see a Transfer Status other than 247, it might indicate an error.
RTU Type The RTU Type is always 1. If you see an RTU Type other than 1, it might indicate an error.
Expiration Date The BIG-IP system does not use the expiration date in this message. Instead, the BIG-IP system enforces the SWG license and the database download works accordingly.
Table of Contents | << Previous Chapter | Next Chapter >>
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5 Networks, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information