Part 4 - Configuring a Web Application or API: Additional Configurations
Now that we have completed the basic information, crawl settings, and default scan configurations, we can shift our attention to additional configurations designed to optimize scanning and provide granular control over how applications and APIs are tested.
Authentication Records
Using authentication to scan applications and APIs allows users to fully assess their assets for vulnerabilities without limiting scans to unauthenticated content. Additionally, in the case of APIs, authentication may be necessary to even interact with the application. Once you toggle the "Authentication Records" option to enable them, you can select any preexisting record to assign to the configuration. Additionally, you can create a new authentication record to use with the web application or API. This process will be covered in greater detail in a future article on authentication, but WAS supports server, form, OAUTH2, and client side certificate based authentication. WAS can also handle complex authentication processes such as Single Sign-On (SSO) and Time-Based One-Time Passwords (TOTP) through the use of Selenium scripts recorded through the Qualys Browser Recorder (QBR), available as a free plugin for both Chrome and Edge browsers. A link to the plugin can be found here: Qualys Browser Recorder.
A common question we receive is if WAS can support any Two Factor Authentication (2FA) or Multi-Factor Authentication (MFA). With the exception of TOTP, we cannot automate 2FA or MFA. If a mechanism existed to automate authentication through 2FA or MFA, that security feature would be obsolete as it's entire purpose is mitigate the risk of password-based authentication (stolen passwords, password reuse, weak passwords, etc…). TOTP is different in that the token or passcode generated only requires a seed value and a hashing algorithm. Using these with the server time that a request is made allows real-time computation of the correct token or passcode to authenticate to an application. Examples of TOTP include RSA Secure ID, Microsoft Authenticator, and Google Authenticator. Details of how to use TOTP with QBR will be covered in a future article deep-diving into authentication. In the meantime, the WAS user-guide offers a detailed explanation on how to set this up: Qualys Browser Recorder User Guide.
In the event a web application or API uses 2FA/MFA besides TOTP, customers can use Header Injection (see below), scan pre-production environments where 2FA/MFA is not implemented, or work with their identity management teams to use service accounts that do not require 2FA/MFA.
Best Practice Tip! When adding authentication records, be sure to click on the "Set as default" selector for the primary record so you do not need to specify it at scan time.
Header Injection
Header injection allows users to configure a request header that will be sent with every request WAS makes during a web application or API scan. There are two primary use cases for doing this, although other unique situations may leverage header injection as well.
API Endpoint Definition
As APIs are just endpoints without HTML or links, it is necessary to define all endpoints, operational methods, and parameters necessary to interact and fully test them. This can be accomplished through Postman Collections, OpenAPI Specification (OAS) / Swagger Files, or Portswigger Burp Proxy Captures. This configuration allows users to upload these files for use in scanning APIs. If accessing a OAS or Swagger file directly by URL, customers do not need to upload any files under API Endpoint Definition (see Part 1 of this series). If using Postman Environment or Global Variables, those files can be uploaded for use as well. Testing APIs are limited to how complete the Postman or OAS/Swagger files are - if endpoints are missing, they will not be tested. Care must be taken when using proxy captures for upload to fully interact with the APIs across all endpoints using all operational methods.
Setup Exclusion Lists
While this category is called exclusion lists, it includes both allow and deny rules for controlling how an application is crawled and tested. Users can select global exclusions, local exclusions, or both.
Default DNS Override
The Domain Name Service (DNS) is the process by which a domain name (e.g. Qualys.com) is turned into a IP address (e.g. 64.39.96.133) used by both a client browser and WAS for reaching web applications or APIs on the world wide web or inside private networks. By default, WAS will use the DNS for the web application URL to identify the IP address for crawling and testing the web application or API. If you select a DNS override record, WAS will use the mappings in your record instead.
There a few reasons a customer might want to do this. For example, a web application may not have a DNS entry if it's in a non-production environment. Or the web application may have a different IP address in a non-production environment (e.g. development or QA) than in production.
Redundant Links
Redundant links are the most efficient method to optimize web application scanning and reduce overall scan times. WAS is designed to test custom source code for web application and API vulnerabilities. When you consider a catalog sites with hundreds, thousands, or even millions of products, no scanning tool will ever completely crawl every single page. And really there is no need to. Many pages in a catalog application use the same identical underlying source code, they just load different content from a database - such as pricing, product descriptions, reviews, pictures, etc. Catalog sites are not the only web applications reusing identical source code - it could be as simple as one or two press releases each week going back 5 years. That's between 260 and 520 URLs using identical source code that do not need to be individually tested. Redundant links can be used to omit testing of identical source code on pages through regular expression matching. A great resource to test out regular expressions can be found at Regex101.
A few useful things to keep in mind: if you are using complete or partial URLs including the "<http://>"" prefix, you can include both http and https by using http(s*) - This allows the engine to match on http or https. In many instances this can be avoided by setting up a regular expression to match on something unique in the URLs themselves. For instance, if there is a collection of press releases and you have identified them as being good candidates for redundant links, you can use wildcards instead of specifying complete URLs.
As an example, consider https://www.qualys.com/company/newsroom/news-releases/usa/2022/… To include both the http and https version of ALL press releases regardless of year of release, a valid regular expression could be: ./newsroom/news-releases. (note the escape character '' in front of the backslashes '/').
Sometimes customers will state that even after adding a redundant link, WAS is still crawling and testing URLs that should be omitted. This usually happens as a result of running a vulnerability scan before setting up valid redundant link regular expressions. If a vulnerability scan identifies a vulnerability on any URL, the scanner will ALWAYS attempt to retest the vulnerability regardless of any redundant link rules. The only way to remove this prior vulnerability information is to purge the application, and then recreate it with the correct redundant link rules in place. Therefore, it is important to run discovery scans and ensure you are crawling only the links you desire before launching a vulnerability scan.
Path Fuzzing Rules
Path fuzzing rules, much like redundant links, can help improve the efficiency of WAS scans by identifying and reducing duplicate source code testing. An excellent article on path fuzzing can be found here. In short, many web applications will take a URL with parameters, such as:
https://www.example.com/articles/display.php?issue=17&section=sports&article=28
and "prettify" it to:
http://www.example.com/articles/issue/17/section/sports/article/28
One can imagine there might be dozens, hundreds, or thousands of these pages where the source code is simply display.php and it uses the URL parameters to specify what content to load. Once again, it is not necessary to test every single one of these pages, and with path fuzzing rules, customers can specify a URI template for WAS to use. For example, the above URI could be rewritten as:
http://www.example.com/articles/issue/{issue}/section/{section}/article/{article}
During a WAS scan, WAS will identify URIs matching the path fuzzing rule and only test a few of them, greatly reducing the overall scan time. WAS will identify the alterable parameters (issue, section, article) and fuzz those values for vulnerabilities like SQL or command injection using the rules specified instead of testing every single path component of every URI.
Form Training
Form training allows users to provide input for specific workflows during a discovery or vulnerability scan. These inputs may be necessary to allow the application to "move on" to the next step. For example, if testing a web application for car insurance, one of the steps may require a legitimate US Driver's License for the application to assess whether or not the license is still valid and the owner can even obtain insurance. WAS users can specify the URL and form values to submit so that the workflow can be completed and the application can move to the next link for crawling and testing.
Best Practice Tip! Selenium scripts can be used to perform the same function as form training, and are often faster and easier to setup. For more details see the "Crawl Links" section of Part 2 of this series.
Malware Monitoring
Malware monitoring is an included service with each WAS subscription. While Qualys WAS identifies vulnerabilities in the source code of web applications and APIs, Qualys Web Malware Detection (MD) identifies malware present on web applications. You can configure a malware scan against any external facing web application and it will run as a separate scan from your normal WAS scanning. While you could have both WAS and MD scans run concurrently, customers can also choose to have them run on their own independent schedules. MD scans simply crawl the application - there are no vulnerability test payloads with any requests. As a result, a MD scan takes significantly less time to complete than a WAS vulnerability scan.
During the crawl, each URL of the web application is inspected for the presence of malware through four methods:
To configure malware monitoring, you can specify the cadence for which the scans take place. Options include a single occurrence, monthly, weekly, daily, or even hourly. You can set the time that the malware scan will take place, and opt to send an email notification to alert that a MD scan is about to take place.
After a scan is complete, an email notification of any findings will be sent to the Qualys WAS web application owner. Additionally, all MD scan reports can be reviewed directly through the MD module included with every WAS subscription.
Now that we have completed these additional configurations, it is time to click Next and go to the final step in adding a new web application or API - Review & Confirm.
Part 5 - Configuring a Web Application or API: Review & Confirm
The final step when adding a new web application or API is to review the settings to make sure it is configured correctly. You can easily navigate back to any of the prior steps to edit the settings and update them. You can also choose to cancel the web application or API configuration and return to the main Web Application tab in WAS if so desired. Otherwise, at this point you can click on the button to either "Save the Web Application" or "Save the Web Application and Scan".
Selecting "Save the Web Application" will save the configuration and return to the main Web Application tab in WAS. Selecting "Save the Web Application and Scan" will save the configuration but also give you the option to launch a Discovery or Vulnerability Scan on-demand to immediately start testing the web application.
Best Practice Tip! Run a discovery scan before a vulnerability scan to make sure all of your configurations and optimization rules are working properly.
Please join us for our next article where we will cover WAS Option Profiles in detail and provide additional best practices for getting the most out of Qualys WAS.