Disabling all discovery checks in WebInspect

In an earlier posting about limiting WebInspect's scan reach, I mentioned the following complaint regarding the scan Policies.  I wanted to use this posting to explain this further and detail a solution.


<< The Audit-Only scan method sounds good, but it is not foolproof.  Currently most of the scan Policies you may use for your Audit will perform a variety of forceful browsing and discovery checks.  To completely disable this possible expansion of your scan target, you will need to make a custom copy of your desired scan Policy.  I will detail that in a separate posting. >>

 The Audit policies utilized by HP WebInspect (currently at version 9.20) are templates that dictate what quantity of the available attack database will be used against your target during this particular scan.  Even when using the Audit-Only scan method, there are still several ways that WebInspect will bypass your desired limitation and perform some crawling/discovery of the site.

To prevent this, the user will need to use a custom scan Policy.  Open the included Policy Manager tool, then click File > New > {select desired HP policy} > and then save this for your own use.  You may want to annotate the Description to remind yourself what makes this custom copy different.  In this custom Policy you will want to disable the following features.


1. Open the Attack Groups view, then expand the Audit Engines branch.  Disable the following engines underneath it.

• Directory Enumeration
• File Extension
• File Prefix
• Fixed Checks
• Known Vulnerabilities
• Local File Include
• Site Search

2. Within the same Audit Engines branch, locate and expand the Audit Options sub-branch.  Disable the following options found there.  These parsers do not directly request these specific resources, but they would fully investigate them if they were found during the scan.

• CVS entries parser
• Robots.txt parser
• ws_ftp.log parser

3. Move up the top of the Attack Groups tree listing and disable the Web Site Discovery branch, a peer of the Audit Engines branch.


Based on how HP (or SPI Dynamics) originally designed the Audit Policies mechanism, it is difficult to fully disable all crawling even for Audit-Only scans.  Their development team has been contacted about this and understands why it should offer a cleaner method, but until that arrives we must use this work around.


~~~~ Habeas Data

How to use HP WebInspect to scan only a part of a web application

I saw a thread on-line regarding this topic, and realized my response would be too large and look silly stuffed into the post comments.  So I will rewrite and expand my response here.  No offense meant, Rohit.

WebInspect is highly configurable for whatever situation you may be encountering.  Please do not feel that you have to perform all of these configurations.  They are just there for when you need them.


Question:  Can anyone explain the way to scan only a certain part of a web application using WebInspect?

Answer:  There are many ways to "shape" your scan with WebInspect (currently at version 9.20), depending on what you are faced with and your end-goal.  I will review them from most common to lesser known (and least used).


Restrict To Folder:

This feature is not found in the scan settings, but it is on page one of the Scan Wizard.  By enabling this option, the user gets three sub-options as follows, defined in the Help guide.

• Directory only - WebInspect will crawl and/or audit only the URL you specify. For example, if you select this option and specify a URL of www.mycompany/one/two/, WebInspect will assess only the "two" directory. 
• Directory and subdirectories - WebInspect will begin crawling and/or auditing at the URL you specify, but will not access any directory that is higher in the directory tree.
• Directory and parent directories - WebInspect will begin crawling and/or auditing at the URL you specify, but will not access any directory that is lower in the directory tree.

A common error with this Restrict To Folder feature is that the user may not realize that the Starting URL field defines the anchor point for their chosen Restriction. The specific folder is identified by the final portion of the Starting URL that is enclosed in slash marks ("/").  This means that both the Starting URLs "../folder1/folder2/" and "../folder1/folder2/index.html" would anchor to "folder2", but that the path "../folder1/folder2" would anchor onto "folder1".



Session Exclusions:

Let's say that rather than focusing on one area, you wish to omit it from being scanned.  For example, the /manuals/ folder within any default Apache installation is rife with samples and various junk text that will add time to your scan without appreciable results.  For that scenario, open the Session Exclusions scan settings panel and add an Exclusion, such as (URL contains "/manuals/"), and the scan should complete faster.  I had an international client use this to split a very large site separated into three languages (English, Chinese, Arabic) into three separate scans.  Their site structure was primarily segregated, so scanning with Session Exclusions for the other two languages kept the scan targeted to one language area.  Post-scan, they were able to combine these three scans into one report.

The Session Exclusions settings were expanded in WebInspect 9.10 or 9.20.  Besides simple keywords or parts of a URI path, the user can also exclude via a variety of Targets (POST parameter, Query parameter, Status Code, et al) and a variety of Matching styles (Contains, Regex, et al).


Scan Log:

As an added feature, when using the Restrict To Folder or Session Exclusions, you may want to watch Scan Log tab found at the bottom of the WebInspect UI (Summary Information pane).  This area should display informational messages when pages found are removed from the testing coverage due to one of your scan configuration.


Scan Methods:

Stepping out of the settings, the Scan Wizard itself offers a variety of methods for controlling the scan, currently found on page one of the wizard.

Crawl-Only:  This will only perform a Discovery of the target, perhaps with some passive auditing of keywords seen in the traffic.   When completed (or Paused), the user can deselect the undesirable pages or folders in the Site Tree and then proceed to the Audit phase by pressing the Audit button found in the toolbar area.  Bear in mind that each folder is selected individually.  To deselect entire branches, use the right-click menu for additional options.

Audit-Only:  Switching from the default Crawl-and-Audit to Audit-Only will prevent the crawl, or discovery, of the rest of the website.

Manual Step-Mode:  This option turns off automated Crawling.  WebInspect will turn itself into a localhost proxy and spawn an instance of IE.   The user will be performing the discovery phase by hand, by browsing.  When finished, return to WebInspect and click the Finish button found at the top of the Site Tree.  You now have an opportunity to deselect undesired folders or branches in the Site Tree (right-click menu!), and then proceed to the Audit phase by clicking the Audit button found in the toolbar area.

List-Driven Scan:  This is a different style from the automated scan where the crawl engine is provided a list of known URLs at the onset of the scan.  This list can be a XML file listing all of the web root files harvested from the target server by its administrator, or it can be a simple TXT file with one full URL per line.  This style of scan can be used with the Audit-Only method to only attack the pages in the list.  Or it can be used with the Crawl-and-Audit to force-feed the crawl engine with that list input.

Workflow-Driven Scan: Identical to the List-Driven scan, except the input used is a pre-recorded "Start Macro", a browser capture of your desired business process.  Using this Macro feeds the crawler or the Audit-Only with the recorded sessions, and then the scan continues from there.


Scan Policy - Audit Only:

The Audit-Only scan method sounds good, but it is not foolproof.  Currently most of the scan Policies you may use for your Audit will perform a variety of forceful browsing and discovery checks.  To completely disable this possible expansion of your scan target, you will need to make a custom copy of your desired scan Policy.  I will detail that in a separate posting.


Filters:


The defaults for Session Exclusions include a variety of exclusions where the URI may indicate a logout page, such as "exit", "logoff", and "logout".  But what if the offending data does not suit the Session Exclusion model, such as dynamically named folders?  For this, you could avoid that specific data by defining a HTTP Request filter that replaces the offending value or data in real-time.  With a regular expression and Filter, you could dynamically alter all live submissions such as "/pda2789/" (regex="\/pda(\d+)\/") to "/pdareplaced/", a nonsensical value that would cause the server to not identify or honor the request and thereby avoid scanning it.

 

~~~~ Habeas Data

WebInspect is the Honey Badger

I have been selling or supporting WebInspect for years. The one aggravating customer question I always hate, especially when it comes from our newly trained Sales reps, is "Can WebInspect handle my website?" As a preface, I will say that WebInspect is designed only for scanning web sites and web services, so if it uses the HTTP/S protocol, WebInspect is designed to attack it. Obviously the attack database has many specific or named vulnerabilities for due diligence, but the majority of the attacks do not care what the system's technology or brand is.

To help my Sales reps, I built the following graphic. I hope you enjoy it.



For those of you not in on the meme, let me introduce the Honey Badger (NSFW).


~~~~ Habeas Data

WebInspect Data Expansion

There were two hidden settings added back during the WebInspect 7.7 release which can provide the expert user more details within a scan. The first is a developer's Memo header, which is now a little more exposed to the user in the more recent releases. The second is an expansion of the existing Details panel found under the Session Detail area.


1. Memo header:

From the Help Guide:

If you select this option, WebInspect includes a "Memo:" header in the HTTP request containing information that can be used by support personnel to diagnose problems. Although the format and content is subject to change without notice, the information may assist advanced users. Two of the more useful constructions are illustrated below.

Previously this header could only be enabled by manually editing the saved scan settings file (XML) prior to performing the assessment, changing the value from "false" to "true". Now it is just a check box found within the scan settings > General panel, and it is enabled by default.

The data shown in this custom header is mostly for the benefit of the HP ASC developers and Customer Support, as it is in their programming lingo. The standard user will be able to get a rough idea of why this HTTP Request was queued up, such as it was part of the recorded Login Macro, part of the Crawl, or triggered by one of the Audit Engines..

The Help Guide also provides these samples to help decipher the provided information.

• Attack memo header example
• Memo: 197:Auditor.SendAsyncronousRequest:Attack(CID:123:AS:2,
  EID:1354e211-9d7d-4cc1-80e6-4de3fd128002,ST:AuditAttack,AT:
  PostParamManipulation,APD:username,I:(1,0),R:False,SM:2,SID:
  FDF074B3AC41D4ABE4114B3C1A114160,PSID:DDAA45FB26C9149DB15AF2D8DDFD5D3A)
• 
  
• Explanation of memo contents
• Requestor thread id handling request:197
• Originating function in scanner: SendAsyncronousRequest
• CheckID:123
• Attack Sequence: 2
• Originating engine:1354e211-9d7d-4cc1-80e6-4de3fd128002
• Session Type: AuditAttack
• Attack Type: PostParamManipulation
• Attack descriptor (what was attacked): username ‘param’ was attacked, it is parameter (1,0) in collection
• Smart Mode: 2
• Attack Session ID: FDF074B3AC41D4ABE4114B3C1A114160
• Parent Session ID :DDAA45FB26C9149DB15AF2D8DDFD5D3A

• Crawl memo header example
• Memo: 180:ProcessSession:Crawler.CreateStateRequest:
  SID:2BC3FC705779A6F201810A1E64F7CF83,PSID:A77674B6A5BF9B3B3CEDAEF583C08262,
  ST:Crawl,CLT:HTML
• 
  
• Explanation of memo contents
• Requestor thread id handling request:180
• Originating function in scanner: ProcessSession:Crawler.CreateStateRequest
• Session Type: Crawl
• Crawl Link Type: HTML
• Session ID: 2BC3FC705779A6F201810A1E64F7CF83
• Parent Session ID : A77674B6A5BF9B3B3CEDAEF583C08262

2. Details Expanded

This is my favorite secret, as it is always available with no need to enable it prior to any assessment. Unfortunately, it is disabled by default and can only be toggled in WebInspect via a modified user.config file. The good news is that action enables this feature for all scans being reviewed within the UI, not just the selected scan. This feature is also known as the "EnableSupportUI" feature, since it primarily aids the Support team when reviewing customer scans for issues.


Enabling:
1. Close WebInspect.
2. Make a back-up copy of the user.config file located at:
* Win7: C:\Users\%CURRENTUSER%\Local Settings\Application Data\SPI Dynamics\WebInspect\7.0\
* XP: C:\Documents and Settings\%CURRENTUSER%\Local Settings\Application Data\SPI Dynamics\WebInspect\7.0\
3. Modify the user.config so that the appropriate setting tag entry has its Value set to "true" rather than "false".
* For WebInspect 8.x, this tag is "EnableSupportUI".
* For WebInspect 7.7, this tag is "DisplayCompleteSessionDetails".
* If you are missing this setting name, you can just insert it at the end of the other tags.
4. Save and exit user.config.
5. Launch WebInspect, open any scan (new or old), select a session, click on the Details session link, see all the gory details.

So you can see what this feature offers, I have attached before and after screen shots for the same session.

~~ Habeas Data

Differences between WebInspect, QAInspect & AMP.

The three dynamic web application security scanner products now produced at the HP Application Security Center (HP ASC) are WebInspect, QAInspect for HP Quality Center, and the Assessment Management Platform (AMP). These were all originally developed by SPI Dynamics and acquired by HP Software in 2007. All share the same vulnerability database that has been in development since 2000, and they share the same scanning engine and auditing framework. They differ on their interfaces and target audiences as will be described below.

A fourth component will be joining these soon, Fortify's code analysis suite of products. This will be integrated and sold via the same development and sales organization, bolstered by the Fortify staff. However, these products have a different heritage/history and target focus. It remains to be seen which vulnerability database will be merged, or if that will be possible given the differences between SAST and DAST testing.

• SAST = Static Analysis Security Testing
• DAST = Dynamic Analysis Security Testing

WebInspect

WebInspect was designed with the penetration tester in mind. It is a workstation implementation meaning that it runs on a Windows laptop, PC, server, or Virtual Machine, using a Microsoft SQL database as its scan repository. It offers a variety of automated scan methodologies as well as List-Driven and Workflow Driven (Start Macro) options to feed the crawler. It also offers a Manual method permitting the expert to perform the crawling with their own browser while WebInspect audits either afterwards or during this activity. WebInspect is capable of Interactive scans, helping deal with anti-automation techniques such as virtual keyboards or CAPTCHAs or two-factor client authentication requirements such as US CAC cards or RSA ID tokens.

WebInspect provides substantial raw details on the results beyond the verbose remediation data. The scan results may also be exported to an XML file for transport to other security systems or reporting databases. The Report engine offers a series of canned templates from HP, each with its own sub-options, as well as a Report Designer tool for maximum control over the output. The Compliance reports include twenty-three industry templates as well as its own tool for customizing templates.

The secondary tools included with WebInspect are similar to those any consultant might have in their workbench, but these have been written by HP and integrated into the product. These include a web intercept proxy, a SQL injector, web fuzzer, server profilers, regular expression studio, as well as tools to record login macros or to manage the crawler's form values.


AMP

AMP is capable of performing the same automated crawling and auditing as offered within WebInspect, but not the Manual Step-Mode method. The AMP interface requires only a browser, Firefox or MSIE supported, permitting multiple users to generate reports, review findings, or start assessments via remote connected scan engines known as AMP Sensors. In this respect AMP serves as a shared platform for generating dynamic assessments.

Separately, AMP performs as a collation point for all the web app vulnerabilities and reporting within the entire organization, and WebInspect, QAInspect, and Fortify 360 can all be linked to it for uploading of their scan results. This allows AMP to collect and manage the scan results from these varied input points. Using granular security controls (Hierarchical and Roles Based), AMP can limit the resources and results that particular teams or individuals have access to within the interface. The Roles are linked with the user's existing Active Directory or LDAP authentication system so the AMP administrators do not have to deal with a usernames/password management system.


QAInspect for Quality Center

QAInspect is a mature integration plug-in for HP Quality Center, going back to the days when HP QC was known as Mercury's Test Director. QAInspect came about when QA testers found their security teams were identifying defects with WebInspect that could not be tested in the QA cycle, and effectively delaying product releases. They asked for a tool that would permit them to find these same issues themselves, however it must A) not require them to become security experts and B) not require them to use and learn some new tool. QAInspect is fully integrated inside QC so that these two requirements have been met.

With QAInspect, security testing can be completed earlier in the lifecycle, with the results being automatically generated within QC's Defect Tracking Module. This automatic behavior can be throttled by the user, such as dropping all vulnerabilities to a Staging Area for manual "publish" to QC, rolling up identical vuln types into single defects, or rolling up all vulns per page into single defects. Even though both WebInspect and AMP have the ability to send select findings into QC, QAInspect manages these in bulk. In addition, when all those defects come back marked as "Fixed", the QA tester simply re-runs the same Test Run and all corrected defects will be automatically updated to a "Closed" status.

QAInspect 8.0 offers a management tool permitting the QC project leader to pre-define all aspects of the Test Run templates. This helps the project team utilize QAInspect tests in a uniform manner as their manager requires. QAInspect offers the same Reporting engine capabilities as WebInspect. Furthermore, linking QAInspect with AMP allows the results to not only become defects within QC, but also to have the test results uploaded and included in the AMP Dashboards automatically.


The primary aim of the HP Application Security Center is to enable organizations to perform automated security testing. The products attempt to be organic in their growth and integration options, as well as to provide equivalent results and descriptions. One may acquire them all at once, or piecemeal as needed. The addition of Fortify's solutions should expand the SAST and DAST capabilities and integrations of this product set going forward.



.

WebInspect Trial versus Evaluation license

Here is something that is not very complicated, but also not well understood by persons first encountering WebInspect, the Trial License process.

Since WebInspect is freely available for download (https://download.hpsmartupdate.com/webinspect/), anyone can install it and request a Trial License. This is done when launching the product for the first time, when the product realizes there is no applied license. A form is offered and the user completes it with a valid e-mail address to receive the 15-day trial key almost immediately via their mailbox. Return to WebInspect, paste in the activation token, and they are off to the races. The Trial version of WebInspect is completely full-featured in all respects except one, it can only scan HP's demo web server, http://zero.webappsecurity.com.

Now this makes sense. As an automated tool that finds flaws in web sites, and whose crawling traffic could conceivably cause undue server load or extraneous effects, WebInspect is essentially a weapon that a L33T script kiddie would love to have for free. So it is hobbled in this one respect. Clients can still exercise its Report engine, configure scans, et al to their heart's content.

Now here's where it gets strange, and many clients and Sales reps seem to misunderstand. The Trial license is not an "Evaluation" license. The Evaluation license is simply the same activation token modified by an HP Sales rep or engineer to permit scans against any desired IP address, as well as extended beyond the simple 15-day period. There is no cost to getting this license either, other than being a serious client and opening a dialog with your HP Sales rep.

Some would say the Evaluation license should never be given out, because the client will just fix their issues and then fade away without purchasing. This is counter to the understanding that security testing needs to be carried out full-time as part of a mature and on-going Software Development Process (SDLC). Sure, the immediate issues were found, but the true value of the tool is in verifying this same correction for all applications, both live and in development, into perpetuity.


As for QAInspect for Quality Center and HP AMP (Assessment Management Platform)? These also offer the Trial vs Evaluation license scenario listed above. Both of these products are a bit more involved in terms of the supporting installations required, so most clients never start off with the Trial licenses without also being in touch with their HP Sales rep.



---

Validating vulnerabilities found in WebInspect

Today I would like to cover how to use WebInspect to validate its findings rather than just reading the remediation details. This is a common stumbling block for users who are new to web application security testing and not simply this HP product. While the tools discussed here are all included with the WebInspect product, the included user documentation is unfortunately aimed at how the features work or which buttons are available and used for and not truly focused on how to use this sort of tool for penetration testing. I guess that's why they call it a product guide and not a job training guide. ;-)


Data At-Hand

WebInspect offers a lot of details right there in the Session Info panes. Raw HTTP Request, HTTP Response, parent and child Links, general Details, et al. The HTTP Request will include a request header called "Memo" by default, unless you disabled that under the General scan settings, and this Memo header lists in HP developer short-hand a description as to what generated this individual request. There is even a way to change a configuration file and expand the Details pane before or after a scan to expose a lot more data regarding this single session. The Request and Response data will frequently have the significant items (attack) colorized with red high-lighting and this will carry over into the vulnerability reports, although this feature is only available on an individual security-check-by-check basis, at the time they were being defined by HP's researchers.


Right-click Menu

To make the review and editing of scan results simpler, WebInspect offers a right-click menu both within the Vulnerabilities pane (bottom of screen) and in the Site Tree pane (left of screen). This menu is nearly identical in both panes except that the one available within the Site Tree starts with four options for managing the branches of the tree, a tree export function, and an option to generate a small Single Session report for this one single page/session.

The right-click menu offers the following general capabilities.

* load original HTTP Request into browser ready for replay.

* edit the vulnerability description, its severity, or add additional findings (factory-provided or customized ones).

* mark the finding as a False Positive or even remove the finding completely. The FP items will then be charted on the Dashboard, and shifted from the Vulnerabilities pane to a new False Positives panel, and will be suppressed from all reports except the False Positives report, naturally.

* add a comment to the Vulnerability or the Session (page), which can be included as part of the Vulnerability Summary report later. Such annotations can be located from the overall Notes panel in WebInspect if one loses track of them.

* load the original HTTP Request into some side tools for replay, e.g. HTTP Editor or SQL Injector.

* Send the finding and all remediation details into your available defect tracking system, if it happens to be HP Quality Center or IBM ClearQuest. This will add a vulnerability note with the resultant defect number and such details to the WebInspect session thus affected. Nice way to close the "air gap" between the security tester and the developers who need the data to fix the issue. Who needs PDFs?


HTTP Editor

This side tool is a raw request editor allowing you to perform hands-on hacking, very similar to using common freeware intercept proxies or the Tamper plug-in for your browser. Send the Request and see the Response in real-time, including Raw, Browser, and Hex views of the results. There is a Show History button so you can compare or review the requests you have been making, and the data can be saved in order to share with another user who also has access to a copy of the HTTP Editor.

This tool can also be used as a step-mode browser, so if session state is fouled up and the single replay is not initially working as expected, you can use the Response's Browser window to log into the application and then go deliver the payload desired. This process is done (slowly) by filling out the login fields shown in the Browser view, notice that the Request gets altered, click the "Send" button, see the Browser view response, click a button or link there to modify the Request accordingly, click "Send", and keep repeating that use of the Send button to walk through the site. Once session state is established, you can jump directly to the page by modifying the URL field and Request as needed.


Intelligent Engines - XSS and SQLi

These patented engines from SPI Dynamics/HP circa 2007 actually send an entire series of appropriately-encoded probing requests against the page/form and review the responses to ensure that the attack was not sanitized but is indeed vulnerable. The trouble for the user is that the WebInspect UI or its scan database can only seem to show/store one single Request/Response pair, and that is typically the final one made. So the user may be looking at something relatively basic like a single-tick OR probe ("'%20OR") with a 200OK or 500 error and then wondering how HP's tool figured that this means there is "Confirmed Blind SQL Injection", when truthfully this Request/Response pair was the last of a series of perhaps up to 20+ probes.

Verifying the XSS findings is usually straight-forward. Load the HTTP Request into your browser or HTTP Editor and replay the attack, reviewing the response closely. The Web Browser View pane within WebInspect's UI can also be used for many of these instances, as it has Active Content enabled in the application settings by default. Some of these probes are simple alert pop-ups, others require User Interaction (e.g. onmouseover), and others will require that you understand Javascript and can "just see" with your developer's eye that the Response is unsafe. If you want to see the XSS engine in full action, you will need to repeat the assessment using the same scan settings but focused just on the page of interest, running the scanner through an intercept proxy such as the included Web Proxy tool.

Verifying SQL Injection is a similar process of replay, with different tools. Bear in mind that HP will identify four primary types of SQLi in two classes: Possible SQLi, Possible Blind SQLi, Confirmed SQLi, and Confirmed Blind SQLi. The two "Possibles" indicate that WebInspect did not get into the database, but saw varying results to its True/False probes. These "Possible" findings generally boil down to input validation issues, such as "Why did the Zip Code field permit script characters and 67 characters of input?" ;-) The "Confirmed" findings indicate that WebInspect went beyond the initial True/False probes and actually got valid SQL server responses to direct SELECT statements. These "Confirmed" items are ripe for a probe with the SQL Injector tool.

And yes, the SQL Injection engine works on more than just MS SQL, specifically MS SQL, MySQL, Oracle, Postgres, DB2, and (sometimes) MS Access.


SQL Injector

Assuming you have a "Confirmed" SQLi finding, I would load the vulnerable session/request into the SQL Injector via the aforementioned right-click menu, as this automatically carries over any Authentication settings you may have had in the original assessment. You could also do all of this set-up the hard way, by manually launching the tool separately from the Tools menu or Windows Start menu and then building out the necessary settings and session state. Once the vulnerable request is loaded in the Injector, simply click the tool's "Send" button and wait for the status on the lower-right section of the Injector's screen. If the tool says injection is possible you may now proceed to dumping the entire database or select quantities of Tables/Columns/Rows using the buttons at the top toolbar of the screen. The stolen data can be saved as a *.SDF (SQL Database File), and that data can be managed or queried using SQL Studio Express for real shock-and-awe at your presentation later.

Sometimes the SQL Injector tool will state some other status message, such as "Data Extraction Not Possible". This does not mean "not vulnerable to SQLi", it simply means that some complication (permitted queries, clustered databases, MS Access in use, et al) is preventing the use of SELECT queries to steal the data. Both WebInspect and the SQL Injector only make use of the SELECT query method, as this is the safest in most conditions. It is quite possible that the injection point will permit you to exploit other methods such as DROP, UNION, et al, all of which will require a bit of coordination with the DBA before you start those types of actions with other tools.

If you want to see what the SQL Injector does, or you want to capture the data so HP's ASC Customer Support can take it to the developers for review, you can capture it in an intercept proxy. First, modify the SQL Injector's settings to use an intercept proxy such as the included Web Proxy tool (default = 127.0.0.1:8080), prepare your proxy, run the SQL Injector attacks through the proxy, save the sessions captured in the proxy, and also save the SQL Injector data (*.SDF) to accompany it.


Repeat Scans

Sometimes you need to reassess the site, or just the portion of it that the developer corrected. This can benefit from the use of a saved scan settings file. If you did not save the settings during the original Scan Wizard, you can still acquire the settings file from your completed scan by opening it on screen and clicking on the Edit menu > Current Scan Settings > Save As. Once saved you can further modify these settings from the Manage Settings item found under the same Edit menu. Repeating a scan then only requires selecting this saved XML file from the available choices shown on the Settings button pick-list at the bottom-left of the Scan Wizard screens. I would select the desired settings file on the first screen, as it will reset everything else and you will need to refill the necessary fields.

If you do not want to scan the whole site just to validate a small portion, there are the following features available, based on your need, and assuming that you are using the saved scan settings.

* Audit-Only scan method - no Crawl or Discovery of the rest of the site!

* Deep-linked (long) Starting URL coupled with one of the Restrict To Folder options and using the Audit-Only scan method.

* List Driven Assessment - use a simple input file, or harvested folder listing from the web server. When used with the Audit-Only, only those pages will be scanned.

* Workflow Driven Assessment - Similar to the List Driven, except it uses a prerecorded Start Macro of the desired pages or business process.

* Session Exclusions - Omit all URI's containing a keyword or partial folder path.



.

WebInspect and Scheduled jobs

Many users may be unaware that WebInspect offers basic scheduling capabilities. I know that I prefer to watch my scans begin and run in case anything goes awry, but scheduling is a great tool for those recurring assessment targets where you have already performed assessments in the past and essentially know what to expect.

Bear in mind that while the WebInspect UI allows up to two simultaneous scans, the Scheduler, CLI, and Enterprise Assessment methods being discussed today are limited to running only single scans in series, one after the other.



1. Scan Settings File:

Obviously having the settings saved ahead of time will make the scheduling or repeating that much simpler. The scan settings reside as XML files on your machine, and they can be accessed or edited from several methods as follows.

* If the site has already been scanned successfully, save its settings as a settings file for re-use later. To do this, open that scan on-screen, click on the Edit menu > Current Scan Settings. The lower left corner offers the choice to "Save Settings As", to XML format.

* Or, pretend you are starting the scan, and on the final/fifth page of the WebInspect 8.1 scan wizard there is now a link that allows you to save the configured settings for later use. At this point you may cancel the scan wizard without running the assessment.

* Or, open the Default Scan Settings, modify everything you want, and then use the "Save Settings As" option found on the lower left-hand corner of that window. Access this screen from the Edit menu > Default Scan Settings, and using the Cancel button afterwards will ensure your actual Defaults are not modified.

* To edit your settings file further just use any of the methods above, or access it directly from the Manage Settings tool found under the Edit menu. Loading it in as your Default Scan Settings, and then re-saving it afterwards works well also.


2. Now set up the Scheduled job:

On the Start Page tab, click on the Manage Schedule button > Add, or just use the Schedule button found in the top toolbar to get to the same wizard. Proceed through the offered Schedule Wizard as follows. Having a saved scan settings file makes this simpler, but that file is not required.

The first screen of this Schedule Wizard deals with "When" to run it (e.g. Recurring, Weekly). The next five screens are the normal Scan Wizard the user always uses when kicking off assessments. The Settings button at the bottom of the screen allows you to simply drop in the saved setting file if you had one. The wizard's sixth page has an option to define a particular Report to auto-generate upon completion.


3. Scheduler background service:

To ensure the Scheduled Scan actually runs, make certain that the Windows service, "WebInspect Scheduler Service" is set to Automatic, i.e. it has not been disabled. If you prefer to set it to Manual, you have to manually start it before you leave for the night/weekend. The "HP ASC Monitor" icon in the System Tray provides a short route to start this service manually, or it can be managed through the Windows Control Panel (Administrative Tools). The scan will run so long as this scheduler service is running, regardless of whether the WebInspect UI is open or not.



Command Line Alternative:

WebInspect also offers the same automated scan methods via the Command Line Interface (CLI) that the Scheduler offers. Details can be found in the UI's Help guide under "Command Line Execution", or via "wi.exe -?". Among others, WI.EXE has an option to use a previously saved scan setting file. This version of WebInspect's scanner can be called from the CLI, any BAT file, the Windows AT command, or similar Windows features/tools that can be used to schedule and run CLI tools. The CLI scan option can be run regardless of the status of the WebInspect Scheduler Service, as they are not related or linked.


Back-to-Back Scan Options:

If you wish to run a series of scans, it is preferable to use the Enterprise Assessment to set that up, since it combines the Scheduler with one of the two primary assessment types (Web Site Assessment or Web Service Assessment). The normal Scheduler can kick off the scan, but it has no regard as to when it will end. That sort of additional awareness and scheduling flexibility was left to the enterprise solution, HP's Assessment Management Platform, which includes scan priorities and Black Out periods. If two (or more) WebInspect jobs are scheduled back-to-back, the second one may not start because the previous one has not yet completed, and when it does the window of opportunity to begin the next assessment may have long passed according to WebInspect's internal programming. The Enterprise Assessment automatically runs the configured scans in series, one after another.


Monitoring the status:

For all of these methods, bear in mind that there is no UI access into the scan's progress. You can see that it is running, and kill the service directly or the WebInspect.exe or WI.EXE process to halt it, but one cannot peek into it in real time. If you end the scan prematurely, the scan can always be opened in the UI and then the Resume button will allow the assessment to complete by running it within that UI interface.


.

WebInspect, background processes, and Windows services

HP WebInspect installs with two active Windows services running by default. Being "old school" and primarily DOS-taught, I dislike unnecessary services eating up memory, even though small by today's standards. Here is the reasoning and purpose for these particular services.


Services Installed

First service is the "WebInspect Scheduler". This is actually used by WebInspect, but only if the user plans to run assessments via the Scheduler tool. This service ensures that the scheduled job will run even if the WebInspect UI is closed. If you never plan to use the Scheduler, or only rarely, change this service from Automatic to Manual or Disabled. If set to Manual, you can always start it up from the HP ASC Monitor process mentioned below.

The second service is completely unnecessary for the WebInspect user, and that is the "AMP Sensor for WebInspect" service. This is only needed if you are connecting this workstation to an HP AMP Manager server to serve as one of its remote scan engines. If that is you, and you also have a copy of WebInspect, you will want to separate these to two separate workstations anyways. The reason is that while you may be happily running up to two assessments in the WebInspect UI, the AMP Sensor service might also be running a third in the background for some user on the AMP infrastructure, resulting in poor performance for you. So if you use AMP, install and configure the Sensor somewhere else, and whether you do or do not use AMP, set this service to Disabled on your WebInspect workstation.

Lastly, there is a helper process that runs in the System Tray known as the "HP ASC Monitor". It looks like a "black hat hacker" icon, and is just a short-cut to stop and start the two services mentioned above, or to configure the AMP Sensor's connection to the AMP Manager. This process can be closed without issue. You can always bring it back from the Windows Start Menu, under the HP WebInspect listing.


Running Processes

Now let's cover the actual processes used while scanning. Obviously the WebInspect.EXE process is the WebInspect application itself. If you happen to run the command-line version, you will see those scans being run by WI.EXE.

You may also see the ScriptServer.EXE from time to time, eating up an especially large amount of resources when scanning script-dense web sites. This is a helper process that parses the scripts for the WebInspect executable. This was built to avoid the natural .NET limitations in per-process memory use. Even if you have loads of installed RAM, the .NET framework has natural limits on the total memory any individual process may use, so by splitting this load between two processes the WebInspect scanner can chew on more of your site. Nice, eh? The ScriptServer process will appear and disappear as it is needed, and if it crashes it just respawns and continues without a hitch from where it left off. If you wish, you can kill this process and it will rebuild itself, which I have to admit to using on some pen tests when resources and screen activity seemed a little frozen.


Special Services

If you were to watch the WebInspect start-up via a network sniffer, you will find a series of requests being made. One series will be DNS checks for any Host Names specified in your Allowed IP Ranges attribute of your product license. Another series will be secure web requests being made to the HP SmartUpdate portal and their license service. If your license was recently updated, this process will fetch your new capabilities. The SmartUpdate service will fetch any updates to the vulnerability database and/or the product binaries, provided your license has not expired. Got to pay that Maintenance fee!

A rarer service that may occur is the Support Channel feature. When marking a False Positive, there is an option to send the item in question as quality feedback to the HP research team anonymously. This bundle of data may be sent immediately or queued to run once a connection can be made. There are similar ways to send enhancement requests or bundles of support-related data from the Help menu, but all of these use the Support Channel function. Those Messages you see in the Start Page tab are also delivered as part of the Support Channel check-in.



.

WebInspect's scan repository and full SQL Server

HP WebInspect defaults to using Microsoft SQL Express as its scan repository and reporting workspace, but one could instead use a full SQL Server.

WebInspect 8 SP2 (January 2010) added support for SQL 2008, not just the existing SQL 2005 allowed. The benefits of using a full SQL Server is that there is theoretically no limit on the individual scan size, compared to the 4 GB limit native to SQL Express. It can also be managed and backed-up via the corporate database administration instead of being at risk stored on one local hard drive.

One additional item is that the HP support team indicates a possible scan performance increase of up to 15% when the full SQL Server component is not installed on the same machine as WebInspect. This has more to do with Microsoft SQL than with WebInspect. I have no such statistics on the comparison between using SQL or SQL Express on the local machine, this fact is just for the case of full SQL version.



To make use of a full SQL Server, you will need the following guidelines.

* Do not install SQL Express, only install WebInspect. If you are already using SQL Express, that is fine, keep it around until after you transfer your saved scans.


* Configure the WebInspect scan repository connection manually by opening the Edit menu > Default Scan Settings > Data Connectivity panel.

Fill in the server name, credentials, and the database name, e.g. "WebInspectScanDB". The database utilized should not be used by any other applications, but it may reside on a SQL Server or cluster alongside other databases without any issue. The dialogs on this screen allow the user to create the new database on-the-fly, assuming that the current Windows user or supplied SQL credentials have DB Owner rights to the target SQL Server.

After the creation of the database, the connection credentials can be downgraded to simply DB Administrator privileges if the SQL Server administrators prefer. Alternatively, the administrators could generate the blank database ahead of time for the WebInspect user, leaving them with only DBA rights to this blank database.


* Configure the WebInspect reporting database connection by manually opening the Edit menu > Application Settings > Reports panel.

Just as before, fill in the server, credentials, and a different database name, e.g. "WebInspectReportingWorkspace". This cannot be the same as the one used to store the scans! This database can be created on-the-fly and has the same DB privileges requirements as the scan repository created in the prior step. This will be a workspace used by WebInspect when generating reports.


* Return to WebInspect's Start Page tab and open its "Manage Scans" section. Configure the Connections button to ensure this screen is now showing the contents of the scan repository database configured previously. If you happen to also use SQL Express for WebInspect, this screen allows you to list the contents from both databases at the same time. Bear in mind that "local" means "SQL Express" and "remote" indicates full SQL Server (whether on localhost or not). If this is not clear, customize the available columns to include the Location and/or Database column.


* Two or more WebInspect may share the same SQL database for scans, but it is not optimal. There could be conflicts if multiple users open the same saved scan simultaneously and begin modifying it. Since the reporting workspace is temporary, it is probably fine to share among several users.


* Lastly, keep the SQL Server logically close to the WebInspect machine. Do not try to store your real-time data across the English Channel or some such enormous distance and then complain of the latency or product instability. I am not making this up.


AMP Sensor

If you happen to be a customer using HP AMP (Assessment Management Platform), all the aspects of installing WebInspect apply equally to the AMP Sensor, those remote workstations that actually perform the AMP assessments. These two products use the same install file, although with slightly differing install folders. AMP Sensors also default to using SQL Express, or can be configured to use full SQL Server. They also benefit from using a remote rather than local SQL Server. And unlike WebInspect, they can share a single scan repository database as they only store their current scan there. Once completed, the scan is uploaded to the AMP Manager server's database and the Sensor's copy is wiped.

WebInspect's scan repository and SQL Server Express

Many HP WebInspect evaluations or anonymous trial downloads have been complicated by the Microsoft SQL Server Express component. In today's posting I will address pretty much everything you need to know to properly configure this database for WebInspect. Any time I refer to "SQL Express" here it should be understood as "Microsoft SQL Server 2008 Express Edition" and/or "Microsoft SQL Server 2005 Express Edition", and not any other versions. My original posting was delayed because WebInspect 8 SP2 had just added support for SQL 2008, but thus far it works the same and so here's your post.


Requirements

By default, WebInspect 8.0 SP2 expects to use SQL Server Express (2008 or 2005) as its scan data repository. This is used in real-time to store the assessment findings. WebInspect expects this service to be running locally with the default named user instance of "SQLEXPRESS". SQL Express is not a true prerequisite that prevents the installation of WebInspect from completing, nor is it required to open the WebInspect UI and poke around. However the moment you try anything that touches the scan database, such as importing the Sample Scan or any other exported scan file, or kicking off a new assessment, you will discover this missing piece. The reason that this is not a hard installation requirement is that the user could instead use any full installation of SQL Server (2008 or 2005) that is at their disposal, whether it is installed on the localhost or elsewhere on their LAN. In such a situation one would not need SQL Express installed at all (see next blog entry for that).

At present, SQL Express 2005 is up to Service Pack 3, which is the preferred version for WebInspect 8, even though 2005 Express' Service Pack 2 will also work. For SQL Express 2008, you will want the Service Pack 1 (SP1) version. SQL Express has the benefits of being free and relatively lightweight on the machine, although it does have a natural 4 GB limitation on the amount of data stored per assessment. Obviously, switching to use a full SQL server would negate this 4 GB upper limit. Most assessments by WebInspect are less that 1 GB in size, so this would only be an issue with large enterprise sites, highly dynamic sites, and/or scans against such sites without optimized scan settings. Another benefit of using a database is that in the event of a crash or shutdown, the scan data is safe and the assessment can simply be Resumed once WebInspect is re-opened.


Installation

Once SQL Express is properly installed and running, there is no other care or feeding required in order to use it with WebInspect. So how to we get there?

1. Assuming a 32-bit OS, acquire the 32-bit version of the SQL Express install file, "SQLEXPR32.EXE". Do not use the installer that supports both 64-bit and 32-bit OS unless you are specifically using a 64-bit OS! I have also seen troubles when using the installer that includes SQL Studio Express. Just get the lean install for now.

2. Run the installation with the following caveats.

2a. Enable the box to "Hide the Advanced Installation Options". The options this would provide are generally not needed and can cause you trouble on your "default" installation. If you need to put SQL Express on a non-C: drive, you will need to use the Advanced settings, but keep your demands/settings simple.

2b. Enable any box asking to Create or Generate "Named User Instances". This is the crux of how SQL Express operates. Do not change the named user instance from "SQLEXPRESS".

2c. Enable any box asking to Add the Current User to the database administrators (dba).

3. You may start using SQL Express and WebInspect immediately after the installation, but a reboot is recommended to shake out any issues from the install.


When running, SQL Express will be represented by the Windows process, "sqlsrvr.exe". If you wish to interface directly with the SQL Express system, there is a CLI tool (SQLCLI.EXE) included with the installation or you may download and install the Microsoft SQL Server Management Studio Express for your version (2008 or 2005). The scan data held within the resulting WebInspect scan files (*.MDF) is visible using either tool, although certain columns or tables may be encoded/encrypted and there is no published database schema. HP's Application Security Center and its developers reserve the right to alter the database schema at any time in order to maintain and implement product features. You can still figure out a good quantity of the filed linkages by yourself, but just expect your assumptions to change over time and releases.


Scan Initialization Failed

By far the biggest error associated with WebInspect and SQL Express is the "Scan Initialization Failed" message. This is the nasty-gram you will receive if SQL Express is not configured correctly or even running, and it will occur only when you seek to interface with the database (importing, launch new scan, view the Manage Scans pane, et al). If you swear that the database is running properly, you can configure WebInspect to use SQL Express as if it were a "remote" database, i.e. full SQL Server installation, by filling in the (full) SQL Server option and pointing to the local "SQLEXPRESS" database by name. This works for verification, but not as a long-term solution as all of your scans will be stuffed inside that one database instance with a total upper limit of 4 GB. But if it works, then you know the problem is with your SQL Express Named User Instances and not the product itself. Below are the primary items to review for the Scan Initialization Failed message.


* Clear the appropriate SQL cache folder:

- XP: C:\Documents and Settings\%CURRENTUSER%\Local Settings\Application Data\Microsoft\Microsoft SQL Server Data\SQLEXPRESS\

- Vista: C:\Users\%CURRENTUSER%\Appdata\Local\Microsoft\Microsoft SQL Server Data\SQLEXPRESS\

Erasing this cache folder and re-opening WebInspect will cause the folder to regenerate. This can be necessary even after you needed to correct something else, as this folder could be holding a corrupted form of the entries, preventing your fix to truly operate. You may need to Stop the SQL Server service and/or exit WebInspect before you will be allowed to delete this folder.


* Ensure you are not using Windows XP SP2 over RDP or Terminal Services. Microsoft had a known issue with this combination in that SQL Express would not know what Named User Instance to use for the remote user. Connecting once locally could fix this, as could a special Microsoft Support patch file. The best fix is to upgrade to XP SP3, since you will be better off in many other ways than running SP2.

Reference: SQL Express RDP Patch - http://support.microsoft.com/?id=896613 - "You cannot connect to Visual Studio SQL Server Express on a remote Windows XP Service Pack 2-based computer "


* (rare) Delete the Scans.XML file used by WebInspect to index the saved scans. This file will be be rebuilt when WebInspect re-opens. Assuming product defaults, this file is located in the following directory. If not using defaults, this file will be wherever you have defined your \ScanData\ folder under WebInspect's Application Settings screens ("Directories").

folder: C:\Documents and Settings\%CURRENTUSER%\Local Settings\Application Data\SPI Dynamics\WebInspect\7.0\ScanData\

Also, verify that there is a "version.txt" file accompanying the Scans.xml file. It can be created by hand by copying the current contents from another up-to-date ASC product installation or from the ASC Support team.


* Verify the user is logged on as a Local Administrator, and not using "Run As". Also rather rare, but it can happen on more secured networks.


* Verify that the "SQL Server (SQLEXPRESS)" Windows service is being run by the default service name, "NT AUTHORITY\NetworkService". This Initialization error can occur when a user attempts to configure the "Run As" a non-Local Administrative domain user or network service account such as "DomainName\SQLServer2005SQLBrowserUser", or if the NetworkService has been limited in some way by Group Policies. Switching to your own (Administrative) user account might correct or test this scenario.


*Even more rarely, the installation is done properly but the Named User Instances are not truly enabled during it, behind the scenes. Looking into the database will report that they are enabled, but then using this quick CLI process you can reset them to enabled. You can run the same two queries from the MS Studio Express as well, but they must be run separately, it will not correct the issue if they are run as one query command.

CLI Location: C:\Program Files\Microsoft SQL Server\90\Tools\Binn\SQLCMD.EXE


Steps for enabling named user instances:

1. Use the -S option to specify the local machine and the SQLEXPRESS default instance name.
2. Enter the first command followed by GO.
3. Enter the second command followed by GO.
4. Type "exit" to leave the program.

* Example:

SQLCMD -S machine_name\SQLEXPRESS
sp_configure 'user instances enabled' ,1
GO
RECONFIGURE
GO
exit


Enjoy!

(eof)

WebInspect's Manual Step-Mode and daisy-chained proxies

This post is a follow-on from the prior post on how to daisy-chain WebInspect with an intercept proxy.

A powerful penetration testing feature of WebInspect is its Manual Step-Mode crawl, suited for very complicated web applications. When this style of crawling is used rather than the automated crawling methods, WebInspect is actually spawning a hidden instance of the included tool, HP Web Proxy, offering even more daisy-chaining fun for the security expert. During this time, WebInspect listens on a dynamically set port on localhost (127.0.0.1) and opens an instance of MSIE pre-configured to that port. This Manual crawl is very similar to how some freeware web scanners operate, with the professional manually driving the scanner via their browser and the tool parsing and auditing the captured sessions in the background. WebInspect's dynamic port behavior for Manual Step-Mode can also be changed to use a static (expected or known) port rather than a dynamic port, which is very useful if you want to use an alternative browser. When doing this, you must leave the triggered MSIE window open (minimized) because that is what WebInspect actively hooks into and signifies to WebInspect that it is in "Manual mode". However, you can then launch and configure your favored alternative browser to the static port (proxy port) you set previously. The scenario below will detail this further.


Let's look at an advanced configuration. I had a call from a counterpart who was on-site with his client. The WebInspect machine they provided him had MSIE 8 installed, and no matter what he did, the development application they were testing crashed that browser, and it simply refused to work with any non-Microsoft browsers. This would normally not affect WebInspect's automated scanning methods as WebInspect operates as its own stand-alone browser. However, this site was completely ActiveX with browser plug-ins and client-within-a-browser configurations that required a human to use the Manual Step-Mode of WebInspect to "crawl" the site by hand with a plug-in enabled browser. Since he could not downgrade the MSIE browser on the workstation nor automate the scan, he called me and here is what resolved the trouble.

In addition to the WebInspect workstation, the tester was provided RDP access to a second machine which still had MSIE 7 on it. This version of MSIE worked great for browsing the development application without any crashing as had been experienced with MSIE 8, but the machine lacked WebInspect. To audit the application we daisy-chained the two workstations across the LAN, using the little-known static port setting for Manual Step-Mode and then opening the Web Proxy to the LAN rather than just the localhost, as follows.

1. On workstation1, open WebInspect's Edit menu > Application Settings > Step-Mode panel > set it to use a static port not currently in use, e.g. 8081. Leave the IP address field as localhost (127.0.0.1).

2. Start the Step-Mode Manual crawl assessment in WebInspect for the target site, and once it is running Manual mode leave WebInspect and the resulting MSIE8 window alone.

3. Now open HP Web Proxy on the same workstation1.
* Within the Proxy Server settings, configure Web Proxy to use the Manual Step-Mode proxy server that is "upstream" at 127.0.0.1:8081
* Within the General settings, configure Web Proxy to listen on workstation1's actual network IP and port 8080, e.g. 192.168.1.105:8080
* Save these settings and now start the Web Proxy service/listener from its toolbar. The lower left-hand corner of the Web Proxy window will show that it is now Listening and the port being used.

4. RDP to workstation2, open MSIE7 there, configure MSIE7 to use the proxy server at workstation1, e.g. 192.168.1.105:8080

5. Browse the test application from MSIE7 on workstation2, verifying that the WebInspect window on workstation1 is showing captured/audited traffic within its Navigation Pane (Site Tree view).

6. When finished testing from workstation2, return to workstation1, close the MSIE8 window that popped up previously, and click the "Finish" (Step-Mode) button found at the top of WebInspect's Navigation Pane. This halts the Manual Step-Mode crawl. It can be Resumed if needed.
* WebInspect is now ready to proceed to Auditing, if you had the Step-Mode set to "Manual Audit" mode.


Representation for our example:

MSIE7 browser (on 192.168.1.104) > HP Web Proxy (192.168.1.105:8080) > HP WebInspect (127.0.0.1:8081) > network proxy (192.168.1.200:80) > target server (192.168.1.106:443)


What about serving Manual Step-Mode directly to workstation2?

If you read my previous article on daisy-chaining WebInspect and Web Proxy, I must point out here that the Manual Step-Mode settings do permit the local LAN to be served directly if the localhost IP address is replaced with the workstation's own live LAN IP address. However, in this scenario, we wanted the ability to monitor the browser traffic from workstation2 within the Web Proxy window on workstation1. This would help diagnose and connectivity troubles, if that traffic was not able to pass through WebInspect and then back from the test application. This offered the added bonus that the captured Web Proxy sessions could be saved as a proxy sessions file (*.PSF) for review at a later time. Those sessions could also be saved as a Start Macro which could be used later in an attempt to run the WebInspect audit automatically rather than manually. Sometimes this automated trick works, sometimes it does not, depending on the application itself, but having the option to try is very useful.

HP WebInspect and proxy daisy-chaining

This is my inaugural posting, so here goes. I am all new to this, and yadda yadda yadda, let's get started. (disclaimer) I have worked with and supported WebInspect, QAInspect, DevInspect, and the Assessment Management Platform for several years. If I abbreviate these products in any postings as WI, DI, QAI, or AMP, you will just have to forgive and read on. Also, I like to share.

HP WebInspect is an advanced penetration testing tool used solely for auditing and exposing vulnerabilities in web sites. It works on the HTTP/S protocol only, for any active port on the target server. One useful technique that is often over-looked by WebInspect users is the ability to daisy-chain the tool through various HTTP intercept proxies. This can be the included Web Proxy ("SPI Proxy") tool, Paros Proxy, BURP Suite, Charles proxy, et al. The value in doing this activity is usually to monitor the scanner's traffic in real-time, or to trouble-shoot when the site is behaving strangely and you need to find out why in order to adjust the scan settings.

The most basic process for configuring this daisy-chain is detailed below. These instructions are based on using the included HP Web Proxy, and the exact setting details can be found within its Help guide. This should also be adequate detail to guide the configuration of your alternative proxy tool of choice.

1. Open the Web Proxy tool (from the Start Menu or WebInspect) > Edit menu > Options.

2. Web Proxy defaults to listening on 127.0.0.1:8080. Verify that here, or change it, and Save. If you need to use a network proxy to reach the target, configure that on the Proxy Servers tab, and then Save.

3. Start the proxy service using the "Play" icon in the toolbar. Minimize or resize the Web Proxy tool as desired.

4. Open WebInspect > Edit menu > Default Scan Settings > Proxy panel.

5. Configure the scanner's proxy to match the port that is listening, e.g. 127.0.0.1:8080.

6. Save the scan settings and run your scan. Verify the traffic is being captured within Web Proxy window. Web Proxy offers a Scroll Lock button to help with its scrolling display.


Representation for our example:

WebInspect (localhost) > HP Web Proxy (127.0.0.1:8080) > network proxy (192.168.1.200:80) > target server (192.168.1.106:443)


The one trouble with daisy-chaining WebInspect is that now the intercept proxy is the weakest link in your assessment. If the scan will take 8 hours and will parse many megabytes of traffic, it is probable your proxy engine will fail and halt the WebInspect scan prematurely (default and configurable setting). No worries, you can always restart your proxy tool and Resume the scan, but that situation can be a pain. The good news is that if you only wish to monitor a small part of the scan, that is also possible. To do this, Pause the assessment > click on the Edit menu > Current Scan Settings (not the Default Scan Settings) > Proxy panel > alter the proxy setting to NO longer use your intercept proxy > Save settings > Resume the scan > verify that now no traffic is being captured in your proxy window.

This sort of daisy-chaining can go all day, so long as you use different ports for all of the services. For example, we could configure the Proxy Server options within Web Proxy to go to another "upstream" proxy tool (local or remote), such as Paros Proxy or a second instance of Web Proxy. Just remember to keep their ports separated if running all of them on the local machine, such as using 127.0.0.1:8080 for Web Proxy #1 and 127.0.0.1:8888 for Web Proxy #2. Why would you want to do this multiple proxy hopping? There are various trouble-shooting reasons which I will not detail now, but the capability is there if you need it.