Security Product Testing

Endpoint Protection Group Test Started

2009-06-25T16:56:00.001-07:00

NSS Labs is continuing its testing of anti-malware products and has started its first group test of endpoint protection products. We are testing the ability to protect against socially engineered malware downloaded from the web. This is a continuous live test that will measure time to protect, and average protection over time. All systems are connected to the live internet and subjected to actual downloads of actual, fresh malware every 4 hours over a period of 12 days.

Both consumer and corporate products are being evaluated. Stay tuned for more information or contact me with any questions (rmoy).

Two acquisitions in two weeks!

2009-05-19T18:40:00.000-07:00

Within the last 2 weeks, two young companies that NSS Labs did independent certifications on were acquired. ThirdBrigade, which makes Host Intrusion Prevention Software (HIPS) was acquired by TrendMicro, one of the major antimalware vendors. This product filled a server-side gap in their product line.
Solidcore Systems, which makes memory firewall/application white listing products, was acquired by McAfee. The #2 antimalware vendor cum security vendor has added whitelisting to its billion dollar portfolio of antimalware, vulnerability and intrusion prevention products. In Q3 of 2008, NSS Labs had evaluated and certified the S3 Control Embedded product as NSS Approved for Host Malware Protection.
In a down economy, strong vendors go shopping for technologies to round out their product lines so they're in positions of strength when the buyers recover. Note, even with all the cost cutting and layoffs, there's always money left for strategic purposes. And if you're a CEO who is going to make a purchase in this economy, there's not much room for forgiveness. So, you can bet they did their homework on all sides: technology, sales execution, management, margins, balance sheet, etc. I'm pleased NSS Labs was able to help these young companies grow their businesses and wish them well in the next stages of their evolution.

NSS Awards First Gold in 5 Years

2009-05-15T14:27:00.000-07:00

Yes, it's true. After a long 5 years of waiting for the next great product, at RSA Conference 2009 this year, we bestowed the prestigious NSS Labs Gold Award to IBM/ISS for it's Proventia Network IPS GX6116. IBM's was the first IPS to pass our new requirements for Gold, including the monthly recurring Security Update Monitor (SUM) program testing.

The GX6116 scored an average of 98.6% over the 3 consecutive months of testing. This new recurring testing program ensures that vendors are keeping up with current threat protection levels as advertised. Each month our engineers add new attacks to the test set according to our modified CVSS ranking of enterprise-relevant vulnerabilities. Unlike other tests, the vendors do not know which exploits will be used in this blind test. So 98.6% is pretty impressive. Most other products don't do nearly as well.

Also to be commended is the 8Gbps of real-world throughput achieved by the GX. Certainly, the IBM team worked hard and should be proud of their accomplishments on this rigorous test program. Here is Brian Truskowski, General Manager of IBM/ISS, accepting the NSS Gold Award; and his team: Dan Holden, John Pirc, Eric York, Greg Adams.

IBM isn't the only participant in the program. You can look forward to monthly testing from McAfee as well (coming soon).

Live Testing, web malware and assumptions...

2009-03-31T16:54:00.000-07:00

NSS labs just uploaded the video archive of the Live Testing Webinar we did on 3/31. This was a webinar with live Q&A as a follow up to the initial browser security test report we performed on 6 different web browsers' ability to block socially engineered malware. As we roll out this new test methodology we wanted to give readers a deeper, interactive look into the testing process. There were a few questions from readers about how we did it, why it's more relevant than static or 'in-lab' dynamic testing, and how to interpret the different measurements, etc.
Interestingly we are hearing from two different camps. A few bloggers/journalists are finding their assumptions challenged about their favorite programs; "how can that be?" Meanwhile, 'hard core' security researchers are telling us they are glad to see more comprehensive empirical validation of some of their own data points. Regardless of whether your assumptions were validated or challenged, the data can now drive the conversation - and future research.

CBS News covers Socially Engineered Malware

2009-03-29T21:15:00.000-07:00

The lead story tonight on CBS News' 60 minutes show was about socially engineered malware pushed by cyber gangs. One can see a good example of how users are tricked into clicking on links sent to them from supposed friends via social networking sites. Symantec's Steve Trilling also explained the workings of the Confiker worm and a keylogger trojan to the CBS anchor, Leslie Stahl. Very timely given the upcoming April 1 'trigger date' for Confiker. NSS Labs of course recently published a report on socially engineered malware testing we performed in early March.

web browser security study - socially engineered malware

2009-03-19T14:36:00.000-07:00

NSS Labs just released a study we did on 6 leading web browsers' ability to stop socially engineered malware attacks. We tested Safari, Chrome, IE7, IE8, Firefox and Opera. This is extremely relevant today since the majority of malware is currently being delivered via the web. Trend Micro research puts it globally at 53%, dwarfing email at just 12%. Oh how times have changed.

Read the full report here: http://nsslabs.com/anti-malware/browser-security

Also notable, this was the industry’s first live test of fresh malware sites. We pulled thousands of URLs off the web in real-time and fed them into 6 different browsers (84 unique machines) every 2 hours. A lot of work went into building this test harness and you'll certainly be hearing more about it shortly. Also keep in mind, while the highest score was Microsoft at 69%, this is nothing to sneeze at. All of the sites were extremely fresh, and the time between detection on the web and testing in the harness was between 30 minutes and 2 hours. Compare this to a VB100, ICSA, West Coast or other wild-list type test where the malware is generally 2+ months old. Our new Live Testing model yields a much more real-world assessment of anti-malware detection rates.

As far as the results, we were pleasantly surprised at just how well IE8 did. Browsers, and IE8 in particular, are becoming a viable extra layer of security on top of anti-malware/endpoint protection.

Note: NSS Labs developed the test methodology and infrastructure independently. Microsoft provided funding.

First 10Gbps IPS certification: McAfee M-8000 receives NSS Labs Approved

2009-01-23T12:31:00.000-08:00

10Gbps:
NSS Labs just released the first 10Gbps IPS certification as part of our 10Gbps IPS group test. A number of vendors are offering 10Gbps appliances: Juniper, McAfee, Enterasys, Force10, Sourcefire. McAfee's M-8000 was the first to pass our extensive testing and receive certification. In addition to meeting the rigorous performance requirements, the product scored exceptionally well on the security effectiveness testing. Read the full report here.

Still other vendors are taking the solution approach by including a load balancer and multiple IPS devices. It should be noted, these could use any reasonable switching approach to stack/VLAN multiple physical IPS devices into one logical unit. Think of products from the likes of: IBM, Cisco, Crossbeam (Chassis/Blade), Sourcefire, TippingPoint, TopLayer, etc. Depending on what a company already has installed, and what their growth/infrstructure plans look like, this model could also work well. It will come down to a TCO and effectiveness decision.

It should be noted that this was an award that was a long time in the making since we announced the testing in the summer of 2008; and many vendors had announced products well before that. Indeed there are many reasons why it takes so long. #1 - It's hard to get right. It is not necessarily easy for a vendor that has a 'successful' 1Gbps IPS to deliver the same quality product that truly meets 10Gbps requirements. We just held a technical webinar on the topic of 10Gbps IPS. We covered the challenges that vendors face when making a 10 Gbps IPS, as well as those faced by buyers who are evaluating these products. The webinar is recorded and available here. I was pleasantly surprised to receive several comments that this was the "best webinar ever," and very informative. If you don't have time to listen to the webinar, you can probably at least peruse the slides.

As we've seen in our testing, there are plenty of gotchas to look out for. And for this large and complex of a purchase, most of the potential buyers do NOT have the capabilities to adequately evaluate and test such a product. In such cases it should really behoove the vendors who have done a good job to have their products validated by a competent 3rd party. So be sure to ask your vendor what kind of testing and certification the product has gone through. (OK, somewhat of a trick question: I must confess I don't know of any other lab capable of doing the level of testing that we do, either in terms of throughput or security ;-)

/rick

The value of "reviews" just went down another notch

2009-01-20T22:40:00.000-08:00

Belkin is today's unfortunate poster child of dishonest marketing, the euphemistic "putting lipstick on a pig".

When I began my career in IT, a while ago, I relied on user reviews to provide me with some guidance. Which products were better than others, more reliable, faster, etc. The world of user-based reviews has slid a long way. Apparently a sales rep at Belkin had been hiring people on the internet to flag negative reviews of his products as "unhelpful" and post positive ones. There are plenty of other journalists and bloggers lambasting the guy, and the company president for denying and then brushing over the transgression. Amazingly, the employee still has his job. PC World covers the pandemic further here.

Folks: This is why a trusted independent 3rd party is so important when it comes to getting good advice about products. The financial motivations for individuals with a sales quota and a boss to please, or companies with investors to show returns for can be tempted to cross the line. "users" can be anyone, write anything, and have almost absolute anonymity, and no accountability. Reviews can be written in such a way that they are generic enough to apply to any product, allowing them to spam such services that host reviews. This "review SPAM" (can we coin RSPAM now?) can appear on any magazine site, or portal, regardless of how trusted the mother brand may be.

To reach back to a 1990's cartoon that has new meaning here, on the internet, you just don't know which reviews are dogs.

Webinar: 10Gbps Intrusion Prevention

2009-01-07T18:34:00.000-08:00

Are 10Gbps network IPS products mature enough for deployment? Depends... Join our upcoming webcast to find out why 10Gbps IPS is more than 10 times more difficult to get right than 1Gbps IPS. NSS Labs' Vik Phatak will also walk through a checklist of criteria to look for when evaluating products. We'll also give a behind the scenes look at how we implement our industry standard IPS test methodology using products like BreakingPoint.

You can also look forward to some real experiences culled from our 10Gbps IPS group test. The first results forthcoming end of January 09 and the full report end of Q1.

Which endpoint protection products stop IE Exploits?

2008-12-22T04:45:00.000-08:00

During the week of Dec 15-18, NSS Labs conducted a series of tests of popular anti-malware and endpoint protection products to evaluate their ability to protect clients from exploits targeting the IE vulnerability. The results are somewhat surprising, showing a broad lack of protection from current enterprise products. Admins are advised to read this and address any gaps ASAP.

Tested antivirus/anti-malware/endpoint protection products include:

AVG Internet Security Network Edition v8.0
Kaspersky Total Space Security v6.0
McAfee Total Protection for Endpoint
Sophos Endpoint Security and Control v8.0
Symantec Endpoint Protection 11.0.2 MR2
Trend Micro Officescan 8.0 SP1 R3

Read the report here.

Exploits vs Drive-by Downloads

2008-12-22T00:11:00.000-08:00

What's a "drive-by download" anyways? Recent discussions and the flurry of media articles about the recent Microsoft Internet Explorer vulnerability have given rise to some discussion. So, we at NSS Labs decided to provide this clarification of exploits vs drive-by downloads in response to some research and discussions we've had with a number of end-users and vendors. Our recent research into the Internet Explorer exploits revealed that some vendors and enterprises were not 'framing' the problem properly.

The "drive-by download" is the result of a successful exploit. It is worth noting that the exploit could have executed any arbitrary code, including returning a shell prompt, deleting or encrypting files, etc. But, more likely than not these days, the perpetrator prefers to go unnoticed so they can continue to leverage the newest memeber of their botnet in their quest for world domination. So, more frequently we see keyloggers, trojans, and other 'quiet' culprits. Come to think of it, drive-bys are usually pretty noisy with all the shooting and screeching of tires and such.

So, when vendors and end-users talk about the "download" it can unduly shift the focus towards the result and away from the cause. There are very few exploits compared to hundreds of thousands of pieces of malware. And the exploits are easier to detect - if you are looking in the right place... Network IPS and Host IPS (which can be part of an endpoint protection product) are two great solutions.

Exploits vs Drive-by Downloads.

Microsoft IE7 zero day exploit - patch released

2008-12-17T16:39:00.000-08:00

Today, just 7 days after the discovery of a critical zero-day exploit in Microsoft's popular Internet Explorer (see Microsoft Security Advisory 961051), Microsoft has released its analysis and a public patch via various Windows Update services.

We at NSS Labs has been following this closely, as live exploits have been circulating and growing rapidly, reaching more than 10,000 infected sites (TrendMicro). There are different implementations, including java script and ActiveX that exploit the XML parser in IE versions 5.01 through IE8 beta 2. See the official description and analysis from Microsoft MS08-078 for a complete list of affected versions and systems. And on the more interesting side, HD Moore at BreakingPoint Systems describes his analysis.

IBM’s Proventia Server for Windows v2 passes NSS Labs PCI Suitability Testing

2008-12-15T01:11:00.000-08:00

IBM’s Proventia Server for Windows v2 has successfully passed NSS Labs’ PCI Suitability testing for Host Intrusion Prevention Systems (HIPS). The security effectiveness of Proventia Server for Windows 2.0 was excellent. NSS Labs tested the product on numerous Windows platforms, and a wide range of applications. Proventia Server for Windows 2.0 detected and blocked a total of 64 exploits (98.5%) – all of which were Attacker Initiated. Support for PCI DSS requirements was excellent. Overall, out of 58 tested requirements, the product supports 57 (98%).

Read the complete report on IBM's Proventia Server

"Strategic" solutions vs. "pure play"

2008-11-19T17:51:00.000-08:00

Vik Phatak of NSS Labs discussed the impact of running IPS within a router in this Network World article about integrated security.

Gartner lists NSS Labs certification as criteria for Magic Quadrant

2008-11-17T23:45:00.000-08:00

In case anyone is wondering what the value of an NSS certification is, Gartner has recently recognized the value of NSS Labs certifications by adding them to the short list of criteria for products to achieve ranking in the coveted Gartner Magic Quadrant for Network IPS. NSS Labs pioneered the Network Intrusion Prevention Systems (IPS) standards and test methodologies as early as 2002, and these are globally recognized as the de facto gold standard for the industry. 3rd party testing such as NSS Labs group test certification is an important measure of product quality, which carries the highest weighting of all the evaluation criteria.

The fact that NSS was listed before Common Criteria was probably not accidental. The difference between the two evaluations is significant; NSS evaluates real-world security effectiveness and performance, whereas CC primarily evaluates the processes used to create a product.

Note:NSS Labs has completed a number of network IPS product evaluations this year on products from IBM, Juniper and others and are currently performing the industry's only 10 Gbps IPS group test.

We hear time and again from information security managers and CISOs that our reports are helping them make informed decisions that they couldn't make with less rigorous evaluations. Such acknowledgement makes what we do all that more rewarding. On behalf of all the staff and engineers at NSS Labs, I'd like to thank the gentlemen at Gartner for acknowledging the efforts of our product analysts.

P.S. We don't plan to stop at IPS...

Test in the "ether"

2008-11-02T21:15:00.000-08:00

We at NSS Labs work pretty hard testing network, host and other information security products. Gruelling but rewarding work. Sometimes we get to have a little fun as well, like this recent "Air-Test."

RSA Conference: Short-term impact of the financial crisis

2008-10-28T20:18:00.001-07:00

Here at the RSA Security Conference 2008 in London's ExCel Centre. In a recent interview with netevents I was asked -
Q: "What's the long-term security outlook?"
A: Long-term it’s good for several reasons.
1. Vendors are constantly developing new and improved products.
2. Users are getting more awareness and practical security training.
3. Companies derive competitive advantages by connecting with suppliers, customers and partners. It's increasingly understood by business managers that 'networking stuff' is needed to make money. And thanks to compliance mandates like PCI DSS, security is getting more attention and funding. Or at least it was.

Short-term there’s an increasing danger secondary ripple effects of the financial crisis. IT Security organizations, and other cost centers, will likely be squeezed to invest less time, resources and finances on solving security problems. This would be a dangerous win for the bad guys, who could have weaker, poorer funded defenses to contend with.

Contrast this with the time when governments on both sides of the axis had a clear focus and funding for cryptographic technologies as a lever in the information warfare of WWII.

Why doesn't NSS Labs have a report on Product X?

2008-10-16T16:14:00.000-07:00

Just because you don't see a product evaluation report on our website, it does not mean we have not evaluated the product. There are several possible scenarios:

NSS Labs is in process of testing the product. However, due to NDA and confidentiality reasons we cannot disclose whether or not we are testing a given product until the vendor decides to make it public.
The product vendor is waiting to release a new major revision before having it (re-)certified.
The product was evaluated by NSS Labs, but issues were found that the vendor opted to fix before completing the public certification.
The product simply has not yet been evaluated. NSS Labs operates meaningful and rigorous product testing. Not every vendor wishes to subject their product to this process.

NSS Labs makes every effort to involve product vendors in our tests. However, for various reasons, we cannot always secure their participation. Since you as a reader may not know which of the above cases is true, we recommend you inquire with the product vendor's PR or product management team.

How long is a product certification valid?

2008-10-10T00:24:00.000-07:00

Recently we have been asked about some of our older product certification reports, whether or not they were still valid; what's changed, etc; some all the way back to 2001. So just how long is a product certification valid?

From an IT Security buyer's perspective, the question is really: how long after the certification does the product still offer similar effectiveness, performance and usability characteristics? How well do they still meet the essential criteria?

Unlike static applications, security products with updates (signatures, heuristics, code, patches) change frequently in order to remain effective. (IPS products generally release new signatures on a weekly or daily basis. Antivirus products are becoming increasingly dynamic: last year Kaspersky was pushing hourly updates, and recently McAfee and Symantec have boasted 'real-time' updates.) Thus, a product could increase or decrease effectiveness significantly even 6 months out.
Performance can change anytime the code is changed. Yes, even a 'little' maintenance patch can have pronounced effects on throughput, state tables, latency, etc. To be fair, the converse is true: a vendor could release a patch that improves performance. Oh, and the more signatures that are turned on by default generally consume more resources and thus negatively affect performance.
Unfortunately, management capabilities don't change often enough. So if an interface is 'so-so', you can probably count on having to live with it for a while. Intuitive, easy-to-use interfaces is one of the underserved areas of security products.

These are all things that buyers should check on, whether it is in an NSS Labs report, or some other evaluation. The short answer (which I saved for last) is that a certification can be leveraged by a vendor for one major release cycle. These are generally 18 months long. Any new major release, and buyers should really ask for an updated report. Beware of certifications that are 2, 3, or even 4 or more years old.

Here's a little-known trick! Carefully scrutinize products that have not changed the major version number in a loooong time. Some vendors keep the same major version and modify minor numbers only for years on end in order to circumvent recertification requirements of painful things like common criteria.

NSS Labs does not withdraw certifications after an arbitrary period of time. Perhaps we should; some other labs do, and we could likely make more money to be blunt. Instead, we rely on vendor willingness to 'step up and show their mettle.'

Greasing the skids of commerce

2008-10-08T11:41:00.000-07:00

"Commerce requires a meeting of the minds between buyer and seller, and it's just not happening. The sellers can't explain what they're selling to the buyers, and the buyers don't buy because they don't understand what the sellers are selling. There's a mismatch between the two; they're so far apart that they're barely speaking the same language." Bruce Schneier on the security industry.

Having been on both sides of the vendor-IT buyer fence, I can definitely relate to both parties frustration. In this vein, some have referred to NSS Labs reports as 'next generation sales collateral', bridging the gap between brochureware and a proof of concept test (and who has time, expertise and resources for all that anyways).

North American PCI Community Meeting

2008-10-06T13:45:00.000-07:00

We just got back from the North American PCI community meeting. The turnout was about double compared to the 2007 meeting, with all the major QSAs and many name brand retailers and banks in attendance. and the SSC has clearly achieved quite a bit in the last year. Changes to the new PCI DSS version 1.2 were discussed, the first in-person Special Interest Group (SIG) meetings took place, and there were even about 40 vendor exhibits. Branden Williams, Director of the PCI Practice at Verisign, and I sat down and talked about some of the trends and changes in DSS 1.2 (watch the video).

The exhibits were a great opportunity to meet face to face with top technical representatives from these vendors and QSAs. And for them they got direct access to key influencers and decision-makers in the PCI community. Interesting note about the marketing banners, just about all claimed to have an easy PCI Compliance solution. Of course the practitioners know there is no magical "PCI Compliance Solution" and that it is more of a process or journey where the multiple layers of details cannot be avoided. But clearly some marketers are going for the standard easy benefit-oriented taglines, because after all, a marketer's goal is to get you to stop and listen. We heard a lot of merchants and card brands talking about the challenge of getting that next layer of information, which was a great segue into what NSS Labs does to validate vendor product functionality and specifically how it relates to PCI DSS.

Vik and I are serving as secretary for the Wireless Security SIG and I was honored to be able to address the community and provide an update of the SIGs activities. The goal of the SIGs is to make recommendations to the council, which will then review the recommendations, ask questions and render the final decisions. Without revealing too much, it is important to know that we are not taking a technology-centric approach that will make life harder for merchants. Rather, the SIG has decided to take a problem-oriented approach to the task, by focusing first on the problems we are trying to solve for specific groups of users. Very similar to the methods taught by pragmatic marketing. So, Level 3 & 4 merchants who believe they do not have wireless in their network would be one use case; Level 1 & 2s with known use of WiFi would be another. Of course there are many details, and there are sub-groups working on implementation guides and advanced technologies (like BlueTooth and Satellite). If you're a participating organization and would like to 'participate' drop me a line - rmoy AT you know where.

How important is a user interface after all?

2008-09-17T12:07:00.000-07:00

One important thing to consider when evaluating security products for any environment is manageability and usability. Having tested a vast array of products, it's probably safe to say we've seen a spectrum of good bad and ugly interfaces. But I'm not just talking about the look and feel. Far more important is the suitability to task: how well thought out are the most important and frequent tasks that a user will have to complete? Is critical information that I need to take action on represented effectively? How many clicks to get to it? Often times we get both excited and scared by large management frameworks. These can easily tend to present data in engineering terms of tables and lists without much thought to the objective. The last thing I want to see in a console is a lot of text in tables or generic plug-ins to meet some requirement to make data available. With so much R&D cost put into developing speeds, feeds and detection, are we as an industry investing appropriately in the equally important human interfaces?

Testing, Testing, 1-2-3

2008-09-12T11:52:00.000-07:00

A recent interview/article with NSS Labs' Vik Phatak on how enterprises can setup a test network to evaluate functionality, performance and interoperability of vendor products prior to purchase and deployment. Article in Processor.

About Deep Packet Inspection

2008-08-13T10:38:00.000-07:00

What is DPI? How can it be used effectively? What are the different use cases and requirements for such products?
We recently hosted a webinar in which we discuss this and the methodologies needed to properly evaluate the DPI functionality of network devices under the demanding network conditions in which they will be deployed. The webinar can be viewed here.

Got an opinion about IPS?

2008-07-24T00:01:00.000-07:00

If you're currently using an IPS, or in the market for one, we want to hear from you.

As an exercise to accompany our 10Gbps Network IPS group test, we decided to ask end-users what they like and dislike about their current IPS products, how they use them, and what they'd wish for in their next go around.

Simply take the short survey, and you could also win a $50 Amazon gift certificate.