Report Archive for

Click on a report title to go to the download page for that report.

SonicWall SonicWall NSa 2700 Next-Generation Firewall (NGFW) Strategic Analysis vs. Fortinet FortiGate 100F NGFW

Document number: 221102
Release Date: 04 Mar 2021

The term firewall has been part of the IT lexicon for over two decades. Today’s threats are radically more sophisticated than in the past and today’s next- generation firewalls have came a long way compared to the original invention - and all NGFWs are not the same. It is important to focus on the security features and performance needs that are most important for your environment. SonicWall has designed its NSa 2700 to excel in protection while maintaining an industry- leading price-performance ratio.

SonicWall commissioned Tolly to review the published specifications of its NSa 2700 and compare it to Fortinet’s FortiGate 100F. Analysis was mainly focused on evaluating key total cost of ownership (TCO) metrics of the comparable solutions.

The Tolly analysis shows that the SonicWall has a dramatically lower 3-year TCO for HA deployments compared to Fortinet while providing comparable or better security feature set and performance characteristics. The TCO is enhanced by SonicWall’s High Availability (HA) licensing policy allowing a single firewall license to be used for both the primary and backup (HA) appliance.
Security
GreatHorn, Inc. GreatHorn Cloud Email Security Platform vs. Proofpoint Essentials

Document number: 221107
Release Date: 01 Mar 2021

Threat actors are always seeking new methods to access and compromise corporate data. Email is a preferred path into organizations for many hackers as virtually every organization depends upon email and, almost universally, email is a communications path between outside and inside. As companies strive to erect barriers to hackers, the hackers strive to find new and often very sophisticated ways to overcome those barriers and access corporate data.

GreatHorn’s Cloud Email Security Platform is a purpose-built security solution focused on meeting the demands of today’s cloud- based email offerings. As organizations have shifted to cloud email platforms (Google Workspace, formerly G-Suite and Microsoft 365), and phishing attacks have evolved to include zero-day attacks and increasingly complex social engineering tactics, a multi-layered approach to more effectively respond to risk across each stage of the entire email lifecycle is required.

GreatHorn commissioned Tolly to evaluate the email security feature set and capabilities of its solution and compare that with capabilities of Proofpoint. Specifically, this comparison was made with Proofpoint Essentials.
Security
Spirion, LLC. Spirion Data Discovery Accuracy Evaluation

Document number: 221104
Release Date: 02 Feb 2021

Knowing the location of personal and sensitive personal data is essential for organizations of all sizes. You cannot secure what you cannot locate. The elements and composition of personal can vary dramatically across applications. One size certainly does not fit all. Business, government, legal, and medical environments have different and overlapping definitions of what is personal data. Thus, it is critically important to be able to customize any data discovery solution. Spirion provides for easy and flexible tuning to enable accurate definition of personal data.

Spirion commissioned Tolly to evaluate its data discovery and protection solution. The evaluation included a proof-of-concept (PoC) accuracy demonstration, installation, setup and filtering configuration.

Testing illustrated that Spirion could deliver 98.5% accuracy in a 200 document corpus with filtering configuration and tuning accomplished in under four hours.
Security
SlashNext, Inc. SlashNext Phishing Protection Accuracy vs. Leading Email Security Providers

Document number: 221108
Release Date: 22 Jan 2021

Phishing attacks are not new but the sophistication of the attacks and, thus, their effectiveness is always growing. Instead of mass attacks many bad actors use targeted, “spear phishing” attacks. SlashNext has developed next-gen AI phishing threat intelligence and protection to help combat this omnipresent threat.

SlashNext commissioned Tolly to benchmark the accuracy of its phishing protection solution and compare that with several leading vendors of email security solutions. Tests were conducted with Microsoft Defender (Safe Links) and Proofpoint Essentials Business (URL Defense). An informal test was also run against the advanced threat protection offering, including their URL rewriting feature, from another market leading provider (Vendor X). The test corpus consisted of zero-hour phishing URLs submitted to the PhishTank community site as well as those discovered by SlashNext’s Threat Labs.

SlashNext delivered significantly greater accuracy on the zero-hour PhishTank URLs than the other solutions. Significantly, the other solutions could only identify 21% to 49% of the recent and zero-hour phishing URLs selected by Tolly from the SlashNext AI Phishing Detection Database.
Security
Infoblox, Inc Infoblox BloxOne Threat Defense vs. Cisco Umbrella DNS-Layer Security Evaluation

Document number: 220141
Release Date: 06 Nov 2020

DNS is used constantly by almost every IP-based device that needs to connect to other IP-based systems (e.g., computers, IoT devices and OT devices), and its presence is essential to the internet and IP networks in general. DNS translates queries for names of resources (like web sites or IoT/OT management servers) and returns IP address information that allows the connection to be established. The whole concept of DNS was to translate IP addresses to common names we can easily remember – like www.infoblox.com or www.tolly.com (each of which requires underlying IP addresses for computers to connect).

Because DNS originated as a relatively simple translation tool, it was considered benign, with DNS traffic typically allowed to pass through security solutions without additional inspection. Unfortunately, cyberattackers have developed new techniques to exploit DNS and use it to steal sensitive data from corporate networks as well as to infiltrate malware into the network.

Infoblox commissioned Tolly to evaluate the effectiveness of the Infoblox BloxOne® Threat Defense solution in key DNS-layer threat scenarios and compare those results against the Cisco Umbrella solution. Two scenarios were created to replicate real- world situations. The first scenario was to infect a point-of-sale (PoS) system with malware, and the second scenario was to attempt the exfiltration of data over DNS (which would typically occur after a system was infected). Additionally, engineers reviewed global and locale-specific threat intelligence options offered by each vendor, as well as ecosystem integration with third-party management solutions designed to streamline security operations and incident response. The Infoblox Threat Defense solution demonstrated greater effectiveness than Cisco Umbrella, as will be detailed shortly, and provided broader threat intelligence and ecosystem integration than the Cisco Umbrella offering.
Security
GreatHorn, Inc. GreatHorn Cloud Email Security Platform vs. Microsoft Defender for Office 365

Document number: 220142
Release Date: 02 Nov 2020

Threat actors are always seeking new methods to access and compromise corporate data. Email is a preferred path into organizations for many hackers as virtually every organization depends upon email and, almost universally, email is a communications path between outside and inside. As companies strive to erect barriers to hackers, the hackers strive to find new and often very sophisticated ways to overcome those barriers and access corporate data.

GreatHorn’s Cloud Email Security Platform is a purpose-built security solution focused on meeting the demands of today’s cloud- based email offerings. As organizations have shifted to cloud email platforms (Google G-Suite and Microsoft Office 365), and phishing attacks have evolved to include zero-day attacks and increasingly complex social engineering tactics, a multi-layered approach to more effectively respond to risk across each stage of the entire email lifecycle is required.

GreatHorn commissioned Tolly to evaluate the email security feature set and capabilities of its solution and compare that with capabilities of Microsoft’s native, two-component Office 365 security solution which consists of basic Exchange Online Protection (EOP) augmented by Advanced Threat Protection (ATP).
Security
Infoblox, Inc Webinar: Infoblox BloxOne DDI: The Key to Management and Control of the Borderless Enterprise

Document number: 220145
Release Date: 21 Oct 2020

Networks continue to expand, particularly at the edge, as organizations add tens, hundreds or more locations as BYOD, mobile & IoT usage increases. As a result, traditional network topologies and management models are falling short, and resource limitations continue to loom as a major barrier to success. So, the need for optimized operations across these environments have become a top priority.

This is a unique opportunity to hear experts Kevin Tolly, Founder of the Tolly Group, and Kanaiya Vasani, EVP at Infoblox, present on how Infoblox, the industry leader in enterprise DNS, DHCP, and IP Address Management (or DDI), recently introduced BloxOne™ DDI, an innovative cloud-managed solution, that optimizes the deployment, and management of expanding environments, with centralized visibility, automation, and administrative control of core network services.

Infoblox Webcast View On-demand Landing Page.

The Tolly report can be downloaded from here.
Security
Xena Networks Safire Enterprise Firewall Testing Evaluation of Enterprise-Class Firewalls

Document number: 220132
Release Date: 23 Sep 2020

Security is an essential element of every network and the firewall is at the core of network security. Firewall architecture, platform and services configuration can impact both security and performance. Thus, to ensure delivering a superior user experience, it is necessary to benchmark firewalls to establish their performance limits. Xena Networks’ Safire Enterprise Firewall Tester has been designed to provide exactly this benchmarking capability.

Xena Networks commissioned Tolly evaluate the capabilities and ease-of-use of the Safire Enterprise Firewall Tester. This was accomplished by running a series of tests on a leading enterprise firewall deployed as a virtual appliance. Additionally, Tolly established the raw throughput limits of the Safire solution.

Tolly tests confirmed the need for benchmarking firewalls as the Safire tests showed dramatically different firewall throughput levels based on the security policy and functions performed by the firewall. Back-to-back tests between ports of the Safire confirmed 10Gbps of throughput. Finally, testers noted the simplicity with which a test could be configured and run with automatic analysis of the raw results.
Security
Infoblox, Inc The Business Case for Infoblox BloxOne DDI

Document number: 220120
Release Date: 14 Sep 2020

We've all recently witnessed how critical distributed networks have become for businesses. Many organizations have had to assess, act, and add tens, hundreds and even thousands of remote sites and locations to their networks in a matter of days. These activities were more than just important, they were essential to business continuity. Decentralization of the enterprise is the “New Normal”, and this trend is showing no signs of slowing as the number of locations, devices, and users continue to increase.

This growing importance of distributed business locations is at odds with traditional, long-standing hierarchal deployments that made the headquarters the center of the universe. In that model, branch-office and satellite sites relied on the central site for services. In many cases, if the WAN access between the remote location and the central office was unavailable, the branch office could see services limited or unavailable, including core networking services such as DNS, DHCP, and IP Address Management (or DDI). This model is no longer feasible, as the need for continued business operations at distributed locations has become a top priority in nearly every environment. Infoblox BloxOne DDI provides the industry’s first comprehensive core network services solution to address these challenges, while streamlining the deployment, administration, and control of distributed enterprises, through a centralized cloud-based interface.

Infoblox commissioned Tolly to analyze key challenges facing enterprise IT organizations, and to identify if and how BloxOne DDI could meet their expanding requirements. Tolly analysts assessed the BloxOne DDI solution in a lab deployment and reviewed technical documentation during the course of the project. Tolly found three key areas where BloxOne DDI enhances the management and control of today's distributed enterprise networks: 1) Centralized Administration, 2) Mobility & Cloud Optimization, and 3) Distributed Site Survivability. Additional benefits include simplified deployments, streamlined operations, reduced TCO, and flexible physical or virtual deployment options provided by the BloxOne Platform.
Security
Infoblox, Inc Infoblox Le business case d'Infoblox BloxOne DDI

Document number: 220120FR
Release Date: 14 Sep 2020

Nous avons tous récemment constaté à quel point les réseaux distribués sont devenus stratégiques pour les entreprises. Nombre d’entre elles ont dû évaluer, agir et ajouter à leurs réseaux des dizaines, des centaines, voire des milliers de sites et d’emplacements distants en quelques jours seulement. Ces opérations importantes se sont avérées essentielles à la continuité de l’activité. La décentralisation de l’entreprise est devenue la « nouvelle norme » et cette tendance ne montre aucun signe de ralentissement alors que le nombre d’emplacements, d’appareils et d’utilisateurs continuent d’augmenter.

Cette multiplication des emplacements distribués vient contrecarrer les anciens déploiements hiérarchiques classiques qui ont fait du siège social le centre de l’univers. Dans ce modèle, les succursales et les sites satellites s’appuyaient sur le site central pour les services. Très souvent, si l’accès WAN entre l’emplacement distant et le siège central n’était pas disponible, les services pouvaient être limités ou indisponibles pour la succursale, notamment les services réseau de base tels que DNS, DHCP et gestion des adresses IP (ou DDI). Ce modèle n’est plus d’actualité car la nécessaire continuité des opérations de l’entreprise sur des emplacements distribués est devenue une priorité absolue dans presque tous les environnements. Infoblox BloxOne DDI propose la première solution complète de services réseau de base du secteur qui permet de relever ces défis, tout en rationalisant le déploiement, l’administration et le contrôle des entreprises distribuées, grâce à une interface centralisée basée sur le cloud.

Infoblox a chargé Tolly d’analyser les principales problématiques auxquelles sont confrontées les entreprises dans le secteur des TI et d’identifier si BloxOne DDI serait en mesure de répondre à leurs besoins croissants, et de quelle manière. Les analystes de Tolly ont évalué la solution BloxOne DDI dans un déploiement en laboratoire et examiné la documentation technique au fil du projet. Tolly a trouvé trois domaines clés dans lesquels BloxOne DDI permet d’améliorer la gestion et le contrôle des réseaux d’entreprise distribués actuels : 1) administration centralisée, 2) mobilité et optimisation du cloud et 3) survivabilité des sites distribués. Cette solution offre également d’autres avantages : déploiements simplifiés, opérations rationalisées, coût total de possession réduit et options de déploiement physique ou virtuel flexibles offertes par la plateforme BloxOne.
Security
Infoblox, Inc Infoblox El caso de negocio para Infoblox BloxOne DDI

Document number: 220120ES
Release Date: 14 Sep 2020

Recientemente, todos hemos sido testigos de la importancia que han cobrado las redes distribuidas para las empresas. Muchas organizaciones han tenido que evaluar la situación, tomar medidas y añadir a sus redes decenas, cientos e incluso miles de ubicaciones y emplazamientos remotos en cuestión de días. Más allá de ser meramente importantes, estas medidas han sido esenciales para la continuidad del negocio. La descentralización de la empresa es la “nueva normalidad”, una tendencia que no muestra signos de ralentización. De esta forma, la cantidad de ubicaciones, dispositivos y usuarios continúa en aumento.

Esta creciente importancia de las ubicaciones corporativas distribuidas es opuesta a las tradicionales implementaciones jerárquicas, en las que las oficinas centrales eran el centro del universo. En el modelo tradicional, las sucursales y los emplazamientos satélite recurrían a las oficinas centrales para acceder a los servicios. En muchos casos, si el acceso WAN entre la ubicación remota y las oficinas centrales no estaba disponible, la sucursal podía ver los servicios limitados o no disponibles, incluidos los servicios de red centrales, como DNS, DHCP y la gestión de direcciones IP (o DDI). Este modelo ya no es viable, ya que la necesidad de continuidad de las operaciones de negocio en las ubicaciones distribuidas se ha convertido en una prioridad principal en prácticamente todos los entornos. Infoblox BloxOne DDI proporciona la primera solución completa de servicios de red centrales de la industria para responder a estos retos, optimizando al mismo tiempo la implementación, la administración y el control de las empresas distribuidas a través de una interfaz centralizada basada en la nube

Infoblox encargó a Tolly el análisis de los principales retos a los que se enfrentan las organizaciones de TI empresariales, y que averiguara si y cómo BloxOne DDI podría cumplir sus crecientes requisitos. Los analistas de Tolly evaluaron la solución BloxOne DDI en una implementación de laboratorio y revisaron la documentación técnica en el transcurso del proyecto. Tolly detectó tres áreas en las que BloxOne DDI mejora la gestión y el control de las redes empresariales distribuidas de hoy en día. 1) Administración centralizada, 2) Optimización de la movilidad y de la nube, y 3) Capacidad de supervivencia de los emplazamientos distribuidos. Entre las ventajas adicionales se incluyen la simplificación de las implementaciones, la optimización de las operaciones, la reducción del TCO y opciones de implementación físicas o virtuales flexibles proporcionadas por la plataforma BloxOne.
Security
Huawei Technologies, Co.Ltd Data Center Autonomous Driving Network - Huawei CloudFabric Openness

Document number: 220113
Release Date: 06 Apr 2020

In recent years, the biggest change within enterprise IT has been the development of traditional physical servers towards virtualization, cloud computing, and containers. The rapid development brings tremendous changes to compute and storage, and drives data center networks (DCNs) towards automation and intelligence. For example, SDN enables dynamic provisioning of network functions and provides elastic networks that collaborate with compute resources to quickly adapt to the development of new services in data centers. This requires DCNs to provide openness at different layers. In this way, customers can quickly customize and change networks based on service requirements to implement quick integration of IT systems and support rapid service rollout.

Tolly evaluated the openness of Huawei CloudFabric Autonomous Driving Network Solution.

Huawei CloudFabric solution consists of the following core components: iMaster NCE-Fabric (the data center autonomous driving network management and control system), and CloudEngine data center switches. Based on standard protocols, the solution implements multi-layer openness of the entire fabric and can quickly integrate and interoperate with third-party platforms and devices.
Security
The Tolly Group SD-WAN Best Practices for Security

Document number: 219501
Release Date: 13 Nov 2019

Perhaps more than in any other part of your network, your SD-WAN strategy needs to be equal measures networking and security. By definition, the SD-WAN interfaces with and runs across external, often public, networks - the source of many threats. Additionally, the site networks, whose traffic flows across your SD-WAN, are likely to be multifaceted. Each site will likely handle critical business applications, provide access to cloud services, handle common web browsing and often handle Internet traffic generated by visitors and guests. Each of these traffic flows has different security needs. And, to these elements we add the need to secure the SD-WAN infrastructure itself.

In this report, Tolly provides a concise summary of important security considerations and options for SD-WAN deployments based on Tolly’s experience and input from leading vendors and test tool companies.

The price of $395.00 is for a single, end-user license of the SD-WAN Security Best Practices report. Multi-user, company license costs provided on request. To purchase please contact sales@tolly.com.

For IT vendor licensing of the SD-WAN Security Best Practices report please contact sales@tolly.com.
Security
Awake Security Webinar - Evaluating Network Traffic Analysis Systems – Requirements and Challenges

Document number: 219148
Release Date: 12 Nov 2019

Awake Security, Inc. commissioned Tolly to evaluate the Awake Security Platform and compare it to the Darktrace Enterprise Immune System. Awake provided test scenarios that its customers have identified as relevant. Testing was performed in a live, high-tech company’s production environment and was comprised of five different scenarios that exercised different methods of data theft, exfiltration and credential theft that ran over common protocols and programs such as browsers, DNS and SMB file protocols.

View the on-demand webcast with Kevin Tolly, Founder at The Tolly Group presenting the study results along with David Pearson, Principal Threat Researcher at Awake Security by clicking on the link below which will take you to the Awake Security webcast landing page.

Awake Security Webcast View On-demand Landing Page.

The Tolly report can be downloaded from here.
(Note: There is no download for this webcast item.)
Security
Automation Anywhere, Inc. Guia do Comprador de Automação Robótica de Processos (RPA) Noções básicas dos recursos de RPA do Automation Anywhere Enterprise

Document number: 219129BR
Release Date: 30 Sep 2019

A automação robótica de processos (RPA) terá um efeito profundo na forma como as empresas são administradas. O "bot" centrado em tarefa é o bloco de construção básico. A Automation Anywhere cria "funcionários digitais" combinando bots com automação cognitiva e análise para potencializar o trabalho humano, realizando funções de negócios completas do início ao fim. A Automation Anywhere encomendou uma análise da Tolly dos elementos-chave de ofertas de RPA para ajudar a fornecer uma abordagem estruturada para compradores avaliarem várias soluções no mercado.

Com novas tecnologias inovadoras como a RPA, a decisão crítica de compra precisa ser feita sem o benefício de experiências longas. As ofertas são tão inovadoras que poucos têm anos de experiência, e é fácil confundir uma etapa tendo um mercado inexperiente como base.

Neste guia, apresentaremos a você o que vemos como os quatro pilares de qualquer oferta de RPA: IA/Cognitiva, Segurança, Escalabilidade e Análise. Nesta seção, resumiremos cada uma das áreas e forneceremos uma visão geral do que o Automation Anywhere Enterprise (AAE) oferece em cada área (resumido na tabela 1). No corpo do relatório, mostraremos mais detalhes adicionais de cada área. Forneceremos funcionalidades específicas e importantes que podem formar a base de uma decisão de compra informada. Isso pode fornecer um modelo para comparação com outras ofertas de RPA.
Security
Automation Anywhere, Inc. Guía del comprador para la RPA Comprenda las funciones de la RPA de Automation Anywhere Enterprise

Document number: 219129ES
Release Date: 30 Sep 2019

La automatización robótica de procesos (RPA) tendrá un efecto profundo en la forma en que se ejecutan las empresas. El “bot” centrado en las tareas es el bloque de creación fundamental. Automation Anywhere diseña “Trabajadores digitales” mediante la combinación de bots con la automatización cognitiva y funciones de análisis para complementar a los trabajadores humanos llevando a cabo funciones empresariales completas de principio a fin. Automation Anywhere le encargó a Tolly que analizara los elementos clave de las ofertas de la RPA para ayudar a proporcionar un enfoque estructurado a fin de que los compradores evalúen diversas soluciones del mercado.

Con tecnologías nuevas y revolucionarias, como la RPA, la decisión de compra fundamental debe tomarse sin el beneficio de una larga experiencia. Las ofertas son tan vanguardistas que pocos cuentan con años de experiencia y es fácil dar un paso en falso cuando solo hay un marketing ostentoso para guiarse.

En esta guía, le presentaremos lo que consideramos los cuatro pilares de cualquier oferta de RPA: AI/Cognitiva, Seguridad, Escalabilidad y Análisis. En esta sección, resumiremos cada una de las áreas y proporcionaremos una descripción general de lo que ofrece Automation Anywhere Enterprise (AAE) en cada área. (Resumido en la Tabla 1). En el cuerpo del documento, profundizaremos en los detalles adicionales de cada área. Le proporcionaremos capacidades específicas e importantes que pueden constituir la base de una decisión de compra informada. Esto puede proporcionar un modelo para comparar otras ofertas de RPA.
Security
Automation Anywhere, Inc. 로보틱 프로세스 자동화(RPA) 구매자 가이드 Automation Anywhere Enterprise RPA 기능이해

Document number: 219129KO
Release Date: 30 Sep 2019

로보틱 프로세스 자동화(RPA)는 비즈니스 운영 방식에 많은 영향을 미치고 있습니다. RPA의 기본 구성 요소는 태스크 중심 “봇”이며, Automation Anywhere는 봇과 인지 자 동화 및 분석 기능을 결합하여 처음부터 끝까지 완벽한 비즈니스 기능을 수행함으로써 인력을 지원하는 "디지털 워크포스"를 구축합니다. Automation Anywhere는 Tolly에 RPA 제품의 주요 구성 요소들을 분석하도록 의뢰하였습니다. 이를 통해 구매자가 시장 내 다양한 솔루션을 평가할 수 있게 도와드립니다.

RPA와 같은 신기술에서는 중요한 구매 결정을 내릴 때 오랜 기간 쌓아온 경험을 활용할 수 없습니다. 이러한 제품들은 몇 년 정도밖에 안 된 최신 기술 이기 때문에 화려한 마 케팅만 쫓다 보면 발을 잘못 들여놓기 쉽습니다.

본 안내서에서는 다음과 같은 RPA 제품의 4가지 핵심 요소인 AI/인지, 보안, 확장성 및 분석 기능에 대해 소개합니다. 본 섹션에서는 각 영역별로 요약하고, AAE(Automation Anywhere Enterprise)가 각 영역에서 제공하고 있는 기능에 대한 개요를 설명합니다. (표 1에 요약됨) 본 안내서의 본문에서는 각 영역에 대한 추가적인 세부 정보를 심층적 으로 설명할 예정입니다. 이를 통해 정보에 근거한 구매 결정의 기반을 갖출 수 있는 구 체적이고 중요한 기능에 대한 정보를 얻을 수 있습니다. 또한 다른 RPA 제품과 비교할 수 있는 템플릿도 제공합니다.
Security
Array Networks, Inc. Array Networks AVX 平台執行多租戶虛擬機測試報告

Document number: 219116ZT
Release Date: 20 Sep 2019

AVX 系列網路功能平台乘載多個 Array 及第三方的虛擬設備(VA),提供虛擬化的靈活性,以及專用硬體的保證效能。

依據不同型號,AVX 系列可於單一硬體平台上執行多達 32 個完全獨立的虛擬設備。專用 CPU、SSL、記憶體,以及 I/O 資源為每個虛擬設備提供服務。因此,AVX 系列在兩個機架單位中提供相當於 32 個專用硬體設備的效能,其中安全及網路服務可確保提供保證與明顯的效能。

Array 委託 Tolly 集團基於次世代防火牆(NGFW)執行效能及延伸擴充的基準測試。

Tolly 集團藉由將測試活動分成三個測試情境來測量效能及規模。測試#1 包含在單一 AVX9800 平台上執行 16 個小型的 NGFW 虛擬機。在單一 AVX9800 平台上,使用 8 個中型的 NGFW 虛擬機進行測試#2 的測量。在單一 AVX9800 平台上,使用 4 個大型的 NGFW 虛擬機進行測試#3 的測量。 測試結果顯示,即使 AVX 平台滿載,每個虛擬機的 throughput 都比 NGFW 設備商的產品簡介宣稱的 throughput 規格至少提高 61%;每個虛擬設備的 TPS 呈線性成長;此外,合計系統 throughput 及 TPS 非常一致。
Security
Array Networks, Inc. Array Networks AVX網路功能平台 網頁應用程式防火牆(WAF)及次世代防火牆(NGFW)SSL/TLS效能數據報告

Document number: 219113ZT
Release Date: 10 Sep 2019

網路設備虛擬化(VA)為網頁應用防火牆(WAF)及次世代防火牆(NGFW)等邊際安全設備,提供了低成本的部署方法。然而,資料的加密及解密(SSL/TLS)處理過程中,其處理效能會嚴重下降。 Array Networks 特殊的 AVX 網路功能平台內建高性能硬體 SSL/TLS晶片完美處理加密資料,讓第三方虛擬網路安全設備完全發揮應有的效能。

Array Networks 委託 Tolly 以市場領先的 WAF 及 NGFW 作為基準,測試 WAF 及 NGFW 網路虛擬資安設備。測試會先基於沒有輔助的情況下進行,而後使用 Array Networks 的 AVX 硬體SSL/TLS 卸載功能,將 SSL/TLS 處理交由 AVX 平台來處理加密資料。測試顯示資料傳遞速度(Transactions per second - TPS)及總處理流量(Throughput)都有極大幅度的增加。
Security
Awake Security Awake Security Platform Evaluating the Evolution of Network Traffic Analysis: Awake Security Platform vs. Darktrace Enterprise Immune System

Document number: 219139
Release Date: 27 Aug 2019

Threats to enterprise network security have evolved in complexity and sophistication. Protecting your network by catching virus fingerprints is a thing of the past. Today, threats are multi-faceted and often try to camouflage themselves within normal traffic flows. Network detection and response (NDR) solutions focus on ferreting out such attacks.

Awake Security, Inc. commissioned Tolly to evaluate the Awake Security Platform and compare it to the Darktrace Enterprise Immune System. Awake provided test scenarios that its customers have identified as relevant. Testing was performed in a live, high-tech company’s production environment and was comprised of five different scenarios that exercised different methods of data theft, exfiltration and credential theft that ran over common protocols and programs such as browsers, DNS and SMB file protocols.

Tests showed that the Awake Security solution detected significantly more threats across the full suite of malicious behavior detection tests.
Security
Huawei Technologies, Co.Ltd Huawei Campus Network Solution Interoperability with Cisco ISE for Network Access Control

Document number: 219114
Release Date: 11 Jul 2019

Campus network’s access become more complex with the exponential growth of different kinds of mobile devices. When enterprises deploy new network equipment, one key requirement will be new equipment’s interoperability with existing Network Access Control (NAC) solutions for user endpoints’ secure access.

Huawei commissioned Tolly to verify its campus network solution’s interoperability with the Cisco Identity Services Engine (ISE), a third-party NAC solution. The Huawei campus network solution includes the WLAN solution which uses the Huawei WLAN access controller as the NAC policy enforcement point for Wi-Fi users, and the Ethernet switch solution which uses the Huawei S series campus switch as the NAC policy enforcement point for wired users. Tests covered user/endpoint authentication (802.1X, MAC and portal), device management authentication (TACACS), authorization, CoA, endpoint profiling, BYOD, posture assessment, and guest lifecycle management. With the Huawei access controller and the Huawei S series campus switches as NAC policy enforcement points, the Huawei campus network solution passed all test cases to interoperate with the Cisco ISE.
Security
Array Networks, Inc. Array AVXプラットフォームのマルチテナント・ベンチマーク 次世代ファイアウォール(NGFW)使用時の性能・拡張ベンチマーク

Document number: 219116JP
Release Date: 02 Jul 2019

AVXシリーズ ネットワークファンクションプラットフォームは、アレイ純正や 3rd Party製の仮想アプライアンス(VA)を複数ホストし、専用ハードウェアア プライアンス同様の性能確約と仮想化のアジリティを実現します。

AVXシリーズは、最大32基の完全独立したVAを単一ハードウェア・プラット フォーム上で実行します。各仮想アプライアンスに専用CPU、SSL、メモリ、I/O リソースが割り当てられるため、AVXシリーズでは2RU(ラックユニット)で最大 32台の専用ハードウェアアプライアンスを使用する場合と同等の性能を実現で き、セキュリティやネットワークのサービスは確かな性能を達成することができ ます。

アレイ・ネットワークスは、ひとつの使用例として、代表的ベンダの次世代ファ イアウォール(NGFW)仮想アプライアンスを用いた際のAVXプラットフォーム のベンチマーキングをTolly Groupに依頼しました。

Tolly Groupではこれを受け、3つのテストシナリオを用いて同製品の性能・拡張 性を測定しました。テスト#1では、16基の”Small”サイズNGFW仮想アプライア ンス・インスタンスを一台のAVX9800プラットフォーム上で実行しました。

テスト#2では、8基の”Medium”サイズNGFW仮想アプライアン ス・インスタンスを一台のAVX9800プラットフォーム上で実行 して測定を行ないました。テスト#3では、4基の”Large”サイズ NGFW仮想アプライアンス・インスタンスを一台のAVX9800プ ラットフォーム上で実行して測定を行ないました。
Security
Array Networks, Inc. アレイ・ネットワークスAVXネットワークファンクションプラットフォームウェブアプリケーションファイアウォール(WAF)&次世代ファイアウォール(NGFW)のSSL/TLSオフロード性能

Document number: 219113JP
Release Date: 24 Jun 2019

仮想アプライアンス(以下VA)は、アプリケーションファイアウォール(以下 WAF)や次世代ファイアウォール(以下NGFW)といった「境界防御型セキュリ ティ」を導入するうえで費用対効果の高い方法です。しかし、SSL(Secure Sockets Layer/TLS(Transport Layer Security)などの暗号化処理を行う場合に は、セキュリティソリューションの処理性能は大きく低下してしまいます。そこ で、SSL/TLSのハードウェアアクセラレーション機能を有するアレイ・ネットワー クスAVXネットワークファンクションプラットフォーム上にデプロイすることで、 3rd Party製仮想セキュリティアプライアンスの性能は大幅に向上します。

アレイ・ネットワークスは、WAFやNGFWのそれぞれの市場におけるトップベン ダのVAを用いた性能ベンチマーキングをTollyに依頼しました。テストは、まず SSL/TLSオフロード機能なしの通常状態で実施され、その後、アレイ・ネットワー クスAVXハードウェアのSSL/TLSオフロード機能を作動させた状態で実施しまし た。その結果、SSL/TLS処理をアレイのAVXプラットフォームにオフロードするこ とでトランザクション処理とスループットが大幅に改善されることが示されまし た。
Security
Array Networks, Inc. Array AVX Platform Multi-Tenancy Benchmarks Performance and Scaling Benchmarks Using Next-Gen Firewall (NGFW)

Document number: 219116
Release Date: 13 May 2019

AVX Series network functions platforms host multiple Array and 3rd-party virtual appliances (VAs), providing the agility of virtualization with the guaranteed performance of dedicated appliances.

Depending on the model, the AVX Series runs up to 32 fully independent VAs in a single hardware platform. Dedicated CPU, SSL, memory, and I/O resources serve each virtual appliance. As a result, the AVX Series offers the equivalent of up to 32 dedicated physical appliances in two rack units, in which security and networking services can be assured of guaranteed and demonstrable performance.

Array Networks commissioned The Tolly Group to benchmark performance and scaling characteristics of the AVX platform using next-gen firewall (NGFW) virtual appliances from a market-leading vendor as an example application.

Tolly Group measured performance and scale by dividing the testing activity into three test scenarios. Test #1 consisted of running 16 small-sized NGFW virtual appliance instances on a single AVX9800 platform. Test #2 measurements were made with 8 medium-sized NGFW virtual appliance instances on a single AVX9800 platform. Test #3 measurements were taken with 4 large-sized NGFW virtual appliance instances on a single AVX 9800 platform.
Security
Symantec Corporation Symantec Cloud-Delivered Integrated Web Security Service Solution Key Feature Comparison Versus A Competitive Cloud Security Solution

Document number: 219105
Release Date: 01 May 2019

Providing enterprise-class security is a multi-faceted challenge. In addition to detecting malicious websites, security solutions need to provide advanced capabilities against web delivered malicious content and phishing attacks, integrate with endpoints, use the latest in secure session technology and more. In particular, integration with cloud-based environments is now essential as they become a standard part of business environments.

Symantec Corp. commissioned Tolly to evaluate the Symantec Cloud-Delivered Web Security Service along with its CloudSOC CASB solution and compare some key features to a leading competing cloud-based security solution. Feature areas include web security interaction with cloud-based applications, advanced phishing protection features, SSL session security and integration with endpoint solutions.

The test showed that the Symantec Cloud-Delivered Web Security Service (WSS) provided significant benefits when integrating with its CloudSOC CASB solution, providing advanced technology to protect against zero-day malware, phishing attacks and integration with endpoint security solutions.

Symantec CloudSOC can identify and control over 30,000 applications, providing greater visibility into the different shadow IT applications in the corporate environment. The competing solution only supports ~250 applications with limited controls, which also limits shadow IT visibility.
Security
Array Networks, Inc. Array Networks AVX Network Functions Platform Web Application Firewall (WAF) & Next-Gen Firewall (NGFW) SSL/TLS Offload Performance

Document number: 219113
Release Date: 09 Apr 2019

Virtual appliances (VAs) provide a cost-effective method for deploying perimeter security elements such as web application firewalls (WAFs) and next-generation firewalls (NGFWs). However, security solutions can suffer from severe performance degradation when cryptographic tasks like secure-sockets layer (SSL)/Transport Layer Security (TLS) processing are performed. By deploying on an Array Networks AVX Network Functions Platform – which provides hardware SSL/TLS acceleration – performance for 3rd-party security virtual appliances can be improved dramatically.

Array Networks commissioned Tolly to benchmark the performance of WAF and NGFW security VAs, from a market leading WAF vendor and from a market leading NGFW vendor. Testing was done first running unassisted and then benefiting from the Array Networks AVX hardware SSL/TLS offload capability. Tests showed dramatic improvement in both transaction processing and overall throughput when SSL/TLS processing was offloaded to the Array AVX platform.
Security
BOLO Network, Inc. BOLO Network Cloud Storage Gateway Random Byte Storage (RBS) Data Security Effectiveness Evaluation

Document number: 219102
Release Date: 08 Jan 2019

Data security is a concern that never goes away. Over the years, encryption technology has gotten more sophisticated but hackers always seem to keep pace. BOLO Network’s unique random byte storage (RBS) technology defeats hackers by eliminating encryption as the protection mechanism and replacing that with its unique technology.

BOLO Network commissioned Tolly to demonstrate the key characteristics and effectiveness of the BOLO Network technology as implemented in its Cloud Storage Gateway offering. Tests were conducted on ten different file types that included text and media files all chosen by Tolly.

Tests illustrated that the RBS solution worked with every file type tested, shuffled test files randomly without adding any bytes to a file and successfully de-shuffled all files to re-create the original file exactly.
Security
Symantec Corporation Webinar: How to Validate Email Security Services For Effectiveness

Document number: 218144
Release Date: 10 Dec 2018

Tolly Group Founder Kevin Tolly hosts a seventeen-minute webinar discussing how to validate email security services for effectiveness.

Symantec commissioned Tolly to evaluate the effectiveness of the Symantec Email Security.cloud product versus two competitors.

The research provided insights into the myriad challenges facing customers when evaluating alternative solutions. These range from feed selection, through sample crafting to avoiding blacklists - and more.

Download is in mp4 format.
Security
Symantec Corporation Optimizing and Protecting Enterprise WAN & Internet with Symantec: Leveraging Symantec PacketShaper to Manage Enterprise/Cloud Traffic

Document number: 218134
Release Date: 15 Oct 2018

Today’s application and traffic environment is already complex. And, with the move to cloud and hybrid services, is becoming even more complex. Services like Microsoft Office 365, Amazon Web Services (AWS), Salesforce and countless others are becoming part of the critical infrastructure for companies large and small. Think about it - Cloud services, voice, video and, in many cases, customers accessing countless apps over your Wi-Fi have all jumped on to your enterprise WAN and Internet access links.

Traditionally, WANs are effectively invisible making problematic users and apps invisible. You can’t fix a problem that you can’t identify, and you can’t identify a problem that you can’t see.

The solution is to bring visibility to the apps that traverse your WAN and Internet links coupled with the ability to control how your bandwidth is used at the application or flow level. Visibility and control are the key elements. As this paper will detail, Symantec PacketShaper brings you all that and more.

PacketShaper led the industry into application-level visibility and the control of the enterprise WAN and Internet links. Its ability to manage network bandwidth by prioritizing application flows is long proven. As new applications appear on your network - and you know they will - PacketShaper can provide ongoing intelligence about the traffic flowing across your network and keep you in control now and in the future.
Security
Cisco Systems, Inc Cisco RV340 Dual WAN Gigabit VPN Router L3, L4 & VPN Performance Evaluation

Document number: 217127
Release Date: 29 Aug 2017

Small businesses can have big needs when it comes to network throughput. Cisco Systems realizes that and designed its RV340 Dual WAN Gigabit VPN Router to deliver a range of connectivity options and to deliver Gigabit throughput across NAT and IPsec connections.

Cisco Systems commissioned Tolly to evaluate its Cisco RV340 Dual WAN Gigabit VPN Router in both single device and paired router configurations. Tests evaluated the Layer 3 NAT throughput of a single router and the throughput across an IPsec VPN tunnel between a pair of routers. Additionally, tests benchmarked the Layer 4 throughput of the device.

The Cisco RV340 delivered up to 99% of theoretical throughput across Gigabit Ethernet in a NAT configuration, up to 95% across an IPsec VPN tunnel and approximately 975Mbps HTTP download speed in Layer 4 testing.
Security
Symantec Corporation Symantec Secure Web Gateway Web Security Effectiveness & Features Versus A Leading NGFW Solution

Document number: 217114
Release Date: 05 May 2017

Enterprise security perimeters are subject to constant and changing threats. Web security gateways/firewalls need to be able to detect pre-existing as well as zero-day attacks. Furthermore, security solutions must be able to detect and intercept hacks that use evasion techniques to try to bypass perimeter security.

Symantec Corp. commissioned Tolly to evaluate the Symantec Secure Web Gateway solution which combines the functionality of the Symantec ProxySG with the intelligence of Symantec Content Analysis and Malware Analysis. The Symantec solution was compared to that of a leading next-generation firewall (NGFW) solution. Testing primarily focused on evaluating the effectiveness of each solution in its detection of malware and stopping hackers from evading the security perimeter. Tolly also profiled key features of each solution.

Tests showed that the Symantec solution detected significantly more threats across the full suite of malware detection tests.
Security
Symantec Corporation Symantec Cloud-Delivered Web Security Service Web Security Effectiveness Versus A Competitive Cloud Security Solution

Document number: 217111
Release Date: 15 Jan 2017

Traditional, premise-based network security is an essential part of any enterprise security strategy. New enterprise requirements, such as the need to secure mobile users and mange employees who are directly accessing cloud-based SaaS business applications, has led to innovative capabilities and flexible deployment approaches that IT and Security teams need to consider as they re-evaluate their security architecture.

Symantec Corp. commissioned Tolly to evaluate the Symantec Cloud-Delivered Web Security Service and compare its effectiveness to that of another competitive cloud-based security solution. Testing primarily focused on evaluating the effectiveness of each solution in its detection of malware. Tolly also analyzed each solution’s ability to avoid false positives, an important consideration when evaluating these products since false positives can quickly overwhelm precious security personnel resources. Finally, Tolly looked at some other important areas, such as deployment flexibility, CASB support, and security policy administration. Tests showed that the Symantec solution’s performance was superior across the full suite of malware detection tests.
Security
Sophos Ltd. Podcast: Next-Gen Endpoint Security with Sophos

Document number: 217102
Release Date: 03 Jan 2017

Tolly Group Founder Kevin Tolly hosts a fourteen-minute podcast discussing Next-Gen Endpoint Security technology with Sophos.

The podcast is part of our series “Next-Gen Endpoint Security”. In this series of podcasts, we’ll hear from leading technology vendors about what to expect with this new generation of technology known as Next-Gen Endpoint Security.

Kevin is joined by Thom Bailey, Sr. Director, Product Management and find out Sophos' perspective on the benefits of Next-Gen Endpoint, key customer needs that would benefit from Next-Gen Endpoint and Sophos' unique approach to bringing Next-Gen Endpoint solutions to market.

Download is in mp3 format.
Security
IBM Corporation Podcast: IBM Security Network Protection XGS 7100 Next-Generation Intrusion Prevention System (IPS)

Document number: 216167
Release Date: 12 Dec 2016

Tolly Group Founder Kevin Tolly hosts a twelve-minute podcast discussing Tolly's IBM Security Network Protection XGS 7100 Next-Generation Intrusion Prevention System (IPS) evaluation published in Tolly report #216106.

The Podcast provides an overview of the XGS 7100 efficacy and performance testing along with a Q&A with IBM's Jordan Carlson, WW Portfolio Marketing Manager, Security Analytics & Network Security, IBM Security.

Download is in mp3 format.
Security
IBM Corporation IBM Webcast - The Four Superpowers Your Access Management Needs

Document number: 216153
Release Date: 28 Sep 2016

IBM commissioned Tolly to evaluate IBM Access Manager for its Web protection effectiveness and performance as well as its identity federation, risk management and mobile one-time password capabilities.

Tolly found that IBM Access Manager provided effective, high-performance threat protection while conveniently providing identity federation features and flexible risk-based access options.

View the on-demand webcast with Kevin Tolly presenting the study results along with Dr. Angelika Steinacker, IBM European Leader for Identity & Access Management by clicking on the link below which will take you to the IBM webcast landing page.

IBM Webcast View On-demand Landing Page.

The Tolly report can be downloaded from here.
(Note: There is no download for this webcast item.)
Security
SiteLock, LLC SiteLock® SMART™ (Secure Malware Alert & Removal Tool) Web-based Malware Comparison Versus McAfee Complete Endpoint Protection Suite

Document number: 216143
Release Date: 19 Sep 2016

Traditional endpoint security is an essential part of any enterprise security strategy, but, today, is it enough? Hackers work 24x7 to exploit systems any way that they can. Web applications are publicly visible, and frequently contain vulnerabilities, making them a prime target.

SiteLock, LLC commissioned Tolly to evaluate the SiteLock SMART web-based malware protection solution and compare its effectiveness to the McAfee Complete Endpoint Protection Suite. Testing focused on evaluating how effective each solution was at detecting and removing web-based malware.

Tests showed that the SiteLock solution detected and cleaned 100% of the samples where the traditional endpoint solution detected under 6% of the samples and deleted less than 2% of the samples.
Security
Akamai Technologies, Inc. Akamai Kona Site Defender vs CloudFlare WAF Web Application Firewall Security Efficacy

Document number: 216116
Release Date: 01 Jul 2016

Because of the critical role played by web applications in so many organizations, securing business websites is often the most important task assigned to the security team. Akamai’s Kona Site Defender provides integrated website protection against DDos and web application attacks as well as from a wide range of online threats, including network- and application-layer DDoS, SQL injection and XSS attacks.

Akamai commissioned Tolly to evaluate the security efficacy of the Akamai Web application firewall (WAF) solution and compare that to the CloudFlare WAF solution. Both solutions are delivered via a content delivery network (CDN). Tests showed that Akamai Kona Site Defender detected and blocked more threats in more categories than the CloudFlare WAF solution.

Security
IBM Corporation IBM Security Access Manager Web Application Protection, Performance, Federation & Risk-Based Access Evaluation

Document number: 216110
Release Date: 01 Jun 2016

Web applications are often the most vulnerable part of a company's infrastructure and yet are typically given direct paths to the internet, thus leaving these vulnerabilities exposed.

IBM commissioned Tolly to evaluate IBM Access Manager for its Web protection effectiveness and performance as well as its identity federation, risk management and mobile one-time password capabilities.

Tolly found that IBM Access Manager provided effective, high-performance threat protection while conveniently providing identity federation features and flexible risk-based access options.
Security
IBM Corporation Webinar - IBM XGS 7100 Next-Generation Intrusion Prevention System (IPS) Efficacy and Performance Evaluation

Document number: 216128
Release Date: 04 May 2016

IBM commissioned Tolly to evaluate the effectiveness and performance of its IBM Security Network Protection XGS 7100 appliance. The IBM XGS 7100 is a next-generation intrusion prevention appliance, and the model tested was outfitted with 8x10GbE ports. Efficacy testing encompassed attack detection/ blocking of various threat corpora, both with and without SSL/TLS inbound/ outbound inspection enabled. Performance testing included multi-protocol throughput with SSL/TLS disabled and SSL/TLS inbound enabled, along with HTTP connections per second. The system detected 100% of publicly-disclosed exploits tested along with successfully blocking 100% of the McAfee Evader test suite.

View the on-demand webcast with Kevin Tolly presenting the study results along with Sr. Product Manager, IBM Security, Eric York by clicking on the link below which will take you to the IBM webcast landing page.

IBM Webcast View On-demand Landing Page.

The Tolly report can be downloaded from here.
(Note: There is no download for this webcast item.)
Security
Sophos Ltd. Sophos Endpoint Protection y Sophos Cloud Endpoint Protection Facilidad de uso y funcionalidad de Endpoint Security Management en comparación con Kaspersky, McAfee, Microsoft, Symantec y Trend Micro

Document number: 215102ES
Release Date: 23 Feb 2016

Pocos negarían la importancia de la seguridad de las estaciones de trabajo en el entorno de TI de cualquier empresa. Pero para que una solución de seguridad sea eficaz, debe ser fácil de instalar y mantener. Algunas soluciones pueden ser tan complejas de implementar que puede darse el caso de que sus funcionalidades se configuren de manera incorrecta o no se utilicen en absoluto. Con una solución que implique menos esfuerzo, es más probable que se usen las funciones de seguridad, y que se usen correctamente.

Sophos encargó a Tolly que evalúe sus soluciones de seguridad para estaciones de trabajo de instalación local y en la nube, Sophos Endpoint Protection y Sophos Cloud Endpoint Protection. Sophos ha diseñado estas soluciones para proporcionar una amplia funcionalidad de seguridad “lista para usar”, que permita a los usuarios responder fácilmente a situaciones de amenazas de seguridad comunes. Estas soluciones se compararon con otras ofertas destacadas de Kaspersky, McAfee (Intel Security), Symantec y Trend Micro. De las soluciones evaluadas, sólo las ofrecidas por Sophos proporcionaron una variedad de funcionalidades, incluidas prácticas recomendadas y plantillas preconfiguradas que estaban listas para usar inmediatamente después de su instalación y resultaban fáciles de administrar. Véase la Figura 1 para obtener un resumen de los resultados.
Security
Sophos Ltd. Sophos Endpoint Protection および Sophos Cloud Endpoint Protection エンドポイント セキュリティ管理製品のユーザビリティ・機能を Kaspersky および McAfee、Microsoft、 Symantec、Trend Micro と比較

Document number: 215102JA
Release Date: 23 Feb 2016

あらゆるビジネス IT 環境におけるエンドポイント セキュリティの重要 性は今や明らかですが、セキュリティ ソリューションを効果的に実施 するには展開と保守が容易でなければなりません。中には、実装方法 の複雑さゆえに設定ミスが起きやすかったり、一部の機能が全く使用 されないソリューションもあります。一般に、セキュリティ機能は、 操作が簡単であるほど正しく使用されます。

Sophos は、同社のオンプレミスとクラウドベースのエンドポイント セ キュリティ ソリューションである Sophos Endpoint Protection および Sophos Cloud Endpoint Protection の評価を Tolly に委託しました。これらの ソリューションは、「購入後すぐに使用できる」幅広いセキュリティ 機能を提供し、ユーザーが一般的なセキュリティ脅威のシナリオに簡 単に対応できるように設計されています。評価においては、これらの ソリューションを、Kaspersky および McAfee (Intel Security)、Symantec、 Trend Micro の有力製品と比較しました。テストされたソリューション のうち、インストール後すぐに使用でき、管理も簡単な事前設定済み のベスト プラクティスとテンプレートをはじめとする広範な機能性を 提供したのは Sophos 製品だけでした。結果の概要は図 1 をご覧くだ さい。
Security
IBM Corporation IBM Security Network Protection XGS 7100 Next-Generation Intrusion Prevention System (IPS) Efficacy and Performance Evaluation

Document number: 216106
Release Date: 12 Feb 2016

Network security threats continue to grow not only in number, but also in type and sophistication. Organizations need protection from zero-day attacks, advanced persistent threats, intricate evasion techniques, and more. And while security is a key objective, maintaining network performance and availability remain top priorities as well. Organizations need both proactive protection and performance from their network security solutions.

IBM commissioned Tolly to evaluate the effectiveness and performance of its IBM Security Network Protection XGS 7100 appliance. The IBM XGS 7100 is a next-generation intrusion prevention appliance, and the model tested was outfitted with 8x10GbE ports. Efficacy testing encompassed attack detection/ blocking of various threat corpora, both with and without SSL/TLS inbound/ outbound inspection enabled. Performance testing included multi-protocol throughput with SSL/TLS disabled and SSL/TLS inbound enabled, along with HTTP connections per second. The system detected 100% of publicly-disclosed exploits tested along with successfully blocking 100% of the McAfee Evader test suite.
Security
Zebra Technologies Zebra AirDefense Wireless IPS (WIPS) Security Effectiveness Evaluation vs. Cisco Meraki Air Marshal

Document number: 216104
Release Date: 15 Jan 2016

Wireless LAN (WLAN) has become a critical service network in healthcare, manufacturing, finance, T&L, retail, and hospitality. As businesses exploit the use of Wi-Fi to improve their operations and reduce friction to interact with their customers, securing the wireless network is paramount. It is not just safeguarding intellectual property and employees, but also the privacy of their customers, shoppers, and guest, while preserving compliance with PCI, HIPAA, SOX, and other industry standards.

Zebra Technologies commissioned Tolly to evaluate the Zebra AirDefense WIPS solution and compare that to Cisco Meraki Air Marshal. Tests illustrated both performance and feature benefits of the Zebra solution.
Security
Zebra Technologies Zebra AirDefense Wireless IPS (WIPS) Security Effectiveness Evaluation vs. Aruba (HP Enterprise) RFProtect

Document number: 215173
Release Date: 05 Jan 2016

Wireless LAN (WLAN) has become a critical service network in healthcare, manufacturing, finance, T&L, retail, and hospitality. As businesses exploit the use of Wi-Fi to improve their operations and reduce friction to interact with their customers, securing the wireless network is paramount. It is not just safeguarding intellectual property and employees, but also the privacy of their customers, shoppers, and guest, while preserving compliance with PCI, HIPAA, SOX, and other industry standards.

Zebra Technologies commissioned Tolly to evaluate the Zebra AirDefense WIPS solution and compare that to RFProtect from Aruba, an HP Enterprise company. Tests illustrated the feature benefits of the Zebra solution.
Security
Intel Security McAfee Endpoint Security 10 Enterprise Endpoint Security Usability vs. Kaspersky, Sophos, Symantec & Trend Micro

Document number: 215110
Release Date: 16 Dec 2015

McAfee Endpoint Security 10 has a new, redesigned architecture that offers a single, integrated platform. Unlike traditional endpoint security approaches where products are isolated from each other, McAfee Endpoint Security 10’s architecture integrates protection technologies allowing the Firewall, Threat Prevention, and Web Control modules to talk to each other in real-time allowing them to learn from each other, analyze and act upon new potential malware and advanced threats.

Intel Security commissioned Tolly to evaluate its Endpoint Security 10 (ES 10) product, implemented using ePolicy Orchestrator cloud and on-premise to compare the Intel Security solutions with similar enterprise endpoint security offerings from Kaspersky, Sophos, Symantec and Trend Micro. Scenarios involved deployment, detection, management interface and reporting, and scanning efficiency. Testing was conducted in March-April 2015.

Deployed and managed using McAfee ePolicy Orchestrator, Endpoint Security 10 provided a consistent management interface whether the cloud-based or on- premise management option was selected.
Security
Zebra Technologies Zebra AirDefense Wireless IPS (WIPS) Security Effectiveness Evaluation vs. Cisco Adaptive wIPS

Document number: 215156
Release Date: 10 Nov 2015

Wireless LAN (WLAN) within enterprises is the common network access for employees, guests and customers. It has become a critical service network in healthcare, manufacturing, finance, transportation and logistics, retail, and hospitality. WLAN is also serving as mobile marketing platforms for shopper and guest engagement. As more businesses exploit the use of Wi-Fi to improve their business and reduce friction to interact with their customers, securing the wireless network is paramount. It is not just safeguarding intellectual property and employees, but also the privacy of their customers, shoppers, and guest, while preserving compliance with PCI and HIPAA. The primary defense of a wireless network is the Wireless Intrusion Prevention System (WIPS). A WIPS solution’s primary role is to protect the company and its employees’, guest’s and customer’s wireless connections and transactions. WIPS ensure users are connected to the correct network, with the required security and are in compliance with corporate policies.

Zebra Technologies commissioned Tolly to evaluate the Zebra AirDefense WIPS solution and compare that to Cisco Adaptive wIPS. Tests illustrated both performance and feature benefits of the Zebra solution.
Security
Allot Communications Ltd. Allot Service Gateway Virtual Edition and Security VNF Carrier-Class Performance Evaluation: L4-7 Classification & URL Filtering

Document number: 215157
Release Date: 12 Oct 2015

Carrier environments need to be able to provide the highest network performance at a competitive price point. Network virtualization and virtualized services hold the key to achieving this goal. High-performance virtual network appliances built on standard, commercial off-the-shelf (COTS) hardware can deliver the “bang for the buck” that is sought after by network operators large and small.

Allot Communications commissioned Tolly to test a virtualized security service (i.e., Web URL Filtering) and the virtualized service framework that supports it - Allot Service Gateway Virtual Edition (Allot SG-VE). Together, they comprise an Allot Security virtualized network function (VNF) that is engineered to operate seamlessly in network functions virtualization (NFV) environments. That’s Allot’s vision for the future – to offer VNFs that have all the underlying capabilities built-in and pre-integrated so all a customer has to do is deploy an instance of the VNF and they are up and running.

The Allot Service Gateway Virtual Edition (referred to as “SG-VE” hereafter) demonstrated nearly 200Gbps of network throughput while running traffic classification and URL filtering functions common to carrier-grade networks.
Security
WINS WINS Sniper ONE 40G Intrusion Prevention System (IPS) Efficacy and Performance Evaluation

Document number: 215144
Release Date: 30 Sep 2015

In a large scale network, security solutions are installed to detect and protect the various network incidents. Moreover, enterprise/ISP administrators are using various security products to protect the service/internal resources by the characteristics of their service/internal resources (DNS Server, VoIP connection, Internet, etc). The security solutions should provide the stable performance without interrupting the data flow. SNIPER ONE is a security solution which stably detects and blocks the attacks in large scale network such as Mobile LTE, higher education, financial, service provider, enterprise data center, etc. The user can activate the different security policy by services like IPS, Anti-DDoS, VoIP, DNS, and so on.

WINS commissioned Tolly to evaluate the effectiveness and performance of its Sniper ONE 40G Intrusion Prevention System (IPS). The Sniper ONE is an inline IPS outfitted with four 10GbE ports. Testing encompassed attack detection/blocking in various application environments, packet level and Layer 7 (HTTP) throughput as well as an evaluation of some key features of the IPS. The system detected 98% of attacks and blocked 93% of attacks
Security
Sophos Ltd. Sophos Endpoint Protection & Sophos Cloud Endpoint Protection Endpoint Security Management Usability & Functionality vs Kaspersky, McAfee, Microsoft, Symantec & Trend Micro

Document number: 215102
Release Date: 15 Sep 2015

Few would argue about the importance of endpoint security in any business IT environment. But, for a security solution to be effective it must be easy to deploy and maintain. Some solutions can be so complex to implement that features are either easily misconfigured or not used at all. The less effort involved, the more likely it is that the security features will be used and used correctly.

Sophos commissioned Tolly to evaluate its on-premise and cloud-based endpoint security solutions, Sophos Endpoint Protection & Sophos Cloud Endpoint Protection. Sophos has designed these solutions to provide broad security functionality “out of the box” and make it easy for users to respond to common security threat scenarios. These solutions were compared to other prominent offerings from Kaspersky, McAfee (Intel Security), Symantec and Trend Micro. Of the solutions tested, only the Sophos offerings provided a range of functionality, including pre-configured best practices and templates, that were ready to use upon installation and simple to manage.
Security
Bromium Webcast with Bromium - Why Endpoint Security Fails

Document number: 215134
Release Date: 05 Jun 2015

Listen to Bromium and Kevin Tolly, founder and CEO of The Tolly Group, the premier independent test lab and gain valuable insights into the current challenges of preventing breaches and protecting your endpoints.

In this on-demand webinar you will learn: 1. The challenges of the current endpoint security landscape 2. Trends in endpoint security strategies, solutions and concerns 3. The impact of micro-virtualization on security

View the webinar with Kevin Tolly and Bromium by clicking on the link below which will take you to the Bromium webcast landing page.

Bromium Webcast On-demand Landing Page.

.
(Note: There is no download for this webcast item.)
Security
Huawei Technologies, Co.Ltd Huawei eSight Unified Management Platform Features Validation and Performance Evaluation

Document number: 215107
Release Date: 16 Apr 2015

Huawei eSight is a new generation of unified operation and management (O&M) system for enterprise network infrastructure, unified communications, telepresence conferencing and video surveillance. Based upon “topology-centric, simplifying management, and improving the O&M efficiency” philosophy, eSight centrally manages corporate resources, services and users. Meanwhile, eSight provides a flexible and open platform that supports enterprise customization development to allow users to build their own personalized intelligent management system.

Huawei commissioned Tolly to evaluate the ease-of-use, wired and wireless converged network management, full lifecycle WLAN management, intelligent network quality monitoring, fine right- and domain-based user management, multi-vendor device management capability and other key features.
Security
IBM Corporation Podcast: IBM Security Access Manager (ISAM) for Web
 - Web Application Protection, Performance, and Ease-of-Use Evaluation

Document number: 215116
Release Date: 06 Feb 2015

Tolly Group Founder Kevin Tolly hosts a six-minute podcast discussing Tolly's IBM Security Access Manager (ISAM) for Web
 evaluation published in Tolly report #214143.

Kevin is joined by IBM's Jason Keenaghan, Senior Product Manager, IBM Security Access Manager Family.

Stream from the Tolly Podcast page.

Download is in mp3 format.
Security
Penta Security Systems Inc. Penta Security WAPPLES Web Application Firewall (WAF) Detection Effectiveness, Performance and Management Functionality Evaluation

Document number: 214147
Release Date: 06 Feb 2015

Because the web is an essential business tool and in constant use by organizations small and large, it is also the most likely path for attacks to enter an organization. The threats are ever-changing and a challenge for most security vendors to detect. Failure to detect attacks can disrupt the flow of business; at the same time, legitimate users need to access the web server for convenience and efficiency. Penta Security has designed the WAPPLES web application firewall security engine to block web attacks, while providing accurate detection and minimizing performance degradation.

Penta Security Systems, Inc. commissioned Tolly to evaluate the effectiveness, performance and functionality of its WAPPLES WAF and compare that to a widely deployed competing product (identified in this report as Vendor X).

Tests showed that the Penta Security solution provided more effective security both at default and maximum settings, delivered higher performance at both settings and delivered greater functionality than the Vendor X solution.
Security
IBM Corporation IBM Security Access Manager (ISAM) for Web
 - Web Application Protection, Performance, and Ease-of-Use Evaluation

Document number: 214143
Release Date: 18 Nov 2014

Web applications are often the most vulnerable part of a company's infrastructure, and yet are typically given direct paths to the internet, thus leaving these vulnerabilities exposed. In recent years, Web application firewalls and access gateways have been utilized in an attempt to secure this route. However, these methods can prove ineffective against vulnerabilities specific to Web applications, riding atop valid user session and HTTP traffic.

IBM commissioned Tolly to evaluate IBM Security Access Manager (ISAM) for Web for its Web protection effectiveness, performance, and ease-of-use. The ISAM appliance is designed to sit between Web application servers and the Internet, inspecting HTTP traffic and user sessions inline and blocking attempted exploits. Testing was conducted in August 2013.
Security
Sophos Ltd. Sophos SafeGuard Enterprise Windows 7 Full Disk Encryption Performance versus Check Point Software, McAfee & Symantec Corp.

Document number: 214126
Release Date: 23 Sep 2014

In the many industries where security is paramount, encrypting the disks of user computers is standard procedure. While encryption certainly enhances security, it is important for disk encryption solutions to avoid degrading performance of the target encrypted systems.

Sophos Ltd. commissioned Tolly to evaluate the performance of the Sophos SafeGuard Enterprise disk encryption solution in a Windows 7 Enterprise OS environment and compare its performance to that of other leading solutions from Check Point Software, McAfee and Symantec Corporation. Tests were conducted using traditional hard disk drives (HDD) as well as solid state drives (SSD).

Sophos encryption delivered both the fastest initial encryption as well as the lowest overall impact on performance.
Security
Sophos Ltd. Sophos Antivirus for vShield v1.0 VMware Server and VDI Client Workload Performance versus Trend Micro Deep Security and McAfee MOVE Agentless

Document number: 214110
Release Date: 07 Apr 2014

A key element in the explosive growth of virtualization is the ability to drive the physical server hardware to higher, and more cost-efficient, utilization levels. With that in mind, it is important that server resources are not wasted by overly demanding antivirus (AV) solutions. VMware’s vShield Endpoint provides access to VM resources via a virtual appliance rather than by requiring agents on each VM.

Sophos commissioned Tolly to evaluate its Antivirus for vShield v1.0 solution and compare its performance to two other vShield-based offerings: McAfee MOVE Agentless 3.0 and Trend Micro Deep Security 9. Tests encompassed a range of VMware ESXi5.5 Microsoft Server 2008 virtual server applications including a web, database and file services. A virtual desktop infrastructure (VDI) environment with 120 Windows 7 Enterprise virtual machines on a host was also evaluated using a VMware View Planner 2.1 standard workload.

The Sophos solution demonstrated consistently better performance and, by inference, lower system resource demands than the McAfee and Trend Micro solution.
Security
IBM Corporation IBM Security Access Manager Proxy (AMP) 5100 - Web Gateway Appliance
 - Web Application Protection, Performance, and Ease-of-Use Evaluation

Document number: 213149
Release Date: 27 Sep 2013

Web applications are often the most vulnerable part of a company's infrastructure, and yet are typically given direct paths to the internet, thus leaving these vulnerabilities exposed. In recent years, Web application firewalls and access gateways have been utilized in an attempt to secure this route. However, these methods can prove ineffective against vulnerabilities specific to Web applications, riding atop valid user session and HTTP traffic.

IBM commissioned Tolly to evaluate the IBM Security Access Manager Proxy (AMP) 5100 Web Gateway Appliance for its Web protection effectiveness, performance, and ease-of-use. The AMP 5100 is designed to sit between Web application servers and the internet, inspecting HTTP traffic and user sessions inline and blocking attempted exploits.
Security
Kaspersky Lab Kaspersky Security for Virtualization v2.0 - Competitive Anti-virus Performance and Effectiveness in VMware vSphere 5.1 Virtual Environments

Document number: 213132
Release Date: 05 Aug 2013

In virtual environments, anti-virus (A/V) solutions can be implemented as a client-based agent, whereby all security processing takes place on the client, a virtual appliance that handles the A/V workload- or some hybrid of the two. As more users of virtual infrastructures begin to understand the advantages of virtualization-specific security solutions over traditional agent-based approaches, leading vendors have begun to take note, adding such virtualization-specific products to their portfolios. Efficient resource usage with minimal impact on the host and virtual infrastructure, specifically, are the primary benefits of using a solution optimized for a virtual environment.

Kaspersky Lab commissioned Tolly to benchmark the performance and effectiveness of its new, agentless Security for Virtualization v2.0 offering in VMware vSphere 5 virtual environments vs. agentless Trend Micro Deep Security 8 SP2 and McAfee MOVE Agentless Security 2.6 and agent-based Symantec SEP 12.1.2.

Tolly found that Kaspersky Security for Virtualization 2.0 blends efficient hypervisor resource usage with solid protection abilities by delivering lower average response times and disk usage than the other products tested. Kaspersky also defended against threats better than the the other agentless offerings under test.
Security
IBM Corporation Podcast: IBM Security Network Intrusion Prevention System GX7800 Comparative Efficacy and Performance Evaluation

Document number: 213130
Release Date: 07 Jun 2013

Tolly Group Director of Engineering John Tolly hosts a nine-minute podcast discussing Tolly's IBM GX7800 comparative evaluation published in Tolly report 212148.

John is joined by IBM's Senior Operations Manager Clinton McFadden.

Stream from the Tolly Podcast page.

Download is in mp3 format.
Security
Lumension, Inc Lumension® Endpoint Management and Security Suite Improving Windows Client Performance and Security: An Impact Comparison of Application Control and Traditional Anti-Virus Solutions

Document number: 213126
Release Date: 03 May 2013

While traditional anti-virus (AV) solutions can provide protection for endpoints, constantly running AV processes along with potentially frequent signature updates can consume resources that could otherwise be used to provide application services to users.

Lumension commissioned Tolly to evaluate the impact on client resources of its alternative application control solution and compare that with traditional AV solutions from Symantec Corp. and McAfee, Inc. Tests showed that the Lumension Application Control solution consumed less CPU and memory resources, thus allowing the endpoint to deliver greater performance across a diverse workload while providing effective security against zero-day threats. As a result, the aggregated time saved by the Lumension solution translates to significant savings to a business by way of time spent. See Figure 1 and Table 1. Additional tests validated the level of protection against malware afforded by the solutions.
Security
Lumension, Inc Lumension® Endpoint Management and Security Suite - Improving Server Performance and Security: An Impact Comparison of Application Control and Traditional Anti-Virus Solutions

Document number: 213121
Release Date: 11 Apr 2013

Server systems are, by definition, more important than individual endpoints, must provide services to hundreds or even thousands of endpoints and, naturally, must be secure. While traditional anti-virus (AV) solutions can provide protection for servers, constantly running AV processes along with potentially frequent signature updates can consume resources that could otherwise be used to provide application services to users.

Lumension commissioned Tolly to evaluate the impact on server resources of its alternative application control solution and compare that with traditional AV solutions from Microsoft Corp., Symantec Corp. and Trend Micro, Inc. Tests showed that the Lumension Application Control solution consumed less server CPU resource allowing the server to deliver greater throughput across a diverse workload while providing effective security against malware.
Security
IBM Corporation IBM Security Network Intrusion Prevention System GX7800 Comparative Efficacy and Performance Evaluation

Document number: 212148
Release Date: 06 Dec 2012

Enterprise-class networks today are facing more advanced threats from a multitude of sources than ever before. Effective threat protection solutions must defend against real-world threats that are evolving quickly, and at the same time deliver high levels of performance and availability. IBM commissioned Tolly to evaluate their protocol-based IBM Security Network Intrusion Prevention System GX7800 and compare its efficacy to that of a Snort-based device, a signature- based platform.

Tolly engineers conducted many different performance tests with the GX7800 and achieved a maximum of 35.7 Gbps throughput under mixed traffic loads. This demonstrates a great tolerance for network surges, growth and capacity over IBM's published performance characteristics. Tolly also evaluated the GX7800’s efficacy and functionality.

Tests showed the GX7800 to be more effective blocking publicly-available exploits than Snort and dramatically more effective when blocking mutated exploits - blocking 100% compared to 52% for Snort.
Security
Symantec Corporation Symantec Endpoint Protection 12.1 Anti-virus Effectiveness in VMware vSphere 5 Virtual Environments

Document number: 212131
Release Date: 03 Dec 2012

As virtualization-aware endpoint security solutions continue to evolve, more and more functionality is offloaded from a single VM to its supporting infrastructure in the form of Virtual Appliances (VA). In addition to considering the performance impact of this re- architecting, administrators must also ensure that the protection offered remains fully-functional, even in virtual environments.

Security
Symantec Corporation Symantec Endpoint Protection 12.1 Competitive Anti-virus Performance in VMware vSphere 5 Virtual Environments

Document number: 212130
Release Date: 03 Dec 2012

As IT architects scale deployments of virtual desktop infrastructure (VDI) solutions, they must be aware of the resource requirements of “always on” and high-use components such as endpoint security systems. In virtual environments, vendors can implement their solution as a client-based agent where all security processing takes place on the client, a virtual appliance that handles the anti-virus (A/V) workload or, possibly, some hybrid of the two approaches.

Symantec, Corp. commissioned Tolly to benchmark the performance of its new Symantec Endpoint Protection (SEP) 12.1 within VMware vSphere 5 virtual environments vs. agentless and agent-based solutions from competing vendors. Specifically, this testing focused on the system resource requirements of each solution when performing on-demand and on-access scanning functions, and during distributed virus definition updates.
Security
Kaspersky Lab Kaspersky Security Center 9 Management of Physical and Virtual Environments vs. Trend Micro

Document number: 212118
Release Date: 21 Aug 2012

The benefits of server virtualization are so compelling that virtualized environments are now a part of business environments large and small. As with their physical counterparts, though, virtualized systems require security software to be installed and managed. Kaspersky Lab has designed their security management system to provide a consistent interface across both physical and virtual systems.

Kaspersky Lab commissioned Tolly to evaluate Kaspersky Security Management and comparable security management products from Trend Micro in a scenario where both physical and virtual systems are managed. Testing confirmed that Kaspersky users can administer both physical and virtual environments from a single security console, while users of the Trend Micro products need to install and use separate management products for physical and virtual environments.
Security
Symantec Corporation Symantec Endpoint Protection 12.1 vs. Trend Micro Deep Security 8 Anti-virus Performance in VMware ESXi Virtual Environments

Document number: 212117
Release Date: 07 May 2012

As IT architects scale deployments of virtual desktop infrastructure (VDI) solutions, they must be aware of the resource requirements of “always on” and high-use components such as endpoint security systems. In virtual environments, vendors can implement their solution as a client-based agent where all security processing takes place on the client, a virtual appliance that handles the anti-virus (A/V) workload or, possibly, some hybrid of the two approaches.

Symantec Corp. commissioned Tolly to benchmark the performance of its new Symantec Endpoint Protection (SEP) 12.1 within VMware ESXi 5 virtual environments vs. Trend Micro Deep Security 8. Specifically, this testing focused on the system resource requirements of each solution when performing on-demand and on-access scanning, and during distributed virus definition updates.
Security
AVG Technologies AVG Internet Security Business Edition 2012 Ease of Management and Usability Evaluation Against Competing Security Suites for SMB Users

Document number: 212102
Release Date: 13 Mar 2012

Endpoint security is just as important for small and medium business (SMB) environments as it is for large enterprises. SMBs, though, are likely to have limited or no dedicated IT resources to manage installation and ongoing policy updates. Thus, understanding the effort required to install a solution and conduct ongoing maintenance tasks is important when choosing an endpoint security solution for SMB deployment.

Testing showed that with AVG users can save valuable time in managing their security solution -- time that can be better used to focus on their business. Testing showed that AVG is a solution that is one of the easiest-to-use for most common tasks such as running scans, installing across a small business network, etc.
Security
Bit9 Bit9 Parity Suite 6.0 Comparison of Bit9 Advanced Threat Solution versus McAfee Endpoint Protection Suite and Symantec Endpoint Protection 12.1

Document number: 212108
Release Date: 12 Mar 2012

Security and network operation center personnel have traditionally had few alternatives when it comes to securing endpoints and servers in their enterprise, with most solutions being some variation of the traditional antivirus blacklisting approach to security. Advanced threats which put valuable intellectual property at risk have quickly become a significant security concern for corporations.

Bit9 commissioned Tolly to evaluate leading endpoint security solutions to compare the effectiveness of traditional antivirus products and the most recent application control approach against malware and zero-day threats. The application control (whitelisting) solution tested was Bit9 Parity Suite 6.0, and the antivirus (blacklisting) solutions tested were McAfee Endpoint Protection Suite and Symantec Endpoint Protection 12.1. Testing of 5 attacks across a number of systems showed that Bit9 Parity Suite protects both Windows clients and Windows Web servers more effectively than the Symantec and McAfee endpoint solutions under test.
Security
Sophos Ltd. Sophos Endpoint Security and Control v9.7: Anti-virus Performance in VMware ESX Virtual Environments

Document number: 211125
Release Date: 26 Aug 2011

As IT architects scale deployments of virtual desktop infrastructure (VDI) solutions, they must be aware of the resource requirements of “always on” and high-use components such as endpoint security systems. In virtual environments, vendors can implement their solution as a client-based agent, where all processing for each client takes place on the client, a virtual appliance that handles the anti- virus (AV) workload or, possibly, some hybrid of the two approaches.

Sophos Ltd. commissioned Tolly to benchmark the performance of its new Sophos Endpoint Security and Control v9.7 within virtual environments. Specifically, this testing focused on the system resource requirements of the Sophos client-based agent when performing on-demand/on-access scanning and virus signature definition update tasks.
Security
Symantec Corporation Symantec Endpoint Protection 12.1 vs. McAfee and Trend Micro Anti-virus Performance in VMware ESX Virtual Environments

Document number: 211123
Release Date: 17 Aug 2011

As IT architects scale deployments of virtual desktop infrastructure (VDI) solutions, they must be aware of the resource requirements of “always on” and high-use components such as endpoint security systems. In virtual environments, vendors can implement their solution as a client-based agent where all processing for each client takes place on the client, an virtual appliance that handles the a/v workload or, possibly, some hybrid of the two approaches.

Symantec Corp. commissioned Tolly to benchmark the performance within virtual environments of its new Symantec Endpoint Protection 12.1 vs. comparable solutions from McAfee and Trend Micro. Specifically, this testing focused on the disk input/output requirements of each solution when performing on-access scan, on- demand scan and virus definition update.

Security
Network Box Corp. M-Series M-285 Internet Security Appliance WildList Malware Detection Evaluation

Document number: 211117
Release Date: 26 May 2011

Businesses of all sizes and types need to be concerned with providing protection from malware to their user base. Irrespective of the delivery mechanism, Unified Threat Management (UTM) firewall appliances aimed at small and medium businesses should deliver immediate and ongoing protection against malware threats propagating on the Internet.

Network Box commissioned Tolly to evaluate how effectively the Network Box M-285 solution detected malware. Using malware samples from the WildList, augmented by additional samples provided by AV-Test, engineers evaluated the detection rates across the HTTP, POP3 and SMTP protocols.

Security
Trend Micro, Inc Trend Micro Deep Security 7.5 vs. McAfee and Symantec Anti-virus Performance in VMware ESX Virtual Environments

Document number: 211101
Release Date: 09 Feb 2011

Server and desktop virtualization are essential elements of any IT strategy that seeks to decrease capital and operational expenditures . In the rush to implement virtualization technologies, many organizations simply deploy the same anti-virus solution that is in use on their physical server and desktop systems. Because these traditional anti-virus solutions are not designed specifically for virtual environments, they can create significant operational issues such as anti-virus (AV) storms, resource wastage and administrative overhead, and hamper the organization’s objective of maximizing VM densities.

Trend Micro, Inc. commissioned Tolly to benchmark the performance within virtual environments of the Trend Micro Deep Security solution vs. McAfee Total Protection for Endpoint and Symantec Endpoint Protection 11.0. Specifically, this testing evaluated the impact each solution had on host system (physical server) resources especially as guest machine density increased to up to 100 virtual machines simultaneously running in a VMware ESX 4.1 environment.
Security
Biometric Signature ID Biometric Signature ID - BioSig-ID 2.0 User Authentication Solution Using Signature Gesture Biometrics Ease of Use, Enrollment, Accuracy and Protection Evaluation

Document number: 211104
Release Date: 24 Jan 2011

Single-stage password security mechanisms that act as the front door to user accounts in enterprise networks are susceptible to imposters who successfully steal legitimate user ID and password data.

To strengthen user account security, Biometric Signature ID (BioSig-ID) developed the BioSig-ID solution for Windows client workstations. The solution records a signature profile of a user’s mouse gestures while writing a code through an enrollment process, using that to validate the user during account logon. BioSig-ID uses a form of dynamic biometrics known as “signature/ gesture dynamics.”

This tokenless approach creates a second layer of account logon verification and guards against the possible use of stolen password and account data to gain entry to the network.

Tolly engineers measured the effectiveness and accuracy of the BioSig-ID solution as tested with 93 test subjects accessing their own accounts and also attempting to access 20 “victim” (other user’s accounts) after being supplied with the victims’ credentials. Over 15,000 logon attempts were monitored during the evaluation.
Security
NETGEAR, Inc. ProSecureTM UTM25 and UTM50 UTM Firewall Appliances for Defense against Web 2.0/ Social Media Threats: Malware Detection Evaluation against Competing Products

Document number: 210153
Release Date: 21 Oct 2010

NETGEAR commissioned Tolly and AV-Test.org to evaluate its Prosecure UTM25 and ProSecure UTM50 UTM firewall appliances as well as comparable solutions from Cisco, Fortinet, SonicWALL and WatchGuard.

Tests focused on detection rates for zoo malware as well as threats found on the Extended WildList across HTTP, POP3 and SMTP protocols.
Security
Trend Micro, Inc Trend Micro Titanium Maximum Security 3.0 Consumer Endpoint Security Performance vs K7, Kaspersky, McAfee & Symantec

Document number: 210142
Release Date: 08 Sep 2010

Trend Micro commissioned Tolly to benchmark the performance of its Titanium Maximum Security endpoint security solution vs several competitors. In addition to the document found on this page, an appendix document (210142A) is available here. This appendix document contains tabular data and additional test methodology details. Security
NETGEAR, Inc. ProSecure™ UTM10 Unified Threat Management Appliance: Malware Detection Evaluation Versus Fortinet, Inc., SonicWALL, Inc., and WatchGuard Technologies, Inc.

Document number: 209131
Release Date: 26 Oct 2009

NETGEAR commissioned Tolly to conduct a malware detection accuracy evaluation of the ProSecure UTM10, Fortinet FortiGate-60B, SonicWALL TZ 100, SonicWALL TZ 210 and WatchGuard Firebox Edge X55e UTM appliances.

Tests focused on the malware detection capabilities of the above mentioned UTM appliances using their default security policies, over the Web traffic and email vectors using HTTP, POP3 and SMTP protocols. Test malware samples consisted of The WildList Organization International’s latest WildList (a list of viruses and worms found propagating on the Internet) along with other major Win32 malware.

Download the free report.
Security
Sunbelt Software, Inc. Sunbelt Software VIPRE Enterprise 3.1: Anti-virus Scanning Performance and System Resource Utilization Comparison Versus McAfee VirusScan Enterprise 8.7i and Symantec Endpoint Protection 12 Small Business Edition

Document number: 209138
Release Date: 28 Sep 2009

Sunbelt Software commissioned Tolly to evaluate the anti-virus scanning performance and system resource utilization of its VIPRE Enterprise 3.1 anti-virus product in comparison to McAfee VirusScan Enterprise 8.7i and Symantec Endpoint Protection 12 Small Business Edition products.

Tests focused on measuring the memory and CPU utilization of the products under test during various stages of product operation: at idle, with the product GUI open, during an on-demand full-system scan, etc.

Security
Lumension, Inc Lumension VMS versus Microsoft WSUS: Total Cost of Ownership Comparison

Document number: 209135
Release Date: 18 Sep 2009

Lumension commissioned Tolly to evaluate the total cost of ownership (TCO) associated with managing system vulnerabilities on Windows environments and to compare that with free Windows Server Update Service (WSUS) offered by Microsoft.

Tests explored patching Microsoft and non-Microsoft applications, discovering new and/or unauthorized clients, CVE patching, software removal and other areas.

Download the free report.
Security
Symantec Corporation Symantec Endpoint Protection Small Business Edition 12.0 Competitive Windows XP Performance Evaluation

Document number: 209110
Release Date: 27 Apr 2009

Symantec commissioned Tolly to evaluate the impact of endpoint security offerings designed for small businesses on PC client responsiveness.

The Tolly Group compared the Windows XP client version of Symantec Endpoint Protection Small Business Edition 12.0 against security offerings from AVG, BitDefender, Kaspersky Lab, McAfee, Inc., Sophos and Trend Micro, Inc.

The Tolly Group examined system start-up time, the impact on Microsoft Office 2007, on Internet Explorer, on local and network file operations and on the time required to decompress a file archive.

Symantec Endpoint Protection Small Business Edition 12.0 consistently delivered faster response time than competing products tested. Detailed test results are documented in a companion “Appendix” document which can be downloaded from tolly.com at Symantec appendix download.

Tests were conducted in March 2009.

Download the free report.
Security
Symantec Corporation Symantec Endpoint Protection Small Business Edition 12.0 Competitive Windows XP Performance Evaluation - APPENDIX

Document number: 209110APPENDIX
Release Date: 25 Apr 2009

Symantec commissioned Tolly to evaluate the impact of endpoint security offerings designed for small businesses on PC client responsiveness.

The Tolly Group compared the Windows XP client version of Symantec Endpoint Protection Small Business Edition 12.0 against security offerings from AVG, BitDefender, Kaspersky Lab, McAfee, Inc., Sophos and Trend Micro, Inc.

The Tolly Group examined system start-up time, the impact on Microsoft Office 2007, on Internet Explorer, on local and network file operations and on the time required to decompress a file archive.

Symantec Endpoint Protection Small Business Edition 12.0 consistently delivered faster response time than competing products tested. This appendix document contains detailed results. Higher-level test results are documented in a companion document which can be downloaded from tolly.com at Symantec test report download.

Tests were conducted in March 2009.

Download the free report.
Security
Red Condor, Inc. Red Condor Message Assurance Gateway 2700: Anti-spam Effectiveness and Feature Comparison versus Solutions From Barracuda, Cisco and Google

Document number: 209107
Release Date: 22 Apr 2009

Red Condor commissioned Tolly to evaluate the anti-spam effectiveness of its Message Assurance Gateway 2700 against competing solutions from Cisco Systems’ IronPort C150 Email Security Appliance, Barracuda Networks’ Spam Firewall 300 and Google’s Message Security powered by Postini.

Testing focused on the anti-spam effectiveness in terms of spam detection percentage, spam error percentage and false positive rate of the solutions under test during a calendar week of testing per solution, while deployed in the live corporate network of The Tolly Group.

The testing was conducted in accordance with Tolly Common Test Plan #1058, Anti-spam Gateway v1.0.

Click below to download the free report.
Security
Open Text Connectivity Solutions Group Évaluation: Connectivity Secure Server 1.0 d’Open Text a été comparé à Reflection for Secure IT Server 6.1 d’Attachmate, ainsi qu’à SSH Tectia Server 6.0, de SSH Communications Security

Document number: 209100FR
Release Date: 03 Mar 2009

Connectivity Secure Server d’Open Text a nettement surclassé les serveurs SSH espectivement conçus par Attachmate Corp. et SSH Communications Security, offrant une vitesse de traitement supérieure et supportant une montee en charge permettant d’assurer plus de 1000 téléchargements par serveur effectués par les utilisateurs. Ces performances lui ont permis de mettre en évidence un coût total de possession sans correspondance avec celui offert par les produits concurrents avec lesquels il était comparé au cours de l’essai.

Au cours de cette batterie d’essais, effectués en décembre 2008, Connectivity Secure Server d’Open Text a systématiquement surclassé Reflection for Secure IT Server d’Attachmate Corp. et le serveur SSH Tectia de SSH Communications Security. Lors des essais de transfert d’un fichier de 36 Mo d’un serveur vers un client qui en avait effectué la requête, Connectivity Secure Server a exécuté la tâche dix fois plus vite que le serveur Tectia Server de SSH et vingtquatre fois plus vite que le serveur Reflection for Secure IT d’Attachmate.

Download the free report.
Security
McAfee McAfee Total Protection for Virtualization Evaluation in VMware ESX and Microsoft Hyper-V Environments

Document number: 208344
Release Date: 30 Jan 2009

McAfee, Inc. commissioned The Tolly Group to evaluate the effectiveness of McAfee Total Protection (ToPS) for Virtualization, managed by ePolicy Orchestrator (ePO) 4.0, in providing a comprehensive suite of security services to virtualized Microsoft Windows Server 2003 and Windows Server 2008 environments — online and offline — under both VMware and Microsoft’s Hyper-V server virtualization platforms.

Tolly engineers built virtual server environments using both VMware ESX Server version 3.5 and Microsoft Hyper-V. In these environments they deployed virtual instances of Microsoft’s Windows Server 2003 and Windows Server 2008. Engineers then exercised an extensive set of functions to illustrate that McAfee could provide extensive management and protection of virtual server environments in both online and offline states.

Click on the report icon to purchase the report.
Security
Passlogy Passlogy - PatternPass 1000, Evaluation of Token-less PASSLOGIC One-time Password Authentication System

Document number: 208290
Release Date: 26 Jan 2009

Passlogy Co., Ltd. commissioned The Tolly Group to evaluate the PASSLOGIC token-less one-time password (OTP) authentication system running on its PatternPass 1000 appliance. The PASSLOGIC technology implemented a patented 2-way, 2-factor, one-time password authentication mechanism that does not require a dedicated hardware token.

Tests focused on the ability of Passlogy’s PatternPass 1000 appliance and PASSLOGIC system to resist security events such as brute-force attacks or random account locking attacks. Engineers also tested the compatibility of the PASSLOGIC system to work in an SSL VPN scenario, as well as using multiple portable devices like portable game devices, PDAs, mobile phones, etc. to generate the one-time password.
Download the free report.
Security
Open Text Connectivity Solutions Group Evaluation: Open Text Connectivity Secure Server 1.0 vs. Attachmate Reflection for Secure IT Server 6.1 and SSH Communications Security SSH Tectia Server 6.0

Document number: 209100
Release Date: 19 Jan 2009

Open Text’s Connectivity Secure Server outperformed secure shell servers from Attachmate Corp. and SSH Communications Security, delivering superior processing speed and scaling to support over 1,000 user downloads per server which enables it to deliver a cost of ownership unmatched by rival products tested.

Open Text’s Connectivity Secure Server consistently outperformed Attachmate Corp.’s Reflection for Secure IT Server and SSH Communications Security’s SSH Tectia Server. In tests transferring a 36MB file from a server to a requesting client, Connectivity Secure Server completed the task 10X faster than SSH’s Tectia Server and 24X faster than Attachmate’s Reflection for Secure IT server.

Testing was conducted in December 2008.
Security
Radware Ltd. Radware OnDemand Switch 1 & OnDemand Switch 2 AppDirector Version 1.06 Competitive Performance Evaluation versus F5 Networks BIG-IP 6400

Document number: 208287
Release Date: 16 Sep 2008

Radware commissioned The Tolly Group to evaluate the performance of its OnDemand Switch 1 & 2 AppDirector, the company’s nextgeneration application switch, in comparison with F5 Network’s BIG-IP 6400.

The goal was to measure performance while handling challenging tasks to determine which device offers the higher Layer 7 transaction rate and faster response time. Tolly Group engineers determined the transactions-per-second (tps) rate and the corresponding throughput and response time for 10 object sizes at Layer 7 in multiple scenarios of a single HTTP request per connection and 10 HTTP requests per connection. They also in conjunction tested the ability of the platforms to maintain performance while dealing with DDoS attack packets.

Tests were conducted in February 2008.

Download the free report.
Security
Radware Ltd. Radware OnDemand Switch 1 & OnDemand Switch 2 AppDirector Version 1.06 Competitive Performance Evaluation versus F5 Networks BIG-IP 6800

Document number: 208288
Release Date: 16 Sep 2008

Radware commissioned The Tolly Group to evaluate the performance of its OnDemand Switch 1 & 2 AppDirector, the company’s nextgeneration application switch, in comparison with F5 Network’s BIG-IP 6800.

The goal was to measure performance while handling challenging tasks to determine which device offers the higher Layer 7 transaction rate and faster response time. Tolly Group engineers determined the transactions-per-second (tps) rate and the corresponding throughput and response time for 10 object sizes at Layer 7 in multiple scenarios of a single HTTP request per connection and 10 HTTP requests per connection. They also in conjunction tested the ability of the platforms to maintain performance while dealing with DDoS attack packets.

Tests were conducted in February 2008.

Download the free report.
Security
Symantec Corporation Symantec Corporation Symantec Endpoint Protection 11.0 vs. McAfee Total Protection for Endpoint Performance Impact on Microsoft Office Usage

Document number: 208321
Release Date: 12 Sep 2008

Symantec Corporation commissioned The Tolly Group to evaluate the impact of two Enterprise class endpoint security offerings on host client performance: Symantec Endpoint Protection 11.0 compared with McAfee Total Protection for Endpoint. The Tolly Group installed Symantec Endpoint Protection 11.0 which provides anti-virus, anti-spyware and host intrusion prevention functionality in a single agent against the corresponding products in the McAfee Total Protection for Endpoint Bundle.

The Tolly Group benchmarked file “open” and “save/close” times, as well as memory usage on an unprotected Microsoft Windows Vista SP1 system and compared these with execution times on the protected systems.

Tests were conducted in July 2008.

Security
Nortel Nortel Secure Network Access Solution Validation of Open and Flexible Network Access Control Features, Enterprise Performance, Scalability and Redundancy

Document number: 208302
Release Date: 09 Sep 2008

Nortel commissioned The Tolly Group to evaluate its Secure Network Access (SNA) Solution, which controls and automates network access for both managed and unmanaged users and devices.

Tolly Group engineers examined two models: the Nortel Secure Network Access Switch (SNAS) 4050 and 4070. Engineers validated an assortment of key network access control features and functions, as well as verified scalability capabilities of the device and energy consumption.

Tests were conducted in August 2008.
Security
Trustwave Trustwave TS-1000 High-Speed Intrusion Prevention Appliance IPS Performance and Security Effectiveness Evaluation

Document number: 208296
Release Date: 25 Jun 2008

Trustwave commissioned The Tolly Group to evaluate its TS- 1000 High-Speed Intrusion Prevention Appliance to determine its performance levels along with its security effectiveness.

The Tolly Group measured the Layer 3 zero-loss (􀀁0.001% acceptable packet loss) throughput and latency of the TS-1000 in firewall only mode, with intrusion prevention active, and while under attack. Engineers also measured the TS-1000’s support for concurrent TCP connections and its connection setup rate. Finally, engineers subjected the TS-1000 to a variety of security attacks and identified the product’s effectiveness at protecting servers.

Tests were conducted in March 2008.

Security
Cymtec Cymtec Systems, Inc. Cymtec Sentry™ Propagation Protection Solution Detection Accuracy and Network Performance Evaluation

Document number: 208279
Release Date: 20 May 2008

Cymtec Systems, Inc. commissioned The Tolly Group to evaluate its Cymtec Sentry Propagation Protection Solution.

The Tolly Group examined the network performance and accuracy of the Cymtec Sentry. Tests measured throughput and latency; accuracy of threat detection; fail-over response; traffic shaping; and session limits. The Cymtec Sentry is composed of hardware appliances that are placed in specific network segments where propagation protection is required. All appliances are then managed by the Cymtec Sentry Management Console software. This controls the Cymtec Sentry Appliances installed on the network including configuration, current threats, alerts, and statistical information.

Tests were conducted in April 2008.

Security
McAfee McAfee®, Inc. TCO-evaluatie van McAfee Total Protection Service1 in vergelijking met Symantec Endpoint Protection Small Business Edition 11.0 en Trend Micro Client Server Messaging Security for SMB

Document number: 208255NL
Release Date: 30 Mar 2008

McAfee, Inc. heeft The Tolly Group opdracht gegeven de effectiviteit te evalueren van haar McAfee Total Protection Service - Advanced. Dit is een SaaS-aanbod (Security as a Service) dat uitgebreide beveiligingsvoorzieningen biedt ter bescherming tegen virussen, spyware, hackers en andere bedreigingen.

Technici van The Tolly Group voerden een gedetailleerde praktijkevaluatie uit van het serviceaanbod van McAfee en vergeleken dit met Symantec Endpoint Protection Small Business Edition 11.0 en Trend Micro Client Server Messaging Security for SMB Ver 7.6. Dit zijn traditionele softwareproducten die hulpbronnen op locatie voor implementatie en ondersteuning vereisen. De technici namen de initiële implementatie, de totale eigendomskosten (Total Cost of Ownership, oftewel TCO), het beheer en de bruikbaarheid van McAfee Total Protection en de twee andere producten zorgvuldig onder de loep. De tests werden uitgevoerd in december 2007.
Security
McAfee McAfee®, Inc. Bewertung der Gesamtbetriebskosten von McAfee Total Protection Service1 im Vergleich zu Symantec Endpoint Protection Small Business Edition 11.0 und Trend Micro Client Server Messaging Security for SMB

Document number: 208255DE
Release Date: 30 Mar 2008

McAfee, Inc. hat The Tolly Group damit beauftragt, die Effektivität von McAfee Total Protection Service – Advanced zu bewerten. Bei diesem Produkt handelt es sich um ein Security-as-a-Service-Angebot (SaaS), das umfassenden Schutz vor Viren, Spyware, Hackern und anderen Bedrohungen bietet.

Die Ingenieure von The Tolly Group führten eine detaillierte praktische Bewertung der McAfee- Servicelösung durch und verglichen die Ergebnisse mit Symantec Endpoint Protection Small Business Edition 11.0 und Trend Micro Client Server Messaging Security for SMB Version 7,6 - herkömmlichen Softwareprodukten, für deren Bereitstellung und Support Ressourcen und Mitarbeiter vor Ort erforderlich sind. Die Ingenieure unterzogen McAfee Total Protection und die beiden anderen Produkte einer genauen Untersuchung hinsichtlich Erstbereitstellung, Gesamtbetriebskosten, Verwaltung und Verwendbarkeit. Die Tests wurden im Dezember 2007 durchgeführt.
Security
NetClarity, Inc. EasyNAC Enterprise™, EasyNAC Branch™ & Endpoint Defender™ Vulnerability Management Appliances and Host-based Intrusion Prevention System Evaluation of Vulnerability Management, Network Admission Control (NAC) Features and Endpoint Security Package

Document number: 208294
Release Date: 06 Mar 2008

NetClarity commissioned The Tolly Group to evaluate its EasyNAC Enterprise™, EasyNAC Branch™ appliances and Endpoint Defender™ endpoint security solution in terms of security vulnerability management, Network Admission Control (NAC) and endpoint security features.

Tests evaluated the various features of the products and evaluated how they interoperated to provide a comprehensive security solution that also helps ensure and document regulatory compliance. Tests also evaluated the ease of deployment and ongoing maintenance of the products. Tests were conducted in January 2007.

Security
McAfee TCO Evaluation of McAfee Total Protection Service vs. Symantec Endpoint Protection Small Business Edition 11.0 and Trend Micro Client Server Messaging Security for SMB

Document number: 208255
Release Date: 27 Feb 2008

McAfee, Inc. commissioned The Tolly Group to evaluate the effectiveness of its McAfee Total Protection Service — Advanced, a Security as a Service (SaaS) offering that provides comprehensive security to protect against viruses, spyware, hackers and other threats.

Tolly Group engineers conducted a detailed hands-on evaluation of the McAfee service offering and compared it against the Symantec Endpoint Protection Small Business Edition 11.0 and Trend Micro Client Server Messaging Security for SMB Ver 7.6, traditional software products that require on-site resources for deployment, and support. Engineers closely examined initial deployment, Total Cost of Ownership (TCO), management and usability factors of McAfee Total Protection and the two other products. Tests were conducted in December 2007.

Security
Intoto iGateway Software Firewall on Intel Multi-Core UDP Throughput and HTTP Connection Rate Performance Evaluation

Document number: 208264
Release Date: 08 Feb 2008

INTOTO, Inc. commissioned The Tolly Group to evaluate the iGateway Firewall. iGateway Firewall is a software-based carrier grade/large enterprise firewall that provides high-performance and scalability while running on off-the-shelf, general purpose Intel multi-core hardware platforms. The software includes stateful inspection with packet integrity checks, access policies, content filtering, local proxy redirection and DDoS attack prevention. Testing focused on evaluating firewall performance with stateful inspection and access policies.

Tests measured the zero-loss (≤0.001%) aggregate UDP throughput with multiple simultaneous sessions and HTTP connection rate with different policy matching for its eight-core, quad-core and dual-core systems. Tests were conducted in November 2007.

Security
Fortinet, Inc. FortiMail-100™ Anti-spam Effectiveness and Feature Comparison vs. Barracuda Networks Spam Firewall 200

Document number: 207259
Release Date: 14 Jan 2008

Fortinet, Inc. commissioned The Tolly Group to measure the effectiveness of the company’s FortiMail-100™ multi-layered E-mail security appliance at blocking spam and virus messages.

Tolly Group engineers tested the spam detection effectiveness of the FortiMail-100 against a Barracuda Networks Spam Firewall 200. In accordance with The Tolly Group’s Fair Testing Charter, Barracuda was invited to review the test methodology, offer suggestions for its product and comment on its results.

Engineers measured the percentage of spam blocked, the number of “false positives,” “false negatives” and virus messages detected per product. Engineers also validated a number of E-mail security features and deployment flexibility. Tests were conducted in October and November 2007.

Security
RSA,The Security Division of EMC RSA® DLP Network Competitive Evaluation of Content Detection Accuracy versus Symantec Mail Security 8240

Document number: 208282
Release Date: 10 Jan 2008

RSA, The Security Division of EMC, commissioned The Tolly Group to measure the accuracy of RSA DLP Network, a server-based appliance designed to prevent the loss of sensitive corporate data by monitoring and blocking transmissions from a corporate network.

Tests focused on measuring “recall,” the product’s ability to correctly identify all documents that contain sensitive information and “precision,” or the percentage of retrieved documents that are sensitive and relevant. The product with the best recall offers the best protection against data loss or misuse; the product with the best precision offers ease of use and lower cost of ownership.

Engineers measured the accuracy of RSA DLP Network against Symantec Corp.’s Mail Security 8240, an appliance that merges anti-spam, anti-virus, content protection and other capabilities (with content detection technology from Vontu, Inc.).

Tests were executed using pre-built policies for Personally Identifiable. Information (PII) and Payment Card Industry (PCI) data included in both products. Use cases were developed by an independent expert at a leading West coast university and have not been shared with the vendors. Tests focused on identification of sensitive content. Tests were conducted in June 2007.

Security
Fortinet, Inc. FortiMail-4000A™ Anti-spam Effectiveness and Feature Comparison vs. IronPort Systems C350 E-mail Security Appliance

Document number: 207258
Release Date: 11 Dec 2007

Fortinet, Inc. commissioned TheTolly Group to measure the effectiveness of the company’s FortiMail-4000A™ multi-layered E-mail security appliance at blocking spam and virus messages.

Tolly Group engineers tested the performance of the FortiMail-4000A against an IronPort Systems C350 E-mail Security Appliance. In accordance with The Tolly Group’s Fair Testing Charter, IronPort was invited to review the test methodology, offer suggestions for optimal configuration of its product and comment on its results.

Engineers measured the percentage of spam blocked, the number of “false positives,” “false negatives” and virus messages detected per product. Engineers also validated a number of E-mail security features and deployment flexibility. Tests were conducted in November 2007.

Security
The Tolly Group "Tolly Benchmarks" Vol. 6 No. 4 - Highlights of 3Com, Reflex, Procera Networks, Nortel, Vyatta, Mirage and NETGEAR projects.

Document number: 207273
Release Date: 19 Nov 2007

Industry notables such as 3Com, Reflex, Procera Networks, Nortel, Vyatta, Mirage and NETGEAR are featured in the October 29, 2007 issue of Tolly Benchmarks, a regular advertising supplement in Network World newspaper.

Tolly Benchmarks is open exclusively to vendors who have completed testing with The Tolly Group in the past year. The color, magazine-style supplement reaches more than 150,000 network decision makers who read Network World. This edition of Tolly Benchmarks focuses on such issues as network applications, access control, security, traffic and service management, as well as ethernet routing and switching.

Security
Mirage Networks Mirage Endpoint Control™ NAC Solution Evaluation of Network Access Control for Real-World Endpoints and Applications

Document number: 207252
Release Date: 18 Oct 2007

Mirage networks, Inc. commissioned The Tolly Group to validate the chief capabilities of the company’s Endpoint Control network access control (NAC) solution.

Tolly Group engineers examined the Mirage Endpoint Control NAC solution (consisting of the Endpoint Control 145 Advanced Compliance Server, the Endpoint Control 245 sensor appliance and the Mirage Operations Console (MOC) management applications) for its ability to detect and restrict network access of both managed and rogue devices and applications that fail to comply with IT security and behavioral policies.

Engineers tested high-risk applications like Peer-to-Peer (P2P) file-sharing services, Instant Messaging (IM) services, unknown or rogue devices like mobile communication devices, game consoles, personal routers and servers, that have the potential to introduce risk inside an organization’s trusted network. Tests were conducted in September 2007.

Security
Tablus, Inc. Tablus, Inc. Content Alarm NW Competitive Evaluation of Content Detection Accuracy versus Symantec Mail Security 8240

Document number: 207210
Release Date: 23 Aug 2007

Tablus, Inc. commissioned The Tolly Group to measure the accuracy of the company’s Content Alarm NW, a server-based appliance designed to prevent the loss of sensitive corporate data by monitoring and blocking transmissions from a corporate network.

Tolly Group engineers measured the accuracy of Content Alarm NW against Symantec Corp.’s Mail Security 8240. Tests focused on measuring “recall,” the product’s ability to correctly identify all documents that contain sensitive information and “precision,” or the percentage of retrieved documents that are sensitive and relevant. Tests were conducted in June 2007.

Security
BioPassword, Inc. (AdmitOne Security) BioPassword, Inc. Enterprise Edition 3.2 Accuracy Evaluation of Keystroke Dynamics

Document number: 207233
Release Date: 07 Aug 2007

BioPassword, Inc. commissioned The Tolly Group to evaluate the accuracy of its BioPassword Enterprise Edition 3.2 software-based biometric authentication solution in a Windows XP business environment.

Tolly Group engineers evaluated the accuracy and effectiveness of the BioPassword solution to allow legitimate end-users to log on to their accounts without issues, while also offering high security for Windows accounts when credentials have been shared or compromised with other users. Testing was conducted in June 2007.

Security
Reflex Security, Inc. Reflex Security, Inc. MG10 Network Security System Performance Evaluation under Severe Attack Strain with No Transaction Loss and High Availability Examination

Document number: 207219
Release Date: 17 Jul 2007

Reflex Security, Inc. commissioned The Tolly Group to measure the performance of the vendor’s Reflex MG10, a network system that employs a blade-based Distributed Security Architecture™ (DSA) that provides scalable throughput from 10 Mbps to 10 Gigabit per second (Gbps).

Engineers measured the multi-Gigabit performance of the MG10, both with and without exposing the device to a serious load of security threats. Engineers also measured the number of open TCP connections sustained across the MG10, and examined how the unit responds during an invoked failure. Testing was conducted in June 2007.

Security
Cymtec Cymtec Sentry™ Fast Ethernet Propagation Protection Solution Detection Accuracy and Network Performance Evaluation

Document number: 207167
Release Date: 14 Jun 2007

Cymtec Systems, Inc. commissioned The Tolly Group to evaluate its Cymtec Sentry Propagation Protection Solution. The Tolly Group examined the network performance and accuracy of the Cymtec Sentry. Tests measured throughput and latency, threat detection accuracy, fail-over response, traffic shaping, and session limits.

Security
Colubris Networks White Paper Sponsored by Colubris Networks: Evaluating Wireless IPS Systems

Document number: 207231
Release Date: 23 May 2007

This Tolly Group white paper, commissioned by Colubris Networks, Inc., focuses on the key issues users must consider when evaluating wireless intrusion prevention systems. For the report, The Tolly Group evaluated the Colubris RF Manager, a multi-faceted WIPS designed to protect enterprise network infrastructures from wireless attacks.

The Tolly Group assessed the capability of the Colubris RF Manager to detect and block a range of wireless threats — from dealing with rogue APs, to detection and prevention of access point (AP) MAC address spoofing, to detection and prevention of Denial of Service (DoS) attacks, and several others.

Tolly Group engineers measured the effectiveness of the Colubris RF Manager against two other products: AirMagnet Inc.’s AirMagnet Enterprise and Aruba Networks Aruba Mobility Controller.

Security
3Com Corp. TollyEdge White Paper Series: Benchmarking Strategies for Wireless Intrusion Prevention Systems

Document number: 207216
Release Date: 19 May 2007

This comprehensive 32-page TollyEdge: Benchmarking Strategies for Wireless Intrusion Prevention Systems white paper identifies the chief factors with regards to protection, performance and ease-of-use for WIPS offerings that users need to address, and The Tolly Group offers its insights into the most practical way to benchmark these essential criteria.

The report provides unique perspectives from 3Com and AirDefense. The report aims to help readers understand the key issues they must consider, and the key metrics and processes they should employ to effectively benchmark any WIPS products.

Security
NetClarity, Inc. NetClarity Auditor Enterprise, Auditor Branch & Protection for Windows Manejo de Vulnerabilidad de Activos y Sistema Preventivo de Intrusos Evaluación en el Manejo de Vulnerabilidad, Características del Network Admission Control (NAC)

Document number: 207183ES
Release Date: 01 Feb 2007

NetClarity delegó a The Tolly Group a desarrollar pruebas en sus productos de seguridad Auditor Enterprise™, Auditor Branch™ y Protection for Windows™ en áreas del manejo de vulnerabilidad, Network Admission Control (NAC) y características de seguridad para terminales.

Las pruebas evaluaron diferentes características e implementacion de los productos al proveer soluciones de seguridad cumpliendo con las regulaciones acorde a la industria. Las pruebas también evaluaron su facilidad de instalación y su continuo manteniemiento en los productos. Las pruebas se realizaron en Enero del 2007.

Security
NetClarity, Inc. NetClarity Auditor Enterprise, Auditor Branch & Protection for Windows Vulnerability Management Appliances and Host-based Intrusion Prevention System Evaluation of Vulnerability Management, Network Admission Control Features and Endpoint Security Package

Document number: 207183
Release Date: 01 Feb 2007

NetClarity commissioned The Tolly Group to evaluate its Auditor Enterprise, Auditor Branch appliances and Protection for Windows endpoint security solution in terms of security vulnerability management, Network Admission control (NAC) and endpoint security features.

Tests evaluated the various features of the products and evaluated how they interoperated to provide a comprehensive security solution that also helps ensure and document regulatory compliance. Tests also evaluated the ease of deployment and ongoing maintenance of the products. Tests were conducted in January 2007.

Security
The Tolly Group TollyEdge White Paper Series: Benchmarking Strategies for Wireless Intrusion Prevention Systems

Document number: 207117
Release Date: 21 Jan 2007

This comprehensive 32-page TollyEdge: Benchmarking Strategies for Wireless Intrusion Prevention Systems white paper identifies the chief factors with regards to protection, performance and ease-of-use for WIPS offerings that users need to address, and The Tolly Group offers its insights into the most practical way to benchmark these essential criteria.

The report provides unique perspectives from AirDefense and AirTight Networks. The report aims to help readers understand the key issues they must consider, and the key metrics and processes they should employ to effectively benchmark any WIPS products.

Security
InfoExpress InfoExpress Dynamic Network Access Control Competitive “Ease-of-Use” Comparison versus Cisco Network Admission Control and Cisco Clean Access

Document number: 207165
Release Date: 19 Jan 2007

InfoExpress commissioned The Tolly Group to evaluate its Dynamic NAC (DNAC) 5 versus Cisco Systemss, Inc.’s Cisco Network Access Control (NAC) 2.0 and Cisco Clean Access (CCA) 4.0.

Tests concentrated on the effort necessary – in terms of number of steps required – to deploy and maintain the NAC solutions under test, and the potential impact of each step on the existing network infrastructure. Tolly Group engineers audited the process of deploying the NAC solution in a representative network, and also documented the effort involved in performing routine maintenance of each NAC solution. Testing was conducted in November 2006.

Security
Enterasys Networks Tolly Group White Paper Series Sponsored by Enterasys: Advanced Security Applications to Secure Any Network

Document number: 207187
Release Date: 18 Jan 2007

Enterasys commissioned The Tolly Group to evaluate Dragon Security Command Console (DSCC), which combines network behavior analysis, security event correlation, vulnerability profiles and directed remediation in a single platform. Engineers evaluated DSCC against the Cisco Security Monitoring, Analysis and Response System (Cisco Security MARS).

Tolly Group engineers subjected both solutions to a battery of tests in the following areas: Accuracy of threat detection, correlation of security events, data reduction, auto discovery and classification of network assets, and multivendor support. Testing took place in October/November 2006.

Security
Q1 Labs Inc. Tolly Group White Paper Series Sponsored by Q1 Labs: Network Security Management for Self-Defending Networks

Document number: 206160
Release Date: 22 Nov 2006

Q1 Labs, Inc. commissioned The Tolly Group in October 2006 to conduct a comprehensive hands-on evaluation of QRadar, a Network Security Management (NSM) solution that combines network behavior analysis, security event correlation, vulnerability profiles and directed remediation in a single platform.

Tolly Group engineers evaluated the QRadar against the Cisco Security Monitoring, Analysis and Response System (Cisco Security MARS). Both solutions were subjected to a battery of tests in the following areas: Accuracy of threat detection, correlation of security events, data reduction, auto discovery and classification of network assets, and multivendor support.

Tests demonstrated that QRadar accurately detected more known attacks, as well as network anomalies, than did Cisco Security MARS. In addition, tests also showed that QRadar is particularly effective when distilling voluminous event and traffic data into manageable and readily understood summary information, with priority events flagged. The report is informed reading for any company considering networking security management.

Security
Viking InterWorks Viking Interworks V 2-Gigabit VPN/Firewall Appliance Firewall/VPN Appliance Performance Evaluation

Document number: 206137
Release Date: 01 Jun 2006

Viking InterWorks, Inc. commissioned The Tolly Group to evaluate its Viking V 2-Gigabit VPN/Firewall Appliance, which employs an entirely new silicon architecture that makes it possible to protect data centers, perimeters and areas inside the LAN with minimal to no impact on throughput.

Tolly Group engineers conducted firewall and VPN throughput tests, measuring the zero-loss bidirectional performance of the Viking 2-Gigabit VPN/Firewall Appliance when subjected to a variety of packet sizes: 64, 128, 256, 512, 768, 1,024, 1,280, 1,518 bytes plus an Internet mix (IMIX) and Tolly IMIX. Tests were conducted during April 2006.
Security
Mirapoint, Inc. Tolly Group White Paper Series: Exploring Exchange-Compatible Appliance-Based Messaging

Document number: 206131
Release Date: 11 May 2006

Mirapoint, Inc. commissioned The Tolly Group to build a microcosm of an enterprise E-mail environment and validate specific capabilities of the vendor’s Message Server appliance related to its ability to provide a “drop in” replacement to Microsoft’s Exchange E-mail application.

This hands-on evaluation proved conclusively that Mirapoint’s Message Server appliance can provide a viable alternative to Microsoft’s Windows-based Exchange Server mail transport application providing full functionalitiy to Outlook clients via Mirapoint’s Outlook SynQ technology while offering platform and security advantages over Microsoft Exchange.

This 23-page white paper comes complete with comparison charts that assess the compatibility of Mirapoint’s management, maintenance, integration and E-mail/calendaring features/functions with Microsoft Outlook. Plus accompanying screen shots provide a window into the Mirapoint Message Server’s “look and feel.”

Security
IntruGuard Devices, Inc. IntruGuard Devices, Inc. IG2000 Rate-Based Intrusion Prevention System, Layer 2-4 DoS/DDoS Attack Mitigation and Performance Evaluation

Document number: 206129
Release Date: 03 May 2006

IntruGuard Devices, Inc. commissioned The Tolly Group to evaluate the performance of its IG2000 Rate-Based Intrusion Prevention System (RBIPS), a stateful security appliance designed to monitor bidirectional traffic, intercept DoS/DDoS floods and other anomalies such as state-anomalies, header anomalies, network scans, dark-address scans, and port scans, etc.

The Tolly Group validated the performance of the IG2000, as well as the appliance’s effectiveness at detecting and mitigating a variety of high-rate attacks at Layers 2, 3 and 4. Additionally, Tolly Group engineers measured the latency of the appliance and effectiveness under attack.

Security
Array Networks, Inc. Array Networks SPX5000 SSL VPN Gateway, SSL VPN Gateway Scalability, Performance and Feature Evaluation

Document number: 206118
Release Date: 01 May 2006

Array Networks commissioned The Tolly Group to evaluate its SPX5000, a purpose-built SSL VPN access gateway that the company says is designed to extend access to remote and local users while maintaining both high security and rapid application response times.

The Tolly Group conducted a number of scalability/performance tests, as well as exercise a number of key SSL VPN features to validate key device capabilities. Engineers examined the scalability of the device in terms of the number of concurrent users supported, the aggregate throughout achieved and latency introduced by the SPX5000 and the ability of the device to work in a variety of popular environments such as Linux, Mac, Windows, Windows Mobile PDA, and Palm-based PDA.

Security
The Tolly Group TollyEdge White Paper Series: Benchmarking Strategies for Wired Intrusion Prevention Systems (IPS)

Document number: 206115
Release Date: 27 Apr 2006

This comprehensive 26-page TollyEdge: Benchmarking Strategies for Wired Intrusion Prevention Systems white paper explains what to look for in an IPS — in terms of protection, performance, ease of use and reliability, quantifying measures and benchmarks where possible, to help you decide how much software or other tools are required to adequately protect your network.

The report provides unique perspectives from vendors such as iPolicy, Karalon, Nortel, SourceFire, and TopLayer. The report maps out the types of threats, how IPS tools are designed to deal with them, and how users can effectively benchmark IPS products to determine the best fit for enterprise security needs.

Security
Sana Security, Inc. Sana Security Primary Response SafeConnect, Malware Detection and Removal/CPU and Memory Utilization

Document number: 206125
Release Date: 25 Apr 2006

Sana Security, Inc. commissioned The Tolly Group to evaluate its Primary Response SafeConnect, a software program designed for Windows-based operating systems (2000, XP) to detect and remove malicious programs.

Primary Response SafeConnect detected and removed 100% — all 183 — of the malware programs that invaded the host system from visiting 4,280 suspicious Web sites. The test results conclusively demonstrate that the program successfully accomplished its main objective — identifying and removing programs showing signs of suspicious behavior, such as trojans, rootkits, Spyware, adware and more. Engineers also measured the CPU utilization and memory utilization of Primary Response SafeConnect during peak program usage.
Security
Citrix Systems, Inc. Citrix Access Gateway 4.2 with Advanced Access Control, SSL/VPN Performance Comparison versus Juniper Networks NetScreen SA-1000

Document number: 206123
Release Date: 18 Apr 2006

Citrix Systems, Inc. commissioned The Tolly Group to evaluate its Citrix Access Gateway, an SSL/VPN appliance that provides a secure, always-on, single point of access for applications and resources. The Tolly Group was asked to compare the Citrix Access Gateway performance against a Juniper Networks NetScreen SA-1000, a remote access security device serving small-to-medium-sized companies.

Tolly Group engineers performed several tests to compare each SSL/VPN appliance. The performance of running common business tasks remotely over Citrix Presentation Server 4, specifically file transfer, remote printing and PowerPoint presentations, was measured by computing the elapsed time (in seconds) and bytes sent. Lastly, Tolly Group engineers examined the effectiveness of endpoint security when a client is connected through the SSL/VPN appliance.
Security
Wiresoft Net, Inc. Tolly Group White Paper Series: Securing SMB Networks Without Breaking the Bank

Document number: 206113
Release Date: 07 Apr 2006

SMBs need a multipurpose security platform that provides complete security protection. The dilemma that SMBs face today is that the IT market abounds with supplier after supplier that offer point solutions for security.

Wiresoft Net, Inc. commissioned The Tolly Group to evaluate its Wiresoft Sentry Security Suite, a versatile platform that offers a variety of security services including transparent virus scanning, challenge response spam blocking, stateful packet firewalling, VPN services (PPTP and IPSec) and more.

Tolly Group engineers conducted a battery of performance tests on the Sentry Security Suite, such as its effectiveness at blocking spam traffic, the aggregate throughput delivered while operating as a firewall, the aggregate throughput delivered across a VPN connection and the aggregate throughput while scanning Web traffic for viruses. Engineers also evaluated a number of functions, such as set up, hardware reliability and failover protection.

Security
Fortress Technologies, Inc Fortress Technologies, Inc. Fortress Security Controller FC-X Encryption and Compression Performance Evaluation of Three Models (FC-1500, FC-500 and FC-250)

Document number: 206104
Release Date: 20 Mar 2006

Fortress Technologies commissioned The Tolly Group to evaluate the performance of the company’s flagship FC-X Security Controller. The Fortress FC-X is a high-performance security appliance with Gigabit Ethernet interfaces and is an integral part of the Fortress Security System designed to provide high performance, secure and reliable connectivity to wireless LANs and fixed wireless networks using Wi-MAX, free space optics, satellite, or other point-to-point links.

The Tolly Group conducted steady-state zero-loss (<0.001%) encryption and compression throughput and latency tests in two FC-X boxes for a variety of Ethernet frame sizes. The FC-X also was subjected to extensive performance tests designed to measure the scalability of the FC-X Security Controller in three selectable performance modules: FC-1500, FC-500 and FC-250.

Finally, The Tolly Group measured the zero-loss throughput in Mbps across the FC-X pair on an encrypted link for different data types (most-compressible traffic and least-compressible traffic) and various numbers of emulated clients for the three FC-X models. For the latency test, The Tolly Group measured the average Store-and-Forward latency for the least-compressible data type and a single client for the FC-1500 model only.

Security
Piolink, Inc PIOLINK Application Switch 4500, Layer 4/7 Load Balancer, Firewall Performance and Worm Attack Protection Evaluation

Document number: 206111
Release Date: 16 Mar 2006

PIOLINK, Inc. commissioned The Tolly Group and TTA to evaluate the PIOLINK Application Switch 4500 (PAS 4500) for Layer 4-7 firewall and load balancing performance, and the impact of the PIOLINK Security Manager (PSM) on the performance.

The PAS 4500 is an intelligent and secure application switch that provides Layer 4-7 load balancing and application/network security with PSM. The PSM is PIOLINK's unique security system and enhances security by offloading the deep packet inspection from the PAS 4500 without imposing noticeable delays on the other traffic in the same data path.

Engineers measured the throughput of the PAS 4500 as a firewall after configuring Layer 4 filters (based on service port number) and Layer 7 filters (based on application payload signature), respectively, for various numbers of filters and frame sizes. Engineers also measured the TCP connection rate of the PAS 4500 when configured as a Layer 4 load balancer, and again as a Layer 7 load balancer. Finally, engineers measured the capability of the PAS 4500 to process legitimate HTTP and UDP traffic while blocking the attack traffic.

Security
AirTight Networks, Inc. White Paper Sponsored by AirTight Networks: Evaluating Wireless IPS Systems

Document number: 206103
Release Date: 06 Feb 2006

This Tolly Group white paper, commissioned by AirTight Networks, Inc., focuses on the key issues users must consider when evaluating wireless intrusion prevention systems. For the report, The Tolly Group evaluated SpectraGuard Enterprise, a multi-faceted WIPS designed to protect enterprise network infrastructures from wireless attacks.

The Tolly Group assessed the capability of SpectraGuard Enterprise to detect and block a range of wireless threats — from dealing with rogue APs, to detection and prevention of access point (AP) MAC address spoofing, to detection and prevention of Denial of Service (DoS) attacks, and several others.

Tolly Group engineers measured the effectiveness of SpectraGuard Enterprise against two other products: AirMagnet Inc.’s AirMagnet Enterprise and Aruba Networks Aruba Mobility Controller. Tests were conducted at AirTight Networks facilities in Mountain View, CA during December 2005.
Security
Symantec Corporation Symantec Gateway Security Version 3.0, Firewall Performance and Security Capability Benchmark versus Cisco ASA 5520 and Juniper NetScreen-500

Document number: 206108
Release Date: 06 Feb 2006

Symantec Corp. commissioned The Tolly Group to evaluate its Symantec Gateway Security solution, a full-inspection firewall with integrated dynamic routing and VLAN support, intrusion prevention, anti-virus, anti-spam, URL and Dynamic Document Review- based (DDR) content filtering, VPN (IPSec and SSL), and intrusion detection.

Tolly Group engineers evaluated the capability of the Symantec Gateway Security (SGS) Version 3.0 software running on a Symantec Gateway Security 5660 to identify and to block network attacks common to enterprise networks. Engineers also examined the SGS capability to isolate and to block suspicious network and audit events, and as well as block common evasion techniques used to deceive security appliances and affect end users. Additionally, engineers measured the aggregate throughput delivered by the SGS 5660 while configured to scan all traffic for attacks, as well as the connection set-up rate and the maximum number of sustained connections supported. Finally, engineers evaluated the graphical user interface (Security Gateway Management Interface) of the SGS. Tests were conducted during September and November 2005.

Engineers benchmarked the SGS appliance against a Cisco Systems, Inc. Adaptive Security 5520 appliance and a Juniper Networks, Inc. NetScreen-500 integrated firewall/IPSec VPN security appliance.
Security
Reflex Security, Inc. "Talking Outside the Box": Podcast Interview with Reflex Security CTO Hezi Moore and Performance Evaluation of Reflex IPS-100 Appliance

Document number: 206101
Release Date: 31 Jan 2006

This 14-minute podcast focuses on intrusion prevention appliances. The podcast reviews the chief findings from a performance evaluation of Reflex Security Inc.’s IPS-100 intrusion prevention appliance.

In addition, Kevin Tolly, President/CEO/Founder of The Tolly Group interviews Reflex Security CTO Hezi Moore on the issues users face when deploying IPS appliances and achieving optimal performance.

Details of the test can be found in document 205136.

Click below to download the "podcast" MP3 audio file.
Security
Verso Technologies, Inc. (TeleMate.Net Software) Verso Technologies, Inc. NetSpective M-Class Content Filter, Content Filter Evaluation for Skype Traffic

Document number: 206105
Release Date: 24 Jan 2006

Verso Technologies, Inc., a global provider of carrier and enterprise solutions, commissioned The Tolly Group to evaluate the NetSpective® M-Class Content Filter, an intelligent signature-based filtering device that maximizes network performance and security by filtering unauthorized or potentially harmful traffic according to user profiles defined by an administrator.

Tolly Group engineers subjected the NetSpective appliance to a test that measures the device’s effectiveness at blocking Skype™ traffic during the login process without affecting other benign traffic or authorized VoIP traffic (i.e. Net2Phone) passing through a test network.

The Tolly Group also examined the capability of the NetSpective appliance to monitor a constant traffic flow in excess of 3 Gbps while still blocking Skype login processes and allowing other VoIP traffic (i.e. Net2Phone) to pass unencumbered.
Security
Future Systems, Inc. Future Systems, Inc. FSC2003 SoC (System on a Chip) in Future Systems RenoGate, Firewall and VPN Performance Evaluation

Document number: 205147
Release Date: 15 Jan 2006

Future Systems, Inc. commissioned The Tolly Group/TTA (Telecommunications Technology Association) to evaluate its FSC2003 System on Chip (SoC) residing in RenoGate, a Fast Ethernet firewall and VPN appliance.

TTA/TTG benchmarked the bidirectional steady-state zero-loss ( 0.001%) firewall and VPN throughput across two Fast Ethernet interfaces when RenoGate equipped with FSC2003 SoC operated as a firewall or a VPN gateway.

Security
Reflex Security, Inc. Reflex Security IPS100 Intrusion Prevention Appliance, Performance, Security and Usability Evaluation

Document number: 205136
Release Date: 20 Nov 2005

Reflex Security, Inc. commissioned The Tolly Group to test the Reflex IPS100 network intrusion prevention appliance. The Reflex IPS blocks a comprehensive range of malicious traffic, including HTTP attacks, Denial-of-Service attempts, scans, backdoor exploits, floods, viruses, and worms. The Tolly Group validated the performance of the Reflex IPS, as well as the appliance’s effectiveness at detecting and preventing a variety of attacks. The Tolly Group also evaluated the system’s reliability, reporting and ease of use.

Tolly Group engineers conducted a battery of performance tests, focusing on HTTP throughput across the Reflex IPS appliance under normal conditions, and when subjected to attack traffic generated by Blade Software IDS Informer. They also performed a security test to measure the number of IDS Informer attacks blocked by the Reflex IPS100 while handling HTTP traffic in the background, and tests were also conducted to verify that the Reflex IPS100 appliance could block E-mails infected with worms and virus.

Security
Nortel Nortel Secure Router 3120, Competitive Performance Evaluation versus Cisco Systems ISR 2821 and ISR 3825

Document number: 205146
Release Date: 01 Nov 2005

Nortel commissioned The Tolly Group to evaluate its Secure Router 3120, a wide-area network router with integrated network services such as Quality of Service, Network Address Translation and Access Control Lists.

Tolly Group engineers measured the multilink Point-to-Point Protocol (PPP) zero-loss throughput of the Nortel Secure Router 3120 with Quality of Service (QoS),Network Address Translation (NAT) and Access Control List (ACL) features enabled. Nortel also directed The Tolly Group to evaluate the Nortel 3120 versus a Cisco Systems 3825 Integrated Services Router and 2821 Integrated Services Router. Testing was performed in September/October 2005.

Test results show that the Nortel Secure Router 3120 delivers superior throughput for the majority of packet sizes tested, especially with regards to smaller packet sizes (64 bytes to 256 bytes), generally delivering up to 4X greater throughput than the Cisco devices tested.
Security
Check Point Software Technologies Ltd. SSL VPN Gateways: Delivering Superior ROI with Integrated Security - JAPANESE VERSION

Document number: 205132JP
Release Date: 12 Oct 2005

Check Point commissioned The Tolly Group to validate the security and functionality claims the company has made concerning the Connectra SSL/VPN appliance. In total, 13 tests were conducted in August 2005 at a Check Point lab in Tel Aviv, Israel, and validated on-site by Tolly Group personnel.

Check Point’s Connectra NGX was tested against three other SSL VPN products: F5 Networks, Inc.’s FirePass 1000, Cisco System Inc.’s VPN Concentrator 3005 and Juniper Networks, Inc. NetScreen-SA 1000. Tests show that Check Point’s Connectra NGX offers much greater depth of protection over SSL VPN links than any of the other three products. Connectra was the only product to pass all of the tests.

Tests results underscore a basic philosohical difference in the architectures of the tested products. Check Point integrates endpoint security with extensive gateway-based security facilities that focus on protecting the network transport, guard against application attacks and protect backend Web servers and applications from network-borne threats. Other products tested provide just basic SSL VPN connectivity.

Japanese translation of Tolly Group document 205132. If there are any issues between the two documents, the English language document takes precedent.
Security
Check Point Software Technologies Ltd. SSL VPN Gateways: Delivering Superior ROI with Integrated Security

Document number: 205132
Release Date: 12 Oct 2005

Check Point commissioned The Tolly Group to validate the security and functionality claims the company has made concerning the Connectra SSL/VPN appliance. In total, 13 tests were conducted in August 2005 at a Check Point lab in Tel Aviv, Israel, and validated on-site by Tolly Group personnel.

Check Point’s Connectra NGX was tested against three other SSL VPN products: F5 Networks, Inc.’s FirePass 1000, Cisco System Inc.’s VPN Concentrator 3005 and Juniper Networks, Inc. NetScreen-SA 1000. Tests show that Check Point’s Connectra NGX offers much greater depth of protection over SSL VPN links than any of the other three products. Connectra was the only product to pass all of the tests.

Tests results underscore a basic philosohical difference in the architectures of the tested products. Check Point integrates endpoint security with extensive gateway-based security facilities that focus on protecting the network transport, guard against application attacks and protect backend Web servers and applications from network-borne threats. Other products tested provide just basic SSL VPN connectivity.
Security
NETASQ NETASQ F2000 IPS-Firewall Multiservice Security Appliance Performance Evaluation

Document number: 205120
Release Date: 10 Jul 2005

NETASQ commissioned The Tolly Group to evaluate the NETASQ F2000 IPS-Firewall, a purpose-built network security appliance that combines real-time intrusion prevention, firewall service, IPSec virtual private networking (VPN), clientless SSL VPNs, advanced content filtering, anti-spam, anti-virus and other integrated security services.

Tolly Group engineers focused testing on the performance of the NETASQ F2000 using a mostly default configuration, measuring the device’s zero-loss throughput (while IPS services were active), benchmarking latency introduced by the device under varying traffic loads and conditions. (In its default state, the NETASQ F2000 enables protocol analysis and signature and port-scan detection, among other IPS capabilities.) Tests were conducted at The Tolly Group’s Boca Raton, FL. facilities in May 2005.
Security
Symantec Corporation Symantec Network Security 7160 Intrusion Prevention Appliance Performance Evaluation

Document number: 205111
Release Date: 23 Jun 2005

Symantec Corp. commissioned The Tolly Group to evaluate its Symantec Network Security 7160, an eight-port Gigabit Ethernet security appliance that offers intrusion protection while delivering throughput in excess of 1 Gbps.

Tolly Group engineers evaluated the capability of the Symantec Network Security 7160 to detect and block network attacks and threats common to enterprise networks. Engineers also examined the Symantec Network Security 7160’s capability to detect and block suspicious network threats and security risks /audit events. Next, engineers tested the ability for the Symantec Network Security 7160 to continue to block threats when common and advanced evasion techniques were used to deceive the security appliance and affect end users. Additionally, engineers measured the aggregate throughput delivered by the Symantec Network Security 7160 while configured to scan all traffic for attacks, as well as the connection set-up rate and the maximum number of sustained connections supported. Finally, engineers evaluated the management capabilities and ease of use for the Symantec Network Security 7160. Tests were conducted in March 2005.

Security
Radware Ltd. White paper: Measuring Key Criteria of Intrusion Prevention Systems

Document number: 205114
Release Date: 20 Jun 2005

Radware, Inc. commissioned The Tolly Group to evaluate its DefensePro 3000, an intrusion prevention switch with DoS protection that combines bandwidth management for attack isolation and traffic shaping to offer enterprise and carrier networks protection against a diverse range of network- and application-level attacks.

The aim of the testing was to evaluate the DefensePro 3000 to determine that it delivers the advanced IPS requirements users need to combat today’s sophisticated security threats.

Tolly Group engineers examined the performance of the DefensePro 3000 in various scenarios to understand the maximum throughput offered by the switch while it actively handled various attacks and processed signature loads. Engineers also evaluated the manner in which the DefensePro 3000 was able to detect and block attacks that utilized common evasion techniques and engineers examined a facility that restricts bandwidth to background applications that otherwise could interfere with strategic application traffic. Finally, The Tolly Group examined a capability of the DefensePro 3000 to implement protection options for different segments of supported networks. Tests were conducted in April 2005 at Radware facilities in Israel.

Also see document 205112.
Security
Nortel Nortel VPN Gateway 3070 SSL VPN Throughput, Scalability and Voice Quality Benchmark Evaluation

Document number: 205113
Release Date: 02 Jun 2005

Nortel commissioned The Tolly Group to evaluate the Nortel VPN Gateway 3070, an enterprise-class device that combines support for IPSec and SSL VPNs into a single device.

Tolly Group engineers benchmarked the throughput of the Nortel VPN Gateway 3070, examined the limits of the device’s scalability and identified the extent of voice quality supported by the gateway. Testing was performed at The Tolly Group's Boca Raton, Fl., facilities in March 2005.

Tests show that the Nortel VPN Gateway 3070 delivers consistent throughput when tested across workgroups scaling from 10 users, to 5,000 users. Scalability tests show that even as the number of users scales to 5,000, the device does not sacrifice significant throughput. Finally, the Nortel VPN Gateway 3070 yielded toll-quality voice during tests focusing on voice quality.
Security
Radware Ltd. Radware, Inc. DefensePro 3000 Throughput Benchmark and Attack Mitigation Evaluation

Document number: 205112
Release Date: 31 May 2005

Radware, Inc. commissioned The Tolly Group to evaluate its DefensePro 3000, an intrusion prevention switch with Denial of Service (DoS) protection that combines bandwidth management for attack isolation and traffic shaping to offer enterprise and carrier networks protection against a diverse range of network- and application-level attacks.

Tests show that the DefensePro 3000 is adept at identifying and blocking attacks with zero instances of false positives while simultaneously handling multi-Gigabit traffic loads. From a performance standpoint, the DefensePro 3000 was able to handle 2.5 Gbps of “real-world” throughput while simultaneously handling either a 40,000-packet per second (pps) worm attack, a 200-Mbps SYN Flood attack or a 100-Mbps DoS attack. Tests also show the DefensePro 3000 is capable of protecting Secure Sockets Layer (SSL) data and can isolate attacks to protect mass mailings and control P2P traffic.

Security
Nortel White Paper – Building a World-Class VPN Solution to Meet Today’s Needs — and Tomorrow’s

Document number: 205103
Release Date: 09 May 2005

This Nortel-focused white paper examines the market trends shaping the adoption of Secure Socket layer (SSL) VPNs, and specifically how Nortel is approaching the market with its VPN Gateway 3070.

Nortel commissioned The Tolly Group in January 2005 to build a hypothetical enterprise environment in which to implement and benchmark the Nortel VPN Gateway 3070, which delivers integrated support for IPSec and SSL VPNs. The goal of this paper is to educate readers about the types of SSL VPN functionality that will be critical in the selection of products and migration to the new technology. Readers will learn the primary functions that should be available in an SSL VPN gateway.

Tolly Group engineers certified the feature/functionality of a series of strategic SSL VPN capabilities in the Nortel VPN gateway 3070, including: basic VPN access, privileged-based access, enhanced clientless access, portal access, scalability and service partitioning.

The Tolly Group also performed SSL VPN throughput tests on the VPN Gateway 3070, subjecting it to tests with either the RC4-MD5 encryption or the much more complex DES-CBC3-SHA (Triple DES) encryption. Further, engineers examined the ability of the VPN Gateway 3070 to handle voice/data convergence by measuring the quality of voice over IP (VoIP) calls placed across the gateway.

Tolly-Nortel SSL VPN Study - View the webcast
Security
Internet Security Systems, Inc. (IBM) Internet Security Systems Proventia Intrusion Prevention Appliance G2000 Throughput, Latency and Failover Performance Evaluation

Document number: 205110
Release Date: 12 Apr 2005

Internet Security Systems Inc. commissioned The Tolly Group to evaluate the performance of its Proventia Intrusion Prevention Appliance G2000, a security appliance designed to monitor all inbound traffic, intercept attack traffic and other security threats and block the attacking stream so it does not reach intended targets. The Proventia G2000 is an eight-port device capable of supporting four monitoring segments with two ports dedicated to each segment. The appliance came with two 10/100/1000 Ethernet management ports and has an advertised throughput of 2 Gbps.

Tolly Group engineers measured the zero-loss Layer 2 bidirectional throughput of the Proventia G2000, as well as the latency of the appliance. In addition, engineers measured the TCP performance in terms of the sustained new connection rate and the maximum simultaneous connections supported. Lastly, Tolly Group engineers evaluated the effectiveness of the Proventia G2000 at thwarting certain Denial of Service (DoS) attacks that otherwise could impact network performance adversely. Tests were conducted in March 2005.

Tests show that the Proventia G2000 is able to deliver network performance on par with typical network switching devices, meaning its presence will not degrade network throughput when it is deployed in an enterprise network. Tests show the Proventia G2000 delivers between 2 Gbps and 5 Gbps of Layer 2, bidirectional, zero-loss throughput (depending on frame size), along with low latency. Moreover the appliance processes TCP connections at rates that are necessary to support scalable enterprise applications and also help repel DoS attacks. Finally, tests show the Proventia G2000 repels DoS, Nimda and Blaster attacks on one network segment without compromising the throughput rate of normal application traffic traversing a second network segment. In essence, good traffic on one network segment remains unaffected by attack traffic that is isolated on a second network segment.

Security
SECUi.COM Corp. SECUi.COM NXG 2000 Evaluation of Gigabit Ethernet Firewall & VPN Performance

Document number: 205102
Release Date: 27 Feb 2005

SECUi.COM Ltd. commissioned The Tolly Group through its South Korean affiliate TTA (Telecommunications Technology Association) to test its NXG 2000, a Gigabit Ethernet firewall & VPN appliance.

TTA/TTG benchmarked the maximum TCP session rate when the NXG 2000 is operated in firewall mode only. The Spirent Communications SmartBits SMB-6000 (Testing S/W: Websuite/ Firewall) was used to establish the TCP connections and to measure the maximum session rate.

TTA/TTG also benchmarked the bi-directional steady-state zero-loss (0.1%) UDP throughput under multiple rules and UDP sessions when the NXG 2000 was operated in firewall or VPN mode respectively. For VPN throughput testing, engineers utilized a variety of frame sizes (64, 128, 256, 512, 1,024, and 1,400 bytes) generated using the SmartBits SMB-6000 (Testing S/W: SmartFlow) equipped with two Gigabit Ethernet interfaces. The VPN test was run three times and the final result was an average of the three test iterations. For the firewall throughput testing, engineers utilized frame sizes of 64, 128, 256 bytes; all frames were generated using same Spirent Communications SmartBits SMB-6000 that was utilized in the VPN throughput test.

Tests show that the NXG 2000 processes up to 83,400 TCP sessions/sec in Layer 2 firewall mode, up to 56,200 TCP sessions/sec in Layer 3 firewall mode. Moreover, the NXG 2000 forwards up to 1,923 Mbps and 1,907 Mbps of bi-directional, zero-loss throughput for a single VPN tunnel and 5,000 VPN tunnels respectively when tested with 1,400-byte frames.
Security
Check Point Software Technologies Ltd. White Paper: Improving Security ROI via an Integrated Application Security Solution

Document number: 205101
Release Date: 23 Feb 2005

Check Point Software Technologies, Inc. commissioned The Tolly Group to conduct a series of tests that demonstrate the effectiveness of the company's Application Intelligence within the Check Point VPN-1 NG Series firewall compared to other offerings and how they handle threatening security exploits. Check Point believes its Check Point VPN-1 NG Series firewall is the only perimeter security gateway to provide protection for the entire perimeter environment -- without requiring the purchase and deployment of a second standalone "intrusion protection" device.

Engineers tested the security attributes of Check Point VPN-1 NG firewall against a Cisco PIX 515E and a Juniper Networks NetScreen-204. The Check Point, Cisco and Juniper security solutions went through 17 rigorous tests that exposed them to a variety of common application-level exploits including SSL, SQL and HTTP-based vulnerabilities. Tests demonstrated that while Cisco’s and Juniper’s solutions are response-based, meaning that they rely on pre-defined signatures to defeat attacks, Check Point’s solution is proactive, protecting the network against attacks before they even occur.

Tests show that the Check Point VPN-1 NG Gateway offers greater depth of protection in comparison to Cisco and Juniper products tested, and also provides application-level security for a greater number of protocols including SQL, HTTP, HTTPS, SOCKS, IPSec, BGP, OSPF, and RIP. Moreover, the Check Point gateway offers integrated IPS, firewall and VPN capabilities in a single device, unlike the rival products that steer users to a companion security device. Finally, the Check Point VPN-1 NG Gateway offers a significant total cost-of-ownership advantage.

Please note that this is a large document - close to 2MB. A shorter version of this document is available, without the 30+ page appendix, from the Check Point web site.
Security
Nokia Nokia Enterprise Solutions Nokia IP2250 Competitive Performance Evaluation versus Juniper Networks NetScreen-5400 - JAPANESE VERSION

Document number: 204150JP
Release Date: 04 Dec 2004

Nokia Enterprise Solutions commissioned The Tolly Group to evaluate its Nokia IP2250 Security Platform, a diskless network processor-based platform that runs the Check Point VPN-1 NG with Application Intelligence firewall. The Nokia IP2250 was tested against a Juniper Networks NetScreen-5400.

Both devices were tested in a number of areas, including firewall throughput using various packet sizes ranging from the taxing 64-byte packets to 1,518-byte packets; VPN throughput, as well as per-second connection and session rates.

Test results show that the Nokia IP2250 outperformed the NetScreen-5400 in every test, offering up to twice the firewall throughput, more than three times the mixed traffic throughput and nearly five times the session rate of the NetScreen-5400. Tests were conducted from August to October 2004.

Click the link below.
Security
Nokia Nokia Enterprise Solutions Nokia IP2250 Competitive Performance Evaluation versus Juniper Networks NetScreen-5400

Document number: 204150
Release Date: 04 Dec 2004

Nokia Enterprise Solutions commissioned The Tolly Group to evaluate its Nokia IP2250 Security Platform, a diskless network processor-based platform that runs the Check Point VPN-1 NG with Application Intelligence firewall. The Nokia IP2250 was tested against a Juniper Networks NetScreen-5400.

Both devices were tested in a number of areas, including firewall throughput using various packet sizes ranging from the taxing 64-byte packets to 1,518-byte packets; VPN throughput, as well as per-second connection and session rates.

Test results show that the Nokia IP2250 outperformed the NetScreen-5400 in every test, offering up to twice the firewall throughput, more than three times the mixed traffic throughput and nearly five times the session rate of the NetScreen-5400. Tests were conducted from August to October 2004.

Click the link below.
Security
Inkra Networks Corp. Inkra Networks Inkra 1504GX and Inkra 1518TX Virtual Service Switch Fast Ethernet/Gigabit Ethernet Security Performance Evaluation

Document number: 204130
Release Date: 03 Dec 2004

Inkra Networks commissioned The Tolly Group to evaluate the performance of the Inkra 1504GX and 1518TX Virtual Service Switches (VSS). The Inkra 1500 Series switches offer an all-in-one real-time security device that offers firewall, intrusion prevention, VPN, SSL and load-balancing services in a single platform.

The Inkra 1504GX is designed for backbones and high-speed applications and supports up to 4 Gbps of aggregate throughput across four GBIC interfaces. The Inkra 1518TX combines high-density Fast Ethernet and Gigabit Ethernet (GbE) uplinks to support multiple server farms and high-speed applications. The Inkra 1518TX comes equipped with up to 16 Fast Ethernet ports and two Gigabit Ethernet ports.

Tolly Group engineers measured the bidirectional zero-loss firewall throughput rates, plus the Layer 7 transaction processing throughput rate of both devices. Tests were conducted in May 2004.
Security
Top Layer Networks Top Layer Networks Attack Mitigator IPS 5500 IPS Evaluation versus TippingPoint UnityOne-2400

Document number: 204146
Release Date: 02 Dec 2004

Top Layer Networks, Inc. commissioned The Tolly Group to evaluate its Attack Mitigator IPS 5500, an intrusion prevention system designed to stop network-based threats while allowing legitimate transactions to complete.

The Tolly Group evaluated the effectiveness of the Attack Mitigator IPS 5500 at dealing with single-protocol and mixed-protocol Distributed Denial-of-Service (DDoS) SYN flood attacks. Moreover, engineers examined the capability of the Attack Mitigator IPS 5500 to handle real-time identification and blocking of embedded worms when mixed in otherwise normal traffic. Tests focused on the capability to filter such traffic while monitoring what, if any, degradation that screening caused to the IPS’ connection rate.

Engineers conducted these tests on the Attack Mitigator IPS 5500 and compared the results to a TippingPoint Technologies Inc. UnityOne-2400 IPS. In every test case the Top Layer IPS5500 outperformed the UnityOne-2400. Tests show the IPS5500 is up to 82.5% more effective at blocking embedded worm attacks than the UnityOne-2400 during tests with worms infecting from 10% to 50% of traffic. Tests also demonstrate that the IPS5500 completes 100% of HTTP and mixed protocol connections attempted while under SYN flood attack. Tests were conducted in October 2004.
Security
Aruba, an HPE company Aruba Wireless Networks’ Aruba 5000 WLAN Security System Competitive Security Evaluation

Document number: 204144
Release Date: 22 Nov 2004

Aruba Wireless Networks commissioned The Tolly Group to evaluate its Aruba 5000 WLAN switch that combines 10/100/1000 Mbps Ethernet switching with stateful LAN-speed firewalling, VPN concentrator features and a variety of wireless security services.

Tolly Group engineers exposed the Aruba 5000 switch to three security scenarios to determine its effectiveness at securing communications between the switch and various access points (APs). The Aruba 5000 was subjected to a secure voice test, a variant of the “man-in-the-middle” attack, and wireless intrusion prevention scenario.

Aruba asked The Tolly Group to compare the security features/functions of the Aruba 5000 against the Airespace 4012 WLAN switch. Airespace threatened legal action if The Tolly Group tested its product without explicit permission which it ultimately declined to give. The Tolly Group did not test the Airespace 4012 but instead relied upon publicly available information and on-the-record Airespace comments to make comparisons between the products.

In every test instance, the Aruba 5000 demonstrated its security effectiveness. The Tolly Group also validated more than 20 key functions on the Aruba 5000 WLAN switch under its Tolly Verified certification program. Click the link at the bottom of the page to download the Test Summary document. Click here to download the document containing Airespace's official statement
Security
iPolicy Networks iPolicy Networks-6420 Competitive Performance Evaluation versus Fortinet FortiGate-3600 Anti-Virus Firewall

Document number: 204138
Release Date: 14 Oct 2004

iPolicy Networks commissioned The Tolly Group to evaluate and compare its iPolicy-6420 Intrusion Prevention Firewall with Fortinet’s FortiGate-3600 anti-virus firewall. Both devices are multifunction security appliances designed to protect data networks from a wide range of security threats with no compromise to network performance. Both products are marketed to service providers and large enterprises.

Tests focused on three main functional areas: UDP frame loss, TCP/UDP per-second connection rate, and the ability to establish new connections per second when the device already has a large number of connections active. Tests were conducted during August 2004.

The iPolicy-6420 consistently delivered high performance in every test scenario, even with a range of security applications running. That was not the case with the FortiGate-3600; it demonstrated significantly high frame loss, significantly lower TCP/UDP connection rates, and was barely able to complete any transactions successfully under such test scenarios.
Security
Intel Corporation Intel Corp. IXP425 Network Processors, Performance Analysis of VPN Devices

Document number: 204132
Release Date: 23 Jul 2004

Intel Corp. commissioned The Tolly Group to benchmark the performance of several commercially available VPN gateways that utilize the Intel® IXP425 network processor, and compare the performance of those devices against other, generally available products based upon alternative chipsets.

The Tolly Group conducted performance tests to validate the Layer 2 throughput of various VPN devices focusing on the CPU/NPU performances using Spirent’s SmartFlow and TeraVPN applications running on a SmartBits 6000B. As expected, the two SmartBits applications reported slightly different throughput results, but they both produced very consistent and similar performance trends for all the devices under test. The best performers in the SmartFlow test were also the best ones in the TeraVPN test. This also proves the integrity and validity of the test.

Among the devices tested, the Intel IXP425-based products outperformed the alternative chipset-based devices (Broadcom BCM4702, Motorola MPC860P, etc) in every scenario.
Security
Astaro Corp. Measuring the Value of Integrated Perimeter Security, A White Paper Commissioned by Astaro Corp.

Document number: 204128
Release Date: 01 Jul 2004

This hands-on study measures the time to deploy and manage an integated security solution versus two “best-of-breed” alternatives.

Engineers compared Astaro Security Linux versus two solution sets, one anchored by Juniper Networks (formerly NetScreen Technologies Inc.) firewall/VPN products, and the other anchored by Check Point Software Technologies, Inc. firewall offerings. Both solutions utilized anti-virus and anti-spam software from Trend Micro Devices, Inc. and URL/content filtering software from Websense, Inc.

These test exercises were intended to compare the effort and complexity required to deploy and to manage a comprehensive perimeter security solution for a typical medium-sized business for a period of 12 months.

The results were quite dramatic. The “best-of-breed” combinations took more than 3X as long to deploy and to configure. On an ongoing basis, the best-of-breed solutions required 2X to 2.5X more effort to manage than the integrated solution.

Click the link at the bottom of the page for the English version. A German version is also available. Click here to download the German version of the complete White Paper

Security
Aventail Corp. Aventail Competitive SSL VPN Feature Analysis Versus F5 Networks FirePass 1000 and Juniper Networks NetScreen-SA EA150 - JAPANESE VERSION

Document number: 204133JP
Release Date: 25 Jun 2004

Aventail Corp. commissioned The Tolly Group to evaluate its Aventail® EX-1500, an SSL VPN appliance that provides users with clientless access from any PC with Internet access to the network applications and resources they need to be productive. The Tolly Group examined the feature/functionality of the EX-1500 versus F5 Networks FirePass 1000 and Juniper Networks NetScreen EA-150, both SSL VPN appliances.

Tolly Group engineers compared the feature/functionality of the three SSL VPN products in three categories: Access control policy, end-point security and system security. All three products were subjected to a battery of feature validation tests as defined under The Tolly Group’s Tolly Verified certification program. Engineers examined the products during May and June 2004.

Tolly Group engineers found that the Aventail EX-1500 offers the most robust set of features/functions of the three products. On the access control side, the EX-1500 supported all nine of the features engineers examined, and even when other vendors matched the feature availability, the EX-1500 offered more robust functionality that makes management simpler

Japanese translation of Tolly Group document 204133. If there are any issues between the two documents, the English language document takes precedent.
Security
Aventail Corp. Aventail Competitive SSL VPN Feature Analysis Versus F5 Networks FirePass 1000 and Juniper Networks NetScreen-SA EA150

Document number: 204133
Release Date: 25 Jun 2004

Aventail Corp. commissioned The Tolly Group to evaluate its Aventail® EX-1500, an SSL VPN appliance that provides users with clientless access from any PC with Internet access to the network applications and resources they need to be productive. The Tolly Group examined the feature/functionality of the EX-1500 versus F5 Networks FirePass 1000 and Juniper Networks NetScreen EA-150, both SSL VPN appliances.

Tolly Group engineers compared the feature/functionality of the three SSL VPN products in three categories: Access control policy, end-point security and system security. All three products were subjected to a battery of feature validation tests as defined under The Tolly Group’s Tolly Verified certification program. Engineers examined the products during May and June 2004.

Tolly Group engineers found that the Aventail EX-1500 offers the most robust set of features/functions of the three products. On the access control side, the EX-1500 supported all nine of the features engineers examined, and even when other vendors matched the feature availability, the EX-1500 offered more robust functionality that makes management simpler.
Security
14 South Networks 14 South Security Appliance Card Performance Functionality Evaluation

Document number: 204119
Release Date: 26 Mar 2004

14 South Networks commissioned The Tolly Group to evaluate its Security Appliance Card™ and determine that the device can operate within a host server without any undue impact on the host performance or resident applications. For this test, the Security Appliance Card was running Check Point Software’s VPN-1/FireWall-1 Next Generation with Application Intelligence security software.

Testing illustrated that 14 South’s SAC, while physically resident in the test server, functions independently from that server. Similarly, tests showed that the presence of the SAC had “zero impact” on the operation and performance of the test server.

Tests also identified the Gigabit Ethernet Firewall and Fast Ethernet VPN performance of the SAC when tested at various frame sizes.

Security
Analog Devices, Inc. Fusiv-Vx200 Competitive Routing, Firewall and VPN Performance Analysis

Document number: 204117
Release Date: 19 Mar 2004

Analog Devices, Inc. (ADI) commissioned The Tolly Group to benchmark the performance of the ADI Fusiv-Vx200 network processor, implemented in a reference platform, and compare that performance with commercially available products based upon rival chipsets. The ADI Fusiv-Vx200 was tested against a Linksys WRV54G broadband access router with an embedded Intel Corp. IXP425 network processor, a NetScreen Technologies, Inc. NetScreen-204, and a Texas Instrument’s TNETV2020 as implemented by Nippon Telegraph & Telephone East Corp.’s WebCaster V100.

Tolly Group engineers used the industry-accepted Spirent SmartFlow test suite to gauge the Fast Ethernet firewall and VPN throughput of the devices. Testing was performed in February 2004.

Test results show that the ADI Fusiv-Vx200 exceeds or matches the throughput of the other devices tested in both firewall and “mixed” VPN (i.e., ADI communicating to the other vendor’s device) throughput tests. Tests also illustrated that the ADI Fusiv-Vx200 can continue processing traffic when the main CPU is taken offline.

The full report is available via the link at the bottom of the page. Click here to download a one page summary document
Security
Inkra Networks Corp. Inkra Networks 4000 Virtual Service Switch Multi-Gigabit, Multi-Services Switch Functionality Evaluation

Document number: 204107
Release Date: 11 Feb 2004

Inkra Networks commissioned The Tolly Group to evaluate its Inkra 4000 Virtual Service Switch (VSS), a 14-slot chassis-based device that supports multiple virtual services, including firewall, VPN, intrusion detection, load balancing and SSL processing. Inkra Networks asked The Tolly Group to validate the single-rule firewall throughput of the Inkra 4000 VSS, as well as validate an array of other supported services.

Engineers configured the Inkra 4000 VSS chassis as a single firewall with one rule, using a 1,518-byte packet and 28 Gigabit Ethernet port pairs. Engineers established 14,336 sessions and flowed bidirectional Layer 3 data across the Inkra 4000 VSS and measured the zero-loss firewall throughput. Engineers also measured device latency and VPN throughput. Testing was performed in September 2003.

The Tolly Group also benchmarked a number of Inkra 4000 VSS device functions including server load balancing, SSL connection rate, and intrusion detection/intrusion prevention processing.
Security
Inkra Networks Corp. Inkra 4000 Virtual Service Switch Multi-Gigabit Ethernet Firewall Throughput and Scalability Evaluation

Document number: 203127
Release Date: 03 Nov 2003

Inkra Networks commissioned The Tolly Group to evaluate its Inkra 4000 Virtual Service Switch (VSS), a 14-slot chassis-based device that supports multiple virtual services, including firewall, VPN, intrusion detection, load balancing and SSL processing. Inkra Networks asked The Tolly Group to validate the single-rule firewall throughput, scalability and latency of the Inkra 4000 VSS. Testing was performed in September 2003.

Tolly Group engineers tested the Inkra 4000 VSS aggregate zero-loss (<0.001%) throughput when using 1,518-byte frames. Tests were also conducted using three different configurations in order to demonstrate the linear slope of the product’s scalability. Finally, Tolly Group engineers measured the latency introduced by the Inkra 4000 VSS as data traverses the switch backplane.
Security
Aruba, an HPE company Aruba Wireless Networks Aruba 5000 Wi-Fi Switch/Aruba 52 Access Point - Encrypted Wireless Performance

Document number: 203126
Release Date: 23 Oct 2003

Aruba Wireless Networks commissioned The Tolly Group to evaluate its Aruba 5000 Wireless LAN Switch and its Aruba 52 access point. The Aruba 5000 is a four-slot WLAN switch with an integrated VPN gateway that operates at 10/100/1000 Mbps and combines firewall and VPN functionality along with wireless service. The Aruba 52 access point is a dual-band AP that supports 802.11a (54 Mbps) and 802.11b (11 Mbps).

Engineers conducted an IPSec throughput test designed to measure the aggregate internal performance of the Aruba 5000 switch. A second test measured the real-world throughput of the Aruba 5000 with an integrated VPN gateway supporting a large-scale wireless client/server network. Lastly, Tolly Group engineers validated 20 different features of the Aruba 5000 WLAN switch and the Aruba 52 AP. All tests were performed in September 2003.
Security
Ingate Systems, AB Ingate Firewall™ 1400, Functionality and Performance Evaluation

Document number: 203118
Release Date: 19 Sep 2003

Ingate Systems commissioned The Tolly Group to conduct a performance evaluation of its Ingate Firewall™ 1400, a firewall designed for use in small- to medium-sized enterprises (SMEs) that supports standard functions plus incorporates a SIP proxy and a SIP registrar to handle SIP-based real-time communications such as VoIP, instant messaging and audio/videoconferencing.

Engineers subjected the Ingate Firewall 1400 to a volley of tests, including throughput benchmarks, an evaluation of the device’s SIP functionality, VPN performance benchmark, and voice call quality capabilities.

Security
Ingate Systems, AB Ingate Systems SIParator™ 40 Functionality Evaluation

Document number: 203119
Release Date: 19 Sep 2003

Ingate Systems commissioned The Tolly Group to conduct a functionality evaluation of its SIParator™ 40, an adjunct processor that works in conjunction with installed firewalls and enables the traversal of real-time SIP-based communications without compromising existing firewall security.

Tolly Group engineers subjected the SIParator 40 to 26 different SIP test scenarios representing common VoIP and instant messaging functions in each of three different deployment configurations: DMZ mode, DMZ/LAN mode and Standalone mode.

Security
Forum Systems, Inc. Forum Sentry™ 1504 XML Security Appliance Web Services Security Evaluation

Document number: 203117
Release Date: 17 Sep 2003

Forum Systems, Inc. commissioned The Tolly Group to evaluate its Forum Sentry™ 1504, a Web services security appliance that utilizes specially designed hardware to provide an enterpirse-class Web services security management and acceleration platform.

The Forum Sentry 1504 integrates three XML Web services security solutions in a single device.
Security
SECUi.COM Corp. SECUi.COM Corp. secuiWALL versus NetScreen Technologies, Inc. NetScreen-500 and NetScreen-1000 Gigabit Ethernet Firewalls Competitive Performance Evaluation

Document number: 203112
Release Date: 06 Jul 2003

Recent tests of three Gigabit Ethernet firewalls reveal a product from SECUi.COM Corp. outperforms a pair of NetScreen Technologies GbE firewalls in zero-loss throughput tests under variable session loads, packet sizes and operating modes. The secuiWALL GbE firewall delivered 15% more bi-directional zero-loss throughput than a NetScreen-1000 when handling 1,518-byte packets with 5,000 UDP sessions and 2,000 active rules. A NetScreen-500 tested was only able to achieve 40% of wire-speed throughput for all conditions tested..

SECUi.COM commissioned the Telecommunication Technology Association (TTA), a Korean-based testing partner of The Tolly Group working in conjunction with The Tolly Group, to evaluate the three GbE firewalls. All products were tested according to the principles, methodologies and fair testing charter employed by The Tolly Group. While testing was conducted in Korea, The Tolly Group reviewed all testing methodologies and results.

TTA exercised all three GbE firewalls in a variety of situations including bi-directional zero-loss Layer 2 throughput tests and bi-directional Layer 3 throughput tests with NAT enabled and then disabled. Test conditions included a variety of packet sizes ranging from 64 bytes to 1,518 bytes with 10 or 5,000 simultaneous UDP sessions with either a single active allow-all rule, or 2,000 active firewall rules.

Test results show that the secuiWALL GbE firewall achieved wire-speed GbE throughput when handling up to 5,000 UDP sessions and 2,000 active rules. The NetScreen-1000 achieved wire speed when handling 1,028-byte packets and handling 10 UDP sessions while the NetScreen-50 only reached 40% of wire speed under all test conditions.

Security
SmartPipes, Inc. The Power of Policy: Streamlining VPN Security through Software: A “hands-on” implementation of SmartPipes IP PolicyPro SecureSite Software

Document number: 203503
Release Date: 29 Apr 2003

The prevalence of IP VPNs to provide connectivity in enterprise networks raises serious issues for network administrators to confront in their deployment and life-cycle maintenance. In the past, network managers were forced to grapple with complex configuration issues manually on an individual-device basis. The time spent to set up even a single VPN tunnel manually is notorious in the industry. Magnify that across a network for a large-scale deployment and you get a hefty chunk of man-hours, which translates into cost.

SmartPipes, Inc. developed its IP PolicyPro platform and SecureSite software to provide network administrators with an easy-to-use, scalable and secure method for simultaneously provisioning hundreds, and even thousands, of VPNs in today’s global network.

In order to provide intellectual and pragmatic analysis to back up its product claims, SmartPipes commissioned The Tolly Group to build a microcosm of a complex VPN deployment with which to illustrate the key benefits of the SmartPipes SecureSite platform.

The Tolly Group’s evaluation of SmartPipes SecureSite was based on factors that included the set-up, implementation and life-cycle management of SmartPipes SecureSite compared to manual configuration of those aspects using an array of vendor devices that included multiple NetScreen and Cisco devices.

Since it was impractical to deploy a test network consisting of thousands of devices in a full-scale deployment, for this project The Tolly Group conducted a mini-deployment of various key features in its own lab to extrapolate the operational and management characteristics of provisioning these VPN solutions. This provided empirical material that could be used to develop a clearer understanding of the costs and resources involved in deploying SmartPipes SecureSite solution compared to manually configuring a host of devices.

This study shows SmartPipes SecureSite to be a secure, scalable, easy-to-use provisioning platform. By reducing the complexity of configuration and providing a powerful single-point-of-control for large-scale multi-platform VPNs, SmartPipes SecureSite significantly reduces VPN Total Cost of Ownership while simultaneously improving “turn up” speed for provisioning new networks or modifying existing connections.

Security
TippingPoint Technologies, a 3Com Company Tipping Point Technologies, Inc.UnityOne Intrusion Prevention Appliances Performance Evaluation

Document number: 203101
Release Date: 01 Feb 2003

TippingPoint Technologies, Inc. commissioned The Tolly Group to evaluate its line of UnityOne Intrusion Prevention Appliances, the UnityOne 2400, 1200 and 400. UnityOne is an intrusion prevention system that blocks worms, Trojans, viruses, hybrid attacks, denial of service attacks and other attacks while delivering switch-like performance. Engineers measured network performance (aggregate throughput and latency), the precision of each device’s security filtering and the efficiency of testing for false positives and false negatives. Product class: Intrusion prevention system Security
The Tolly Group User Technology Study: Re-Engineering Enterprise Network Security for 2003

Document number: 203400
Release Date: 10 Jan 2003

In this report detailing results of a survey of 52 enterprise network architects conducted during December 2002, The Tolly Group captures a snapshot of the technology choices users plan to make in their deployment of network security technologies across enterprise networks. Security
WatchGuard Technologies, Inc. WatchGuard Firebox V60 Competitive Firewall/VON Benchmark Evaluation Versus Cisco PIX 515E and NetScreen-50

Document number: 202164
Release Date: 01 Jan 2003

WatchGuard Technologies, Inc. commissioned The Tolly Group to evaluate the Firebox® V60, a four-port Fast Ethernet firewall/VPN security appliance in a variety of real-world zero-loss throughput scenarios and to compare its performance against a Cisco PIX 515E two-port appliance and a NetScreen Technologies, Inc. NetScreen-50 four-port appliance. Security
Enterasys Networks Enterasys Networks XSR-1805 Security Router versus Cisco Systems, Inc. 1751 Modular Access Router Competitive Performance Evaluation

Document number: 202154
Release Date: 15 Nov 2002

Enterasys Networks commissioned The Tolly Group to evaluate its XSR-1805 Security Router, a multi-function routing device, and to compare its performance against that of a Cisco Systems, Inc. 1751 router in a LAN-to-LAN environment designed to stress the system’s processing power to the maximum allowable bandwidth. While security routers are typically deployed in conjunction with a WAN, this series of tests attempts to gauge the processing power of the devices; therefore, tests were conducted in a LAN-to-LAN, two-port router configuration using 10 Mbit/s interfaces and 100 Mbit/s interfaces. The Tolly Group conducted zero-loss (<0.001%) bi-directional (full-duplex) Layer 3 throughput tests and zero-loss, Layer 3 throughput tests when QoS and Access Control Lists (ACLs), respectively, were enabled. Security
Enterasys Networks Enterasys Networks XSR-1850 Security Router versus Cisco Systems, Inc. 2651XM Multiservice Router Competitive Performance Evaluation

Document number: 202155
Release Date: 13 Nov 2002

Enterasys Networks commissioned The Tolly Group to evaluate its XSR-1850 Security Router, and to compare its performance against that of a Cisco Systems, Inc. 2651XM router in a LAN-to-LAN environment designed to stress the system's processing power to the maximum allowable bandwidth. While security routers are typically deployed in conjunction with a WAN, the series of tests attempts to gauge the processing power of the devices; therefore, tests were conducted in a LAN-to-LAN, two-port router configuration using 100 Mbits/ interfaces. The Tolly Group conducted zero-loss bidirectional (full-duplex) Layer 3 throughput tests and zero-loss Layer 3 throughput when QoS and Access Control Lists (ACLs), respectively, were enabled. Security
NetScreen Technologies, Inc (Juniper) NetScreen Technologies, Inc. Netscreen 5200 versus Nokia IP740 and Cisco Systems PIX 535 Competitive Evaluation of Multi-Gigabit Firewall/VPN Multifunction Devices

Document number: 202121
Release Date: 01 Sep 2002

NetScreen Technologies commissioned The Tolly Group to benchmark the NetScreen-5200, a purpose-built high-performance Internet security system outfitted with Gigabit Ethernet interfaces, and to compare the results with those of a similarly outfitted Cisco PIX 535 firewall/VPN and Nokia IP740 firewall device. The Tolly Group conducted application throughput and zero-loss throughput tests, as well as standard latency tests for both firewall and VPN tunnel configurations, the latter incurring the extra processing factored in with support for 3DES and SHA-1. Each of the devices was subjected to a range of session loads, escalating from 1,000 sessions to as many as 500,000 sessions in firewall tests. Security
Inkra Networks Corp. Inkra Networks Corp. Inkra 4000 Virtual Service Switch HardWall Technology Evaluation

Document number: 202145
Release Date: 02 Aug 2002

Inkra Networks Corp. commissioned The Tolly Group to evaluate the HardWall feature of its Inkra 4000 Virtual Service Switch, a carrier-class, multi-service device. The Inkra 4000 is designed to deliver multiple virtualized IP services simultaneously, including firewall, server load balancing, SSL acceleration and VPN services for multiple, dedicated clients. HardWall is Inkra's patent-pending hardware-enforces resource management technology (implemented in an ASIC-based Virtual Rack Processor) that isolates each virtual rack on the system from the others to guarantee the performance levels of individual virtual racks. Tests included validation of the system's capability to sustain functionality despite individual rack failure, demonstration of the switch's capability to control virtual rack bandwidth, virtual rack session over-subscription and virtual rack burst-priority. Security
Bivio Networks, Inc. Bivio Networks, Inc. Bivio 1000 Firewall Performance Evaluation

Document number: 202143
Release Date: 01 Aug 2002

Bivio Networks, Inc. commissioned The Tolly Group to evaluate its Bivio 1000, a high-speed security platform that is integrated with Check Point Software's FireWall-1 NG software. The Bivio 1000 is designed for large enterprises and carriers in order to deliver gigabit-class firewall performance. The Tolly Group tested the Bivio 1000 both as a single-rule, allow-all firewall configured with two Gigabit Ethernet interfaces. Security
Check Point Software Technologies Ltd. Check Point Software Technologies Ltd. VPN-1 Pro vs. Cisco Systems PIX-535 and NetScreen Technologies NetScreen-500 Competitive Evaluation of Enterprise-Class Internet Security Solutions

Document number: 202132
Release Date: 01 Aug 2002

Check Point Software Technologies Ltd. commissioned The Tolly Group to evaluate its VPN-Pro, an enterprise-class security solution that integrates firewall and VPN capabilities against two other enterprise products: a Cisco Systems, inc. PIX-535 and a NetScreen Technologies, Inc. NetScreen-500. Engineers subjected the devices to a battery of tests focusing on UDP firewall throughput, UDP VPN performance and a real-world mix of Layer 7 application traffic. Security
Quarry Technologies (Reef Point) Quarry Technologies iQ8000 Service Edge Switch Performance Evaluation

Document number: 202112
Release Date: 01 Jan 2002

Quarry Technologies commissioned The Tolly Group to evaluate the performance of its iQ8000 Service Edge Switch, a carrier-class, chassis-based routing system designed to deliver security services at the edge of large, high-speed networks. The Tolly Group conducted bidirectional zero-loss throughput tests in both firewall-only and IPSec tunnel configurations with Quality of Service (QoS) and firewall services enabled. Security
ServGate Technologies, Inc. ServGate Technologies, Inc. SG2000 Firewall Performance Benchmark

Document number: 201126
Release Date: 01 Sep 2001

ServGate Technologies, Inc. commissioned The Tolly Group to evaluate its SG2000, a multifunction firewall designed for large enterprises and carriers that is designed to deliver gigabit performance. The Tolly Group tested the SG2000 as a single-rule firewall configured with two Gigabit Ethernet interfaces and benchmarked its bidirectional packet per second UDP/IP packet throughput. Security
WatchGuard Technologies, Inc. WatchGuard Technologies, Inc. Firebox III Model 1000 versus Nokia IP330 and SonicWALL PRO-VX Performance Evaluation

Document number: 201132
Release Date: 01 Sep 2001

WatchGuard Technologies, Inc. commissioned The Tolly Group to evaluate its Firebox III Model 1000, an enterprise-class firewall and IPSec gateway designed to serve mid-size business or branch offices. This purpose-built, Fast Ethernet security device was benchmarked by The Tolly Group, which compared the Firebox 1000 performance to a Nokia Corp. IP330 and a SonicWALL PRO-VX. The Tolly Group conducted zero-loss throughput tests in both IPSec tunnel and firewall configurations with a range of packet sizes. Security
RapidStream, Inc. RapidStream, Inc. RapidStream 6000 (Fast Ethernet) and RapidStream 8000 (Gigabit Ethernet) Security Appliance Performance Evaluation

Document number: 201101
Release Date: 01 Aug 2001

RapidStream, Inc. commissioned The Tolly Group to benchmark the performance of its RapidStream 8000 (Gigabit Ethernet) and RapidStream 6000 (Fast Ethernet) firewall/VPN security appliances. The Tolly Group subjected both devices to a battery of tests to determine bidirectional packet per second (pps) UDP packet throughput, bidirectional file-transfer application throughput and bidirectional UDP packet latency during maximum zero-loss utilization. Security
NetScreen Technologies, Inc (Juniper) NetScreen Technologies, Inc NetScreen-500 vs. Cisco Systems Inc. PIX 535 Competitive Evaluation of Enterprise-Class Internet Security Devices

Document number: 201111
Release Date: 01 Jul 2001

NetScreen Technologies, Inc. commissioned The Tolly Group to benchmark the NetScreen-500 against the Cisco PIX 535 firewall outfitted with an optional VPN Accelerator Card. The Tolly Group conducted application throughput and zero-loss throughput tests, as well as standard latency tests for both firewall and VPN tunnel configurations under various session loads. Security
2Wire, Inc. 2Wire HomePortal 100W Wireless Residential Gateway versus The Linksys Group, Inc. BEFSR41 - EtherFast 4-Port Cable/DSL Router and SonicWALL SOHO/10 Firewall Competitive Evaluation

Document number: 201110
Release Date: 01 Jul 2001

2Wire, Inc. commissioned The Tolly Group to evaluate the firewall security of the 2Wire HomePortal 100W and to compare the results with those of The Linksys Group, Inc. BEFSR41 and a SonicWALL SOHO/10. The Tolly Group evaluated the capability of each device to detect and to protect user home networks from Denial of Service (DoS) attacks. Moreover, testing was designed to reveal the security differences between application-lay gateway (ALG) and demilitarized zone (DMZ) mode for hosting applications on the home network. Security
Nortel Nortel Networks Contivity 2600 VPN Switch Firewall/VPN Multi-Service Performance Evaluation

Document number: 201130
Release Date: 01 Jul 2001

Nortel Networks, Inc. commissioned The Tolly Group to evaluate its Contivity 2600, which is designed to serve large branch offices or data centers that support up to 1,000 VPN tunnels. Tolly Group engineers subjected the Contivity 2600 to a battery of tests to determine the switch's single-rule firewall and IPSec gateway bidirectional zero-loss performance, as well as to benchmark switch performance when both services are contending for bandwidth. Security
OpenReach, Inc. OpenReach, Inc. OpenReach Gateway Software IP Routing and IPSec Gateway Performance Evaluation

Document number: 200236
Release Date: 01 Mar 2001

OpenReach, Inc. commissioned The Tolly Group to evaluate version 2.2 of its OpenReach Gateway Software, an IP routing and IPSec software that runs in conjunction with LINUX on Intel computing platforms. The Tolly Group conducted throughput tests of OpenReach gateway’s IP routing and its (paired) IPSec gateway throughput. Zero-loss packet throughput and application throughput tests were conducted using Red Hat LINUX version 6.1 operating system running on either a 1-GHz Pentium III PC or a 500-MHz Celeron platform. Security
NetScreen Technologies, Inc (Juniper) NetScreen Technologies, Inc NetScreen-5 versus SonicWALL, inc. SOHO/50 and WatchGuard Technologies, Inc. SOHO Competitive Evaluation of SOHO Internet Security Devices

Document number: 200230
Release Date: 01 Jan 2001

NetScreen Technologies, Inc. commissioned The Tolly Group to evaluate the performance of its NetScreen-5, an Internet security appliance integrating firewall and virtual private networking (VPN) in a SOHO environment. NetScreen requested that The Tolly Group evaluate the NetScreen-5 along with the following Internet appliances: a SonicWALL, Inc. SOHO/50 and a WatchGuard Technologies, Inc. SOHO. The Tolly Group conducted tests of devices as IPSec tunnels for application and zero-loss throughput. Tolly engineers also tested each device as a firewall and measured the zero-loss throughput when using UDP packets. For zero-loss performance tests, The Tolly Group measured steady-state throughput at 0.001%, the same metric The Tolly Group uses to test Layer 2 and Layer 3 networking devices. Testing was performed in July through November, 2000. Security
NetScreen Technologies, Inc (Juniper) NetScreen Technologies, Inc NetScreen-100 versus Check Point Software Technologies, Ltd. FireWall-1/VON-1, Nokia IP650 and Cisco Systems, Inc. Firewall Series PIX-515 Competitive Evaluation of Enterprise Class Internet Security Devices

Document number: 200225
Release Date: 01 Jan 2001

NetScreen Technologies, Inc. commissioned The Tolly Group to evaluate its NetScreen-100, an enterprise class firewall and Internet Protocol Security (IPSec) Virtual Private Network gateway. This purpose-built, Fast Ethernet security device was benchmarked by The Tolly Group and compared to the following three devices: a Check Point Software Technologies, Ltd. FireWall-1/VPN-1; a Nokia IP650; and a Cisco Systems, Inc. Firewall Series PIX-515. For all devices under test, The Tolly Group conducted application throughput and zero-loss throughput tests in an IPSec tunnel configuration. Engineers also measured zero-loss throughput and TCP/IP session-processing rate in a firewall configuration. Security
Cylink Corporation Cylink Corporation NetHawk IPSec Gateway Fast Ethernet Performance

Document number: 200224
Release Date: 01 Sep 2000

Cylink Corporation commissioned The Tolly Group to benchmark the performance of a pair of NetHawks in a LAN-to-LAN (IPSec) VPN gateway. The NetHawks, equipped with two full duplex, Fast Ethernet interfaces, were tested for packet per second throughput, file transfer performance, interactive application performance, session capacity and latency. In tests, all traffic was forwarded within a single IPSec Security Association (SA) and was encrypted using DES-3 (triple Data Encryption Standard); data integrity verification was performed with DESMAC (DES Message Authentication Code). Security
Indus River Networks, Inc.(Enterasys) Indus River Networks, Inc. RiverWorks Enterprise VPN ver. 1.2 versus Nortel Networks Contivity Extranet Switch 4000 and TimeStep Corp. PERMIT/Gate 7520 VPN Tunneling Competitive Evaluation

Document number: 199125
Release Date: 01 Aug 1999

Indus River Networks, Inc. commissioned The Tolly Group to test its RiverWorks Enterprise VPN against Nortel Networks’ Contivity Extranet Switch 4000 (CES 4000) and TimeStep Corp.’s PERMIT/Gate 7520. Tests were conducted to determine the single-tunnel throughput for IPSec and the point-to-point tunneling protocol (PPTP) connections over a simulated Internet. Tests shipping ASCII text data across 56 Kbit/s links, and using compression where available, reveal that RiverWorks delivers an average 20% to 60% greater IPSec throughput than Nortel’s CES 4000 or TimeStep’s PERMIT/ Gate 7520. Results also demonstrate that RiverWorks delivered 50% greater throughput when using PPTP than the CES 4000. For VPN vendors who use the Internet as a transport for remote-user VPN tunnel sessions as opposed to more costly direct-dial lines to a traditional remote access server, RiverWorks makes this a viable option because it can effectively optimize data transport over the Internet. Security
Nortel Nortel Networks Contivity Extranet Switch 4000 Fast Ethernet-to-Fast Ethernet Layer 2 Tunneling Protocol Thoughput

Document number: 199104
Release Date: 01 Jan 1999

Nortel Networks commissioned The Tolly Group to benchmark the performance characteristics of each L2TP Network Server (LNS) under test, while handling up to 2,000 active L2TP tunnels in a Fast Ethernet environment. Switches and routers configured as LNSs are commonly used to establish secure tunnels across the Internet, enabling remote users to access internal resources at a central office. The Tolly Group examined the zero-packet-loss (+/-2%) Fast Ethernet-to-Fast Ethernet L2TP throughput of Nortel's Continuity Extranet Switch 4000 (CES 4000) versus the Cisco 7206 router. The test configuration emulated VPN solutions that are also referred to as LAN-to-LAN or branch-to-headquarters VPN solutions. All tests used a single L2TP session per tunnel. Security