NGFW Firewall Security Benchmark 2024

Firewall Security Efficacy Competitive Assessment

Summary Lab Report

for

Check Point Software

January 2024 SR240117C

miercom.com/checkpoint

https://miercom.com/checkpoint

Check Point Quantum Cyber Security 2 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

Table of Contents 1.0 Executive Summary 3

2.0 Testing Summary Results 5

2.1 Malware Prevention and Detection Summary 5

2.1.1 Malware Prevention vs Detection-Only Zero+1 Day Malware 5 2.1.2 Malware Prevention Efficacy Zero+1 Day Malware 6

2.2 Malicious Phishing URLs Prevention and Detection Summary 7

2.2.1 Phishing and Malicious URL Prevention 7

3.0 False Positive Detection 8

3.1 False Positive Testing Summary 8

3.1.1 False Positive Rate for Malware Detection 8

4.0 Products Tested 9

5.0 Test Setup 10

5.1 Miercom Advanced Offensive Threat Detection 10

5.2 VirusTotal 11

5.3 Testing Environment 12

6.0 About Miercom 13

7.0 Use of This Report 13

Check Point Quantum Cyber Security 3 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

1.0 Executive Summary Miercom was engaged by Check Point to conduct competitive security effectiveness testing of the Check Point Next Generation Firewalls (NGFW) as compared to products from Cisco, Fortinet, and Palo Alto Networks. Testing with Zscaler involved their SWG (Secure Web Gateway). Testing included verifying the effectiveness of anti-virus, anti-malware, Intrusion Prevention System (IPS), anti-bot, URL Filtering (URLF), sandboxing, machine learning, and phishing protection. We conducted tests with all security services enabled and challenged each solution’s ability to detect and block modern-day malware.

Modern threats like web-based malware attacks, targeted phishing attacks, application-layer attacks, and others increase the threat level against organizations globally. The majority of new malware and intrusion attempts to exploit weaknesses in applications, as opposed to networking components and services. NGFWs with advanced threat prevention offer the best protection against the latest generation of cyberattacks.

Our testing specifically focused on the ability to detect and prevent new malware variants within the first 24 hours of their discovery as well as detecting and preventing new phishing sites.

In this report, Zero+1 Day Malware (one day past Zero-Day discovery) means newly discovered malware on the first day of discovery. These malware samples are less likely to be known by any vendor’s signature detection mechanisms in the first 24 hours

Key Findings

Critical Prevention Rate in the first 24 hours: Check Point led in the group test for immediate prevention of the total malware samples. The first 24 hours of a malware campaign are the most dangerous, and this is the critical time to stop an attack before it quickly spreads and creates widespread damage. A security system with a higher block rate in the first 24 hours means an enterprise will spend less time, money, and energy responding to and remediating infected servers and endpoints. • Zero+1 Day Malware Prevent vs. Detect Tests: Check Point prevented over 99.8% of

new malware from a comprehensive set of files and file types, including executables, documents, and archived files that were no more than one day old.

Check Point led with the highest score preventing 99.8% of malware downloads Fortinet had 84.0% prevention and 9.4% detect-only

Terms used in this report include Prevent vs. Detect-Only. Prevent means malware was blocked. Detect-Only means malware was identified but not blocked.



Check Point Quantum Cyber Security 4 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

Zscaler had 75.4% prevention Palo Alto Networks had 69.4% prevention and 8.7% detect-only Cisco had 47.8% prevention and 37.2% detect-only 2.1.1 Prevention vs Detection-Only for Zero+1 Day Malware

• Zero+1 Day Malware Prevent (First to Block) Results

Check Point led with a 99.8% prevention rate Fortinet had an 84.0% prevention rate Zscaler had a 75.4% prevention rate Palo Alto Networks had a 69.4% prevention rate Cisco had a 47.8% prevention rate 2.1.2 Prevention Efficacy for Zero+1 Day Malware

● Phishing Prevention: Again, the first 24 hours are the most critical time to block attacks.

Check Point proved to have the best overall prevention against phishing URLs, making use of (R81.20) advanced AI deep learning capabilities.

Check Point led with a 100.0% phishing and malicious URL prevention rate Zscaler had a 97.2% prevention rate Palo Alto Networks had a 96.5% prevention rate Fortinet had a 95.9% prevention rate Cisco had a 53.1%. prevention rate 2.2.1 Phishing and Malicious URL Prevention

• False Positive Malware Detection: Content falsely reported as malicious creates unnecessary workload and stress on security teams. This, in turn, creates complacency and reduces an organization’s overall security posture and security efficacy.

Check Point led the group with the lowest false positive detection rate of 0.13% Zscaler had a 0.20% false positive rate Fortinet had a 0.23% false positive rate Cisco had a 0.27% false positive rate Palo Alto Networks had a 0.30% false positive rate 3.1.1 False Positive Detection Rate for Malware

• Integrated Machine Learning Technology: We evaluated each vendor’s product with the

most aggressive detection settings and detection features available, including Machine Learning (ML). Check Point and Fortinet uniquely incorporate ML capabilities into the immediate detection and block response.

Check Point Quantum Cyber Security 5 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

2.0 Testing Summary Results

2.1 Malware Prevention and Detection Summary

Summary of NGFW Test Results: Blocking and Detection Efficacy comparing test results from Zero+1 Day recently discovered malware between products.

2.1.1 Malware Prevention vs Detection-Only Zero+1 Day Malware

New Variant Malware Prevention success rate: In our Zero+1 Day Malware test, Check Point prevented over 99.8% of malware from a large set of files and file types including executables, documents, and archives. Palo Alto Networks, Fortinet, Cisco, and Zscaler had prevention rates of 69.4%, 84.0%, 47.8%, and 75.4% respectively.

The chart above reflects how each vendor's firewall performed in Prevention vs. Detection-Only in the first 24 hours of an attack. Prevent means the solution identified malware and immediately blocked it from entering the network. Detect-Only means the solution identified malware but did not prevent that malware from entering the network.

Check Point Quantum Cyber Security 6 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

2.1.2 Malware Prevention Efficacy Zero+1 Day Malware

The chart above reflects how each vendor's firewall performed prevention in the first 24 hours of an attack. Prevent means the solution identified the malware and immediately blocked it from entering the network.

Check Point Quantum Cyber Security 7 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

2.2 Malicious Phishing URLs Prevention and Detection Summary

Summary of NGFW Test Results: Blocking and Detection Efficacy comparing test results from recently discovered phishing and other malicious URLs.

2.2.1 Phishing and Malicious URL Prevention

Missed malicious URLs, less is better. The chart above shows how each vendor's NGFW product performs in Detecting and Preventing of newly discovered (less than 24-Hour known) phishing and other malicious URLs. Check Point demonstrated not only static detection ability but could also detect phishing websites dynamically with AI-based phishing protection, based on analysis of web page content such as corporate logos/icons, suspicious fields, irregular spellings, redirection, and many other obscured maleficent components of these websites. This double layer of protection (reputation- based and content analysis) for phishing detection is important as many phishing websites change their IP address locations and domain names to defeat static reputation-based forms of protection.

Check Point Quantum Cyber Security 8 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

3.0 False Positive Detection

3.1 False Positive Testing Summary False positive occurrences are non-malicious (or benign) files that are misidentified as malicious. Some files fall into a gray category due to their possible misuse on a network. An example of this is a password recovery tool, which, while not technically malicious, is often detected for its malicious potential.

Samples evaluated the granularity of the NGFWs AV engine. An intelligent AV engine flags only malicious files (true positive), so that users can continue clean file transactions. If the false positive detection is too high, the AV engine is considered overly aggressive, hindering network activity and productivity. An intelligent AV engine knows when to pass and not pass samples.

We sent a mixture of false positive samples (clean, suspicious files) and true positives (malware files) via HTTP. Of the clean files sent, we calculated the percentage of samples the NGFW mistakenly flagged as malicious.

3.1.1 False Positive Rate for Malware Detection

For false positive detection, less is better. We examined each of the NGFW products for incidents of false positive detection in the malware tests. These are sample files that may be challenging for the NGFW products to determine whether they are malicious or not when they are not actually malicious. Check Point scored the best, with lowest false positive detection compared to Palo Alto Networks, Fortinet, Cisco, and Zscaler. Testing included review of thousands of samples over the previous 90 days.

Check Point Quantum Cyber Security 9 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

4.0 Products Tested

Check Point Version: R82 and R81.20

Data sheet and specifications:

https://downloads.checkpoint.com/fileserver/SOURCE/direct/ID/103832/FILE/CP_R81_ReleaseNotes.pdf

Palo Alto Networks

Version: PAN-OS 11.1.1

Data sheet and specifications:

https://docs.paloaltonetworks.com/content/dam/techdocs/en_US/pdf/pan-os/11-1/pan-os-release-

notes/pan-os-release-notes.pdf

Fortinet

Version: FortiGate/FortiOS 7.4.2

https://docs.fortinet.com/document/fortigate/7.4.2/administration-guide/954635

Cisco Systems Version: 7.4.1

Data sheet and specifications:

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-

center/admin/740/management-center-admin-74/get-started-overview.html

Zscaler

Platform version: 6.2

Client Version: 3.9.0.156

Data sheet and specifications:

https://help.zscaler.com/zia/step-step-configuration-guide-zia

https://downloads.checkpoint.com/fileserver/SOURCE/direct/ID/103832/FILE/CP_R81_ReleaseNotes.pdf https://docs.paloaltonetworks.com/content/dam/techdocs/en_US/pdf/pan-os/11-1/pan-os-release-notes/pan-os-release-notes.pdf https://docs.paloaltonetworks.com/content/dam/techdocs/en_US/pdf/pan-os/11-1/pan-os-release-notes/pan-os-release-notes.pdf https://docs.fortinet.com/document/fortigate/7.4.2/administration-guide/954635 https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/admin/740/management-center-admin-74/get-started-overview.html https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/admin/740/management-center-admin-74/get-started-overview.html https://help.zscaler.com/zia/step-step-configuration-guide-zia

Check Point Quantum Cyber Security 10 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

5.0 Test Setup The testing conducted was designed to determine the strengths and weaknesses of each NGFW product. In addition to generating traffic patterns and attacks from industry test tools, we use unique, verified malicious samples for a customized, open-source approach. High detection efficacy against this blend of malicious samples indicates well-rounded protection from multiple attack vectors. Over the course of 90 days, we repeatedly downloaded sets of 500 malicious files from VirusTotal (most recently submitted) - with over 25 engines with verdict malicious (high probability of being valid malware). These malicious samples consisted of Office docx, Office xlsx, pdf, exe, and dll and archived files. We assessed each NGFW solution using AV + Anti-Malware, IPS, anti-bot, URLF, sandboxing, and machine learning inline detection mechanisms. Testing was run concurrently on each of the vendor’s NGFW solutions. To further challenge the signature detection mechanisms of the devices under test (DUTs) the malicious samples were also slightly modified to ensure a new hash would be determined for these samples. The modification was done without affecting the malicious payload execution. This allowed the known malware samples to be discovered as new variants, which better challenged the “signature” engines for the NGFWs.

5.1 Miercom Advanced Offensive Threat Detection The threat landscape evolves each day and with more complexity, requiring not only more offensive security but also more dynamic methods of testing. Miercom’s Advanced Offensive Security Testing incorporates scenario-driven methods to provide consumers with relevant data regarding their security. These tests assess the ability of the DUT to detect and prevent specific types of sensitive data from leaving the network without introducing performance degradation. Targeted traffic flows consist of emails that we generate to contain criteria such as user accounts, keywords, and randomized numeric strings formatted, like credit card numbers or tax identification numbers. Simulated targeted traffic is sent in simultaneously with real-world benign background traffic to evaluate detection efficacy and check for false positive detection.

Check Point Quantum Cyber Security 11 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

5.2 VirusTotal

Malware samples from VirusTotal were downloaded and were later used for evaluating all the NGFW products. A user can select a file from their computer using a web browser and send it to VirusTotal. VirusTotal offers many file submission methods, including the primary public web interface, desktop uploaders, browser extensions, and a programmatic API. The web interface has the highest scanning priority among the publicly available submission methods. Submissions may be scripted in any programming language using the HTTP-based public API. The rule set for selecting the VirusTotal samples features in testing is shown below. The sample set for Zero + 1 Malware consisted of sets of 500 randomly selected, freshly submitted samples within 24 hours with at least 25 of VirusTotal’s ~80. The rule set for selecting the VirusTotal samples is shown below:

Check Point Quantum Cyber Security 12 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

5.3 Testing Environment

Vendor Product Version Feature Bundles

Check Point

Quantum

Cyber Security Platform

R82 and R81.20 SNBT

Cisco Systems Cisco Firepower 7.4.1 TMC

Fortinet FortiGate FortiOS 7.4.2 Enterprise

Palo Alto Networks PAN NGFW PAN-OS 11.1.1 Core-Security

Bundle

Zscaler ZIA 6.2 Transformation

Bundle

Test servers configured for payload delivery and to monitor detection and blocking of the open-source malware and phishing URLs.

Source: Miercom

NGFW

Check Point Quantum Cyber Security 13 SR240117C Miercom NGFW Security Benchmark 29 January 2024 Miercom 2024

6.0 About Miercom Miercom has published hundreds of network product analyzes in leading trade periodicals and other publications. Miercom’s reputation as the leading, independent product test center is undisputed.

Private test services available from Miercom include competitive product analyzes, as well as individual product evaluations. Miercom features comprehensive certification and test programs, including Certified Interoperable™, Certified Reliable™, Certified Secure™, and Certified Green™. Products may also be evaluated under the Performance Verified™ program, the industry’s most thorough and trusted assessment of product usability and performance.

7.0 Use of This Report Every effort was made to ensure the accuracy of the data contained in this report, but errors and/or oversights can occur. The information documented in this report may also rely on various test tools, the accuracy of which is beyond our control. Furthermore, the document may contain certain vendors’ representations that Miercom reasonably verified but is beyond our control to verify with 100 percent certainty.

This document is provided “as is”, by Miercom and gives no warranty, representation, or undertaking, whether express or implied, and accepts no legal responsibility, whether direct or indirect, for the accuracy, completeness, usefulness, or suitability of any information contained in this report.

All trademarks used in the document are owned by their respective owners. You agree not to use any trademark in or as the whole or part of your trademarks in connection with any activities, products, or services that are not ours or in a manner that may be confusing, misleading, or deceptive or in a manner that disparages us or our information, projects, or developments.

Miercom’s Fair Test Policy allows for any vendor evaluated to challenge or retest these results in accordance with Miercom Terms of Use Agreement if there is any disagreement in findings. Miercom did not acquire products for this review, nor has agreed to any vendor’s EULA or any other overly restrictive terms limiting free press, product evaluations, editorial works, or publishing. We believe in allowing customers to make informed purchasing decisions.

By downloading, circulating, or using this report in any way, you agree to Miercom’s Terms of Use. For full disclosure of Miercom’s terms, visit miercom.com/tou

© 2024 Miercom. All Rights Reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please email reviews@miercom.com for additional information.

https://miercom.com/tou mailto:reviews@miercom.com

1.0 Executive Summary 2.0 Testing Summary Results 2.1 Malware Prevention and Detection Summary 2.1.1 Malware Prevention vs Detection-Only Zero+1 Day Malware 2.1.2 Malware Prevention Efficacy Zero+1 Day Malware

2.2 Malicious Phishing URLs Prevention and Detection Summary 2.2.1 Phishing and Malicious URL Prevention

3.0 False Positive Detection 3.1 False Positive Testing Summary 3.1.1 False Positive Rate for Malware Detection

4.0 Products Tested 5.0 Test Setup 5.1 Miercom Advanced Offensive Threat Detection 5.2 VirusTotal 5.3 Testing Environment

6.0 About Miercom 7.0 Use of This Report