Updated 8/6/17: Many changes to reflect process changes.
Since mesh / distributed Wi-Fi systems are here to stay, it's time to get a better test process in place for them. So we're replacing the open air method used in the Mesh Mashup and other previous reviews of Wi-Fi systems. The new process can handle products containing up to three nodes and can also test router / extender designs like NETGEAR's Orbi.
The process consists of five sets of benchmarks:
- Wired routing
- Wi-Fi Throughput vs. attenuation
- Wi-Fi System
- Wi-Fi System Backhaul
- Wi-Fi System Capacity
We run two benchmarks from the Version 10 router process. Please refer to that document for details. Note that since many Wi-Fi system products require a constant WAN connection, the wired routing tests use a switch between the product under test WAN port and the WAN-side testbed machines.
- 30 second Routing Throughput test (using iperf3)
- HTTP Score
Wi-Fi Throughput vs. Attenuation (RvR)
This benchmark tests how wireless throughput changes as path loss (signal level) changes between the product under test and a standard test client device.
Since most Wi-Fi System testing takes place in the small octoScope Box-18 test chambers described below, we perform throughput vs. attenuation testing there, too. The smaller chambers don't have a turntable, so the device under test is not rotated for this test. The simplified setup also does not use the MPE-2 multipath emulator and uses unity-gain dipole antennas instead of the high-gain antennas used in our Revision 10 Wireless test process.
Wi-Fi Systems consist of multiple copies of essentially the same hardware that act as routers or access points. So we test only one module in these types of systems. For router / extender type systems, we run this test on the router module only using the dual-band Pal-245 as the station device.
The Pal is allowed to adjust all parameters automatically except for number of MIMO streams, which is set to two. This means this test uses a 2x2 802.11ac client that supports up to 256 QAM on both bands. Maximum 2.4 GHz link rate is 400 Mbps; maximum 5 GHz link rate is 867 Mbps.
Separate tests are run for downlink throughput and uplink throughput on the 2.4 and 5 GHz bands. Each test starts with 0 dB attenuation between the device under test and Pal client device and is increased in 3 dB steps to a maximum of 60 dB for 2.4 GHz or 48 dB for 5 GHz, until the connection is broken or 0 Mbps throughput is measured. Results are recorded as four benchmarks:
- 2.4 GHz Downlink
- 2.4 GHz Uplink
- 5 GHz Downlink
- 5 GHz Uplink
Wi-Fi System Test
These tests assess the performance of Wi-Fi Systems as systems. They require the new test configuration shown below. Each mesh node (AP) is placed in an RF-tight octoScope Box-18. Three Box-18's are connected by five octoScope quadAttenuators and 4x4 MIMO 1:2 or 1:4 RF splitters.
Three of the quadAttens connect the Box-18's to an octoScope Pal-24, Pal-5 and Pal-245 via an octoScope 1:4 splitter. Note all RF connections contain four separate signal paths. Although only two of those paths are used for the test STA, Wi-Fi Systems, NETGEAR's Orbi specifically, may use up to four streams, which the testbed accommodates.
The Pals are octoScope's partner devices that can function in AP, STA or Monitor mode and support up to 32 virtual STAs. Each is based on a Qualcomm Atheros chipset, has an onboard processor and runs both an iperf3 server and IxChariot endpoint.
The Pal-245 is a selectable dual-band 3x3 device that supports up to three transmit / receive chains, 20, 40 and 80 MHz bandwidth and 256 QAM on both bands. It does not support MU-MIMO or 80+80 or 160 MHz bandwidth.
The Pal-24 and Pal-5 are 2.4 and 5 GHz 4x4 devices, respectively, that support up to four transmit / receive chains, 20, 40, 80, 80+80 and 160 MHz channel bandwidth and 256 QAM. Pal-5 also supports 1x1 and 2x2 MU-MIMO.
Wi-Fi system test configuration
The Pal-245 is our test client (STA) for most tests. The quadAttens set path loss to simulate distance between the STA and each AP. The other two quadAttens set the box-to-box path loss to simulate distance between APs.
Box-18 Root has two gigabit Ethernet connections. One is used to connect the root mesh node to the internet, which most mesh products require for connection to their companion cloud service for proper operation. The second Ethernet connection goes to the root mesh node's LAN connection, which is where the test traffic generator is connected.
The dashed lines going to the other mesh nodes indicate optional connections for their LAN ports, for backhaul / wireless bridge throughput tests.
The smaller Box-18's make it possible to fit a lot of test flexibility into the same space occupied by our previous testbed. The new stack, shown below, handles throughput vs. attenuation (rate vs. range) testing in the top half and Wi-Fi System testing in the bottom.
SmallNetBuilder Wireless Testbed
The octoScope TS-MESH module sits with the Box-18's and holds the quadAttens, Pal and PoE gigabit switches used to provide control and power to them. It also contains the splitters that provide RF connections.
The inside of the Box-18 is pretty tight, so unity-gain dual-band dipole antennas are used, mounted as shown below. For low-profile products, the antennas are angled at 45 degrees as shown; for taller products, they are set straight up. This setup provides typical beacon RSSIs reported by the Pal devices of -30 dBm for 2.4 GHz and -45 dBm for 5 GHz with attenuators set to 0 dB.