Camera test

How we test vacuums – CNET

We test vacuum cleaners.

Tyler Lizenby/CNET

We test all sizes and shapes of vacuum cleaners here at CNET — life size modelslightweight wireless alternativesand the most high-tech of the group, robot vacuum cleaners. Our testing, which takes place at the CNET Testing Lab in Louisville, Kentucky, helps us recommend the best vacuum cleaners for your home — and it can be a lot of fun, too. So here’s a behind-the-curtain look at our setup and testing process of the wide variety of different vacuums available.

Vacuum, but make it high tech

The main test we use on all vacuum cleaners is known as the straight line test. Yes, this is primarily for vacuuming, but we do a lot of calculations to make sure this is a fair test for all models that pass through our lab. Our process is closely aligned with the standard set by the International Electrotechnical Commission. The purpose of a straight line test is to measure the percentage of dirt the vacuum cleaner is able to pick up. We use three types of soil (we call it soil in testing): black rice, sand, and animal hair. We perform the test on three different types of flooring: hardwood, high pile carpet and low pile carpet.

animal hair

Our pet hair test is the easiest. We use 2 grams of humanely collected animal hair spread over each test bed and take before and after photos for visual comparison. We call it an anecdotal test – we’re just comparing how many clumps of hair each vacuum can pick up.

rice and sand

For black rice and sand, we start by weighing the empty dustbin of the vacuum cleaner. Then we adjust the width of this platform we made according to the nozzle size of the unit. The goal is to equalize soil density, which is the amount of mass per soil surface, for all different nozzle widths. The length of our test bed remains the same. We change the width to be one inch less than the length of the vacuum nozzle. This is so that the vacuum cleaner has enough space to cover the entire width of the floor in one pass. Think of it as giving it a half inch margin of error on each side.

robot-vacuum-test-photos-4

Our platform for distributing the soil on the test bench.

Tyler Lizenby/CNET

Then we weigh a certain amount of soil based on a simple equation with width as the variable. For each vacuum cleaner, we want a soil density of 125 grams per square meter of soil on carpet – a number established by the IEC dust collection test. We use 40% for hardwood.

Then we vacuum! Each machine makes a straight line pass, covering our test area. Then we weigh the bin again and calculate the percentage of the initial dirt that was picked up by the vacuum cleaner. To ensure that the bin, and therefore the vacuum cleaner, does not retain dirt inside, we clean and weigh it after each pass, ensuring that this quantity is equal to the initial weight of the bin.

Rinse and repeat three times per soil type for rice and sand, but only once per soil type for animal hair – again, this is more anecdotal. But it all adds up. Between all the cleaning and measuring, just testing one unit equals a total of 21 passes.

We do all of this for every vacuum we test, and it’s our primary form of testing for any type of user-operated vacuum.

Navigation test for vacuum robots

For robot vacuums, we do all of the tests above, but we’re just getting started. Our second test for robot vacuums is a navigation test. This test is important because you need to know which model will clean your living room the fastest and which will cover each spot while it works. Different models use very different techniques to navigate, and this is one of the ways the higher end models separate themselves.

Less expensive models sail with random shocks. They run in a straight line until they hit something, then change direction. Basically they keep going until the battery dies.

robot-vacuum-lab-pics-9

We closely monitor what the robot vacuums are doing in the test room.

Tyler Lizenby/CNET

High-end vacuums use lasers that come out of a turret to map their surroundings. Then the navigation software uses this information to tell the vacuum cleaner where to go and where not to go. You can usually see the maps created by these templates in the corresponding apps. Some models also use pictures of the ceiling, but in general the more expensive models work with an actual plan and this results in a much more efficient job.

We can see it all in action in this room we’ve built to mimic the layout of a real home from a robot vacuum’s perspective. This probably sounds weird to you, but for a vacuum cleaner, these represent table legs, chairs, and other furniture to navigate. The size and dimensions of the room also reflect what other national and global testing organizations use for vacuum robot navigation testing.

Our goal is to use long exposure shooting to evaluate and compare each unit’s navigation efficiency. We mount an LED bar on top of the robot vacuum in the same location and size as the cleaning nozzle on the bottom, and turn off all the lights in the lab. We have a high-resolution ceiling-mounted camera that captures every movement of the robot vacuum as it navigates the room, dodging obstacles, stopping, turning, and trying its best to reach every spot.

We end up with images like these. Below is an example of a low-end vacuum using bump navigation. On it, you can not only see the amount of room covered by the robot vacuum cleaner, but also its navigation pattern and the duration of its run, which are all taken into account when comparing the units with each other. . Note that the vacuum has pretty good coverage, but there’s no noticeable pattern in its operation.

random bump

You can see that this robot vacuum navigates by bumping into walls and moving in another direction.

Gianmarco Chumbe/CNET

Compare that to this image below of a high-end vacuum cleaner at work. Here you can clearly see a systematic pattern. Coverage is the most important thing we evaluate. You want your vacuum to get to every place, but systematic navigation makes a big difference. The first vacuum completed the test in 55 minutes. The second covered the same area in 24 minutes.

neatod10-a

This premium robot vacuum maps the floor with lasers to clean more efficiently.

Gianmarco Chumbe/CNET

Other vacuum tests

These tests represent the total of our standardized vacuum tests. Sometimes products offer unique features that encourage us to perform additional testing. Read our more in-depth look at how we test robot vacuums for more information on our pet waste detection features. Or watch the affectionately titled poo test in action in the video below.

This is our vacuum test, with lots of math and lasers. We turn all the data we get in the lab into performance charts, so we can see results side-by-side and understand which vacuums perform best on which tests. And we play with plastic poo so you don’t have to worry about smears of the real stuff, all to help you make an informed buying decision when it’s time to spend big bucks on a new technology.