Mad Scientist Matt's Lair

Many people these days do not know how to design a proper science experiment. It's quite understandable, since unless you are a professional scientist, you can probably make it through the week without having to perform any science experiments. However, if you want to get more gas mileage from your car, you are going to need a good way to measure if you have improved anything. After all, there are quite a few outright scammers out there hawking bogus mileage gadgets. If you put something on your car that actually makes your mileage worse, you're going to have a very costly mistake.

Sometimes I am much more skeptical about certain devices that claim to enhance mileage than I am about others. I'll be a lot more inclined to believe someone who replaced his original oxygen sensor with a wideband oxygen sensor and reprogrammed his fuel injection to run lean than someone who put a magnet on his fuel line and claimed it made a difference. The difference is that the first is scientifically sound – injecting less fuel can certainly be a reasonable way of burning less fuel. On the other hand, gasoline is not magnetic, so it seems very unlikely that a magnet would affect gas mileage. If you claim that a fuel economy gadget has improved your mileage but the principle the device supposedly works on violates several laws of physics, I will want to see a much higher standard of proof.

To see why this is the case, imagine a doctor has a test that correctly detects a disease 90% of the time. The other 10%, it gives the wrong answer. And suppose that 10% of the doctor's patients have this disease. A little math can predict the results the doctor will have. 90% of her patients will be disease-free. 90% of them will be correctly told that they are healthy. 10% of the healthy patients will be told, wrongly, that they are sick. Of the 10% of her patients that actually have the disease, 90% will correctly be diagnosed. 10% of the sick patients will be told that they are healthy. You can divide the patients into four groups. 81% do not have the disease, and have been correctly told that they do not. 9% are false positives, patients do not have the disease, but have been told that they have it. 9% of the patients do have the disease and have been told that they do. The remaining 1% are false negatives, people who have the disease but were told they do not.

Take a close look at these results. The percentage of patients who actually have the disease and have tested positive is exactly the same as the number of false positives! That means that a patient who has a positive test result only has a 50-50 chance of actually having the disease that the test is supposed to find. In the world of medicine, this would mean that the doctor would order a follow-up test, preferably by a more accurate method. Things would be even worse if only 5% of the population had the disease; in that case, more than two thirds of the people who tested positive for the disease would not actually have it.

The lesson to learn is that if the odds of a device actually working are less than the odds that an experiment will give the wrong results, a good result is actually more likely to be the result of experimental error than it is to be a result of the device actually working. It can be painful to admit a mistake. But this applies to good research as well as bad. If you are testing for something that has a 1 in 1000 chance of working, and your research has a 1 in 100 chance of going wrong, you're still going to have to make it better to be sure your results aren't false positives.

To find out if a change improves your gas mileage, you will need to do your best to eliminate any possibility that the change in mileage is due to any other factor. Here are some important techniques scientists use to make their experiments reliable.

• Measure the original condition (also called a control group) to see how where you are to start with.


• Only change one factor at a time.


• Take as many measurements as you can and see how repeatable the measurements are.


• Measure things with the most accurate measuring device you can find.


• Keep as many things about the test constant as possible.

Consumer Reports has a page where they explain how they test gas mileage compared to how the government tests gas mileage. If you're able to view that page, you will notice that in both cases, the testers try to make their tests as repeatable as possible. The EPA tests cars in a climate-controlled chamber to avoid variations in temperature and humidity that might affect the test. Consumer Reports conducts their tests outside but uses a complex mathematical formula to correct for changes in the weather. The EPA measures how much fuel is burned with a precision instrument that counts all the hydrocarbons coming out of the exhaust. Consumer Reports uses a calibrated fuel flow meter. The EPA uses a pre-programmed test sequence on a chassis dyno. Consumer Reports drives their cars around a test track according to an identical pattern each time. Both of them try to prevent any factor from influencing their measurements, except the car itself.

Most people I've seen who have tested for mileage improvements have some idea of what their mileage was before trying out a mod, so there is not much to elaborate on there. The other points, however, deserve a more in-depth look.

One factor at a time is also relatively self-explanatory. If during one tune-up, you replace the spark plug wires with new ones, install Splitfire plugs, replace the oil with Mobile One synthetic, put in a Tornado Fuel Saver, fit your car with water injection, and replace a burned out oxygen sensor, you won't know which of these has caused any change in gas mileage. In fact, you may have even put in something that could have caused your mileage to go down along with changes that improve it. Serious car guys will note that the list of changes above is a deliberate mixture of measures that may work along with some that are almost certainly scams.

The repeatability of measurements is a much more common issue with amateur gas mileage experiements. If you've kept track of your car's mileage over many tanks, you've probably noticed that it rises and falls depending on what sort of driving you have been doing, and possibly due to other factors such as climate. For example, my Focus usually averages around 28 in the sort of driving I do. Suppose I install some sort of gizmo and find the next time I fill it up, I have recorded 31 miles per gallon. If I just stopped there, it seems like I've had a 3 mpg increase – not bad. Unfortunately, I've measured as low as 25 mpg and as high as 33 mpg without making any changes to the car itself. A measurement of 31 mpg may just be a fluke due to differences in how I have been driving it.

Accuracy of the measurements is also an issue that sometimes trips up the unwary. One well-meaning tester tried to assess the effectiveness of a mod by judging how many miles his car could go before the gas gauge reached Empty. Unfortunately, normal gas gauges are not very well calibrated instruments. On my Focus, an Empty reading seems to mean I could put in anywhere from 11 to 12 and a half gallons of fuel.

So, what is a precise indicator? If I were doing tests, I would love to get my hands on a genuine miles-per-gallon gauge that has a flow meter in the supply and return fuel lines along with tapping into the vehicle speed sensor. If your car has a built-in miles per gallon trip computer, that will probably also work reasonably well. A gas pump's volume measurements are very precise, although it does not always shut off at exactly the same point. But it's pretty close. One test by Popular Mechanics ran the fuel tank dry and then filled it with a carefully measured amount of gas – a technique I'd only repeat if I could shut the fuel pump off as soon as the tank ran out, and had a container at ready to refill it.

Holding all outside factors constant may be one of the trickiest things to do. If you are just measuring your miles per gallon every time you fill up at the pump, you'll have been driving through different kinds of traffic, in different weather, over different terrain each time. And you may even be putting a different blend of gasoline in the tank, as they change the ingredients a little depending on the time of year. Averaging over many tanks may be more accurate, especially if you switch out whatever it is you're doing to try and change your mileage each time you fill up. That way you won't make the mistake of testing your car unmodified in summer and modified in winter, and find that the change in mileage was due to the onset of winter instead of the changes you made to the car. Or due to something on your car wearing out at about the same time you installed the mileage gadget.

Some tests I could see working without any special equipment would include making several passes over the same stretch of pavement going the same way while carefully recording the readings on a mileage gauge, or pouring a measured quantity of gas into the tank and driving it around a flat oval at constant speed until it runs out of gas and using the trip odometer to see how far the car drove. Using cruise control to hold the car at constant speed can eliminate one more source of human error. With a little creativity, you may be able to come up with some other rigorous testing methods if you keep the principles of accuracy and repeatability in mind.