Yes, the engine must be out of the car. I had my engine on the engine dyno before it went in the car. It was fun watching the process. The engine is hooked up in a little room. It looks like it is in intensive care with all the wires and stuff around it. From another room, the engine is ran and monitored. I think more things can be monitored on an engine dyno. Mine was tuned on the engine dyno, although once in the car, it was set too lean. I had to bump up a couple of jet sizes.