According to The Verge, it's battery life:

Battery life is, arguably, the biggest pain point in wearables right now. You could certainly make a case for slow apps, or less-than-compelling software, or just plain unattractiveness being the biggest drawback. But those at least have the potential to change. Battery life is an altogether different problem.

You're right, I would like to argue the point. As I noted on Twitter a couple days ago, battery life on my Apple Watch is not even something I think about. I put in on the charger while I shave and shower every morning, and that's all it takes for the thing to never run out of battery on me¹. Other than that 20-30 minutes everyday, I wear the thing 24/7.

I haven't worn all smartwatches to say that this isn't a problem for some of them, but it's definitely not true for the Apple Watch. The biggest problem with the Apple Watch is that it's a quintessential 1.0 product². The hardware is nice, and most everything works as intended, but they simply didn't understand how people would actually use the damn thing in the real world.

The general pitch was that in addition to telling the time and tracking your activity, the Apple Watch would let you do almost everything your iPhone could do, but faster and easier. They nailed the first two aspects, but the third pillar is very hit and miss. The Apple Watch excels at presenting notifications, the weather, and a few other things that have on my watch face, but more advanced tasks are just too much for the hardware to handle.

It's pretty astounding how many things the Apple Watch can do, but it's disappointing how many of these things are made less compelling due to how long they take to do on the current Apple Watch hardware. Part of the problem is with raw processing power, the S1 SoC is just too slow to do anything more than simple functions. The big thing Apple needs to figure out is how to know what wearers are going to want to see before they look for it. Much like how iOS 9's Proactive features try to guess what apps/people/news you want to access, the Apple Watch should do a better job of knowing what information you want to see before you even raise your wrist. This isn't an easy problem to solve, but they may want to take some notes from how Android Wear handles this. It's something you'll always hear from Wear fans as the thing they enjoy most about their watches.

Would I like to have a smart watch that had a year long battery life? Sure, of course that would be better, but it's not the biggest concern for me as an owner, and I don't think it's the biggest concern of prospective customers. It sounds incredibly obvious, but I think the primary focus for smart watch makers right now is to make them smarter. Battery improvements will come, but that's absolutely a secondary concern at the moment.