You end up at the same HP no matter when you take Tough.

(Twice your level + 2/level thereafter) works out to the same number no matter the level when you take it.

Consider: if you take Tough at 1st level you immediately bump your HP max by 2, then another 2 at 2nd, another 2 at 3rd, another 2 at 4th for a total HP max bump from Tough of 8HP.

Alternately: if you take Tough at 4th level you immediately bump your HP max by 8.

Thereafter: in both cases you increase HP max by an additional 2HP at each level.

Worked Example:

Assuming a CON mod of +3 and the no-roll-for-HP option, here it is charted out. (I’ll also assume you never take another CON ASI, just to make the L20 prediction easier. But as long as you assume both “paths” take the same CON ASIs, nothing in the comparison changes.)

Level12345⋮20HP max,Tough at 1st12+3+2=1719+6+4=2926+9+6=4133+12+8=5340+15+10=65⋮145+60+40=245HP max,Tough at 4th12+3+0=1519+6+0=2526+9+0=3533+12+8=5340+15+10=65⋮145+60+40=245

So does it matter?

The consideration, then, needs to be in how much you value that HP earlier, vs. whatever the opportunity cost of taking Tough is. In other words, how does earlier HP bump compare to the other feat(s) you are considering? That’s for you to decide.