Would some weaver just see a sett he liked and simply make something roughly similar
Were most tartans based off of setts that already existed at the time?
We have no idea how setts were developed pre-1750 but it's reasonable to assume that some patterns existed at any given time and that there would have been (new) variations based on colours available and/or preference.

were there only certain existing tartans inside specific "rules?"
There is no evidence that there were any 'rules' until the early C19th.

How much room was there for variation?
As there were no rules there was every oportunity for variation.