This article from Florida Today has a comprehensive look at the particular violation on Sunday. In this case, the violator was a tugboat hauling a barge.
The Coast Guard said the tug boat and barge were about 30 miles form
shore and within the safety zone... violators of the safety zone can
be fined if they are within 12 miles U.S. territorial waters. The
Coast Guard does not have the authority to fine boats outside
territorial waters. A boat could, however, still face fines for
equipment violations... A commercial vessel such as the tugboat is
required to have an automatic identification system and radio. If the
vessel strayed into the area because it did not have the required
equipment it could be fined $1,000, even if it was outside territorial
waters. "A lot of people are asking if they could be fined," he said.
"They could be fined for not using AIS system." West said the Coast
Guard had not yet determined what, if any, action would be taken
against the tugboat owners.
A notice regarding a launch from Wallops Flight Center listed the penalty for violation of the launch exclusion zone:
Every person and every corporation which shall violate such
regulations shall be deemed guilty of a misdemeanor and, on conviction
thereof in any district court of the United States within whose
territorial jurisdiction such offense may have been committed, shall
be punished by a fine not exceeding $500, or by imprisonment not
exceeding six months, in the discretion of the court.
Because it mentions 'any district court in the United States', this would seem to be a federal regulation in the U.S..
A case of a leisure boat that caused the scrub of the Antares launch of Cygnus last October (which later exploded at liftoff) was described in this Air and Space article this way:
“Boats out that far in the ocean are supposed to be monitoring marine
band channel 16,” Beutel says. The effort to warn away at-risk boasts
is robust: Nine hours prior to launch time, Wallops begins
broadcasting a warning on channel 16. Four hours before, a complement
of “seven or eight” boats from the U.S. Coast Guard, Coast Guard
Auxiliary, and Virginia Marine Police begins patrolling and clearing
the launch area. An hour after that, they’re joined by three
aircraft—a radar airplane, a spotter airplane, and a helicopter—all on
the lookout for stray boats.
The leisure boat that scuttled last night’s planned launch was spotted
early in the launch count, but it did not respond to numerous attempts
at radio contact. The spotter plane’s subsequent attempt to signal to
the boat by circling it and dipping one wing, as is standard practice
when radio contact fails, also went unanswered. The boat was traveling
at only about four knots. “There was no way it was going to be able to
clear the hazard zone in time for the launch,” Beutel says.
There was no mention of anyone being fined. Elsewhere it was implied that violators are given some leeway and fines may be deferred depending on circumstances. From SpaceFlightNow:
Boats violating restricted rocket safety zones periodically cause
launch delays, but missions with long launch windows give vessels more
time to vacate restricted areas... An Atlas rocket launch from Cape
Canaveral — a spaceport run by the U.S. Air Force — was scrubbed in
2000 after boats strayed into restricted waters during a fishing
tournament scheduled the same day.