As the ship gets underway the first time, the order is given to "hoist the main sail". Fore-and-aft sails are hoisted, but the ship's main sail is square-rigged. Square sails on ships of that time were unfurled, not hoisted.
After Jim has killed Israel Hands he leaves the Hispaniola on the beach with the mainsail unfurled and returns to the stockade. When he and the others return to the ship, the sail is furled.
There is an aerial shot of the Hispaniola in the open sea on her outbound voyage with no sails set at all yet there is clearly a wake behind the ship suggesting that she has an engine.