The shuttle was not certified to enter with the payload bay doors open. In fact, it was not even certified to enter with more than one of the latches that held the doors closed failed open.
Space Shuttle Flight rule A10-23 states that with even a single latch gang failed open steps will be taken to minimize entry structural loads. For more than one failed open, an EVA would be performed to install latch tools to hold the doors closed. (sorry about all caps, that's the way they wrote the rules)
FOR FAILURE OF ANY TWO PLBD LATCH GANGS TO FULLY ENGAGE
(CENTERLINE OR BULKHEAD), AN EVA WILL BE PERFORMED TO INSTALL
MANUAL LATCH TOOLS ON AT LEAST ONE OF THE FAILED LATCH GANGS.
B. FOR FAILURE OF ANY SINGLE OR MULTIPLE PLBD LATCH GANGS TO
FULLY ENGAGE (CENTERLINE OR BULKHEAD), ENTRY LOADS WILL BE
MINIMIZED PER THE FOLLOWING:
1. ENTRY TEST MANEUVERS WILL NOT BE PERFORMED, AND ANY
PROGRAMMED TEST INPUTS (PTI’S) WILL BE INHIBITED.
2. CONSIDERATION WILL BE GIVEN TO TARGETING A RUNWAY/HAC
APPROACH AT THE PLS OR SLS TO MINIMIZE TAILWIND AT HAC
INITIATION AND TURBULENCE/SURFACE WINDS.
The rationale section is as follows:
There is no analysis which indicates that entry with multiple latch
gang failures is an acceptable condition. Although the latch tools
have not been proven to provide the same structural integrity as a
nominal latch gang, performing an EVA to install these latch tools
provides a workaround to recover at least some load-carrying
capability. Two latch tools are required to secure a single failed
latch gang; however, only two centerline and two bulkhead latch tools
are flown. When two of the same type gangs have failed, only one gang
can be secured, as both available tools are required to secure one of
these latch gangs.
A 1991 Rockwell assessment (which refined analyses performed in 1981
and 1987/1988) for a PLBD
single latch-out (centerline or bulkhead) certified a nominal entry trajectory in AUTO flight control by
exhibiting structural, thermal, and aerodynamic margins above the required 1.4 safety factor. This
analysis shows that steps and gaps may occur as a result of the payload bay-to-ambient delta pressure
and latch disengagement. Temperature exceedances may occur at these steps and gaps, but the resulting
thermal damage poses no flight safety concerns. The extent of the postflight repairs can be limited by
minimizing side slip and steady pitch-up maneuvers, according to the 1991 analysis. (Reference
Structural Assessment of Shuttle Payload Bay Doors for Entry with a Latch Gang Out, SSD91D0276,
May 1991, and A/E FTP #96, 12/18/92.)
Although entry test maneuvers and PTI’s should not expose the vehicle
to conditions outside of its
certification limits, flying a nominal entry trajectory minimizes adverse mechanical, thermal, and
aerodynamic loads. Vehicle accelerations, heating, and compartment-to-ambient pressure differentials
can be limited by avoiding these maneuvers. Selecting a runway/HAC that minimizes HAC winds (most
importantly, tailwind at HAC initiation) will help to minimize normal loads, and selecting a landing site
with benign turbulence and surface wind conditions reduces the need for any abrupt directional control
Source: Generic Flight Rules
To sum up, analysis showed that with even one latch gang out, a safe landing could be performed, but there would be thermal damage to the doors. Since the shuttle was by nature reusable, damage to its structure was highly undesireable, so there was no need to perform further analysis about the effects of multiple latches open (or the doors open). One latch out was bad enough. So steps were taken to generate procedures that tried to ensure that a maximum of one latch would be out for entry.
Therefore, no exact analysis was ever done of the case you describe, but even the first steps of getting into that case were highly undesirable and to be avoided.