Laika was never expected to make it back alive because the Sputnik 2 capsule it was launched in didn't have any means of deorbiting. From Anatoly Zak's Russian Space Web page on Sputnik-2:
As remembered by Yevgeniy Shabarov, "after placing Laika in the
container and before closing the hatch, we kissed her nose and wished
her bon voyage, knowing that she would not survive the flight."
Its orbit eventually, in a bit under half a year (November 3, 1957 - April 14, 1958), decayed and the capsule reentered the Earth's atmosphere and most of it if not all would have burnt on reentry. It didn't have a reentry heat shield, no parachutes, and even if it did, the 500 kg something capsule wouldn't be able to keep sufficient life support (breathable air, scrubbing carbon dioxide, food, water, temperature and humidity control) provisions to keep even a small dog alive for such a long time.
Soviets, motivated to some degree also by the public outcry of mostly the Western media and later JFK's protest to Nikita Khrushchev in Vienna in 1961, even though there's documents attesting that such decision to sacrifice Laika for science was difficult to make on its own, nobody enjoyed it, some scientists working on the project publicly declared that science return did not justify her sacrifice, and later made every conceivable effort (and it has to be said they were years ahead at the time compared to US efforts) to equip the following capsules with deorbit motors, heat shield and parachutes, something that had yet to be developed, and they succeeded in returning canine visitors to the outer space back home alive.
Nikita Khrushchev later presented a gift to JFK's daughter Caroline, a puppy named Pushinka (Fluffy) that was one of Stelka's (Arrow) offspring. Strelka and Belka (Whitey) were in 1960 second and third dogs in Earth's orbit.
To answer your questions more directly:
What were expectations of scientists working on the project of sending a dog to space?
To learn more about untill then mostly just theoretical limitations of travel to space and conditions in Earth's orbit. That Laika survived for only a few hours instead of the planned ten days and died due to overheating attests to our lack of knowledge of it at the time. Of course, there was also the prestige of sending first living being into orbit, but that was somewhat spoiled by the fact that Laika didn't survive it.
Did they expect Laika to survive until reentry?
No. See above. Laika was selected by Vladimir Yazdovsky ten days before flight out of ten available dogs, one of which (but not Laika) had two previous suborbital flight experiences. Oleg Gazenko, one of project scientists, was later quoted:
Work with animals is a source of suffering to all of us. We treat them
like babies who cannot speak. The more time passes, the more I'm sorry
about it. We shouldn't have done it ... We did not learn enough from
this mission to justify the death of the dog.
Reentry was roughly half a year away from the date of the launch. While the mission also served to measure orbital decay rate, Sputnik-1 as a precursor orbital mission did teach us enough not to expect Sputnik-2 to reenter any time soon enough for Laika's life support provisions to last. As rumor has it (and this isn't confirmed!), her last food ration was poisoned to euthanize her so she doesn't have to suffer any of the number of worse ways to go. It was one of the options being considered, but it is unknown if it has actually been implemented.
What effects of space flight did they consider most threatening to dog's life in the situation?
Nobody really knew, but there were many theories. We didn't really know if microgravity is even survivable on its own for such a long time (Laika was expected to survive for 10 days). All microgravity experience with mammals we had before Sputnik-2 flight was on short, suborbital flight experiments. But most expected that either solar and cosmic rays radiation dose, or her heart giving up will end Laika's life, and there was instrumentation aboard Sputnik 2 to measure these sources of radiation and Laika's heart rate, blood pressure and breathing were measured.
Did they perform any (not necessarily humane) experiments to determine the chances of success?
Small surgical procedure was necessary before flight to route cables from breathing, pulse and blood-pressure sensors to the transmitters. As stated before, these procedures wouldn't be inhumane and they're routinely performed nowadays on dogs before more difficult surgical procedures. I've seen it done many times. It involves tiny punctures through epidermis and shaving off a small spot of the dog's fur. It might be a bit unpleasant for a minute or two, but that's it.
