10
$\begingroup$

Sparked by the comments to this answer, how many memory banks does Opportunity have? I'm specially interested in the flash memory but any other info is also welcome.

According to this JPL press release, fixing memory bank 7 left Opportunity with 6 useable banks.

The team received confirmation from Mars on March 20 that the reformatting completed successfully. The rover switched to updated software earlier this month that will avoid using one of the seven banks of onboard flash memory. Some of the flash-memory problems that prompted the team to adopt a no-flash mode of operations in late 2014 were traced to Bank 7. The remaining six banks provide more nonvolatile memory capacity than the rover has used on all but a few days since landing on Mars in January 2004.

But to any computer scientist that seems a bit suspicious:

  1. Computer scientists start counting at 0, so bank 7 would actually be the 8th bank in the range, leaving at least other 7 banks available.
  2. Computers like powers of 2, and dislike prime numbers. 7 is too close to 8 to ignore.
  3. According to NASA, the rover has 256MB. That's not divisible by 7.
  4. For anyone not used to counting from 0, bank 7 would indeed be the 7th bank, leaving 6 before it.

It may well be that they skipped bank 0 for technical reasons, or that it is restricted to some specific use so they can't usually count on it, but it all points at someone at PR making an understandable mistake.

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ I agree - it looks a lot like a mistake in reporting $\endgroup$
    – Jack
    Jun 14, 2018 at 11:47
  • $\begingroup$ But it is also possible to provide decoding for a maximum of eight memory banks and to install only seven banks and leave the eigth bank free. 256 MB would be the maximum of installable flash memory and 224 MB of installed memory. If bank seven is defect, there will be six banks remaining. $\endgroup$
    – Uwe
    Jun 14, 2018 at 12:52
  • 1
    $\begingroup$ @Uwe: NASA says it has 256MiB installed. I made a big update to my answer after reading both links in the question and realizing that there was still a real mystery. $\endgroup$
    – Peter Cordes
    Jun 14, 2018 at 14:15
  • $\begingroup$ @Uwe and Diego: updated my answer with stuff from links I found while researching, in case anyone's interested in a bigger picture of Opportunity's hardware / software. $\endgroup$
    – Peter Cordes
    Jul 29, 2018 at 18:53

3 Answers 3

Reset to default
10
$\begingroup$

As shown in the picture below, there are 8 banks of flash out of which 7 banks are used as data storage and bank 0 is used for flight software (FSW) and Entry, Descent and Landing (EDL).

Opportunity rover flash file system

Source: R. Sosland, M. Seibert, E. Ferguson, R. Steele and K. Zittle, "Getting back on the road: Reformatting Flash memory on-board the Mars Exploration Rover Opportunity," 2015 IEEE Aerospace Conference, Big Sky, MT, 2015, pp. 1-12. doi: 10.1109/AERO.2015.7119257

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ So all 8 banks are flash memory, but bank 0 is used for infrequently changing software and banks 1 to 7 are used for frequently changing data storage. The risk of memory wear is near zero for bank 0. The hardware for all banks could be identical. $\endgroup$
    – Uwe
    Jun 15, 2018 at 16:29
5
$\begingroup$

Glenn Reeves talks about troubleshooting the flash issue, and specifically mentions a 224 MB flash filesystem. At 7x32 MB, this indeed looks like it's 7 banks.

In a more scientific paper, he confirms the 224 MB allocated to the file system.

I can imagine that the 8th bank of flash holds the Spirit firmware, not a filesystem. Spirit has >30 MB of code, and that has to be stored somewhere. An alternative explanation would be that the last 32 MB is used for wear levelling.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Your wear-leveling hypothesis being that each bank could have its own flash translation layer, exposing 32MiB of logical bytes from 32+4MB of physical bytes? If leveling was at the whole-bank level, you'd expect that bank remapping would have solved the problem more easily by just not mapping the faulting bank. $\endgroup$
    – Peter Cordes
    Jun 14, 2018 at 22:01
  • 2
    $\begingroup$ But nice find, 7x32MB of flash used for a filesystem + a separate 32MB used for the firmware explains everything nicely. I had guessed that would be a less likely config, because it would make that one 32MB region special and unable to be substituted by a still-working bank. (We know there's 3MB of EEPROM, so recovery-mode code to handle the case of failing program-memory flash could go there. My answer was assuming that the entire firmware probably fit in that 3MB, so no wonder I was left with mysteries :P) $\endgroup$
    – Peter Cordes
    Jun 14, 2018 at 22:05
  • $\begingroup$ @PeterCordes: Speculation, really, I can't find information about the actual mapping. $\endgroup$
    – MSalters
    Jun 14, 2018 at 22:06
  • $\begingroup$ updated my answer with stuff from links I found while researching, in case anyone's interested in a bigger picture of Opportunity's hardware / software. $\endgroup$
    – Peter Cordes
    Jul 29, 2018 at 18:54
4
$\begingroup$

This is a broader look at Opportunity's computer hardware and how it works, since I had found a bunch of these sources before @wooooooooosh posted the right answer. Seems a shame to not post it somewhere.

An earlier article (https://www.jpl.nasa.gov/news/news.php?feature=4406) also says 7:

The incidents of Opportunity's flash memory not accepting data for storage have occurred in only one of the seven banks of flash microchip circuitry on board.

This agrees with the "one of the seven banks of onboard flash memory" phrase in the quoted article. Unfortunately it turns out this is a misrepresentation. The first bank (bank 0) is not part of the 7x32MiB filesystem used for storage of scientific data, but it exists and it is also NAND flash.

Probably they think of it separately because it holds the firmware (~30MiB): the program code that runs on the CPU, and the data it needs. Leaving bank 0 as read-only (except for firmware updates) means that wearing out those flash chips from too many write cycles is very unlikely. Bad bits or other failures in bank 0 for other reasons could maybe be dealt with by booting from another bank, if the system can boot far enough to be reprogrammed remotely and update the bootloader.

NAND flash can only be read in large blocks (e.g. 512 or 4096 bytes). I don't know the page size of the flash used in Spirit / Opportunity, but according to one source they use flash chips manufactured by AMD.

Spirit's HW is similar but not identical to Opportunity's. I was able to dig up these details about their hardware, most of it consistent:

  • EETimes: Flash-memory snafu cripples Mars rover Spirit, says NASA:

    The two Mars rovers consist of a single board computer called the RAD6000, which is reportedly built by BAE Systems in Manassas, Va. The microprocessor on the system is an older, 25-MHz part, based on IBM Corp.'s PowerPC family. It is a radiation-hardened device.

    The RAD6000 does not use a hard drive for storage. Instead, data are kept in 128-MB of random access memory, although it's unclear which vendor supplies those parts. The computer runs a real-time operating system from Wind River Systems Inc.

    [... also Intersil custom silicon, and Xilinx FPGAs in motor controllers etc.]

  • ExtremeTech: The ultimate remote admin: NASA will reformat Opportunity’s dodgy flash memory from 125 million miles away

    Both Opportunity and Spirit are powered by a radiation-hardened 20MHz BAE RAD6000 computer, with 128MB of RAM, 3MB of EEPROM (firmware), and 256MB of NAND flash. The operating system (VxWorks) and critical subroutines are stored in the non-volatile EEPROM, while the flash memory is used to store less-important data — mostly captured imagery and other scientific data — while it awaits upload to NASA HQ via the Mars Global Surveyor orbiter or the Earth-based Deep Space Network. Curiosity, incidentally, has a virtually identical setup — but it uses a faster CPU (132MHz) and has lots more RAM and flash. Curiosity also has an identical backup failsafe computer that can be switched in if a fault develops — the earlier rovers only have a single computer.

    Confirms that it has NAND-flash. And that the OS is Wind River's VxWorks.

    (Another source said Spirit doesn't have EEPROM.) The article has a picture of a BAE single-board computer, similar to what's in Opportunity.

  • Tom's IT Pro: NASA Reformatted an SSD... on Mars!

    ...with the culprit most likely being dead NAND flash cells...

    Another confirmation of using all NAND flash, as opposed to byte-addressable NOR flash which would allow execute-from-flash. So we know it must copy code from NAND into RAM on boot.

    Opportunity contains 8 memory banks with 4 modules in each bank and four flash chips in each module for a total of 256 MBs of memory manufactured by AMD. The rover also contains 128 MB of RAM, 3 MB of EEPROM and runs on a 20MHz CPU.

  • ITworld: The day a software bug almost killed the Spirit rover

    Spirit ran some hacked-together software that mirrored the DOS filesystem (FAT16 I guess?) in RAM, which failed when the FS filled up to bigger than RAM capacity.

    a DOS library design flaw, a bug in some 3rd party software, and several configuration errors

    Ouch. Presumably Opportunity uses FAT16 as its flash filesystem, too.

The CPU:

The RAD6000 CPUs has 8kiB of unified L1 cache. The PowerPC 601 it's based on also uses a unified cache, not split instruction/data caches like most CPU designs.

Some previous guesses would have been unlikely or impossible designs:

Now we know that bank 0 is special.

Before we knew that, there was speculation that it was used as parity for the other 7 banks. That would be an unlikely design: it would get 7x the amount of writes as other banks. (RAID-5 style distributed parity would be possible in theory to distribute that wear, but with flash erase-blocks being much bigger than write-blocks, not really. e.g. 256k erase block vs. 4k write block. And besides, it would be the same cells every time getting the extra writes in each stripe.)

I guessed that maybe the 8th bank was RAM, and it was on the same bus as flash. But that's not plausible for NAND-flash. It's not byte-addressable; only readable in whole pages through a flash controller chip that handle error-correction codes with page granularity. Hooking it up to the same address lines as the system RAM wouldn't make sense.

Execute-from-flash, aka execute-in-place is only done with NOR flash, not NAND. So the rover must copy copy + data it needs for whatever task its doing (e.g. orbital insertion vs. entry vs. roving on the ground) from flash bank 0 into RAM. Presumably the "no flash mode" just meant no writing data to flash, not avoiding reads of bank 0.

NAND flash can only be erased in large blocks, for example 256 kB, and can only be written in pages (e.g. 4kB at once).

The OS itself (VxWorks, a real-time embedded operating system) probably runs directly from the 3MB EEPROM even in normal operation. This counts as part of the firmware, even though it's stored separately from the code+data in flash bank 0. Code can run directly from an EEPROM; it's byte-addressable like RAM. Probably that includes boot-time functions that load flash into RAM.

I don't know if updates from Earth ever modified the EEPROM contents, but that is plausible because EEPROM = Electrically Erasable Programmable ROM. But it's risky: any interruption or corruption could brick the rover, just like why updating the BIOS on your motherboard warns you not to turn off the power, and make sure the power is stable. Still, uploading a new image to the rover's RAM and then checksumming it would make an update pretty safe and not dependent on anything external after it started. They'd probably only do if if necessary to fix a problem they couldn't work around another way.

Apparently Spirit doesn't have a separate EEPROM, only Opportunity does. So IDK how Spirit boots. There must be something to let Spirit boot, unless the NAND flash controller automatically copies some flash to RAM, or supports reading from memory-mapped flash.

More links

Onboard memory includes 128 megabytes of random access memory, augmented by 256 megabytes of flash memory and smaller amounts of other non-volatile memory, which allows the system to retain data even without power.

Improve this answer
$\endgroup$
10
  • $\begingroup$ A cite from wikipedia: " 128 MB of DRAM, 3 MB of EEPROM, and 256 MB of flash memory". There is more DRAM than 1/8 of 256 MB, it is 4/8. $\endgroup$
    – Uwe
    Jun 14, 2018 at 13:43
  • $\begingroup$ A couple of things to keep in mind: this little fella was launched in 2003, think late 90 tech level at the latest. Also, flash memory is specifically used for storage; it can perform all its duties with flash disabled (as it did for several months), it just has to transmit data immediately to earth. $\endgroup$
    – Diego Sánchez
    Jun 14, 2018 at 14:41
  • $\begingroup$ Just guessing, but they may have hit some limit with their original configuration (e.g. space, power), that caused them to use reduce the number of chips they installed and/or the size of some of the individual chips. Or maybe the chips are all from different manufacturers for redundancy reasons and some makers didn't offer the capacity they wanted. $\endgroup$
    – Rikki-Tikki-Tavi
    Jun 14, 2018 at 16:22
  • $\begingroup$ @Rikki-Tikki-Tavi: none of that explains 256MiB of flash total vs. 32MiB per bank. 7 banks of 32MiB would be easy to explain, as the first half of my answer does. $\endgroup$
    – Peter Cordes
    Jun 14, 2018 at 16:28
  • 4
    $\begingroup$ The RAD6000 CPU is derived from the PowerPC 601. This CPU has a von Neumann architecture and a 32 bit physical memory address for up to 4 GB of memory. No problem to address 128 MB of DRAM, 3 MB of EEPROM, and 256 MB of flash memory. All that addressable with 29 bit for up to 512 MB. 3 of the 32 bit address bus remains unused. $\endgroup$
    – Uwe
    Jun 14, 2018 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.