Email or username:

Password:

Forgot your password?
AkaSci's posts Post Back to profile
Top-level
AkaSci

For those interested in failures and recovery in far away spacecraft, check out this thread in August, when Voyager 2 lost contact with earth due to a mispointed antenna (caused by operator error :mastodon_oops: ).
fosstodon.org/@AkaSci/11083140
#Voyager
5/n

13 Dec 2023 at 4:27 | Open on fosstodon.org
27 comments
AkaSci

Richard Stephenson of DSN Canberra explains on twitter how NASA verified that the uplink is working.

They sent a command to Voyager 1 to switch between non-coherent mode and coherent mode transmission.

In coherent mode, the Transmission clock is derived from the Rx signal instead of from the AUX oscillator. This changes the Tx RF frequency a bit which was detected at the DSN.

science.nasa.gov/learn/basics-
descanso.jpl.nasa.gov/DPSummar
#Voyager
6/n

Richard Stephenson of DSN Canberra explains on twitter how NASA verified that the uplink is working.

They sent a command to Voyager 1 to switch between non-coherent mode and coherent mode transmission.

In coherent mode, the Transmission clock is derived from the Rx signal instead of from the AUX oscillator. This changes the Tx RF frequency a bit which was detected at the DSN.

In coherent mode, the Tx clock is derived from the Rx signal instead of from the AUX oscillator
13 Dec 2023 at 5:01 | Open on fosstodon.org
Steve's Place

@AkaSci I was going to say maybe the aux oscillators, but now I'm stumped. That is the sort of stream that might be used for testing before flight, to show the pathway was working. We use similar dummy data at companies developing hardware, fwiw.

13 Dec 2023 at 5:19 | Open on mastodon.stevesworld.co
AkaSci

In the blog post at blogs.nasa.gov/sunspot/, Voyager engineers point out the difficulty in diagnosing problems and crafting solutions for a spacecraft with a signal round-trip-time of almost 2 days and hardware/software developed over 46 years ago using technology long since obsolete.

"Finding solutions to challenges the probes encounter often entails consulting original, decades-old documents written by engineers who didn’t anticipate the issues that are arising today."

#Voyager
7/n

In the blog post at blogs.nasa.gov/sunspot/, Voyager engineers point out the difficulty in diagnosing problems and crafting solutions for a spacecraft with a signal round-trip-time of almost 2 days and hardware/software developed over 46 years ago using technology long since obsolete.

"Finding solutions to challenges the probes encounter often entails consulting original, decades-old documents written by engineers who didn’t anticipate the issues that are arising today."

Screen shot of blurb from NASA blog
13 Dec 2023 at 12:51 | Open on fosstodon.org
AkaSci

NASA DSN in Goldstone, CA is currently receiving the downlink from Voyager 1 at a reduced rate of 40 bps. No uplink at this moment.

Apparently, Voyager 1 switched data rate (160 -> 40 bps) & did a full memory read-out of her Attitude and Articulation Control Subsystem, Flight Data Subsystem, and Command Computer Subsystems A&B.
Transmission time = 6 hours
Download size = ~108 kBytes

Here's hoping that the received data is not 0101... 🤞

eyes.nasa.gov/dsn/dsn.html
#Voyager
8/n

NASA DSN in Goldstone, CA is currently receiving the downlink from Voyager 1 at a reduced rate of 40 bps. No uplink at this moment.

Apparently, Voyager 1 switched data rate (160 -> 40 bps) & did a full memory read-out of her Attitude and Articulation Control Subsystem, Flight Data Subsystem, and Command Computer Subsystems A&B.
Transmission time = 6 hours
Download size = ~108 kBytes

Screen shot of DSN web page
13 Dec 2023 at 15:39 | Open on fosstodon.org
AkaSci

A similar but not identical problem afflicted Voyager 2 in 2010. Received science data (but not engg data?) was garbled.

The problem was traced to a flipped bit in the program stored in the FDS. A command was sent to flip the bit.

The issue was diagnosed by downloading a full memory image, which implies that engg data download was working.

This is probably what was done today with Voyager 1 today. Hopefully, it is a similar problem.

voyager.jpl.nasa.gov/news/deta
@destevez
#Voyager
9/n

A similar but not identical problem afflicted Voyager 2 in 2010. Received science data (but not engg data?) was garbled.

The problem was traced to a flipped bit in the program stored in the FDS. A command was sent to flip the bit.

The issue was diagnosed by downloading a full memory image, which implies that engg data download was working.

14 Dec 2023 at 1:39 | Open on fosstodon.org
AkaSci

NASA did not provide a date but it looks like this issue was discovered and acted upon on Dec 7 or 8.

The graphic below shows the schedule for Voyager 1 comms via DSN, generated on Dec 7. Normally, the downlink rate is 160 bps. On Dec 8, it was switched to 40 bps. And again on Dec 10. Some special commands for the FDS were also sent.
Since then, the D/L rate has been switched between 160 bps and 40 bps a few times with additional FDS commands uploaded.

voyager.jpl.nasa.gov/pdf/sfos2
#Voyager
10/n

NASA did not provide a date but it looks like this issue was discovered and acted upon on Dec 7 or 8.

The graphic below shows the schedule for Voyager 1 comms via DSN, generated on Dec 7. Normally, the downlink rate is 160 bps. On Dec 8, it was switched to 40 bps. And again on Dec 10. Some special commands for the FDS were also sent.
Since then, the D/L rate has been switched between 160 bps and 40 bps a few times with additional FDS commands uploaded.

DSN schedule diagrams for Voyager 1
14 Dec 2023 at 14:17 | Open on fosstodon.org
AkaSci

Two-way comms happening now between Voyager 1 and NASA DSN Canberra.

Of course, the results of the uplink commands will arrive 45 hours from now. The data arriving now left Voyager 1 22.5 hours ago.

Downlink rate is the lower 40 bps rate.
The DSN schedule for Voyager 1 shown below was modified and published yesterday.

Here's hoping that Voyager engineers are getting closer to a solution 🤞

eyes.nasa.gov/dsn/dsn.html
voyager.jpl.nasa.gov/pdf/sfos2
#Voyager
11/n

Two-way comms happening now between Voyager 1 and NASA DSN Canberra.

Of course, the results of the uplink commands will arrive 45 hours from now. The data arriving now left Voyager 1 22.5 hours ago.

Downlink rate is the lower 40 bps rate.
The DSN schedule for Voyager 1 shown below was modified and published yesterday.

1. Screenshot of DSN site 2. Today's DSN schedule for Voyager 1
15 Dec 2023 at 1:46 | Open on fosstodon.org
DELETED replied to AkaSci

@AkaSci 🤞👍🏻 very familiar with that display. I miss working cool space stuff.

15 Dec 2023 at 1:50 | Open on mastodon.social
AkaSci replied to AkaSci

NASA JPL provided a minor update today about the status of the Voyager 1 spacecraft, indicating that the comm. problem that started more than 2 months ago has not been resolved yet. No other details.

Please check out the rest of this thread for more info on the problem where instead of sending science and engg. data, Voyager 1 has been stuck sending a 0101 bit pattern.

@NSFVoyager2
#Voyager
12/n

Image of tweet from JPL indicating that engineers are still working to resolve the data issue on Voyager 1
6 February at 23:28 | Open on fosstodon.org
AkaSci replied to AkaSci

No new info on the status of the Voyager 1 spacecraft, which since Sep 2023 has been sending a 1010 bit pattern instead of real data.

Several popular science outfits have been covering it lately. A bit flip in the FDS is suspected, but it is difficult to identify since the memory cannot be read back.

Several commands were sent yesterday to Voyager 1; responses will arrive 45 hours later tomorrow.

Wonder why they cannot overwrite all prog and data memory.

arstechnica.com/space/2024/02/
#Space
13/n

No new info on the status of the Voyager 1 spacecraft, which since Sep 2023 has been sending a 1010 bit pattern instead of real data.

Several popular science outfits have been covering it lately. A bit flip in the FDS is suspected, but it is difficult to identify since the memory cannot be read back.

DSN schedule for Voyager 1 showing timeline and commands sent on Friday and data expected on Sunday. From https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_02_15-24_03_04.sfos.pdf
17 February at 16:35 | Open on fosstodon.org
AkaSci replied to AkaSci

Good news from the Voyager 1 spacecraft that has been stuck sending a 0101 pattern since Nov 2023.

The team has long suspected the root cause to be a corrupted area of memory in the FDS computer. On Mar 1, they sent some commands to make the FDS skip around sections of memory. The data stream rcvd 45 hours later looked different and was decoded to contain a read-out of the entire FDS memory!

Hopefully, they can now identify and fix the offending memory words.
🤞
blogs.nasa.gov/sunspot/
14/n

Good news from the Voyager 1 spacecraft that has been stuck sending a 0101 pattern since Nov 2023.

The team has long suspected the root cause to be a corrupted area of memory in the FDS computer. On Mar 1, they sent some commands to make the FDS skip around sections of memory. The data stream rcvd 45 hours later looked different and was decoded to contain a read-out of the entire FDS memory!

1. Graphic of the Voyager 1 spacecraft with a starry background 2. High level schematic of Voyager computers and telecom systems
14 March at 3:11 | Open on fosstodon.org
AkaSci replied to AkaSci

Voyager is not out of the woods yet, but the lesson for all of us is to never ever give up.

Here is the schedule for comms with Voyager 1 via NASA DSN this weekend. Some new commands will be sent on Friday, with responses expected 45 hours later on Sunday.

voyager.jpl.nasa.gov/pdf/sfos2
15/n

Schedule for DSN comms with Voyager 1 for Friday and Sunday.
14 March at 4:15 | Open on fosstodon.org
AkaSci replied to AkaSci

Some tech. info on the Voyager FDS computer –
- There was a backup FDS unit but it failed in 1981.
- Custom CMOS CPU - 36 instructions. 80 KIPS, 115 kbps data rate.
- 128 registers, kept in memory.
- CMOS memory, a first in space, 8KB.
- No separate memory for program storage vs execution. The CMOS memory is non-volatile kept powered on by the RTG.
- DMA access to memory by hardware. Instead of “cycle-stealing”, the instructions indicated cycles where DMA can occur.

ntrs.nasa.gov/api/citations/19
16/n

Some tech. info on the Voyager FDS computer –
- There was a backup FDS unit but it failed in 1981.
- Custom CMOS CPU - 36 instructions. 80 KIPS, 115 kbps data rate.
- 128 registers, kept in memory.
- CMOS memory, a first in space, 8KB.
- No separate memory for program storage vs execution. The CMOS memory is non-volatile kept powered on by the RTG.
- DMA access to memory by hardware. Instead of “cycle-stealing”, the instructions indicated cycles where DMA can occur.

Pic of the Flight Data Subsystem (FDS) computer board KIPS = Kilo instructions per second DMA = Direct memory Access RTG = Radioisotope Thermoelectric generator Image source: https://www.allaboutcircuits.com/news/voyager-mission-anniversary-computers-command-data-attitude-control/
15 March at 13:02 | Open on fosstodon.org
AkaSci replied to AkaSci

Status update on the Voyager 1 spacecraft which has been sending a 0101 pattern since Nov 2023.

The problem seems to be a failed memory part in the FDS computer; engineers are planning to move ~200 words of software from one region to another, according to Joseph Westlake, director of NASA’s heliophysics division, who was speaking at a March 20 meeting of the National Academies’ Committee on Solar and Space Physics.

Westlake sounded very optimistic.
👍
nationalacademies.org/document
17/n

Status update on the Voyager 1 spacecraft which has been sending a 0101 pattern since Nov 2023.

The problem seems to be a failed memory part in the FDS computer; engineers are planning to move ~200 words of software from one region to another, according to Joseph Westlake, director of NASA’s heliophysics division, who was speaking at a March 20 meeting of the National Academies’ Committee on Solar and Space Physics.

Image of slide on Voyager 1 status update at the meeting. Source: https://vimeo.com/event/4088825
28 March at 3:50 | Open on fosstodon.org
AkaSci replied to AkaSci

It looks like the Voyager team is preparing for a new "memory upload" to the FDS computer on Friday, as evident from the DSN schedule and instructions shown below for Voyager 1.

I am guessing that this is to rearrange the software so that it no longer uses the locations in the faulty memory chip in the FDS. If true, then hopefully we will hear Voyager 1's true voice on Sunday, 45 hours later. OTOH, this may be just one of many steps on the road to recovery.
🤞
voyager.jpl.nasa.gov/pdf/sfos2
18/n

It looks like the Voyager team is preparing for a new "memory upload" to the FDS computer on Friday, as evident from the DSN schedule and instructions shown below for Voyager 1.

I am guessing that this is to rearrange the software so that it no longer uses the locations in the faulty memory chip in the FDS. If true, then hopefully we will hear Voyager 1's true voice on Sunday, 45 hours later. OTOH, this may be just one of many steps on the road to recovery.
🤞
voyager.jpl.nasa.gov/pdf/sfos2

1. Schedule of DSN antennas for Voyager 1 comms for Friday. 2. Schematic of Voyager computer and comm systems.
28 March at 21:54 | Open on fosstodon.org
AkaSci replied to AkaSci

Looks like the "memory upload" to the Flight Data Subsystem (FDS) on Voyager 1 is taking place at this time from the NASA DSN site in Canberra.
Go Voyager!

eyes.nasa.gov/dsn/dsn.html
en.wikipedia.org/wiki/Canberra
19/n

1. Screenshot of NASA DSN site showing comm antenna and parameters for link to Voyager 1 from Canberra 2. Aerial shot of Canberra DSN site
29 March at 17:49 | Open on fosstodon.org
AkaSci replied to AkaSci

It's been 6 hours since the "memory upload" data was transmitted to Voyager 1 from the NASA DSN site in Canberra.

During that time, the signal has traveled about a quarter of the way to Voyager 1, about the average distance to Pluto. The response will arrive at earth on Sunday around 1500 UTC (RTT = 45 hours).

Let's imagine a spacecraft sent to the nearest star Proxima Centauri, 4.2 light-years away. How would we diagnose problems and upload new software to it?

eyes.nasa.gov/apps/solar-syste
20/n

It's been 6 hours since the "memory upload" data was transmitted to Voyager 1 from the NASA DSN site in Canberra.

During that time, the signal has traveled about a quarter of the way to Voyager 1, about the average distance to Pluto. The response will arrive at earth on Sunday around 1500 UTC (RTT = 45 hours).

Map of solar system and the Voyager 1 spacecraft location in it
30 March at 1:13 | Open on fosstodon.org
AkaSci replied to AkaSci

All eyes and ears on Voyager 1 as data is being downloaded from it in response to the "memory upload" commands and data sent 45 hours ago.

The DSN Goldstone 70m antenna is receiving data now at 40 bits/s.

Hopefully, the problem with its transmission being stuck at 0101 has been fixed. We will find out this week ...
🤞

eyes.nasa.gov/dsn/dsn.html
21/n

1. DSN schedule for Voyager 1 for today 2. Screen shot of NASA DSN site
31 March at 16:45 | Open on fosstodon.org
AkaSci replied to AkaSci

NASA Voyager twitter site confirming that Friday's memory upload was intended as a fix for the Voyager 1 transmission problem caused by the failed memory chip in the FDS computer.

"Sister @NASAVoyager's reply to Friday's upload should b arriving now @ Goldstone's 70m antenna DSS-14, hopefully confirming that the Flight Data Subsystem memory update was successful. If so, telemetry should now give clearly interpretable signals with science & engineering data!"

22/n

Image of tweet from NASA Voyager site
31 March at 17:27 | Open on fosstodon.org
AkaSci replied to AkaSci

Great news on Voyager 1.
Richard Stephenson of DSN Canberra reports that engineering data was being received from Voyager 1 last night at 40 bps.

No science data yet, perhaps because they did not switch to the higher 160 bps rate, but this is a major step towards recovery and validates the diagnosis (failed memory chip in the FDS computer) and fix (rearrange software to bypass the failed memory area).

Now waiting for a status update from NASA.
23/n

Great news on Voyager 1.
Richard Stephenson of DSN Canberra reports that engineering data was being received from Voyager 1 last night at 40 bps.

No science data yet, perhaps because they did not switch to the higher 160 bps rate, but this is a major step towards recovery and validates the diagnosis (failed memory chip in the FDS computer) and fix (rearrange software to bypass the failed memory area).

Image of tweet by Richard Stevenson - "The FDS MEMUPD worked as planned. No attempts were made to change bit rate or return to the lower subcarrier frequency but now engineering data is being received. Baby steps, for the first time in a couple of months I'm feeling really optermistic."
2 April at 11:18 | Open on fosstodon.org
AkaSci replied to AkaSci

Minor update from the NASA Voyager 1 team -

- Confirmation that the problem is due to a faulty memory chip, which affects about 3% of the FDS memory. We have known about this diagnosis since March 27 but if was not officially announced until today.
- No info on the recent report that engg data was received last Sunday.
- "It may take weeks or months" to fix the problem so that Voyager 1 can operate normally without the unusable memory hardware.

blogs.nasa.gov/voyager/2024/04
24/n

Minor update from the NASA Voyager 1 team -

- Confirmation that the problem is due to a faulty memory chip, which affects about 3% of the FDS memory. We have known about this diagnosis since March 27 but if was not officially announced until today.
- No info on the recent report that engg data was received last Sunday.
- "It may take weeks or months" to fix the problem so that Voyager 1 can operate normally without the unusable memory hardware.

4 April at 18:51 | Open on fosstodon.org
AkaSci replied to AkaSci

The DSN schedule for Voyager 1 next week shows commands to be sent on Tuesday to relocate some FDS code around "VIM5". Presumably, VIM5 is a memory module.

Additional uploads will take place on Thursday.

So, there is still work that lies ahead to rearrange the code around the failed memory chip. And we suppose, it has to be done incrementally and meticulously so as to not accidentally brick the FDS computer.

voyager.jpl.nasa.gov/pdf/sfos2
25/n

The DSN schedule for Voyager 1 next week shows commands to be sent on Tuesday to relocate some FDS code around "VIM5". Presumably, VIM5 is a memory module.

Additional uploads will take place on Thursday.

So, there is still work that lies ahead to rearrange the code around the failed memory chip. And we suppose, it has to be done incrementally and meticulously so as to not accidentally brick the FDS computer.

DSN schedule for Voyager 1 on Apr 8 and Apr 11
5 April at 0:51 | Open on fosstodon.org
AkaSci replied to AkaSci

The Voyager 1 team sent some more commands today to the spacecraft to relocate code around the failed memory area in the FDS computer. The command will reach Voyager 1 tomorrow, 22.5 hours later; we will know the results on Saturday.
🤞

The tweet from the Voyager team indicates that the software being relocated is the "program for proper encoding and compression of engineering data (EL-40)."

Would an analogy of replacing spark plugs on a running car be accurate?

voyager.jpl.nasa.gov/pdf/sfos2
26/n

The Voyager 1 team sent some more commands today to the spacecraft to relocate code around the failed memory area in the FDS computer. The command will reach Voyager 1 tomorrow, 22.5 hours later; we will know the results on Saturday.
🤞

The tweet from the Voyager team indicates that the software being relocated is the "program for proper encoding and compression of engineering data (EL-40)."

1. DSN schedule for Voyager 1 on Apr 17, 2024 2. Tweet from Voyager twitter site: Sister Voyager 1 should currently be relocating its program for proper encoding and compression of engineering data (EL-40) to a new memory location that hopefully avoids the problems that arose in November 2023.
18 April at 21:23 | Open on fosstodon.org
AkaSci replied to AkaSci

The NASA DSN site in Canberra is currently receiving data from the Voyager 1 spacecraft. The data will ascertain whether the commands sent on Thu to relocate code around the failed memory area in the FDS computer, worked as expected or not.

Some additional commands are scheduled to be sent today contingent on the results, although the uplink is active now.

Here's hoping that we get good engineering and science data from Voyager 1 today 🤞

eyes.nasa.gov/dsn/dsn.html
voyager.jpl.nasa.gov/mission/s
26/n

The NASA DSN site in Canberra is currently receiving data from the Voyager 1 spacecraft. The data will ascertain whether the commands sent on Thu to relocate code around the failed memory area in the FDS computer, worked as expected or not.

Some additional commands are scheduled to be sent today contingent on the results, although the uplink is active now.

1. Screen cap of NASA DSN website 2. Today's NASA DSN schedule for Voyager
20 April at 13:38 | Open on fosstodon.org
AkaSci replied to AkaSci

Hallelujah! And Congrats all around.

JPL just announced that engineering data was successfully received from the Voyager 1 spacecraft on Saturday April 20, the first time since Nov 2023.

The commands sent on Thu to relocate some code around the failed memory chip in the FDS worked as expected.

Science data is not being received yet; it will require relocation of some more code in the sparse free memory areas in the FDS.

(Engg data = spacecraft health data)
jpl.nasa.gov/news/nasas-voyage
27/n

Hallelujah! And Congrats all around.

JPL just announced that engineering data was successfully received from the Voyager 1 spacecraft on Saturday April 20, the first time since Nov 2023.

The commands sent on Thu to relocate some code around the failed memory chip in the FDS worked as expected.

Science data is not being received yet; it will require relocation of some more code in the sparse free memory areas in the FDS.

1. After receiving data about the health and status of Voyager 1 for the first time in five months, members of the Voyager flight team celebrate in a conference room at NASA’s Jet Propulsion Laboratory on April 20. Credit: NASA/JPL-Caltech 2. Schematic of Voyager computers and telecom systems
22 April at 21:33 | Open on fosstodon.org
AkaSci replied to AkaSci

DSN in Canberra should be receiving a full memory read-out of the Voyager 1 FDS computer as we speak. This will help with the planning for further code relocations to enable transmission of science data soon.

eyes.nasa.gov/dsn/dsn.html
28/n
23 April at 15:30 | Open on fosstodon.org
AkaSci replied to AkaSci

Six months after it suffered a serious brain injury and after months of mind-boggling ultra-long-distance surgery, the Voyager 1 spacecraft walked and talked at full data rate today!

After transmitting a full memory readout on Friday at 40 bps, Voyager 1 switched to the science-mode 160 bps rate, which presumably the DSN site at Goldstone was able to receive and decode today.

Congrats and kudos to all who made it happen.
👏 :mastodance:
eyes.nasa.gov/dsn/dsn.html
29/n

Six months after it suffered a serious brain injury and after months of mind-boggling ultra-long-distance surgery, the Voyager 1 spacecraft walked and talked at full data rate today!

After transmitting a full memory readout on Friday at 40 bps, Voyager 1 switched to the science-mode 160 bps rate, which presumably the DSN site at Goldstone was able to receive and decode today.

Screen cap of DSN site
19 May at 12:57 | Open on fosstodon.org
Go Up