Email or username:

Password:

Forgot your password?
jeff's posts Post Back to profile
jeff

Why do I have to restart my masto services after just one or two days? It's like everything is frozen. Requests default to pending. And I have plenty of resources to spare, and on current version.

I just don't get it.
@Gargron
25 Apr 2021 at 18:36 | Open on my0.net
25 comments
Eugen Rochko

@jeff I don't know what you mean by pending requests and what that has to do with restarting.

25 Apr 2021 at 19:52 | Open on mastodon.social
jeff

@Gargron sorry. wasn't trying to be unclear.

sometimes everything on the instance stops. as in no new content in federated timeline or home. if I try to follow someone, it says pending even if they don't require approval.

But if I restart those 3x services, my home screen and timeline will fill up and display new content, and I'm able to follow those same accounts with one click.

25 Apr 2021 at 20:00 | Open on my0.net
Eugen Rochko

@jeff Did you check if restarting only mastodon-sidekiq did the trick?

25 Apr 2021 at 20:01 | Open on mastodon.social
jeff

@Gargron no, but will try that next time. I suppose I could make a cron job to just restart nightly. just figured maybe something could be addressed. pretty new to all of this. So sorry for my ambiguity 😂 wasn't trying to pester you in anyway

25 Apr 2021 at 20:02 | Open on my0.net
Eugen Rochko

@jeff Well, I'm not restarting anything here, it's obviously not supposed to require that. Just trying to figure out what could be wrong because the observations described could be anything.

25 Apr 2021 at 20:03 | Open on mastodon.social
jeff

@Gargron if it acts up again is there a log somewhere I should check? Or post it somewhere that would be more productive?

25 Apr 2021 at 20:04 | Open on my0.net
Eugen Rochko

@jeff It's either a) the streaming server isn't up or can't be connected to for some reason or b) the sidekiq service is not executing jobs. There's a link to the Sidekiq dashboard from the admin UI.

25 Apr 2021 at 20:06 | Open on mastodon.social
jeff

@Gargron okay will keep that in mind. Thanks again for the response appreciate your time 🍻

25 Apr 2021 at 20:10 | Open on my0.net
jeff

@Gargron hey wanted to follow-up on this. looks like your hunch was correct - the sidekiq service was not executing jobs. under "Dashboard" the top timeline was all blank.

When I ran "systemctl restart mastodon-sidekiq" my timeline started to fill in and it functions now. just strange behavior considering it has only been up for 3 days.

27 Apr 2021 at 16:27 | Open on my0.net
spla
@jeff seems related to what's happening to mine:
my Mastodon instance was running great on Centos 8 but I migrated it to a new server running Ubuntu 20.04 LTS.
Don't know why after several hours running fine in this new server, Sidekiq/Redis start throwing lines like this:
'WARN: Your Redis network connection is performing extremely poorly. Last RTT readings were. [100632, 99845, 100015, 99969, 100037], ideally these should be < 1000.'
And the federated timeline get freezed.
I must restart Sidekiq & and Redis to back to normal.
Happened three times in two days. For some reason Redis is performing really bad for unknown reason.

@Gargron
@jeff seems related to what's happening to mine:
my Mastodon instance was running great on Centos 8 but I migrated it to a new server running Ubuntu 20.04 LTS.
Don't know why after several hours running fine in this new server, Sidekiq/Redis start throwing lines like this:
27 Apr 2021 at 17:36 | Open on pl.mastodont.cat
Eugen Rochko

@spla @jeff It looks like Redis stops responding. Does the Redis process log anything about the matter? Any clues in RAM or CPU usage? I'm assuming it's on the same machine so no actual network involved

27 Apr 2021 at 17:36 | Open on mastodon.social
jeff

@Gargron @spla I don't recall any warnings like that - but how do I check?

I'm running ubuntu18 with 6vcpu, 6gb ram, and 160 gb storage -- for a single user instance.

27 Apr 2021 at 18:17 | Open on my0.net
jeff

@Gargron @spla happened again, still can't find any errors. But apparently this Eugen guy is where it says Retry? And all 25 workers are busy? 🤔
29 Apr 2021 at 0:18 | Open on my0.net
Eugen Rochko

@jeff @spla Watch the dashboard and see if there's more failures than processed jobs, or if no jobs are being processed (e.g. all stuck)

29 Apr 2021 at 0:33 | Open on mastodon.social
jeff

@Gargron @spla failures graph/line is just flatlining at the bottom. But I can't click on the word to see failure specifics if they exist.

29 Apr 2021 at 0:39 | Open on my0.net
Eugen Rochko

@jeff @spla Well, is there a "processed" line?

29 Apr 2021 at 0:39 | Open on mastodon.social
jeff

@Gargron @spla not when everything stops its just all at the bottom.

29 Apr 2021 at 0:40 | Open on my0.net
Eugen Rochko

@jeff @spla Go on the Busy tab and look at the Started column, is it basically 25 jobs that have been going for a long time? What are they?

29 Apr 2021 at 0:42 | Open on mastodon.social
jeff

@Gargron @spla this is what's started. It's showing only 5 this time after a sidekiq restart 🤔
29 Apr 2021 at 0:45 | Open on my0.net
Eugen Rochko

@jeff @spla What do you mean after restart? Did you restart while investigating. This will make finding the cause more difficult. That being said none of those jobs are supposed to take an hour, or more than like a minute, realistically.

You will want to send the Sidekiq process a TTIN signal to make it print backtraces of what it's currently executing

github.com/mperham/sidekiq/wik

29 Apr 2021 at 0:47 | Open on mastodon.social
jeff replied to Eugen

@Gargron @spla had to restart the service in order see the responses here 😬 sorry I'm so awful at this. Do I use the kill command for backtrace? How do I retrieve the pid? I only see jit & tid 😬😬

29 Apr 2021 at 0:54 | Open on my0.net
Eugen Rochko replied to jeff

@jeff @spla See the pid through `systemctl status mastodon-sidekiq`, execute `kill -TTIN pid`, view the backtrace dump with `journalctl -u mastodon-sidekiq --since="1 minute ago"`

29 Apr 2021 at 0:56 | Open on mastodon.social
jeff replied to Eugen

@Gargron @spla ok this is what it says. but tbh I don't really understand anything. are you able to decipher any issues?

pastebin.com/jbQNFbt0

29 Apr 2021 at 1:11 | Open on my0.net
Eugen Rochko replied to jeff

@jeff @spla What are the TIDs of the jobs that are stuck for an hour? Search for those in the log output.

From a cursory glance I see some jobs are in the middle of executing ffmpeg conversions. But I don't know if those are the ones that are stuck since you restarted Sidekiq.

29 Apr 2021 at 1:15 | Open on mastodon.social
jeff replied to Eugen

@Gargron @spla the first row didn't even return any results. The 2nd row down tid search just came up with 1 line:

Apr 28 21:09:08 ubuntu bundle[13509]: 2021-04-29T01:09:08.212Z pid=13509 tid=58x WARN: Thread TID-2hth processor

Just doesn't make sense why something can be stuck without much of a description (now showing 6 stuck)
29 Apr 2021 at 1:27 | Open on my0.net
Go Up