@jeff @spla What do you mean after restart? Did you...

@jeff @spla What do you mean after restart? Did you restart while investigating. This will make finding the cause more difficult. That being said none of those jobs are supposed to take an hour, or more than like a minute, realistically.

You will want to send the Sidekiq process a TTIN signal to make it print backtraces of what it's currently executing

https://github.com/mperham/sidekiq/wiki/Signals#ttin

Like 29 Apr 2021 at 0:47 | Wall-to-wall | Open on mastodon.social

5 comments

jeff replied to Eugen

@Gargron @spla had to restart the service in order see the responses here 😬 sorry I'm so awful at this. Do I use the kill command for backtrace? How do I retrieve the pid? I only see jit & tid 😬😬

29 Apr 2021 at 0:54 | Open on my0.net

Eugen Rochko replied to jeff

@jeff @spla See the pid through `systemctl status mastodon-sidekiq`, execute `kill -TTIN pid`, view the backtrace dump with `journalctl -u mastodon-sidekiq --since="1 minute ago"`

29 Apr 2021 at 0:56 | Open on mastodon.social

jeff replied to Eugen

@Gargron @spla ok this is what it says. but tbh I don't really understand anything. are you able to decipher any issues?

https://pastebin.com/jbQNFbt0

29 Apr 2021 at 1:11 | Open on my0.net

Eugen Rochko replied to jeff

@jeff @spla What are the TIDs of the jobs that are stuck for an hour? Search for those in the log output.

From a cursory glance I see some jobs are in the middle of executing ffmpeg conversions. But I don't know if those are the ones that are stuck since you restarted Sidekiq.

29 Apr 2021 at 1:15 | Open on mastodon.social

jeff replied to Eugen

@Gargron @spla the first row didn't even return any results. The 2nd row down tid search just came up with 1 line:

Apr 28 21:09:08 ubuntu bundle[13509]: 2021-04-29T01:09:08.212Z pid=13509 tid=58x WARN: Thread TID-2hth processor

Just doesn't make sense why something can be stuck without much of a description (now showing 6 stuck)

29 Apr 2021 at 1:27 | Open on my0.net

Go Up