Email or username:

Password:

Forgot your password?
Top-level
jeff

@Gargron I have no clue either. I just ran the 3 "stuck" workers in ruby console:

pastebin.com/r8cGLiLY

AND FROM WHAT I CAN TELL THEY ARE RUNNING FINE

maybe I should just let this go for now. if you're out of ideas, I certainly wouldnt be able to figure it out

I'll just set up a cron to restart sidekiq every other hour or so.

But please if you think of anything else to try, please let me know!

22 comments
Clairement crevée

@jeff @Gargron this issue really is odd…

if you happen to get stuck jobs again, could you check the following:

pstree -a -t mastodon

check if the ffmpeg processes are currently using CPU

find one of the ffmpeg processes and do strace -p PID where PID is the PID of one such process

jeff

@Claire @Gargron my instance is cursed. be warned - I'm shitty with computers. let me finish dinner and will check. Thanks for reaching out!

jeff

@Claire @Gargron hi Clouine. I ran the pstree but I fail to see the PID of the process using up ffmpeg:

pastebin.com/pBxx0TB9

am i just blind?

Clairement crevée

@jeff @Gargron sorry i meant to say to look at CPU usage etc. with things like “htop”

to display PIDs in pstree i think it's the -p switch, so pstree -p -a -t mastodon

jeff

@Claire @Gargron no worries. attached a screenshot of htop, I think it looks normal.

Here is the pstree output:
pastebin.com/85gcbN4Y

But when I pick a PID, I don't get much output from strace:

root@ubuntu:~# strace -p 2850
strace: Process 2850 attached
futex(0x56159724c270, FUTEX_WAIT_PRIVATE, 0, NULL

(unless I'm doing something wrong?)

jeff

@Claire @Gargron my strace doesn't really give much info:

Clairement crevée

@jeff yeah by the time you attached it must have already been waiting on stdout/stderr, but i don't understand how it could have happened with the code you merged from @Gargron

jeff

@Claire @Gargron I don't get it either. I literally followed these instructions:

docs.joinmastodon.org/admin/in
(minus 1 command due mimemagic errors, so it installed from source)

is just restarting the sidekiq service hourly from crontab my best bet?

jeff replied to jeff

@Claire @Gargron if you were in my shoes, would you just un-comment out the sidekiq line here:
pastebin.com/H0CrSnHu

or is there something else I can try

jeff replied to jeff

@Claire @Gargron okay. so not surprisingly, it is impossible to upload an animated gif, gets stuck at the progress bar.

But each time an attempt made, it gets a worker stuck with PostProcessMediaWorker.

When manually running the ruby command, it has a message saying "trying to..."
pastebin.com/4AjZ6waR

At least this is predictable 😬

Clairement crevée replied to jeff

@jeff @Gargron odd… there have been a few changes to the fix-hang-on-stdin branch, could you pull it again and try again?

jeff replied to Clairement

@Claire @Gargron yeah I can. What exactly do you want me to try again?

Eugen Rochko replied to jeff

@jeff @Claire Just run it, check out the updated branch like this:

git fetch
git checkout origin/fix-hang-on-stdin

And then restart sidekiq

jeff replied to Eugen

@Gargron @Claire ok. so here are the results when I pull git hang on stdin:

pastebin.com/4gWYrhaw

uploading a gif is still hanging and not completing. but we can let this sit for a bit to see if other media behaves the same way

Eugen Rochko replied to jeff

@jeff @Claire Okay, let's try this: Change Sidekiq log level to debug (you can add RAILS_LOG_LEVEL=debug to .env.production and restart Sidekiq), then watch the logs with:

journalctl -u mastodon-sidekiq -f

And try uploading a GIF. You should see output similar to when running stuff from the console. And you should notice at which step it hangs...

jeff replied to Eugen

@Gargron @Claire ok I had to use zerobin (because pastebin said it contained adult languege) sorry if this isnt very readable:

zerobin.net/?f1a9cd18ffded8dc#

uploaded image around 10:23A system time if that helps you find it. but there is a ton of info there about other statuses.

according to my sidekiq gui, the stuck worker has TID of gokotqmd8

Eugen Rochko replied to jeff

@jeff @Claire Question, does your mastodon-sidekiq.service systemd file contain "SystemCallFilter" line? Just wondering the version you copied when installing...

jeff replied to Eugen

@Gargron @Claire it does, here is the full file:

pastebin.com/taAKLsVB

vi /etc/systemd/system/mastodon-sidekiq.service

Eugen Rochko replied to jeff

@jeff @Claire Can you try replacing it with this file: github.com/tootsuite/mastodon/

Then restarting Sidekiq this way:

systemctl daemon-reload
systemctl restart mastodon-sidekiq

jeff replied to Eugen

@Gargron @Claire

sure, looks like theres two now in the gui. do you want me to try to upload again - with the detailed log?

Eugen Rochko replied to jeff

@jeff @Claire You might want to `kill 3972` since it seems to be stuck. But my suspicion is that things should work now

jeff replied to Eugen

@Gargron @Claire you're right it just went away sorry

Go Up