Email or username:

Password:

Forgot your password?
jeff

@Gargron OK, so have some encouraging news. Changing the Ruby version seems to fix the DNS lookups. when I test it out now I get the following:
pastebin.com/k2LNCfKz

^^ on the 2nd to last line, it says the table size is empty which is good news.

Unfortunately the workers are still "stuck", but the TID# is much longer than before. I'm assuming to manually try this in the ruby console, would it be correct to test it with:

ActivityPub::ProcessingWorker.new.perform(106...)
?? (or no)

24 comments
Eugen Rochko

@jeff :AAAAAA:

I don't really know where to go from here. I suppose you could keep trying to run those things manually until one of them gets stuck in the console, in which case we might learn where it gets stuck.

Or maybe the server is cursed

To answer your question, yes

jeff

@Gargron I have no clue either. I just ran the 3 "stuck" workers in ruby console:

pastebin.com/r8cGLiLY

AND FROM WHAT I CAN TELL THEY ARE RUNNING FINE

maybe I should just let this go for now. if you're out of ideas, I certainly wouldnt be able to figure it out

I'll just set up a cron to restart sidekiq every other hour or so.

But please if you think of anything else to try, please let me know!

Clairement crevée

@jeff @Gargron this issue really is odd…

if you happen to get stuck jobs again, could you check the following:

pstree -a -t mastodon

check if the ffmpeg processes are currently using CPU

find one of the ffmpeg processes and do strace -p PID where PID is the PID of one such process

jeff

@Claire @Gargron my instance is cursed. be warned - I'm shitty with computers. let me finish dinner and will check. Thanks for reaching out!

jeff

@Claire @Gargron hi Clouine. I ran the pstree but I fail to see the PID of the process using up ffmpeg:

pastebin.com/pBxx0TB9

am i just blind?

Clairement crevée

@jeff @Gargron sorry i meant to say to look at CPU usage etc. with things like “htop”

to display PIDs in pstree i think it's the -p switch, so pstree -p -a -t mastodon

jeff

@Claire @Gargron no worries. attached a screenshot of htop, I think it looks normal.

Here is the pstree output:
pastebin.com/85gcbN4Y

But when I pick a PID, I don't get much output from strace:

root@ubuntu:~# strace -p 2850
strace: Process 2850 attached
futex(0x56159724c270, FUTEX_WAIT_PRIVATE, 0, NULL

(unless I'm doing something wrong?)

jeff

@Claire @Gargron my strace doesn't really give much info:

Clairement crevée

@jeff yeah by the time you attached it must have already been waiting on stdout/stderr, but i don't understand how it could have happened with the code you merged from @Gargron

jeff

@Claire @Gargron I don't get it either. I literally followed these instructions:

docs.joinmastodon.org/admin/in
(minus 1 command due mimemagic errors, so it installed from source)

is just restarting the sidekiq service hourly from crontab my best bet?

jeff replied to jeff

@Claire @Gargron if you were in my shoes, would you just un-comment out the sidekiq line here:
pastebin.com/H0CrSnHu

or is there something else I can try

jeff replied to jeff

@Claire @Gargron okay. so not surprisingly, it is impossible to upload an animated gif, gets stuck at the progress bar.

But each time an attempt made, it gets a worker stuck with PostProcessMediaWorker.

When manually running the ruby command, it has a message saying "trying to..."
pastebin.com/4AjZ6waR

At least this is predictable 😬

Clairement crevée replied to jeff

@jeff @Gargron odd… there have been a few changes to the fix-hang-on-stdin branch, could you pull it again and try again?

jeff replied to Clairement

@Claire @Gargron yeah I can. What exactly do you want me to try again?

Eugen Rochko replied to jeff

@jeff @Claire Just run it, check out the updated branch like this:

git fetch
git checkout origin/fix-hang-on-stdin

And then restart sidekiq

jeff replied to Eugen

@Gargron @Claire ok. so here are the results when I pull git hang on stdin:

pastebin.com/4gWYrhaw

uploading a gif is still hanging and not completing. but we can let this sit for a bit to see if other media behaves the same way

Eugen Rochko replied to jeff

@jeff @Claire Okay, let's try this: Change Sidekiq log level to debug (you can add RAILS_LOG_LEVEL=debug to .env.production and restart Sidekiq), then watch the logs with:

journalctl -u mastodon-sidekiq -f

And try uploading a GIF. You should see output similar to when running stuff from the console. And you should notice at which step it hangs...

jeff replied to Eugen

@Gargron @Claire ok I had to use zerobin (because pastebin said it contained adult languege) sorry if this isnt very readable:

zerobin.net/?f1a9cd18ffded8dc#

uploaded image around 10:23A system time if that helps you find it. but there is a ton of info there about other statuses.

according to my sidekiq gui, the stuck worker has TID of gokotqmd8

Eugen Rochko replied to jeff

@jeff @Claire Question, does your mastodon-sidekiq.service systemd file contain "SystemCallFilter" line? Just wondering the version you copied when installing...

jeff replied to Eugen

@Gargron @Claire it does, here is the full file:

pastebin.com/taAKLsVB

vi /etc/systemd/system/mastodon-sidekiq.service

Eugen Rochko replied to jeff

@jeff @Claire Can you try replacing it with this file: github.com/tootsuite/mastodon/

Then restarting Sidekiq this way:

systemctl daemon-reload
systemctl restart mastodon-sidekiq

jeff replied to Eugen

@Gargron @Claire

sure, looks like theres two now in the gui. do you want me to try to upload again - with the detailed log?

Eugen Rochko replied to jeff

@jeff @Claire You might want to `kill 3972` since it seems to be stuck. But my suspicion is that things should work now

jeff replied to Eugen

@Gargron @Claire you're right it just went away sorry

Go Up