@Gargron OK, so have some encouraging news. Changing...

All posts jeff's posts Post Back to profile

@Gargron OK, so have some encouraging news. Changing the Ruby version seems to fix the DNS lookups. when I test it out now I get the following:
https://pastebin.com/k2LNCfKz

^^ on the 2nd to last line, it says the table size is empty which is good news.

Unfortunately the workers are still "stuck", but the TID# is much longer than before. I'm assuming to manually try this in the ruby console, would it be correct to test it with:

ActivityPub::ProcessingWorker.new.perform(106...)
?? (or no)

Like 4 May 2021 at 21:15 | Open on my0.net

24 comments

Eugen Rochko

@jeff :AAAAAA:

I don't really know where to go from here. I suppose you could keep trying to run those things manually until one of them gets stuck in the console, in which case we might learn where it gets stuck.

Or maybe the server is cursed

To answer your question, yes

4 May 2021 at 21:17 | Open on mastodon.social

jeff

@Gargron I have no clue either. I just ran the 3 "stuck" workers in ruby console:

https://pastebin.com/r8cGLiLY

AND FROM WHAT I CAN TELL THEY ARE RUNNING FINE

maybe I should just let this go for now. if you're out of ideas, I certainly wouldnt be able to figure it out

I'll just set up a cron to restart sidekiq every other hour or so.

But please if you think of anything else to try, please let me know!

4 May 2021 at 21:31 | Open on my0.net

Clairement crevée

@jeff @Gargron this issue really is odd…

if you happen to get stuck jobs again, could you check the following:

pstree -a -t mastodon

check if the ffmpeg processes are currently using CPU

find one of the ffmpeg processes and do strace -p PID where PID is the PID of one such process

4 May 2021 at 21:40 | Open on social.sitedethib.com

jeff

@Claire @Gargron my instance is cursed. be warned - I'm shitty with computers. let me finish dinner and will check. Thanks for reaching out!

4 May 2021 at 22:50 | Open on my0.net

jeff

@Claire @Gargron hi Clouine. I ran the pstree but I fail to see the PID of the process using up ffmpeg:

https://pastebin.com/pBxx0TB9

am i just blind?

5 May 2021 at 0:17 | Open on my0.net

Clairement crevée

@jeff @Gargron sorry i meant to say to look at CPU usage etc. with things like “htop”

to display PIDs in pstree i think it's the -p switch, so pstree -p -a -t mastodon

5 May 2021 at 0:36 | Open on social.sitedethib.com

jeff

@Claire @Gargron no worries. attached a screenshot of htop, I think it looks normal.

Here is the pstree output:
https://pastebin.com/85gcbN4Y

But when I pick a PID, I don't get much output from strace:

root@ubuntu:~# strace -p 2850
strace: Process 2850 attached
futex(0x56159724c270, FUTEX_WAIT_PRIVATE, 0, NULL

(unless I'm doing something wrong?)

5 May 2021 at 0:50 | Open on my0.net

jeff

@Claire @Gargron my strace doesn't really give much info:

5 May 2021 at 0:51 | Open on my0.net

Clairement crevée

@jeff yeah by the time you attached it must have already been waiting on stdout/stderr, but i don't understand how it could have happened with the code you merged from @Gargron

5 May 2021 at 0:53 | Open on social.sitedethib.com

jeff

@Claire @Gargron I don't get it either. I literally followed these instructions:

https://docs.joinmastodon.org/admin/install/
(minus 1 command due mimemagic errors, so it installed from source)

is just restarting the sidekiq service hourly from crontab my best bet?

5 May 2021 at 0:58 | Open on my0.net

jeff replied to jeff

@Claire @Gargron if you were in my shoes, would you just un-comment out the sidekiq line here:
https://pastebin.com/H0CrSnHu

or is there something else I can try

5 May 2021 at 1:07 | Open on my0.net

jeff replied to jeff

@Claire @Gargron okay. so not surprisingly, it is impossible to upload an animated gif, gets stuck at the progress bar.

But each time an attempt made, it gets a worker stuck with PostProcessMediaWorker.

When manually running the ruby command, it has a message saying "trying to..."
https://pastebin.com/4AjZ6waR

At least this is predictable 😬

5 May 2021 at 3:22 | Open on my0.net

Clairement crevée replied to jeff

@jeff @Gargron odd… there have been a few changes to the fix-hang-on-stdin branch, could you pull it again and try again?

5 May 2021 at 9:13 | Open on social.sitedethib.com

jeff replied to Clairement

@Claire @Gargron yeah I can. What exactly do you want me to try again?

5 May 2021 at 13:35 | Open on my0.net

Eugen Rochko replied to jeff

@jeff @Claire Just run it, check out the updated branch like this:

git fetch
git checkout origin/fix-hang-on-stdin

And then restart sidekiq

5 May 2021 at 13:56 | Open on mastodon.social

jeff replied to Eugen

@Gargron @Claire ok. so here are the results when I pull git hang on stdin:

https://pastebin.com/4gWYrhaw

uploading a gif is still hanging and not completing. but we can let this sit for a bit to see if other media behaves the same way

5 May 2021 at 14:15 | Open on my0.net

Eugen Rochko replied to jeff

@jeff @Claire Okay, let's try this: Change Sidekiq log level to debug (you can add RAILS_LOG_LEVEL=debug to .env.production and restart Sidekiq), then watch the logs with:

journalctl -u mastodon-sidekiq -f

And try uploading a GIF. You should see output similar to when running stuff from the console. And you should notice at which step it hangs...

5 May 2021 at 14:17 | Open on mastodon.social

jeff replied to Eugen

@Gargron @Claire ok I had to use zerobin (because pastebin said it contained adult languege) sorry if this isnt very readable:

https://zerobin.net/?f1a9cd18ffded8dc#MSF1QE7wDYC9yvg7NqlegdIOHIbB3zpI6uw6xnxh30Y=

uploaded image around 10:23A system time if that helps you find it. but there is a ton of info there about other statuses.

according to my sidekiq gui, the stuck worker has TID of gokotqmd8

5 May 2021 at 14:31 | Open on my0.net

Eugen Rochko replied to jeff

@jeff @Claire Question, does your mastodon-sidekiq.service systemd file contain "SystemCallFilter" line? Just wondering the version you copied when installing...