@danilo I honestly don't buy the "useful" argument (cf. https://www.baldurbjarnason.com/2023/ai-research-again/), still, although in most cases I've seen it's more elaborate than that, and goes hand-in-hand with the "black box" argument you gave:

Basically, we can partition the set of tasks into those that AI can solve and those it cannot – and this is not something that changes through technological innovation much. AI gets better at solving tasks it previously could do only mediocre, but the actual set doesn't change because of its nature.

The tasks that AI can solve however are either things that don't need solving (recreational arts), that wouldn't be a problem if AI does it if we had proper base income (commissioned arts/illustrations) and problems that can be solved a lot more efficient by actually solving the root cause (trains instead of automatic driving cars, the classic employee-uses-AI-to-expand-bullets-into-text and manager does the opposite case). This means that AI is either not useful much for society at all except for being a play thing and doesn't warrant wasting resources.

The second set, tasks AI cannot solve, also contains tasks that AI can pretend to solve. Arguably worse.