Email or username:

Password:

Forgot your password?
Asahi Linya (朝日りにゃ〜)

I regretfully completely understand Wedson's frustrations.

lore.kernel.org/lkml/202408282

A subset of C kernel developers just seem determined to make the lives of the Rust maintainers as difficult as possible. They don't see Rust as having value and would rather it just goes away.

When I tried to upstream the DRM abstractions last year, that all was blocked on basic support for the concept of a "Device" in Rust. Even just a stub wrapper for struct device would be enough.

That simple concept only recently finally got merged, over one year later.

When I wrote the DRM scheduler abstractions, I ran into many memory safety issues caused by bad design of the underlying C code. The lifetime requirements were undocumented and boiled down to "design your driver like amdgpu to make it work, or else".

My driver is not like amdgpu, it fundamentally can't work the same way. When I tried to upstream minor fixes to the C code to make the behavior more robust and the lifetime requirements sensible, the maintainer blocked it and said I should just do "what other drivers do".

Even when I pointed out that other C drivers also triggered the same bugs because the API is just bad and unintuitive and there are many secret hidden lifetime requirements, he wouldn't budge.

One C driver works, so Rust drivers must work the same way.

186 comments
Asahi Linya (朝日りにゃ〜)

Making the Rust bindings safe would have required duplicating much of the functionality of the C code just to track things to uphold the lifetime requirements. It made no sense. It would have been easier to just rewrite the whole thing in Rust (I might end up doing that).

To this day, bugs in the DRM scheduler have been the only causes of kernel panics triggered via my Apple GPU driver in production.

The design of that component is just bad. But because I come from the Rust world, the maintainer didn't want to listen to my suggestions.

If it takes a whole year to get a concept as simple as a trivial "device" wrapper upstreamed (not any device model functionality, literally just an object wrapping a struct device so we can pass it around) then how is Rust for Linux ever going to take off?

Rust works. I'm pretty sure I'm the only person ever to single handedly write a complex GPU kernel driver that has never had a memory safety kernel panic bug (itself) in production, running on thousands of users' systems for 1.5 years now.

Because I wrote it in Rust.

But I get the feeling that some Linux kernel maintainers just don't care about future code quality, or about stability or security any more. They just want to keep their C code and wish us Rust folks would go away. And that's really sad... and isn't helping make Linux better.

Making the Rust bindings safe would have required duplicating much of the functionality of the C code just to track things to uphold the lifetime requirements. It made no sense. It would have been easier to just rewrite the whole thing in Rust (I might end up doing that).

To this day, bugs in the DRM scheduler have been the only causes of kernel panics triggered via my Apple GPU driver in production.

bitzero
@lina At this point, I guess it’s a cultural clash that has little to do with Rust’s supposed pros and cons. I myself don’t like the language that much, but plainly ignoring it is probably extreme. And is not helping anyone.
romu

@lina Keep it up 😺, you can do it 😸

Scherzog von Beast Oil

@lina There is a *fuckton* of pride at stake among C (and C++) developers.

"C(++) is a perfectly good language, just write your code correctly from the start and never make mistakes" is a statement I hear with almost frightening regularity. Of course that usually also correlates with said people being rather found dead than admitting they may have made bad design decisions in the past (or worse, mistakes while implementing said designs).

crazyeddie

@scherzog @lina A lot of C++ developer actually hate the language and refuse to use its modern features, but I would be shocked to hear that said by one who does use them. C++ is an evolving language that's trying more and more to leverage compile time safety assertions. Rust is more or less just an extreme application of the same concept: it's better if the compiler can catch your errors.

Aren't the kernel developers hostile to anything not C or asm? Linus really hates us C++ devs...

rastilin

@crazyeddie @scherzog @lina

The worst part is that in benchmarks, Rust beats out C++ and is only very slightly slower than C (by 7%), but would probably be more secure and more reliable in practice.

It might also be faster than C in production since it can better optimize itself.

crazyeddie

@rastilin @scherzog @lina I can readily beat the crap out of a great amount of C code, performance wise, with C++ so I'm not sure what you're measuring there. Look at how a C vs. C++ developer might make a binary tree for example: C++ version is faster and safer. If you want to do build time processing in C you're also stuck with the preprocessor. Maybe the C++ code was crap, or maybe it's something C++ is really bad at...I dunno, but it sounds suspect.

Scherzog von Beast Oil

@crazyeddie @lina Idk, the person I hear it most from is very much a C++Next fanboy. The usual reactions to any complaints about the language are "ah but C++25/30/40 will address that" and that failing again "skill issue, shoulda used $feature".

My biggest problem with C++ is that nothing really pushed people towards the modern helper features so all the old style code remains mainstream.

crazyeddie

@scherzog @lina Yep. The number of C++ developers who actually get to use modern features of the language is way smaller than the total number of C++ devs. Community members are one thing, but in the field it's all old shit and nobody wants to update for fear the whole house of cards will collapse if some intern flies off the handle using new features they don't understand...you know, because code reviews are in the process but it all just gets checked off.

That's how those discussions go

crazyeddie

@scherzog @lina But whoever said, "Just make it right the first time," had to be pure moron. Doesn't matter the language. An experienced developer has experienced too much failure to think remotely like that anymore. You only think you can write consistently elegant code and designs that always just work is if you've never tried to do it. In ANY language. I'd have a hard time working with a person that said that. I'd be always thinking how stupid they must be.

tizilogic

@lina I just wish there were fewer people that go religious crazy about such things and more people trying to argue on well researched and as objective as possible facts..

saddest problem is that these things tend to spark some form of "war" where it is one group against the other.. can't we just get along?

tizilogic

@lina tbc, not accusing you of "going religious crazy" about rust..

also DISCLAIMER: I love C simply because it just works everywhere and compiles fast as heck, even with all its downsides it comes with by design.
To me, Rust currently fulfills 0.9 out of those 2 criteria and once it is >1 I likely would dive in head first into it and probably forget I ever had any love for C in the first place

Ben Leggett

@lina

The longer you work in this stuff the more you realize a lot of people are just acting like children about code. That's all it is. It's not great skill or genius. It's just childish obsessiveness.

Sean

@dadregga @lina yeah my experience is that people tend to get defensive about their projects and suggesting that they might be able to improve on something or that you have a need they didn’t consider gets an immediate negative reaction, no matter what community the report comes from. This even happens between teams working at the same company.

Raul

@lina Even Linus himself recently pointed his own finger directly at memory management issues in Linux, and yet this perennial resistance to change persists:

"You'd think that all the basics would have been fixed long ago, but they're not. We're still dealing with basic issues such as memory management."

zdnet.com/article/linus-torval

Orca🌻 | 🏴🏳️‍⚧️

@lina@vt.social last time I heard there's even work in progress to proactively remove all Rust code from the kernel for religious-like reasons... These people should really just get the f out and handle the maintaining to someone else with a right state of mind.

Jessica's new Main

@lina@vt.social C developers not beating the "old as dirt gatekeepers" allegation

Timo Zimmermann

@lina I have seen many disputes over the last two decades Linux kernel maintainers had.

Most didn’t appear like anyone wants to improve things or actually do something useful for users.

It came down to personal preference, ego, ideology and power trips.

Personally I don’t know how anyone can find the mental energy to deal with that, but I truly appreciate everyone who still tries to improve things!

ludiofines

@lina the simplest solution is usually the best, 5$ shovel...

jokes aside, i see this around me as well and believe is more about "don't make me work since i am close to retire". i see this in people close to 60 taking care of providing for their family and rewriting something in Rust "just because" will not change anything in their routines.

keep it up and wait until they retire when you will be the second in command and take charge. It is the most painful, i know.

Lizzie

@lina@vt.social this kind of stuff is currently my biggest issue with linux. We haven't seen many innovations in the kernel for a long time. The hardware is getting more complex and requires more effort into making sure the kernel can understand what it is and how to use it at least basic enough to get some drivers installed. But its always the drivers have to work with the kernel, and not the other way around, and that stops so much from working as well as it could. And these people refuse to even budge a centimeter on the idea that they might be wrong in that decision. And those kinds of people tend to ruin everything they hold any power in.

@lina@vt.social this kind of stuff is currently my biggest issue with linux. We haven't seen many innovations in the kernel for a long time. The hardware is getting more complex and requires more effort into making sure the kernel can understand what it is and how to use it at least basic enough to get some drivers installed. But its always the drivers have to work with the kernel, and not the other way around, and that stops so much from working as well as it could. And these people refuse to even...

Tᴀᴋᴀɴᴀsʜɪ Hᴏʀᴏ

@lina@vt.social The one C driver doesn't work either. Look at how many memory errors amdgpu spits out every single day ​:blobcatnotlike:​ ​:bocchi_glitch:​

Ertain

@lina It's unfortunate that you and other kernel programmers have to go through that.

In Re: the 🏳️‍⚧️✨ of Rylie

@lina yeah... Hate it when folks do shit like this, try not to do it myself. New or different ideas are not inherently wrong...

I don't know how people put up with it for long.

In Re: the 🏳️‍⚧️✨ of Rylie

@lina to be fair I'm sitting on some year old patches but it's not inherently from resistance, it's from overload.

Charadon

@lina "design your driver like amdgpu to make it work, or else"

Which is concerning, because the amdgpu driver isn't even that good. I can't tell you how many times back when I had an AMD gpu I got a system lockup because of Ring Timeouts. Meanwhile on my Intel Arc, I haven't had a *single* gpu driver crash since I got it, while with AMD it was a weekly occurance

GD2

@lina wait, amdgpu works?

Lyude🌹#BLM

@lina @desttinghim what bothers me about this is it also feels like a leadership failure. The reality is if someone like this doesn't see "the value" in rust then they should be required to find someone else who can be trusted as a maintainer to handle the code. I don't know how folks like Linus actually expect rust to pick up in the kernel if we don't address the maintainers who clearly want nothing to do with this. There's plenty of other contributors capable of reviewing this code

Lyude🌹#BLM

@lina @desttinghim yet: the sentiment we seem to agree on is that adopting rust is quite literally important to the future of Linux, since it's not just about the benefits the language brings: but the fact it actually makes kernel development appealing to a wider audience and ensures it remains sustainable

Orman

@Lyude @lina @desttinghim yeah, if it's prompting pushback so intense people are explicitly giving up on "the future of the project" the people causing the pushback need held accountable

Neal Gompa (ニール・ゴンパ) :fedora:

@Lyude @lina @desttinghim If we take Rust out of the equation and look at any other large scale modernization effort, we see pretty much the same problems.

I like to pick on folios because I'm mostly a storage guy, but that transition has been long and painful because there's fundamentally no real leadership in Linux. A lot of storage developers just cannot handle working in an organized fashion.

DRM is the most organized subsystem and even it suffers through these kinds of problems.

Raul

@Lyude @lina @desttinghim I kind of hope Linus himself will master Rust and introduce a bunch of it in quite critical points.

Lyude🌹#BLM

@raulinbonn @lina @desttinghim to be honest - my bigger hope would be that he actually sets rules around "If there has already been a consensus in the kernel to accepting a project, and part of that project falls on your lap for review - you either have to review it in good faith or hand it to someone else who will.". Because like - folks like Neal are right, this has definitely been a problem in many areas of the kernel for a while. I'm all for thorough review of patches but I've seen/had discourse sent to me that was clearly just "I have a disdain for rust and am buying time", or sometimes literally just "I have a disdain for rust and I will bother you because you sent patches related to rust"

@raulinbonn @lina @desttinghim to be honest - my bigger hope would be that he actually sets rules around "If there has already been a consensus in the kernel to accepting a project, and part of that project falls on your lap for review - you either have to review it in good faith or hand it to someone else who will.". Because like - folks like Neal are right, this has definitely been a problem in many areas of the kernel for a while. I'm all for thorough review of patches but I've seen/had discourse...

Lyude🌹#BLM

@raulinbonn @lina @desttinghim It even extends outside of just being obstinate - subsystems like input have become pretty difficult to get a lot of work into simply because there's not enough maintainers, yet despite this being a problem for ages now I've seen basically no effort from the single maintainer there to actually bring in new maintainers when other subsystems like DRM very rarely ever have only single maintainer for anything

Phantasm
@Lyude @lina @desttinghim
>The reality is if someone like this doesn't see "the value" in rust then they should be required to find someone else who can be trusted as a maintainer to handle the code
Replacing maintainers that worked on a piece of code for years with someone else only because 'they don't see "the value" in rust" is insane.

>I don't know how folks like Linus actually expect rust to pick up in the kernel if we don't address the maintainers who clearly want nothing to do with this.
There are reasons why maintainers want nothing to do with Rust. Maybe, just maybe, improve Rust instead of complaining about Linux maintainers on social media, because in the end shouting into the void about it will improve absolutely nothing about the situation.

I'm calling it now, Rust in Linux will be gone in few years just like C++. The only reason why it hasn't been done yet is because more people are interested in Rust than in the hell called C++.
@Lyude @lina @desttinghim
>The reality is if someone like this doesn't see "the value" in rust then they should be required to find someone else who can be trusted as a maintainer to handle the code
Replacing maintainers that worked on a piece of code for years with someone else only because 'they don't see "the value" in rust" is insane.
Sven Peter

@lina a few weeks ago I considered rewriting atcphy in rust because that would’ve been more fun but then stumbled across exactly that: it’s been over a year and there still wasn’t any platform device binding upstream. I would’ve been fine writing phy bindings but there‘s still a lot of infrastructure missing unfortunately :(

(Not that it matters much for atcphy: the driver doesn’t gain anything from rust since it’s fairly simple and doesn’t even need any memory allocation outside of probe. It just would’ve been more fun than rewriting the current C version to me)

@lina a few weeks ago I considered rewriting atcphy in rust because that would’ve been more fun but then stumbled across exactly that: it’s been over a year and there still wasn’t any platform device binding upstream. I would’ve been fine writing phy bindings but there‘s still a lot of infrastructure missing unfortunately :(

Asahi Linya (朝日りにゃ〜)

@sven Yeah... I've been waiting for someone else to figure that out because the whole device probe design goes over my head... but with how long it took to even get struct device in, I don't know how long that is going to take...

karolherbst 🐧 🦀

@lina @sven it's going to be interesting in regards to nova and how much it can push maintainers here.

At least "we don't want to spend more time on the old nouveau driver, so if you want to have support for new nvidia GPUs, it's gonna be a rust driver" isn't something you can simply ignore.

People can go "macs? why do I care", but doing the same for Nvidia GPUs?

That can lead to quite some discussions upstream if people keep blocking work.

chaos_princess

@lina @sven Kinda had the same thoughts when starting to write adpdrm, looked at if using rust would be possible and then quickily went "Ok, i want this thing to work, not spend 10x the time bikeshedding bindings"

Lun

@lina A solid tenth of the amdgpu mailing list is randomly moving fences around playing whackamole and hoping they'll finally get it not to hang; being forced to copy them is scary.

Asahi Linya (朝日りにゃ〜)

@Lun I wonder if they've figured this one out yet? I'm pretty sure it can cause random crashes in all drivers using drm_sched with certain kernel builds...

github.com/AsahiLinux/linux/co

I'm too tired of this code to come up with a proper fix and send it upstream though, I don't want to deal with that guy any more... I'll probably end up writing a Rust scheduler anyway so...

Avuton Olrich

@lina Personally, I found the linked that YouTube video was pretty brutal. The audience hijacked the session from the speaker, some with offtopic rage, when the speakers appeared to be cooly interested in cooperation. Others expecting the perfect API before it becomes "battle tested".

Gabriel Adrian Samfira

@avuton @lina yeah. It was a bit hard to watch. Misplaced rage. Completely off base. Can't blame the maintainer for stepping down after that.

Gratuitous hostility. Meh.

Asahi Linya (朝日りにゃ〜)

@avuton Wanting the "perfect API" is the trick these anti-Rust people use to stall things... ;;

Like they think it's the Rust team's job to fix the existing lifetime/recounting messes of the kernel before they're allowed to introduce safe APIs... even though an imperfect Rust abstraction that leaks some unsoundness in corner cases (due to underlying C problems or design) is still better than literally no Rust code at all and writing more C code...

Peter Brett

@avuton @lina I watched the video and… wow, I am so glad that I don't do FOSS stuff any more

gudenau

@lina Just because Rust is an awful language doesn't mean you shouldn't improve the C codebase...

dana :blobhaj_witch:

@lina @thephd the kernel is not alone in this. The biggest challenges in integrating c/c++ and rust are not technical ones.

PJB

@lina Every time I am forced to look at a large C codebase I see not only a gigantic pile of nightmares, I see a gigantic pile of nightmares created by people that refuse to accept the flaws of their tools and refuse attempts to change them.

To hear that Linux is the same is unsurprising but saddening at the same time.

Ugh.

JayF

re: last RT -- we have to be careful as open source projects to accept new ideas and include them, especially when people do the work to go through the process. New is scary, but running away anyone else who can help improve it is way, way scarier to me.

Pierre Bourdon

@lina that video linked in Wedson's retirement mail is... damning, to say the least. Wow. And the fact that this is apparently just acceptable behavior in that community...

Karl

@delroth @lina God, that's textbook gatekeeping.

Whoever uttered those words should be deeply ashamed (but they're not, I'm sure).

40whirl

@karl @delroth @lina apparently that's who maintains ext4 code :blobcat_dundundun:

Karl

@40whirl @delroth @lina You'd imagine someone maintaining such a widely used filesystem would be interested in the properties of a memory-safe language, especially given ext4's track record on the matter.

40whirl

@karl @delroth @lina i mean, even if they actively hate rust and people writing rust, they should at least use valid points in an argument...

Imikoy

@delroth@mastodon.delroth.net It felt like the output of active-on-twitter US conservatives...

edit: the one-word description here is "reactionary"

Daniel Leigh

@delroth @lina what is it with prominent C developers insisting that there is no reason to learn anything new and that C is the only good way to write software? It isn't just kernel people, this is a huge issue on the userland side of open source too. This gatekeeping is super toxic, and leads to dramatically worse software overall.

Charadon

@delroth @lina As far as I know, not a lot of people like the guy who's making an ass of himself (He's the maintainer of ext4 iirc, his name is Ted Ts’o) and he's really only kept around because he maintains ext4.

If you noticed, no one clapped after he was done with his hissy fit ;)

FC (Fay) 🏳️‍🌈

@lina that video fragment... *sigh*

kinda reminds me of being forced to write ruby code that looks like python instead of idiomatic ruby because the other devs only knew python but still wanted custom jekyll plugins.

Schamschula

@lina As a MacPorts maintainer and a FreeBSD sysadmin, I run into two problems with rust:
1) It is unwieldy to maintain rust and cargo, so both are often out of date on MacPorts.
2) On FreeBSD some random library (e.g. librsvg) that depends on rust/cargo causes Poudriere to rebuild rust every time some build dependency is updated. This usually takes several hours on my build machine. For this reason, and that it adds a lot of extra packages, MacPorts still doesn't use librsvg built using rust.

Asahi Linya (朝日りにゃ〜)

@schamschula Every modern Linux distro has already figured out Rust packaging since it's required for a lot of modern packages, so I don't think that's a concern...

Schamschula

@lina Chuckles, I only use Linux on Raspberry Pis.
And yes, I could wait to download the packages from FreeBSD's build/package system but I prefer to keep my systems more current, and hence run Poudriere.

Håkon Lorentzen

@lina good lord that video is infuriating. I don't think I would be able to be so diplomatic.

Lux under a pile of leaves 🍂🦊🍂

@lina this brings back memories of a couple of us early Rust netdev folks trying to push for a branch where we would be allowed to experiment and share abstractions that just did not have any user yet, but could be improved and reworked by people who eventually picked them up for use. It took multiple hours-long meetings to finally ending up agreeing that we would just all share links to our respective code and pray that anymore who wasn't aware of it wouldn't have to reinvent the wheel again abstraction-wise.

I don't want to fall into tribalism, or paint all C (or even netdev) developers as stuck-up assh*les, far from that, but it's clear that the requirements imposed on Rust developers to adhere to the way things have always been done in C, and effectively considering Rust as a guest who should be grateful to be in the project, rather than a system capable of making informed decisions about other systems*, places a ton of constraints that just make our development harder and slows down the inertia to get the project to mature. Meanwhile, a lot of time has to be spent by the heads of the project (bless them) doing diplomacy because otherwise patches don't get reviewed, and C devs don't see our point.

Plus the random stubborn and very opinionated dude every now and then of course, but it's the linux dev community, ofc it'll happen.

- (eg. Rust being used must not motivate large changes on the C side, Rust has to reproduce the way things are done in C however bad they are etc)

@lina this brings back memories of a couple of us early Rust netdev folks trying to push for a branch where we would be allowed to experiment and share abstractions that just did not have any user yet, but could be improved and reworked by people who eventually picked them up for use. It took multiple hours-long meetings to finally ending up agreeing that we would just all share links to our respective code and pray that anymore who wasn't aware of it wouldn't have to reinvent the wheel again abstraction-wise.

Sobex

@SharpLimefox @lina Isn’t there a Rust-in-Linux GitHub repo, that could serve as a central point to keep track of all those abstraction and shared work ? (Possibly with more experimental branches).

Lux under a pile of leaves 🍂🦊🍂

@Sobex @lina there is a repo, it's where the previous work resides in the rust branch from before the merge in 6.1. That was where we would have wanted a branch, but review rules are what they are and we would have needed to go through LKMLs with netdev people and netdev people will not review code that is not aiming to be merged (which i understand, but then why force us through LKMLs?)

And it was all just diplomacy and sticking to how things are done

GhostOnTheHalfShell

@lina man, as someone with an embedded background, that stinks to high heaven. Everything starts and ends with lifecycle and ownership.

Sam

@lina I made the mistake of reading the phoronix comments about this news, won't be going there again. On average people are way too protective of how things have always been done. I know perfectly well from my career that "C is perfectly safe if you use it correctly" is just stupid gatekeeping that doesn't even work. I make mistakes as well and I want to be warned as early as possible without needing to add on additional static analysers (which still can't check everything).

Avuton Olrich

@annun @lina Yeah, Phoronix comment toxicity rates between YouTube and 4chan comments.

Yvan DS 🗺️ :ferris: :go:

@lina This is insane.

The 3 minutes video link is absolutely stupid to the highest level. Describing things as "religion" when it has nothing to do with that is stupidity and bike shedding.

And people wonder why CEO's are pushing hard to replace engineers by LLM's .................

I am sorry, I don't mind people not knowing, however, people refusing to learn, WTH isn't that our job ???
Here is langs I don't like that I learned:
- prolog
- nix
- python
- java
- c#
- haskell
- ada
- c++
- js

pkprotoplasm

@lina I truly wish I didn’t know who this post was about. Sorry for the trouble you were put through.

Greg Brooks

@lina Its just frustrating because he was gate keeping ultimately got his way and everyone else will suffer in the long run.

shine

@lina wow, watching that linked conference talk, people there are literally hostile. I can't even imagine how mentally exhausting it must be to try to build something in such an environment. I have huge respect for you and everyone who works on Rust in the Linux Kernel space.

New Divide

@lina I saw the video, ideals and opinions brought forth on the open. I don't have too keen an understanding, but I can definitely see how change can be difficult. Hopefully that bridge to communication is improved to be able to move forward, taking the concerns of both parties into account.

Simon

@lina the guy in the linked video is absolutely insane. pretending to be an "engineer" while at the same time being the only one to talk about "religion" when it comes to Rust. it's actual madness.

Antoine Büsch

@lina Holy crap, that youtube excerpt in the linked message is painful to watch :(

yugaego 🌻

@lina
> the maintainer blocked it and said I should just do "what other drivers do".

The maintainer must have been haunted by Grace Hopper after that.

youtube.com/watch?v=si9iqF5uTF

Tzafrir

@lina

Reading through the discussion in LWN, I see reasonable arguments as to why there are problems. See e.g. lwn.net/Articles/987817/ (a subthread in that discussion).

fluke

@lina It smacks of what I experienced as a heavy fabrication welder fitter turner. All over again. Old farts refuse to adopt newer safer practices because “we’ve always done it this way and it’s worked for us why do we need to change”.

Go Up