I wish there was an actual thriving business model like this -- just fixing most annoying bugs, for a price, of commonly used desktop software. Why proprietary software companies cannot or do not want to provide this service is over me. Perhaps I'm too much used to consulting.
Given that “fixing this issue required weeks of intensive work from multiple people”, the price would have to be prohibitively high.
More generally, software is really, really expensive to produce and maintain. The economics only work at scale, in particular for B2C. (Maybe AI will change that, if it becomes more reliable.)
For many large companies or even teams, there exists a class of bugs / issues / features where dropping 5-10k on a bounty is extremely cost efficient compared to working around the issue or internal development. That might not fund development outright, but at worst it would point out the features people want and serve to inform what to work on next. I think there are a couple reasons why that is not prevalent. Most important one is that highly compensated enterprise teams that would benefit the most from placing bounties tend to avoid software that is lacking features or has bugs. Secondary is not implemented here ego and general disconnect between people in the trenches that know what needs to be done and people controlling ability to place bounties.
Imagine FAANG assigning $500 per engineer per year to allocate to feature / bug bounties.
Bounties for security holes make sense because you don’t need to submit the patch, just find the hole.
And bounties for open source (like in this case) also make sense because you have everything you need to submit a patch.
But for everything else (like big tech, startups, and so on) bounties can’t fix bugs because even if I find a bug, how am I going to patch it without access to the source code? How can someone submit a patch to Netflix or whatever?
IME your average SV startup has a long list of bugs they are aware of, but just haven’t gotten around to fixing because other priorities are in the way. But people can’t help patch unless you have an open development process.
You can fix bugs without source lots of ways, although many are arcane and finicky. An example of a healthy and productive ecosystem for this is in game modding. Sometimes this relies on vendor supplied tools (like a modkit, e.g. Elder Scrolls games), messing with bytecode directly (Minecraft until recently), or some cooperation from the vendor (Dwarf Fortress).
In all of those cases users/players were able to fix bugs and add desired functionality (mostly) independently, on a closed-source program.
For industrial software you don't see as much, even though arguably cracks (to skip license check) qualify here.
Most larger companies would probably find it way easier and more sensible to contract with some outside consultancy to work on these issues than just posting a random bounty, even if the latter might potentially be cheaper. See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
Yes, when my org needed a very specific feature from an open source project the company reached out to the authors. I don’t know the terms, but they dropped a chunk of cash. No strings either on the new feature and everyone benefited in the end.
> See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
That potential for failure is there for any "subcontractors". I wonder if anyone has any stats on this.
While you are completely correct about the bounty price, sometimes there are people who work deeply in the field and can solve those things relatively fast because they have already done similar things in the past.
Especially if you’re talking about a business who takes on these types of bounties routinely. I imagine you’d be able to build up a body of historical knowledge about fixing common issues. You could see how that could be a viable business model.
Eh, I think you're underestimating some people perseverance.
You generally only need multiple people for timely action, and it usually even slows you down (from the perspective of total hours spent)
Like 2k bug bounty? I guarantee you some people would be willing to spend a lot of time for that. But yeah, people which are gainfully employed and have a decent salary - likely not.
People will have fun spending their free time on such projects. But it’s virtually impossible to turn it into “an actual thriving business model” that people can make a living on.
This $1900 bug bounty is quite an outlier, you generally won’t find one per month. An additional challenge is that it’s hard to predict how much work something will take, or whether there are any showstoppers. Also, if you don’t live in the same country as the client, it will be more difficult to get legal assurance that you’ll receive your money (or for the client that they won’t lose their money).
For small stuff, the cost is just going to be too much for people to want to pay it. This bug had a $1900 bounty attached. Let's put the cost of one software engineer (salary plus overheads) at $200,000 a year, which I think is an underestimate. That's $3850 a week, so unless your bug can definitely be fixed (including getting any necessary hardware, investigation, fixing, code review overhead, etc) in two or three days it doesn't pay. And if it could obviously be done in two days then it's likely somebody would have already done that.
The above back of envelope maths ignores the overheads of interacting with the people who posted the bounties to get them to agree to pay up, and of the cost overruns on the class of bugs that look like two day fixes but take two weeks.
I assumed the commonly cited 2x markup, so that would be a $100k salary, which is less than various websites say is the average US software dev salary. You could probably find cheaper elsewhere in the world, but even if you cut the salary in half that's still "bug must be doable in a week", which isn't going to cover many of the bugs people will care about.
Paying for software developers is really weird. State governments for example struggle to pay for a FTE that makes $140k. But they can pay me over $200/hour for consulting services for multiple years. The technical FTE employees that they have generally aren't qualified to evaluate their consulting needs so you get multi-million dollar contracts with very little actual oversight. I was really impressed with the folks I was working with at this particular state government and looked into what it would look like if I joined them full time as a FTE technology leader. I would have to take almost a 50% pay cut. The top senior IT position that oversees all of the state resources makes 70% of what I do. It's crazy. Unless you're working in medicine or sports, government pay sucks.
I've seen similar but less extreme examples play out in the private sector. 16 year senior architect making less than freshly hired software dev that was just an intern within the same company. Software developer pay is largely based on what you're demanding. In a lot of companies, there is a wide range of pay for folks doing literally the same job. They will hire a dev at $180k because that dev wouldn't go lower and turn around and push back to get another dev at $120k for the same level of unproven experience.
200k is a fairly high salaried software eng in expensive markets, a bounty program like this would be open worldwide and many people would be willing to work for a fraction of that, quality control is another concern but take a look at prices on sites like upwork and bids for this type of work and realize 200k is nowhere near the lower baseline.
$200k in cost to the company is a lot different than $200k in salary. It probably relates to someone making like $140k, depending on the various tax rates.
$200k is on the extreme high-end of software engineers. For example in eastern europe $30k is normal. And that's not even the floor. You can go to india or africa to get even cheaper. The problem with this bug bounty though is that it requires pretty rare expertise. It's not a "throw any developer at it" type of thing.
Did you realize that you didn't include 'open source' in your statement? This is exactly what the desktop OS makers -Microsoft and Apple- do every single day. Their prices are mostly B2B and therefore hidden, but there is a steady income for each person involved in making the fix.
it's the management structure that's broken. Plenty of decent engineers around microsoft who could fix it, plenty of customer and enterprises willing to pay, but they are not allowed to work on it because of prioritization bullshit, allegedly they could get more money elsewhere
That's literally the issue, management by KPI frameworks
I think it has more to do with bundling reducing the need to compete to zero. Change that and the economics of competition would take over and the changes would get prioritized but nobody at Teams needs to sell a single license, so the priorities become the bs like internal status and visibility and not product success.
How many companies have Teams for basically free with their 365 license but still pay for Slack? The marginal value of Teams is nearly zero.
There is also a matter of selective effort by staff senior enough to make their own choices. Many SDE3 (or whatever MS equivalent is) wouldn’t want to be associated with a dumpster fire product like Teams.
The economics for something like MS teams is not what you'd expect.
It has to be good enough that other options are not worth the hassle to switch over to, for enterprise customers. The quality doesn't matter in the slightest, because making it 5-10% better would cost double or triple.
Where quality does matter for these customers, backward compatibility, Microsoft does pretty well.
Most recently I had it put meetings on a different day because something was broken with it's outlook integration w.r.t starting the week on a Sunday vs Monday.
"Lots" is a relative term, but the overwhelming majority of kernel developers are employed and usually do kernel work as part of their job (usually at least ~80% but it could be argued as high as 97% depending on how you interpret the breakdown done by LWN of each release[1]).
And I would guess that most of the kernel devs who are "working for free" are doing the stuff they personally enjoy and find satisfaction in working on, because it's a hobby -- so many of them are probably not interested in fixing random bugs for cash either.
Since you are talking about proprietary software, I assume you mean fixing bugs by the corpo devs themselves.
Well, this would imply broken software. You already payed for the software, now you are required to pay to get bugs fixed? Bad optics, although not beyond contemporary sentiments... Inherently shady incentives: https://en.wikipedia.org/wiki/Perverse_incentive
This kinda only works best for FOSS, incentivizing external devs IMO.
Sort of unrelated, but I've been thinking a lot of founding a non-profit that fund raises just to undercut the usual shitty consultancy companies that build government websites and apps just to build them properly.
I wish there was regulation that you have to sell and maintain a working product, so that open source devs don't have to waste their time fixing proprietary products.
It looks like these laptops are usually sold with Windows; are you saying that every manufacturer should be obligated to develop drivers for every software which is theoretically compatible with it? Or are you just saying that we need even more caveats in the interminable EULAs we all just click through?
Maybe the obligation should be to provide adequate information about the hardware, so anyone could make a driver for their own software if they so desire.
Yeah, you'd want some sort of micro-kickstarting website where users can pool money that goes into paying for some fix or feature if the committed money crosses a threshold.
Out of all bugs and feature requests, this one is an outlier in that it requires specific hardware to work on and has an obvious success condition. This means that every man and his dog is not going to be throwing an LLM at this to see if their particular slop wins the prize. People get weird when money is on the line and managing a bounty is a job for which I would never volunteer.
People spam the most minimal viable patch to collect the bounty and move on. And these days they are sending an AI slop solution. It doesn’t promote good code like actually hiring someone.
I'd gladly pay a couple hundred to have Swift-like optionals in Godot's GDScript, among other things that are just a pain to convince all the random idiots on their official spaces of, but GitHub doesn't have a way to offer that :(
That thread is a fun (though frustrating for them!) conversation to read through.
After about a hundred back-and-forths getting the guy with the actual hardware to try different commands, I was thinking to myself man, maybe he should just give him remote access to work on the target PC, this is torture for both of them. And then I see him comment:
> Honestly I'm thinking of this and maybe something insane like organizing ssh access or something to quit torturing Nadim with building and rebooting all the time
And Nadim replies:
> Haha, sorry, but there's no way I'm giving you SSH access!
> I’m fine with continuing with tests!
Which is fair enough! But was funny to see right when I was thinking the same thing. Great perseverance from both of them.
Was slightly disappointing they they moved off GitHub to Discord eventually so after all that, we miss the moment of them actually getting it working!
This has been an old problem with Legion laptops. All this will be available free of cost to everyone! Mad respect for people who are pledging their own money and the person who fixed it.
Also, Lenovo Legion Pro 7* are not cheap (not that this would have been justified for cheap laptops).
Shame on Lenovo/<big company> who should have fixed this years ago.
I heard from a frontier coder at deepmind that Gemini, whilst not great at novel frontier coding, is actually pretty good at debugging. Wonder if someone is going to automate a bug bounty pipeline with AI tools.
Intel HDA was supposed to be a better standard than the AC'97 it was meant to replace. IMHO the blame lies solely on the codec makers for not working with the default settings (they can add additional functionality, but the base audio I/O should work with a generic HDA driver.)
There is a common problem with Realtek ALC3306 on Linux (Kernel Bug 213159). This affects many Lenovo laptop models. For example, my fairly old Legion S7 15IMH5 laptop also does not work.
I'm not willing to pay $1000 for a fix (it's easier for me to buy a new laptop that will work with Linux), but $100 is probably okay. :)
If there's some Lenovo EU rep with enough budget for their department, this person should really be contracted to fix the audio from their laptops one by one. This seemed to take 2 weeks with a dedicated person and remote access but debugging with access to the device could be much faster!
Those two weeks probably would've been two days had the main developer been able to reboot and test the device directly.
Outsourcing this work to outside developers on the regular probably would make the media claim something like "Lenovo is too lazy to pay their people to make their speakers work so they make strangers on the internet do it instead" which isn't half false.
It'd probably be better and cheaper if they'd just hire someone to make their speakers work on Linux.
> Approximately 95% of the engineering work was done by Lyapsus. Lyapsus improved an incomplete kernel driver, wrote new kernel codecs and side-codecs, and contributed much more. I want to emphasize his incredible kindness and dedication to solving this issue. He is the primary force behind this fix, and without him, it would never have been possible.
> I (Nadim Kobeissi) conducted the initial investigation that identified the missing components needed for audio to work on the 16IAX10H on Linux. Building on what I learned from Lyapsus's work, I helped debug and clean up his kernel code, tested it, and made minor improvements. I also contributed the solution to the volume control issue documented in Step 8, and wrote this guide.
IF $1900 is the bounty, it means it doesn't hurt enough.
I remember going for the highest paying bounty in the Ethereum VM several years ago (I think it was ~$400 DAI/SAI). I did it because I wanted to force myself to learn the internals and to see for myself if the bounty system works. I think I spent a few weeks debugging and ended up splitting the bounty.
As long as the user-facing issues are disconnected from the technical issues, it's going to be hard to get the true value.
Many of these firmware blobs are non redistributable, if you require this blob, you should grab it and back it up. I know a lot of firwmare can't be legally distributable in linux-firmware.
Lenovo may not be as friendly as IBM to its opensource.
I have a couple old-ish Samsung Galaxy Book x86 tablets that have a similar issue, that I have never quite goaded myself into trying to reverse engineer. I'd love some better material on trying to reverse engineer windows drivers: presumably maybe running windows in qemu with some kind of intercepting pass through?
Actually, the mercenary garbage that Lenovo started doing was absolutely hateful. I'd been tearing my hair out forever trying to figure out why nothing I could do would make my bluetooth work right on an old T430, even when I upgraded the chip. Assumed it was a Linux bug.
Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.) I have no idea why you would do that except out of spite; I don't remember them selling bluetooth upgrades or anything. They were just keeping their options open? This is aside from having to hack the bios in order to upgrade wireless or use generic batteries.
I would be awesome if the people that sold me products weren't awful people. They don't have to feel bad about it, but they should.
I don't care that you're angrier at Apple than at Lenovo. I'm angrier at the electric company, but I don't bring it up to defend my local alderman. I also don't care that FOSS hasn't solved all your problems for free like they apparently promised you they would.
> Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.)
I used to Hackintosh Lenovos -- I thought this was at the bios level, so even if you did DSDT patching (linux or mac wise) it wouldn't work?
I use a T530, which is the larger brother of the T430. My overall impression is that it's a fine machine. (A lot of that opinion is motivated by the fact that when I bought it, it represented the very last of the plain black business-ey 15" PC laptops that wasn't cursed by the inclusion of a numeric keypad. Nowadays, Framework might fill that niche.)
Anyway: IIRC, a lot of the reason for locking down wireless hardware support is the FCC. AFAIK, the machine is tested and certified as a whole: The entire combination of its chassis and antennas and shielding and transmitting radios forms the item-under-test.
It's not just the sum of its parts; the whole thing gets tested. Deviation by end-users can result in having a non-compliant device, and the BIOS seeks to restrict that deviation.
And no, I don't like that aspect either.
But it only cares about the radios[1]. The rest of the guts are pretty freely exchanged: Upgrade to a different optical drive? SSD? Hard drive? RAM? CPU? Hack in a different keyboard? It's not picky at all about those things; have at it. (They could have locked down the entire system so that only parts matching a secret sauce would work, but they didn't.)
Anyway, Bluetooth module is garbage even under Windows. It's a bad design.
I bought my T530 from some random seller on eBay. It was obviously rebuilt by an outfit where they have a pile of variously-destroyed computers and take the cleanest chassis and put it with the fanciest screen and the nicest motherboard and sell it with some manner of RAM and an SSD for as much as they can get, even if that particular combination of stuff was never sold by Lenovo.
And that's fine, except: The Bluetooth module didn't work. I became convinced that it didn't even have one, even though it was advertised as including Bluetooth. So I bought a Bluetooth module (after validating the correct Lenovo FRU from the service manual, of course) and tore the thing apart to install it.
And once I was in there, I discovered that there was already a Bluetooth module present. It just wasn't installed properly.
IIRC there's one screw, one alignment pin, and one board-to-board connector for that part. The screw was tight, the pin was lined up fine, but the connector wasn't quite seated. It looked OK at a glance, but the whole module, in-situ, was very slightly bent by doing all 3 of those things concurrently.
That was annoying because only a bad design would have allowed for this to happen in the first place.
But I put my "new" non-bent module in and... it worked. (I haven't used it in Linux yet, though. Day job requires Windows software that talks to external hardware and I don't like dual-boot systems, so I'm kind of stuck.)
In terms of upgrading that part: It looks like the next wireless card is almost certain to also include Bluetooth by default, so after I hack the BIOS (I have a flashy-tool to poke at it with and an entire spare motherboard) to get around the trickery and plug a different wifi card in, I'll also have a newer Bluetooth radio.
[1]: I deliberately didn't mention batteries. My T530 still has the official Lenovo-minted BIOS; I've even updated it myself at one point using a Lenovo download. But it came to me with an aftermarket battery that worked great for a couple of years, and it now has a different aftermarket battery that also works great. I've heard the storied ruminations about aftermarket battery woes but simply have not experienced them myself. Indeed, the positive reviews on the Amazon listing I bought from suggests that it wasn't an issue for any of those folks, either -- it was like a sea of people who were just blissfully unaware of the issue.
I'm guessing it's the fact that linux has in-tree drivers, so you necessarily need to "patch the kernel" in order to write/fix a driver for a non-standard compliant device?
I can’t see any blocker to publishing this as a prebuilt kernel module honestly.
For driver developers the above where you rebuild the kernel is a necessary step in developing the driver but now the above is done someone should make the trivial next step to make this into a prebuilt kernel module which are trivial to install for end users with no rebuild/reboot required. (I have built kernel modules before but I don’t have this laptop myself, sorry!).
That's how it works in Linux land for two reasons. One, drivers live in the kernel (roughly). Two, Linux is aftermarket for many hardware, in which cases there's hardware first, then the support.
I agree. I can’t see any reason this couldn’t be packaged as a prebuilt kernel
module so end users can trivially install it. The instructions and code here can be used to build the kernel module.
I don’t have this laptop but have built kernel modules in the past to give context. It’s a tiny step to publish this as a kernel module so end users can trivially install this (this reduces the instructions to downloading one file, running one command, with no reboot or rebuild needed) so it’s quite reasonable to call this out and ask someone to do it.
It’s a bit like publishing a windows driver as raw source code. Great work but there’s no reason not to ship the prebuilt driver right?
> I can’t see any reason this couldn’t be packaged as a prebuilt kernel module so end users can trivially install it.
I don't think you have to be the original developer to create packages one can distribute. Go for it!
From TFA:
> This guide is currently for Linux kernel version 6.17.8. It will be updated for future kernel versions as they are released, until the fix is fully integrated into the kernel.
So it sounds like the plan is to get it into the mainline kernel, at which point it will get to all the distributions. So I sent see the problem here.
Some device classes can be supported in userspace because no matter how an adversarial driver might get the device to misbehave, it cannot possibly break the kernel's security model. This might even apply to some audio devices, depending on how exactly they're hooked up to the rest of your system. But the more typical devices, especially those in your average SoC and those connected to a PCIe bus or the like, have full privileges within the system and will need kernel-level support for the foreseeable future.
Kernel modules absolutely run in kernel space though.
I’ve literally written kernel modules for high speed networking devices that have full access to the memory bus and enumerate pci devices. There’s no userspace or kernel space question here. It’s merely a matter of someone turning this into an easily installable kernel module
Kernel modules are not going to be "easily installable" anyway because their whole purpose is to poke at kernel-internal structures that will change all the time as the kernel evolves. With source code, you'll hopefully get notified if there is breakage - the module fails to build and you need to forward-port it to the current kernel.
They have great stability between kernels by design. Better than Windows dll based drivers IMHO.
As someone who actually writes drivers I'm a little frustrated at this whole thread with people claiming Linux drivers have to be distributed this way.
Kernel modules exist for a reason, literally to allow end users as easy and as forwards compatible of a way to install drivers as windows dll based drivers. This whole thread has a lot of know nothings chiming in if I'm blunt.
I wish there was an actual thriving business model like this -- just fixing most annoying bugs, for a price, of commonly used desktop software. Why proprietary software companies cannot or do not want to provide this service is over me. Perhaps I'm too much used to consulting.
Given that “fixing this issue required weeks of intensive work from multiple people”, the price would have to be prohibitively high.
More generally, software is really, really expensive to produce and maintain. The economics only work at scale, in particular for B2C. (Maybe AI will change that, if it becomes more reliable.)
For many large companies or even teams, there exists a class of bugs / issues / features where dropping 5-10k on a bounty is extremely cost efficient compared to working around the issue or internal development. That might not fund development outright, but at worst it would point out the features people want and serve to inform what to work on next. I think there are a couple reasons why that is not prevalent. Most important one is that highly compensated enterprise teams that would benefit the most from placing bounties tend to avoid software that is lacking features or has bugs. Secondary is not implemented here ego and general disconnect between people in the trenches that know what needs to be done and people controlling ability to place bounties.
Imagine FAANG assigning $500 per engineer per year to allocate to feature / bug bounties.
I’m confused.
Bounties for security holes make sense because you don’t need to submit the patch, just find the hole.
And bounties for open source (like in this case) also make sense because you have everything you need to submit a patch.
But for everything else (like big tech, startups, and so on) bounties can’t fix bugs because even if I find a bug, how am I going to patch it without access to the source code? How can someone submit a patch to Netflix or whatever?
IME your average SV startup has a long list of bugs they are aware of, but just haven’t gotten around to fixing because other priorities are in the way. But people can’t help patch unless you have an open development process.
Am I missing something?
You can fix bugs without source lots of ways, although many are arcane and finicky. An example of a healthy and productive ecosystem for this is in game modding. Sometimes this relies on vendor supplied tools (like a modkit, e.g. Elder Scrolls games), messing with bytecode directly (Minecraft until recently), or some cooperation from the vendor (Dwarf Fortress).
In all of those cases users/players were able to fix bugs and add desired functionality (mostly) independently, on a closed-source program.
For industrial software you don't see as much, even though arguably cracks (to skip license check) qualify here.
Most larger companies would probably find it way easier and more sensible to contract with some outside consultancy to work on these issues than just posting a random bounty, even if the latter might potentially be cheaper. See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
Yes, when my org needed a very specific feature from an open source project the company reached out to the authors. I don’t know the terms, but they dropped a chunk of cash. No strings either on the new feature and everyone benefited in the end.
> See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
That potential for failure is there for any "subcontractors". I wonder if anyone has any stats on this.
While you are completely correct about the bounty price, sometimes there are people who work deeply in the field and can solve those things relatively fast because they have already done similar things in the past.
Especially if you’re talking about a business who takes on these types of bounties routinely. I imagine you’d be able to build up a body of historical knowledge about fixing common issues. You could see how that could be a viable business model.
Eh, I think you're underestimating some people perseverance.
You generally only need multiple people for timely action, and it usually even slows you down (from the perspective of total hours spent)
Like 2k bug bounty? I guarantee you some people would be willing to spend a lot of time for that. But yeah, people which are gainfully employed and have a decent salary - likely not.
People will have fun spending their free time on such projects. But it’s virtually impossible to turn it into “an actual thriving business model” that people can make a living on.
Why not? In much of the world, working on one of those a month would provide a comfortable living.
This $1900 bug bounty is quite an outlier, you generally won’t find one per month. An additional challenge is that it’s hard to predict how much work something will take, or whether there are any showstoppers. Also, if you don’t live in the same country as the client, it will be more difficult to get legal assurance that you’ll receive your money (or for the client that they won’t lose their money).
You bought up a lot of points. And I think they are all negligible, compared to the gigantic elephant in the room.
Which is, in order for some rando to fix the bug; a company would need to give access to their codebase to some rando.
And they don't wanna do that.
lt could become some sort of leetcode final boss and/or something that you can put on your resume.
And also scarce skills.
For small stuff, the cost is just going to be too much for people to want to pay it. This bug had a $1900 bounty attached. Let's put the cost of one software engineer (salary plus overheads) at $200,000 a year, which I think is an underestimate. That's $3850 a week, so unless your bug can definitely be fixed (including getting any necessary hardware, investigation, fixing, code review overhead, etc) in two or three days it doesn't pay. And if it could obviously be done in two days then it's likely somebody would have already done that.
The above back of envelope maths ignores the overheads of interacting with the people who posted the bounties to get them to agree to pay up, and of the cost overruns on the class of bugs that look like two day fixes but take two weeks.
$200k is one expensive software engineer. On average, you can get people to work for much less.
I assumed the commonly cited 2x markup, so that would be a $100k salary, which is less than various websites say is the average US software dev salary. You could probably find cheaper elsewhere in the world, but even if you cut the salary in half that's still "bug must be doable in a week", which isn't going to cover many of the bugs people will care about.
I believe that the $200k figure was meant to express what such a person might cost the company, not what that person would be paid as salary.
(And it's just a placeholder. $200k seems like it's at least in the direction of the right ballpark.)
Paying for software developers is really weird. State governments for example struggle to pay for a FTE that makes $140k. But they can pay me over $200/hour for consulting services for multiple years. The technical FTE employees that they have generally aren't qualified to evaluate their consulting needs so you get multi-million dollar contracts with very little actual oversight. I was really impressed with the folks I was working with at this particular state government and looked into what it would look like if I joined them full time as a FTE technology leader. I would have to take almost a 50% pay cut. The top senior IT position that oversees all of the state resources makes 70% of what I do. It's crazy. Unless you're working in medicine or sports, government pay sucks.
I've seen similar but less extreme examples play out in the private sector. 16 year senior architect making less than freshly hired software dev that was just an intern within the same company. Software developer pay is largely based on what you're demanding. In a lot of companies, there is a wide range of pay for folks doing literally the same job. They will hire a dev at $180k because that dev wouldn't go lower and turn around and push back to get another dev at $120k for the same level of unproven experience.
They give up pay for guaranteed work and benefits, maybe a pension? Most likely little risk of being fired or laid off.
You have to keep finding clients (I'm sure it's easy now, will it always?) and pay all your expenses.
You don't need to hire American expensive, but not so productive, engineers, there are lot of other countries. Also, there are ML models.
200k is a fairly high salaried software eng in expensive markets, a bounty program like this would be open worldwide and many people would be willing to work for a fraction of that, quality control is another concern but take a look at prices on sites like upwork and bids for this type of work and realize 200k is nowhere near the lower baseline.
$200k in cost to the company is a lot different than $200k in salary. It probably relates to someone making like $140k, depending on the various tax rates.
also, don't forget to include QA and release management overhead, as well as projectmanagement etc...
the 60k buffer probably just covers the salaries of the multiple layers of management and facilities (building, cleaning...)
$200k is on the extreme high-end of software engineers. For example in eastern europe $30k is normal. And that's not even the floor. You can go to india or africa to get even cheaper. The problem with this bug bounty though is that it requires pretty rare expertise. It's not a "throw any developer at it" type of thing.
You are forgetting that typically many users want a bug fixed.
Did you realize that you didn't include 'open source' in your statement? This is exactly what the desktop OS makers -Microsoft and Apple- do every single day. Their prices are mostly B2B and therefore hidden, but there is a steady income for each person involved in making the fix.
and yet, Microsoft Teams is a total trash fire full of bugs that users hate. So something is broken (Teams. It's Teams that is busted).
it's the management structure that's broken. Plenty of decent engineers around microsoft who could fix it, plenty of customer and enterprises willing to pay, but they are not allowed to work on it because of prioritization bullshit, allegedly they could get more money elsewhere
That's literally the issue, management by KPI frameworks
I think it has more to do with bundling reducing the need to compete to zero. Change that and the economics of competition would take over and the changes would get prioritized but nobody at Teams needs to sell a single license, so the priorities become the bs like internal status and visibility and not product success.
How many companies have Teams for basically free with their 365 license but still pay for Slack? The marginal value of Teams is nearly zero.
There is also a matter of selective effort by staff senior enough to make their own choices. Many SDE3 (or whatever MS equivalent is) wouldn’t want to be associated with a dumpster fire product like Teams.
The economics for something like MS teams is not what you'd expect.
It has to be good enough that other options are not worth the hassle to switch over to, for enterprise customers. The quality doesn't matter in the slightest, because making it 5-10% better would cost double or triple.
Where quality does matter for these customers, backward compatibility, Microsoft does pretty well.
I have used it every day for past 3-4 years. What bugs? I don’t love it but I don’t hate it either. I don’t understand the Teams hate
Most recently I had it put meetings on a different day because something was broken with it's outlook integration w.r.t starting the week on a Sunday vs Monday.
If you had made the same complaint about Win11 and you wouldn't be so far off. Microsoft is great at driver support which is the subject at hand.
I think that 2k is really really cheap for the expertise in kernel development
It is, but it's amazing how cheap kernel expertise is relative to comparable experience in other specialties like frontend.
there are a lot more kernel programmers than kernel work
But also lots of kernel developers work for free, so the average price of their work is very low
"Lots" is a relative term, but the overwhelming majority of kernel developers are employed and usually do kernel work as part of their job (usually at least ~80% but it could be argued as high as 97% depending on how you interpret the breakdown done by LWN of each release[1]).
[1]: https://lwn.net/Articles/1038358/
And I would guess that most of the kernel devs who are "working for free" are doing the stuff they personally enjoy and find satisfaction in working on, because it's a hobby -- so many of them are probably not interested in fixing random bugs for cash either.
Well, if one person spent a month on this, they’d be making about $10/hr.
Makes StarBucks barista pay look good…
Of course, if they can churn this out closer to 2 days, maybe there is something there.
Such a talented person would probably prefer a more certain and higher income.
For a lot of people in the world $10/hr is a fantastic wage. And you get to work at your own pace, probably from home.
And for a lot of people it is fantastic to have one slice of bread per day. What is your point?
I don't think this argument is accurate. There are other reasons to do this work even for free such as self-promotion, community-building, hobby, etc.
I think the real blockers are the legal implications of reverse engineering.
Since you are talking about proprietary software, I assume you mean fixing bugs by the corpo devs themselves.
Well, this would imply broken software. You already payed for the software, now you are required to pay to get bugs fixed? Bad optics, although not beyond contemporary sentiments... Inherently shady incentives: https://en.wikipedia.org/wiki/Perverse_incentive
This kinda only works best for FOSS, incentivizing external devs IMO.
There is some—VueScan for example, where the developer reverse-engineer scanning protocols, re-implement and sell it.
Sort of unrelated, but I've been thinking a lot of founding a non-profit that fund raises just to undercut the usual shitty consultancy companies that build government websites and apps just to build them properly.
I wish there was regulation that you have to sell and maintain a working product, so that open source devs don't have to waste their time fixing proprietary products.
It looks like these laptops are usually sold with Windows; are you saying that every manufacturer should be obligated to develop drivers for every software which is theoretically compatible with it? Or are you just saying that we need even more caveats in the interminable EULAs we all just click through?
Maybe the obligation should be to provide adequate information about the hardware, so anyone could make a driver for their own software if they so desire.
The problem is one-offs don't make steady, predictable, recurring revenue. Owning a consulting business is hard: you have to have customers waiting.
Yeah, you'd want some sort of micro-kickstarting website where users can pool money that goes into paying for some fix or feature if the committed money crosses a threshold.
Out of all bugs and feature requests, this one is an outlier in that it requires specific hardware to work on and has an obvious success condition. This means that every man and his dog is not going to be throwing an LLM at this to see if their particular slop wins the prize. People get weird when money is on the line and managing a bounty is a job for which I would never volunteer.
People spam the most minimal viable patch to collect the bounty and move on. And these days they are sending an AI slop solution. It doesn’t promote good code like actually hiring someone.
I'd gladly pay a couple hundred to have Swift-like optionals in Godot's GDScript, among other things that are just a pain to convince all the random idiots on their official spaces of, but GitHub doesn't have a way to offer that :(
I think the real issues are attributing work, and fear of doing a ton of work only to be pipped at the post.
The paperwork.
And the person who did the implementation, Lyapsus, did it without access to the hardware?? https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/i...
That thread is a fun (though frustrating for them!) conversation to read through.
After about a hundred back-and-forths getting the guy with the actual hardware to try different commands, I was thinking to myself man, maybe he should just give him remote access to work on the target PC, this is torture for both of them. And then I see him comment:
> Honestly I'm thinking of this and maybe something insane like organizing ssh access or something to quit torturing Nadim with building and rebooting all the time
And Nadim replies:
> Haha, sorry, but there's no way I'm giving you SSH access!
> I’m fine with continuing with tests!
Which is fair enough! But was funny to see right when I was thinking the same thing. Great perseverance from both of them.
Was slightly disappointing they they moved off GitHub to Discord eventually so after all that, we miss the moment of them actually getting it working!
I just read it too and now I think I know what the suspense novels robots write for each other will be like.
I also enjoyed reading through it, but wish I'd seen this comment first and avoided missing out on the moment of success too. :)
This has been an old problem with Legion laptops. All this will be available free of cost to everyone! Mad respect for people who are pledging their own money and the person who fixed it.
Also, Lenovo Legion Pro 7* are not cheap (not that this would have been justified for cheap laptops).
Shame on Lenovo/<big company> who should have fixed this years ago.
I heard from a frontier coder at deepmind that Gemini, whilst not great at novel frontier coding, is actually pretty good at debugging. Wonder if someone is going to automate a bug bounty pipeline with AI tools.
Can someone explain exactly what's happening here? https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/i...
It seems like there's a lot of personal information being asked for / thrown around... including a debit/credit card number?
Is there no better way to handle the bounty payment?
I guess this here is what the title is describing: https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/b...
Intel HDA was supposed to be a better standard than the AC'97 it was meant to replace. IMHO the blame lies solely on the codec makers for not working with the default settings (they can add additional functionality, but the base audio I/O should work with a generic HDA driver.)
There is a common problem with Realtek ALC3306 on Linux (Kernel Bug 213159). This affects many Lenovo laptop models. For example, my fairly old Legion S7 15IMH5 laptop also does not work.
I'm not willing to pay $1000 for a fix (it's easier for me to buy a new laptop that will work with Linux), but $100 is probably okay. :)
$100 gets you a usb sound card and $90 for something else. It's not a good solution, but it's easy.
If there's some Lenovo EU rep with enough budget for their department, this person should really be contracted to fix the audio from their laptops one by one. This seemed to take 2 weeks with a dedicated person and remote access but debugging with access to the device could be much faster!
Those two weeks probably would've been two days had the main developer been able to reboot and test the device directly.
Outsourcing this work to outside developers on the regular probably would make the media claim something like "Lenovo is too lazy to pay their people to make their speakers work so they make strangers on the internet do it instead" which isn't half false.
It'd probably be better and cheaper if they'd just hire someone to make their speakers work on Linux.
Oh it's written by Nadim Kobeissi, such a huge fan of his work didn't expect him see him here
In the README:
> Approximately 95% of the engineering work was done by Lyapsus. Lyapsus improved an incomplete kernel driver, wrote new kernel codecs and side-codecs, and contributed much more. I want to emphasize his incredible kindness and dedication to solving this issue. He is the primary force behind this fix, and without him, it would never have been possible.
> I (Nadim Kobeissi) conducted the initial investigation that identified the missing components needed for audio to work on the 16IAX10H on Linux. Building on what I learned from Lyapsus's work, I helped debug and clean up his kernel code, tested it, and made minor improvements. I also contributed the solution to the volume control issue documented in Step 8, and wrote this guide.
For those wondering:
> Sincere thanks to everyone who pledged a reward for solving this problem. The reward goes to Lyapsus.
I didn't mean he wrote the fix but the read me instead, looking back at my comment people might have assumed that Nadim made the fix
Maybe related:
- https://gist.github.com/felipelalli/6179aac72735fd35ea3a9854... - https://github.com/NixOS/nixos-hardware/issues/1039
Title should be: $2000 Bug Bounty to Fix the Lenovoe Legion Pro 7 16IAX10H's Speakers on Linux
IF $1900 is the bounty, it means it doesn't hurt enough.
I remember going for the highest paying bounty in the Ethereum VM several years ago (I think it was ~$400 DAI/SAI). I did it because I wanted to force myself to learn the internals and to see for myself if the bounty system works. I think I spent a few weeks debugging and ended up splitting the bounty.
As long as the user-facing issues are disconnected from the technical issues, it's going to be hard to get the true value.
Where are LLMs now?
They're not useful for fixing things like this. Only frontend React.js
You're wrong:
https://words.filippo.io/claude-debugging/
https://dmitrybrant.com/2025/09/07/using-claude-code-to-mode...
> Only frontend React.js
Good suggestion, but I discovered that React was not able to fix my Linux kernel, either, for some reason.
Have you tried asking an llm to use react to fix your Linux kernel?
Could an AI agent kidnap Torvalds and force him to do it?
Yes, and its shit.
Vibe coder: "ChatGPT, please fix the Lenovo Legion Pro 7 16IAX10H's Speakers on Linux"
Hal3000: "Great request—here is your React version 20XX* TODO list"
*20XX is a year+ old version of React
if there were data sheets available I expect they actually could do a bunch of the work here.
If only people would do these bounties for ITE sensor hubs.
Many of these firmware blobs are non redistributable, if you require this blob, you should grab it and back it up. I know a lot of firwmare can't be legally distributable in linux-firmware.
Lenovo may not be as friendly as IBM to its opensource.
Funnily enough the sound doesn't work properly on Windows 10 on this laptop. In exactly same manner as on Linux. Did open source win again?
Interestingly the Awinc aw88399 smart amplifier chip at the core of this issue allegedly got supporting in 6.7, in 2023. https://www.phoronix.com/news/Linux-6.7-Sound
I have a couple old-ish Samsung Galaxy Book x86 tablets that have a similar issue, that I have never quite goaded myself into trying to reverse engineer. I'd love some better material on trying to reverse engineer windows drivers: presumably maybe running windows in qemu with some kind of intercepting pass through?
I would stay away from anything that's using Nvidia, especially a laptop.
Here is another[1] fine read that was post recently. Author fixed media button issue for old 2005 Fujitsu keyboard and pushed it to kernel.
[1] https://news.ycombinator.com/item?id=45490652
[flagged]
People might be willing to interact with your idea of you removed the snark, if you think it's a discussion worth having.
I am literally unable to understand what's being said here
Angry racist shouting at people.
The machine easily cost a million to develop and it's easy to throw money at a problem for the unemployed tinkerers out there.
Actually, the mercenary garbage that Lenovo started doing was absolutely hateful. I'd been tearing my hair out forever trying to figure out why nothing I could do would make my bluetooth work right on an old T430, even when I upgraded the chip. Assumed it was a Linux bug.
Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.) I have no idea why you would do that except out of spite; I don't remember them selling bluetooth upgrades or anything. They were just keeping their options open? This is aside from having to hack the bios in order to upgrade wireless or use generic batteries.
I would be awesome if the people that sold me products weren't awful people. They don't have to feel bad about it, but they should.
I don't care that you're angrier at Apple than at Lenovo. I'm angrier at the electric company, but I don't bring it up to defend my local alderman. I also don't care that FOSS hasn't solved all your problems for free like they apparently promised you they would.
> Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.)
I used to Hackintosh Lenovos -- I thought this was at the bios level, so even if you did DSDT patching (linux or mac wise) it wouldn't work?
I use a T530, which is the larger brother of the T430. My overall impression is that it's a fine machine. (A lot of that opinion is motivated by the fact that when I bought it, it represented the very last of the plain black business-ey 15" PC laptops that wasn't cursed by the inclusion of a numeric keypad. Nowadays, Framework might fill that niche.)
Anyway: IIRC, a lot of the reason for locking down wireless hardware support is the FCC. AFAIK, the machine is tested and certified as a whole: The entire combination of its chassis and antennas and shielding and transmitting radios forms the item-under-test.
It's not just the sum of its parts; the whole thing gets tested. Deviation by end-users can result in having a non-compliant device, and the BIOS seeks to restrict that deviation.
And no, I don't like that aspect either.
But it only cares about the radios[1]. The rest of the guts are pretty freely exchanged: Upgrade to a different optical drive? SSD? Hard drive? RAM? CPU? Hack in a different keyboard? It's not picky at all about those things; have at it. (They could have locked down the entire system so that only parts matching a secret sauce would work, but they didn't.)
Anyway, Bluetooth module is garbage even under Windows. It's a bad design.
I bought my T530 from some random seller on eBay. It was obviously rebuilt by an outfit where they have a pile of variously-destroyed computers and take the cleanest chassis and put it with the fanciest screen and the nicest motherboard and sell it with some manner of RAM and an SSD for as much as they can get, even if that particular combination of stuff was never sold by Lenovo.
And that's fine, except: The Bluetooth module didn't work. I became convinced that it didn't even have one, even though it was advertised as including Bluetooth. So I bought a Bluetooth module (after validating the correct Lenovo FRU from the service manual, of course) and tore the thing apart to install it.
And once I was in there, I discovered that there was already a Bluetooth module present. It just wasn't installed properly.
IIRC there's one screw, one alignment pin, and one board-to-board connector for that part. The screw was tight, the pin was lined up fine, but the connector wasn't quite seated. It looked OK at a glance, but the whole module, in-situ, was very slightly bent by doing all 3 of those things concurrently.
That was annoying because only a bad design would have allowed for this to happen in the first place.
But I put my "new" non-bent module in and... it worked. (I haven't used it in Linux yet, though. Day job requires Windows software that talks to external hardware and I don't like dual-boot systems, so I'm kind of stuck.)
In terms of upgrading that part: It looks like the next wireless card is almost certain to also include Bluetooth by default, so after I hack the BIOS (I have a flashy-tool to poke at it with and an entire spare motherboard) to get around the trickery and plug a different wifi card in, I'll also have a newer Bluetooth radio.
[1]: I deliberately didn't mention batteries. My T530 still has the official Lenovo-minted BIOS; I've even updated it myself at one point using a Lenovo download. But it came to me with an aftermarket battery that worked great for a couple of years, and it now has a different aftermarket battery that also works great. I've heard the storied ruminations about aftermarket battery woes but simply have not experienced them myself. Indeed, the positive reviews on the Amazon listing I bought from suggests that it wasn't an issue for any of those folks, either -- it was like a sea of people who were just blissfully unaware of the issue.
Patching up the kernel to get some sound coming out of the speakers.. Very on brand for Linux.
How do you know somebody has no idea what they're talking about? They'll tell you.
deep
Show us on the doll where they’re wrong.
I'm guessing it's the fact that linux has in-tree drivers, so you necessarily need to "patch the kernel" in order to write/fix a driver for a non-standard compliant device?
I can’t see any blocker to publishing this as a prebuilt kernel module honestly.
For driver developers the above where you rebuild the kernel is a necessary step in developing the driver but now the above is done someone should make the trivial next step to make this into a prebuilt kernel module which are trivial to install for end users with no rebuild/reboot required. (I have built kernel modules before but I don’t have this laptop myself, sorry!).
How else is that supposed to work?
You either fix a driver in the kernel or a driver outside the kernel, it's not going to make that big of a difference to the person who has to fix it.
The difference is that the end user doesn't have to do it. Someone else is going to do it. Just like it is on Windows.
That only works because Windows has a large marketshare, the day it drops to 10%, users will have to write their own drivers as well
I think the original comment was suggesting that Linux typically has end-user visible bugs like sound not working, not commenting on where they live.
That's how it works in Linux land for two reasons. One, drivers live in the kernel (roughly). Two, Linux is aftermarket for many hardware, in which cases there's hardware first, then the support.
I agree. I can’t see any reason this couldn’t be packaged as a prebuilt kernel module so end users can trivially install it. The instructions and code here can be used to build the kernel module.
I don’t have this laptop but have built kernel modules in the past to give context. It’s a tiny step to publish this as a kernel module so end users can trivially install this (this reduces the instructions to downloading one file, running one command, with no reboot or rebuild needed) so it’s quite reasonable to call this out and ask someone to do it.
It’s a bit like publishing a windows driver as raw source code. Great work but there’s no reason not to ship the prebuilt driver right?
> I can’t see any reason this couldn’t be packaged as a prebuilt kernel module so end users can trivially install it.
I don't think you have to be the original developer to create packages one can distribute. Go for it!
From TFA:
> This guide is currently for Linux kernel version 6.17.8. It will be updated for future kernel versions as they are released, until the fix is fully integrated into the kernel.
So it sounds like the plan is to get it into the mainline kernel, at which point it will get to all the distributions. So I sent see the problem here.
Some device classes can be supported in userspace because no matter how an adversarial driver might get the device to misbehave, it cannot possibly break the kernel's security model. This might even apply to some audio devices, depending on how exactly they're hooked up to the rest of your system. But the more typical devices, especially those in your average SoC and those connected to a PCIe bus or the like, have full privileges within the system and will need kernel-level support for the foreseeable future.
Kernel modules absolutely run in kernel space though.
I’ve literally written kernel modules for high speed networking devices that have full access to the memory bus and enumerate pci devices. There’s no userspace or kernel space question here. It’s merely a matter of someone turning this into an easily installable kernel module
Kernel modules are not going to be "easily installable" anyway because their whole purpose is to poke at kernel-internal structures that will change all the time as the kernel evolves. With source code, you'll hopefully get notified if there is breakage - the module fails to build and you need to forward-port it to the current kernel.
They have great stability between kernels by design. Better than Windows dll based drivers IMHO.
As someone who actually writes drivers I'm a little frustrated at this whole thread with people claiming Linux drivers have to be distributed this way.
Kernel modules exist for a reason, literally to allow end users as easy and as forwards compatible of a way to install drivers as windows dll based drivers. This whole thread has a lot of know nothings chiming in if I'm blunt.