What happened to the driver in the main kernel? It barely works!

On April 26th, I send a patch to the linux-usb-devel@ mailing list that would (finally) intregrate the open source part of the Philips webcam driver into the main kernel tree. After waiting a while and seeing a few 2.4.5-pre kernels come by without my driver, I nudged Johannes Erdfelt, the current USB maintainer, a bit and he re-sent the patch, and indeed it made it into 2.4.5-pre5. At that point, Alan Cox (no introduction necessary) freaked out, and unpatched the most crucial part of the driver: the colour conversion routines.

What seems to be the problem? The camera natively delivers the YUV 4:2:0 colour format, but not many video tools understand that. So I wrote a couple of routines in both C and x86 assembly to convert to RGB and some other YUV look-a-likes. This allows the driver to work with 99.5% of all the tools out there. However, this conversion is done in kernel space, between the pwc core and the user-interface calls like read() and mmap(), and that is what Alan does not want. He says it should be done in the user programs, that they should be able to pick the 'smartest' format for them, and that kernel drivers should not contain this kind of crap (his words). He also claims this policy has been there since 2.0.

Now while I agree that the kernel may not be the best place to put this colour conversion routines in (and instead write an easy to use library for the video tools), I think it's simply unfair to break my driver for this reason alone, especially considering the following:

Alan says he is planning on removing the routines from the other drivers as well for some time. But so far he hasn't, and if my modified driver would go in, it would suddenly be a lot less functional than before, and even less functional than the other drivers! This is simply unacceptable to me: I will not have a crippled driver into the kernel. Therefor, I told Johannes to remove my driver from the source tree until all other drivers have their conversion routines removed; if this policy will be enforced, fine, but then it should be applied to all drivers equally. Frankly, I think this will not happen for a long time, because it will instantly break 90% of the video tools available to end-users (see the E-mail thread below).

I've pleaded, I've argumented, I've written long mails about it, and I cannot seem to get through... So far, I've tried not to get mad about this, and stayed polite and all that (this attitude is what got me the deal with Philips in the first place), but having 1.5 years of work rejected about some policy that never got enforced before, is really infuriating.


The full E-mail conversion related to this subject is reproduced here, so you can read for yourself. I've tried to put the mails in thread-order below.


Date: Wed, 23 May 2001 14:52:08 +0100 (BST)
From: Alan Cox <alan @ lxorguk.ukuu.org.uk>
To: (Linus Torvalds) <torvalds @ transmeta.com>
Subject: Re: Serious bad taste in pre5 patch
Cc: (Alan Cox) <alan @ lxorguk.ukuu.org.uk>,
 (Johannes Erdfelt) <jerdfelt @ valinux.com>, webcam @ smcc.demon.nl

> On Wed, 23 May 2001, Alan Cox wrote:
> > WTF did you merge it ?
> Because it was sent to me by the USB maintainer.
> > It needs serious brain damage removal before it should go into the tree
> I have nothing against an anti-patch, but maybe there should be some
> discussion (that I don't wan tto be part of ;)

Johannes this is the basic fix up patch. It does the following

- Make the driver compile with gcc 2.96/3.0  (fix incorrect abuse of ##)
- Remove various other warnings by making a couple of code paths clearer
- Remove all the format conversions
- Remove the conversion library
- Make the compressor symbols versioned. I can see what its trying to do with
  its binary modules but there is no guarantee the structures will be constant
  across builds/setups so it doesn't seem wise for the mainstream kernel

Format conversions are explicitly user space and this is very important both
from avoding kernel crap (count the number of '-' in the patch) and also to
apps because video conferencing tools _need_ to know which formats are native 
so they can make intelligent performance decisions. H.263 at 15fps is hard
enough without the driver lying and doing an expensive cache trashing 
reprocess of the data first.

[As an aside - it might make sense to move all the magic uncompressors into
 a library not kernel space and use private video formats to indicate them]

(replies set to exclude Linus as requested)

Alan

[patch snipped]

Date: Wed, 23 May 2001 10:53:06 -0700
From: Johannes Erdfelt <jerdfelt @ valinux.com>
To: alan @ lxorguk.ukuu.org.uk, webcam @ smcc.demon.nl
Subject: Re: Serious bad taste in pre5 patch

On Wed, May 23, 2001, Alan Cox <alan @ lxorguk.ukuu.org.uk> wrote:
> > On Wed, 23 May 2001, Alan Cox wrote:
> > > WTF did you merge it ?
> > Because it was sent to me by the USB maintainer.
> > > It needs serious brain damage removal before it should go into the tree
> > I have nothing against an anti-patch, but maybe there should be some
> > discussion (that I don't wan tto be part of ;)
> 
> Johannes this is the basic fix up patch. It does the following
> 
> - Make the driver compile with gcc 2.96/3.0  (fix incorrect abuse of ##)
> - Remove various other warnings by making a couple of code paths clearer
> - Remove all the format conversions
> - Remove the conversion library
> - Make the compressor symbols versioned. I can see what its trying to do with
>   its binary modules but there is no guarantee the structures will be constant
>   across builds/setups so it doesn't seem wise for the mainstream kernel
> 
> Format conversions are explicitly user space and this is very important both
> from avoding kernel crap (count the number of '-' in the patch) and also to
> apps because video conferencing tools _need_ to know which formats are native 
> so they can make intelligent performance decisions. H.263 at 15fps is hard
> enough without the driver lying and doing an expensive cache trashing 
> reprocess of the data first.

I think the patch is missing something. The patch you attached only removes
vcvt.h, vcvt_c.c and vcvt_i386.S. It doesn't seem to touch any Makefile's,
any symbol versioning or any macro's.

Aside from the actual patch, I agree with you. The color conversion in
kernel space was a bad precedent on my part. When I wrote the original
CPiA driver, the only applications available made horrible assumptions
about the device it was using and blindly assumed it would support RGB.

I'll go through the other USB v4l apps and see which ones we should remove
YUV -> RGB for as well.

> [As an aside - it might make sense to move all the magic uncompressors into
>  a library not kernel space and use private video formats to indicate them]

That might be reasonable. I'm not too worried about this since it is just
a driver afterall and the extra kernel memory wasted is at the discretion
of the user.

JE


Date: Fri, 25 May 2001 10:48:44 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: Johannes Erdfelt <jerdfelt @ valinux.com>
Subject: Re: Serious bad taste in pre5 patch

Greetings,

(I´m reading this after my previous mail to Alan)

On 23-May-01 Johannes Erdfelt wrote:
> 
> I think the patch is missing something. The patch you attached only
> removes
> vcvt.h, vcvt_c.c and vcvt_i386.S. It doesn't seem to touch any Makefile's,
> any symbol versioning or any macro's.
> 
> Aside from the actual patch, I agree with you. The color conversion in
> kernel space was a bad precedent on my part. When I wrote the original
> CPiA driver, the only applications available made horrible assumptions
> about the device it was using and blindly assumed it would support RGB.
> 
> I'll go through the other USB v4l apps and see which ones we should remove
> YUV -> RGB for as well.

Okay, if you do that, I´ll accept removal of the conversion code out of my
driver as well. But boy, are you going to get spammed.... :(

 - Nemosoft


Date: Fri, 25 May 2001 09:58:03 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: Alan Cox <alan @ lxorguk.ukuu.org.uk>
Subject: Re: Serious bad taste in pre5 patch
Cc: webcam @ smcc.demon.nl, (Johannes Erdfelt) <jerdfelt @ valinux.com>,
 (Linus Torvalds) <torvalds @ transmeta.com>

Greetings,

On 23-May-01 Alan Cox wrote:
>> On Wed, 23 May 2001, Alan Cox wrote:
>> > WTF did you merge it ?
>> Because it was sent to me by the USB maintainer.
>> > It needs serious brain damage removal before it should go into the tree
>> I have nothing against an anti-patch, but maybe there should be some
>> discussion (that I don't wan tto be part of ;)

Agreed. But I *do* want to be part of it.

> Johannes this is the basic fix up patch. It does the following
> 
> - Make the driver compile with gcc 2.96/3.0  (fix incorrect abuse of ##)

I´m not sure what you mean by ´abuse´, since what I do with ## is IMHO 
correct by reading the gcc documentation (and other examples).

> - Remove various other warnings by making a couple of code paths clearer

Patch does not seem to be complete...

> - Remove all the format conversions
> - Remove the conversion library
> 
> Format conversions are explicitly user space and this is very important
> both
> from avoding kernel crap (count the number of '-' in the patch) and also

Okay, now I´m pissed. I know all your arguments against format conversions
in kernel and I even partly agree, but this cannot be removed. Not without
breaking 90% of the tools out there. Call them brain-dead, call them stupid,
but they are out there, and in wide spread use.

> to
> apps because video conferencing tools _need_ to know which formats are
> native 
> so they can make intelligent performance decisions. 

Then fix the Video4Linux1 API so tools CAN get a good list of formats. All
you can do at the moment is try a bunch of palettes and see if the ioctl
call fails. Wow. Way to go.

> H.263 at 15fps is hard
> enough without the driver lying and doing an expensive cache trashing 
> reprocess of the data first.

My driver isn´t lying. You can query the palettes and make a decision. You
want YUV? It´s there. You prefer RGB? No problem. Some drivers didn´t even
tell you if the palette was supported, leaving programs in the dark.

I am against this anti-patch since it´s a serious degradation of the
functionality of the driver. If this goes in, the mail will pour in
endlessly about programs suddenly not working, or producing garbage
pictures. I really don´t want that. If it goes in, I will CC: Alan with
every mail and reply about this, including the explanatation: ¨I had a great
driver, but the kernel maintainers butchered it.¨

I´ve gone a long way to make sure the conversion is working reliable, fast,
and available in both C and assembly format, which is also properly working
on non-intel platform.  If you want to really fix this problem, then remove
the Video4Linux1 API and come up with something better. In the mean time, can
you please fix up the aprox 12 programs on my ¨Working stuff¨ page that
don´t understand the native YUV420 palette and don´t have a conversion
routine, and therefor will not work anymore? 

Face it, this is a legacy we will have to carry this along in the kernel for
a long time. If you really want to wake up developers of videotools, then
break all such tools, in such a grand manner that they HAVE to fix things
(if they care), but don´t pick this one just because I wanted to create the
best driver possible, and accomodating as much programs as possible. Fix the
problem, not the drivers.

Oh, and thank you for calling my code ´kernel crap´.

[deep breath]

> [As an aside - it might make sense to move all the magic uncompressors
> into
>  a library not kernel space and use private video formats to indicate
> them]

This sounds like a good idea, except for one little thing: a tool that wants
to support all cameras would either have to collect all uncompressor modules
on his site, and/or link through. This reminds me a bit of the situation with
xanim. But it could be done and it would solve at least one thing: the
module versioning problem.

> - Make the compressor symbols versioned. I can see what its trying to do
> with
>   its binary modules but there is no guarantee the structures will be
> constant
>   across builds/setups so it doesn't seem wise for the mainstream kernel

Binaries are always a problem; I think I made the impact as low as possible
(i.e. not using structures, only simple datatypes and pointers). Indeed a
compiler has the right to shuffle around variables in a structure, but that
won´t affect this decompressor.

The sad thing about module versionins is that I have never seen it work.
Trying to load a versions 2.4.3 compiled module on even 2.4.4 is simply
impossible... I was hoping I could cut down on compiling binary stuff by
making a module that could be loaded everywhere, and not compile for 2.4.0,
.1, .2, .3, etc.

Anyway, I hope we can come up with a sensible solution for this. Just
trashing the conversion code is unacceptable to me, and if you really want
to do this, I think I´ll have to reconsider supporting the driver
alltogether...

 - Nemosoft


A second thread started on linux-kernel after Alan's unpatch broke the compilation process for the pwc module; I replied somewhat sarcasticly:

Date: Fri, 25 May 2001 08:50:24 +0200
From: Norbert Preining <preining @ logic.at>
To: linux-kernel @ vger.kernel.org, webcam @ smcc.demon.nl
Subject: ac15 and 2.4.5-pre6, pwc format conversion

Hi!

According to ac ChangeLog:
o       Rip format conversion out of the pwc driver     (me)
        | It belongs in user space..

This change is included in 2.4.5-pre6, but
	drivers/usb/pwc-uncompress.c
still relies on this files:
gcc -D__KERNEL__ -I/usr/src/linux-2.4.5.6-packet/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe -mpreferred-stack-boundary=2 -march=k6 -DMODULE   -DEXPORT_SYMTAB -c pwc-uncompress.c
pwc-uncompress.c:25: vcvt.h: No such file or directory
pwc-uncompress.c: In function `pwc_decompress':
pwc-uncompress.c:158: warning: implicit declaration of function `vcvt_420i_rgb24'
pwc-uncompress.c:161: warning: implicit declaration of function `vcvt_420i_bgr24'
pwc-uncompress.c:164: warning: implicit declaration of function `vcvt_420i_rgb32'
pwc-uncompress.c:167: warning: implicit declaration of function `vcvt_420i_bgr32'
pwc-uncompress.c:171: warning: implicit declaration of function `vcvt_420i_yuyv'
pwc-uncompress.c:185: warning: implicit declaration of function `vcvt_420i_420p'

Best wishes

Norbert

PS: Please reply by email since I am not subscribed and I skim the
mailing list via the web archive.
THANKS.



Date: Fri, 25 May 2001 10:46:52 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: Norbert Preining <preining @ logic.at>
Subject: RE: ac15 and 2.4.5-pre6, pwc format conversion
Cc: linux-kernel @ vger.kernel.org

Greetings,

On 25-May-01 Norbert Preining wrote:
> Hi!
> 
> According to ac ChangeLog:
> o       Rip format conversion out of the pwc driver     (me)
>         | It belongs in user space..
> 
> This change is included in 2.4.5-pre6, but
>       drivers/usb/pwc-uncompress.c
> still relies on this files:
> gcc -D__KERNEL__ -I/usr/src/linux-2.4.5.6-packet/include -Wall
> -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe
> -mpreferred-stack-boundary=2 -march=k6 -DMODULE   -DEXPORT_SYMTAB -c
> pwc-uncompress.c
> pwc-uncompress.c:25: vcvt.h: No such file or directory
> pwc-uncompress.c: In function `pwc_decompress':
> pwc-uncompress.c:158: warning: implicit declaration of function
> `vcvt_420i_rgb24'
> pwc-uncompress.c:161: warning: implicit declaration of function
> `vcvt_420i_bgr24'
> pwc-uncompress.c:164: warning: implicit declaration of function
> `vcvt_420i_rgb32'
> pwc-uncompress.c:167: warning: implicit declaration of function
> `vcvt_420i_bgr32'
> pwc-uncompress.c:171: warning: implicit declaration of function
> `vcvt_420i_yuyv'
> pwc-uncompress.c:185: warning: implicit declaration of function
> `vcvt_420i_420p'

That´s what you get for ripping out the guts of a driver. Have a nice day.




Date: Fri, 25 May 2001 13:15:02 +0200
From: Erik Mouw <J.A.K.Mouw @ ITS.TUDelft.NL>
To: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: Norbert Preining <preining @ logic.at>, linux-kernel @ vger.kernel.org

On Fri, May 25, 2001 at 10:48:12AM +0200, Nemosoft Unv. wrote:
> On 25-May-01 Norbert Preining wrote:
> > According to ac ChangeLog:
> > o       Rip format conversion out of the pwc driver     (me)
> >         | It belongs in user space..
> >=20
> > This change is included in 2.4.5-pre6, but
> >       drivers/usb/pwc-uncompress.c
> > pwc-uncompress.c:185: warning: implicit declaration of function
> > `vcvt_420i_420p'
>=20
> That=B4s what you get for ripping out the guts of a driver. Have a nice=
 day.

The format conversion shouldn't be there in the first place. Format
conversion is policy, so it doesn't belong in kernel. Note for example
that none of the sound drivers does sample rate conversion although
some sound chips are locked at 48kHz only.


Erik



Date: Fri, 25 May 2001 14:21:19 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: Erik Mouw <J.A.K.Mouw @ ITS.TUDelft.NL>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: linux-kernel @ vger.kernel.org, Norbert Preining <preining @ logic.at>,
 linux-usb-devel @ lists.sourceforge.net

Greetings,

On 25-May-01 Erik Mouw wrote:
> On Fri, May 25, 2001 at 10:48:12AM +0200, Nemosoft Unv. wrote:
>> That´s what you get for ripping out the guts of a driver. Have a nice
>> day.
> 
> The format conversion shouldn't be there in the first place. Format
> conversion is policy, so it doesn't belong in kernel. Note for example
> that none of the sound drivers does sample rate conversion although
> some sound chips are locked at 48kHz only.

True, but a number of audio tools will break, because they don´t support that
samplerate (mpg123 0.59r, one of the more popular MP3 players, segfaults
when it has to do conversion of 44.1 KHz to 48 KHz). Recording from such a
source is usually less of a problem, although some post-processing is
then necessary (not all tools support samplerate conversion natively).

The situation for video devices is worse. 80% of the video tools will break
if you limit the number of available palettes per driver. Plus, you will get
severe fragmentation of which program works with which driver. Which is
unacceptable, in my opinion (and certainly NOT the idea behind a common API!)

You can blame the authors of those video tools, but that´s not really fair.
The original Video4Linux API was based upon TV grabber cards. Most of them
do YUV/RGB conversion in hardware, so most tools expected that all or most
palettes would always be available, since supporting a palette would not
require any extra CPU cycles [1]. Some tools like RGB, and others YUV, so
the authors happily selected the palette of their choice and never even
considered building in functions for doing conversion. And then I am not even
talking about the RGB/BGR mess.

Then along came parallel-port and USB webcams, which have a number of
´native´ formats which only sometimes match the defined V4L palettes. So
some conversion, albeit trivial, is necessary. Might as well do the full
conversion to various YUV/RGB palettes then, to accomodate as much programs
as possible (granted, some tools are really braindead).

Which is exactly what I did, and therefor I was quite sarcastic in my
previous mail because my driver was targeted for removal of ALL conversion
routines, while (nearly) nothing has been said about other (USB) webcam
drivers which have conversion routines.

Johannes Erdfeld (current USB maintainer) said he would go over the other
USB drivers and see which ones are eligeble for removal of YUV->RGB
conversion (which are, btw: ov511 and cpia; ibmcam flatly refuses anything
except RGB24). I warned him that if he does so, he will get a lot of angry
looks from users.

I do agree that the kernel may not be the proper place for these kind of
conversions. But as I wrote to Alan Cox earlier today that if he wanted to
fix this, he should fix the problem, which are ´simplistic´ programs and a
hardware specific API (V4L1, and V4L2 isn´t that much better), and not
target the drivers. But it will take something really radical (like removing
the V4L API altogether), that will wake up the authors of ALL video
tools and ´fix´ their programs. Alan Cox certainly isn´t going to do that
himself, and neither am I :)

In other words: if you want to break it, break it well. Don´t smash the
saucer without the cup.

 - Nemosoft


[1] There´s an additional problem, which has plagued my driver for quite
some time: TV cards can scale the image to any size, again in hardware. For
webcams which only have a few fixed sizes, some padding or cropping may be
required (this can usually be programmed very easily, and requires only a few
extra CPU cycles). Scaling on the fly is definitely out of the question, even
for me.




Date: Fri, 25 May 2001 16:58:19 +0100 (BST)
From: Alan Cox <alan @ lxorguk.ukuu.org.uk>
To: (Nemosoft Unv.) <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: (Erik Mouw) <J.A.K.Mouw @ ITS.TUDelft.NL>, linux-kernel @ vger.kernel.org,
 (Norbert Preining) <preining @ logic.at>,
 linux-usb-devel @ lists.sourceforge.net

> > that none of the sound drivers does sample rate conversion although
> > some sound chips are locked at 48kHz only.
> 
> True, but a number of audio tools will break, because they don=B4t supp=

So fix the tools

> if you limit the number of available palettes per driver. Plus, you wil=
> l get
> severe fragmentation of which program works with which driver. Which is
> unacceptable, in my opinion (and certainly NOT the idea behind a common=
>  API!)

Fix the tools.

> routines, while (nearly) nothing has been said about other (USB) webcam
> drivers which have conversion routines.

I have those in my firing line too. It has always been the policy that format
conversions go in user space. The kernel is an arbitrator of resources it is
not a shit bucket for solving other peoples incompetence.

Alan



Date: Fri, 25 May 2001 08:37:26 -0700
From: "Dunlap, Randy" <randy.dunlap @ intel.com>
To: Erik Mouw <J.A.K.Mouw @ ITS.TUDelft.NL>,
 Nemosoft Unv. <nemosoft @ smcc.demon.nl>
Subject: RE: ac15 and 2.4.5-pre6, pwc format conversion
Cc: Norbert Preining <preining @ logic.at>, linux-kernel @ vger.kernel.org

> From: Erik Mouw [mailto:J.A.K.Mouw @ ITS.TUDelft.NL]
>=20
> On Fri, May 25, 2001 at 10:48:12AM +0200, Nemosoft Unv. wrote:
> > On 25-May-01 Norbert Preining wrote:
> > > According to ac ChangeLog:
> > > o       Rip format conversion out of the pwc driver     (me)
> > >         | It belongs in user space..
> > >=20
> > > This change is included in 2.4.5-pre6, but
> > >       drivers/usb/pwc-uncompress.c
> > > pwc-uncompress.c:185: warning: implicit declaration of function
> > > `vcvt_420i_420p'
> >=20
> > That=B4s what you get for ripping out the guts of a driver.=20
> Have a nice day.
>=20
> The format conversion shouldn't be there in the first place. Format
> conversion is policy, so it doesn't belong in kernel. Note for =
example
> that none of the sound drivers does sample rate conversion although
> some sound chips are locked at 48kHz only.

Once upon a time there was an agreement (understanding ?) that this
was to be a major 2.5 change (moving video conversion from kernel
drivers to user space) and that lots of apps would need to be
changed for this to be successful.

~Randy



Date: Fri, 25 May 2001 17:03:11 +0100 (BST)
From: Alan Cox <alan @ lxorguk.ukuu.org.uk>
To: (Dunlap, Randy) <randy.dunlap @ intel.com>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: ('Erik Mouw') <J.A.K.Mouw @ ITS.TUDelft.NL>,
 (Nemosoft Unv.) <nemosoft @ smcc.demon.nl>,
 (Norbert Preining) <preining @ logic.at>, linux-kernel @ vger.kernel.org

> Once upon a time there was an agreement (understanding ?) that this
> was to be a major 2.5 change (moving video conversion from kernel
> drivers to user space) and that lots of apps would need to be
> changed for this to be successful.

Nope. Its been policy since 2.0. Its both v4l1 and v4l2 policy. In fact its
fairly nonsensical to handle any format conversions in kernel unless the
device outputs a unique format of its own.

It breaks apps by doing conversions, and it breaks important apps like H263
codecs not silly little camera viewers, because you trash the performance





Date: Fri, 25 May 2001 21:03:56 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: Alan Cox <alan @ lxorguk.ukuu.org.uk>, johannes @ erdfelt.com
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: linux-usb-devel @ lists.sourceforge.net, linux-kernel @ vger.kernel.org,
 (Erik Mouw) <J.A.K.Mouw @ ITS.TUDelft.NL>, randy.dunlap @ intel.com

Greetings,

Coalescing two messages in one:

On 25-May-01 Alan Cox wrote:
> Nope. Its been policy since 2.0. Its both v4l1 and v4l2 policy. In fact its
> fairly nonsensical to handle any format conversions in kernel unless the
> device outputs a unique format of its own.

Oh come on. 2.0 has been out since, what? 1996? Methinks you have had plenty
of time to really do something about it then. You are now simply too late.

> It breaks apps by doing conversions, and it breaks important apps like H263
> codecs not silly little camera viewers, because you trash the performance

This is a NULL argument. First, it doesn´t break anything; I think H263 is
smart to pick a YUV format if it´s available. Second, conversion has to be
done at one point or another. If the native format of the camera is RGB,
H263 will have to convert to YUV. Wether or not this is done in kernel space
or user space doesn´t matter: it has to be done. And in case the native
format of the camera doesn´t resemble anything in V4L, you will have TWO
conversion: first, in kernel from native to V4L palette, and then in your
tool from V4L to whatever format you need, while maybe the driver could do
it all in one stage.

On 25-May-01 Alan Cox wrote:
>> > that none of the sound drivers does sample rate conversion although
>> > some sound chips are locked at 48kHz only.
>> 
>> True, but a number of audio tools will break, because they don=B4t supp=
> 
> So fix the tools
> 
>> if you limit the number of available palettes per driver. Plus, you wil=
>> l get
>> severe fragmentation of which program works with which driver. Which is
>> unacceptable, in my opinion (and certainly NOT the idea behind a common=
>>  API!)
> 
> Fix the tools.

Gee, ¨Fix the tools¨ seems to be your mantra :-(. Anyway, breaking the tools
doesn´t necessarely mean they are going to get fixed; not all are
being actively maintained at the moment (okay, that´s a pity; still it´s a
shame).

And you are probably not going to get the endless streams of mails to the
webcam developers which state: ¨I upgraded my kernel to 2.4.X and my webcam
broke!¨ Do you really think we will enjoy replying time after time: ¨Yes, I
know, but Alan Cox told me to do so¨. No. To a lot of users, and even
application developers, this just plainly sucks.

>> routines, while (nearly) nothing has been said about other (USB) webcam
>> drivers which have conversion routines.
> 
> I have those in my firing line too. It has always been the policy that
> format
> conversions go in user space. The kernel is an arbitrator of resources it
> is not a shit bucket for solving other peoples incompetence.

Now you are being insulting, really. There are a lot of competent people
who put a lot of time and effort into these tools. But the Video4Linux API
is not complete and not completely suitable for webcams. For example, it
misses the quite fundamental call to query the available palettes; nor does
the V4L API give any hint that the number of palettes might be severely
limited. The first tools that were created had the luxury of hardware that
did all the work for them, so why not use it? Webcams are not so lucky.
There´s no need to penalize them, because if the change goes through, a lot
of tools will not work with webcams but with TV cards.

                                         ...

Anyway, I am not going to debate this any further at this point. Johannes,
please remove my webcam driver from the USB source tree, until the software
YUV/RGB conversion has been removed from ALL other video devices (preferably
all at the same time). I will *not* have a crippled driver into the Linux
kernel. If this is going to be policy, fine. But then this policy will apply
to ALL drivers equally, not just mine because suddenly Alan realizes he has
been sleeping for the past 5 years and decides to implement a policy hardly
noone ever knew existed.

 - Nemosoft




Date: Fri, 25 May 2001 15:51:02 -0400
From: Jeff Garzik <jgarzik @ mandrakesoft.com>
To: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: Alan Cox <alan @ lxorguk.ukuu.org.uk>, johannes @ erdfelt.com,
 linux-usb-devel @ lists.sourceforge.net, linux-kernel @ vger.kernel.org,
 Erik Mouw <J.A.K.Mouw @ ITS.TUDelft.NL>, randy.dunlap @ intel.com

"Nemosoft Unv." wrote:
> On 25-May-01 Alan Cox wrote:
> > It breaks apps by doing conversions, and it breaks important apps like H263
> > codecs not silly little camera viewers, because you trash the performance
> 
> This is a NULL argument. First, it doesnt break anything; I think H263 is
> smart to pick a YUV format if its available. Second, conversion has to be
> done at one point or another. If the native format of the camera is RGB,
> H263 will have to convert to YUV. Wether or not this is done in kernel space
> or user space doesnt matter: it has to be done. And in case the native
> format of the camera doesnt resemble anything in V4L, you will have TWO
> conversion: first, in kernel from native to V4L palette, and then in your
> tool from V4L to whatever format you need, while maybe the driver could do
> it all in one stage.

Sorry this is a slippery slope argument and it won't wash.

The kernel is intended as the arbiter between userspace and hardware,
and userspace and userspace.  Format conversion has nothing to do with
arbitration.

Format conversion in kernelspace is far less flexible than userspace: 
you cannot replace your algorithms at will nor fix bugs at will.  You
cannot support assembly optimizations for format conversions without
bloating the kernel.

Finally, the example you describe is invalid.  If your tool is doing
-two- format conversions, then [again] the tool should be fixed.  The
kernel most definitely should not work around stupid shortcomings of
userspace software.


> Anyway, I am not going to debate this any further at this point. Johannes,
> please remove my webcam driver from the USB source tree,

whatever.  I don't see Alan or Linus accepting such a change, even if
Johannes does.


> until the software
> YUV/RGB conversion has been removed from ALL other video devices (preferably
> all at the same time).

Send a patch for this instead!

Format conversion should not be in the kernel...

	Jeff




Date: Fri, 25 May 2001 23:57:06 +0200 (MEST)
From: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
To: linux-usb-devel @ lists.sourceforge.net
Subject: RE: [linux-usb-devel] Re: ac15 and 2.4.5-pre6, pwc format conver

Greetings,

On 25-May-01 Jeff Garzik wrote:
> 
> Sorry this is a slippery slope argument and it won't wash.
> 
> The kernel is intended as the arbiter between userspace and hardware,
> and userspace and userspace.  Format conversion has nothing to do with
> arbitration.
> 
> Format conversion in kernelspace is far less flexible than userspace: 
> you cannot replace your algorithms at will nor fix bugs at will.  

Well, the kernel is in source form too. So I don´t see why can´t fix that
too.

> Finally, the example you describe is invalid.  If your tool is doing
> -two- format conversions, then [again] the tool should be fixed.  

No, the conversion is in two separate places; one to get something sensible
from the native format which doesn´t fit anything in the current V4L API. Or
do you suggest extending this API with every oddball format that comes
along? Please no... 

> The
> kernel most definitely should not work around stupid shortcomings of
> userspace software.

*pfrt*. I already had to. And I´m sure I´m not the first to have a
´fix-in-kernel´.

>> Anyway, I am not going to debate this any further at this point.
>> Johannes,
>> please remove my webcam driver from the USB source tree,
> 
> whatever.  I don't see Alan or Linus accepting such a change, even if
> Johannes does.

It´s not a change, but an addition. I´m still somewhat ticked off by the
fact that the current drivers have such conversion routines, and nobody did
anything about it. And now suddenly this becomes THE issue to reject a
driver. It´s not fair. I´m not a paranoid person, but I am beginning to
wonder if this driver is being blocked for other reasons, like having a
binary only part (which is purely optional anyway).

>> until the software
>> YUV/RGB conversion has been removed from ALL other video devices
>> (preferably
>> all at the same time).
> 
> Send a patch for this instead!

If I do that now, hell will break loose. To me, this course of action is
perfectly acceptable for 2.5; but not while 2.4.* is slowly making its way
to more and more distributions and desktop machines. It would instantly
break all USB webcams. 

> Format conversion should not be in the kernel...

I still maintain that this should have been fixed a long time ago then. It
would have saved me the disappointment of working 1.5 years on this, only to
be rejected because of some bad-ass policy nobody adhered to in the first
place.

 - Nemosoft



Date: Sat, 26 May 2001 00:03:33 +0100 (BST)
From: Alan Cox <alan @ lxorguk.ukuu.org.uk>
To: (Nemosoft Unv.) <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: (Alan Cox) <alan @ lxorguk.ukuu.org.uk>, johannes @ erdfelt.com,
 linux-usb-devel @ lists.sourceforge.net,
 (Erik Mouw) <linux-kernel @ vger.kernel.org>, J.A.K.Mouw @ ITS.TUDelft.NL,
 randy.dunlap @ intel.com

> all at the same time). I will *not* have a crippled driver into the Lin=
> ux
> kernel. If this is going to be policy, fine. But then this policy will =
> apply
> to ALL drivers equally, not just mine because suddenly Alan realizes he=

I've had the others on the list for a while too, jus this one was very easy
to fix and mindbogglingly annoying

> been sleeping for the past 5 years and decides to implement a policy ha=
> rdly
> noone ever knew existed.

The policy has been there since day 1. This is a kernel not a library. You've
written some very nice conversion library routines and I do hope you contribute
those in userspace where they belong



Date: Sat, 26 May 2001 10:51:18 -0400 (EDT)
From: Mark Hahn <hahn @ coffee.psychology.mcmaster.ca>
To: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion

> > I have those in my firing line too. It has always been the policy that
> > format
> > conversions go in user space. The kernel is an arbitrator of resources it
> > is not a shit bucket for solving other peoples incompetence.
> 
> Now you are being insulting, really. There are a lot of competent people

you are misunderstand what he said.  he said that dozens of hacked,
inefficient, possibly buggy format converters should not be in the kernel.
he is not claiming your driver is bad, or that you are bad.

> who put a lot of time and effort into these tools. But the Video4Linux API
> is not complete and not completely suitable for webcams. For example, it
> misses the quite fundamental call to query the available palettes; nor does
> the V4L API give any hint that the number of palettes might be severely

excellent; this is a constructive comment that can improve the API.

> Anyway, I am not going to debate this any further at this point. Johannes,

this is a mistake on your part.

> all at the same time). I will *not* have a crippled driver into the Linux

why do you object to factoring?  it's a fundamental sw design goal.

> to ALL drivers equally, not just mine because suddenly Alan realizes he has
> been sleeping for the past 5 years and decides to implement a policy hardly
> noone ever knew existed.

I'm quite distant from this topic, having only played with webcams 
a few times, and even I know that no-format-conversion-in-kernel 
is the policy, and why it's appropriate, and why apps will not see
any degredation.



Date: Fri, 25 May 2001 18:52:35 +0200
From: Erik Mouw <J.A.K.Mouw @ ITS.TUDelft.NL>
To: "Nemosoft Unv." <nemosoft @ smcc.demon.nl>
Subject: Re: ac15 and 2.4.5-pre6, pwc format conversion
Cc: linux-kernel @ vger.kernel.org, Norbert Preining <preining @ logic.at>,
 linux-usb-devel @ lists.sourceforge.net

On Fri, May 25, 2001 at 03:22:44PM +0200, Nemosoft Unv. wrote:
> On 25-May-01 Erik Mouw wrote:
> > On Fri, May 25, 2001 at 10:48:12AM +0200, Nemosoft Unv. wrote:
> > The format conversion shouldn't be there in the first place. Format
> > conversion is policy, so it doesn't belong in kernel. Note for exampl=
e
> > that none of the sound drivers does sample rate conversion although
> > some sound chips are locked at 48kHz only.
>=20
> True, but a number of audio tools will break, because they don=B4t supp=
ort that
> samplerate (mpg123 0.59r, one of the more popular MP3 players, segfault=
s
> when it has to do conversion of 44.1 KHz to 48 KHz). Recording from suc=
h a
> source is usually less of a problem, although some post-processing is
> then necessary (not all tools support samplerate conversion natively).

I consider that a bug in mpg123. Examples for rate conversion have been
available for years, so fix mpg123.

> The situation for video devices is worse. 80% of the video tools will b=
reak
> if you limit the number of available palettes per driver. Plus, you wil=
l get
> severe fragmentation of which program works with which driver. Which is
> unacceptable, in my opinion (and certainly NOT the idea behind a common=
 API!)

I consider this a bug in the tools. I've been working with real-time
video on sgi IRIX machines, and they do a couple of things right:

- The hardware supports a limited set of video formats. Most high end
  stuff (Onyx+Sirius, Octane+DV, Onyx2+DIVO) only supports a couple of
  YUV or YUVA formats (in studios YUV422 or YUV422+A is the most used).
  Low end workstations (Indy, O2) supports some "consumer" formats like
  RGB or Y as well.
- There are a couple of libraries (vl, cl, dmedia, audio, audiofile)
  that allows you to do all kinds of manipulations in userland.

Fortunately sgi also recognises this, and they're porting their
libraries to Linux (see oss.sgi.com).

> You can blame the authors of those video tools, but that=B4s not really=
 fair.
> The original Video4Linux API was based upon TV grabber cards. Most of t=
hem
> do YUV/RGB conversion in hardware, so most tools expected that all or m=
ost
> palettes would always be available, since supporting a palette would no=
t
> require any extra CPU cycles [1]. Some tools like RGB, and others YUV, =
so
> the authors happily selected the palette of their choice and never even
> considered building in functions for doing conversion. And then I am no=
t even
> talking about the RGB/BGR mess.

The original V4L API was *implemented* first on TV grabber cards, but
was certainy written with other video equipment in mind. I remember I
looked at the API with the sgi stuff in mind, and considered it quite
sane.


Erik


Main page