Proof-of-concept CFFI-OpenGL running

The above image shows a run of NeHe tutorial 1 making use of the CFFI binding for OpenGL rather than the UFFI binding that's in the original cl-sdl. It's the only one I have tested sofar, but it seems I'm on the right track.

Status report

I haven't had much time to spend, but I did get a version of gl.lisp and glu.lisp in CFFI and mostly autogenerated by SWIG to compile without errors. It still needs to be tested in practice, and I first have to check out how to best do this as I need to grab a rendering context and window through SDL...

On the upside, I have done some "real" Lisp programming: the foreign function names are now lispified in Lisp itself.

swig -cffi

So I installed SWIG and tried out the CFFI mapping, and it seems to work beautifully for the gl.h header (still have to try the others). I initially had some problems with SWIG seemingly discarding hexadecimal values for constants, but this was due to Mac OS X having an older version of SWIG which I was unwittingly using instead. The latest SWIG works just fine and the generated code looks perfect.

I'll do some more tests, and then figure out what my new plan should be.

Spurious work ?

While looking around for info on function pointer types in UFFI, I found CFFI (a newer project with similar goals) and fetter/Verrazano, which generates CFFI/UFFI from C/C++ header files!

I have some reading to do! Until I've delved into this a bit further, my own generator script will be on hold. If fetter is mature enough I'll dump my own script, and use it instead.

I wish I had found this when looking at UFFI a month ago...

[edit] Even better, SWIG supposedly also is able to generate CFFI code. I'll definitely need to try out that one!

glu.lisp progress

Haven't had much time to spend on it, but I think my generator now covers glu.h. I still need to check its coverage by taking measurements as with gl.h. And then, of course, I need to test it for real.

I'm mostly worried about the typing differences between the generated glu.lisp and the original. The original replaced all occurrences of quadric with (* void). The generated version, on the other hand, leaves them be but adds a define-foreign-type mapping quadric to (* void). My guess —hope really— is that this will have the same effect, but I don't understand enough of UFFI to be sure.

There is also a function pointer type that I don't know how to handle yet. I'll have to take a look at the UFFI documentation.

Started on glu.lisp

I have done some preliminary work on generating glu.lisp from the glu.h header on my system. It's not doing too badly (misunderstanding only 76 lines out of 340), but there are lots of procedures it doesn't "get" yet. I will also have to take a closer look at the types being defined in gl.h. If the generation works, this will be extra valuable, as the original glu.lisp is quite minimal. It means I'll be able to cover quite a bit more of the available functionality.

I'm spending more time on this than I had hoped, but I still think it's valuable. My first feature of the engine will likely be character models, making use of the CAL3D library. If so, then I'm already hoping to apply the generator to CAL3D to set up the bindings.

[edit] I have most all procedures covered now. I still need to take a closer look at the type mappings, though.

[PS] As for the peasant magic: I finished dead last. :-)

't Works

Succes! I got my version of cl-sdl running using the generated UFFI bindings for gl.lisp instead of the original ones. I tried some of the demoes, and things seem to be fine.

Allowing for overrides took about half an hour of work. I then spent another half hour fixing some other problems relating to names of parameters. But that was basically it.

Next up, I'll take a look at using the generator for glu.lisp as well. But that won't be for today. I have some peasant magic planned this afternoon. :-)

Here are the results of the Belgian jury...

The above graph shows the (condensed) results of the measurements. For each kind of foreign definition in cl-sdl's "gl.lisp" (constants, routines and types) you can see the number of identifiers which are not used (in blue), the number of identifiers which are used and covered by my generator (in yellow) and the number of identifiers which are used but which my generator does not cover (in red). As you can see, only a very small percentage of all identifiers is causing problems; nine identifiers in all. They are: light-fv, fog-fv, fog-f, scale-f, translate-f, fog-i, rotate-f, material-fv and tex-parameter-i.

Possible choices before me now are (1) to refactor cl-sdl to have these nine identifiers match those generated by my script, or (2) to have my script treat these nine identifiers as exceptions defined by the user. The first choice entails updating 139 use-sites in cl-sdl, the second an extension to my generator script. I think I'll go for the latter solution as I expect it will involve less work.

I'll be keeping the analysis scripts around, as it will allow me to check when my generator is covering all the cases. It also doubles as a very basic coverage calculation tool, which might also prove interesting down the line.

Taking measurements...

I decided to try to quantify the naming scheme problem, so I'm in the process of setting up some analysis scripts. The scripts are virtually complete; it's now really mostly a matter of processing the results. I also think that I can reuse most of the analysis scripts to automate the update of cl-sdl. This should save on time, manual labour, and make sure of better results. But, again, first comes taking a closer look at the extent of the problem.

Naming schemes

I had a first try at substituting the original gl.lisp in cl-sdl with my generated one, and, of course, it failed. So I started taking a look at what differences are left between both, and here is a nasty one: cl-sdl uses an inconsistent naming scheme. Mine, on the other hand, is very strict due to its automated nature. Here is an example: cl-sdl lispifies "glGetIntegerv" into "get-integerv" on the one hand, and "glLoadMatrixd" into "load-matrix-d" on the other hand. My generator script always uses the first variant, and would turn "glLoadMatrixd" into "load-matrixd". This is simply because it doesn't recognise the final "d" as a separate item. I think I'll check the cl-sdl codebase to see if both variants are being actively used. The gl.lisp file seems to favour the second variant (as would I), and so, if the first one is hardly used I can just update those so that the naming scheme becomes uniform. I will then also have to update my script to use this naming scheme, which will require some extra input from the user to figure out where to place the extra dashes... Because of that complication, it might also be worthwhile to just standardise on the first, be it uglier, variant. But if the second variant is more widely used, then I'll have a lot of updating to do... I also need to start thinking on whether the extra work is justified. I still want an up-to-date opengl binding, and using the generation approach still seems a smart thing to do. Having a consistent naming scheme is also a good thing. I just hadn't counted on having to update the use-sites.

sdlstub is autoloading again

I did some tracing and found that the foreign library was, in fact, getting loaded as it should. I then started working on the idea that placing my own library in between the initialisation might have screwed it up somehow. So I did some rearranging, as well as adding explicit dependencies between the libraries. I found the right incantations after about half an hour, and now... it works.

Next up is to finally try that uffi generation script of mine, and then start doing some real 3D programming.

sdlstub still won't autoload

I have been trying some variations, but I still can't get the sdlstub dylib to autoload as it should. The strange thing is that it seems to autoload the sdl library just fine, as well as the sd-aqua fix library I set up. There is no difference in code for loading any of these other than the values used, which I think are quite correct...

(require 'sdl-demos)

So, another day, another step forward, and... another step backward. Let's start with the good news: I integrated my sdl-cocoa bugfix into cl-sdl, which means that I can now skip some tedious manual steps that I had to do before. That's when I started going over the different demos to make sure that everything is still ok. Here are captures of the ones which worked: Looks nice right ? OK, so they're no Unreal 3 demos, but it's still nice to see them running. Unfortunately, there are about as many demos which do not run as there are which do. For those I'm getting an "SB-KERNEL::UNDEFINED-ALIEN-FUNCTION-ERROR: Attempt to call an undefined alien function". With the help of the traceback I can see that something is going wrong with the load-bmp function. I have had bitmaps loading in earlier experiments, but I haven't tried it since starting this blog. So I guess I'm of on another wild duck hunt... [edit] I managed to get the textures to load. Here is a screenshot of (nehe:run-tutorial 9): I think something is going wrong with the loading of the foreign library. I have to do a manual uffi:load-foreign-library; if not, I still get the earlier error. I'll have to take a closer look at why it won't load the right library all by itself anymore.