Context: The Objective-CL Project.
Currently, there is no way of loading nib files without knowing their exact location on the host filesystem. This is unacceptable but a bit hard to solve.
Every Cocoa application is normally located in its own application bundle, a directory containing resources such as nib files, among others. In contrast, simply loading Objective-CL into your Lisp image does not cause any reasonable bundle path to be registered, as the bundle path is determined from the running executable's name, which usually belongs to your Lisp system.
There are two approaches to handle this situation:
The disadvantage of option 1 is that it's no good for interactive development, so option 2 is more or less needed additionally either way.
Either approach might need to defer loading libobjcl
until after the
application's launch. This in turn would mean that Objective-CL would
have to be split into two parts: a loader and the rest. The loader
might have to provide notinline
-declared stubs of all of Objective-CL's
public functions as well as all public macro definitions. This would
get ugly fast... An alternative is needed.
#1=(quote #1#)
+--------------------+ | | v | +------+------+ +--|---+------+ | | | | | | | | | --------->| | ---------> NIL | | | | | | | +--|---+------+ +------+------+ | v QUOTE
(Update 2008-06-10, 15:20 CEST: Michael Weber noticed that I had erroneously drawn #1=(quote . #1#). I've fixed the drawing.)
It just dawned on me that by changing the way objective-c-generic-function dispatch works, it may be possible to give call-next-method a useful semantics.
Assuming MLKFnord
is a Lisp-backed Objective-C class which mulk
is an
instance of, and MLKSlack
is a Lisp-backed superclass of MLKFnord
, then
the call [mulk foo: 10 bar: nil]
is done as follows:
MLKFnord's
foo:bar:
callback is entered.(#/foo:bar: 'mlk-fnord mulk 10 nil)
.MLKSlack's
foo:bar:
callback to be entered.(#/foo:bar: 'mlk-slack mulk 10 nil)
. And so on.I'm experimenting with an alternative approach that does away with the extra class name argument. Here are some random thoughts (which may be completely wrong), in no particular order:
NSObject's
version of a method will never call super and
therefore the call chain will always end there, regardless of what we
do on the Lisp side.The nice thing is that this gives call-next-method a meaning that isn't completely bogus. The bad thing is that it might make method calls (call-next-method and super, at least) slow. This new idea is, after all, way off the usual way of invoking the CLOS dispatch mechanism.
Note that we could still use normal CLOS generic function dispatch if the object's class is the same as the callback's definition class.
After moving (object-get-class receiver) from invoke-by-name-super-v to retrieve-method-signature-info:
Evaluation took: 3.764 seconds of real time 3.760235 seconds of user run time 0.012001 seconds of system run time [Run times include 0.096 seconds GC run time.] 0 calls to %EVAL 0 page faults and 72,809,568 bytes consed.
Finally, an excuse to use the &aux lambda list keyword. Hooray!
The benchmark:
(let ((x (invoke (find-objc-class 'ns-method-signature) :method-signature-for-selector 'new))) (time (dotimes (i 100000) (invoke x :get-argument-type-at-index 0))))
Before:
Evaluation took: 7.727 seconds of real time 7.136446 seconds of user run time 0.080005 seconds of system run time [Run times include 0.288 seconds GC run time.] 0 calls to %EVAL 0 page faults and 218,448,224 bytes consed.
After:
Evaluation took: 5.868 seconds of real time 5.824364 seconds of user run time 0.032002 seconds of system run time [Run times include 0.256 seconds GC run time.] 0 calls to %EVAL 0 page faults and 122,487,656 bytes consed.
What I did was add a name slot to class selector so that selector-name need only access a slot now instead of calling a foreign function and converting the returned value to a Lisp string.
After that, I enhanced intern-pointer-wrapper to intern classes, because object-get-class, another frequently called function, had to first acquire the class name associated with a class pointer and then finally call find-objc-class-by-name with that name. The result:
Evaluation took: 4.058 seconds of real time 4.020251 seconds of user run time 0.016001 seconds of system run time [Run times include 0.148 seconds GC run time.] 0 calls to %EVAL 0 page faults and 76,830,440 bytes consed.
That's gonna be it for now. I'll call it a night.
(By the way, this is using CFFI 0.9.2. The Darcs version is significantly slower. Remember the CFFI speed hack I mentioned sometime way back in the beginning? It's needed for a Darcs CFFI, but I don't think it's needed for CFFI 0.9.2.)
I always profile invoke like this:
(let ((x (invoke (find-objc-class 'ns-method-signature) :method-signature-for-selector 'new))) (sb-sprof:with-profiling (:max-samples 500 :loop t :report :flat) (dotimes (i 100) (invoke x :get-argument-type-at-index 0))))
Of course, this is a ridiculous microbenchmark that doesn't yield much information about actual Objective-C usage, but it's certainly useful for finding the bottlenecks of invoke calls.
Whether the performance of repeated invocation of the very same method on the very same object is all that interesting is, of course, an entirely different matter.
This is what the GNUstep manual says about the #dealloc
method:
In some circumstances, an object may wish to prevent itself from being deallocated, it can do this simply [by] refraining from calling the superclass implementation.
Maybe we could use this in the case of a garbage-collected runtime.
I have just discovered a discouraging bug in CMUCL's version of PCL that is confirmed by what Closer-to-MOP's feature list says: pcl:set-funcallable-instance-function does not accept a closure as its second argument, it only accepts pure functions that don't close over things.
Really. That is a horrible bug. It means that Objective-CL is broken
on CMUCL in a fundamental way (see the definition of shared-initialize
:after (selector ...) in data-types.lisp
) and I don't have the slightest
idea how to work around it.
Collecting all Objective-C classes and creating CLOS classes out of them, which is what the only recently introduced function collect-classes does, works reliably both on Allegro CL and on GNU CLISP. There are some serious problems on CMUCL and SBCL, though.
First, some stats (this is on Wirselkraut, my Dell Inspiron 6400).
OBJCL[3]> (time (collect-classes)) Real time: 14.182854 sec. Run time: 14.180886 sec. Space: 189717640 Bytes GC: 123, GC time: 2.412143 sec. 0
OBJCL(3): (time (collect-classes)) ; cpu time (non-gc) 2,660 msec user, 10 msec system ; cpu time (gc) 620 msec user, 0 msec system ; cpu time (total) 3,280 msec user, 10 msec system ; real time 3,291 msec ; space allocation: ; 1,816,989 cons cells, 44,365,376 other bytes, 72,082 static bytes 0
Note that Allegro CL defers class finalisation until the first instance of a class is created. Then again, half the classes created are metaclasses which are immediately instantiated, so the speed is impressive, either way.
\* (time (collect-classes)) STYLE-WARNING: slot names with the same SYMBOL-NAME but different SYMBOL-PACKAGE (possible package problem) for class #<OBJECTIVE-C-META-CLASS NS:++NS-OBJECT {B75E4440}>: (SB-PCL::NAME NS:NAME) STYLE-WARNING: slot names with the same SYMBOL-NAME but different SYMBOL-PACKAGE (possible package problem) for class #<NS:++NS-OBJECT NS:+NS-OBJECT {B75E4440}>: (SB-PCL::NAME NS:NAME) STYLE-WARNING: slot names with the same SYMBOL-NAME but different SYMBOL-PACKAGE (possible package problem) for class #<NS:+NS-OBJECT NS:+GSXMLP-LIST-PARSER {B75ED3E0}>: (SB-PCL::NAME NS:NAME)
That's how it starts off. The style-warnings go on and on. At first, they rush by fast, but each new class seems to take more time than the previous one to create. I'm not going to wait for it to complete in order to tell you the time stats because it could well take decades...
\* (time (collect-classes)) ; Compiling LAMBDA NIL: ; Compiling Top-Level Form: Type-error in KERNEL::INVALID-ARRAY-INDEX-ERROR-HANDLER: 4 is not of type (INTEGER 0 (0)) [Condition of type TYPE-ERROR] Restarts: 0: [ABORT ] Return to Top-Level. 1: [DESTROY] Destroy the process Debug (type H for help) (MAPHASH #<Closure Over Function "DEFUN MAKE-ACCESSOR-TABLE" {5974DC09}> #<HASH-TABLE :TEST EQ :WEAK-P NIL :COUNT 65 {2824FEBD}>) Source: (AREF KV-VECTOR (* 2 I)) 0]
I don't have the slightest idea what that could mean. A bug in CMUCL's hash table code?
Not timed yet. Will do that someday.
I'll have to ask some CMUCL and SBCL gurus what's going on in their respective variations of PCL.
I wonder whether being API-compatible with Clozure CL's Objective-C bridge would be a good or bad thing. On the one hand, compatibility means application portabilitiy, which is nice. On the other hand, using the same package names and reader macros as Clozure CL's bridge makes it hard to have both loaded at the same time.
Then again, if someone wants to compare Objective-CL with Clozure CL's bridge side-by-side, it's their responsibility to rename packages as needed. Reader macros aren't essential for using Objective-CL, so they may be left disabled in such a case, anyway. I'm going to try hard to use Objective-CL-specific package names within my own code (i.e. objective-c-classes rather than NS), so renaming packages won't break things.
I'll be opting for direct API compatibility for now. It seems to be the right choice.
In the Objective-C 2.0 runtime, the functions
class_add{Method,Protocol,Ivar}
, class_copyMethodList
,
{class,protocol}_copyProtocolList
, protocol_copyMethodDescriptionList
,
and class_copyPropertyList
are probably our friends.
class_copyMethodList
may be used in combination with
method_setImplementation
for good effect.
But... What about the GNU runtime? Is it okay to inspect a Class'
methods
member (see objc.h
and objc-api.h
) and change the IMPs
that
the individual members point to? Is it possible to add new methods at
runtime by using class_add_method_list
and thus
ObjcUtilities_register_method_list
? Is the behaviour of these two
functions specified if a method list for a given class has already been
registered in the past? If so, do they replace the original list or
amend it?
Changing ivars after class creation seems generally impossible. This applies to all the supported runtimes, and I think I can actually see why. I don't think it's a serious problem, but I do consider it regrettable.
I've added the following files as a first step to support class definition.
For the GNU runtime (from JIGS, the Java Interface to GNUstep):
JIGS/ObjcRuntimeUtilities.c
JIGS/ObjcRuntimeUtilities.h
JIGS/ObjcRuntimeUtilities2.m
For the NeXT runtime (from PyObjC):
PyObjC/pyobjc-compat.h
PyObjC/objc-runtime-compat.h
PyObjC/objc-runtime-compat.m
Both the JIGS and PyObjC codebases are impressively modular. Those guys know what they're doing.
Objective-CL now includes its own version of libffi, imported from an older version of PyObjC. (The current version only supports x86 and PowerPC.) Interestingly, according to the Web, Mac OS X 10.5 includes its own version of libffi. This is really good news! We can finally rest assured that Objective-CL does not break on Mac OS X because of libffi bitrot anytime soon. In order to take advantage of this new state of the world, the build system has been changed so as to only compile our own version of libffi if we can't find any on the system.
The downside? We now depend on autoconf. I don't consider this a problem, though.
I've cleaned the Objective-C code up by making the NeXT and GNU
runtime-specific code converge a bit. This also makes find-selector
return nil for unknown selectors on the NeXT runtime, so compile-time
warnings about unknown methods are possible there now. The latter
relies on sel_isMapped
, whose semantics are not entirely clear to me.
On the one hand, Apple's reference manual states: “You can use this
function to determine whether a given address is a valid selector,”
which I interpret as meaning that it takes a selector pointer as an
argument, not a string. On the other hand, in the preceding section,
the same document states: “You can still use the sel_isMapped function
to determine whether a method name is mapped to a selector.”
So if I have two strings that aren't the same under pointer-eq, but that
both name the same valid selector that is registered with the runtime,
like "self"
, say, does sel_isMapped
work reliably in this case? I'm not
sure.
On another note, I wonder what the difference between
sel_get_uid/sel_getUid
and sel_register_name/sel_registerName
might be.
They seem to do the same thing.
Maybe this whole #ifdef
mess isn't even strictly necessary, anyway. I
could just copy objc-gnu2next.h
from the GNUstep project (LGPLv3, so the
licensing is fine).
http://svn.gna.org/svn/gnustep/libs/base/trunk/Headers/Additions/GNUstepBase/objc-gnu2next.h
The latest changes made the test cases fail on GNUstep/x86, which either
means that the PyObjC code is wrong, or the GNU
runtime has very weird calling conventions that use ints
as wrappers for
chars
or something. Anyway, I have reverted the changes for GNUstep and
left them in place for Mac OS X (but note that I left the
PyObjC code as it is, which means that libffi is
still directed to treats chars as ints). As a result, both NeXT/PowerPC
and GNUstep/x86 work for now, but I'm uncertain about the status of
other architectures as well as calling methods with chars and shorts as
arguments, which I've got no test cases for. I'm not confident that
either GNUstep/PowerPC/SPARC/whatever or NeXT/x86 work the way my code
expects them to.
There's a good chance that I've figured out what to do about the
char/int
mess. As it turns out, it isn't even limited to chars
, as
shorts
are affected, too. According to the code I took from
PyObjC, specifically the typespec conversion
functions in libffi_support.m
, both GNUstep and NeXT/PowerPC treat chars
and shorts
as ints
. The only platform that isn't brain-damaged in this
way seems to be NeXT/x86. Or maybe it's even more brain-damaged, as it
treats shorts
and chars
normally when they are used as arguments, but as
ints
when they're used as return values! At least GNUstep and
NeXT/PowerPC are brain-damaged in a consistent manner.
I figure the reason I never saw this problem in GNUstep is probably
endianness. The little-endian x86 lets you treat pointers to ints
as
pointers to chars
without breaking anything, but that doesn't work in
big-endian machines.
In principle, the typespec "c" is supposed indicate a char
. Now look at
the following SLIME session transcript (SBCL/PowerPC on Mac OS X):
OBJECTIVE-CL> (defparameter *tmp* (invoke (find-objc-class 'ns-string) :string-with-u-t-f-8-string "Mulk.")) *TMP* OBJECTIVE-CL> (defparameter *tmp2* (invoke (find-objc-class 'ns-string) :string-with-u-t-f-8-string "Mulk.")) *TMP2* OBJECTIVE-CL> (second ;return type specifier (multiple-value-list (retrieve-method-signature-info (find-objc-class 'ns-string) (selector :is-equal)))) "c" OBJECTIVE-CL> (invoke *tmp* :is-equal *tmp2*) 0 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :char *tmp2*) 0 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :int *tmp2*) 1 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :long *tmp2*) 1 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :long-long *tmp2*) 4294967296
Now, I see why the last value is bogus (I'd be surprised if it weren't,
actually), but why the heck is the correct value (1, because, you see,
the strings are equal and +yes+ is 1 on my machine) returned only for
the wrong return type? The return type is specified as "c"
, but it's
actually an int
! What's going on here? And rather more importantly:
What can I do about this? I don't feel exactly comfortable about
cheating and treating "c"
as specifying an int
on all systems based on
the NeXT runtime without having any indication about what else there is
in the NeXT runtime that has to be special-cased. I haven't seen this
weird behaviour documented anywhere. Even this specific case is
non-trivial, for I don't know whether this applies to all chars
, or only
to chars
that are booleans, or only to chars
that are returned, or even
only to chars
that are returned and are actually booleans.
Licensing is another open question. For the moment, I'm releasing this project under the terms of the GPLv3. This seems like a reasonable choice, because it gives me the option of giving people more permissions later by applying the LGPLv3 to my code. I must be aware that only I am allowed to do this, though, and even then only if all contributors agree (if someone actually makes a contribution, that is). I may want to require all contributors to dual-license their contributions, or maybe to make them available under the terms of the LGPLv3 in the first place (though the latter would make marking them difficult).
Open question: Should NSArray
instances be converted to lists or arrays
automatically? If so, we ought to make functions like objc-class-of
behave in a reasonable way for those kinds of objects, i.e. return
NSArray
or NSMutableArray
(whatever it is that invoke makes out of them
when converting them into Objective-C instances again).
Note that we must not convert NSMutableArray
instances or any other
mutable objects in this way! Note also that our decision must be based
on the dynamic type of the object, not the static one, because a method
whose return type is NSArray
may as well return an NSMutableArray
that
we've fed it sometime earlier. This is okay for immutable objects, but
mutable objects are bound to cause trouble when such a thing happens.
Related types of objects are strings (NSString
), hash tables
(NSDictionary
), and numbers (NSNumber
).
Note that such behaviour would make it impossible to fully identify CLOS classes with Objective-C classes, as arrays would have no Objective-C class to belong to. Then again, why would you want to distinguish Objective-C arrays from Lisp arrays in your Lisp code, anyway? Real integration means not having to worry about such things.
On the other hand, conversion of large NSArrays
may be prohibitively
expensive, so a switch is needed, either way. The real question is what
the default behaviour should look like.
There's an alternative to consider, too. For NSArrays
, there is
Christophe Rhodes' user-extensible sequence proposal, but even without
support for that, we can provide a conduit (a package) that looks like
the common-lisp package, but overloads all sequence and hash-table
functions. Overloading all sequence functions might be a lot of work,
though.
Up until now, the second-generation method invocation procedures (low-level-invoke and primitive-invoke) simply called make-instance for every object received from Objective-C, which meant that although a lookup in the caching hash tables was done, method dispatch for make-instance was needed. Therefore, everything just worked, but did so slowly.
I realised yesterday, after having profiled the code and detected that make-instance method dispatch was the speed bottleneck of invoke calls now, that overriding make-instance wasn't really necessary for memory management, as we could put instances into the hash tables and register finalisers for them just after they were fully created.
So that's what I made the program do. One of the results is much
shorter and clearer code, but the more interesting one is a speed
improvement of around the factor 3, making 100'000 calls to
NSMethodSignature#getArgumentTypeAtIndex:
, which previously called
make-instance for each returned value, take around 10s on my machine.
With the CFFI speed hack enabled, caching cffi::parse-type results, this
figure even goes down to around 2s (that's 50'000 method calls per
second).
I think that's pretty cool. I'm quite satisfied with method invocation performance now. Compared to C, We're still off by a factor of 22 or so (0.9s for 1'000'000 method calls). Most of the time is spent on memory allocation for argument passing and typespec strings. By introducing a global pool of preallocated memory spaces for these purposes (one argument space per thread and maybe a bunch of string buffers, with a fallback mechanism for method calls that take too much space), we might be able to cut the run time by another factor of 5. After that, we can't optimise the Lisp code any further, because most of the rest of the time is spent within the Objective-C function objcl_invoke_with_types (or maybe in calling it via CFFI, which would be even worse, optimisationwise).
It's probably best not to spend too much time pondering this, though, because without the CFFI speed hack, the improvement would probably not be noticeable, anyway (cffi::parse-type is most often called by cffi:mem-ref and cffi:mem-aref, not by the allocation routines).
There are three things left to do that are showstoppers against actually using Objective-CL productively. One is support for structs. This one is actually quite a bit harder than it looks, because we don't necessarily know the structure of foreign objects. Objective-C tells us about the structure (though not the member naming!) of structures as well as pointers to structures that are returned by methods, but any more indirection (that is, pointers to pointers to structs or something even hairier) makes the Objective-C runtime conceal the internals of the structs pointed to. This is probably not a problem in practise, though, as pointers to pointers to structs will usually mean a pointer that the user may alter in order to point to other structs, not that the user will access the structs that are pointed to. In fact, it will probably be best to just pass pointers on to the user.
The second thing left to do is support for defining Objective-C classes. I think this is going to be hard. I've not looked at the problem in detail yet, but it looks like creating methods and classes, and registering methods and classes are all different actions that are all handled differently depending on the runtime. In the case of GNUstep, I don't even know how to register new selectors yet.
Third, varargs. These are easy to implement, but I'm not sure how they
should look like in the case of invoke. Maybe a special keyword
indicator like :*
would work for indicating the end of the method name,
but I think that could be a bit ugly.
We shall see.
On another note, I briefly checked out OpenMCL's support for Objective-C by randomly typing a bunch of method invocations into the listener and calling apropos a lot. Here's what stuck:
NSString
objects, not C strings. Why?All in all, what struck me the most was the fact that the OpenMCL Objective-C bridge does not seem to make use of the concept of designators as much as Objective-CL does. You have to define C strings and selectors explicitely, which I consider a minor annoyance. It's faster, though. Then again, considering that it's integrated into the compiler, I was bit disappointed by the speed, because I figured that a native-code compiler could do better than libffi (which is still a lot slower than directly calling stuff from Objective-C).
Gorm rules. We need to make Objective-CL fully Gorm-compatible.
Matthias Benkard, 2008-06-10, 15:28 CEST