How can I simplify my deserialization framework?
===

I have a Serialization interface which is designed to encapsulate the differences between XML/JSON/binary serialization for my application. It looks something like this:
    
    interface Serialization {
        bool isObject();
        int opApply(int delegate(string member, Serialization value) del); //iterate object
        ...
        int toInt();   //this part is ugly, but without template member overloading, I
        long toLong(); //figure out any way to apply generics here, so all basic types
        ...            //have a toType primitive
        string toString();
    }
    class JSONSerialization : Serialization {
        private JSON json;
        ...
        long toLong() {
            enforce(json.type == JSON_TYPE.NUMBER, SerializationException.IncorrectType);
            return cast(long)json.toNumber();
        }
        ...
    }

So, what I then set up is a set of templates for registering type deserializers and calling them:
    
    ...
    registerTypeDeserializer!Vec3(delegate Vec3(Serialization s) {
        return Vec3(s[0].toFloat, s[1].toFloat, s[2].toFloat);
    });
    ...
    auto v = parseJSON("some file").deserialize!Vec3;
    ...
    registerTypeDeserializer!Light(delegate Light(Serialization s) {
        return new Light(s["intensity"].toFloat, s["position"].deserialize!Vec3);
    });

This works well for structs and simple classes, and with the new parameter identifier tuple and parameter default value tuple I should even be able to add automatic deserializer generation. However, I don't really like the inconsistency between basic and user defined types, and more importantly, complex types have to rely on global state to acquire references:
    
    static MaterialLibrary materials;
    registerTypeDeserializer!Model(delegate Model(Serialization s) {
        return new Model(materials.borrow(s["material"].toString), ...);
    });

That's where it really falls apart. Because I can't (without a proliferation of register deserializer functions) pass other parameters to the deserializer, I'm having difficulty avoiding ugly global factories. I've thought about eliminating the deserialize template, and requiring a deserialize function (which could accept multiple parameters) for each user defined type, but that seems like a lot of work for e.g. POD structs.

So, how can I simplify this design, and hopefully avoid tons of boilerplate deserializers, while still allowing me to inject object factories appropriately, instead of assigning them globally?